| <rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache Beam – Learn about Beam</title><link>/documentation/</link><description>Recent content in Learn about Beam on Apache Beam</description><generator>Hugo -- gohugo.io</generator><language>en</language><atom:link href="/documentation/index.xml" rel="self" type="application/rss+xml"/><item><title>Documentation: About Beam ML</title><link>/documentation/ml/about-ml/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/ml/about-ml/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="about-beam-ml">About Beam ML</h1> |
| <table> |
| <tr> |
| <td> |
| <a> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.ml.inference.html#apache_beam.ml.inference.RunInference" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p> |
| </a> |
| </td> |
| <td> |
| <a target="_blank" class="button" |
| href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/extensions/python/transforms/RunInference.html"> |
| <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="30px" |
| alt="Javadoc" /> |
| Javadoc |
| </a> |
| </td> |
| </tr> |
| </table> |
| <p>You can use Apache Beam to:</p> |
| <ul> |
| <li>Process large volumes of data, both for preprocessing and for inference.</li> |
| <li>Experiment with your data during the exploration phase of your project.</li> |
| <li>Upscale your data pipelines as part of your ML ops ecosystem in a production environment.</li> |
| <li>Run your model in production on a varying data load, both in batch and streaming.</li> |
| </ul> |
| <h2 id="aiml-workloads">AI/ML workloads</h2> |
| <p>You can use Apache Beam for data validation, data preprocessing, model validation, and model deployment and inference.</p> |
| <p><img src="/images/ml-workflows.svg" alt="Overview of AI/ML building blocks and where Apache Beam can be used"></p> |
| <ol> |
| <li>Data ingestion: Incoming new data is either stored in your file system or database, or published to a messaging queue.</li> |
| <li><strong>Data validation</strong>: After you receieve your data, check the quality of the data. For example, you might want to detect outliers and calculate standard deviations and class distributions.</li> |
| <li><strong>Data preprocessing</strong>: After you validate your data, transform the data so that it&rsquo;s ready to use to train your model.</li> |
| <li>Model training: When your data is ready, train your AI/ML model. This step is typically repeated multiple times, depending on the quality of your trained model.</li> |
| <li>Model validation: Before you deploy your model, validate its performance and accuracy.</li> |
| <li><strong>Model deployment</strong>: Deploy your model, using it to run inference on new or existing data.</li> |
| </ol> |
| <p>To keep your model up to date and performing well as your data grows and evolves, run these steps multiple times. In addition, you can apply ML ops to your project to automate the AI/ML workflows throughout the model and data lifecycle. Use orchestrators to automate this flow and to handle the transition between the different building blocks in your project.</p> |
| <h2 id="use-runinference">Use RunInference</h2> |
| <p>The RunInference API is a <code>PTransform</code> optimized for machine learning inferences that lets you efficiently use ML models in your pipelines. The API includes the following features:</p> |
| <ul> |
| <li>To efficiently feed your model, dynamically batches inputs based on pipeline throughput using Apache Beam&rsquo;s <code>BatchElements</code> transform.</li> |
| <li>To balance memory and throughput usage, determines the optimal number of models to load using a central model manager. Shares these models across threads and processes as needed to maximize throughput.</li> |
| <li>Ensures that your pipeline uses the most recently deployed version of your model with the <a href="#automatic-model-refresh">Automatic model refresh</a> feature.</li> |
| <li>Supports <a href="#use-pre-trained-models">multiple frameworks and model hubs</a>, including Tensorflow, Pytorch, Sklearn, XGBoost, Hugging Face, TensorFlow Hub, Vertex AI, TensorRT, and ONNX.</li> |
| <li>Supports arbitrary frameworks using a <a href="#use-custom-models">custom model handler</a>.</li> |
| <li>Supports <a href="#multi-model-pipelines">multi-model pipelines</a>.</li> |
| <li>Lets you use GPUs on supported runners to increase inference speed. For more information, see <a href="https://cloud.google.com/dataflow/docs/gpu">GPUs with Dataflow</a> in the Dataflow documentation.</li> |
| </ul> |
| <h3 id="support-and-limitations">Support and limitations</h3> |
| <ul> |
| <li>The RunInference API is supported in Apache Beam 2.40.0 and later versions.</li> |
| <li>Model handlers are available for PyTorch, scikit-learn, TensorFlow, Hugging Face, Vertex AI, ONNX, TensorRT, and XGBoost. You can also use a custom model handler.</li> |
| <li>The RunInference API supports batch and streaming pipelines.</li> |
| <li>The RunInference API supports both remote inference and inteference local to the runner worker.</li> |
| </ul> |
| <h3 id="batchelements-ptransform">BatchElements PTransform</h3> |
| <p>To take advantage of the optimizations of vectorized inference that many models implement, the <code>BatchElements</code> transform is used as an intermediate step before making the prediction for the model. This transform batches elements together. The batched elements are then applied with a transformation for the particular framework of RunInference. For example, for numpy <code>ndarrays</code>, we call <code>numpy.stack()</code>, and for torch <code>Tensor</code> elements, we call <code>torch.stack()</code>.</p> |
| <p>To customize the settings for <code>beam.BatchElements</code>, in <code>ModelHandler</code>, override the <code>batch_elements_kwargs</code> function. For example, use <code>min_batch_size</code> to set the lowest number of elements per batch or <code>max_batch_size</code> to set the highest number of elements per batch.</p> |
| <p>For more information, see the <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.BatchElements"><code>BatchElements</code> transform documentation</a>.</p> |
| <h3 id="shared-helper-class">Shared helper class</h3> |
| <p>Using the <code>Shared</code> class within the RunInference implementation makes it possible to load the model only once per process and share it with all DoFn instances created in that process. This feature reduces memory consumption and model loading time. For more information, see the |
| <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/shared.py#L20"><code>Shared</code> class documentation</a>.</p> |
| <h3 id="modify-a-python-pipeline-to-use-an-ml-model">Modify a Python pipeline to use an ML model</h3> |
| <p>To use the RunInference transform, add the following code to your pipeline:</p> |
| <pre tabindex="0"><code>from apache_beam.ml.inference.base import RunInference |
| with pipeline as p: |
| predictions = ( p | &#39;Read&#39; &gt;&gt; beam.ReadFromSource(&#39;a_source&#39;) |
| | &#39;RunInference&#39; &gt;&gt; RunInference(&lt;model_handler&gt;) |
| </code></pre><p>Replace <code>model_handler</code> with the model handler setup code.</p> |
| <p>To import models, you need to configure a <code>ModelHandler</code> object that wraps the underlying model. Which model handler you import depends on the framework and type of data structure that contains the inputs. The <code>ModelHandler</code> object also allows you to set environment variables needed for inference using the <code>env_vars</code> keyword argument. The following examples show some model handlers that you might want to import.</p> |
| <pre tabindex="0"><code>from apache_beam.ml.inference.sklearn_inference import SklearnModelHandlerNumpy |
| from apache_beam.ml.inference.sklearn_inference import SklearnModelHandlerPandas |
| from apache_beam.ml.inference.pytorch_inference import PytorchModelHandlerTensor |
| from apache_beam.ml.inference.pytorch_inference import PytorchModelHandlerKeyedTensor |
| from tfx_bsl.public.beam.run_inference import CreateModelHandler |
| </code></pre><h3 id="use-pre-trained-models">Use pre-trained models</h3> |
| <p>The section provides requirements for using pre-trained models with PyTorch, Scikit-learn, and Tensorflow.</p> |
| <h4 id="pytorch">PyTorch</h4> |
| <p>You need to provide a path to a file that contains the model&rsquo;s saved weights. This path must be accessible by the pipeline. To use pre-trained models with the RunInference API and the PyTorch framework, complete the following steps:</p> |
| <ol> |
| <li>Download the pre-trained weights and host them in a location that the pipeline can access.</li> |
| <li>Pass the path of the model weights to the PyTorch <code>ModelHandler</code> by using the following code: <code>state_dict_path=&lt;path_to_weights&gt;</code>.</li> |
| </ol> |
| <p>See <a href="https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_pytorch.ipynb">this notebook</a> |
| that illustrates running PyTorch models with Apache Beam.</p> |
| <h4 id="scikit-learn">Scikit-learn</h4> |
| <p>You need to provide a path to a file that contains the pickled Scikit-learn model. This path must be accessible by the pipeline. To use pre-trained models with the RunInference API and the Scikit-learn framework, complete the following steps:</p> |
| <ol> |
| <li>Download the pickled model class and host it in a location that the pipeline can access.</li> |
| <li>Pass the path of the model to the Sklearn <code>ModelHandler</code> by using the following code: |
| <code>model_uri=&lt;path_to_pickled_file&gt;</code> and <code>model_file_type: &lt;ModelFileType&gt;</code>, where you can specify |
| <code>ModelFileType.PICKLE</code> or <code>ModelFileType.JOBLIB</code>, depending on how the model was serialized.</li> |
| </ol> |
| <p>See <a href="https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_sklearn.ipynb">this notebook</a> |
| that illustrates running Scikit-learn models with Apache Beam.</p> |
| <h4 id="tensorflow">TensorFlow</h4> |
| <p>To use TensorFlow with the RunInference API, you have two options:</p> |
| <ol> |
| <li>Use the built-in TensorFlow Model Handlers in Apache Beam SDK - <code>TFModelHandlerNumpy</code> and <code>TFModelHandlerTensor</code>. |
| <ul> |
| <li>Depending on the type of input for your model, use <code>TFModelHandlerNumpy</code> for <code>numpy</code> input and <code>TFModelHandlerTensor</code> for <code>tf.Tensor</code> input respectively.</li> |
| <li>Use tensorflow 2.7 or later.</li> |
| <li>Pass the path of the model to the TensorFlow <code>ModelHandler</code> by using <code>model_uri=&lt;path_to_trained_model&gt;</code>.</li> |
| <li>Alternatively, you can pass the path to saved weights of the trained model, a function to build the model using <code>create_model_fn=&lt;function&gt;</code>, and set the <code>model_type=ModelType.SAVED_WEIGHTS</code>. |
| See <a href="https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_tensorflow.ipynb">this notebook</a> that illustrates running Tensorflow models with Built-in model handlers.</li> |
| </ul> |
| </li> |
| <li>Using <code>tfx_bsl</code>. |
| <ul> |
| <li>Use this approach if your model input is of type <code>tf.Example</code>.</li> |
| <li>Use <code>tfx_bsl</code> version 1.10.0 or later.</li> |
| <li>Create a model handler using <code>tfx_bsl.public.beam.run_inference.CreateModelHandler()</code>.</li> |
| <li>Use the model handler with the <a href="/releases/pydoc/current/apache_beam.ml.inference.base.html"><code>apache_beam.ml.inference.base.RunInference</code></a> transform. |
| See <a href="https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_tensorflow.ipynb">this notebook</a> |
| that illustrates running TensorFlow models with Apache Beam and tfx-bsl.</li> |
| </ul> |
| </li> |
| </ol> |
| <h3 id="use-custom-models">Use custom models</h3> |
| <p>If you would like to use a model that isn&rsquo;t specified by one of the supported frameworks, the RunInference API is designed flexibly to allow you to use any custom machine learning models. |
| You only need to create your own <code>ModelHandler</code> or <code>KeyedModelHandler</code> with logic to load your model and use it to run the inference.</p> |
| <p>A simple example can be found in <a href="https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_custom_inference.ipynb">this notebook</a>. |
| The <code>load_model</code> method shows how to load the model using a popular <code>spaCy</code> package while <code>run_inference</code> shows how to run the inference on a batch of examples.</p> |
| <h3 id="runinference-patterns">RunInference Patterns</h3> |
| <p>This section suggests patterns and best practices that you can use to make your inference pipelines simpler, |
| more robust, and more efficient.</p> |
| <h4 id="use-a-keyed-modelhandler-object">Use a keyed ModelHandler object</h4> |
| <p>If a key is attached to the examples, wrap <code>KeyedModelHandler</code> around the <code>ModelHandler</code> object:</p> |
| <pre tabindex="0"><code>from apache_beam.ml.inference.base import KeyedModelHandler |
| keyed_model_handler = KeyedModelHandler(PytorchModelHandlerTensor(...)) |
| with pipeline as p: |
| data = p | beam.Create([ |
| (&#39;img1&#39;, torch.tensor([[1,2,3],[4,5,6],...])), |
| (&#39;img2&#39;, torch.tensor([[1,2,3],[4,5,6],...])), |
| (&#39;img3&#39;, torch.tensor([[1,2,3],[4,5,6],...])), |
| ]) |
| predictions = data | RunInference(keyed_model_handler) |
| </code></pre><p>If you are unsure if your data is keyed, you can use <code>MaybeKeyedModelHandler</code>.</p> |
| <p>You can also use a <code>KeyedModelHandler</code> to load several different models based on their associated key. |
| The following example loads a model by using <code>config1</code>. That model is used for inference for all examples associated |
| with <code>key1</code>. It loads a second model by using <code>config2</code>. That model is used for all examples associated with <code>key2</code> and <code>key3</code>.</p> |
| <pre tabindex="0"><code>from apache_beam.ml.inference.base import KeyedModelHandler |
| keyed_model_handler = KeyedModelHandler([ |
| KeyModelMapping([&#39;key1&#39;], PytorchModelHandlerTensor(&lt;config1&gt;)), |
| KeyModelMapping([&#39;key2&#39;, &#39;key3&#39;], PytorchModelHandlerTensor(&lt;config2&gt;)) |
| ]) |
| with pipeline as p: |
| data = p | beam.Create([ |
| (&#39;key1&#39;, torch.tensor([[1,2,3],[4,5,6],...])), |
| (&#39;key2&#39;, torch.tensor([[1,2,3],[4,5,6],...])), |
| (&#39;key3&#39;, torch.tensor([[1,2,3],[4,5,6],...])), |
| ]) |
| predictions = data | RunInference(keyed_model_handler) |
| </code></pre><p>For a more detailed example, see the notebook |
| <a href="https://colab.sandbox.google.com/github/apache/beam/blob/master/examples/notebooks/beam-ml/per_key_models.ipynb">Run ML inference with multiple differently-trained models</a>.</p> |
| <p>Loading multiple models at the same times increases the risk of out of memory errors (OOMs). By default, <code>KeyedModelHandler</code> doesn&rsquo;t |
| limit the number of models loaded into memory at the same time. If the models don&rsquo;t all fit into memory, |
| your pipeline might fail with an out of memory error. To avoid this issue, use the <code>max_models_per_worker_hint</code> parameter |
| to set the maximum number of models that can be loaded into memory at the same time.</p> |
| <p>The following example loads at most two models per SDK worker process at a time. It unloads models that aren&rsquo;t |
| currently in use.</p> |
| <pre tabindex="0"><code>mhs = [ |
| KeyModelMapping([&#39;key1&#39;], PytorchModelHandlerTensor(&lt;config1&gt;)), |
| KeyModelMapping([&#39;key2&#39;, &#39;key3&#39;], PytorchModelHandlerTensor(&lt;config2&gt;)), |
| KeyModelMapping([&#39;key4&#39;], PytorchModelHandlerTensor(&lt;config3&gt;)), |
| KeyModelMapping([&#39;key5&#39;, &#39;key6&#39;, &#39;key7&#39;], PytorchModelHandlerTensor(&lt;config4&gt;)), |
| ] |
| keyed_model_handler = KeyedModelHandler(mhs, max_models_per_worker_hint=2) |
| </code></pre><p>Runners that have multiple SDK worker processes on a given machine load at most |
| <code>max_models_per_worker_hint*&lt;num worker processes&gt;</code> models onto the machine.</p> |
| <p>Leave enough space for the models and any additional memory needs from other transforms. |
| Because the memory might not be released immediately after a model is offloaded, |
| leaving an additional buffer is recommended.</p> |
| <p><strong>Note</strong>: Having many models but a small <code>max_models_per_worker_hint</code> can cause <em>memory thrashing</em>, where |
| a large amount of execution time is used to swap models in and out of memory. To reduce the likelihood and impact |
| of memory thrashing, if you&rsquo;re using a distributed runner, insert a |
| <a href="https://beam.apache.org/documentation/transforms/python/aggregation/groupbykey/"><code>GroupByKey</code></a> transform before your |
| inference step. The <code>GroupByKey</code> transform reduces thrashing by ensuring that elements with the same key and model are |
| collocated on the same worker.</p> |
| <p>For more information, see <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.ml.inference.base.html#apache_beam.ml.inference.base.KeyedModelHandler"><code>KeyedModelHander</code></a>.</p> |
| <h4 id="use-the-predictionresult-object">Use the PredictionResult object</h4> |
| <p>When doing a prediction in Apache Beam, the output <code>PCollection</code> includes both the keys of the input examples and the inferences. Including both these items in the output allows you to find the input that determined the predictions.</p> |
| <p>The <code>PredictionResult</code> object is a <code>NamedTuple</code> that contains both the input and the inferences, named <code>example</code> and <code>inference</code>, respectively. When keys are passed with the input data to the RunInference transform, the output <code>PCollection</code> returns a <code>Tuple[str, PredictionResult]</code>, which is the key and the <code>PredictionResult</code> object. Your pipeline interacts with a <code>PredictionResult</code> object in steps after the RunInference transform.</p> |
| <pre tabindex="0"><code>class PostProcessor(beam.DoFn): |
| def process(self, element: Tuple[str, PredictionResult]): |
| key, prediction_result = element |
| inputs = prediction_result.example |
| predictions = prediction_result.inference |
| # Post-processing logic |
| result = ... |
| yield (key, result) |
| with pipeline as p: |
| output = ( |
| p | &#39;Read&#39; &gt;&gt; beam.ReadFromSource(&#39;a_source&#39;) |
| | &#39;PyTorchRunInference&#39; &gt;&gt; RunInference(&lt;keyed_model_handler&gt;) |
| | &#39;ProcessOutput&#39; &gt;&gt; beam.ParDo(PostProcessor())) |
| </code></pre><p>If you need to use this object explicitly, include the following line in your pipeline to import the object:</p> |
| <pre tabindex="0"><code>from apache_beam.ml.inference.base import PredictionResult |
| </code></pre><p>For more information, see the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/ml/inference/base.py#L65"><code>PredictionResult</code> documentation</a>.</p> |
| <h4 id="automatic-model-refresh">Automatic model refresh</h4> |
| <p>To automatically update the model being used with the RunInference <code>PTransform</code> without stopping the pipeline, pass a <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.ml.inference.base.html#apache_beam.ml.inference.base.ModelMetadata"><code>ModelMetadata</code></a> side input <code>PCollection</code> to the RunInference input parameter <code>model_metadata_pcoll</code>.</p> |
| <p><code>ModelMetdata</code> is a <code>NamedTuple</code> containing:</p> |
| <ul> |
| <li> |
| <p><code>model_id</code>: Unique identifier for the model. This can be a file path or a URL where the model can be accessed. It is used to load the model for inference. The URL or file path must be in the compatible format so that the respective <code>ModelHandlers</code> can load the models without errors.</p> |
| <p>For example, <code>PyTorchModelHandler</code> initially loads a model using weights and a model class. If you pass in weights from a different model class when you update the model using side inputs, the model doesn&rsquo;t load properly, because it expects the weights from the original model class.</p> |
| </li> |
| <li> |
| <p><code>model_name</code>: Human-readable name for the model. You can use this name to identify the model in the metrics generated by the RunInference transform.</p> |
| </li> |
| </ul> |
| <p>Use cases:</p> |
| <ul> |
| <li>Use <code>WatchFilePattern</code> as side input to the RunInference <code>PTransform</code> to automatically update the ML model. For more information, see <a href="https://beam.apache.org/documentation/ml/side-input-updates">Use <code>WatchFilePattern</code> as side input to auto-update ML models in RunInference</a>.</li> |
| </ul> |
| <p>The side input <code>PCollection</code> must follow the <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.pvalue.html?highlight=assingleton#apache_beam.pvalue.AsSingleton"><code>AsSingleton</code></a> view to avoid errors.</p> |
| <p><strong>Note</strong>: If the main <code>PCollection</code> emits inputs and a side input has yet to receive inputs, the main <code>PCollection</code> is buffered until there is |
| an update to the side input. This could happen with global windowed side inputs with data driven triggers, such as <code>AfterCount</code> and <code>AfterProcessingTime</code>. Until the side input is updated, emit the default or initial model ID that is used to pass the respective <code>ModelHandler</code> as a side input.</p> |
| <h4 id="preprocess-and-postprocess-your-records">Preprocess and postprocess your records</h4> |
| <p>With RunInference, you can add preprocessing and postprocessing operations to your transform. |
| To apply preprocessing operations, use <code>with_preprocess_fn</code> on your model handler:</p> |
| <pre tabindex="0"><code>inference = pcoll | RunInference(model_handler.with_preprocess_fn(lambda x : do_something(x))) |
| </code></pre><p>To apply postprocessing operations, use <code>with_postprocess_fn</code> on your model handler:</p> |
| <pre tabindex="0"><code>inference = pcoll | RunInference(model_handler.with_postprocess_fn(lambda x : do_something_to_result(x))) |
| </code></pre><p>You can also chain multiple pre- and postprocessing operations:</p> |
| <pre tabindex="0"><code>inference = pcoll | RunInference( |
| model_handler.with_preprocess_fn( |
| lambda x : do_something(x) |
| ).with_preprocess_fn( |
| lambda x : do_something_else(x) |
| ).with_postprocess_fn( |
| lambda x : do_something_after_inference(x) |
| ).with_postprocess_fn( |
| lambda x : do_something_else_after_inference(x) |
| )) |
| </code></pre><p>The preprocessing function is run before batching and inference. This function maps your input <code>PCollection</code> |
| to the base input type of the model handler. If you apply multiple preprocessing functions, they run on your original |
| <code>PCollection</code> in the order of last applied to first applied.</p> |
| <p>The postprocessing function runs after inference. This function maps the output type of the base model handler |
| to your desired output type. If you apply multiple postprocessing functions, they run on your original |
| inference result in the order of first applied to last applied.</p> |
| <h4 id="handle-errors">Handle errors</h4> |
| <p>To handle errors robustly while using RunInference, you can use a <em>dead-letter queue</em>. The dead-letter queue outputs failed records into a separate <code>PCollection</code> for further processing. |
| This <code>PCollection</code> can then be analyzed and sent to a storage system, where it can be reviewed and resubmitted to the pipeline, or discarded. |
| RunInference has built-in support for dead-letter queues. You can use a dead-letter queue by applying <code>with_exception_handling</code> to your RunInference transform:</p> |
| <pre tabindex="0"><code>main, other = pcoll | RunInference(model_handler).with_exception_handling() |
| other.failed_inferences | beam.Map(print) # insert logic to handle failed records here |
| </code></pre><p>You can also apply this pattern to RunInference transforms with associated pre- and postprocessing operations:</p> |
| <pre tabindex="0"><code>main, other = pcoll | RunInference(model_handler.with_preprocess_fn(f1).with_postprocess_fn(f2)).with_exception_handling() |
| other.failed_preprocessing[0] | beam.Map(print) # handles failed preprocess operations, indexed in the order in which they were applied |
| other.failed_inferences | beam.Map(print) # handles failed inferences |
| other.failed_postprocessing[0] | beam.Map(print) # handles failed postprocess operations, indexed in the order in which they were applied |
| </code></pre><h4 id="run-inference-from-a-java-pipeline">Run inference from a Java pipeline</h4> |
| <p>The RunInference API is available with the Beam Java SDK versions 2.41.0 and later through Apache Beam&rsquo;s <a href="/documentation/programming-guide/#multi-language-pipelines">Multi-language Pipelines framework</a>. For information about the Java wrapper transform, see <a href="https://github.com/apache/beam/blob/master/sdks/java/extensions/python/src/main/java/org/apache/beam/sdk/extensions/python/transforms/RunInference.java">RunInference.java</a>. To try it out, see the <a href="https://github.com/apache/beam/tree/master/examples/multi-language">Java Sklearn Mnist Classification example</a>. Additionally, see <a href="https://beam.apache.org/documentation/ml/multi-language-inference/">Using RunInference from Java SDK</a> for an example of a composite Python transform that uses the RunInference API along with preprocessing and postprocessing from a Beam Java SDK pipeline.</p> |
| <h2 id="custom-inference">Custom Inference</h2> |
| <p>The RunInference API doesn&rsquo;t currently support making remote inference calls using, for example, the Natural Language API or the Cloud Vision API. Therefore, in order to use these remote APIs with Apache Beam, you need to write custom inference calls. The <a href="https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/custom_remote_inference.ipynb">Remote inference in Apache Beam notebook</a> shows how to implement a custom remote inference call using <code>beam.DoFn</code>. When you implement a remote inference for real life projects, consider the following factors:</p> |
| <ul> |
| <li> |
| <p>API quotas and the heavy load you might incur on your external API. To optimize the calls to an external API, you can confgure <code>PipelineOptions</code> to limit the parallel calls to the external remote API.</p> |
| </li> |
| <li> |
| <p>Be prepared to encounter, identify, and handle failure as gracefully as possible. Use techniques like exponential backoff and dead-letter queues (unprocessed messages queues).</p> |
| </li> |
| <li> |
| <p>When running inference with an external API, batch your input together to allow for more efficient execution.</p> |
| </li> |
| <li> |
| <p>Consider monitoring and measuring the performance of a pipeline when deploying, because monitoring can provide insight into the status and health of the application.</p> |
| </li> |
| </ul> |
| <h2 id="multi-model-pipelines">Multi-model pipelines</h2> |
| <p>Use the RunInference transform to add multiple inference models to your pipeline. Multi-model pipelines can be useful for A/B testing or for building out cascade models made up of models that perform tokenization, sentence segmentation, part-of-speech tagging, named entity extraction, language detection, coreference resolution, and more. For more information, see <a href="https://beam.apache.org/documentation/ml/multi-model-pipelines/">Multi-model pipelines</a>.</p> |
| <h3 id="ab-pattern">A/B Pattern</h3> |
| <pre tabindex="0"><code>with pipeline as p: |
| data = p | &#39;Read&#39; &gt;&gt; beam.ReadFromSource(&#39;a_source&#39;) |
| model_a_predictions = data | RunInference(&lt;model_handler_A&gt;) |
| model_b_predictions = data | RunInference(&lt;model_handler_B&gt;) |
| </code></pre><p>Where <code>model_handler_A</code> and <code>model_handler_B</code> are the model handler setup code.</p> |
| <h3 id="cascade-pattern">Cascade Pattern</h3> |
| <pre tabindex="0"><code>with pipeline as p: |
| data = p | &#39;Read&#39; &gt;&gt; beam.ReadFromSource(&#39;a_source&#39;) |
| model_a_predictions = data | RunInference(&lt;model_handler_A&gt;) |
| model_b_predictions = model_a_predictions | beam.Map(some_post_processing) | RunInference(&lt;model_handler_B&gt;) |
| </code></pre><p>Where <code>model_handler_A</code> and <code>model_handler_B</code> are the model handler setup code.</p> |
| <h3 id="use-resource-hints-for-different-model-requirements">Use Resource Hints for Different Model Requirements</h3> |
| <p>When using multiple models in a single pipeline, different models may have different memory or worker SKU requirements. |
| Resource hints allow you to provide information to a runner about the compute resource requirements for each step in your |
| pipeline.</p> |
| <p>For example, the following snippet extends the previous cascade pattern with hints for each RunInference call |
| to specify RAM and hardware accelerator requirements:</p> |
| <pre tabindex="0"><code>with pipeline as p: |
| data = p | &#39;Read&#39; &gt;&gt; beam.ReadFromSource(&#39;a_source&#39;) |
| model_a_predictions = data | RunInference(&lt;model_handler_A&gt;).with_resource_hints(min_ram=&#34;20GB&#34;) |
| model_b_predictions = model_a_predictions |
| | beam.Map(some_post_processing) |
| | RunInference(&lt;model_handler_B&gt;).with_resource_hints( |
| min_ram=&#34;4GB&#34;, |
| accelerator=&#34;type:nvidia-tesla-k80;count:1;install-nvidia-driver&#34;) |
| </code></pre><p>For more information on resource hints, see <a href="/documentation/runtime/resource-hints/">Resource hints</a>.</p> |
| <h2 id="model-validation">Model validation</h2> |
| <p>Model validation allows you to benchmark your model’s performance against a previously unseen dataset. You can extract chosen metrics, create visualizations, log metadata, and compare the performance of different models with the end goal of validating whether your model is ready to deploy. Beam provides support for running model evaluation on a TensorFlow model directly inside your pipeline.</p> |
| <p>The <a href="/documentation/ml/model-evaluation">ML model evaluation</a> page shows how to integrate model evaluation as part of your pipeline by using <a href="https://www.tensorflow.org/tfx/guide/tfma">TensorFlow Model Analysis (TFMA)</a>.</p> |
| <h2 id="troubleshooting">Troubleshooting</h2> |
| <p>If you run into problems with your pipeline or job, this section lists issues that you might encounter and provides suggestions for how to fix them.</p> |
| <h3 id="unable-to-batch-tensor-elements">Unable to batch tensor elements</h3> |
| <p>RunInference uses dynamic batching. However, the RunInference API cannot batch tensor elements of different sizes, so samples passed to the <code>RunInference</code> transform must be the same dimension or length. If you provide images of different sizes or word embeddings of different lengths, the following error might occur:</p> |
| <p><code>File &quot;/beam/sdks/python/apache_beam/ml/inference/pytorch_inference.py&quot;, line 232, in run_inference batched_tensors = torch.stack(key_to_tensor_list[key]) RuntimeError: stack expects each tensor to be equal size, but got [12] at entry 0 and [10] at entry 1 [while running 'PyTorchRunInference/ParDo(_RunInferenceDoFn)']</code></p> |
| <p>To avoid this issue, either use elements of the same size, or disable batching.</p> |
| <p><strong>Option 1: Use elements of the same size</strong></p> |
| <p>Use elements of the same size or resize the inputs. For computer vision applications, resize image inputs so that they have the same dimensions. For natural language processing (NLP) applications that have text of varying length, resize the text or word embeddings to make them the same length. When working with texts of varying length, resizing might not be possible. In this scenario, you could disable batching (see option 2).</p> |
| <p><strong>Option 2: Disable batching</strong></p> |
| <p>Disable batching by overriding the <code>batch_elements_kwargs</code> function in your ModelHandler and setting the maximum batch size (<code>max_batch_size</code>) to one: <code>max_batch_size=1</code>. For more information, see |
| <a href="/documentation/ml/about-ml/#batchelements-ptransform">BatchElements PTransforms</a>. For an example, see our <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/pytorch_language_modeling.py">language modeling example</a>.</p> |
| <h2 id="related-links">Related links</h2> |
| <ul> |
| <li><a href="/documentation/transforms/python/elementwise/runinference">RunInference transforms</a></li> |
| <li><a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference">RunInference API pipeline examples</a></li> |
| <li><a href="https://colab.sandbox.google.com/github/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_basic.ipynb">RunInference public codelab</a></li> |
| <li><a href="https://github.com/apache/beam/tree/master/examples/notebooks/beam-ml">RunInference notebooks</a></li> |
| <li><a href="http://s.apache.org/beam-community-metrics/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1">RunInference benchmarks</a></li> |
| </ul> |
| <table> |
| <tr> |
| <td> |
| <a> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.ml.inference.html#apache_beam.ml.inference.RunInference" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p> |
| </a> |
| </td> |
| <td> |
| <a target="_blank" class="button" |
| href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/extensions/python/transforms/RunInference.html"> |
| <img src="https://beam.apache.org/images/logos/sdks/java.png" width="20px" height="30px" |
| alt="Javadoc" /> |
| Javadoc |
| </a> |
| </td> |
| </tr> |
| </table></description></item><item><title>Documentation: AI Platform integration patterns</title><link>/documentation/patterns/ai-platform/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/patterns/ai-platform/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="ai-platform-integration-patterns">AI Platform integration patterns</h1> |
| <p>This page describes common patterns in pipelines with Google Cloud AI Platform transforms.</p> |
| <nav class="language-switcher"> |
| <strong>Adapt for:</strong> |
| <ul> |
| <li data-value="java" class="active">Java SDK</li> |
| <li data-value="py">Python SDK</li> |
| </ul> |
| </nav> |
| <h2 id="analysing-the-structure-and-meaning-of-text">Analysing the structure and meaning of text</h2> |
| <p>This section shows how to use <a href="https://cloud.google.com/natural-language">Google Cloud Natural Language API</a> to perform text analysis.</p> |
| <p>Beam provides a PTransform called <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.ml.gcp.naturallanguageml.html#apache_beam.ml.gcp.naturallanguageml.AnnotateText">AnnotateText</a>. The transform takes a PCollection of type <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.ml.gcp.naturallanguageml.html#apache_beam.ml.gcp.naturallanguageml.Document">Document</a>. Each Document object contains various information about text. This includes the content, whether it is a plain text or HTML, an optional language hint and other settings. |
| <code>AnnotateText</code> produces response object of type <code>AnnotateTextResponse</code> returned from the API. <code>AnnotateTextResponse</code> is a protobuf message which contains a lot of attributes, some of which are complex structures.</p> |
| <p>Here is an example of a pipeline that creates in-memory PCollection of strings, changes each string to Document object and invokes Natural Language API. Then, for each response object, a function is called to extract certain results of analysis.</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">features</span> <span class="o">=</span> <span class="n">nlp</span><span class="o">.</span><span class="n">types</span><span class="o">.</span><span class="n">AnnotateTextRequest</span><span class="o">.</span><span class="n">Features</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">extract_entities</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">extract_document_sentiment</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">extract_entity_sentiment</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">extract_syntax</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">responses</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;My experience so far has been fantastic! &#39;</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;I</span><span class="se">\&#39;</span><span class="s1">d really recommend this product.&#39;</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">])</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">nlp</span><span class="o">.</span><span class="n">Document</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="s1">&#39;PLAIN_TEXT&#39;</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">nlp</span><span class="o">.</span><span class="n">AnnotateText</span><span class="p">(</span><span class="n">features</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">_</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">responses</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="n">extract_sentiments</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Parse sentiments to JSON&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Write sentiments&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">WriteToText</span><span class="p">(</span><span class="s1">&#39;sentiments.txt&#39;</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">_</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">responses</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="n">extract_entities</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Parse entities to JSON&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Write entities&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">WriteToText</span><span class="p">(</span><span class="s1">&#39;entities.txt&#39;</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">_</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">responses</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="n">analyze_dependency_tree</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Parse adjacency list to JSON&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Write adjacency list&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">WriteToText</span><span class="p">(</span><span class="s1">&#39;adjancency_list.txt&#39;</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">AnnotateTextRequest</span><span class="o">.</span><span class="na">Features</span> <span class="n">features</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">AnnotateTextRequest</span><span class="o">.</span><span class="na">Features</span><span class="o">.</span><span class="na">newBuilder</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">setExtractEntities</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">setExtractDocumentSentiment</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">setExtractEntitySentiment</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">setExtractSyntax</span><span class="o">(</span><span class="kc">true</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">build</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">AnnotateText</span> <span class="n">annotateText</span> <span class="o">=</span> <span class="n">AnnotateText</span><span class="o">.</span><span class="na">newBuilder</span><span class="o">().</span><span class="na">setFeatures</span><span class="o">(</span><span class="n">features</span><span class="o">).</span><span class="na">build</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">AnnotateTextResponse</span><span class="o">&gt;</span> <span class="n">responses</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Create</span><span class="o">.</span><span class="na">of</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;My experience so far has been fantastic, &#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">+</span> <span class="s">&#34;I\&#39;d really recommend this product.&#34;</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">TypeDescriptor</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">Document</span><span class="o">.</span><span class="na">class</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">via</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">(</span><span class="n">SerializableFunction</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Document</span><span class="o">&gt;)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">input</span> <span class="o">-&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Document</span><span class="o">.</span><span class="na">newBuilder</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">setContent</span><span class="o">(</span><span class="n">input</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">setType</span><span class="o">(</span><span class="n">Document</span><span class="o">.</span><span class="na">Type</span><span class="o">.</span><span class="na">PLAIN_TEXT</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">build</span><span class="o">()))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">annotateText</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">responses</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">MapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">TypeDescriptor</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">TextSentiments</span><span class="o">.</span><span class="na">class</span><span class="o">)).</span><span class="na">via</span><span class="o">(</span><span class="n">extractSentiments</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">via</span><span class="o">((</span><span class="n">SerializableFunction</span><span class="o">&lt;</span><span class="n">TextSentiments</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;)</span> <span class="n">TextSentiments</span><span class="o">::</span><span class="n">toJson</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">TextIO</span><span class="o">.</span><span class="na">write</span><span class="o">().</span><span class="na">to</span><span class="o">(</span><span class="s">&#34;sentiments.txt&#34;</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">responses</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">maps</span><span class="o">(</span><span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">(),</span> <span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">()))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">via</span><span class="o">(</span><span class="n">extractEntities</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">MapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">()).</span><span class="na">via</span><span class="o">(</span><span class="n">mapEntitiesToJson</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">TextIO</span><span class="o">.</span><span class="na">write</span><span class="o">().</span><span class="na">to</span><span class="o">(</span><span class="s">&#34;entities.txt&#34;</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">responses</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">lists</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">maps</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">(),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">lists</span><span class="o">(</span><span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">()))))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">via</span><span class="o">(</span><span class="n">analyzeDependencyTree</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">MapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">()).</span><span class="na">via</span><span class="o">(</span><span class="n">mapDependencyTreesToJson</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">TextIO</span><span class="o">.</span><span class="na">write</span><span class="o">().</span><span class="na">to</span><span class="o">(</span><span class="s">&#34;adjacency_list.txt&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="extracting-sentiments">Extracting sentiments</h3> |
| <p>This is a part of response object returned from the API. Sentence-level sentiments can be found in <code>sentences</code> attribute. <code>sentences</code> behaves like a standard Python sequence, therefore all core language features (like iteration or slicing) will work. Overall sentiment can be found in <code>document_sentiment</code> attribute.</p> |
| <pre tabindex="0"><code>sentences { |
| text { |
| content: &#34;My experience so far has been fantastic!&#34; |
| } |
| sentiment { |
| magnitude: 0.8999999761581421 |
| score: 0.8999999761581421 |
| } |
| } |
| sentences { |
| text { |
| content: &#34;I\&#39;d really recommend this product.&#34; |
| begin_offset: 41 |
| } |
| sentiment { |
| magnitude: 0.8999999761581421 |
| score: 0.8999999761581421 |
| } |
| } |
| ...many lines omitted |
| document_sentiment { |
| magnitude: 1.899999976158142 |
| score: 0.8999999761581421 |
| } |
| </code></pre><p>The function for extracting information about sentence-level and document-level sentiments is shown in the next code snippet.</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">return</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;sentences&#39;</span><span class="p">:</span> <span class="p">[{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">sentence</span><span class="o">.</span><span class="n">text</span><span class="o">.</span><span class="n">content</span><span class="p">:</span> <span class="n">sentence</span><span class="o">.</span><span class="n">sentiment</span><span class="o">.</span><span class="n">score</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> <span class="k">for</span> <span class="n">sentence</span> <span class="ow">in</span> <span class="n">response</span><span class="o">.</span><span class="n">sentences</span><span class="p">],</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;document_sentiment&#39;</span><span class="p">:</span> <span class="n">response</span><span class="o">.</span><span class="n">document_sentiment</span><span class="o">.</span><span class="n">score</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">extractSentiments</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"><span class="o">(</span><span class="n">SerializableFunction</span><span class="o">&lt;</span><span class="n">AnnotateTextResponse</span><span class="o">,</span> <span class="n">TextSentiments</span><span class="o">&gt;)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">annotateTextResponse</span> <span class="o">-&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TextSentiments</span> <span class="n">sentiments</span> <span class="o">=</span> <span class="k">new</span> <span class="n">TextSentiments</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">sentiments</span><span class="o">.</span><span class="na">setDocumentSentiment</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">annotateTextResponse</span><span class="o">.</span><span class="na">getDocumentSentiment</span><span class="o">().</span><span class="na">getMagnitude</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Map</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Float</span><span class="o">&gt;</span> <span class="n">sentenceSentimentsMap</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">annotateTextResponse</span><span class="o">.</span><span class="na">getSentencesList</span><span class="o">().</span><span class="na">stream</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">collect</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Collectors</span><span class="o">.</span><span class="na">toMap</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">(</span><span class="n">Sentence</span> <span class="n">s</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">s</span><span class="o">.</span><span class="na">getText</span><span class="o">().</span><span class="na">getContent</span><span class="o">(),</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">(</span><span class="n">Sentence</span> <span class="n">s</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">s</span><span class="o">.</span><span class="na">getSentiment</span><span class="o">().</span><span class="na">getMagnitude</span><span class="o">()));</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">sentiments</span><span class="o">.</span><span class="na">setSentenceSentiments</span><span class="o">(</span><span class="n">sentenceSentimentsMap</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">sentiments</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">};</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>The snippet loops over <code>sentences</code> and, for each sentence, extracts the sentiment score.</p> |
| <p>The output is:</p> |
| <pre tabindex="0"><code>{&#34;sentences&#34;: [{&#34;My experience so far has been fantastic!&#34;: 0.8999999761581421}, {&#34;I&#39;d really recommend this product.&#34;: 0.8999999761581421}], &#34;document_sentiment&#34;: 0.8999999761581421} |
| </code></pre><h3 id="extracting-entities">Extracting entities</h3> |
| <p>The next function inspects the response for entities and returns the names and the types of those entities.</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">return</span> <span class="p">[{</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;name&#39;</span><span class="p">:</span> <span class="n">entity</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;type&#39;</span><span class="p">:</span> <span class="n">nlp</span><span class="o">.</span><span class="n">enums</span><span class="o">.</span><span class="n">Entity</span><span class="o">.</span><span class="n">Type</span><span class="p">(</span><span class="n">entity</span><span class="o">.</span><span class="n">type</span><span class="p">)</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> <span class="k">for</span> <span class="n">entity</span> <span class="ow">in</span> <span class="n">response</span><span class="o">.</span><span class="n">entities</span><span class="p">]</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">extractEntities</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"><span class="o">(</span><span class="n">SerializableFunction</span><span class="o">&lt;</span><span class="n">AnnotateTextResponse</span><span class="o">,</span> <span class="n">Map</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;&gt;)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">annotateTextResponse</span> <span class="o">-&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">annotateTextResponse</span><span class="o">.</span><span class="na">getEntitiesList</span><span class="o">().</span><span class="na">stream</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">collect</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Collectors</span><span class="o">.</span><span class="na">toMap</span><span class="o">(</span><span class="n">Entity</span><span class="o">::</span><span class="n">getName</span><span class="o">,</span> <span class="o">(</span><span class="n">Entity</span> <span class="n">e</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">e</span><span class="o">.</span><span class="na">getType</span><span class="o">().</span><span class="na">toString</span><span class="o">()));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Entities can be found in <code>entities</code> attribute. Just like before, <code>entities</code> is a sequence, that&rsquo;s why list comprehension is a viable choice. The most tricky part is interpreting the types of entities. Natural Language API defines entity types as enum. In a response object, entity types are returned as integers. That&rsquo;s why a user has to instantiate <code>naturallanguageml.enums.Entity.Type</code> to access a human-readable name.</p> |
| <p>The output is:</p> |
| <pre tabindex="0"><code>[{&#34;name&#34;: &#34;experience&#34;, &#34;type&#34;: &#34;OTHER&#34;}, {&#34;name&#34;: &#34;product&#34;, &#34;type&#34;: &#34;CONSUMER_GOOD&#34;}] |
| </code></pre><h3 id="accessing-sentence-dependency-tree">Accessing sentence dependency tree</h3> |
| <p>The following code loops over the sentences and, for each sentence, builds an adjacency list that represents a dependency tree. For more information on what dependency tree is, see <a href="https://cloud.google.com/natural-language/docs/morphology#dependency_trees">Morphology &amp; Dependency Trees</a>.</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">defaultdict</span> |
| </span></span><span class="line"><span class="cl"><span class="n">adjacency_lists</span> <span class="o">=</span> <span class="p">[]</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">index</span> <span class="o">=</span> <span class="mi">0</span> |
| </span></span><span class="line"><span class="cl"><span class="k">for</span> <span class="n">sentence</span> <span class="ow">in</span> <span class="n">response</span><span class="o">.</span><span class="n">sentences</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">adjacency_list</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">list</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">sentence_begin</span> <span class="o">=</span> <span class="n">sentence</span><span class="o">.</span><span class="n">text</span><span class="o">.</span><span class="n">begin_offset</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">sentence_end</span> <span class="o">=</span> <span class="n">sentence_begin</span> <span class="o">+</span> <span class="nb">len</span><span class="p">(</span><span class="n">sentence</span><span class="o">.</span><span class="n">text</span><span class="o">.</span><span class="n">content</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">while</span> <span class="n">index</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">tokens</span><span class="p">)</span> <span class="ow">and</span> \ |
| </span></span><span class="line"><span class="cl"> <span class="n">response</span><span class="o">.</span><span class="n">tokens</span><span class="p">[</span><span class="n">index</span><span class="p">]</span><span class="o">.</span><span class="n">text</span><span class="o">.</span><span class="n">begin_offset</span> <span class="o">&lt;=</span> <span class="n">sentence_end</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">token</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">tokens</span><span class="p">[</span><span class="n">index</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">head_token_index</span> <span class="o">=</span> <span class="n">token</span><span class="o">.</span><span class="n">dependency_edge</span><span class="o">.</span><span class="n">head_token_index</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">head_token_text</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">tokens</span><span class="p">[</span><span class="n">head_token_index</span><span class="p">]</span><span class="o">.</span><span class="n">text</span><span class="o">.</span><span class="n">content</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">adjacency_list</span><span class="p">[</span><span class="n">head_token_text</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">token</span><span class="o">.</span><span class="n">text</span><span class="o">.</span><span class="n">content</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">index</span> <span class="o">+=</span> <span class="mi">1</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">adjacency_lists</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">adjacency_list</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">analyzeDependencyTree</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">(</span><span class="n">SerializableFunction</span><span class="o">&lt;</span><span class="n">AnnotateTextResponse</span><span class="o">,</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">Map</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;&gt;&gt;&gt;)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">response</span> <span class="o">-&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">List</span><span class="o">&lt;</span><span class="n">Map</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;&gt;&gt;</span> <span class="n">adjacencyLists</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o">&lt;&gt;();</span> |
| </span></span><span class="line"><span class="cl"> <span class="kt">int</span> <span class="n">index</span> <span class="o">=</span> <span class="n">0</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="o">(</span><span class="n">Sentence</span> <span class="n">s</span> <span class="o">:</span> <span class="n">response</span><span class="o">.</span><span class="na">getSentencesList</span><span class="o">())</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Map</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;&gt;</span> <span class="n">adjacencyMap</span> <span class="o">=</span> <span class="k">new</span> <span class="n">HashMap</span><span class="o">&lt;&gt;();</span> |
| </span></span><span class="line"><span class="cl"> <span class="kt">int</span> <span class="n">sentenceBegin</span> <span class="o">=</span> <span class="n">s</span><span class="o">.</span><span class="na">getText</span><span class="o">().</span><span class="na">getBeginOffset</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="kt">int</span> <span class="n">sentenceEnd</span> <span class="o">=</span> <span class="n">sentenceBegin</span> <span class="o">+</span> <span class="n">s</span><span class="o">.</span><span class="na">getText</span><span class="o">().</span><span class="na">getContent</span><span class="o">().</span><span class="na">length</span><span class="o">()</span> <span class="o">-</span> <span class="n">1</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">while</span> <span class="o">(</span><span class="n">index</span> <span class="o">&lt;</span> <span class="n">response</span><span class="o">.</span><span class="na">getTokensCount</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&amp;&amp;</span> <span class="n">response</span><span class="o">.</span><span class="na">getTokens</span><span class="o">(</span><span class="n">index</span><span class="o">).</span><span class="na">getText</span><span class="o">().</span><span class="na">getBeginOffset</span><span class="o">()</span> <span class="o">&lt;=</span> <span class="n">sentenceEnd</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Token</span> <span class="n">token</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="na">getTokensList</span><span class="o">().</span><span class="na">get</span><span class="o">(</span><span class="n">index</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="kt">int</span> <span class="n">headTokenIndex</span> <span class="o">=</span> <span class="n">token</span><span class="o">.</span><span class="na">getDependencyEdge</span><span class="o">().</span><span class="na">getHeadTokenIndex</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">String</span> <span class="n">headTokenContent</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">response</span><span class="o">.</span><span class="na">getTokens</span><span class="o">(</span><span class="n">headTokenIndex</span><span class="o">).</span><span class="na">getText</span><span class="o">().</span><span class="na">getContent</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">List</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">adjacencyList</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">adjacencyMap</span><span class="o">.</span><span class="na">getOrDefault</span><span class="o">(</span><span class="n">headTokenContent</span><span class="o">,</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o">&lt;&gt;());</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">adjacencyList</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">token</span><span class="o">.</span><span class="na">getText</span><span class="o">().</span><span class="na">getContent</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">adjacencyMap</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">headTokenContent</span><span class="o">,</span> <span class="n">adjacencyList</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">index</span><span class="o">++;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">adjacencyLists</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">adjacencyMap</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">adjacencyLists</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">};</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>The output is below. For better readability, indexes are replaced by text which they refer to:</p> |
| <pre tabindex="0"><code>[ |
| { |
| &#34;experience&#34;: [ |
| &#34;My&#34; |
| ], |
| &#34;been&#34;: [ |
| &#34;experience&#34;, |
| &#34;far&#34;, |
| &#34;has&#34;, |
| &#34;been&#34;, |
| &#34;fantastic&#34;, |
| &#34;!&#34; |
| ], |
| &#34;far&#34;: [ |
| &#34;so&#34; |
| ] |
| }, |
| { |
| &#34;recommend&#34;: [ |
| &#34;I&#34;, |
| &#34;&#39;d&#34;, |
| &#34;really&#34;, |
| &#34;recommend&#34;, |
| &#34;product&#34;, |
| &#34;.&#34; |
| ], |
| &#34;product&#34;: [ |
| &#34;this&#34; |
| ] |
| } |
| ] |
| </code></pre><h2 id="getting-predictions">Getting predictions</h2> |
| <p>This section shows how to use <a href="https://cloud.google.com/ai-platform/prediction/docs/overview">Google Cloud AI Platform Prediction</a> to make predictions about new data from a cloud-hosted machine learning model.</p> |
| <p><a href="https://github.com/tensorflow/tfx-bsl">tfx_bsl</a> is a library with a Beam PTransform called <code>RunInference</code>. <code>RunInference</code> is able to perform an inference that can use an external service endpoint for receiving data. When using a service endpoint, the transform takes a PCollection of type <code>tf.train.Example</code> and, for every batch of elements, sends a request to AI Platform Prediction. The size of a batch is automatically computed. For more details on how Beam finds the best batch size, refer to a docstring for <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html?highlight=batchelements#apache_beam.transforms.util.BatchElements">BatchElements</a>. Currently, the transform does not support using <code>tf.train.SequenceExample</code> as input, but the work is in progress.</p> |
| <p>The transform produces a PCollection of type <code>PredictionLog</code>, which contains predictions.</p> |
| <p>Before getting started, deploy a TensorFlow model to AI Platform Prediction. The cloud service manages the infrastructure needed to handle prediction requests in both efficient and scalable way. Do note that only TensorFlow models are supported by the transform. For more information, see <a href="https://cloud.google.com/ai-platform/prediction/docs/exporting-savedmodel-for-prediction">Exporting a SavedModel for prediction</a>.</p> |
| <p>Once a machine learning model is deployed, prepare a list of instances to get predictions for. To send binary data, make sure that the name of an input ends in <code>_bytes</code>. This will base64-encode data before sending a request.</p> |
| <h3 id="example">Example</h3> |
| <p>Here is an example of a pipeline that reads input instances from the file, converts JSON objects to <code>tf.train.Example</code> objects and sends data to AI Platform Prediction. The content of a file can look like this:</p> |
| <pre tabindex="0"><code>{&#34;input&#34;: &#34;the quick brown&#34;} |
| {&#34;input&#34;: &#34;la bruja le&#34;} |
| </code></pre><p>The example creates <code>tf.train.BytesList</code> instances, thus it expects byte-like strings as input. However, other data types, like <code>tf.train.FloatList</code> and <code>tf.train.Int64List</code>, are also supported by the transform.</p> |
| <p>Here is the code:</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">json</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="k">as</span> <span class="nn">beam</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">tensorflow</span> <span class="k">as</span> <span class="nn">tf</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">tfx_bsl.beam.run_inference</span> <span class="kn">import</span> <span class="n">RunInference</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">tfx_bsl.proto</span> <span class="kn">import</span> <span class="n">model_spec_pb2</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">convert_json_to_tf_example</span><span class="p">(</span><span class="n">json_obj</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">samples</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">json_obj</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">text</span> <span class="ow">in</span> <span class="n">samples</span><span class="o">.</span><span class="n">items</span><span class="p">():</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">value</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">Feature</span><span class="p">(</span><span class="n">bytes_list</span><span class="o">=</span><span class="n">tf</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">BytesList</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">value</span><span class="o">=</span><span class="p">[</span><span class="n">text</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">&#39;utf-8&#39;</span><span class="p">)]))</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">feature</span> <span class="o">=</span> <span class="p">{</span><span class="n">name</span><span class="p">:</span> <span class="n">value</span><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">tf</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">Example</span><span class="p">(</span><span class="n">features</span><span class="o">=</span><span class="n">tf</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">Features</span><span class="p">(</span><span class="n">feature</span><span class="o">=</span><span class="n">feature</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">p</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">_</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ReadFromText</span><span class="p">(</span><span class="s1">&#39;gs://my-bucket/samples.json&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="n">convert_json_to_tf_example</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">RunInference</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">model_spec_pb2</span><span class="o">.</span><span class="n">InferenceEndpoint</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">model_endpoint_spec</span><span class="o">=</span><span class="n">model_spec_pb2</span><span class="o">.</span><span class="n">AIPlatformPredictionModelSpec</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">project_id</span><span class="o">=</span><span class="s1">&#39;my-project-id&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">model_name</span><span class="o">=</span><span class="s1">&#39;my-model-name&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">version_name</span><span class="o">=</span><span class="s1">&#39;my-model-version&#39;</span><span class="p">))))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Getting predictions is not yet available for Java. [https://github.com/apache/beam/issues/20001] |
| </span></span></span></code></pre></div> |
| </div> |
| </div></description></item><item><title>Documentation: Anomaly Detection</title><link>/documentation/ml/anomaly-detection/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/ml/anomaly-detection/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="anomaly-detection-example">Anomaly Detection Example</h1> |
| <p>The anomaly detection example demonstrates how to set up an anomaly detection pipeline that reads text from Pub/Sub in real time, and then detects anomalies using a trained HDBSCAN clustering model.</p> |
| <h2 id="dataset-for-anomaly-detection">Dataset for Anomaly Detection</h2> |
| <p>This example uses a dataset called <a href="https://huggingface.co/datasets/emotion">emotion</a> that contains 20,000 English Twitter messages with 6 basic emotions: anger, fear, joy, love, sadness, and surprise. The dataset has three splits: train (for training), validation, and test (for performance evaluation). Because it contains the text and the category (class) of the dataset, it is a supervised dataset. You can use the <a href="https://huggingface.co/docs/datasets/index">Hugging Face datasets page</a> to access this dataset.</p> |
| <p>The following text shows examples from the train split of the dataset:</p> |
| <table> |
| <thead> |
| <tr> |
| <th style="text-align:left">Text</th> |
| <th style="text-align:center">Type of emotion</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td style="text-align:left">im grabbing a minute to post i feel greedy wrong</td> |
| <td style="text-align:center">Anger</td> |
| </tr> |
| <tr> |
| <td style="text-align:left">i am ever feeling nostalgic about the fireplace i will know that it is still on the property</td> |
| <td style="text-align:center">Love</td> |
| </tr> |
| <tr> |
| <td style="text-align:left">ive been taking or milligrams or times recommended amount and ive fallen asleep a lot faster but i also feel like so funny</td> |
| <td style="text-align:center">Fear</td> |
| </tr> |
| <tr> |
| <td style="text-align:left">on a boat trip to denmark</td> |
| <td style="text-align:center">Joy</td> |
| </tr> |
| <tr> |
| <td style="text-align:left">i feel you know basically like a fake in the realm of science fiction</td> |
| <td style="text-align:center">Sadness</td> |
| </tr> |
| <tr> |
| <td style="text-align:left">i began having them several times a week feeling tortured by the hallucinations moving people and figures sounds and vibrations</td> |
| <td style="text-align:center">Fear</td> |
| </tr> |
| </tbody> |
| </table> |
| <h2 id="anomaly-detection-algorithm">Anomaly Detection Algorithm</h2> |
| <p><a href="https://hdbscan.readthedocs.io/en/latest/how_hdbscan_works.html">HDBSCAN</a> is a clustering algorithm that extends DBSCAN by converting it into a hierarchical clustering algorithm and then extracting a flat clustering based in the stability of clusters. When trained, the model predicts <code>-1</code> if a new data point is an outlier, otherwise it predicts one of the existing clusters.</p> |
| <h2 id="ingestion-to-pubsub">Ingestion to Pub/Sub</h2> |
| <p>Ingest the data into <a href="https://cloud.google.com/pubsub/docs/overview">Pub/Sub</a> so that while clustering, the model can read the tweets from Pub/Sub. Pub/Sub is a messaging service for exchanging event data among applications and services. Streaming analytics and data integration pipelines use Pub/Sub to ingest and distribute data.</p> |
| <p>You can see the full example code for ingesting data into Pub/Sub in <a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/anomaly_detection/write_data_to_pubsub_pipeline/">GitHub</a></p> |
| <p>The file structure for the ingestion pipeline is shown in the following diagram:</p> |
| <pre><code>write_data_to_pubsub_pipeline/ |
| ├── pipeline/ |
| │ ├── __init__.py |
| │ ├── options.py |
| │ └── utils.py |
| ├── __init__.py |
| ├── config.py |
| ├── main.py |
| └── setup.py |
| </code></pre> |
| <p><code>pipeline/utils.py</code> contains the code for loading the emotion dataset and two <code>beam.DoFn</code> that are used for data transformation.</p> |
| <p><code>pipeline/options.py</code> contains the pipeline options to configure the Dataflow pipeline.</p> |
| <p><code>config.py</code> defines variables that are used multiple times, like Google Cloud PROJECT_ID and NUM_WORKERS.</p> |
| <p><code>setup.py</code> defines the packages and requirements for the pipeline to run.</p> |
| <p><code>main.py</code> contains the pipeline code and additional functions used for running the pipeline.</p> |
| <h3 id="run-the-pipeline">Run the Pipeline</h3> |
| <p>To run the pipeline, install the required packages.For this example, you need access to a Google Cloud project, and you need to configure the Google Cloud variables, like <code>PROJECT_ID</code>, <code>REGION</code>, <code>PubSub TOPIC_ID</code>, and others in the <code>config.py</code> file.</p> |
| <ol> |
| <li>Locally on your machine: <code>python main.py</code></li> |
| <li>On GCP for Dataflow: <code>python main.py --mode cloud</code></li> |
| </ol> |
| <p>The <code>write_data_to_pubsub_pipeline</code> contains four different transforms:</p> |
| <ol> |
| <li>Load the emotion dataset using Hugging Face datasets (for simplicity, we take samples from three classes instead of six).</li> |
| <li>Associate each piece of text with a unique identifier (UID).</li> |
| <li>Convert the text into the format that Pub/Sub expects.</li> |
| <li>Write the formatted message to Pub/Sub.</li> |
| </ol> |
| <h2 id="anomaly-detection-on-streaming-data">Anomaly Detection on Streaming Data</h2> |
| <p>After ingesting the data to Pub/Sub, run the anomaly detection pipeline. This pipeline reads the streaming message from Pub/Sub, converts the text to an embedding using a language model, and feeds the embedding to an already trained clustering model to predict whether the message is an anomaly. One prerequisite for this pipeline is to have an HDBSCAN clustering model trained on the training split of the dataset.</p> |
| <p>You can find the full example code for anomaly detection in <a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/anomaly_detection/anomaly_detection_pipeline/">GitHub</a></p> |
| <p>The following diagram shows the file structure for the anomaly_detection pipeline:</p> |
| <pre><code>anomaly_detection_pipeline/ |
| ├── pipeline/ |
| │ ├── __init__.py |
| │ ├── options.py |
| │ └── transformations.py |
| ├── __init__.py |
| ├── config.py |
| ├── main.py |
| └── setup.py |
| </code></pre> |
| <p><code>pipeline/transformations.py</code> contains the code for different <code>beam.DoFn</code> and additional functions that are used in pipeline.</p> |
| <p><code>pipeline/options.py</code> contains the pipeline options to configure the Dataflow pipeline.</p> |
| <p><code>config.py</code> defines variables that are used multiple times, like the Google Cloud PROJECT_ID and NUM_WORKERS.</p> |
| <p><code>setup.py</code> defines the packages and requirements for the pipeline to run.</p> |
| <p><code>main.py</code> contains the pipeline code and additional functions used to run the pipeline.</p> |
| <h3 id="run-the-pipeline-1">Run the Pipeline</h3> |
| <p>Install the required packages and push the data to Pub/Sub. For this example, you need access to a Google Cloud project, and you need to configure the Google Cloud variables, like <code>PROJECT_ID</code>, <code>REGION</code>, <code>PubSub SUBSCRIPTION_ID</code>, and others in the <code>config.py</code> file.</p> |
| <ol> |
| <li>Locally on your machine: <code>python main.py</code></li> |
| <li>On GCP for Dataflow: <code>python main.py --mode cloud</code></li> |
| </ol> |
| <p>The pipeline includes the following steps:</p> |
| <ol> |
| <li>Read the message from Pub/Sub.</li> |
| <li>Convert the Pub/Sub message into a <code>PCollection</code> of dictionaries where the key is the UID and the value is the Twitter text.</li> |
| <li>Encode the text into transformer-readable token ID integers using a tokenizer.</li> |
| <li>Use RunInference to get the vector embedding from a transformer-based language model.</li> |
| <li>Normalize the embedding.</li> |
| <li>Use RunInference to get anomaly prediction from a trained HDBSCAN clustering model.</li> |
| <li>Write the prediction to BigQuery so that the clustering model can be retrained when needed.</li> |
| <li>Send an email alert if an anomaly is detected.</li> |
| </ol> |
| <p>The following code snippet shows the first two steps of the pipeline:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code> docs = ( |
| pipeline |
| | &#34;Read from PubSub&#34; |
| &gt;&gt; ReadFromPubSub(subscription=cfg.SUBSCRIPTION_ID, with_attributes=True) |
| | &#34;Decode PubSubMessage&#34; &gt;&gt; beam.ParDo(Decode()) |
| )</code></pre> |
| </div> |
| </div> |
| <p>The next section describes the following pipeline steps:</p> |
| <ul> |
| <li>Tokenizing the text</li> |
| <li>Getting embedding using RunInference</li> |
| <li>Getting predictions from the HDBSCAN model</li> |
| </ul> |
| <h3 id="get-embedding-from-a-language-model">Get Embedding from a Language Model</h3> |
| <p>In order to do clustering with text data, first map the text into vectors of numerical values suitable for statistical analysis. This example uses a transformer-based language model called <a href="https://huggingface.co/sentence-transformers/stsb-distilbert-base">sentence-transformers/stsb-distilbert-base/stsb-distilbert-base</a>. This model maps sentences and paragraphs to a 768 dimensional dense vector space, and you can use it for tasks like clustering or semantic search.</p> |
| <p>Because the language model is expecting a tokenized input instead of raw text, start by tokenizing the text. Tokenization is a preprocessing task that transforms text so that it can be fed into the model for getting predictions.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code> normalized_embedding = ( |
| docs |
| | &#34;Tokenize Text&#34; &gt;&gt; beam.Map(tokenize_sentence)</code></pre> |
| </div> |
| </div> |
| <p>Here, <code>tokenize_sentence</code> is a function that takes a dictionary with a text and an ID, tokenizes the text, and returns a tuple of the text and ID as well as the tokenized output.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>Tokenizer = AutoTokenizer.from_pretrained(cfg.TOKENIZER_NAME) |
| def tokenize_sentence(input_dict): |
| &#34;&#34;&#34; |
| Takes a dictionary with a text and an id, tokenizes the text, and |
| returns a tuple of the text and id and the tokenized text |
| Args: |
| input_dict: a dictionary with the text and id of the sentence |
| Returns: |
| A tuple of the text and id, and a dictionary of the tokens. |
| &#34;&#34;&#34; |
| text, uid = input_dict[&#34;text&#34;], input_dict[&#34;id&#34;] |
| tokens = Tokenizer([text], padding=True, truncation=True, return_tensors=&#34;pt&#34;) |
| tokens = {key: torch.squeeze(val) for key, val in tokens.items()} |
| return (text, uid), tokens</code></pre> |
| </div> |
| </div> |
| <p>Tokenized output is then passed to the language model to get the embeddings. To get embeddings from the language model, we use <code>RunInference()</code> from Apache Beam.</p> |
| <p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code> | &#34;Get Embedding&#34; &gt;&gt; RunInference(KeyedModelHandler(embedding_model_handler))</code></pre> |
| </div> |
| </div> |
| where <code>embedding_model_handler</code> is:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code> embedding_model_handler = PytorchNoBatchModelHandler( |
| state_dict_path=cfg.MODEL_STATE_DICT_PATH, |
| model_class=ModelWrapper, |
| model_params={&#34;config&#34;: AutoConfig.from_pretrained(cfg.MODEL_CONFIG_PATH)}, |
| device=&#34;cpu&#34;, |
| )</code></pre> |
| </div> |
| </div> |
| <p>We define <code>PytorchNoBatchModelHandler</code> as a wrapper to <code>PytorchModelHandler</code> to limit batch size to one.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code># Can be removed once: https://github.com/apache/beam/issues/21863 is fixed |
| class PytorchNoBatchModelHandler(PytorchModelHandlerKeyedTensor): |
| &#34;&#34;&#34;Wrapper to PytorchModelHandler to limit batch size to 1. |
| The tokenized strings generated from BertTokenizer may have different |
| lengths, which doesn&#39;t work with torch.stack() in current RunInference |
| implementation since stack() requires tensors to be the same size. |
| Restricting max_batch_size to 1 means there is only 1 example per `batch` |
| in the run_inference() call. |
| &#34;&#34;&#34; |
| def batch_elements_kwargs(self): |
| return {&#34;max_batch_size&#34;: 1}</code></pre> |
| </div> |
| </div> |
| <p>Because the <code>forward()</code> for <code>DistilBertModel</code> doesn&rsquo;t return the embeddings, we custom define the model_class <code>ModelWrapper</code> to get the vector embedding.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>class ModelWrapper(DistilBertModel): |
| &#34;&#34;&#34;Wrapper to DistilBertModel to get embeddings when calling |
| forward function.&#34;&#34;&#34; |
| def forward(self, **kwargs): |
| output = super().forward(**kwargs) |
| sentence_embedding = ( |
| self.mean_pooling(output, |
| kwargs[&#34;attention_mask&#34;]).detach().cpu().numpy()) |
| return sentence_embedding |
| # Mean Pooling - Take attention mask into account for correct averaging |
| def mean_pooling(self, model_output, attention_mask): |
| &#34;&#34;&#34; |
| Calculates the mean of token embeddings |
| Args: |
| model_output: The output of the model. |
| attention_mask: This is a tensor that contains 1s for all input tokens and |
| 0s for all padding tokens. |
| Returns: |
| The mean of the token embeddings. |
| &#34;&#34;&#34; |
| token_embeddings = model_output[ |
| 0] # First element of model_output contains all token embeddings |
| input_mask_expanded = ( |
| attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()) |
| return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp( |
| input_mask_expanded.sum(1), min=1e-9)</code></pre> |
| </div> |
| </div> |
| <p>After getting the embedding for each piece of Twitter text, the embeddings are normalized, because the trained model is expecting normalized embeddings.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code> | &#34;Normalize Embedding&#34; &gt;&gt; beam.ParDo(NormalizeEmbedding())</code></pre> |
| </div> |
| </div> |
| <h3 id="get-predictions">Get Predictions</h3> |
| <p>The normalized embeddings are then forwarded to the trained HDBSCAN model to get the predictions.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code> predictions = ( |
| normalized_embedding |
| | &#34;Get Prediction from Clustering Model&#34; |
| &gt;&gt; RunInference(model_handler=clustering_model_handler) |
| )</code></pre> |
| </div> |
| </div> |
| <p>where <code>clustering_model_handler</code> is:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code> clustering_model_handler = KeyedModelHandler( |
| CustomSklearnModelHandlerNumpy( |
| model_uri=cfg.CLUSTERING_MODEL_PATH, model_file_type=ModelFileType.JOBLIB |
| ) |
| )</code></pre> |
| </div> |
| </div> |
| <p>We define <code>CustomSklearnModelHandlerNumpy</code> as a wrapper to <code>SklearnModelHandlerNumpy</code> to limit batch size to one and to override <code>run_inference</code> so that <code>hdbscan.approximate_predict()</code> is used to get anomaly predictions.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>class CustomSklearnModelHandlerNumpy(SklearnModelHandlerNumpy): |
| # limit batch size to 1 can be removed once: https://github.com/apache/beam/issues/21863 is fixed |
| def batch_elements_kwargs(self): |
| &#34;&#34;&#34;Limit batch size to 1 for inference&#34;&#34;&#34; |
| return {&#34;max_batch_size&#34;: 1} |
| # run_inference can be removed once: https://github.com/apache/beam/issues/22572 is fixed |
| def run_inference(self, batch, model, inference_args=None): |
| &#34;&#34;&#34;Runs inferences on a batch of numpy arrays. |
| Args: |
| batch: A sequence of examples as numpy arrays. They should |
| be single examples. |
| model: A numpy model or pipeline. Must implement predict(X). |
| Where the parameter X is a numpy array. |
| inference_args: Any additional arguments for an inference. |
| Returns: |
| An Iterable of type PredictionResult. |
| &#34;&#34;&#34; |
| _validate_inference_args(inference_args) |
| vectorized_batch = np.vstack(batch) |
| predictions = hdbscan.approximate_predict(model, vectorized_batch) |
| return [PredictionResult(x, y) for x, y in zip(batch, predictions)]</code></pre> |
| </div> |
| </div> |
| <p>After getting the model predictions, decode the output from <code>RunInference</code> into a dictionary. Next, store the prediction in a BigQuery table for analysis, update the HDBSCAN model, and send an email alert if the prediction is an anomaly.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code> _ = ( |
| predictions |
| | &#34;Decode Prediction&#34; &gt;&gt; beam.ParDo(DecodePrediction()) |
| | &#34;Write to BQ&#34; &gt;&gt; beam.io.WriteToBigQuery( |
| table=cfg.TABLE_URI, |
| schema=cfg.TABLE_SCHEMA, |
| write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND, |
| create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED, |
| )) |
| _ = predictions | &#34;Alert by Email&#34; &gt;&gt; beam.ParDo(TriggerEmailAlert())</code></pre> |
| </div> |
| </div></description></item><item><title>Documentation: Apache Beam: Developing I/O connectors for Java</title><link>/documentation/io/developing-io-java/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/io/developing-io-java/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="developing-io-connectors-for-java">Developing I/O connectors for Java</h1> |
| <p><strong>IMPORTANT:</strong> Use <code>Splittable DoFn</code> to develop your new I/O. For more details, read the |
| <a href="/documentation/io/developing-io-overview/">new I/O connector overview</a>.</p> |
| <p>To connect to a data store that isn’t supported by Beam’s existing I/O |
| connectors, you must create a custom I/O connector that usually consist of a |
| source and a sink. All Beam sources and sinks are composite transforms; however, |
| the implementation of your custom I/O depends on your use case. Before you |
| start, read the |
| <a href="/documentation/io/developing-io-overview/">new I/O connector overview</a> |
| for an overview of developing a new I/O connector, the available implementation |
| options, and how to choose the right option for your use case.</p> |
| <p>This guide covers using the <code>Source</code> and <code>FileBasedSink</code> interfaces using Java. |
| The Python SDK offers the same functionality, but uses a slightly different API. |
| See <a href="/documentation/io/developing-io-python/">Developing I/O connectors for Python</a> |
| for information specific to the Python SDK.</p> |
| <h2 id="basic-code-reqs">Basic code requirements</h2> |
| <p>Beam runners use the classes you provide to read and/or write data using |
| multiple worker instances in parallel. As such, the code you provide for |
| <code>Source</code> and <code>FileBasedSink</code> subclasses must meet some basic requirements:</p> |
| <ol> |
| <li> |
| <p><strong>Serializability:</strong> Your <code>Source</code> or <code>FileBasedSink</code> subclass, whether |
| bounded or unbounded, must be Serializable. A runner might create multiple |
| instances of your <code>Source</code> or <code>FileBasedSink</code> subclass to be sent to |
| multiple remote workers to facilitate reading or writing in parallel.</p> |
| </li> |
| <li> |
| <p><strong>Immutability:</strong> |
| Your <code>Source</code> or <code>FileBasedSink</code> subclass must be effectively immutable. |
| All private fields must be declared final, and all private variables of |
| collection type must be effectively immutable. If your class has setter |
| methods, those methods must return an independent copy of the object with |
| the relevant field modified.</p> |
| <p>You should only use mutable state in your <code>Source</code> or <code>FileBasedSink</code> |
| subclass if you are using lazy evaluation of expensive computations that |
| you need to implement the source or sink; in that case, you must declare |
| all mutable instance variables transient.</p> |
| </li> |
| <li> |
| <p><strong>Thread-Safety:</strong> Your code must be thread-safe. If you build your source |
| to work with dynamic work rebalancing, it is critical that you make your |
| code thread-safe. The Beam SDK provides a helper class to make this easier. |
| See <a href="#bounded-dynamic">Using Your BoundedSource with dynamic work rebalancing</a> |
| for more details.</p> |
| </li> |
| <li> |
| <p><strong>Testability:</strong> It is critical to exhaustively unit test all of your |
| <code>Source</code> and <code>FileBasedSink</code> subclasses, especially if you build your |
| classes to work with advanced features such as dynamic work rebalancing. A |
| minor implementation error can lead to data corruption or data loss (such |
| as skipping or duplicating records) that can be hard to detect.</p> |
| <p>To assist in testing <code>BoundedSource</code> implementations, you can use the |
| SourceTestUtils class. <code>SourceTestUtils</code> contains utilities for automatically |
| verifying some of the properties of your <code>BoundedSource</code> implementation. You |
| can use <code>SourceTestUtils</code> to increase your implementation&rsquo;s test coverage |
| using a wide range of inputs with relatively few lines of code. For |
| examples that use <code>SourceTestUtils</code>, see the |
| <a href="https://github.com/apache/beam/blob/master/sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/io/AvroSourceTest.java">AvroSourceTest</a> and |
| <a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOReadTest.java">TextIOReadTest</a> |
| source code.</p> |
| </li> |
| </ol> |
| <p>In addition, see the <a href="/contribute/ptransform-style-guide/">PTransform style guide</a> |
| for Beam&rsquo;s transform style guidance.</p> |
| <h2 id="implementing-the-source-interface">Implementing the Source interface</h2> |
| <p>To create a data source for your pipeline, you must provide the format-specific |
| logic that tells a runner how to read data from your input source, and how to |
| split your data source into multiple parts so that multiple worker instances can |
| read your data in parallel. If you&rsquo;re creating a data source that reads |
| unbounded data, you must provide additional logic for managing your source&rsquo;s |
| watermark and optional checkpointing.</p> |
| <p>Supply the logic for your source by creating the following classes:</p> |
| <ul> |
| <li> |
| <p>A subclass of <code>BoundedSource</code> if you want to read a finite (batch) data set, |
| or a subclass of <code>UnboundedSource</code> if you want to read an infinite (streaming) |
| data set. These subclasses describe the data you want to read, including the |
| data&rsquo;s location and parameters (such as how much data to read).</p> |
| </li> |
| <li> |
| <p>A subclass of <code>Source.Reader</code>. Each Source must have an associated Reader that |
| captures all the state involved in reading from that <code>Source</code>. This can |
| include things like file handles, RPC connections, and other parameters that |
| depend on the specific requirements of the data format you want to read.</p> |
| </li> |
| <li> |
| <p>The <code>Reader</code> class hierarchy mirrors the Source hierarchy. If you&rsquo;re extending |
| <code>BoundedSource</code>, you&rsquo;ll need to provide an associated <code>BoundedReader</code>. if you&rsquo;re |
| extending <code>UnboundedSource</code>, you&rsquo;ll need to provide an associated |
| <code>UnboundedReader</code>.</p> |
| </li> |
| <li> |
| <p>One or more user-facing wrapper composite transforms (<code>PTransform</code>) that |
| wrap read operations. <a href="#ptransform-wrappers">PTransform wrappers</a> discusses |
| why you should avoid exposing your sources.</p> |
| </li> |
| </ul> |
| <h3 id="implementing-the-source-subclass">Implementing the Source subclass</h3> |
| <p>You must create a subclass of either <code>BoundedSource</code> or <code>UnboundedSource</code>, |
| depending on whether your data is a finite batch or an infinite stream. In |
| either case, your <code>Source</code> subclass must override the abstract methods in the |
| superclass. A runner might call these methods when using your data source. For |
| example, when reading from a bounded source, a runner uses these methods to |
| estimate the size of your data set and to split it up for parallel reading.</p> |
| <p>Your <code>Source</code> subclass should also manage basic information about your data |
| source, such as the location. For example, the example <code>Source</code> implementation |
| in Beam’s <a href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/io/gcp/datastore/DatastoreIO.html">DatastoreIO</a> |
| class takes host, datasetID, and query as arguments. The connector uses these |
| values to obtain data from Cloud Datastore.</p> |
| <h4 id="boundedsource">BoundedSource</h4> |
| <p><code>BoundedSource</code> represents a finite data set from which a Beam runner may read, |
| possibly in parallel. <code>BoundedSource</code> contains a set of abstract methods that |
| the runner uses to split the data set for reading by multiple workers.</p> |
| <p>To implement a <code>BoundedSource</code>, your subclass must override the following |
| abstract methods:</p> |
| <ul> |
| <li> |
| <p><code>split</code>: The runner uses this method to split your finite data |
| into bundles of a given size.</p> |
| </li> |
| <li> |
| <p><code>getEstimatedSizeBytes</code>: The runner uses this method to estimate the total |
| size of your data, in bytes.</p> |
| </li> |
| <li> |
| <p><code>createReader</code>: Creates the associated <code>BoundedReader</code> for this |
| <code>BoundedSource</code>.</p> |
| </li> |
| </ul> |
| <p>You can see a model of how to implement <code>BoundedSource</code> and the required |
| abstract methods in Beam’s implementations for Cloud BigTable |
| (<a href="https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java">BigtableIO.java</a>) |
| and BigQuery (<a href="https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java">BigQuerySourceBase.java</a>).</p> |
| <h4 id="unboundedsource">UnboundedSource</h4> |
| <p><code>UnboundedSource</code> represents an infinite data stream from which the runner may |
| read, possibly in parallel. <code>UnboundedSource</code> contains a set of abstract methods |
| that the runner uses to support streaming reads in parallel; these include |
| <em>checkpointing</em> for failure recovery, <em>record IDs</em> to prevent data duplication, |
| and <em>watermarking</em> for estimating data completeness in downstream parts of your |
| pipeline.</p> |
| <p>To implement an <code>UnboundedSource</code>, your subclass must override the following |
| abstract methods:</p> |
| <ul> |
| <li> |
| <p><code>split</code>: The runner uses this method to generate a list of |
| <code>UnboundedSource</code> objects which represent the number of sub-stream instances |
| from which the service should read in parallel.</p> |
| </li> |
| <li> |
| <p><code>getCheckpointMarkCoder</code>: The runner uses this method to obtain the Coder for |
| the checkpoints for your source (if any).</p> |
| </li> |
| <li> |
| <p><code>requiresDeduping</code>: The runner uses this method to determine whether the data |
| requires explicit removal of duplicate records. If this method returns true, |
| the runner will automatically insert a step to remove duplicates from your |
| source&rsquo;s output. This should return true if and only if your source |
| provides record IDs for each record. See <code>UnboundedReader.getCurrentRecordId</code> |
| for when this should be done.</p> |
| </li> |
| <li> |
| <p><code>createReader</code>: Creates the associated <code>UnboundedReader</code> for this |
| <code>UnboundedSource</code>.</p> |
| </li> |
| </ul> |
| <h3 id="implementing-the-reader-subclass">Implementing the Reader subclass</h3> |
| <p>You must create a subclass of either <code>BoundedReader</code> or <code>UnboundedReader</code> to be |
| returned by your source subclass&rsquo;s <code>createReader</code> method. The runner uses the |
| methods in your <code>Reader</code> (whether bounded or unbounded) to do the actual reading |
| of your dataset.</p> |
| <p><code>BoundedReader</code> and <code>UnboundedReader</code> have similar basic interfaces, which |
| you&rsquo;ll need to define. In addition, there are some additional methods unique to |
| <code>UnboundedReader</code> that you&rsquo;ll need to implement for working with unbounded data, |
| and an optional method you can implement if you want your <code>BoundedReader</code> to |
| take advantage of dynamic work rebalancing. There are also minor differences in |
| the semantics for the <code>start()</code> and <code>advance()</code> methods when using |
| <code>UnboundedReader</code>.</p> |
| <h4 id="reader-methods-common-to-both-boundedreader-and-unboundedreader">Reader methods common to both BoundedReader and UnboundedReader</h4> |
| <p>A runner uses the following methods to read data using <code>BoundedReader</code> or |
| <code>UnboundedReader</code>:</p> |
| <ul> |
| <li> |
| <p><code>start</code>: Initializes the <code>Reader</code> and advances to the first record to be read. |
| This method is called exactly once when the runner begins reading your data, |
| and is a good place to put expensive operations needed for initialization.</p> |
| </li> |
| <li> |
| <p><code>advance</code>: Advances the reader to the next valid record. This method must |
| return false if there is no more input available. <code>BoundedReader</code> should stop |
| reading once advance returns false, but <code>UnboundedReader</code> can return true in |
| future calls once more data is available from your stream.</p> |
| </li> |
| <li> |
| <p><code>getCurrent</code>: Returns the data record at the current position, last read by |
| start or advance.</p> |
| </li> |
| <li> |
| <p><code>getCurrentTimestamp</code>: Returns the timestamp for the current data record. You |
| only need to override <code>getCurrentTimestamp</code> if your source reads data that has |
| intrinsic timestamps. The runner uses this value to set the intrinsic |
| timestamp for each element in the resulting output <code>PCollection</code>.</p> |
| </li> |
| </ul> |
| <h4 id="reader-methods-unique-to-unboundedreader">Reader methods unique to UnboundedReader</h4> |
| <p>In addition to the basic <code>Reader</code> interface, <code>UnboundedReader</code> has some |
| additional methods for managing reads from an unbounded data source:</p> |
| <ul> |
| <li> |
| <p><code>getCurrentRecordId</code>: Returns a unique identifier for the current record. |
| The runner uses these record IDs to filter out duplicate records. If your |
| data has logical IDs present in each record, you can have this method return |
| them; otherwise, you can return a hash of the record contents, using at |
| least a 128-bit hash. It is incorrect to use Java&rsquo;s <code>Object.hashCode()</code>, as |
| a 32-bit hash is generally insufficient for preventing collisions, and |
| <code>hasCode()</code> is not guaranteed to be stable across processes.</p> |
| <p>Implementing <code>getCurrentRecordId</code> is optional if your source uses a |
| checkpointing scheme that uniquely identifies each record. For example, if |
| your splits are files and the checkpoints are file positions up to which all |
| data has been read, you do not need record IDs. However, record IDs can |
| still be useful if upstream systems writing data to your source occasionally |
| produce duplicate records that your source might then read.</p> |
| </li> |
| <li> |
| <p><code>getWatermark</code>: Returns a watermark that your <code>Reader</code> provides. The watermark |
| is the approximate lower bound on timestamps of future elements to be read |
| by your <code>Reader</code>. The runner uses the watermark as an estimate of data |
| completeness. Watermarks are used in windowing and triggers.</p> |
| </li> |
| <li> |
| <p><code>getCheckpointMark</code>: The runner uses this method to create a checkpoint in |
| your data stream. The checkpoint represents the progress of the |
| <code>UnboundedReader</code>, which can be used for failure recovery. Different data |
| streams may use different checkpointing methods; some sources might require |
| received records to be acknowledged, while others might use positional |
| checkpointing. You&rsquo;ll need to tailor this method to the most appropriate |
| checkpointing scheme. For example, you might have this method return the |
| most recently acked record(s).</p> |
| </li> |
| <li> |
| <p><code>getCheckpointMark</code> is optional; you don&rsquo;t need to implement it if your data |
| does not have meaningful checkpoints. However, if you choose not to |
| implement checkpointing in your source, you may encounter duplicate data or |
| data loss in your pipeline, depending on whether your data source tries to |
| re-send records in case of errors.</p> |
| </li> |
| </ul> |
| <p>You can read a bounded <code>PCollection</code> from an <code>UnboundedSource</code> by specifying |
| either <code>.withMaxNumRecords</code> or <code>.withMaxReadTime</code> when you read from your |
| source. <code>.withMaxNumRecords</code> reads a fixed maximum number of records from your |
| unbounded source, while <code>.withMaxReadTime</code> reads from your unbounded source for |
| a fixed maximum time duration.</p> |
| <h4 id="bounded-dynamic">Using your BoundedSource with dynamic work rebalancing</h4> |
| <p>If your source provides bounded data, you can have your <code>BoundedReader</code> work |
| with dynamic work rebalancing by implementing the method <code>splitAtFraction</code>. The |
| runner may call <code>splitAtFraction</code> concurrently with start or advance on a given |
| reader so that the remaining data in your <code>Source</code> can be split and |
| redistributed to other workers.</p> |
| <p>When you implement <code>splitAtFraction</code>, your code must produce a |
| mutually-exclusive set of splits where the union of those splits matches the |
| total data set.</p> |
| <p>If you implement <code>splitAtFraction</code>, you must implement both <code>splitAtFraction</code> |
| and <code>getFractionConsumed</code> in a thread-safe manner, or data loss is possible. You |
| should also unit-test your implementation exhaustively to avoid data duplication |
| or data loss.</p> |
| <p>To ensure that your code is thread-safe, use the <code>RangeTracker</code> thread-safe |
| helper object to manage positions in your data source when implementing |
| <code>splitAtFraction</code> and <code>getFractionConsumed</code>.</p> |
| <p>We highly recommended that you unit test your implementations of |
| <code>splitAtFraction</code> using the <code>SourceTestUtils</code> class. <code>SourceTestUtils</code> contains |
| a number of methods for testing your implementation of <code>splitAtFraction</code>, |
| including exhaustive automatic testing.</p> |
| <h3 id="convenience-source-and-reader-base-classes">Convenience Source and Reader base classes</h3> |
| <p>The Beam SDK contains some convenient abstract base classes to help you create |
| <code>Source</code> and <code>Reader</code> classes that work with common data storage formats, like |
| files.</p> |
| <h4 id="filebasedsource">FileBasedSource</h4> |
| <p>If your data source uses files, you can derive your <code>Source</code> and <code>Reader</code> |
| classes from the <code>FileBasedSource</code> and <code>FileBasedReader</code> abstract base classes. |
| <code>FileBasedSource</code> is a bounded source subclass that implements code common to |
| Beam sources that interact with files, including:</p> |
| <ul> |
| <li>File pattern expansion</li> |
| <li>Sequential record reading</li> |
| <li>Split points</li> |
| </ul> |
| <h2 id="using-filebasedsink">Using the FileBasedSink abstraction</h2> |
| <p>If your data source uses files, you can implement the <code>FileBasedSink</code> |
| abstraction to create a file-based sink. For other sinks, use <code>ParDo</code>, |
| <code>GroupByKey</code>, and other transforms offered by the Beam SDK for Java. See the |
| <a href="/documentation/io/developing-io-overview/">developing I/O connectors overview</a> |
| for more details.</p> |
| <p>When using the <code>FileBasedSink</code> interface, you must provide the format-specific |
| logic that tells the runner how to write bounded data from your pipeline&rsquo;s |
| <code>PCollection</code>s to an output sink. The runner writes bundles of data in parallel |
| using multiple workers.</p> |
| <p>Supply the logic for your file-based sink by implementing the following classes:</p> |
| <ul> |
| <li> |
| <p>A subclass of the abstract base class <code>FileBasedSink</code>. <code>FileBasedSink</code> |
| describes a location or resource that your pipeline can write to in |
| parallel. To avoid exposing your sink to end-users, your <code>FileBasedSink</code> |
| subclass should be protected or private.</p> |
| </li> |
| <li> |
| <p>A user-facing wrapper <code>PTransform</code> that, as part of the logic, calls |
| <a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java">WriteFiles</a> |
| and passes your <code>FileBasedSink</code> as a parameter. A user should not need to |
| call <code>WriteFiles</code> directly.</p> |
| </li> |
| </ul> |
| <p>The <code>FileBasedSink</code> abstract base class implements code that is common to Beam |
| sinks that interact with files, including:</p> |
| <ul> |
| <li>Setting file headers and footers</li> |
| <li>Sequential record writing</li> |
| <li>Setting the output MIME type</li> |
| </ul> |
| <p><code>FileBasedSink</code> and its subclasses support writing files to any Beam-supported |
| <code>FileSystem</code> implementations. See the following Beam-provided <code>FileBasedSink</code> |
| implementations for examples:</p> |
| <ul> |
| <li><a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSink.java">TextSink</a> and</li> |
| <li><a href="https://github.com/apache/beam/blob/master/sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroSink.java">AvroSink</a>.</li> |
| </ul> |
| <h2 id="ptransform-wrappers">PTransform wrappers</h2> |
| <p>When you create a source or sink that end-users will use, avoid exposing your |
| source or sink code. To avoid exposing your sources and sinks to end-users, your |
| new classes should be protected or private. Then, implement a user-facing |
| wrapper <code>PTransform</code>. By exposing your source or sink as a transform, your |
| implementation is hidden and can be arbitrarily complex or simple. The greatest |
| benefit of not exposing implementation details is that later on, you can add |
| additional functionality without breaking the existing implementation for users.</p> |
| <p>For example, if your users’ pipelines read from your source using |
| <code>read</code> and you want to insert a reshard into the pipeline, all |
| users would need to add the reshard themselves (using the <code>GroupByKey</code> |
| transform). To solve this, we recommended that you expose the source as a |
| composite <code>PTransform</code> that performs both the read operation and the reshard.</p> |
| <p>See Beam’s <a href="/contribute/ptransform-style-guide/#exposing-a-ptransform-vs-something-else">PTransform style guide</a> |
| for additional information about wrapping with a <code>PTransform</code>.</p></description></item><item><title>Documentation: Apache Beam: Developing I/O connectors for Python</title><link>/documentation/io/developing-io-python/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/io/developing-io-python/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="developing-io-connectors-for-python">Developing I/O connectors for Python</h1> |
| <p><strong>IMPORTANT:</strong> Please use <code>Splittable DoFn</code> to develop your new I/O. For more details, please read |
| the <a href="/documentation/io/developing-io-overview/">new I/O connector overview</a>.</p> |
| <p>To connect to a data store that isn’t supported by Beam’s existing I/O |
| connectors, you must create a custom I/O connector that usually consist of a |
| source and a sink. All Beam sources and sinks are composite transforms; however, |
| the implementation of your custom I/O depends on your use case. Before you |
| start, read the <a href="/documentation/io/developing-io-overview/">new I/O connector overview</a> |
| for an overview of developing a new I/O connector, the available implementation |
| options, and how to choose the right option for your use case.</p> |
| <p>This guide covers using the <a href="https://beam.apache.org/releases/pydoc/2.56.0/apache_beam.io.iobase.html">Source and FileBasedSink interfaces</a> |
| for Python. The Java SDK offers the same functionality, but uses a slightly |
| different API. See <a href="/documentation/io/developing-io-java/">Developing I/O connectors for Java</a> |
| for information specific to the Java SDK.</p> |
| <h2 id="basic-code-reqs">Basic code requirements</h2> |
| <p>Beam runners use the classes you provide to read and/or write data using |
| multiple worker instances in parallel. As such, the code you provide for |
| <code>Source</code> and <code>FileBasedSink</code> subclasses must meet some basic requirements:</p> |
| <ol> |
| <li> |
| <p><strong>Serializability:</strong> Your <code>Source</code> or <code>FileBasedSink</code> subclass must be |
| serializable. The service may create multiple instances of your <code>Source</code> |
| or <code>FileBasedSink</code> subclass to be sent to multiple remote workers to |
| facilitate reading or writing in parallel. The <em>way</em> the source and sink |
| objects are serialized is runner specific.</p> |
| </li> |
| <li> |
| <p><strong>Immutability:</strong> Your <code>Source</code> or <code>FileBasedSink</code> subclass must be |
| effectively immutable. You should only use mutable state in your <code>Source</code> |
| or <code>FileBasedSink</code> subclass if you are using lazy evaluation of expensive |
| computations that you need to implement the source.</p> |
| </li> |
| <li> |
| <p><strong>Thread-Safety:</strong> Your code must be thread-safe. The Beam SDK for Python |
| provides the <code>RangeTracker</code> class to make this easier.</p> |
| </li> |
| <li> |
| <p><strong>Testability:</strong> It is critical to exhaustively unit-test all of your |
| <code>Source</code> and <code>FileBasedSink</code> subclasses. A minor implementation error can |
| lead to data corruption or data loss (such as skipping or duplicating |
| records) that can be hard to detect. You can use test harnesses and utility |
| methods available in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/source_test_utils.py">source_test_utils module</a> |
| to develop tests for your source.</p> |
| </li> |
| </ol> |
| <p>In addition, see the <a href="/contribute/ptransform-style-guide/">PTransform style guide</a> |
| for Beam&rsquo;s transform style guidance.</p> |
| <h2 id="implementing-the-source-interface">Implementing the Source interface</h2> |
| <p>To create a new data source for your pipeline, you&rsquo;ll need to provide the format-specific logic that tells the service how to read data from your input source, and how to split your data source into multiple parts so that multiple worker instances can read your data in parallel.</p> |
| <p>Supply the logic for your new source by creating the following classes:</p> |
| <ul> |
| <li>A subclass of <code>BoundedSource</code>. <code>BoundedSource</code> is a source that reads a |
| finite amount of input records. The class describes the data you want to |
| read, including the data&rsquo;s location and parameters (such as how much data to |
| read).</li> |
| <li>A subclass of <code>RangeTracker</code>. <code>RangeTracker</code> is a thread-safe object used to |
| manage a range for a given position type.</li> |
| <li>One or more user-facing wrapper composite transforms (<code>PTransform</code>) that |
| wrap read operations. <a href="#ptransform-wrappers">PTransform wrappers</a> discusses |
| why you should avoid exposing your sources, and walks through how to create |
| a wrapper.</li> |
| </ul> |
| <p>You can find these classes in the |
| <a href="https://beam.apache.org/releases/pydoc/2.56.0/apache_beam.io.iobase.html">apache_beam.io.iobase module</a>.</p> |
| <h3 id="implementing-the-boundedsource-subclass">Implementing the BoundedSource subclass</h3> |
| <p><code>BoundedSource</code> represents a finite data set from which the service reads, possibly in parallel. <code>BoundedSource</code> contains a set of methods that the service uses to split the data set for reading by multiple remote workers.</p> |
| <p>To implement a <code>BoundedSource</code>, your subclass must override the following methods:</p> |
| <ul> |
| <li> |
| <p><code>estimate_size</code>: Services use this method to estimate the <em>total size</em> of your data, in bytes. This estimate is in terms of external storage size, before performing decompression or other processing.</p> |
| </li> |
| <li> |
| <p><code>split</code>: Service use this method to split your finite data into bundles of a given size.</p> |
| </li> |
| <li> |
| <p><code>get_range_tracker</code>: Services use this method to get the <code>RangeTracker</code> for a given position range, and use the information to report progress and perform dynamic splitting of sources.</p> |
| </li> |
| <li> |
| <p><code>read</code>: This method returns an iterator that reads data from the source, with respect to the boundaries defined by the given <code>RangeTracker</code> object.</p> |
| </li> |
| </ul> |
| <h3 id="implementing-the-rangetracker-subclass">Implementing the RangeTracker subclass</h3> |
| <p>A <code>RangeTracker</code> is a thread-safe object used to manage the current range and current position of the reader of a <code>BoundedSource</code> and protect concurrent access to them.</p> |
| <p>To implement a <code>RangeTracker</code>, you should first familiarize yourself with the following definitions:</p> |
| <ul> |
| <li> |
| <p><strong>Position-based sources</strong> - A position-based source can be described by a range of positions of an ordered type, and the records read by the source can be described by positions of that type. For example, for a record within a file, the position can be the starting byte offset of the record. The position type for the record in this case is <code>long</code>.</p> |
| <p>The main requirement for position-based sources is <strong>associativity</strong>: Reading records in position range &lsquo;[A, B)&rsquo; and records in position range &lsquo;[B, C)&rsquo; should give the same records as reading records in position range &lsquo;[A, C)&rsquo;, where &lsquo;A&rsquo; &lt;= &lsquo;B&rsquo; &lt;= &lsquo;C&rsquo;. This property ensures that no matter how many arbitrary sub-ranges a range of positions is split into, the total set of records they describe stays the same.</p> |
| <p>The other important property is how the source&rsquo;s range relates to positions of records in the source. In many sources each record can be identified by a unique starting position. In this case:</p> |
| <ul> |
| <li>All records returned by a source &lsquo;[A, B)&rsquo; must have starting positions in this range.</li> |
| <li>All but the last record should end within this range. The last record may or may not extend past the end of the range.</li> |
| <li>Records must not overlap.</li> |
| </ul> |
| <p>Such sources should define &ldquo;read &lsquo;[A, B)&rsquo;&rdquo; as &ldquo;read from the first record starting at or after &lsquo;A&rsquo;, up to but not including the first record starting at or after &lsquo;B&rsquo;&rdquo;.</p> |
| <p>Some examples of such sources include reading lines or CSV from a text file, reading keys and values from a database, etc.</p> |
| <p>The concept of <em>split points</em> allows to extend the definitions for dealing with sources where some records cannot be identified by a unique starting position.</p> |
| </li> |
| <li> |
| <p><strong>Split points</strong> - A split point describes a record that is the first one returned when reading the range from and including position <strong>A</strong> up to infinity (i.e. [A, infinity)).</p> |
| <p>Some sources may have records that are not directly addressable. For example, imagine a file format consisting of a sequence of compressed blocks. Each block can be assigned an offset, but records within the block cannot be directly addressed without decompressing the block. Let us refer to this hypothetical format as <em>CBF (Compressed Blocks Format)</em>.</p> |
| <p>Many such formats can still satisfy the associativity property. For example, in CBF, reading [A, B) can mean &ldquo;read all the records in all blocks whose starting offset is in [A, B)&rdquo;.</p> |
| <p>To support such complex formats, Beam introduces the notion of <em>split points</em>. A record is a split point if there exists a position <strong>A</strong> such that the record is the first one to be returned when reading the range [A, infinity). In CBF, the only split points would be the first records in each block.</p> |
| <p>Split points allow us to define the meaning of a record&rsquo;s position and a source&rsquo;s range in the following cases:</p> |
| <ul> |
| <li>For a record that is at a split point, its position is defined to be the largest <strong>A</strong> such that reading a source with the range [A, infinity) returns this record.</li> |
| <li>Positions of other records are only required to be non-decreasing.</li> |
| <li>Reading the source [A, B) must return records starting from the first split point at or after <strong>A</strong>, up to but not including the first split point at or after <strong>B</strong>. In particular, this means that the first record returned by a source MUST always be a split point.</li> |
| <li>Positions of split points must be unique.</li> |
| </ul> |
| <p>As a result, for any decomposition of the full range of the source into position ranges, the total set of records will be the full set of records in the source, and each record will be read exactly once.</p> |
| </li> |
| <li> |
| <p><strong>Consumed positions</strong> - Consumed positions refer to records that have been read.</p> |
| <p>As the source is being read, and records read from it are being passed to the downstream transforms in the pipeline, we say that positions in the source are being <em>consumed</em>. When a reader has read a record (or promised to a caller that a record will be returned), positions up to and including the record&rsquo;s start position are considered <em>consumed</em>.</p> |
| <p>Dynamic splitting can happen only at <em>unconsumed</em> positions. If the reader just returned a record at offset 42 in a file, dynamic splitting can happen only at offset 43 or beyond. Otherwise, that record could be read twice (by the current reader and the reader of the new task).</p> |
| </li> |
| </ul> |
| <h4 id="rangetracker-methods">RangeTracker methods</h4> |
| <p>To implement a <code>RangeTracker</code>, your subclass must override the following methods:</p> |
| <ul> |
| <li> |
| <p><code>start_position</code>: Returns the starting position of the current range, inclusive.</p> |
| </li> |
| <li> |
| <p><code>stop_position</code>: Returns the ending position of the current range, exclusive.</p> |
| </li> |
| <li> |
| <p><code>try_claim</code>: This method is used to determine if a record at a split point is within the range. This method should modify the internal state of the <code>RangeTracker</code> by updating the last-consumed position to the given starting <code>position</code> of the record being read by the source. The method returns true if the given position falls within the current range.</p> |
| </li> |
| <li> |
| <p><code>set_current_position</code>: This method updates the last-consumed position to the given starting position of a record being read by a source. You can invoke this method for records that do not start at split points, and this should modify the internal state of the <code>RangeTracker</code>. If the record starts at a split point, you must invoke <code>try_claim</code> instead of this method.</p> |
| </li> |
| <li> |
| <p><code>position_at_fraction</code>: Given a fraction within the range [0.0, 1.0), this method will return the position at the given fraction compared to the position range [<code>self.start_position</code>, <code>self.stop_position</code>).</p> |
| </li> |
| <li> |
| <p><code>try_split</code>: This method attempts to split the current range into two parts around a suggested position. It is allowed to split at a different position, but in most cases it will split at the suggested position.</p> |
| </li> |
| </ul> |
| <p>This method splits the current range [<code>self.start_position</code>, <code>self.stop_position</code>) into a &ldquo;primary&rdquo; part [<code>self.start_position</code>, <code>split_position</code>), and a &ldquo;residual&rdquo; part [<code>split_position</code>, <code>self.stop_position</code>), assuming that <code>split_position</code> has not been consumed yet.</p> |
| <p>If <code>split_position</code> has already been consumed, the method returns <code>None</code>. Otherwise, it updates the current range to be the primary and returns a tuple (<code>split_position</code>, <code>split_fraction</code>). <code>split_fraction</code> should be the fraction of size of range [<code>self.start_position</code>, <code>split_position</code>) compared to the original (before split) range [<code>self.start_position</code>, <code>self.stop_position</code>).</p> |
| <ul> |
| <li><code>fraction_consumed</code>: Returns the approximate fraction of consumed positions in the source.</li> |
| </ul> |
| <p><strong>Note:</strong> Methods of class <code>iobase.RangeTracker</code> may be invoked by multiple threads, hence this class must be made thread-safe, for example, by using a single lock object.</p> |
| <h3 id="convenience-source-base-classes">Convenience Source base classes</h3> |
| <p>The Beam SDK for Python contains some convenient abstract base classes to help you easily create new sources.</p> |
| <h4 id="filebasedsource">FileBasedSource</h4> |
| <p><code>FileBasedSource</code> is a framework for developing sources for new file types. You can derive your <code>BoundedSource</code> class from the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/filebasedsource.py">FileBasedSource</a> class.</p> |
| <p>To create a source for a new file type, you need to create a sub-class of <code>FileBasedSource</code>. Sub-classes of <code>FileBasedSource</code> must implement the method <code>FileBasedSource.read_records()</code>.</p> |
| <p>See <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/avroio.py">AvroSource</a> for an example implementation of <code>FileBasedSource</code>.</p> |
| <h3 id="reading-from-a-new-source">Reading from a new Source</h3> |
| <p>The following example, <code>CountingSource</code>, demonstrates an implementation of <code>BoundedSource</code> and uses the SDK-provided <code>RangeTracker</code> called <code>OffsetRangeTracker</code>.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>class CountingSource(iobase.BoundedSource): |
| def __init__(self, count): |
| self.records_read = Metrics.counter(self.__class__, &#39;recordsRead&#39;) |
| self._count = count |
| def estimate_size(self): |
| return self._count |
| def get_range_tracker(self, start_position, stop_position): |
| if start_position is None: |
| start_position = 0 |
| if stop_position is None: |
| stop_position = self._count |
| return OffsetRangeTracker(start_position, stop_position) |
| def read(self, range_tracker): |
| for i in range(range_tracker.start_position(), |
| range_tracker.stop_position()): |
| if not range_tracker.try_claim(i): |
| return |
| self.records_read.inc() |
| yield i |
| def split(self, desired_bundle_size, start_position=None, stop_position=None): |
| if start_position is None: |
| start_position = 0 |
| if stop_position is None: |
| stop_position = self._count |
| bundle_start = start_position |
| while bundle_start &lt; stop_position: |
| bundle_stop = min(stop_position, bundle_start + desired_bundle_size) |
| yield iobase.SourceBundle( |
| weight=(bundle_stop - bundle_start), |
| source=self, |
| start_position=bundle_start, |
| stop_position=bundle_stop) |
| bundle_start = bundle_stop</code></pre> |
| </div> |
| </div> |
| <p>To read data from the source in your pipeline, use the <code>Read</code> transform:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>with beam.Pipeline() as pipeline: |
| numbers = pipeline | &#39;ProduceNumbers&#39; &gt;&gt; beam.io.Read(CountingSource(count))</code></pre> |
| </div> |
| </div> |
| <p><strong>Note:</strong> When you create a source that end-users are going to use, we |
| recommended that you do not expose the code for the source itself as |
| demonstrated in the example above. Use a wrapping <code>PTransform</code> instead. |
| <a href="#ptransform-wrappers">PTransform wrappers</a> discusses why you should avoid |
| exposing your sources, and walks through how to create a wrapper.</p> |
| <h2 id="using-the-filebasedsink-abstraction">Using the FileBasedSink abstraction</h2> |
| <p>If your data source uses files, you can implement the <a href="https://beam.apache.org/releases/pydoc/2.56.0/apache_beam.io.filebasedsink.html">FileBasedSink</a> |
| abstraction to create a file-based sink. For other sinks, use <code>ParDo</code>, |
| <code>GroupByKey</code>, and other transforms offered by the Beam SDK for Python. See the |
| <a href="/documentation/io/developing-io-overview/">developing I/O connectors overview</a> |
| for more details.</p> |
| <p>When using the <code>FileBasedSink</code> interface, you must provide the format-specific |
| logic that tells the runner how to write bounded data from your pipeline&rsquo;s |
| <code>PCollection</code>s to an output sink. The runner writes bundles of data in parallel |
| using multiple workers.</p> |
| <p>Supply the logic for your file-based sink by implementing the following classes:</p> |
| <ul> |
| <li> |
| <p>A subclass of the abstract base class <code>FileBasedSink</code>. <code>FileBasedSink</code> |
| describes a location or resource that your pipeline can write to in |
| parallel. To avoid exposing your sink to end-users, use the <code>_</code> prefix when |
| creating your <code>FileBasedSink</code> subclass.</p> |
| </li> |
| <li> |
| <p>A user-facing wrapper <code>PTransform</code> that, as part of the logic, calls |
| <code>Write</code> and passes your <code>FileBasedSink</code> as a parameter. A user should not |
| need to call <code>Write</code> directly.</p> |
| </li> |
| </ul> |
| <p>The <code>FileBasedSink</code> abstract base class implements code that is common to Beam |
| sinks that interact with files, including:</p> |
| <ul> |
| <li>Setting file headers and footers</li> |
| <li>Sequential record writing</li> |
| <li>Setting the output MIME type</li> |
| </ul> |
| <p><code>FileBasedSink</code> and its subclasses support writing files to any Beam-supported |
| <code>FileSystem</code> implementations. See the following Beam-provided <code>FileBasedSink</code> |
| implementation for an example:</p> |
| <ul> |
| <li><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/textio.py">TextSink</a></li> |
| </ul> |
| <h2 id="ptransform-wrappers">PTransform wrappers</h2> |
| <p>When you create a source or sink that end-users will use, avoid exposing your |
| source or sink code. To avoid exposing your sources and sinks to end-users, your |
| new classes should use the <code>_</code> prefix. Then, implement a user-facing |
| wrapper <code>PTransform</code>.`By exposing your source or sink as a transform, your |
| implementation is hidden and can be arbitrarily complex or simple. The greatest |
| benefit of not exposing implementation details is that later on, you can add |
| additional functionality without breaking the existing implementation for users.</p> |
| <p>For example, if your users’ pipelines read from your source using |
| <code>beam.io.Read</code> and you want to insert a reshard into the pipeline, all |
| users would need to add the reshard themselves (using the <code>GroupByKey</code> |
| transform). To solve this, we recommended that you expose the source as a |
| composite <code>PTransform</code> that performs both the read operation and the reshard.</p> |
| <p>See Beam’s <a href="/contribute/ptransform-style-guide/#exposing-a-ptransform-vs-something-else">PTransform style guide</a> |
| for additional information about wrapping with a <code>PTransform</code>.</p> |
| <p>The following examples change the source and sink from the above sections so |
| that they are not exposed to end-users. For the source, rename <code>CountingSource</code> |
| to <code>_CountingSource</code>. Then, create the wrapper <code>PTransform</code>, called |
| <code>ReadFromCountingSource</code>:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>class ReadFromCountingSource(PTransform): |
| def __init__(self, count): |
| super().__init__() |
| self._count = count |
| def expand(self, pcoll): |
| return pcoll | iobase.Read(_CountingSource(self._count))</code></pre> |
| </div> |
| </div> |
| <p>Finally, read from the source:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>with beam.Pipeline() as pipeline: |
| numbers = pipeline | &#39;ProduceNumbers&#39; &gt;&gt; ReadFromCountingSource(count)</code></pre> |
| </div> |
| </div> |
| <p>For the sink, rename <code>SimpleKVSink</code> to <code>_SimpleKVSink</code>. Then, create the wrapper <code>PTransform</code>, called <code>WriteToKVSink</code>:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>class WriteToKVSink(PTransform): |
| def __init__(self, simplekv, url, final_table_name): |
| self._simplekv = simplekv |
| super().__init__() |
| self._url = url |
| self._final_table_name = final_table_name |
| def expand(self, pcoll): |
| return pcoll | iobase.Write( |
| _SimpleKVSink(self._simplekv, self._url, self._final_table_name))</code></pre> |
| </div> |
| </div> |
| <p>Finally, write to the sink:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>with beam.Pipeline(options=PipelineOptions()) as pipeline: |
| kvs = pipeline | &#39;CreateKVs&#39; &gt;&gt; beam.core.Create(KVs) |
| kvs | &#39;WriteToSimpleKV&#39; &gt;&gt; WriteToKVSink( |
| simplekv, &#39;http://url_to_simple_kv/&#39;, final_table_name)</code></pre> |
| </div> |
| </div></description></item><item><title>Documentation: Apache Hadoop Input/Output Format IO</title><link>/documentation/io/built-in/hadoop/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/io/built-in/hadoop/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="hadoop-inputoutput-format-io">Hadoop Input/Output Format IO</h1> |
| <blockquote> |
| <p><strong>IMPORTANT!</strong> Previous implementation of Hadoop Input Format IO, called <code>HadoopInputFormatIO</code>, is deprecated starting from <em>Apache Beam 2.10</em>. Please, use current <code>HadoopFormatIO</code> which supports both <code>InputFormat</code> and <code>OutputFormat</code>.</p> |
| </blockquote> |
| <p>A <code>HadoopFormatIO</code> is a transform for reading data from any source or writing data to any sink that implements Hadoop&rsquo;s <code>InputFormat</code> or <code>OutputFormat</code> accordingly. For example, Cassandra, Elasticsearch, HBase, Redis, Postgres, etc.</p> |
| <p><code>HadoopFormatIO</code> allows you to connect to many data sources/sinks that do not yet have a Beam IO transform. However, <code>HadoopFormatIO</code> has to make several performance trade-offs in connecting to <code>InputFormat</code> or <code>OutputFormat</code>. So, if there is another Beam IO transform for connecting specifically to your data source/sink of choice, we recommend you use that one.</p> |
| <h3 id="reading-using-hadoopformatio">Reading using HadoopFormatIO</h3> |
| <p>You will need to pass a Hadoop <code>Configuration</code> with parameters specifying how the read will occur. Many properties of the <code>Configuration</code> are optional and some are required for certain <code>InputFormat</code> classes, but the following properties must be set for all <code>InputFormat</code> classes:</p> |
| <ul> |
| <li><code>mapreduce.job.inputformat.class</code> - The <code>InputFormat</code> class used to connect to your data source of choice.</li> |
| <li><code>key.class</code> - The <code>Key</code> class returned by the <code>InputFormat</code> in <code>mapreduce.job.inputformat.class</code>.</li> |
| <li><code>value.class</code> - The <code>Value</code> class returned by the <code>InputFormat</code> in <code>mapreduce.job.inputformat.class</code>.</li> |
| </ul> |
| <p>For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">Configuration</span> <span class="n">myHadoopConfiguration</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Configuration</span><span class="o">(</span><span class="kc">false</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Set Hadoop InputFormat, key and value class in configuration |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">myHadoopConfiguration</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;mapreduce.job.inputformat.class&#34;</span><span class="o">,</span> <span class="n">InputFormatClass</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">InputFormat</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">myHadoopConfiguration</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;key.class&#34;</span><span class="o">,</span> <span class="n">InputFormatKeyClass</span><span class="o">,</span> <span class="n">Object</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">myHadoopConfiguration</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;value.class&#34;</span><span class="o">,</span> <span class="n">InputFormatValueClass</span><span class="o">,</span> <span class="n">Object</span><span class="o">.</span><span class="na">class</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>You will need to check if the <code>Key</code> and <code>Value</code> classes output by the <code>InputFormat</code> have a Beam <code>Coder</code> available. If not, you can use <code>withKeyTranslation</code> or <code>withValueTranslation</code> to specify a method transforming instances of those classes into another class that is supported by a Beam <code>Coder</code>. These settings are optional and you don&rsquo;t need to specify translation for both key and value.</p> |
| <p>For example:</p> |
| <p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">SimpleFunction</span><span class="o">&lt;</span><span class="n">InputFormatKeyClass</span><span class="o">,</span> <span class="n">MyKeyClass</span><span class="o">&gt;</span> <span class="n">myOutputKeyType</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"><span class="k">new</span> <span class="n">SimpleFunction</span><span class="o">&lt;</span><span class="n">InputFormatKeyClass</span><span class="o">,</span> <span class="n">MyKeyClass</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">MyKeyClass</span> <span class="nf">apply</span><span class="o">(</span><span class="n">InputFormatKeyClass</span> <span class="n">input</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ...logic to transform InputFormatKeyClass to MyKeyClass |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">};</span> |
| </span></span><span class="line"><span class="cl"><span class="n">SimpleFunction</span><span class="o">&lt;</span><span class="n">InputFormatValueClass</span><span class="o">,</span> <span class="n">MyValueClass</span><span class="o">&gt;</span> <span class="n">myOutputValueType</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"><span class="k">new</span> <span class="n">SimpleFunction</span><span class="o">&lt;</span><span class="n">InputFormatValueClass</span><span class="o">,</span> <span class="n">MyValueClass</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">MyValueClass</span> <span class="nf">apply</span><span class="o">(</span><span class="n">InputFormatValueClass</span> <span class="n">input</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ...logic to transform InputFormatValueClass to MyValueClass |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">};</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| </p> |
| <h4 id="read-data-only-with-hadoop-configuration">Read data only with Hadoop configuration.</h4> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;read&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">HadoopFormatIO</span><span class="o">.&lt;</span><span class="n">InputFormatKeyClass</span><span class="o">,</span> <span class="n">InputFormatKeyClass</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withConfiguration</span><span class="o">(</span><span class="n">myHadoopConfiguration</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="read-data-with-configuration-and-key-translation">Read data with configuration and key translation</h4> |
| <p>For example, a Beam <code>Coder</code> is not available for <code>Key</code> class, so key translation is required.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;read&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">HadoopFormatIO</span><span class="o">.&lt;</span><span class="n">MyKeyClass</span><span class="o">,</span> <span class="n">InputFormatKeyClass</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withConfiguration</span><span class="o">(</span><span class="n">myHadoopConfiguration</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyTranslation</span><span class="o">(</span><span class="n">myOutputKeyType</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="read-data-with-configuration-and-value-translation">Read data with configuration and value translation</h4> |
| <p>For example, a Beam <code>Coder</code> is not available for <code>Value</code> class, so value translation is required.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;read&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">HadoopFormatIO</span><span class="o">.&lt;</span><span class="n">InputFormatKeyClass</span><span class="o">,</span> <span class="n">MyValueClass</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withConfiguration</span><span class="o">(</span><span class="n">myHadoopConfiguration</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueTranslation</span><span class="o">(</span><span class="n">myOutputValueType</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="read-data-with-configuration-value-translation-and-key-translation">Read data with configuration, value translation and key translation</h4> |
| <p>For example, Beam Coders are not available for both <code>Key</code> class and <code>Value</code> classes of <code>InputFormat</code>, so key and value translation are required.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;read&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">HadoopFormatIO</span><span class="o">.&lt;</span><span class="n">MyKeyClass</span><span class="o">,</span> <span class="n">MyValueClass</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withConfiguration</span><span class="o">(</span><span class="n">myHadoopConfiguration</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyTranslation</span><span class="o">(</span><span class="n">myOutputKeyType</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueTranslation</span><span class="o">(</span><span class="n">myOutputValueType</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h1 id="examples-for-specific-inputformats">Examples for specific InputFormats</h1> |
| <h3 id="cassandra---cqlinputformat">Cassandra - CqlInputFormat</h3> |
| <p>To read data from Cassandra, use <code>org.apache.cassandra.hadoop.cql3.CqlInputFormat</code>, which needs the following properties to be set:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">Configuration</span> <span class="n">cassandraConf</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Configuration</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">cassandraConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;cassandra.input.thrift.port&#34;</span><span class="o">,</span> <span class="s">&#34;9160&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">cassandraConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;cassandra.input.thrift.address&#34;</span><span class="o">,</span> <span class="n">CassandraHostIp</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">cassandraConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;cassandra.input.partitioner.class&#34;</span><span class="o">,</span> <span class="s">&#34;Murmur3Partitioner&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">cassandraConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;cassandra.input.keyspace&#34;</span><span class="o">,</span> <span class="s">&#34;myKeySpace&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">cassandraConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;cassandra.input.columnfamily&#34;</span><span class="o">,</span> <span class="s">&#34;myColumnFamily&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">cassandraConf</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;key.class&#34;</span><span class="o">,</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">Long</span> <span class="n">Long</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">Object</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">cassandraConf</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;value.class&#34;</span><span class="o">,</span> <span class="n">com</span><span class="o">.</span><span class="na">datastax</span><span class="o">.</span><span class="na">driver</span><span class="o">.</span><span class="na">core</span><span class="o">.</span><span class="na">Row</span> <span class="n">Row</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">Object</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">cassandraConf</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;mapreduce.job.inputformat.class&#34;</span><span class="o">,</span> <span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">cassandra</span><span class="o">.</span><span class="na">hadoop</span><span class="o">.</span><span class="na">cql3</span><span class="o">.</span><span class="na">CqlInputFormat</span> <span class="n">CqlInputFormat</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">InputFormat</span><span class="o">.</span><span class="na">class</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Call Read transform as follows:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;&gt;</span> <span class="n">cassandraData</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;read&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">HadoopFormatIO</span><span class="o">.&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withConfiguration</span><span class="o">(</span><span class="n">cassandraConf</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueTranslation</span><span class="o">(</span><span class="n">cassandraOutputValueType</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>The <code>CqlInputFormat</code> key class is <code>java.lang.Long</code> <code>Long</code>, which has a Beam <code>Coder</code>. The <code>CqlInputFormat</code> value class is <code>com.datastax.driver.core.Row</code> <code>Row</code>, which does not have a Beam <code>Coder</code>. Rather than write a new coder, you can provide your own translation method, as follows:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">SimpleFunction</span><span class="o">&lt;</span><span class="n">Row</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">cassandraOutputValueType</span> <span class="o">=</span> <span class="n">SimpleFunction</span><span class="o">&lt;</span><span class="n">Row</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;()</span> |
| </span></span><span class="line"><span class="cl"><span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">String</span> <span class="nf">apply</span><span class="o">(</span><span class="n">Row</span> <span class="n">row</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">row</span><span class="o">.</span><span class="na">getString</span><span class="o">(</span><span class="err">&#39;</span><span class="n">myColName</span><span class="err">&#39;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">};</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="elasticsearch---esinputformat">Elasticsearch - EsInputFormat</h3> |
| <p>To read data from Elasticsearch, use <code>EsInputFormat</code>, which needs following properties to be set:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">Configuration</span> <span class="n">elasticsearchConf</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Configuration</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">elasticsearchConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;es.nodes&#34;</span><span class="o">,</span> <span class="n">ElasticsearchHostIp</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">elasticsearchConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;es.port&#34;</span><span class="o">,</span> <span class="s">&#34;9200&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">elasticsearchConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;es.resource&#34;</span><span class="o">,</span> <span class="s">&#34;ElasticIndexName/ElasticTypeName&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">elasticsearchConf</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;key.class&#34;</span><span class="o">,</span> <span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">hadoop</span><span class="o">.</span><span class="na">io</span><span class="o">.</span><span class="na">Text</span> <span class="n">Text</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">Object</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">elasticsearchConf</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;value.class&#34;</span><span class="o">,</span> <span class="n">org</span><span class="o">.</span><span class="na">elasticsearch</span><span class="o">.</span><span class="na">hadoop</span><span class="o">.</span><span class="na">mr</span><span class="o">.</span><span class="na">LinkedMapWritable</span> <span class="n">LinkedMapWritable</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">Object</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">elasticsearchConf</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;mapreduce.job.inputformat.class&#34;</span><span class="o">,</span> <span class="n">org</span><span class="o">.</span><span class="na">elasticsearch</span><span class="o">.</span><span class="na">hadoop</span><span class="o">.</span><span class="na">mr</span><span class="o">.</span><span class="na">EsInputFormat</span> <span class="n">EsInputFormat</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">InputFormat</span><span class="o">.</span><span class="na">class</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Call Read transform as follows:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">Text</span><span class="o">,</span> <span class="n">LinkedMapWritable</span><span class="o">&gt;&gt;</span> <span class="n">elasticData</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;read&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">HadoopFormatIO</span><span class="o">.&lt;</span><span class="n">Text</span><span class="o">,</span> <span class="n">LinkedMapWritable</span><span class="o">&gt;</span><span class="n">read</span><span class="o">().</span><span class="na">withConfiguration</span><span class="o">(</span><span class="n">elasticsearchConf</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>The <code>org.elasticsearch.hadoop.mr.EsInputFormat</code>&rsquo;s <code>EsInputFormat</code> key class is <code>org.apache.hadoop.io.Text</code> <code>Text</code>, and its value class is <code>org.elasticsearch.hadoop.mr.LinkedMapWritable</code> <code>LinkedMapWritable</code>. Both key and value classes have Beam Coders.</p> |
| <h3 id="hcatalog---hcatinputformat">HCatalog - HCatInputFormat</h3> |
| <p>To read data using HCatalog, use <code>org.apache.hive.hcatalog.mapreduce.HCatInputFormat</code>, which needs the following properties to be set:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">Configuration</span> <span class="n">hcatConf</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Configuration</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">hcatConf</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;mapreduce.job.inputformat.class&#34;</span><span class="o">,</span> <span class="n">HCatInputFormat</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">InputFormat</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">hcatConf</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;key.class&#34;</span><span class="o">,</span> <span class="n">LongWritable</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">Object</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">hcatConf</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;value.class&#34;</span><span class="o">,</span> <span class="n">HCatRecord</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">Object</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">hcatConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;hive.metastore.uris&#34;</span><span class="o">,</span> <span class="s">&#34;thrift://metastore-host:port&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">hive</span><span class="o">.</span><span class="na">hcatalog</span><span class="o">.</span><span class="na">mapreduce</span><span class="o">.</span><span class="na">HCatInputFormat</span><span class="o">.</span><span class="na">setInput</span><span class="o">(</span><span class="n">hcatConf</span><span class="o">,</span> <span class="s">&#34;my_database&#34;</span><span class="o">,</span> <span class="s">&#34;my_table&#34;</span><span class="o">,</span> <span class="s">&#34;my_filter&#34;</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Call Read transform as follows:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="n">HCatRecord</span><span class="o">&gt;&gt;</span> <span class="n">hcatData</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;read&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">HadoopFormatIO</span><span class="o">.&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="n">HCatRecord</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withConfiguration</span><span class="o">(</span><span class="n">hcatConf</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="amazon-dynamodb---dynamodbinputformat">Amazon DynamoDB - DynamoDBInputFormat</h3> |
| <p>To read data from Amazon DynamoDB, use <code>org.apache.hadoop.dynamodb.read.DynamoDBInputFormat</code>. |
| DynamoDBInputFormat implements the older <code>org.apache.hadoop.mapred.InputFormat</code> interface and to make it compatible with HadoopFormatIO which uses the newer abstract class <code>org.apache.hadoop.mapreduce.InputFormat</code>, |
| a wrapper API is required which acts as an adapter between HadoopFormatIO and DynamoDBInputFormat (or in general any InputFormat implementing <code>org.apache.hadoop.mapred.InputFormat</code>) |
| The below example uses one such available wrapper API - <a href="https://github.com/twitter/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/MapReduceInputFormatWrapper.java">https://github.com/twitter/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/MapReduceInputFormatWrapper.java</a></p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">Configuration</span> <span class="n">dynamoDBConf</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Configuration</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">Job</span> <span class="n">job</span> <span class="o">=</span> <span class="n">Job</span><span class="o">.</span><span class="na">getInstance</span><span class="o">(</span><span class="n">dynamoDBConf</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">com</span><span class="o">.</span><span class="na">twitter</span><span class="o">.</span><span class="na">elephantbird</span><span class="o">.</span><span class="na">mapreduce</span><span class="o">.</span><span class="na">input</span><span class="o">.</span><span class="na">MapReduceInputFormatWrapper</span><span class="o">.</span><span class="na">setInputFormat</span><span class="o">(</span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">hadoop</span><span class="o">.</span><span class="na">dynamodb</span><span class="o">.</span><span class="na">read</span><span class="o">.</span><span class="na">DynamoDBInputFormat</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">job</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">dynamoDBConf</span> <span class="o">=</span> <span class="n">job</span><span class="o">.</span><span class="na">getConfiguration</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">dynamoDBConf</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;key.class&#34;</span><span class="o">,</span> <span class="n">Text</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">WritableComparable</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">dynamoDBConf</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;value.class&#34;</span><span class="o">,</span> <span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">hadoop</span><span class="o">.</span><span class="na">dynamodb</span><span class="o">.</span><span class="na">DynamoDBItemWritable</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">Writable</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">dynamoDBConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;dynamodb.servicename&#34;</span><span class="o">,</span> <span class="s">&#34;dynamodb&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">dynamoDBConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;dynamodb.input.tableName&#34;</span><span class="o">,</span> <span class="s">&#34;table_name&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">dynamoDBConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;dynamodb.endpoint&#34;</span><span class="o">,</span> <span class="s">&#34;dynamodb.us-west-1.amazonaws.com&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">dynamoDBConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;dynamodb.regionid&#34;</span><span class="o">,</span> <span class="s">&#34;us-west-1&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">dynamoDBConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;dynamodb.throughput.read&#34;</span><span class="o">,</span> <span class="s">&#34;1&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">dynamoDBConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;dynamodb.throughput.read.percent&#34;</span><span class="o">,</span> <span class="s">&#34;1&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">dynamoDBConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;dynamodb.version&#34;</span><span class="o">,</span> <span class="s">&#34;2011-12-05&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">dynamoDBConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">DynamoDBConstants</span><span class="o">.</span><span class="na">DYNAMODB_ACCESS_KEY_CONF</span><span class="o">,</span> <span class="s">&#34;aws_access_key&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">dynamoDBConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">DynamoDBConstants</span><span class="o">.</span><span class="na">DYNAMODB_SECRET_KEY_CONF</span><span class="o">,</span> <span class="s">&#34;aws_secret_key&#34;</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Call Read transform as follows:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Text</span><span class="o">,</span> <span class="n">DynamoDBItemWritable</span><span class="o">&gt;</span> <span class="n">dynamoDBData</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;read&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">HadoopFormatIO</span><span class="o">.&lt;</span><span class="n">Text</span><span class="o">,</span> <span class="n">DynamoDBItemWritable</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withConfiguration</span><span class="o">(</span><span class="n">dynamoDBConf</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="apache-hbase---tablesnapshotinputformat">Apache HBase - TableSnapshotInputFormat</h3> |
| <p>To read data from an HBase table snapshot, use <code>org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat</code>. |
| Reading from a table snapshot bypasses the HBase region servers, instead reading HBase data files directly from the filesystem. |
| This is useful for cases such as reading historical data or offloading of work from the HBase cluster. |
| There are scenarios when this may prove faster than accessing content through the region servers using the <code>HBaseIO</code>.</p> |
| <p>A table snapshot can be taken using the HBase shell or programmatically:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="k">try</span> <span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Connection</span> <span class="n">connection</span> <span class="o">=</span> <span class="n">ConnectionFactory</span><span class="o">.</span><span class="na">createConnection</span><span class="o">(</span><span class="n">hbaseConf</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Admin</span> <span class="n">admin</span> <span class="o">=</span> <span class="n">connection</span><span class="o">.</span><span class="na">getAdmin</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">admin</span><span class="o">.</span><span class="na">snapshot</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;my_snaphshot&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TableName</span><span class="o">.</span><span class="na">valueOf</span><span class="o">(</span><span class="s">&#34;my_table&#34;</span><span class="o">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">HBaseProtos</span><span class="o">.</span><span class="na">SnapshotDescription</span><span class="o">.</span><span class="na">Type</span><span class="o">.</span><span class="na">FLUSH</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>A <code>TableSnapshotInputFormat</code> is configured as follows:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Construct a typical HBase scan |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">Scan</span> <span class="n">scan</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Scan</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">scan</span><span class="o">.</span><span class="na">setCaching</span><span class="o">(</span><span class="n">1000</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">scan</span><span class="o">.</span><span class="na">setBatch</span><span class="o">(</span><span class="n">1000</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">scan</span><span class="o">.</span><span class="na">addColumn</span><span class="o">(</span><span class="n">Bytes</span><span class="o">.</span><span class="na">toBytes</span><span class="o">(</span><span class="s">&#34;CF&#34;</span><span class="o">),</span> <span class="n">Bytes</span><span class="o">.</span><span class="na">toBytes</span><span class="o">(</span><span class="s">&#34;col_1&#34;</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"><span class="n">scan</span><span class="o">.</span><span class="na">addColumn</span><span class="o">(</span><span class="n">Bytes</span><span class="o">.</span><span class="na">toBytes</span><span class="o">(</span><span class="s">&#34;CF&#34;</span><span class="o">),</span> <span class="n">Bytes</span><span class="o">.</span><span class="na">toBytes</span><span class="o">(</span><span class="s">&#34;col_2&#34;</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">Configuration</span> <span class="n">hbaseConf</span> <span class="o">=</span> <span class="n">HBaseConfiguration</span><span class="o">.</span><span class="na">create</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">hbaseConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">HConstants</span><span class="o">.</span><span class="na">ZOOKEEPER_QUORUM</span><span class="o">,</span> <span class="s">&#34;zk1:2181&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">hbaseConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;hbase.rootdir&#34;</span><span class="o">,</span> <span class="s">&#34;/hbase&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">hbaseConf</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;mapreduce.job.inputformat.class&#34;</span><span class="o">,</span> <span class="n">TableSnapshotInputFormat</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">InputFormat</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">hbaseConf</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;key.class&#34;</span><span class="o">,</span> <span class="n">ImmutableBytesWritable</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">Writable</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">hbaseConf</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;value.class&#34;</span><span class="o">,</span> <span class="n">Result</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">Writable</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">ClientProtos</span><span class="o">.</span><span class="na">Scan</span> <span class="n">proto</span> <span class="o">=</span> <span class="n">ProtobufUtil</span><span class="o">.</span><span class="na">toScan</span><span class="o">(</span><span class="n">scan</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">hbaseConf</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">TableInputFormat</span><span class="o">.</span><span class="na">SCAN</span><span class="o">,</span> <span class="n">Base64</span><span class="o">.</span><span class="na">encodeBytes</span><span class="o">(</span><span class="n">proto</span><span class="o">.</span><span class="na">toByteArray</span><span class="o">()));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Make use of existing utility methods |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">Job</span> <span class="n">job</span> <span class="o">=</span> <span class="n">Job</span><span class="o">.</span><span class="na">getInstance</span><span class="o">(</span><span class="n">hbaseConf</span><span class="o">);</span> <span class="c1">// creates internal clone of hbaseConf |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">TableSnapshotInputFormat</span><span class="o">.</span><span class="na">setInput</span><span class="o">(</span><span class="n">job</span><span class="o">,</span> <span class="s">&#34;my_snapshot&#34;</span><span class="o">,</span> <span class="k">new</span> <span class="n">Path</span><span class="o">(</span><span class="s">&#34;/tmp/snapshot_restore&#34;</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"><span class="n">hbaseConf</span> <span class="o">=</span> <span class="n">job</span><span class="o">.</span><span class="na">getConfiguration</span><span class="o">();</span> <span class="c1">// extract the modified clone |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Call Read transform as follows:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">ImmutableBytesWritable</span><span class="o">,</span> <span class="n">Result</span><span class="o">&gt;</span> <span class="n">hbaseSnapshotData</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;read&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">HadoopFormatIO</span><span class="o">.&lt;</span><span class="n">ImmutableBytesWritable</span><span class="o">,</span> <span class="n">Result</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withConfiguration</span><span class="o">(</span><span class="n">hbaseConf</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="writing-using-hadoopformatio">Writing using HadoopFormatIO</h3> |
| <p>You will need to pass a Hadoop <code>Configuration</code> with parameters specifying how the write will occur. Many properties of the <code>Configuration</code> are optional, and some are required for certain <code>OutputFormat</code> classes, but the following properties must be set for all <code>OutputFormat</code>s:</p> |
| <ul> |
| <li><code>mapreduce.job.id</code> - The identifier of the write job. E.g.: end timestamp of window.</li> |
| <li><code>mapreduce.job.outputformat.class</code> - The <code>OutputFormat</code> class used to connect to your data sink of choice.</li> |
| <li><code>mapreduce.job.output.key.class</code> - The key class passed to the <code>OutputFormat</code> in <code>mapreduce.job.outputformat.class</code>.</li> |
| <li><code>mapreduce.job.output.value.class</code> - The value class passed to the <code>OutputFormat</code> in <code>mapreduce.job.outputformat.class</code>.</li> |
| <li><code>mapreduce.job.reduces</code> - Number of reduce tasks. Value is equal to number of write tasks which will be generated. This property is not required for <code>Write.PartitionedWriterBuilder#withoutPartitioning()</code> write.</li> |
| <li><code>mapreduce.job.partitioner.class</code> - Hadoop partitioner class which will be used for distributing of records among partitions. This property is not required for <code>Write.PartitionedWriterBuilder#withoutPartitioning()</code> write.</li> |
| </ul> |
| <p><em>Note</em>: All mentioned values have appropriate constants. E.g.: <code>HadoopFormatIO.OUTPUT_FORMAT_CLASS_ATTR</code>.</p> |
| <p>For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">Configuration</span> <span class="n">myHadoopConfiguration</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Configuration</span><span class="o">(</span><span class="kc">false</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Set Hadoop OutputFormat, key and value class in configuration |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">myHadoopConfiguration</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;mapreduce.job.outputformat.class&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MyDbOutputFormatClass</span><span class="o">,</span> <span class="n">OutputFormat</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">myHadoopConfiguration</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;mapreduce.job.output.key.class&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MyDbOutputFormatKeyClass</span><span class="o">,</span> <span class="n">Object</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">myHadoopConfiguration</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;mapreduce.job.output.value.class&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MyDbOutputFormatValueClass</span><span class="o">,</span> <span class="n">Object</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">myHadoopConfiguration</span><span class="o">.</span><span class="na">setClass</span><span class="o">(</span><span class="s">&#34;mapreduce.job.partitioner.class&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MyPartitionerClass</span><span class="o">,</span> <span class="n">Object</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">myHadoopConfiguration</span><span class="o">.</span><span class="na">setInt</span><span class="o">(</span><span class="s">&#34;mapreduce.job.reduces&#34;</span><span class="o">,</span> <span class="n">2</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>You will need to set <code>OutputFormat</code> key and value class (i.e. &ldquo;mapreduce.job.output.key.class&rdquo; and &ldquo;mapreduce.job.output.value.class&rdquo;) in Hadoop <code>Configuration</code> which are equal to <code>KeyT</code> and <code>ValueT</code>. If you set different <code>OutputFormat</code> key or value class than <code>OutputFormat</code>&rsquo;s actual key or value class then, it will throw <code>IllegalArgumentException</code>.</p> |
| <h4 id="batch-writing">Batch writing</h4> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Data which will we want to write |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">Text</span><span class="o">,</span> <span class="n">LongWritable</span><span class="o">&gt;&gt;</span> <span class="n">boundedWordsCount</span> <span class="o">=</span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Hadoop configuration for write |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// We have partitioned write, so Partitioner and reducers count have to be set - see withPartitioning() javadoc |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">Configuration</span> <span class="n">myHadoopConfiguration</span> <span class="o">=</span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Path to directory with locks |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">String</span> <span class="n">locksDirPath</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">boundedWordsCount</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;writeBatch&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">HadoopFormatIO</span><span class="o">.&lt;</span><span class="n">Text</span><span class="o">,</span> <span class="n">LongWritable</span><span class="o">&gt;</span><span class="n">write</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withConfiguration</span><span class="o">(</span><span class="n">myHadoopConfiguration</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPartitioning</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withExternalSynchronization</span><span class="o">(</span><span class="k">new</span> <span class="n">HDFSSynchronization</span><span class="o">(</span><span class="n">locksDirPath</span><span class="o">)));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="stream-writing">Stream writing</h4> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Data which will we want to write |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">Text</span><span class="o">,</span> <span class="n">LongWritable</span><span class="o">&gt;&gt;</span> <span class="n">unboundedWordsCount</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Transformation which transforms data of one window into one hadoop configuration |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PTransform</span><span class="o">&lt;</span><span class="n">PCollection</span><span class="o">&lt;?</span> <span class="kd">extends</span> <span class="n">KV</span><span class="o">&lt;</span><span class="n">Text</span><span class="o">,</span> <span class="n">LongWritable</span><span class="o">&gt;&gt;,</span> <span class="n">PCollectionView</span><span class="o">&lt;</span><span class="n">Configuration</span><span class="o">&gt;&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">configTransform</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">unboundedWordsCount</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;writeStream&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">HadoopFormatIO</span><span class="o">.&lt;</span><span class="n">Text</span><span class="o">,</span> <span class="n">LongWritable</span><span class="o">&gt;</span><span class="n">write</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withConfigurationTransform</span><span class="o">(</span><span class="n">configTransform</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withExternalSynchronization</span><span class="o">(</span><span class="k">new</span> <span class="n">HDFSSynchronization</span><span class="o">(</span><span class="n">locksDirPath</span><span class="o">)));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support Hadoop Input/Output Format IO.</span></span></span></code></pre></div> |
| </div> |
| </div></description></item><item><title>Documentation: Apache HCatalog I/O connector</title><link>/documentation/io/built-in/hcatalog/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/io/built-in/hcatalog/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="hcatalog-io">HCatalog IO</h1> |
| <p>An <code>HCatalogIO</code> is a transform for reading and writing data to an HCatalog managed source.</p> |
| <h3 id="reading-using-hcatalogio">Reading using HCatalogIO</h3> |
| <p>To configure an HCatalog source, you must specify a metastore URI and a table name. Other optional parameters are database and filter.</p> |
| <p>For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">Map</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">configProperties</span> <span class="o">=</span> <span class="k">new</span> <span class="n">HashMap</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">configProperties</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="s">&#34;hive.metastore.uris&#34;</span><span class="o">,</span><span class="s">&#34;thrift://metastore-host:port&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">HCatalogIO</span><span class="o">.</span><span class="na">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withConfigProperties</span><span class="o">(</span><span class="n">configProperties</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withDatabase</span><span class="o">(</span><span class="s">&#34;default&#34;</span><span class="o">)</span> <span class="c1">//optional, assumes default if none specified |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">withTable</span><span class="o">(</span><span class="s">&#34;employee&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withFilter</span><span class="o">(</span><span class="n">filterString</span><span class="o">))</span> <span class="c1">//optional, may be specified if the table is partitioned |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support HCatalogIO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="writing-using-hcatalogio">Writing using HCatalogIO</h3> |
| <p>To configure an <code>HCatalog</code> sink, you must specify a metastore URI and a table name. Other |
| optional parameters are database, partition and batchsize. |
| The destination table should exist beforehand as the transform will not create a new table if missing.</p> |
| <p>For example:</p> |
| <p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">Map</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">configProperties</span> <span class="o">=</span> <span class="k">new</span> <span class="n">HashMap</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">configProperties</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="s">&#34;hive.metastore.uris&#34;</span><span class="o">,</span><span class="s">&#34;thrift://metastore-host:port&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(...)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">HCatalogIO</span><span class="o">.</span><span class="na">write</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withConfigProperties</span><span class="o">(</span><span class="n">configProperties</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withDatabase</span><span class="o">(</span><span class="s">&#34;default&#34;</span><span class="o">)</span> <span class="c1">//optional, assumes default if none specified |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">withTable</span><span class="o">(</span><span class="s">&#34;employee&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPartition</span><span class="o">(</span><span class="n">partitionValues</span><span class="o">)</span> <span class="c1">//optional, may be specified if the table is partitioned |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">withBatchSize</span><span class="o">(</span><span class="n">1024L</span><span class="o">))</span> <span class="c1">//optional, assumes a default batch size of 1024 if none specified |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># The Beam SDK for Python does not support HCatalogIO.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| </p> |
| <h3 id="using-older-versions-of-hcatalog-1x">Using older versions of HCatalog (1.x)</h3> |
| <p><code>HCatalogIO</code> is built for Apache HCatalog versions 2 and up and will not work out of the box for older versions of HCatalog. |
| The following illustrates a workaround to work with Hive 1.1.</p> |
| <p>Include the following Hive 1.2 jars in the uber jar you build. |
| The 1.2 jars provide the necessary methods for Beam while remain compatible with Hive 1.1.</p> |
| <pre tabindex="0"><code>&lt;dependency&gt; |
| &lt;groupId&gt;org.apache.beam&lt;/groupId&gt; |
| &lt;artifactId&gt;beam-sdks-java-io-hcatalog&lt;/artifactId&gt; |
| &lt;version&gt;${beam.version}&lt;/version&gt; |
| &lt;/dependency&gt; |
| &lt;dependency&gt; |
| &lt;groupId&gt;org.apache.hive.hcatalog&lt;/groupId&gt; |
| &lt;artifactId&gt;hive-hcatalog-core&lt;/artifactId&gt; |
| &lt;version&gt;1.2&lt;/version&gt; |
| &lt;/dependency&gt; |
| &lt;dependency&gt; |
| &lt;groupId&gt;org.apache.hive&lt;/groupId&gt; |
| &lt;artifactId&gt;hive-metastore&lt;/artifactId&gt; |
| &lt;version&gt;1.2&lt;/version&gt; |
| &lt;/dependency&gt; |
| &lt;dependency&gt; |
| &lt;groupId&gt;org.apache.hive&lt;/groupId&gt; |
| &lt;artifactId&gt;hive-exec&lt;/artifactId&gt; |
| &lt;version&gt;1.2&lt;/version&gt; |
| &lt;/dependency&gt; |
| &lt;dependency&gt; |
| &lt;groupId&gt;org.apache.hive&lt;/groupId&gt; |
| &lt;artifactId&gt;hive-common&lt;/artifactId&gt; |
| &lt;version&gt;1.2&lt;/version&gt; |
| &lt;/dependency&gt; |
| </code></pre><p>Relocate <em>only</em> the following hive packages:</p> |
| <pre tabindex="0"><code>&lt;plugin&gt; |
| &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt; |
| &lt;artifactId&gt;maven-shade-plugin&lt;/artifactId&gt; |
| &lt;version&gt;${maven-shade-plugin.version}&lt;/version&gt; |
| &lt;configuration&gt; |
| &lt;createDependencyReducedPom&gt;false&lt;/createDependencyReducedPom&gt; |
| &lt;filters&gt; |
| &lt;filter&gt; |
| &lt;artifact&gt;*:*&lt;/artifact&gt; |
| &lt;excludes&gt; |
| &lt;exclude&gt;META-INF/*.SF&lt;/exclude&gt; |
| &lt;exclude&gt;META-INF/*.DSA&lt;/exclude&gt; |
| &lt;exclude&gt;META-INF/*.RSA&lt;/exclude&gt; |
| &lt;/excludes&gt; |
| &lt;/filter&gt; |
| &lt;/filters&gt; |
| &lt;/configuration&gt; |
| &lt;executions&gt; |
| &lt;execution&gt; |
| &lt;phase&gt;package&lt;/phase&gt; |
| &lt;goals&gt; |
| &lt;goal&gt;shade&lt;/goal&gt; |
| &lt;/goals&gt; |
| &lt;configuration&gt; |
| &lt;shadedArtifactAttached&gt;true&lt;/shadedArtifactAttached&gt; |
| &lt;shadedClassifierName&gt;shaded&lt;/shadedClassifierName&gt; |
| &lt;transformers&gt; |
| &lt;transformer implementation=&#34;org.apache.maven.plugins.shade.resource.ServicesResourceTransformer&#34;/&gt; |
| &lt;/transformers&gt; |
| &lt;relocations&gt; |
| &lt;!-- Important: Do not relocate org.apache.hadoop.hive --&gt; |
| &lt;relocation&gt; |
| &lt;pattern&gt;org.apache.hadoop.hive.conf&lt;/pattern&gt; |
| &lt;shadedPattern&gt;h12.org.apache.hadoop.hive.conf&lt;/shadedPattern&gt; |
| &lt;/relocation&gt; |
| &lt;relocation&gt; |
| &lt;pattern&gt;org.apache.hadoop.hive.ql&lt;/pattern&gt; |
| &lt;shadedPattern&gt;h12.org.apache.hadoop.hive.ql&lt;/shadedPattern&gt; |
| &lt;/relocation&gt; |
| &lt;relocation&gt; |
| &lt;pattern&gt;org.apache.hadoop.hive.metastore&lt;/pattern&gt; |
| &lt;shadedPattern&gt;h12.org.apache.hadoop.hive.metastore&lt;/shadedPattern&gt; |
| &lt;/relocation&gt; |
| &lt;/relocations&gt; |
| &lt;/configuration&gt; |
| &lt;/execution&gt; |
| &lt;/executions&gt; |
| &lt;/plugin&gt; |
| </code></pre><p>This has been testing to read SequenceFile and ORCFile file backed tables running with |
| Beam 2.4.0 on Spark 2.3 / YARN in a Cloudera CDH 5.12.2 managed environment.</p></description></item><item><title>Documentation: Apache Parquet I/O connector</title><link>/documentation/io/built-in/parquet/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/io/built-in/parquet/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p><a href="/documentation/io/built-in/">Built-in I/O Transforms</a></p> |
| <h1 id="apache-parquet-io-connector">Apache Parquet I/O connector</h1> |
| <nav class="language-switcher"> |
| <strong>Adapt for:</strong> |
| <ul> |
| <li data-value="java" class="active">Java SDK</li> |
| <li data-value="py">Python SDK</li> |
| </ul> |
| </nav> |
| <p>The Beam SDKs include built-in transforms that can read data from and write data |
| to <a href="https://parquet.apache.org">Apache Parquet</a> files.</p> |
| <h2 id="before-you-start">Before you start</h2> |
| <!-- Java specific --> |
| <p class="language-java">To use ParquetIO, add the Maven artifact dependency to your <code>pom.xml</code> file.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="o">&lt;</span><span class="n">dependency</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">groupId</span><span class="o">&gt;</span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">beam</span><span class="o">&lt;/</span><span class="n">groupId</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">artifactId</span><span class="o">&gt;</span><span class="n">beam</span><span class="o">-</span><span class="n">sdks</span><span class="o">-</span><span class="n">java</span><span class="o">-</span><span class="n">io</span><span class="o">-</span><span class="n">parquet</span><span class="o">&lt;/</span><span class="n">artifactId</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">version</span><span class="o">&gt;</span><span class="n">2</span><span class="o">.</span><span class="na">56</span><span class="o">.</span><span class="na">0</span><span class="o">&lt;/</span><span class="n">version</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"><span class="o">&lt;/</span><span class="n">dependency</span><span class="o">&gt;</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">Additional resources:</p> |
| <span class="language-java"><ul> |
| <li><a href="https://github.com/apache/beam/blob/master/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java">ParquetIO source code</a></li> |
| <li><a href="https://beam.apache.org/releases/javadoc/2.56.0/org/apache/beam/sdk/io/parquet/ParquetIO.html">ParquetIO Javadoc</a></li> |
| </ul> |
| </span> |
| <!-- Python specific --> |
| <p class="language-py">ParquetIO comes preinstalled with the Apache Beam python sdk..2.56.0</p> |
| <p class="language-py">Additional resources:</p> |
| <span class="language-py"><ul> |
| <li><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/parquetio.py">ParquetIO source code</a></li> |
| <li><a href="https://beam.apache.org/releases/pydoc/2.56.0/apache_beam.io.parquetio.html">ParquetIO Pydoc</a></li> |
| </ul> |
| </span> |
| <p class="language-java"><h4 id="using-parquetio-with-spark-before-24">Using ParquetIO with Spark before 2.4</h4> |
| </p> |
| <p class="language-java"><code>ParquetIO</code> depends on an API introduced in Apache Parquet 1.10.0. <strong>Spark 2.4.x is compatible and no additional steps are necessary</strong>. Older versions of Spark will not work out of the box since a pre-installed version of Parquet libraries will take precedence during execution. The following workaround should be applied.</p> |
| <p class="language-java"><blockquote> |
| <p><strong>Note</strong>: The following technique allows you to execute your pipeline with <code>ParquetIO</code> correctly. |
| The Parquet files that are consumed or generated by this Beam connector should remain interoperable with the other tools on your cluster.</p> |
| </blockquote> |
| </p> |
| <p class="language-java">Include the Parquet artifact normally and ensure that it brings in the correct version of Parquet as a transitive dependency.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="o">&lt;</span><span class="n">dependency</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">groupId</span><span class="o">&gt;</span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">beam</span><span class="o">&lt;/</span><span class="n">groupId</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">artifactId</span><span class="o">&gt;</span><span class="n">beam</span><span class="o">-</span><span class="n">sdks</span><span class="o">-</span><span class="n">java</span><span class="o">-</span><span class="n">io</span><span class="o">-</span><span class="n">parquet</span><span class="o">&lt;/</span><span class="n">artifactId</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">version</span><span class="o">&gt;</span><span class="n">$</span><span class="o">{</span><span class="n">beam</span><span class="o">.</span><span class="na">version</span><span class="o">}&lt;/</span><span class="n">version</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"><span class="o">&lt;/</span><span class="n">dependency</span><span class="o">&gt;</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">Relocate the following packages:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="o">&lt;</span><span class="n">plugin</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">groupId</span><span class="o">&gt;</span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">maven</span><span class="o">.</span><span class="na">plugins</span><span class="o">&lt;/</span><span class="n">groupId</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">artifactId</span><span class="o">&gt;</span><span class="n">maven</span><span class="o">-</span><span class="n">shade</span><span class="o">-</span><span class="n">plugin</span><span class="o">&lt;/</span><span class="n">artifactId</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">configuration</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">createDependencyReducedPom</span><span class="o">&gt;</span><span class="kc">false</span><span class="o">&lt;/</span><span class="n">createDependencyReducedPom</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">filters</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">filter</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">artifact</span><span class="o">&gt;*:*&lt;/</span><span class="n">artifact</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">excludes</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">exclude</span><span class="o">&gt;</span><span class="n">META</span><span class="o">-</span><span class="n">INF</span><span class="o">/*.</span><span class="na">SF</span><span class="o">&lt;/</span><span class="n">exclude</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">exclude</span><span class="o">&gt;</span><span class="n">META</span><span class="o">-</span><span class="n">INF</span><span class="o">/*.</span><span class="na">DSA</span><span class="o">&lt;/</span><span class="n">exclude</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">exclude</span><span class="o">&gt;</span><span class="n">META</span><span class="o">-</span><span class="n">INF</span><span class="o">/*.</span><span class="na">RSA</span><span class="o">&lt;/</span><span class="n">exclude</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;/</span><span class="n">excludes</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;/</span><span class="n">filter</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;/</span><span class="n">filters</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;/</span><span class="n">configuration</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">executions</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">execution</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">phase</span><span class="o">&gt;</span><span class="n">package</span><span class="o">&lt;/</span><span class="n">phase</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">goals</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">goal</span><span class="o">&gt;</span><span class="n">shade</span><span class="o">&lt;/</span><span class="n">goal</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;/</span><span class="n">goals</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">configuration</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">shadedArtifactAttached</span><span class="o">&gt;</span><span class="kc">true</span><span class="o">&lt;/</span><span class="n">shadedArtifactAttached</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">shadedClassifierName</span><span class="o">&gt;</span><span class="n">shaded</span><span class="o">&lt;/</span><span class="n">shadedClassifierName</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">relocations</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">relocation</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">pattern</span><span class="o">&gt;</span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">parquet</span><span class="o">&lt;/</span><span class="n">pattern</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">shadedPattern</span><span class="o">&gt;</span><span class="n">shaded</span><span class="o">.</span><span class="na">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">parquet</span><span class="o">&lt;/</span><span class="n">shadedPattern</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;/</span><span class="n">relocation</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;!--</span> <span class="n">Some</span> <span class="n">packages</span> <span class="n">are</span> <span class="n">shaded</span> <span class="n">already</span><span class="o">,</span> <span class="n">and</span> <span class="n">on</span> <span class="n">the</span> <span class="n">original</span> <span class="n">spark</span> <span class="n">classpath</span><span class="o">.</span> <span class="n">Shade</span> <span class="n">them</span> <span class="n">more</span><span class="o">.</span> <span class="o">--&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">relocation</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">pattern</span><span class="o">&gt;</span><span class="n">shaded</span><span class="o">.</span><span class="na">parquet</span><span class="o">&lt;/</span><span class="n">pattern</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">shadedPattern</span><span class="o">&gt;</span><span class="n">reshaded</span><span class="o">.</span><span class="na">parquet</span><span class="o">&lt;/</span><span class="n">shadedPattern</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;/</span><span class="n">relocation</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">relocation</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">pattern</span><span class="o">&gt;</span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">avro</span><span class="o">&lt;/</span><span class="n">pattern</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">shadedPattern</span><span class="o">&gt;</span><span class="n">shaded</span><span class="o">.</span><span class="na">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">avro</span><span class="o">&lt;/</span><span class="n">shadedPattern</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;/</span><span class="n">relocation</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;/</span><span class="n">relocations</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">transformers</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">transformer</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">implementation</span><span class="o">=</span><span class="s">&#34;org.apache.maven.plugins.shade.resource.ServicesResourceTransformer&#34;</span><span class="o">/&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;/</span><span class="n">transformers</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;/</span><span class="n">configuration</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;/</span><span class="n">execution</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;/</span><span class="n">executions</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"><span class="o">&lt;/</span><span class="n">plugin</span><span class="o">&gt;</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">This technique has been tested to work on Spark 2.2.3, Spark 2.3.3 and Spark 2.4.3 (although it is optional for Spark 2.4+).</p></description></item><item><title>Documentation: Apache SingleStore I/O connector</title><link>/documentation/io/built-in/singlestore/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/io/built-in/singlestore/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p><a href="/documentation/io/built-in/">Built-in I/O Transforms</a></p> |
| <h1 id="singlestoredb-io">SingleStoreDB I/O</h1> |
| <p>Pipeline options and general information about using and running SingleStoreDB I/O.</p> |
| <h2 id="before-you-start">Before you start</h2> |
| <p>To use SingleStoreDB I/O, add the Maven artifact dependency to your <code>pom.xml</code> file.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>&lt;dependency&gt; |
| &lt;groupId&gt;org.apache.beam&lt;/groupId&gt; |
| &lt;artifactId&gt;beam-sdks-java-io-singlestore&lt;/artifactId&gt; |
| &lt;version&gt;2.56.0&lt;/version&gt; |
| &lt;/dependency&gt;</code></pre> |
| </div> |
| </div> |
| <p>Additional resources:</p> |
| <ul> |
| <li><a href="https://github.com/apache/beam/tree/master/sdks/java/io/singlestore/src/main/java/org/apache/beam/sdk/io/singlestore">SingleStoreIO source code</a></li> |
| <li><a href="https://beam.apache.org/releases/javadoc/2.56.0/org/apache/beam/sdk/io/singlestore/SingleStoreIO.html">SingleStoreIO Javadoc</a></li> |
| <li><a href="https://docs.singlestore.com/">SingleStore documentation</a></li> |
| </ul> |
| <h2 id="authentication">Authentication</h2> |
| <p>DataSource configuration is required for configuring SingleStoreIO connection properties.</p> |
| <p>Create the DataSource configuration: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>SingleStoreIO.DataSourceConfiguration |
| .create(&#34;myHost:3306&#34;) |
| .withDatabase(&#34;db&#34;) |
| .withConnectionProperties(&#34;connectTimeout=30000;useServerPrepStmts=FALSE&#34;) |
| .withPassword(&#34;password&#34;) |
| .withUsername(&#34;admin&#34;);</code></pre> |
| </div> |
| </div> |
| </p> |
| <p>Where parameters can be:</p> |
| <ul> |
| <li><code>.create(endpoint)</code> |
| <ul> |
| <li>Hostname or IP address of the SingleStoreDB in the form host:[port] (port is optional).</li> |
| <li>Required parameter.</li> |
| <li>Example: <code>.create(&quot;myHost:3306&quot;)</code>.</li> |
| </ul> |
| </li> |
| <li><code>.withUsername(username)</code> |
| <ul> |
| <li>SingleStoreDB username.</li> |
| <li>Default - <code>root</code>.</li> |
| <li>Example: <code>.withUsername(&quot;USERNAME&quot;)</code>.</li> |
| </ul> |
| </li> |
| <li><code>.withPassword(password)</code> |
| <ul> |
| <li>Password of the SingleStoreDB user.</li> |
| <li>Default - empty String.</li> |
| <li>Example: <code>.withPassword(&quot;PASSWORD&quot;)</code>.</li> |
| </ul> |
| </li> |
| <li><code>.withDatabase(database)</code> |
| <ul> |
| <li>Name of the SingleStoreDB database to use.</li> |
| <li>Example: <code>.withDatabase(&quot;MY_DATABASE&quot;)</code>.</li> |
| </ul> |
| </li> |
| <li><code>.withConnectionProperties(connectionProperties)</code> |
| <ul> |
| <li>List of properties that are used by JDBC Driver.</li> |
| <li>The format is “key1=value1;key2=value2;&hellip;”.</li> |
| <li>A full list of supported properties can be found <a href="https://docs.singlestore.com/managed-service/en/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.html#connection-string-parameters">here</a>.</li> |
| <li>Example: <code>.withConnectionProperties(&quot;connectTimeout=30000;useServerPrepStmts=FALSE&quot;)</code>.</li> |
| </ul> |
| </li> |
| </ul> |
| <p><strong>Note</strong> - <code>.withDatabase(...)</code> <strong>is required for <code>.readWithPartitions()</code></strong>.</p> |
| <h2 id="reading-from-singlestoredb">Reading from SingleStoreDB</h2> |
| <p>One of the functions of SingleStoreIO is reading from SingleStoreDB tables. |
| SingleStoreIO supports two types of reading:</p> |
| <ul> |
| <li>Sequential data reading (<code>.read()</code>)</li> |
| <li>Parallel data reading (<code>.readWithPartitions()</code>)</li> |
| </ul> |
| <p>In many cases, parallel data reading is preferred over sequential data reading because of performance reasons.</p> |
| <h3 id="sequential-data-reading">Sequential data reading</h3> |
| <p>The basic <code>.read()</code> operation usage is as follows: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>PCollection&lt;USER_DATA_TYPE&gt; items = pipeline.apply( |
| SingleStoreIO.&lt;USER_DATA_TYPE&gt;read() |
| .withDataSourceConfiguration(dc) |
| .withTable(&#34;MY_TABLE&#34;) // or .withQuery(&#34;QUERY&#34;) |
| .withStatementPreparator(statementPreparator) |
| .withOutputParallelization(true) |
| .withRowMapper(mapper) |
| );</code></pre> |
| </div> |
| </div> |
| </p> |
| <p>Where parameters can be:</p> |
| <ul> |
| <li><code>.withDataSourceConfiguration(dataSourceConfiguration)</code> |
| <ul> |
| <li><code>DataSourceConfiguration</code> object with all information needed to establish a connection to the database. See <a href="#authentication">authentication</a> for more information.</li> |
| <li>Required parameter.</li> |
| </ul> |
| </li> |
| <li><code>.withTable(table)</code> |
| <ul> |
| <li>Table to read data from.</li> |
| <li>Example: <code>.withTable(&quot;MY_TABLE&quot;)</code>.</li> |
| </ul> |
| </li> |
| <li><code>.withQuery(query)</code> |
| <ul> |
| <li>SQL query to execute.</li> |
| <li>Example: <code>.withTable(&quot;SELECT * FROM MY_TABLE&quot;)</code>.</li> |
| </ul> |
| </li> |
| <li><code>.withStatementPreparator(statementPreparator)</code> |
| <ul> |
| <li><a href="#statementpreparator">StatementPreparator</a> object.</li> |
| </ul> |
| </li> |
| <li><code>.withRowMapper(rowMapper)</code> |
| <ul> |
| <li><a href="#rowmapper">RowMapper</a> object.</li> |
| <li>Required parameter.</li> |
| </ul> |
| </li> |
| <li><code>.withOutputParallelization(outputParallelization)</code> |
| <ul> |
| <li>Boolean value that indicates whether to reshuffle the result.</li> |
| <li>Default - <code>true</code>.</li> |
| <li>Example: <code>.withOutputParallelization(true)</code>.</li> |
| </ul> |
| </li> |
| </ul> |
| <p><strong>Note</strong> - either <code>.withTable(...)</code> or <code>.withQuery(...)</code> <strong>is required</strong>.</p> |
| <h3 id="parallel-data-reading">Parallel data reading</h3> |
| <p>The basic <code>.readWithPartitions()</code> operation usage is as follows: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>PCollection&lt;USER_DATA_TYPE&gt; items = pipeline.apply( |
| SingleStoreIO.&lt;USER_DATA_TYPE&gt;readWithPartitions() |
| .withDataSourceConfiguration(dc) |
| .withTable(&#34;MY_TABLE&#34;) // or .withQuery(&#34;QUERY&#34;) |
| .withRowMapper(mapper) |
| );</code></pre> |
| </div> |
| </div> |
| </p> |
| <p>Where parameters can be:</p> |
| <ul> |
| <li><code>.withDataSourceConfiguration(dataSourceConfiguration)</code> |
| <ul> |
| <li><code>DataSourceConfiguration</code> object with all information needed to establish a connection to the database. See <a href="#authentication">DataSource Configuration</a> for more information.</li> |
| <li>Required parameter.</li> |
| </ul> |
| </li> |
| <li><code>.withTable(table)</code> |
| <ul> |
| <li>Table to read data from.</li> |
| <li>Example: <code>.withTable(&quot;MY_TABLE&quot;)</code>.</li> |
| </ul> |
| </li> |
| <li><code>.withQuery(query)</code> |
| <ul> |
| <li>SQL query to execute.</li> |
| <li>Example: <code>.withTable(&quot;SELECT * FROM MY_TABLE&quot;)</code>.</li> |
| </ul> |
| </li> |
| <li><code>.withRowMapper(rowMapper)</code> |
| <ul> |
| <li><a href="#rowmapper">RowMapper</a> object.</li> |
| <li>Required parameter.</li> |
| </ul> |
| </li> |
| </ul> |
| <p><strong>Note</strong> - either <code>.withTable(...)</code> or <code>.withQuery(...)</code> <strong>is required</strong>.</p> |
| <h3 id="statementpreparator">StatementPreparator</h3> |
| <p>The <code>StatementPreparator</code> is used by <code>read()</code> to set the parameters of the <code>PreparedStatement</code>. |
| For example: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>public static class MyStatmentPreparator implements SingleStoreIO.StatementPreparator { |
| @Override |
| public void setParameters(PreparedStatement preparedStatement) throws Exception { |
| preparedStatement.setInt(1, 10); |
| } |
| }</code></pre> |
| </div> |
| </div> |
| </p> |
| <h3 id="rowmapper">RowMapper</h3> |
| <p>The <code>RowMapper</code> is used by <code>read()</code> and <code>readWithPartitions()</code> for converting each row of the <code>ResultSet</code> |
| into an element of the resulting <code>PCollection</code>. |
| For example: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>public static class MyRowMapper implements SingleStoreIO.RowMapper&lt;MyRow&gt; { |
| @Override |
| public MyRow mapRow(ResultSet resultSet) throws Exception { |
| return MyRow.create(resultSet.getInt(1), resultSet.getString(2)); |
| } |
| }</code></pre> |
| </div> |
| </div> |
| </p> |
| <h2 id="writing-to-singlestoredb-tables">Writing to SingleStoreDB tables</h2> |
| <p>One of the functions of SingleStoreIO is writing to SingleStoreDB tables. |
| This transformation enables you to send the user&rsquo;s <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/values/PCollection.html">PCollection</a> to your SingleStoreDB database. |
| It returns number of rows written by each batch of elements.</p> |
| <p>The basic <code>.write()</code> operation usage is as follows: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>data.apply( |
| SingleStoreIO.&lt;USER_DATA_TYPE&gt;write() |
| .withDataSourceConfiguration(dc) |
| .withTable(&#34;MY_TABLE&#34;) |
| .withUserDataMapper(mapper) |
| .withBatchSize(100000) |
| );</code></pre> |
| </div> |
| </div> |
| </p> |
| <p>Where parameters can be:</p> |
| <ul> |
| <li><code>.withDataSourceConfiguration(dataSourceConfiguration)</code> |
| <ul> |
| <li><code>DataSourceConfiguration</code> object with all information needed to establish a connection to the database. See <a href="#authentication">DataSource Configuration</a> for more information.</li> |
| <li>Required parameter.</li> |
| </ul> |
| </li> |
| <li><code>.withTable(table)</code> |
| <ul> |
| <li>Table in which data should be saved.</li> |
| <li>Required parameter.</li> |
| <li>Example: <code>.withTable(&quot;MY_TABLE&quot;)</code>.</li> |
| </ul> |
| </li> |
| <li><code>.withBatchSize(batchSize)</code> |
| <ul> |
| <li>Number of rows loaded by one <code>LOAD DATA</code> query.</li> |
| <li>Default - 100000.</li> |
| <li>Example: <code>.withBatchSize(100000)</code>.</li> |
| </ul> |
| </li> |
| <li><code>.withUserDataMapper(userDataMapper)</code> |
| <ul> |
| <li><a href="#userdatamapper">UserDataMapper</a> object.</li> |
| <li>Required parameter.</li> |
| </ul> |
| </li> |
| </ul> |
| <h3 id="userdatamapper">UserDataMapper</h3> |
| <p>The <code>UserDataMapper</code> is required to map data from a <code>PCollection</code> to an array of <code>String</code> values before the <code>write()</code> operation saves the data. |
| For example: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>public static class MyRowDataMapper implements SingleStoreIO.UserDataMapper&lt;MyRow&gt; { |
| @Override |
| public List&lt;String&gt; mapRow(MyRow element) { |
| List&lt;String&gt; res = new ArrayList&lt;&gt;(); |
| res.add(element.id().toString()); |
| res.add(element.name()); |
| return res; |
| } |
| }</code></pre> |
| </div> |
| </div> |
| </p></description></item><item><title>Documentation: Apache Snowflake I/O connector</title><link>/documentation/io/built-in/snowflake/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/io/built-in/snowflake/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p><a href="/documentation/io/built-in/">Built-in I/O Transforms</a></p> |
| <h1 id="snowflake-io">Snowflake I/O</h1> |
| <p>Pipeline options and general information about using and running Snowflake IO.</p> |
| <h2 id="before-you-start">Before you start</h2> |
| <p>To use SnowflakeIO, add the Maven artifact dependency to your <code>pom.xml</code> file.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>&lt;dependency&gt; |
| &lt;groupId&gt;org.apache.beam&lt;/groupId&gt; |
| &lt;artifactId&gt;beam-sdks-java-io-snowflake&lt;/artifactId&gt; |
| &lt;version&gt;2.56.0&lt;/version&gt; |
| &lt;/dependency&gt;</code></pre> |
| </div> |
| </div> |
| <p>Additional resources:</p> |
| <ul> |
| <li><a href="https://github.com/apache/beam/tree/master/sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake">SnowflakeIO source code</a></li> |
| <li><a href="https://beam.apache.org/releases/javadoc/2.56.0/org/apache/beam/sdk/io/snowflake/SnowflakeIO.html">SnowflakeIO Javadoc</a></li> |
| <li><a href="https://docs.snowflake.com/en/">Snowflake documentation</a></li> |
| </ul> |
| <h2 id="authentication">Authentication</h2> |
| <p>Reading and batch writing supports the following authentication methods:</p> |
| <ul> |
| <li>Username and password</li> |
| <li>Key pair</li> |
| <li>OAuth token</li> |
| </ul> |
| <p>Streaming writing supports only key pair authentication. For details, see: <a href="https://issues.apache.org/jira/browse/BEAM-3304">BEAM-3304</a>.</p> |
| <p>Passing credentials is done via Pipeline options used to instantiate <code>SnowflakeIO.DataSourceConfiguration</code> class. Each authentication method has different ways to configure this class.</p> |
| <h3 id="username-and-password">Username and password</h3> |
| <p>To use username/password authentication in SnowflakeIO, invoke your pipeline with the following Pipeline options: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>--username=&lt;USERNAME&gt; --password=&lt;PASSWORD&gt;</code></pre> |
| </div> |
| </div> |
| </p> |
| <p>Passing credentials is done via Pipeline options used to instantiate <code>SnowflakeIO.DataSourceConfiguration</code> class. |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>SnowflakeIO.DataSourceConfiguration datasource = SnowflakeIO.DataSourceConfiguration.create() |
| .withUsernamePasswordAuth( |
| options.getUsername(), |
| options.getPassword()) |
| .withServerName(options.getServerName()) |
| .withDatabase(options.getDatabase()) |
| .withRole(options.getRole()) |
| .withWarehouse(options.getWarehouse()) |
| .withSchema(options.getSchema());</code></pre> |
| </div> |
| </div> |
| </p> |
| <h3 id="key-pair">Key pair</h3> |
| <p>To use this authentication method, you must first generate a key pair and associate the public key with the Snowflake user that will connect using the IO transform. For instructions, see the <a href="https://docs.snowflake.com/en/user-guide/key-pair-auth.html">Key Pair Authentication &amp; Key Pair Rotation</a> in Snowflake documentation.</p> |
| <p>To use key pair authentication with SnowflakeIO, invoke your pipeline with one of the following set of Pipeline options:</p> |
| <ul> |
| <li>with passing the key as a path: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code> --username=&lt;USERNAME&gt; --privateKeyPath=&lt;PATH_TO_P8_FILE&gt; --privateKeyPassphrase=&lt;PASSWORD_FOR_KEY&gt; |
| </code></pre> |
| </div> |
| </div> |
| The initialization of an <code>SnowflakeIO.DataSourceConfiguration</code> class may be as follows: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code> SnowflakeIO.DataSourceConfiguration datasource = SnowflakeIO.DataSourceConfiguration.create() |
| .withKeyPairPathAuth( |
| options.getUsername(), |
| options.getPrivateKeyPath(), |
| options.getPrivateKeyPassphrase()) |
| .withServerName(options.getServerName()) |
| .withDatabase(options.getDatabase()) |
| .withRole(options.getRole()) |
| .withWarehouse(options.getWarehouse()) |
| .withSchema(options.getSchema()); |
| </code></pre> |
| </div> |
| </div> |
| </li> |
| <li>with passing the key as a value: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code> --username=&lt;USERNAME&gt; --rawPrivateKey=&lt;PRIVATE_KEY&gt; --privateKeyPassphrase=&lt;PASSWORD_FOR_KEY&gt; |
| </code></pre> |
| </div> |
| </div> |
| The initialization of an <code>SnowflakeIO.DataSourceConfiguration</code> class may be as follows: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code> SnowflakeIO.DataSourceConfiguration datasource = SnowflakeIO.DataSourceConfiguration.create() |
| .withKeyPairRawAuth( |
| options.getUsername(), |
| options.getRawPrivateKey(), |
| options.getPrivateKeyPassphrase()) |
| .withServerName(options.getServerName()) |
| .withDatabase(options.getDatabase()) |
| .withRole(options.getRole()) |
| .withWarehouse(options.getWarehouse()) |
| .withSchema(options.getSchema()); |
| </code></pre> |
| </div> |
| </div> |
| </li> |
| </ul> |
| <h3 id="oauth-token">OAuth token</h3> |
| <p>SnowflakeIO also supports OAuth token.</p> |
| <p><strong>IMPORTANT</strong>: SnowflakeIO requires a valid OAuth access token. It will neither be able to refresh the token nor obtain it using a web-based flow. For information on configuring an OAuth integration and obtaining the token, see the <a href="https://docs.snowflake.com/en/user-guide/oauth-intro.html">Snowflake documentation</a>.</p> |
| <p>Once you have the token, invoke your pipeline with following Pipeline Options: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>--oauthToken=&lt;TOKEN&gt;</code></pre> |
| </div> |
| </div> |
| The initialization of an <code>SnowflakeIO.DataSourceConfiguration</code> class may be as follows: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code> SnowflakeIO.DataSourceConfiguration datasource = SnowflakeIO.DataSourceConfiguration |
| .create() |
| .withUrl(options.getUrl()) |
| .withServerName(options.getServerName()) |
| .withDatabase(options.getDatabase()) |
| .withWarehouse(options.getWarehouse()) |
| .withSchema(options.getSchema());</code></pre> |
| </div> |
| </div> |
| </p> |
| <h2 id="datasource-configuration">DataSource Configuration</h2> |
| <p>DataSource configuration is required in both read and write object for configuring Snowflake connection properties for IO purposes.</p> |
| <h3 id="general-usage">General usage</h3> |
| <p>Create the DataSource configuration: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code> SnowflakeIO.DataSourceConfiguration |
| .create() |
| .withUrl(options.getUrl()) |
| .withServerName(options.getServerName()) |
| .withDatabase(options.getDatabase()) |
| .withWarehouse(options.getWarehouse()) |
| .withSchema(options.getSchema());</code></pre> |
| </div> |
| </div> |
| Where parameters can be:</p> |
| <ul> |
| <li><code> .withUrl(...)</code> |
| <ul> |
| <li>JDBC-like URL for your Snowflake account, including account name and region, without any parameters.</li> |
| <li>Example: <code>.withUrl(&quot;jdbc:snowflake://account.snowflakecomputing.com&quot;)</code></li> |
| </ul> |
| </li> |
| <li><code>.withServerName(...)</code> |
| <ul> |
| <li>Server Name - full server name with account, zone and domain.</li> |
| <li>Example: <code>.withServerName(&quot;account.snowflakecomputing.com&quot;)</code></li> |
| </ul> |
| </li> |
| <li><code>.withDatabase(...)</code> |
| <ul> |
| <li>Name of the Snowflake database to use.</li> |
| <li>Example: <code>.withDatabase(&quot;MY_DATABASE&quot;)</code></li> |
| </ul> |
| </li> |
| <li><code>.withWarehouse(...)</code> |
| <ul> |
| <li>Name of the Snowflake warehouse to use. This parameter is optional. If no warehouse name is specified, the default warehouse for the user is used.</li> |
| <li>Example: <code>.withWarehouse(&quot;MY_WAREHOUSE&quot;)</code></li> |
| </ul> |
| </li> |
| <li><code>.withSchema(...)</code> |
| <ul> |
| <li>Name of the schema in the database to use. This parameter is optional.</li> |
| <li>Example: <code>.withSchema(&quot;PUBLIC&quot;)</code></li> |
| </ul> |
| </li> |
| <li><code>.withUsernamePasswordAuth(username, password)</code> |
| <ul> |
| <li>Sets username/password authentication.</li> |
| <li>Example: <code>.withUsernamePasswordAuth(&quot;USERNAME&quot;, &quot;PASSWORD&quot;)</code></li> |
| </ul> |
| </li> |
| <li><code>.withOAuth(token)</code> |
| <ul> |
| <li>Sets OAuth authentication.</li> |
| <li>Example: <code>.withOAuth(&quot;TOKEN&quot;)</code></li> |
| </ul> |
| </li> |
| <li><code>.withKeyPairAuth(username, privateKey)</code> |
| <ul> |
| <li>Sets key pair authentication using username and <a href="https://docs.oracle.com/javase/8/docs/api/java/security/PrivateKey.html">PrivateKey</a></li> |
| <li>Example: <code>.withKeyPairAuth(&quot;USERNAME&quot;,</code> <a href="https://docs.oracle.com/javase/8/docs/api/java/security/PrivateKey.html">PrivateKey</a><code>)</code></li> |
| </ul> |
| </li> |
| <li><code>.withKeyPairPathAuth(username, privateKeyPath, privateKeyPassphrase)</code> |
| <ul> |
| <li>Sets key pair authentication using username, path to private key file and passphrase.</li> |
| <li>Example: <code>.withKeyPairPathAuth(&quot;USERNAME&quot;, &quot;PATH/TO/KEY.P8&quot;, &quot;PASSPHRASE&quot;)</code></li> |
| </ul> |
| </li> |
| <li><code>.withKeyPairRawAuth(username, rawPrivateKey, privateKeyPassphrase)</code> |
| <ul> |
| <li>Sets key pair authentication using username, private key and passphrase.</li> |
| <li>Example: <code>.withKeyPairRawAuth(&quot;USERNAME&quot;, &quot;PRIVATE_KEY&quot;, &quot;PASSPHRASE&quot;)</code></li> |
| </ul> |
| </li> |
| </ul> |
| <p><strong>Note</strong> - either <code>.withUrl(...)</code> or <code>.withServerName(...)</code> <strong>is required</strong>.</p> |
| <h2 id="pipeline-options">Pipeline options</h2> |
| <p>Use Beam’s <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/options/PipelineOptions.html">Pipeline options</a> to set options via the command line.</p> |
| <h3 id="snowflake-pipeline-options">Snowflake Pipeline options</h3> |
| <p>Snowflake IO library supports following options that can be passed via the <a href="/documentation/io/built-in/snowflake/#running-main-command-with-pipeline-options">command line</a> by default when a Pipeline uses them:</p> |
| <p><code>--url</code> Snowflake&rsquo;s JDBC-like url including account name and region without any parameters.</p> |
| <p><code>--serverName</code> Full server name with account, zone and domain.</p> |
| <p><code>--username</code> Required for username/password and Private Key authentication.</p> |
| <p><code>--oauthToken</code> Required for OAuth authentication only.</p> |
| <p><code>--password</code> Required for username/password authentication only.</p> |
| <p><code>--privateKeyPath</code> Path to Private Key file. Required for Private Key authentication only.</p> |
| <p><code>--rawPrivateKey</code> Private Key. Required for Private Key authentication only.</p> |
| <p><code>--privateKeyPassphrase</code> Private Key&rsquo;s passphrase. Required for Private Key authentication only.</p> |
| <p><code>--stagingBucketName</code> External bucket path ending with <code>/</code>. I.e. <code>{gs,s3}://bucket/</code>. Sub-directories are allowed.</p> |
| <p><code>--storageIntegrationName</code> Storage integration name</p> |
| <p><code>--warehouse</code> Warehouse to use. Optional.</p> |
| <p><code>--database</code> Database name to connect to. Optional.</p> |
| <p><code>--schema</code> Schema to use. Optional.</p> |
| <p><code>--table</code> Table to use. Optional.</p> |
| <p><code>--query</code> Query to use. Optional.</p> |
| <p><code>--role</code> Role to use. Optional.</p> |
| <p><code>--authenticator</code> Authenticator to use. Optional.</p> |
| <p><code>--portNumber</code> Port number. Optional.</p> |
| <p><code>--loginTimeout</code> Login timeout. Optional.</p> |
| <p><code>--snowPipe</code> SnowPipe name. Optional.</p> |
| <h3 id="running-main-command-with-pipeline-options">Running main command with Pipeline options</h3> |
| <p>To pass Pipeline options via the command line, use <code>--args</code> in a gradle command as follows:</p> |
| <p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>./gradle run |
| --args=&#34; |
| --serverName=&lt;SNOWFLAKE SERVER NAME&gt; |
| Example: --serverName=account.region.gcp.snowflakecomputing.com |
| --username=&lt;SNOWFLAKE USERNAME&gt; |
| Example: --username=testuser |
| --password=&lt;SNOWFLAKE PASSWORD&gt; |
| Example: --password=mypassword |
| --database=&lt;SNOWFLAKE DATABASE&gt; |
| Example: --database=TEST_DATABASE |
| --schema=&lt;SNOWFLAKE SCHEMA&gt; |
| Example: --schema=public |
| --table=&lt;SNOWFLAKE TABLE IN DATABASE&gt; |
| Example: --table=TEST_TABLE |
| --query=&lt;IF NOT TABLE THEN QUERY&gt; |
| Example: --query=‘SELECT column FROM TABLE’ |
| --storageIntegrationName=&lt;SNOWFLAKE STORAGE INTEGRATION NAME&gt; |
| Example: --storageIntegrationName=my_integration |
| --stagingBucketName=&lt;GCS OR S3 BUCKET&gt; |
| Example: --stagingBucketName={gs,s3}://bucket/ |
| --runner=&lt;DirectRunner/DataflowRunner&gt; |
| Example: --runner=DataflowRunner |
| --project=&lt;FOR DATAFLOW RUNNER: GCP PROJECT NAME&gt; |
| Example: --project=my_project |
| --tempLocation=&lt;FOR DATAFLOW RUNNER: GCS TEMP LOCATION STARTING |
| WITH gs://…&gt; |
| Example: --tempLocation=gs://bucket/temp/ |
| --region=&lt;FOR DATAFLOW RUNNER: GCP REGION&gt; |
| Example: --region=us-east-1 |
| --appName=&lt;OPTIONAL: DATAFLOW JOB NAME PREFIX&gt; |
| Example: --appName=my_job&#34;</code></pre> |
| </div> |
| </div> |
| Then in the code it is possible to access the parameters with arguments using the <code>options.getStagingBucketName()</code> command.</p> |
| <h3 id="running-test-command-with-pipeline-options">Running test command with Pipeline options</h3> |
| <p>To pass Pipeline options via the command line, use <code>-DintegrationTestPipelineOptions</code> in a gradle command as follows: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>./gradlew test --tests nameOfTest |
| -DintegrationTestPipelineOptions=&#39;[ |
| &#34;--serverName=&lt;SNOWFLAKE SERVER NAME&gt;&#34;, |
| Example: --serverName=account.region.gcp.snowflakecomputing.com |
| &#34;--username=&lt;SNOWFLAKE USERNAME&gt;&#34;, |
| Example: --username=testuser |
| &#34;--password=&lt;SNOWFLAKE PASSWORD&gt;&#34;, |
| Example: --password=mypassword |
| &#34;--schema=&lt;SNOWFLAKE SCHEMA&gt;&#34;, |
| Example: --schema=PUBLIC |
| &#34;--table=&lt;SNOWFLAKE TABLE IN DATABASE&gt;&#34;, |
| Example: --table=TEST_TABLE |
| &#34;--database=&lt;SNOWFLAKE DATABASE&gt;&#34;, |
| Example: --database=TEST_DATABASE |
| &#34;--storageIntegrationName=&lt;SNOWFLAKE STORAGE INTEGRATION NAME&gt;&#34;, |
| Example: --storageIntegrationName=my_integration |
| &#34;--stagingBucketName=&lt;GCS OR S3 BUCKET&gt;&#34;, |
| Example: --stagingBucketName={gs,s3}://bucket |
| &#34;--externalLocation=&lt;GCS BUCKET URL STARTING WITH GS://&gt;&#34;, |
| Example: --tempLocation=gs://bucket/temp/ |
| ]&#39; --no-build-cache</code></pre> |
| </div> |
| </div> |
| </p> |
| <p>Where all parameters are starting with “&ndash;”, they are surrounded with double quotation and separated with comma:</p> |
| <ul> |
| <li> |
| <p><code>--serverName=&lt;SNOWFLAKE SERVER NAME&gt;</code></p> |
| <ul> |
| <li>Specifies the full name of your account (provided by Snowflake). Note that your full account name might include additional segments that identify the region and cloud platform where your account is hosted.</li> |
| <li>Example: <code>--serverName=xy12345.eu-west-1.gcp..snowflakecomputing.com</code></li> |
| </ul> |
| </li> |
| <li> |
| <p><code>--username=&lt;SNOWFLAKE USERNAME&gt;</code></p> |
| <ul> |
| <li>Specifies the login name of the user.</li> |
| <li>Example: <code>--username=my_username</code></li> |
| </ul> |
| </li> |
| <li> |
| <p><code>--password=&lt;SNOWFLAKE PASSWORD&gt;</code></p> |
| <ul> |
| <li>Specifies the password for the specified user.</li> |
| <li>Example: <code>--password=my_secret</code></li> |
| </ul> |
| </li> |
| <li> |
| <p><code>--schema=&lt;SNOWFLAKE SCHEMA&gt;</code></p> |
| <ul> |
| <li>Specifies the schema to use for the specified database once connected. The specified schema should be an existing schema for which the specified user’s role has privileges.</li> |
| <li>Example: <code>--schema=PUBLIC</code></li> |
| </ul> |
| </li> |
| <li> |
| <p><code>--table=&lt;SNOWFLAKE TABLE IN DATABASE&gt;</code></p> |
| <ul> |
| <li>Example: <code>--table=MY_TABLE</code></li> |
| </ul> |
| </li> |
| <li> |
| <p><code>--database=&lt;SNOWFLAKE DATABASE&gt;</code></p> |
| <ul> |
| <li>Specifies the database to use once connected. The specified database should be an existing database for which the specified user’s role has privileges.</li> |
| <li>Example: <code>--database=MY_DATABASE</code></li> |
| </ul> |
| </li> |
| <li> |
| <p><code>--storageIntegrationName=&lt;SNOWFLAKE STORAGE INTEGRATION NAME&gt;</code></p> |
| <ul> |
| <li>Name of storage integration created in <a href="https://docs.snowflake.com/en/sql-reference/sql/create-storage-integration.html">Snowflake</a> for a cloud storage of choice.</li> |
| <li>Example: <code>--storageIntegrationName=my_google_integration</code></li> |
| </ul> |
| </li> |
| </ul> |
| <h2 id="running-pipelines-on-dataflow">Running pipelines on Dataflow</h2> |
| <p>By default, pipelines are run on <a href="/documentation/runners/direct/">Direct Runner</a> on your local machine. To run a pipeline on <a href="https://cloud.google.com/dataflow/">Google Dataflow</a>, you must provide the following Pipeline options:</p> |
| <ul> |
| <li> |
| <p><code>--runner=DataflowRunner</code></p> |
| <ul> |
| <li>The Dataflow’s specific runner.</li> |
| </ul> |
| </li> |
| <li> |
| <p><code>--project=&lt;GCS PROJECT&gt;</code></p> |
| <ul> |
| <li>Name of the Google Cloud Platform project.</li> |
| </ul> |
| </li> |
| <li> |
| <p><code>--stagingBucketName=&lt;GCS OR S3 BUCKET&gt;</code></p> |
| <ul> |
| <li>Google Cloud Services bucket or AWS S3 bucket where the Beam files will be staged.</li> |
| </ul> |
| </li> |
| <li> |
| <p><code>--maxNumWorkers=5</code></p> |
| <ul> |
| <li>(optional) Maximum number of workers.</li> |
| </ul> |
| </li> |
| <li> |
| <p><code>--appName=&lt;JOB NAME&gt;</code></p> |
| <ul> |
| <li>(optional) Prefix for the job name in the Dataflow Dashboard.</li> |
| </ul> |
| </li> |
| </ul> |
| <p>More pipeline options for Dataflow can be found <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.html">here</a>.</p> |
| <p><strong>Note</strong>: To properly authenticate with Google Cloud, please use <a href="https://cloud.google.com/sdk/gcloud/">gcloud</a> or follow the <a href="https://cloud.google.com/docs/authentication/">Google Cloud documentation</a>.</p> |
| <p><strong>Important</strong>: Please acknowledge <a href="https://cloud.google.com/dataflow/pricing">Google Dataflow pricing</a></p> |
| <h3 id="running-pipeline-templates-on-dataflow">Running pipeline templates on Dataflow</h3> |
| <p>Google Dataflow is supporting <a href="https://cloud.google.com/dataflow/docs/guides/templates/overview">template</a> creation which means staging pipelines on Cloud Storage and running them with ability to pass runtime parameters that are only available during pipeline execution.</p> |
| <p>The process of creating own Dataflow template is following</p> |
| <ol> |
| <li>Create your own pipeline.</li> |
| <li>Create <a href="https://cloud.google.com/dataflow/docs/guides/templates/creating-templates#creating-and-staging-templates">Dataflow template</a> with checking which options SnowflakeIO is supporting at runtime.</li> |
| <li>Run a Dataflow template using <a href="https://cloud.google.com/dataflow/docs/guides/templates/running-templates#using-the-cloud-console">Cloud Console</a>, <a href="https://cloud.google.com/dataflow/docs/guides/templates/running-templates#using-the-rest-api">REST API</a> or <a href="https://cloud.google.com/dataflow/docs/guides/templates/running-templates#using-gcloud">gcloud</a>.</li> |
| </ol> |
| <p>Currently, SnowflakeIO supports following options at runtime:</p> |
| <ul> |
| <li> |
| <p><code>--serverName</code> Full server name with account, zone and domain.</p> |
| </li> |
| <li> |
| <p><code>--username</code> Required for username/password and Private Key authentication.</p> |
| </li> |
| <li> |
| <p><code>--password</code> Required for username/password authentication only.</p> |
| </li> |
| <li> |
| <p><code>--rawPrivateKey</code> Private Key file. Required for Private Key authentication only.</p> |
| </li> |
| <li> |
| <p><code>--privateKeyPassphrase</code> Private Key&rsquo;s passphrase. Required for Private Key authentication only.</p> |
| </li> |
| <li> |
| <p><code>--stagingBucketName</code> external bucket path ending with <code>/</code>. I.e. <code>{gs,s3}://bucket/</code>. Sub-directories are allowed.</p> |
| </li> |
| <li> |
| <p><code>--storageIntegrationName</code> Storage integration name.</p> |
| </li> |
| <li> |
| <p><code>--warehouse</code> Warehouse to use. Optional.</p> |
| </li> |
| <li> |
| <p><code>--database</code> Database name to connect to. Optional.</p> |
| </li> |
| <li> |
| <p><code>--schema</code> Schema to use. Optional.</p> |
| </li> |
| <li> |
| <p><code>--table</code> Table to use. Optional. Note: table is not in default pipeline options.</p> |
| </li> |
| <li> |
| <p><code>--query</code> Query to use. Optional. Note: query is not in default pipeline options.</p> |
| </li> |
| <li> |
| <p><code>--role</code> Role to use. Optional.</p> |
| </li> |
| <li> |
| <p><code>--snowPipe</code> SnowPipe name. Optional.</p> |
| </li> |
| </ul> |
| <p>Currently, SnowflakeIO <strong>doesn&rsquo;t support</strong> following options at runtime:</p> |
| <ul> |
| <li> |
| <p><code>--url</code> Snowflake&rsquo;s JDBC-like url including account name and region without any parameters.</p> |
| </li> |
| <li> |
| <p><code>--oauthToken</code> Required for OAuth authentication only.</p> |
| </li> |
| <li> |
| <p><code>--privateKeyPath</code> Path to Private Key file. Required for Private Key authentication only.</p> |
| </li> |
| <li> |
| <p><code>--authenticator</code> Authenticator to use. Optional.</p> |
| </li> |
| <li> |
| <p><code>--portNumber</code> Port number. Optional.</p> |
| </li> |
| <li> |
| <p><code>--loginTimeout</code> Login timeout. Optional.</p> |
| </li> |
| </ul> |
| <h2 id="writing-to-snowflake-tables">Writing to Snowflake tables</h2> |
| <p>One of the functions of SnowflakeIO is writing to Snowflake tables. This transformation enables you to finish the Beam pipeline with an output operation that sends the user&rsquo;s <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/values/PCollection.html">PCollection</a> to your Snowflake database.</p> |
| <h3 id="batch-write-from-a-bounded-source">Batch write (from a bounded source)</h3> |
| <p>The basic .<code>write()</code> operation usage is as follows: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>data.apply( |
| SnowflakeIO.&lt;type&gt;write() |
| .withDataSourceConfiguration(dc) |
| .to(&#34;MY_TABLE&#34;) |
| .withStagingBucketName(&#34;BUCKET&#34;) |
| .withStorageIntegrationName(&#34;STORAGE INTEGRATION NAME&#34;) |
| .withUserDataMapper(mapper) |
| )</code></pre> |
| </div> |
| </div> |
| Replace type with the data type of the <code>PCollection</code> object to write; for example, <code>SnowflakeIO.&lt;String&gt;</code> for an input <code>PCollection</code> of Strings.</p> |
| <p>All the below parameters are required:</p> |
| <ul> |
| <li> |
| <p><code>.withDataSourceConfiguration()</code> Accepts a DatasourceConfiguration object.</p> |
| </li> |
| <li> |
| <p><code>.to()</code> Accepts the target Snowflake table name.</p> |
| </li> |
| <li> |
| <p><code>.withStagingBucketName()</code> Accepts a cloud bucket path ended with slash. |
| -Example: <code>.withStagingBucketName(&quot;{gs,s3}://bucket/my/dir/&quot;)</code></p> |
| </li> |
| <li> |
| <p><code>.withStorageIntegrationName()</code> Accepts a name of a Snowflake storage integration object created according to Snowflake documentation. Examples: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>CREATE OR REPLACE STORAGE INTEGRATION &#34;test_integration&#34; |
| TYPE = EXTERNAL_STAGE |
| STORAGE_PROVIDER = GCS |
| ENABLED = TRUE |
| STORAGE_ALLOWED_LOCATIONS = (&#39;gcs://bucket/&#39;);</code></pre> |
| </div> |
| </div> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>CREATE STORAGE INTEGRATION &#34;test_integration&#34; |
| TYPE = EXTERNAL_STAGE |
| STORAGE_PROVIDER = S3 |
| ENABLED = TRUE |
| STORAGE_AWS_ROLE_ARN = &#39;&lt;ARN ROLE NAME&gt;&#39; |
| STORAGE_ALLOWED_LOCATIONS = (&#39;s3://bucket/&#39;)</code></pre> |
| </div> |
| </div> |
| Then: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>.withStorageIntegrationName(&#34;test_integration&#34;)</code></pre> |
| </div> |
| </div> |
| </p> |
| </li> |
| <li> |
| <p><code>.withUserDataMapper()</code> Accepts the UserDataMapper function that will map a user&rsquo;s PCollection to an array of String values <code>(String[])</code>.</p> |
| </li> |
| </ul> |
| <p><strong>Note</strong>: |
| SnowflakeIO uses <code>COPY</code> statements behind the scenes to write (using <a href="https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-table.html">COPY to table</a>). StagingBucketName will be used to save CSV files which will end up in Snowflake. Those CSV files will be saved under the “stagingBucketName” path.</p> |
| <p><strong>Optional</strong> for batching:</p> |
| <ul> |
| <li><code>.withQuotationMark()</code> |
| <ul> |
| <li>Default value: <code>‘</code> (single quotation mark).</li> |
| <li>Accepts String with one character. It will surround all text (String) fields saved to CSV. It should be one of the accepted characters by <a href="https://docs.snowflake.com/en/sql-reference/sql/create-file-format.html">Snowflake’s</a> <a href="https://docs.snowflake.com/en/sql-reference/sql/create-file-format.html">FIELD_OPTIONALLY_ENCLOSED_BY</a> parameter (double quotation mark, single quotation mark or none).</li> |
| <li>Example: <code>.withQuotationMark(&quot;'&quot;)</code></li> |
| </ul> |
| </li> |
| </ul> |
| <h3 id="streaming-write-from-unbounded-source">Streaming write (from unbounded source)</h3> |
| <p>It is required to create a <a href="https://docs.snowflake.com/en/user-guide/data-load-snowpipe.html">SnowPipe</a> in the Snowflake console. SnowPipe should use the same integration and the same bucket as specified by <code>.withStagingBucketName</code> and <code>.withStorageIntegrationName</code> methods. The write operation might look as follows: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>data.apply( |
| SnowflakeIO.&lt;type&gt;write() |
| .withStagingBucketName(&#34;BUCKET&#34;) |
| .withStorageIntegrationName(&#34;STORAGE INTEGRATION NAME&#34;) |
| .withDataSourceConfiguration(dc) |
| .withUserDataMapper(mapper) |
| .withSnowPipe(&#34;MY_SNOW_PIPE&#34;) |
| .withFlushTimeLimit(Duration.millis(time)) |
| .withFlushRowLimit(rowsNumber) |
| .withShardsNumber(shardsNumber) |
| )</code></pre> |
| </div> |
| </div> |
| </p> |
| <h4 id="parameters">Parameters</h4> |
| <p><strong>Required</strong> for streaming:</p> |
| <ul> |
| <li> |
| <p><code> .withDataSourceConfiguration()</code></p> |
| <ul> |
| <li>Accepts a DatasourceConfiguration object.</li> |
| </ul> |
| </li> |
| <li> |
| <p><code>.to()</code></p> |
| <ul> |
| <li>Accepts the target Snowflake table name.</li> |
| <li>Example: <code>.to(&quot;MY_TABLE&quot;)</code></li> |
| </ul> |
| </li> |
| <li> |
| <p><code>.withStagingBucketName()</code></p> |
| <ul> |
| <li>Accepts a cloud bucket path ended with slash.</li> |
| <li>Example: <code>.withStagingBucketName(&quot;{gs,s3}://bucket/my/dir/&quot;)</code></li> |
| </ul> |
| </li> |
| <li> |
| <p><code>.withStorageIntegrationName()</code></p> |
| <ul> |
| <li>Accepts a name of a Snowflake storage integration object created according to Snowflake documentation.</li> |
| <li>Example: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>CREATE OR REPLACE STORAGE INTEGRATION &#34;test_integration&#34; |
| TYPE = EXTERNAL_STAGE |
| STORAGE_PROVIDER = GCS |
| ENABLED = TRUE |
| STORAGE_ALLOWED_LOCATIONS = (&#39;gcs://bucket/&#39;);</code></pre> |
| </div> |
| </div> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>CREATE STORAGE INTEGRATION &#34;test_integration&#34; |
| TYPE = EXTERNAL_STAGE |
| STORAGE_PROVIDER = S3 |
| ENABLED = TRUE |
| STORAGE_AWS_ROLE_ARN = &#39;&lt;ARN ROLE NAME&gt;&#39; |
| STORAGE_ALLOWED_LOCATIONS = (&#39;s3://bucket/&#39;)</code></pre> |
| </div> |
| </div> |
| Then: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>.withStorageIntegrationName(&#34;test_integration&#34;)</code></pre> |
| </div> |
| </div> |
| </li> |
| </ul> |
| </li> |
| <li> |
| <p><code>.withSnowPipe()</code></p> |
| <ul> |
| <li> |
| <p>Accepts the target SnowPipe name. <code>.withSnowPipe()</code> accepts the exact name of snowpipe. |
| Example: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>CREATE OR REPLACE PIPE &#34;test_database&#34;.&#34;public&#34;.&#34;test_gcs_pipe&#34; |
| AS COPY INTO stream_table from @streamstage;</code></pre> |
| </div> |
| </div> |
| </p> |
| </li> |
| <li> |
| <p>Then: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>.withSnowPipe(&#34;test_gcs_pipe&#34;)</code></pre> |
| </div> |
| </div> |
| </p> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| <p><strong>Note</strong>: this is important to provide <strong>schema</strong> and <strong>database</strong> names.</p> |
| <ul> |
| <li><code>.withUserDataMapper()</code> |
| <ul> |
| <li>Accepts the <a href="/documentation/io/built-in/snowflake/#userdatamapper-function">UserDataMapper</a> function that will map a user&rsquo;s PCollection to an array of String values <code>(String[]).</code></li> |
| </ul> |
| </li> |
| </ul> |
| <p><strong>Note</strong>:</p> |
| <p>As mentioned before SnowflakeIO uses <a href="https://docs.snowflake.com/en/user-guide/data-load-snowpipe.html">SnowPipe REST calls</a> |
| behind the scenes for writing from unbounded sources. StagingBucketName will be used to save CSV files which will end up in Snowflake. |
| SnowflakeIO is not going to delete created CSV files from path under the “stagingBucketName” either during or after finishing streaming.</p> |
| <p><strong>Optional</strong> for streaming:</p> |
| <ul> |
| <li> |
| <p><code>.withFlushTimeLimit()</code></p> |
| <ul> |
| <li>Default value: 30 seconds</li> |
| <li>Accepts Duration objects with the specified time after each the streaming write will be repeated</li> |
| <li>Example: <code>.withFlushTimeLimit(Duration.millis(180000))</code></li> |
| </ul> |
| </li> |
| <li> |
| <p><code>.withFlushRowLimit()</code></p> |
| <ul> |
| <li>Default value: 10,000 rows</li> |
| <li>Limit of rows written to each staged file</li> |
| <li>Example: <code>.withFlushRowLimit(500000)</code></li> |
| </ul> |
| </li> |
| <li> |
| <p><code>.withShardNumber()</code></p> |
| <ul> |
| <li>Default value: 1 shard</li> |
| <li>Number of files that will be saved in every flush (for purposes of parallel write).</li> |
| <li>Example: <code>.withShardNumber(5)</code></li> |
| </ul> |
| </li> |
| <li> |
| <p><code>.withQuotationMark()</code></p> |
| <ul> |
| <li>Default value: <code>‘</code> (single quotation mark).</li> |
| <li>Accepts String with one character. It will surround all text (String) fields saved to CSV. It should be one of the accepted characters by <a href="https://docs.snowflake.com/en/sql-reference/sql/create-file-format.html">Snowflake’s</a> <a href="https://docs.snowflake.com/en/sql-reference/sql/create-file-format.html">FIELD_OPTIONALLY_ENCLOSED_BY</a> parameter (double quotation mark, single quotation mark or none). Example: .withQuotationMark(&quot;&quot;) (no quotation marks)</li> |
| </ul> |
| </li> |
| <li> |
| <p><code>.withDebugMode()</code></p> |
| <ul> |
| <li>Accepts: |
| <ul> |
| <li><code>SnowflakeIO.StreamingLogLevel.INFO</code> - shows whole info about loaded files</li> |
| <li><code>SnowflakeIO.StreamingLogLevel.ERROR</code> - shows only errors.</li> |
| </ul> |
| </li> |
| <li>Shows logs about streamed files to Snowflake similarly to <a href="https://docs.snowflake.com/en/user-guide/data-load-snowpipe-rest-apis.html#endpoint-insertreport">insertReport</a>. Enabling debug mode may influence performance.</li> |
| <li>Example: <code>.withDebugMode(SnowflakeIO.StreamingLogLevel.INFO)</code></li> |
| </ul> |
| </li> |
| </ul> |
| <p><strong>Important notice</strong>:</p> |
| <ol> |
| <li>Streaming accepts only <strong>key pair authentication</strong>. For details, see: <a href="https://github.com/apache/beam/issues/21287">Issue 21287</a>.</li> |
| <li>The role parameter configured in <code>SnowflakeIO.DataSourceConfiguration</code> object is ignored for streaming writing. For details, see: <a href="https://github.com/apache/beam/issues/21365">Issue 21365</a></li> |
| </ol> |
| <h4 id="flush-time-duration--number-of-rows">Flush time: duration &amp; number of rows</h4> |
| <p>Duration: streaming write will write periodically files on stage according to time duration specified in flush time limit (for example. every 1 minute).</p> |
| <p>Number of rows: files staged for write will have number of rows specified in flush row limit unless the flush time limit will be reached (for example if the limit is 1000 rows and buffer collected 99 rows and the 1-minute flush time passes, the rows will be sent to SnowPipe for insertion).</p> |
| <p>Size of staged files will depend on the rows size and used compression (GZIP).</p> |
| <h3 id="userdatamapper-function">UserDataMapper function</h3> |
| <p>The <code>UserDataMapper</code> function is required to map data from a <code>PCollection</code> to an array of String values before the <code>write()</code> operation saves the data to temporary <code>.csv</code> files. For example: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>public static SnowflakeIO.UserDataMapper&lt;Long&gt; getCsvMapper() { |
| return (SnowflakeIO.UserDataMapper&lt;Long&gt;) recordLine -&gt; new String[] {recordLine.toString()}; |
| }</code></pre> |
| </div> |
| </div> |
| </p> |
| <h3 id="additional-write-options">Additional write options</h3> |
| <h4 id="transformation-query">Transformation query</h4> |
| <p>The <code>.withQueryTransformation()</code> option for the <code>write()</code> operation accepts a SQL query as a String value, which will be performed while transfering data staged in CSV files directly to the target Snowflake table. For information about the transformation SQL syntax, see the <a href="https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-table.html#transformation-parameters">Snowflake Documentation</a>.</p> |
| <p>Usage: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>String query = &#34;SELECT t.$1 from YOUR_TABLE;&#34;; |
| data.apply( |
| SnowflakeIO.&lt;~&gt;write() |
| .withDataSourceConfiguration(dc) |
| .to(&#34;MY_TABLE&#34;) |
| .withStagingBucketName(&#34;BUCKET&#34;) |
| .withStorageIntegrationName(&#34;STORAGE INTEGRATION NAME&#34;) |
| .withUserDataMapper(mapper) |
| .withQueryTransformation(query) |
| )</code></pre> |
| </div> |
| </div> |
| </p> |
| <h4 id="write-disposition">Write disposition</h4> |
| <p>Define the write behaviour based on the table where data will be written to by specifying the <code>.withWriteDisposition(...)</code> option for the <code>write()</code> operation. The following values are supported:</p> |
| <ul> |
| <li> |
| <p><code>APPEND</code> - Default behaviour. Written data is added to the existing rows in the table,</p> |
| </li> |
| <li> |
| <p><code>EMPTY</code> - The target table must be empty; otherwise, the write operation fails,</p> |
| </li> |
| <li> |
| <p><code>TRUNCATE</code> - The write operation deletes all rows from the target table before writing to it.</p> |
| </li> |
| </ul> |
| <p>Example of usage: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>data.apply( |
| SnowflakeIO.&lt;~&gt;write() |
| .withDataSourceConfiguration(dc) |
| .to(&#34;MY_TABLE&#34;) |
| .withStagingBucketName(&#34;BUCKET&#34;) |
| .withStorageIntegrationName(&#34;STORAGE INTEGRATION NAME&#34;) |
| .withUserDataMapper(mapper) |
| .withWriteDisposition(TRUNCATE) |
| )</code></pre> |
| </div> |
| </div> |
| </p> |
| <h4 id="create-disposition">Create disposition</h4> |
| <p>The <code>.withCreateDisposition()</code> option defines the behavior of the write operation if the target table does not exist . The following values are supported:</p> |
| <ul> |
| <li> |
| <p><code>CREATE_IF_NEEDED</code> - default behaviour. The write operation checks whether the specified target table exists; if it does not, the write operation attempts to create the table Specify the schema for the target table using the <code>.withTableSchema()</code> option.</p> |
| </li> |
| <li> |
| <p><code>CREATE_NEVER</code> - The write operation fails if the target table does not exist.</p> |
| </li> |
| </ul> |
| <p>Usage: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>data.apply( |
| SnowflakeIO.&lt;~&gt;write() |
| .withDataSourceConfiguration(dc) |
| .to(&#34;MY_TABLE&#34;) |
| .withStagingBucketName(&#34;BUCKET&#34;) |
| .withStorageIntegrationName(&#34;STORAGE INTEGRATION NAME&#34;) |
| .withUserDataMapper(mapper) |
| .withCreateDisposition(CREATE_NEVER) |
| )</code></pre> |
| </div> |
| </div> |
| </p> |
| <h4 id="table-schema-disposition">Table schema disposition</h4> |
| <p>When the <code>.withCreateDisposition()</code> option is set to <code>CREATE_IF_NEEDED</code>, the <code>.withTableSchema()</code> option enables specifying the schema for the created target table. |
| A table schema is a list of <code>SnowflakeColumn</code> objects with name and type corresponding to column type for each column in the table.</p> |
| <p>Usage: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>SnowflakeTableSchema tableSchema = |
| new SnowflakeTableSchema( |
| SnowflakeColumn.of(&#34;my_date&#34;, new SnowflakeDate(), true), |
| new SnowflakeColumn(&#34;id&#34;, new SnowflakeNumber()), |
| SnowflakeColumn.of(&#34;name&#34;, new SnowflakeText(), true)); |
| data.apply( |
| SnowflakeIO.&lt;~&gt;write() |
| .withDataSourceConfiguration(dc) |
| .to(&#34;MY_TABLE&#34;) |
| .withStagingBucketName(&#34;BUCKET&#34;) |
| .withStorageIntegrationName(&#34;STORAGE INTEGRATION NAME&#34;) |
| .withUserDataMapper(mapper) |
| .withTableSchema(tableSchema) |
| )</code></pre> |
| </div> |
| </div> |
| </p> |
| <h2 id="reading-from-snowflake">Reading from Snowflake</h2> |
| <p>One of the functions of SnowflakeIO is reading Snowflake tables - either full tables via table name or custom data via query. Output of the read transform is a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/values/PCollection.html">PCollection</a> of user-defined data type.</p> |
| <h3 id="general-usage-1">General usage</h3> |
| <p>The basic <code>.read()</code> operation usage: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>PCollection&lt;USER_DATA_TYPE&gt; items = pipeline.apply( |
| SnowflakeIO.&lt;USER_DATA_TYPE&gt;read() |
| .withDataSourceConfiguration(dc) |
| .fromTable(&#34;MY_TABLE&#34;) // or .fromQuery(&#34;QUERY&#34;) |
| .withStagingBucketName(&#34;BUCKET&#34;) |
| .withStorageIntegrationName(&#34;STORAGE INTEGRATION NAME&#34;) |
| .withCsvMapper(mapper) |
| .withCoder(coder)); |
| )</code></pre> |
| </div> |
| </div> |
| Where all below parameters are required:</p> |
| <ul> |
| <li> |
| <p><code>.withDataSourceConfiguration(...)</code></p> |
| <ul> |
| <li>Accepts a DataSourceConfiguration object.</li> |
| </ul> |
| </li> |
| <li> |
| <p><code>.fromTable(...) or .fromQuery(...)</code></p> |
| <ul> |
| <li>Specifies a Snowflake table name or custom SQL query.</li> |
| </ul> |
| </li> |
| <li> |
| <p><code>.withStagingBucketName()</code></p> |
| <ul> |
| <li>Accepts a cloud bucket name.</li> |
| </ul> |
| </li> |
| <li> |
| <p><code>.withStorageIntegrationName()</code></p> |
| </li> |
| <li> |
| <p>Accepts a name of a Snowflake storage integration object created according to Snowflake documentation. Example: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>CREATE OR REPLACE STORAGE INTEGRATION test_integration |
| TYPE = EXTERNAL_STAGE |
| STORAGE_PROVIDER = GCS |
| ENABLED = TRUE |
| STORAGE_ALLOWED_LOCATIONS = (&#39;gcs://bucket/&#39;);</code></pre> |
| </div> |
| </div> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>CREATE STORAGE INTEGRATION test_integration |
| TYPE = EXTERNAL_STAGE |
| STORAGE_PROVIDER = S3 |
| ENABLED = TRUE |
| STORAGE_AWS_ROLE_ARN = &#39;&lt;ARN ROLE NAME&gt;&#39; |
| STORAGE_ALLOWED_LOCATIONS = (&#39;s3://bucket/&#39;)</code></pre> |
| </div> |
| </div> |
| Then: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>.withStorageIntegrationName(test_integration)</code></pre> |
| </div> |
| </div> |
| </p> |
| </li> |
| <li> |
| <p><code>.withCsvMapper(mapper)</code></p> |
| <ul> |
| <li>Accepts a <a href="/documentation/io/built-in/snowflake/#csvmapper">CSVMapper</a> instance for mapping String[] to USER_DATA_TYPE.</li> |
| </ul> |
| </li> |
| <li> |
| <p><code>.withCoder(coder)</code></p> |
| <ul> |
| <li>Accepts the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/coders/Coder.html">Coder</a> for USER_DATA_TYPE.</li> |
| </ul> |
| </li> |
| </ul> |
| <p><strong>Note</strong>: |
| SnowflakeIO uses <code>COPY</code> statements behind the scenes to read (using <a href="https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-location.html">COPY to location</a>) files staged in cloud storage.StagingBucketName will be used as a temporary location for storing CSV files. Those temporary directories will be named <code>sf_copy_csv_DATE_TIME_RANDOMSUFFIX</code> and they will be removed automatically once Read operation finishes.</p> |
| <h3 id="csvmapper">CSVMapper</h3> |
| <p>SnowflakeIO uses a <a href="https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-location.html">COPY INTO <location></a> statement to move data from a Snowflake table to GCS/S3 as CSV files. These files are then downloaded via <a href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/io/FileIO.html">FileIO</a> and processed line by line. Each line is split into an array of Strings using the <a href="https://opencsv.sourceforge.net/">OpenCSV</a> library.</p> |
| <p>The CSVMapper’s job is to give the user the possibility to convert the array of Strings to a user-defined type, ie. GenericRecord for Avro or Parquet files, or custom POJO.</p> |
| <p>Example implementation of CsvMapper for GenericRecord: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>static SnowflakeIO.CsvMapper&lt;GenericRecord&gt; getCsvMapper() { |
| return (SnowflakeIO.CsvMapper&lt;GenericRecord&gt;) |
| parts -&gt; { |
| return new GenericRecordBuilder(PARQUET_SCHEMA) |
| .set(&#34;ID&#34;, Long.valueOf(parts[0])) |
| .set(&#34;NAME&#34;, parts[1]) |
| [...] |
| .build(); |
| }; |
| }</code></pre> |
| </div> |
| </div> |
| </p> |
| <h2 id="using-snowflakeio-with-aws-s3">Using SnowflakeIO with AWS S3</h2> |
| <p>To be able to use AWS S3 bucket as <code>stagingBucketName</code> is required to:</p> |
| <ol> |
| <li>Create <code>PipelineOptions</code> interface which is <a href="/documentation/io/built-in/snowflake/#extending-pipeline-options">extending</a> <code>SnowflakePipelineOptions</code> and <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/aws2/options/S3Options.html">S3Options</a> |
| with <code>AwsAccessKey</code> and <code>AwsSecretKey</code> options. Example:</li> |
| </ol> |
| <p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>public interface AwsPipelineOptions extends SnowflakePipelineOptions, S3Options { |
| @Description(&#34;AWS Access Key&#34;) |
| @Default.String(&#34;access_key&#34;) |
| String getAwsAccessKey(); |
| void setAwsAccessKey(String awsAccessKey); |
| @Description(&#34;AWS secret key&#34;) |
| @Default.String(&#34;secret_key&#34;) |
| String getAwsSecretKey(); |
| void setAwsSecretKey(String awsSecretKey); |
| }</code></pre> |
| </div> |
| </div> |
| 2. Set <code>AwsCredentialsProvider</code> option by using <code>AwsAccessKey</code> and <code>AwsSecretKey</code> options.</p> |
| <p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>options.setAwsCredentialsProvider( |
| new AWSStaticCredentialsProvider( |
| new BasicAWSCredentials(options.getAwsAccessKey(), options.getAwsSecretKey()) |
| ) |
| );</code></pre> |
| </div> |
| </div> |
| 3. Create pipeline</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>Pipeline p = Pipeline.create(options);</code></pre> |
| </div> |
| </div> |
| <p><strong>Note</strong>: Remember to set <code>awsRegion</code> from <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/aws2/options/S3Options.html">S3Options</a>.</p> |
| <h2 id="using-snowflakeio-in-python-sdk">Using SnowflakeIO in Python SDK</h2> |
| <h3 id="intro">Intro</h3> |
| <p>Snowflake cross-language implementation is supporting both reading and writing operations for Python programming language, thanks to |
| cross-language which is part of <a href="/roadmap/portability/">Portability Framework Roadmap</a> which aims to provide full interoperability |
| across the Beam ecosystem. From a developer perspective it means the possibility of combining transforms written in different languages(Java/Python/Go).</p> |
| <p>For more information about cross-language please see <a href="/roadmap/connectors-multi-sdk/">multi sdk efforts</a> |
| and <a href="/roadmap/connectors-multi-sdk/#cross-language-transforms-api-and-expansion-service">Cross-language transforms API and expansion service</a> articles.</p> |
| <p>Additional resources:</p> |
| <ul> |
| <li><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/snowflake.py">SnowflakeIO source code</a></li> |
| <li><a href="https://beam.apache.org/releases/pydoc/2.56.0/apache_beam.io.snowflake.html">SnowflakeIO Pydoc</a></li> |
| <li><a href="https://docs.snowflake.com/en">Snowflake documentation</a></li> |
| </ul> |
| <h3 id="reading-from-snowflake-1">Reading from Snowflake</h3> |
| <p>One of the functions of SnowflakeIO is reading Snowflake tables - either full tables via table name or custom data via query. Output of the read transform is a <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.pvalue.html#apache_beam.pvalue.PCollection">PCollection</a> of user-defined data type.</p> |
| <h4 id="general-usage-2">General usage</h4> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>OPTIONS = [&#34;--runner=FlinkRunner&#34;] |
| with TestPipeline(options=PipelineOptions(OPTIONS)) as p: |
| (p |
| | ReadFromSnowflake(...) |
| | &lt;FURTHER TRANSFORMS&gt;)</code></pre> |
| </div> |
| </div> |
| <h4 id="required-parameters">Required parameters</h4> |
| <ul> |
| <li> |
| <p><code>server_name</code> Full Snowflake server name with an account, zone, and domain.</p> |
| </li> |
| <li> |
| <p><code>schema</code> Name of the Snowflake schema in the database to use.</p> |
| </li> |
| <li> |
| <p><code>database</code> Name of the Snowflake database to use.</p> |
| </li> |
| <li> |
| <p><code>staging_bucket_name</code> Name of the Google Cloud Storage bucket or AWS S3 bucket. Bucket will be used as a temporary location for storing CSV files. Those temporary directories will be named <code>sf_copy_csv_DATE_TIME_RANDOMSUFFIX</code> and they will be removed automatically once Read operation finishes.</p> |
| </li> |
| <li> |
| <p><code>storage_integration_name</code> Is the name of a Snowflake storage integration object created according to <a href="https://docs.snowflake.net/manuals/sql-reference/sql/create-storage-integration.html">Snowflake documentation</a>.</p> |
| </li> |
| <li> |
| <p><code>csv_mapper</code> Specifies a function which must translate user-defined object to array of strings. SnowflakeIO uses a <a href="https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-location.html">COPY INTO <location></a> statement to move data from a Snowflake table to GCS/S3 as CSV files. These files are then downloaded via <a href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/io/FileIO.html">FileIO</a> and processed line by line. Each line is split into an array of Strings using the <a href="https://opencsv.sourceforge.net/">OpenCSV</a> library. The csv_mapper function job is to give the user the possibility to convert the array of Strings to a user-defined type, ie. GenericRecord for Avro or Parquet files, or custom objects. |
| Example: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>def csv_mapper(strings_array): |
| return User(strings_array[0], int(strings_array[1])))</code></pre> |
| </div> |
| </div> |
| </p> |
| </li> |
| <li> |
| <p><code>table</code> or <code>query</code> Specifies a Snowflake table name or custom SQL query</p> |
| </li> |
| </ul> |
| <h4 id="authentication-parameters">Authentication parameters</h4> |
| <p>It’s required to pass one of the following combinations of valid parameters for authentication:</p> |
| <ul> |
| <li> |
| <p><code>username</code> and <code>password</code> Specifies username and password for username/password authentication method.</p> |
| </li> |
| <li> |
| <p><code>private_key_path</code> and <code>private_key_passphrase</code> Specifies a path to private key and passphrase for key/pair authentication method.</p> |
| </li> |
| <li> |
| <p><code>raw_private_key</code> and <code>private_key_passphrase</code> Specifies a private key and passphrase for key/pair authentication method.</p> |
| </li> |
| <li> |
| <p><code>o_auth_token</code> Specifies access token for OAuth authentication method.</p> |
| </li> |
| </ul> |
| <h4 id="additional-parameters">Additional parameters</h4> |
| <ul> |
| <li> |
| <p><code>role</code> specifies Snowflake role. If not specified the user&rsquo;s default will be used.</p> |
| </li> |
| <li> |
| <p><code>warehouse</code> specifies Snowflake warehouse name. If not specified the user&rsquo;s default will be used.</p> |
| </li> |
| <li> |
| <p><code>expansion_service</code> specifies URL of expansion service.</p> |
| </li> |
| </ul> |
| <h3 id="writing-to-snowflake">Writing to Snowflake</h3> |
| <p>One of the functions of SnowflakeIO is writing to Snowflake tables. This transformation enables you to finish the Beam pipeline with an output operation that sends the user&rsquo;s <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.pvalue.html#apache_beam.pvalue.PCollection">PCollection</a> to your Snowflake database.</p> |
| <h4 id="general-usage-3">General usage</h4> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>OPTIONS = [&#34;--runner=FlinkRunner&#34;] |
| with TestPipeline(options=PipelineOptions(OPTIONS)) as p: |
| (p |
| | &lt;SOURCE OF DATA&gt; |
| | WriteToSnowflake( |
| server_name=&lt;SNOWFLAKE SERVER NAME&gt;, |
| username=&lt;SNOWFLAKE USERNAME&gt;, |
| password=&lt;SNOWFLAKE PASSWORD&gt;, |
| o_auth_token=&lt;OAUTH TOKEN&gt;, |
| private_key_path=&lt;PATH TO P8 FILE&gt;, |
| raw_private_key=&lt;PRIVATE_KEY&gt; |
| private_key_passphrase=&lt;PASSWORD FOR KEY&gt;, |
| schema=&lt;SNOWFLAKE SCHEMA&gt;, |
| database=&lt;SNOWFLAKE DATABASE&gt;, |
| staging_bucket_name=&lt;GCS OR S3 BUCKET&gt;, |
| storage_integration_name=&lt;SNOWFLAKE STORAGE INTEGRATION NAME&gt;, |
| create_disposition=&lt;CREATE DISPOSITION&gt;, |
| write_disposition=&lt;WRITE DISPOSITION&gt;, |
| table_schema=&lt;SNOWFLAKE TABLE SCHEMA&gt;, |
| user_data_mapper=&lt;USER DATA MAPPER FUNCTION&gt;, |
| table=&lt;SNOWFLAKE TABLE&gt;, |
| query=&lt;IF NOT TABLE THEN QUERY&gt;, |
| role=&lt;SNOWFLAKE ROLE&gt;, |
| warehouse=&lt;SNOWFLAKE WAREHOUSE&gt;, |
| expansion_service=&lt;EXPANSION SERVICE ADDRESS&gt;))</code></pre> |
| </div> |
| </div> |
| <h4 id="required-parameters-1">Required parameters</h4> |
| <ul> |
| <li> |
| <p><code>server_name</code> Full Snowflake server name with account, zone and domain.</p> |
| </li> |
| <li> |
| <p><code>schema</code> Name of the Snowflake schema in the database to use.</p> |
| </li> |
| <li> |
| <p><code>database</code> Name of the Snowflake database to use.</p> |
| </li> |
| <li> |
| <p><code>staging_bucket_name</code> Path to Google Cloud Storage bucket or AWS S3 bucket ended with slash. Bucket will be used to save CSV files which will end up in Snowflake. Those CSV files will be saved under “staging_bucket_name” path.</p> |
| </li> |
| <li> |
| <p><code>storage_integration_name</code> Is the name of a Snowflake storage integration object created according to <a href="https://docs.snowflake.net/manuals/sql-reference/sql/create-storage-integration.html">Snowflake documentation</a>.</p> |
| </li> |
| <li> |
| <p><code>user_data_mapper</code> Specifies a function which maps data from a PCollection to an array of String values before the write operation saves the data to temporary .csv files. |
| Example: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>def user_data_mapper(user): |
| return [user.name, str(user.age)]</code></pre> |
| </div> |
| </div> |
| </p> |
| </li> |
| <li> |
| <p><code>table</code> or <code>query</code> Specifies a Snowflake table name or custom SQL query</p> |
| </li> |
| </ul> |
| <h4 id="authentication-parameters-1">Authentication parameters</h4> |
| <p>It’s required to pass one of the following combination of valid parameters for authentication:</p> |
| <ul> |
| <li> |
| <p><code>username</code> and <code>password</code> Specifies username/password authentication method.</p> |
| </li> |
| <li> |
| <p><code>private_key_path</code> and <code>private_key_passphrase</code> Specifies a path to private key and passphrase for key/pair authentication method.</p> |
| </li> |
| <li> |
| <p><code>raw_private_key</code> and <code>private_key_passphrase</code> Specifies a private key and passphrase for key/pair authentication method.</p> |
| </li> |
| <li> |
| <p><code>o_auth_token</code> Specifies access token for OAuth authentication method.</p> |
| </li> |
| </ul> |
| <h4 id="additional-parameters-1">Additional parameters</h4> |
| <ul> |
| <li> |
| <p><code>role</code> specifies Snowflake role. If not specified the user&rsquo;s default will be used.</p> |
| </li> |
| <li> |
| <p><code>warehouse</code> specifies Snowflake warehouse name. If not specified the user&rsquo;s default will be used.</p> |
| </li> |
| <li> |
| <p><code>create_disposition</code> Defines the behaviour of the write operation if the target table does not exist. The following values are supported:</p> |
| <ul> |
| <li><code>CREATE_IF_NEEDED</code> - default behaviour. The write operation checks whether the specified target table exists; if it does not, the write operation attempts to create the table Specify the schema for the target table using the table_schema parameter.</li> |
| <li><code>CREATE_NEVER</code> - The write operation fails if the target table does not exist.</li> |
| </ul> |
| </li> |
| <li> |
| <p><code>write_disposition</code> Defines the write behaviour based on the table where data will be written to. The following values are supported:</p> |
| <ul> |
| <li><code>APPEND</code> - Default behaviour. Written data is added to the existing rows in the table,</li> |
| <li><code>EMPTY</code> - The target table must be empty; otherwise, the write operation fails,</li> |
| <li><code>TRUNCATE</code> - The write operation deletes all rows from the target table before writing to it.</li> |
| </ul> |
| </li> |
| <li> |
| <p><code>table_schema</code> When the <code>create_disposition</code> parameter is set to CREATE_IF_NEEDED, the table_schema parameter enables specifying the schema for the created target table. A table schema is a JSON array with the following structure: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>{&#34;schema&#34;: [ |
| { |
| &#34;dataType&#34;:{&#34;type&#34;:&#34;&lt;COLUMN DATA TYPE&gt;&#34;}, |
| &#34;name&#34;:&#34;&lt;COLUMN NAME&gt; &#34;, |
| &#34;nullable&#34;: &lt;NULLABLE&gt; |
| }, |
| ... |
| ]}</code></pre> |
| </div> |
| </div> |
| All supported data types: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>{&#34;type&#34;:&#34;date&#34;}, |
| {&#34;type&#34;:&#34;datetime&#34;}, |
| {&#34;type&#34;:&#34;time&#34;}, |
| {&#34;type&#34;:&#34;timestamp&#34;}, |
| {&#34;type&#34;:&#34;timestamp_ltz&#34;}, |
| {&#34;type&#34;:&#34;timestamp_ntz&#34;}, |
| {&#34;type&#34;:&#34;timestamp_tz&#34;}, |
| {&#34;type&#34;:&#34;boolean&#34;}, |
| {&#34;type&#34;:&#34;decimal&#34;,&#34;precision&#34;:38,&#34;scale&#34;:1}, |
| {&#34;type&#34;:&#34;double&#34;}, |
| {&#34;type&#34;:&#34;float&#34;}, |
| {&#34;type&#34;:&#34;integer&#34;,&#34;precision&#34;:38,&#34;scale&#34;:0}, |
| {&#34;type&#34;:&#34;number&#34;,&#34;precision&#34;:38,&#34;scale&#34;:1}, |
| {&#34;type&#34;:&#34;numeric&#34;,&#34;precision&#34;:38,&#34;scale&#34;:2}, |
| {&#34;type&#34;:&#34;real&#34;}, |
| {&#34;type&#34;:&#34;array&#34;}, |
| {&#34;type&#34;:&#34;object&#34;}, |
| {&#34;type&#34;:&#34;variant&#34;}, |
| {&#34;type&#34;:&#34;binary&#34;,&#34;size&#34;:null}, |
| {&#34;type&#34;:&#34;char&#34;,&#34;length&#34;:1}, |
| {&#34;type&#34;:&#34;string&#34;,&#34;length&#34;:null}, |
| {&#34;type&#34;:&#34;text&#34;,&#34;length&#34;:null}, |
| {&#34;type&#34;:&#34;varbinary&#34;,&#34;size&#34;:null}, |
| {&#34;type&#34;:&#34;varchar&#34;,&#34;length&#34;:100}]</code></pre> |
| </div> |
| </div> |
| You can read about Snowflake data types at <a href="https://docs.snowflake.com/en/sql-reference/data-types.html">Snowflake data types</a>.</p> |
| </li> |
| <li> |
| <p><code>expansion_service</code> Specifies URL of expansion service.</p> |
| </li> |
| </ul> |
| <h2 id="limitations">Limitations</h2> |
| <p>SnowflakeIO currently has the following limitations.</p> |
| <ol> |
| <li> |
| <p>Streaming writing supports only pair key authentication. For details, see: <a href="https://github.com/apache/beam/issues/21287">Issue 21287</a>.</p> |
| </li> |
| <li> |
| <p>The role parameter configured in <code>SnowflakeIO.DataSourceConfiguration</code> object is ignored for streaming writing. For details, see: <a href="https://github.com/apache/beam/issues/21365">Issue 21365</a></p> |
| </li> |
| </ol></description></item><item><title>Documentation: ApproximateQuantiles</title><link>/documentation/transforms/java/aggregation/approximatequantiles/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/java/aggregation/approximatequantiles/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="approximatequantiles">ApproximateQuantiles</h1> |
| <table align="left"> |
| <a target="_blank" class="button" |
| href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/ApproximateQuantiles.html"> |
| <img src="/images/logos/sdks/java.png" width="20px" height="20px" |
| alt="Javadoc" /> |
| Javadoc |
| </a> |
| </table> |
| <br><br> |
| <p>Takes a comparison function and the desired number of quantiles <em>n</em>, either |
| globally or per-key. Using an approximation algorithm, it returns the |
| minimum value, <em>n-2</em> intermediate values, and the maximum value.</p> |
| <h2 id="examples">Examples</h2> |
| <p><strong>Example</strong>: to compute the quartiles of a <code>PCollection</code> of integers, we |
| would use <code>ApproximateQuantiles.globally(5)</code>. This will produce a list |
| containing 5 values: the minimum value, Quartile 1 value, Quartile 2 |
| value, Quartile 3 value, and the maximum value.</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-java playground-snippet" |
| data-sdk="java" |
| data-path="SDK_JAVA_ApproximateQuantiles" |
| data-show="main_section" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_JAVA_ApproximateQuantiles%22%2c%22sdk%22%3a%22java%22%2c%22show%22%3a%22main_section%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2> |
| <ul> |
| <li><a href="/documentation/transforms/java/aggregation/approximateunique">ApproximateUnique</a> |
| estimates the number of distinct elements or distinct values in key-value pairs</li> |
| <li><a href="/documentation/transforms/java/aggregation/combine">Combine</a></li> |
| </ul></description></item><item><title>Documentation: ApproximateQuantiles</title><link>/documentation/transforms/python/aggregation/approximatequantiles/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/python/aggregation/approximatequantiles/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="approximatequantiles">ApproximateQuantiles</h1> |
| <script type="text/javascript"> |
| localStorage.setItem("language", "language-py") |
| </script> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.stat.html#apache_beam.transforms.stat.ApproximateQuantile" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p> |
| <h2 id="examples">Examples</h2> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_ApproximateQuantiles" |
| data-show="approximatequantiles" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_ApproximateQuantiles%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22approximatequantiles%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2></description></item><item><title>Documentation: ApproximateUnique</title><link>/documentation/transforms/java/aggregation/approximateunique/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/java/aggregation/approximateunique/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="approximateunique">ApproximateUnique</h1> |
| <table align="left"> |
| <a target="_blank" class="button" |
| href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/ApproximateUnique.html"> |
| <img src="/images/logos/sdks/java.png" width="20px" height="20px" |
| alt="Javadoc" /> |
| Javadoc |
| </a> |
| </table> |
| <br><br> |
| <p>Transforms for estimating the number of distinct elements in a collection |
| or the number of distinct values associated with each key in a collection |
| of key-value pairs.</p> |
| <h2 id="examples">Examples</h2> |
| <p>See <a href="https://issues.apache.org/jira/browse/BEAM-7703">BEAM-7703</a> for updates.</p> |
| <h2 id="related-transforms">Related transforms</h2> |
| <ul> |
| <li><a href="/documentation/transforms/java/aggregation/hllcount">HllCount</a> |
| estimates the number of distinct elements and creates re-aggregatable sketches using the HyperLogLog++ algorithm.</li> |
| <li><a href="/documentation/transforms/java/aggregation/count">Count</a> |
| counts the number of elements within each aggregation.</li> |
| <li><a href="/documentation/transforms/java/aggregation/distinct">Distinct</a></li> |
| </ul></description></item><item><title>Documentation: ApproximateUnique</title><link>/documentation/transforms/python/aggregation/approximateunique/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/python/aggregation/approximateunique/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="approximateunique">ApproximateUnique</h1> |
| <script type="text/javascript"> |
| localStorage.setItem("language", "language-py") |
| </script> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.stat.html#apache_beam.transforms.stat.ApproximateUnique" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p> |
| <h2 id="examples">Examples</h2> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_ApproximateUnique" |
| data-show="approximateunique" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_ApproximateUnique%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22approximateunique%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2></description></item><item><title>Documentation: Auto Update ML models using WatchFilePattern</title><link>/documentation/ml/side-input-updates/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/ml/side-input-updates/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="use-watchfilepattern-to-auto-update-ml-models-in-runinference">Use WatchFilePattern to auto-update ML models in RunInference</h1> |
| <p>The pipeline in this example uses a <a href="https://beam.apache.org/documentation/transforms/python/elementwise/runinference/">RunInference</a> <code>PTransform</code> to run inference on images using TensorFlow models. It uses a <a href="https://beam.apache.org/documentation/programming-guide/#side-inputs">side input</a> <code>PCollection</code> that emits <code>ModelMetadata</code> to update the model.</p> |
| <p>Using side inputs, you can update your model (which is passed in a <code>ModelHandler</code> configuration object) in real-time, even while the Beam pipeline is still running. This can be done either by leveraging one of Beam&rsquo;s provided patterns, such as the <code>WatchFilePattern</code>, |
| or by configuring a custom side input <code>PCollection</code> that defines the logic for the model update.</p> |
| <p>For more information about side inputs, see the <a href="https://beam.apache.org/documentation/programming-guide/#side-inputs">Side inputs</a> section in the Apache Beam Programming Guide.</p> |
| <p>This example uses <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.ml.inference.utils.html#apache_beam.ml.inference.utils.WatchFilePattern"><code>WatchFilePattern</code></a> as a side input. <code>WatchFilePattern</code> is used to watch for the file updates matching the <code>file_pattern</code> |
| based on timestamps. It emits the latest <a href="https://beam.apache.org/documentation/transforms/python/elementwise/runinference/"><code>ModelMetadata</code></a>, which is used in |
| the RunInference <code>PTransform</code> to automatically update the ML model without stopping the Beam pipeline.</p> |
| <h2 id="set-up-the-source">Set up the source</h2> |
| <p>To read the image names, use a Pub/Sub topic as the source. The Pub/Sub topic emits a <code>UTF-8</code> encoded model path that is used to read and preprocess images to run the inference.</p> |
| <h2 id="models-for-image-segmentation">Models for image segmentation</h2> |
| <p>For the purpose of this example, use TensorFlow models saved in <a href="https://www.tensorflow.org/tutorials/keras/save_and_load#hdf5_format">HDF5</a> format.</p> |
| <h2 id="pre-process-images-for-inference">Pre-process images for inference</h2> |
| <p>The Pub/Sub topic emits an image path. We need to read and preprocess the image to use it for RunInference. The <code>read_image</code> function is used to read the image for inference.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">io</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">PIL</span> <span class="kn">import</span> <span class="n">Image</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.io.filesystems</span> <span class="kn">import</span> <span class="n">FileSystems</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">numpy</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">tensorflow</span> <span class="k">as</span> <span class="nn">tf</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">read_image</span><span class="p">(</span><span class="n">image_file_name</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">with</span> <span class="n">FileSystems</span><span class="p">()</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">image_file_name</span><span class="p">,</span> <span class="s1">&#39;r&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">file</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">data</span> <span class="o">=</span> <span class="n">Image</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">io</span><span class="o">.</span><span class="n">BytesIO</span><span class="p">(</span><span class="n">file</span><span class="o">.</span><span class="n">read</span><span class="p">()))</span><span class="o">.</span><span class="n">convert</span><span class="p">(</span><span class="s1">&#39;RGB&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">img</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">resize</span><span class="p">((</span><span class="mi">224</span><span class="p">,</span> <span class="mi">224</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">img</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">img</span><span class="p">)</span> <span class="o">/</span> <span class="mf">255.0</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">img_tensor</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">cast</span><span class="p">(</span><span class="n">tf</span><span class="o">.</span><span class="n">convert_to_tensor</span><span class="p">(</span><span class="n">img</span><span class="p">[</span><span class="o">...</span><span class="p">]),</span> <span class="n">dtype</span><span class="o">=</span><span class="n">tf</span><span class="o">.</span><span class="n">float32</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">img_tensor</span> |
| </span></span></code></pre></div><p>Now, let&rsquo;s jump into the pipeline code.</p> |
| <p><strong>Pipeline steps</strong>:</p> |
| <ol> |
| <li>Get the image names from the Pub/Sub topic.</li> |
| <li>Read and pre-process the images using the <code>read_image</code> function.</li> |
| <li>Pass the images to the RunInference <code>PTransform</code>. RunInference takes <code>model_handler</code> and <code>model_metadata_pcoll</code> as input parameters.</li> |
| </ol> |
| <p>For the <a href="https://github.com/apache/beam/blob/07f52a478174f8733c7efedb7189955142faa5fa/sdks/python/apache_beam/ml/inference/base.py#L308"><code>model_handler</code></a>, we use <a href="https://github.com/apache/beam/blob/186973b110d82838fb8e5ba27f0225a67c336591/sdks/python/apache_beam/ml/inference/tensorflow_inference.py#L184">TFModelHandlerTensor</a>.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.ml.inference.tensorflow_inference</span> <span class="kn">import</span> <span class="n">TFModelHandlerTensor</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># initialize TFModelHandlerTensor with a .h5 model saved in a directory accessible by the pipeline.</span> |
| </span></span><span class="line"><span class="cl"><span class="n">tf_model_handler</span> <span class="o">=</span> <span class="n">TFModelHandlerTensor</span><span class="p">(</span><span class="n">model_uri</span><span class="o">=</span><span class="s1">&#39;gs://&lt;your-bucket&gt;/&lt;model_path.h5&gt;&#39;</span><span class="p">)</span> |
| </span></span></code></pre></div><p>The <code>model_metadata_pcoll</code> is a <a href="https://beam.apache.org/documentation/programming-guide/#side-inputs">side input</a> <code>PCollection</code> to the RunInference <code>PTransform</code>. This side input is used to update the models in the <code>model_handler</code> without needing to stop the beam pipeline. |
| We will use <code>WatchFilePattern</code> as side input to watch a glob pattern matching <code>.h5</code> files.</p> |
| <p><code>model_metadata_pcoll</code> expects a <code>PCollection</code> of ModelMetadata compatible with <a href="https://beam.apache.org/releases/pydoc/2.4.0/apache_beam.pvalue.html#apache_beam.pvalue.AsSingleton">AsSingleton</a>. Because the pipeline uses <code>WatchFilePattern</code> as side input, it will take care of windowing and wrapping the output into <code>ModelMetadata</code>.</p> |
| <p>After the pipeline starts processing data and when you see some outputs emitted from the RunInference <code>PTransform</code>, upload a <code>.h5</code> <code>TensorFlow</code> model that matches the <code>file_pattern</code> to the Google Cloud Storage bucket. RunInference will update the <code>model_uri</code> of <code>TFModelHandlerTensor</code> using <code>WatchFilePattern</code> as a side input.</p> |
| <p><strong>Note</strong>: Side input update frequency is non-deterministic and can have longer intervals between updates.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="k">as</span> <span class="nn">beam</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.ml.inference.utils</span> <span class="kn">import</span> <span class="n">WatchFilePattern</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.ml.inference.base</span> <span class="kn">import</span> <span class="n">RunInference</span> |
| </span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">file_pattern</span> <span class="o">=</span> <span class="s1">&#39;gs://&lt;your-bucket&gt;/*.h5&#39;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pubsub_topic</span> <span class="o">=</span> <span class="s1">&#39;&lt;topic_emitting_image_names&gt;&#39;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">side_input_pcoll</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s2">&#34;FilePatternUpdates&#34;</span> <span class="o">&gt;&gt;</span> <span class="n">WatchFilePattern</span><span class="p">(</span><span class="n">file_pattern</span><span class="o">=</span><span class="n">file_pattern</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">images_pcoll</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s2">&#34;ReadFromPubSub&#34;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ReadFromPubSub</span><span class="p">(</span><span class="n">topic</span><span class="o">=</span><span class="n">pubsub_topic</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s2">&#34;DecodeBytes&#34;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;utf-8&#39;</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s2">&#34;PreProcessImage&#34;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="n">read_image</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">inference_pcoll</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">images_pcoll</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s2">&#34;RunInference&#34;</span> <span class="o">&gt;&gt;</span> <span class="n">RunInference</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">model_handler</span><span class="o">=</span><span class="n">tf_model_handler</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">model_metadata_pcoll</span><span class="o">=</span><span class="n">side_input_pcoll</span><span class="p">))</span> |
| </span></span></code></pre></div><h2 id="post-process-the-predictionresult-object">Post-process the <code>PredictionResult</code> object</h2> |
| <p>When the inference is complete, RunInference outputs a <code>PredictionResult</code> object that contains <code>example</code>, <code>inference</code>, and <code>model_id</code> fields. The <code>model_id</code> is used to identify which model is used for running the inference.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.ml.inference.base</span> <span class="kn">import</span> <span class="n">PredictionResult</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">PostProcessor</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;&#34;&#34; |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> Process the PredictionResult to get the predicted label and model id used for inference. |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> &#34;&#34;&#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">:</span> <span class="n">PredictionResult</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">typing</span><span class="o">.</span><span class="n">Iterable</span><span class="p">[</span><span class="nb">str</span><span class="p">]:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">predicted_class</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">argmax</span><span class="p">(</span><span class="n">element</span><span class="o">.</span><span class="n">inference</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">axis</span><span class="o">=-</span><span class="mi">1</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">labels_path</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">keras</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">get_file</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;ImageNetLabels.txt&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt&#39;</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">imagenet_labels</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="nb">open</span><span class="p">(</span><span class="n">labels_path</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">splitlines</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">predicted_class_name</span> <span class="o">=</span> <span class="n">imagenet_labels</span><span class="p">[</span><span class="n">predicted_class</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">predicted_class_name</span><span class="o">.</span><span class="n">title</span><span class="p">(),</span> <span class="n">element</span><span class="o">.</span><span class="n">model_id</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">post_processor_pcoll</span> <span class="o">=</span> <span class="p">(</span><span class="n">inference_pcoll</span> <span class="o">|</span> <span class="s2">&#34;PostProcessor&#34;</span> <span class="o">&gt;&gt;</span> <span class="n">PostProcessor</span><span class="p">())</span> |
| </span></span></code></pre></div><h2 id="run-the-pipeline">Run the pipeline</h2> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">result</span> <span class="o">=</span> <span class="n">pipeline</span><span class="o">.</span><span class="n">run</span><span class="p">()</span><span class="o">.</span><span class="n">wait_until_finish</span><span class="p">()</span> |
| </span></span></code></pre></div><p><strong>Note</strong>: The <code>model_name</code> of the <code>ModelMetaData</code> object will be attached as prefix to the <a href="https://beam.apache.org/documentation/ml/runinference-metrics/">metrics</a> calculated by the RunInference <code>PTransform</code>.</p> |
| <h2 id="final-remarks">Final remarks</h2> |
| <p>You can use this example as a pattern when using side inputs with the RunInference <code>PTransform</code> to auto-update the models without stopping the pipeline. You can see a similar example for PyTorch on <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/pytorch_image_classification_with_side_inputs.py">GitHub</a>.</p></description></item><item><title>Documentation: Basics of the Beam model</title><link>/documentation/basics/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/basics/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="basics-of-the-beam-model">Basics of the Beam model</h1> |
| <p>Apache Beam is a unified model for defining both batch and streaming |
| data-parallel processing pipelines. To get started with Beam, you&rsquo;ll need to |
| understand an important set of core concepts:</p> |
| <ul> |
| <li><a href="#pipeline"><em>Pipeline</em></a> - A pipeline is a user-constructed graph of |
| transformations that defines the desired data processing operations.</li> |
| <li><a href="#pcollection"><em>PCollection</em></a> - A <code>PCollection</code> is a data set or data |
| stream. The data that a pipeline processes is part of a PCollection.</li> |
| <li><a href="#ptransform"><em>PTransform</em></a> - A <code>PTransform</code> (or <em>transform</em>) represents a |
| data processing operation, or a step, in your pipeline. A transform is |
| applied to zero or more <code>PCollection</code> objects, and produces zero or more |
| <code>PCollection</code> objects.</li> |
| <li><a href="#aggregation"><em>Aggregation</em></a> - Aggregation is computing a value from |
| multiple (1 or more) input elements.</li> |
| <li><a href="#user-defined-function-udf"><em>User-defined function (UDF)</em></a> - Some Beam |
| operations allow you to run user-defined code as a way to configure the |
| transform.</li> |
| <li><a href="#schema"><em>Schema</em></a> - A schema is a language-independent type definition for |
| a <code>PCollection</code>. The schema for a <code>PCollection</code> defines elements of that |
| <code>PCollection</code> as an ordered list of named fields.</li> |
| <li><a href="/documentation/sdks/java/"><em>SDK</em></a> - A language-specific library that lets |
| pipeline authors build transforms, construct their pipelines, and submit |
| them to a runner.</li> |
| <li><a href="#runner"><em>Runner</em></a> - A runner runs a Beam pipeline using the capabilities of |
| your chosen data processing engine.</li> |
| <li><a href="#window"><em>Window</em></a> - A <code>PCollection</code> can be subdivided into windows based on |
| the timestamps of the individual elements. Windows enable grouping operations |
| over collections that grow over time by dividing the collection into windows |
| of finite collections.</li> |
| <li><a href="#watermark"><em>Watermark</em></a> - A watermark is a guess as to when all data in a |
| certain window is expected to have arrived. This is needed because data isn’t |
| always guaranteed to arrive in a pipeline in time order, or to always arrive |
| at predictable intervals.</li> |
| <li><a href="#trigger"><em>Trigger</em></a> - A trigger determines when to aggregate the results of |
| each window.</li> |
| <li><a href="#state-and-timers"><em>State and timers</em></a> - Per-key state and timer callbacks |
| are lower level primitives that give you full control over aggregating input |
| collections that grow over time.</li> |
| <li><a href="#splittable-dofn"><em>Splittable DoFn</em></a> - Splittable DoFns let you process |
| elements in a non-monolithic way. You can checkpoint the processing of an |
| element, and the runner can split the remaining work to yield additional |
| parallelism.</li> |
| </ul> |
| <p>The following sections cover these concepts in more detail and provide links to |
| additional documentation.</p> |
| <h2 id="pipeline">Pipeline</h2> |
| <p>A Beam pipeline is a graph (specifically, a |
| <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph">directed acyclic graph</a>) |
| of all the data and computations in your data processing task. This includes |
| reading input data, transforming that data, and writing output data. A pipeline |
| is constructed by a user in their SDK of choice. Then, the pipeline makes its |
| way to the runner either through the SDK directly or through the Runner API&rsquo;s |
| RPC interface. For example, this diagram shows a branching pipeline:</p> |
| <p><img src="/images/design-your-pipeline-multiple-pcollections.svg" alt="The pipeline applies two transforms to a single input collection. Eachtransform produces an output collection."></p> |
| <p>In this diagram, the boxes represent the parallel computations called |
| <a href="#ptransform"><em>PTransforms</em></a> and the arrows with the circles represent the data |
| (in the form of <a href="#pcollection"><em>PCollections</em></a>) that flows between the |
| transforms. The data might be bounded, stored, data sets, or the data might also |
| be unbounded streams of data. In Beam, most transforms apply equally to bounded |
| and unbounded data.</p> |
| <p>You can express almost any computation that you can think of as a graph as a |
| Beam pipeline. A Beam driver program typically starts by creating a <code>Pipeline</code> |
| object, and then uses that object as the basis for creating the pipeline’s data |
| sets and its transforms.</p> |
| <p>For more information about pipelines, see the following pages:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#overview">Beam Programming Guide: Overview</a></li> |
| <li><a href="/documentation/programming-guide/#creating-a-pipeline">Beam Programming Guide: Creating a pipeline</a></li> |
| <li><a href="/documentation/pipelines/design-your-pipeline">Design your pipeline</a></li> |
| <li><a href="/documentation/pipelines/create-your-pipeline">Create your pipeline</a></li> |
| </ul> |
| <h2 id="pcollection">PCollection</h2> |
| <p>A <code>PCollection</code> is an unordered bag of elements. Each <code>PCollection</code> is a |
| potentially distributed, homogeneous data set or data stream, and is owned by |
| the specific <code>Pipeline</code> object for which it is created. Multiple pipelines |
| cannot share a <code>PCollection</code>. Beam pipelines process PCollections, and the |
| runner is responsible for storing these elements.</p> |
| <p>A <code>PCollection</code> generally contains &ldquo;big data&rdquo; (too much data to fit in memory on |
| a single machine). Sometimes a small sample of data or an intermediate result |
| might fit into memory on a single machine, but Beam&rsquo;s computational patterns and |
| transforms are focused on situations where distributed data-parallel computation |
| is required. Therefore, the elements of a <code>PCollection</code> cannot be processed |
| individually, and are instead processed uniformly in parallel.</p> |
| <p>The following characteristics of a <code>PCollection</code> are important to know.</p> |
| <p><strong>Bounded vs. unbounded</strong>:</p> |
| <p>A <code>PCollection</code> can be either bounded or unbounded.</p> |
| <ul> |
| <li>A <em>bounded</em> <code>PCollection</code> is a dataset of a known, fixed size (alternatively, |
| a dataset that is not growing over time). Bounded data can be processed by |
| batch pipelines.</li> |
| <li>An <em>unbounded</em> <code>PCollection</code> is a dataset that grows over time, and the |
| elements are processed as they arrive. Unbounded data must be processed by |
| streaming pipelines.</li> |
| </ul> |
| <p>These two categories derive from the intuitions of batch and stream processing, |
| but the two are unified in Beam and bounded and unbounded PCollections can |
| coexist in the same pipeline. If your runner can only support bounded |
| PCollections, you must reject pipelines that contain unbounded PCollections. If |
| your runner is only targeting streams, there are adapters in Beam&rsquo;s support code |
| to convert everything to APIs that target unbounded data.</p> |
| <p><strong>Timestamps</strong>:</p> |
| <p>Every element in a <code>PCollection</code> has a timestamp associated with it.</p> |
| <p>When you execute a primitive connector to a storage system, that connector is |
| responsible for providing initial timestamps. The runner must propagate and |
| aggregate timestamps. If the timestamp is not important, such as with certain |
| batch processing jobs where elements do not denote events, the timestamp will be |
| the minimum representable timestamp, often referred to colloquially as &ldquo;negative |
| infinity&rdquo;.</p> |
| <p><strong>Watermarks</strong>:</p> |
| <p>Every <code>PCollection</code> must have a <a href="#watermark">watermark</a> that estimates how |
| complete the <code>PCollection</code> is.</p> |
| <p>The watermark is a guess that &ldquo;we&rsquo;ll never see an element with an earlier |
| timestamp&rdquo;. Data sources are responsible for producing a watermark. The runner |
| must implement watermark propagation as PCollections are processed, merged, and |
| partitioned.</p> |
| <p>The contents of a <code>PCollection</code> are complete when a watermark advances to |
| &ldquo;infinity&rdquo;. In this manner, you can discover that an unbounded PCollection is |
| finite.</p> |
| <p><strong>Windowed elements</strong>:</p> |
| <p>Every element in a <code>PCollection</code> resides in a <a href="#window">window</a>. No element |
| resides in multiple windows; two elements can be equal except for their window, |
| but they are not the same.</p> |
| <p>When elements are written to the outside world, they are effectively placed back |
| into the global window. Transforms that write data and don&rsquo;t take this |
| perspective risk data loss.</p> |
| <p>A window has a maximum timestamp. When the watermark exceeds the maximum |
| timestamp plus the user-specified allowed lateness, the window is expired. All |
| data related to an expired window might be discarded at any time.</p> |
| <p><strong>Coder</strong>:</p> |
| <p>Every <code>PCollection</code> has a coder, which is a specification of the binary format |
| of the elements.</p> |
| <p>In Beam, the user&rsquo;s pipeline can be written in a language other than the |
| language of the runner. There is no expectation that the runner can actually |
| deserialize user data. The Beam model operates principally on encoded data, |
| &ldquo;just bytes&rdquo;. Each <code>PCollection</code> has a declared encoding for its elements, |
| called a coder. A coder has a URN that identifies the encoding, and might have |
| additional sub-coders. For example, a coder for lists might contain a coder for |
| the elements of the list. Language-specific serialization techniques are |
| frequently used, but there are a few common key formats (such as key-value pairs |
| and timestamps) so the runner can understand them.</p> |
| <p><strong>Windowing strategy</strong>:</p> |
| <p>Every <code>PCollection</code> has a windowing strategy, which is a specification of |
| essential information for grouping and triggering operations. The <code>Window</code> |
| transform sets up the windowing strategy, and the <code>GroupByKey</code> transform has |
| behavior that is governed by the windowing strategy.</p> |
| <br/> |
| <p>For more information about PCollections, see the following page:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#pcollections">Beam Programming Guide: PCollections</a></li> |
| </ul> |
| <h2 id="ptransform">PTransform</h2> |
| <p>A <code>PTransform</code> (or transform) represents a data processing operation, or a step, |
| in your pipeline. A transform is usually applied to one or more input |
| <code>PCollection</code> objects. Transforms that read input are an exception; these |
| transforms might not have an input <code>PCollection</code>.</p> |
| <p>You provide transform processing logic in the form of a function object |
| (colloquially referred to as “user code”), and your user code is applied to each |
| element of the input PCollection (or more than one PCollection). Depending on |
| the pipeline runner and backend that you choose, many different workers across a |
| cluster might execute instances of your user code in parallel. The user code |
| that runs on each worker generates the output elements that are added to zero or |
| more output <code>PCollection</code> objects.</p> |
| <p>The Beam SDKs contain a number of different transforms that you can apply to |
| your pipeline’s PCollections. These include general-purpose core transforms, |
| such as <code>ParDo</code> or <code>Combine</code>. There are also pre-written composite transforms |
| included in the SDKs, which combine one or more of the core transforms in a |
| useful processing pattern, such as counting or combining elements in a |
| collection. You can also define your own more complex composite transforms to |
| fit your pipeline’s exact use case.</p> |
| <p>The following list has some common transform types:</p> |
| <ul> |
| <li>Source transforms such as <code>TextIO.Read</code> and <code>Create</code>. A source transform |
| conceptually has no input.</li> |
| <li>Processing and conversion operations such as <code>ParDo</code>, <code>GroupByKey</code>, |
| <code>CoGroupByKey</code>, <code>Combine</code>, and <code>Count</code>.</li> |
| <li>Outputting transforms such as <code>TextIO.Write</code>.</li> |
| <li>User-defined, application-specific composite transforms.</li> |
| </ul> |
| <p>For more information about transforms, see the following pages:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#overview">Beam Programming Guide: Overview</a></li> |
| <li><a href="/documentation/programming-guide/#transforms">Beam Programming Guide: Transforms</a></li> |
| <li>Beam transform catalog (<a href="/documentation/transforms/java/overview/">Java</a>, |
| <a href="/documentation/transforms/python/overview/">Python</a>)</li> |
| </ul> |
| <h2 id="aggregation">Aggregation</h2> |
| <p>Aggregation is computing a value from multiple (1 or more) input elements. In |
| Beam, the primary computational pattern for aggregation is to group all elements |
| with a common key and window then combine each group of elements using an |
| associative and commutative operation. This is similar to the &ldquo;Reduce&rdquo; operation |
| in the <a href="https://en.wikipedia.org/wiki/MapReduce">MapReduce</a> model, though it is |
| enhanced to work with unbounded input streams as well as bounded data sets.</p> |
| <img src="/images/aggregation.png" alt="Aggregation of elements." width="120px"> |
| <p><em>Figure 1: Aggregation of elements. Elements with the same color represent those |
| with a common key and window.</em></p> |
| <p>Some simple aggregation transforms include <code>Count</code> (computes the count of all |
| elements in the aggregation), <code>Max</code> (computes the maximum element in the |
| aggregation), and <code>Sum</code> (computes the sum of all elements in the aggregation).</p> |
| <p>When elements are grouped and emitted as a bag, the aggregation is known as |
| <code>GroupByKey</code> (the associative/commutative operation is bag union). In this case, |
| the output is no smaller than the input. Often, you will apply an operation such |
| as summation, called a <code>CombineFn</code>, in which the output is significantly smaller |
| than the input. In this case the aggregation is called <code>CombinePerKey</code>.</p> |
| <p>In a real application, you might have millions of keys and/or windows; that is |
| why this is still an &ldquo;embarrassingly parallel&rdquo; computational pattern. In those |
| cases where you have fewer keys, you can add parallelism by adding a |
| supplementary key, splitting each of your problem&rsquo;s natural keys into many |
| sub-keys. After these sub-keys are aggregated, the results can be further |
| combined into a result for the original natural key for your problem. The |
| associativity of your aggregation function ensures that this yields the same |
| answer, but with more parallelism.</p> |
| <p>When your input is unbounded, the computational pattern of grouping elements by |
| key and window is roughly the same, but governing when and how to emit the |
| results of aggregation involves three concepts:</p> |
| <ul> |
| <li><a href="#window">Windowing</a>, which partitions your input into bounded subsets that |
| can be complete.</li> |
| <li><a href="#watermark">Watermarks</a>, which estimate the completeness of your input.</li> |
| <li><a href="#trigger">Triggers</a>, which govern when and how to emit aggregated results.</li> |
| </ul> |
| <p>For more information about available aggregation transforms, see the following |
| pages:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#core-beam-transforms">Beam Programming Guide: Core Beam transforms</a></li> |
| <li>Beam Transform catalog |
| (<a href="/documentation/transforms/java/overview/#aggregation">Java</a>, |
| <a href="/documentation/transforms/python/overview/#aggregation">Python</a>)</li> |
| </ul> |
| <h2 id="user-defined-function-udf">User-defined function (UDF)</h2> |
| <p>Some Beam operations allow you to run user-defined code as a way to configure |
| the transform. For example, when using <code>ParDo</code>, user-defined code specifies what |
| operation to apply to every element. For <code>Combine</code>, it specifies how values |
| should be combined. By using <a href="/documentation/patterns/cross-language/">cross-language transforms</a>, |
| a Beam pipeline can contain UDFs written in a different language, or even |
| multiple languages in the same pipeline.</p> |
| <p>Beam has several varieties of UDFs:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#pardo"><em>DoFn</em></a> - per-element processing |
| function (used in <code>ParDo</code>)</li> |
| <li><a href="/documentation/programming-guide/#setting-your-pcollections-windowing-function"><em>WindowFn</em></a> - |
| places elements in windows and merges windows (used in <code>Window</code> and |
| <code>GroupByKey</code>)</li> |
| <li><a href="/documentation/programming-guide/#side-inputs"><em>ViewFn</em></a> - adapts a |
| materialized <code>PCollection</code> to a particular interface (used in side inputs)</li> |
| <li><a href="/documentation/programming-guide/#side-inputs-windowing"><em>WindowMappingFn</em></a> - |
| maps one element&rsquo;s window to another, and specifies bounds on how far in the |
| past the result window will be (used in side inputs)</li> |
| <li><a href="/documentation/programming-guide/#combine"><em>CombineFn</em></a> - associative and |
| commutative aggregation (used in <code>Combine</code> and state)</li> |
| <li><a href="/documentation/programming-guide/#data-encoding-and-type-safety"><em>Coder</em></a> - |
| encodes user data; some coders have standard formats and are not really UDFs</li> |
| </ul> |
| <p>Each language SDK has its own idiomatic way of expressing the user-defined |
| functions in Beam, but there are common requirements. When you build user code |
| for a Beam transform, you should keep in mind the distributed nature of |
| execution. For example, there might be many copies of your function running on a |
| lot of different machines in parallel, and those copies function independently, |
| without communicating or sharing state with any of the other copies. Each copy |
| of your user code function might be retried or run multiple times, depending on |
| the pipeline runner and the processing backend that you choose for your |
| pipeline. Beam also supports stateful processing through the |
| <a href="/blog/stateful-processing/">stateful processing API</a>.</p> |
| <p>For more information about user-defined functions, see the following pages:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms">Requirements for writing user code for Beam transforms</a></li> |
| <li><a href="/documentation/programming-guide/#pardo">Beam Programming Guide: ParDo</a></li> |
| <li><a href="/documentation/programming-guide/#setting-your-pcollections-windowing-function">Beam Programming Guide: WindowFn</a></li> |
| <li><a href="/documentation/programming-guide/#combine">Beam Programming Guide: CombineFn</a></li> |
| <li><a href="/documentation/programming-guide/#data-encoding-and-type-safety">Beam Programming Guide: Coder</a></li> |
| <li><a href="/documentation/programming-guide/#side-inputs">Beam Programming Guide: Side inputs</a></li> |
| </ul> |
| <h2 id="schema">Schema</h2> |
| <p>A schema is a language-independent type definition for a <code>PCollection</code>. The |
| schema for a <code>PCollection</code> defines elements of that <code>PCollection</code> as an ordered |
| list of named fields. Each field has a name, a type, and possibly a set of user |
| options.</p> |
| <p>In many cases, the element type in a <code>PCollection</code> has a structure that can be |
| introspected. Some examples are JSON, Protocol Buffer, Avro, and database row |
| objects. All of these formats can be converted to Beam Schemas. Even within a |
| SDK pipeline, Simple Java POJOs (or equivalent structures in other languages) |
| are often used as intermediate types, and these also have a clear structure that |
| can be inferred by inspecting the class. By understanding the structure of a |
| pipeline’s records, we can provide much more concise APIs for data processing.</p> |
| <p>Beam provides a collection of transforms that operate natively on schemas. For |
| example, <a href="/documentation/dsls/sql/overview/">Beam SQL</a> is a common transform |
| that operates on schemas. These transforms allow selections and aggregations in |
| terms of named schema fields. Another advantage of schemas is that they allow |
| referencing of element fields by name. Beam provides a selection syntax for |
| referencing fields, including nested and repeated fields.</p> |
| <p>For more information about schemas, see the following pages:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#schemas">Beam Programming Guide: Schemas</a></li> |
| <li><a href="/documentation/patterns/schema/">Schema Patterns</a></li> |
| </ul> |
| <h2 id="runner">Runner</h2> |
| <p>A Beam runner runs a Beam pipeline on a specific platform. Most runners are |
| translators or adapters to massively parallel big data processing systems, such |
| as Apache Flink, Apache Spark, Google Cloud Dataflow, and more. For example, the |
| Flink runner translates a Beam pipeline into a Flink job. The Direct Runner runs |
| pipelines locally so you can test, debug, and validate that your pipeline |
| adheres to the Apache Beam model as closely as possible.</p> |
| <p>For an up-to-date list of Beam runners and which features of the Apache Beam |
| model they support, see the runner |
| <a href="/documentation/runners/capability-matrix/">capability matrix</a>.</p> |
| <p>For more information about runners, see the following pages:</p> |
| <ul> |
| <li><a href="/documentation/#choosing-a-runner">Choosing a Runner</a></li> |
| <li><a href="/documentation/runners/capability-matrix/">Beam Capability Matrix</a></li> |
| </ul> |
| <h2 id="window">Window</h2> |
| <p>Windowing subdivides a <code>PCollection</code> into <em>windows</em> according to the timestamps |
| of its individual elements. Windows enable grouping operations over unbounded |
| collections by dividing the collection into windows of finite collections.</p> |
| <p>A <em>windowing function</em> tells the runner how to assign elements to one or more |
| initial windows, and how to merge windows of grouped elements. Each element in a |
| <code>PCollection</code> can only be in one window, so if a windowing function specifies |
| multiple windows for an element, the element is conceptually duplicated into |
| each of the windows and each element is identical except for its window.</p> |
| <p>Transforms that aggregate multiple elements, such as <code>GroupByKey</code> and <code>Combine</code>, |
| work implicitly on a per-window basis; they process each <code>PCollection</code> as a |
| succession of multiple, finite windows, though the entire collection itself may |
| be of unbounded size.</p> |
| <p>Beam provides several windowing functions:</p> |
| <ul> |
| <li><strong>Fixed time windows</strong> (also known as &ldquo;tumbling windows&rdquo;) represent a consistent |
| duration, non-overlapping time interval in the data stream.</li> |
| <li><strong>Sliding time windows</strong> (also known as &ldquo;hopping windows&rdquo;) also represent time |
| intervals in the data stream; however, sliding time windows can overlap.</li> |
| <li><strong>Per-session windows</strong> define windows that contain elements that are within a |
| certain gap duration of another element.</li> |
| <li><strong>Single global window</strong>: by default, all data in a <code>PCollection</code> is assigned to |
| the single global window, and late data is discarded.</li> |
| <li><strong>Calendar-based windows</strong> (not supported by the Beam SDK for Python)</li> |
| </ul> |
| <p>You can also define your own windowing function if you have more complex |
| requirements.</p> |
| <p>For example, let&rsquo;s say we have a <code>PCollection</code> that uses fixed-time windowing, |
| with windows that are five minutes long. For each window, Beam must collect all |
| the data with an event time timestamp in the given window range (between 0:00 |
| and 4:59 in the first window, for instance). Data with timestamps outside that |
| range (data from 5:00 or later) belongs to a different window.</p> |
| <p>Two concepts are closely related to windowing and covered in the following |
| sections: <a href="#watermark">watermarks</a> and <a href="#trigger">triggers</a>.</p> |
| <p>For more information about windows, see the following page:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#windowing">Beam Programming Guide: Windowing</a></li> |
| <li><a href="/documentation/programming-guide/#setting-your-pcollections-windowing-function">Beam Programming Guide: WindowFn</a></li> |
| </ul> |
| <h2 id="watermark">Watermark</h2> |
| <p>In any data processing system, there is a certain amount of lag between the time |
| a data event occurs (the “event time”, determined by the timestamp on the data |
| element itself) and the time the actual data element gets processed at any stage |
| in your pipeline (the “processing time”, determined by the clock on the system |
| processing the element). In addition, data isn’t always guaranteed to arrive in |
| a pipeline in time order, or to always arrive at predictable intervals. For |
| example, you might have intermediate systems that don&rsquo;t preserve order, or you |
| might have two servers that timestamp data but one has a better network |
| connection.</p> |
| <p>To address this potential unpredictability, Beam tracks a <em>watermark</em>. A |
| watermark is a guess as to when all data in a certain window is expected to have |
| arrived in the pipeline. You can also think of this as “we’ll never see an |
| element with an earlier timestamp”.</p> |
| <p>Data sources are responsible for producing a watermark, and every <code>PCollection</code> |
| must have a watermark that estimates how complete the <code>PCollection</code> is. The |
| contents of a <code>PCollection</code> are complete when a watermark advances to |
| “infinity”. In this manner, you might discover that an unbounded <code>PCollection</code> |
| is finite. After the watermark progresses past the end of a window, any further |
| element that arrives with a timestamp in that window is considered <em>late data</em>.</p> |
| <p><a href="#trigger">Triggers</a> are a related concept that allow you to modify and refine |
| the windowing strategy for a <code>PCollection</code>. You can use triggers to decide when |
| each individual window aggregates and reports its results, including how the |
| window emits late elements.</p> |
| <p>For more information about watermarks, see the following page:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#watermarks-and-late-data">Beam Programming Guide: Watermarks and late data</a></li> |
| </ul> |
| <h2 id="trigger">Trigger</h2> |
| <p>When collecting and grouping data into windows, Beam uses <em>triggers</em> to |
| determine when to emit the aggregated results of each window (referred to as a |
| <em>pane</em>). If you use Beam’s default windowing configuration and default trigger, |
| Beam outputs the aggregated result when it estimates all data has arrived, and |
| discards all subsequent data for that window.</p> |
| <p>At a high level, triggers provide two additional capabilities compared to |
| outputting at the end of a window:</p> |
| <ol> |
| <li>Triggers allow Beam to emit early results, before all the data in a given |
| window has arrived. For example, emitting after a certain amount of time |
| elapses, or after a certain number of elements arrives.</li> |
| <li>Triggers allow processing of late data by triggering after the event time |
| watermark passes the end of the window.</li> |
| </ol> |
| <p>These capabilities allow you to control the flow of your data and also balance |
| between data completeness, latency, and cost.</p> |
| <p>Beam provides a number of pre-built triggers that you can set:</p> |
| <ul> |
| <li><strong>Event time triggers</strong>: These triggers operate on the event time, as |
| indicated by the timestamp on each data element. Beam’s default trigger is |
| event time-based.</li> |
| <li><strong>Processing time triggers</strong>: These triggers operate on the processing time, |
| which is the time when the data element is processed at any given stage in |
| the pipeline.</li> |
| <li><strong>Data-driven triggers</strong>: These triggers operate by examining the data as it |
| arrives in each window, and firing when that data meets a certain property. |
| Currently, data-driven triggers only support firing after a certain number of |
| data elements.</li> |
| <li><strong>Composite triggers</strong>: These triggers combine multiple triggers in various |
| ways. For example, you might want one trigger for early data and a different |
| trigger for late data.</li> |
| </ul> |
| <p>For more information about triggers, see the following page:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#triggers">Beam Programming Guide: Triggers</a></li> |
| </ul> |
| <h2 id="state-and-timers">State and timers</h2> |
| <p>Beam’s windowing and triggers provide an abstraction for grouping and |
| aggregating unbounded input data based on timestamps. However, there are |
| aggregation use cases that might require an even higher degree of control. State |
| and timers are two important concepts that help with these uses cases. Like |
| other aggregations, state and timers are processed per window.</p> |
| <p><strong>State</strong>:</p> |
| <p>Beam provides the State API for manually managing per-key state, allowing for |
| fine-grained control over aggregations. The State API lets you augment |
| element-wise operations (for example, <code>ParDo</code> or <code>Map</code>) with mutable state. Like |
| other aggregations, state is processed per window.</p> |
| <p>The State API models state per key. To use the state API, you start out with a |
| keyed <code>PCollection</code>. A <code>ParDo</code> that processes this <code>PCollection</code> can declare |
| persistent state variables. When you process each element inside the <code>ParDo</code>, |
| you can use the state variables to write or update state for the current key or |
| to read previous state written for that key. State is always fully scoped only |
| to the current processing key.</p> |
| <p>Beam provides several types of state, though different runners might support a |
| different subset of these states.</p> |
| <ul> |
| <li><strong>ValueState</strong>: ValueState is a scalar state value. For each key in the |
| input, a ValueState stores a typed value that can be read and modified inside |
| the <code>DoFn</code>.</li> |
| <li>A common use case for state is to accumulate multiple elements into a group: |
| <ul> |
| <li><strong>BagState</strong>: BagState allows you to accumulate elements in an unordered |
| bag. This lets you add elements to a collection without needing to read any |
| of the previously accumulated elements.</li> |
| <li><strong>MapState</strong>: MapState allows you to accumulate elements in a map.</li> |
| <li><strong>SetState</strong>: SetState allows you to accumulate elements in a set.</li> |
| <li><strong>OrderedListState</strong>: OrderedListState allows you to accumulate elements in |
| a timestamp-sorted list.</li> |
| </ul> |
| </li> |
| <li><strong>CombiningState</strong>: CombiningState allows you to create a state object that |
| is updated using a Beam combiner. Like BagState, you can add elements to an |
| aggregation without needing to read the current value, and the accumulator |
| can be compacted using a combiner.</li> |
| </ul> |
| <p>You can use the State API together with the Timer API to create processing tasks |
| that give you fine-grained control over the workflow.</p> |
| <p><strong>Timers</strong>:</p> |
| <p>Beam provides a per-key timer callback API that enables delayed processing of |
| data stored using the State API. The Timer API lets you set timers to call back |
| at either an event-time or a processing-time timestamp. For more advanced use |
| cases, your timer callback can set another timer. Like other aggregations, |
| timers are processed per window. You can use the timer API together with the |
| State API to create processing tasks that give you fine-grained control over the |
| workflow.</p> |
| <p>The following timers are available:</p> |
| <ul> |
| <li><strong>Event-time timers</strong>: Event-time timers fire when the input watermark for |
| the <code>DoFn</code> passes the time at which the timer is set, meaning that the runner |
| believes that there are no more elements to be processed with timestamps |
| before the timer timestamp. This allows for event-time aggregations.</li> |
| <li><strong>Processing-time timers</strong>: Processing-time timers fire when the real wall-clock |
| time passes. This is often used to create larger batches of data before |
| processing. It can also be used to schedule events that should occur at a |
| specific time.</li> |
| <li><strong>Dynamic timer tags</strong>: Beam also supports dynamically setting a timer tag. This |
| allows you to set multiple different timers in a <code>DoFn</code> and dynamically |
| choose timer tags (for example, based on data in the input elements).</li> |
| </ul> |
| <p>For more information about state and timers, see the following pages:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#state-and-timers">Beam Programming Guide: State and Timers</a></li> |
| <li><a href="/blog/stateful-processing/">Stateful processing with Apache Beam</a></li> |
| <li><a href="/blog/timely-processing/">Timely (and Stateful) Processing with Apache Beam</a></li> |
| </ul> |
| <h2 id="splittable-dofn">Splittable DoFn</h2> |
| <p>Splittable <code>DoFn</code> (SDF) is a generalization of <code>DoFn</code> that lets you process |
| elements in a non-monolithic way. Splittable <code>DoFn</code> makes it easier to create |
| complex, modular I/O connectors in Beam.</p> |
| <p>A regular <code>ParDo</code> processes an entire element at a time, applying your regular |
| <code>DoFn</code> and waiting for the call to terminate. When you instead apply a |
| splittable <code>DoFn</code> to each element, the runner has the option of splitting the |
| element&rsquo;s processing into smaller tasks. You can checkpoint the processing of an |
| element, and you can split the remaining work to yield additional parallelism.</p> |
| <p>For example, imagine you want to read every line from very large text files. |
| When you write your splittable <code>DoFn</code>, you can have separate pieces of logic to |
| read a segment of a file, split a segment of a file into sub-segments, and |
| report progress through the current segment. The runner can then invoke your |
| splittable <code>DoFn</code> intelligently to split up each input and read portions |
| separately, in parallel.</p> |
| <p>A common computation pattern has the following steps:</p> |
| <ol> |
| <li>The runner splits an incoming element before starting any processing.</li> |
| <li>The runner starts running your processing logic on each sub-element.</li> |
| <li>If the runner notices that some sub-elements are taking longer than others, |
| the runner splits those sub-elements further and repeats step 2.</li> |
| <li>The sub-element either finishes processing, or the user chooses to |
| checkpoint the sub-element and the runner repeats step 2.</li> |
| </ol> |
| <p>You can also write your splittable <code>DoFn</code> so the runner can split the unbounded |
| processing. For example, if you write a splittable <code>DoFn</code> to watch a set of |
| directories and output filenames as they arrive, you can split to subdivide the |
| work of different directories. This allows the runner to split off a hot |
| directory and give it additional resources.</p> |
| <p>For more information about Splittable <code>DoFn</code>, see the following pages:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#splittable-dofns">Splittable DoFns</a></li> |
| <li><a href="/blog/splittable-do-fn-is-available/">Splittable DoFn in Apache Beam is Ready to Use</a></li> |
| </ul> |
| <h2 id="whats-next">What&rsquo;s next</h2> |
| <p>Take a look at our <a href="/documentation/">other documentation</a> such as the Beam |
| programming guide, pipeline execution information, and transform reference |
| catalogs.</p></description></item><item><title>Documentation: BatchElements</title><link>/documentation/transforms/python/aggregation/batchelements/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/python/aggregation/batchelements/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="batchelements">BatchElements</h1> |
| <script type="text/javascript"> |
| localStorage.setItem("language", "language-py") |
| </script> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.stat.html#apache_beam.transforms.stat.BatchElements" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p> |
| <h2 id="examples">Examples</h2> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_BatchElements" |
| data-show="batchelements" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_BatchElements%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22batchelements%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2> |
| <ul> |
| <li><a href="/documentation/transforms/python/aggregation/groupintobatches">GroupIntoBatches</a> batches elements by key</li> |
| </ul></description></item><item><title>Documentation: BatchElements</title><link>/documentation/transforms/python/aggregation/tolist/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/python/aggregation/tolist/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="batchelements">BatchElements</h1> |
| <script type="text/javascript"> |
| localStorage.setItem("language", "language-py") |
| </script> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.ToList" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p> |
| <h2 id="examples">Examples</h2> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_ToList" |
| data-show="tolist" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_ToList%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22tolist%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div></description></item><item><title>Documentation: Beam glossary</title><link>/documentation/glossary/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/glossary/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="apache-beam-glossary">Apache Beam glossary</h1> |
| <h2 id="aggregation">Aggregation</h2> |
| <p>A transform pattern for computing a value from multiple input elements. Aggregation is similar to the reduce operation in the <a href="https://en.wikipedia.org/wiki/MapReduce">MapReduce</a> model. Aggregation transforms include Combine (applies a user-defined function to all elements in the aggregation), Count (computes the count of all elements in the aggregation), Max (computes the maximum element in the aggregation), and Sum (computes the sum of all elements in the aggregation).</p> |
| <p>For a list of built-in aggregation transforms, see:</p> |
| <ul> |
| <li><a href="/documentation/transforms/java/overview/#aggregation">Java Transform catalog</a></li> |
| <li><a href="/documentation/transforms/python/overview/#aggregation">Python Transform catalog</a></li> |
| </ul> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/basics/#aggregation">Basics of the Beam model: Aggregation</a></li> |
| </ul> |
| <h2 id="apply">Apply</h2> |
| <p>A method for invoking a transform on an input PCollection (or set of PCollections) to produce one or more output PCollections. The <code>apply</code> method is attached to the PCollection (or value). Invoking multiple Beam transforms is similar to method chaining, but with a difference: You apply the transform to the input PCollection, passing the transform itself as an argument, and the operation returns the output PCollection. Because of Beam’s deferred execution model, applying a transform does not immediately execute that transform.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#applying-transforms">Applying transforms</a></li> |
| </ul> |
| <h2 id="batch-processing">Batch processing</h2> |
| <p>A data processing paradigm for working with finite, or bounded, datasets. A bounded PCollection represents a dataset of a known, fixed size. Reading from a batch data source, such as a file or a database, creates a bounded PCollection. A batch processing job eventually ends, in contrast to a streaming job, which runs until cancelled.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#size-and-boundedness">Size and boundedness</a></li> |
| </ul> |
| <h2 id="bounded-data">Bounded data</h2> |
| <p>A dataset of a known, fixed size (alternatively, a dataset that is not growing over time). A PCollection can be bounded or unbounded, depending on the source of the data that it represents. Reading from a batch data source, such as a file or a database, creates a bounded PCollection. Beam also supports reading a bounded amount of data from an unbounded source.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#size-and-boundedness">Size and boundedness</a></li> |
| </ul> |
| <h2 id="bundle">Bundle</h2> |
| <p>The processing and commit/retry unit for elements in a PCollection. Instead of processing all elements in a PCollection simultaneously, Beam processes the elements in bundles. The runner handles the division of the collection into bundles, and in doing so it may optimize the bundle size for the use case. For example, a streaming runner might process smaller bundles than a batch runner.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/runtime/model/#bundling-and-persistence">Bundling and persistence</a></li> |
| </ul> |
| <h2 id="coder">Coder</h2> |
| <p>A component that describes how the elements of a PCollection can be encoded and decoded. To support distributed processing and cross-language portability, Beam needs to be able to encode each element of a PCollection as bytes. The Beam SDKs provide built-in coders for common types and language-specific mechanisms for specifying the encoding of a PCollection.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#data-encoding-and-type-safety">Data encoding and type safety</a></li> |
| </ul> |
| <h2 id="cogroupbykey">CoGroupByKey</h2> |
| <p>A PTransform that takes two or more PCollections and aggregates the elements by key. In effect, CoGroupByKey performs a relational join of two or more key/value PCollections that have the same key type. While GroupByKey performs this operation over a single input collection, CoGroupByKey operates over multiple input collections.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#cogroupbykey">CoGroupByKey</a></li> |
| <li><a href="/documentation/transforms/java/aggregation/cogroupbykey/">CoGroupByKey (Java)</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/cogroupbykey/">CoGroupByKey (Python)</a></li> |
| </ul> |
| <h2 id="collection">Collection</h2> |
| <p>See <a href="/documentation/glossary/#pcollection">PCollection</a>.</p> |
| <h2 id="combine">Combine</h2> |
| <p>A PTransform for combining all elements of a PCollection or all values associated with a key. When you apply a Combine transform, you have to provide a user-defined function (UDF) that contains the logic for combining the elements or values. The combining function should be <a href="https://en.wikipedia.org/wiki/Commutative_property">commutative</a> and <a href="https://en.wikipedia.org/wiki/Associative_property">associative</a>, because the function is not necessarily invoked exactly once on all values with a given key.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#combine">Combine</a></li> |
| <li><a href="/documentation/transforms/java/aggregation/combine/">Combine (Java)</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/combineglobally/">CombineGlobally (Python)</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/combineperkey/">CombinePerKey (Python)</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/combinevalues/">CombineValues (Python)</a></li> |
| </ul> |
| <h2 id="composite-transform">Composite transform</h2> |
| <p>A PTransform that expands into many PTransforms. Composite transforms have a nested structure, in which a complex transform applies one or more simpler transforms. These simpler transforms could be existing Beam operations like ParDo, Combine, or GroupByKey, or they could be other composite transforms. Nesting multiple transforms inside a single composite transform can make your pipeline more modular and easier to understand. Many of the built-in transforms are composite transforms.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#composite-transforms">Composite transforms</a></li> |
| </ul> |
| <h2 id="counter-metric">Counter (metric)</h2> |
| <p>A metric that reports a single long value and can be incremented. In the Beam model, metrics provide insight into the state of a pipeline, potentially while the pipeline is running.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#types-of-metrics">Types of metrics</a></li> |
| </ul> |
| <h2 id="cross-language-transforms">Cross-language transforms</h2> |
| <p>Transforms that can be shared across Beam SDKs. With cross-language transforms, you can use transforms written in any supported SDK language (currently, Java and Python) in a pipeline written in a different SDK language. For example, you could use the Apache Kafka connector from the Java SDK in a Python streaming pipeline. Cross-language transforms make it possible to provide new functionality simultaneously in different SDKs.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#multi-language-pipelines">Multi-language pipelines</a></li> |
| </ul> |
| <h2 id="deferred-execution">Deferred execution</h2> |
| <p>A feature of the Beam execution model. Beam operations are deferred, meaning that the result of a given operation may not be available for control flow. Deferred execution allows the Beam API to support parallel processing of data and perform pipeline-level optimizations.</p> |
| <h2 id="distribution-metric">Distribution (metric)</h2> |
| <p>A metric that reports information about the distribution of reported values. In the Beam model, metrics provide insight into the state of a pipeline, potentially while the pipeline is running.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#types-of-metrics">Types of metrics</a></li> |
| </ul> |
| <h2 id="dofn">DoFn</h2> |
| <p>A function object used by ParDo (or some other transform) to process the elements of a PCollection, often producing elements for an output PCollection. A DoFn is a user-defined function, meaning that it contains custom code that defines a data processing task in your pipeline. The Beam system invokes a DoFn one or more times to process some arbitrary bundle of elements, but Beam doesn’t guarantee an exact number of invocations.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#pardo">ParDo</a></li> |
| </ul> |
| <h2 id="driver">Driver</h2> |
| <p>A program that defines your pipeline, including all of the inputs, transforms, and outputs. To use Beam, you need to create a driver program using classes from one of the Beam SDKs. The driver program creates a pipeline and specifies the execution options that tell the pipeline where and how to run. These options include the runner, which determines what backend your pipeline will run on.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#overview">Overview</a></li> |
| </ul> |
| <h2 id="element">Element</h2> |
| <p>The unit of data in a PCollection. Elements in a PCollection can be of any type, but they must all have the same type. This allows parallel computations to operate uniformly across the entire collection. Some element types have a structure that can be introspected (for example, JSON, Protocol Buffer, Avro, and database records).</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#pcollection-characteristics">PCollection characteristics</a></li> |
| </ul> |
| <h2 id="element-wise">Element-wise</h2> |
| <p>A type of transform that independently processes each element in an input PCollection. Element-wise is similar to the map operation in the <a href="https://en.wikipedia.org/wiki/MapReduce">MapReduce</a> model. An element-wise transform might output 0, 1, or multiple values for each input element. This is in contrast to aggregation transforms, which compute a single value from multiple input elements. Element-wise operations include Filter, FlatMap, and ParDo.</p> |
| <p>For a complete list of element-wise transforms, see:</p> |
| <ul> |
| <li><a href="/documentation/transforms/java/overview/#element-wise">Java Transform catalog</a></li> |
| <li><a href="/documentation/transforms/python/overview/#element-wise">Python Transform catalog</a></li> |
| </ul> |
| <h2 id="engine">Engine</h2> |
| <p>A data-processing system, such as Dataflow, Spark, or Flink. A Beam runner for an engine executes a Beam pipeline on that engine.</p> |
| <h2 id="event-time">Event time</h2> |
| <p>The time a data event occurs, determined by a timestamp on an element. This is in contrast to processing time, which is when an element is processed in a pipeline. An event could be, for example, a user interaction or a write to an error log. There’s no guarantee that events will appear in a pipeline in order of event time, but windowing and timers let you reason correctly about event time.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#watermarks-and-late-data">Watermarks and late data</a></li> |
| <li><a href="/documentation/programming-guide/#triggers">Triggers</a></li> |
| </ul> |
| <h2 id="expansion-service">Expansion Service</h2> |
| <p>A service that enables a pipeline to apply (expand) cross-language transforms defined in other SDKs. For example, by connecting to a Java expansion service, the Python SDK can apply transforms implemented in Java. Currently, SDKs typically start up expansion services as local processes, but in the future Beam may support long-running expansion services. The development of expansion services is part of the ongoing effort to support multi-language pipelines.</p> |
| <h2 id="flatten">Flatten</h2> |
| <p>One of the core PTransforms. Flatten merges multiple PCollections into a single logical PCollection.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#flatten">Flatten</a></li> |
| <li><a href="/documentation/transforms/java/other/flatten/">Flatten (Java)</a></li> |
| <li><a href="/documentation/transforms/python/other/flatten/">Flatten (Python)</a></li> |
| </ul> |
| <h2 id="fn-api">Fn API</h2> |
| <p>An interface that lets a runner invoke SDK-specific user-defined functions. The Fn API, together with the Runner API, supports the ability to mix and match SDKs and runners. Used together, the Fn and Runner APIs let new SDKs run on every runner, and let new runners run pipelines from every SDK.</p> |
| <h2 id="fusion">Fusion</h2> |
| <p>An optimization that Beam runners can apply before running a pipeline. When one transform outputs a PCollection that’s consumed by another transform, or when two or more transforms take the same PCollection as input, a runner may be able to fuse the transforms together into a single processing unit (a <em>stage</em> in Dataflow). The consuming DoFn processes elements as they are emitted by the producing DoFn, rather than waiting for the entire intermediate PCollection to be computed. Fusion can make pipeline execution more efficient by preventing I/O operations.</p> |
| <h2 id="gauge-metric">Gauge (metric)</h2> |
| <p>A metric that reports the latest value out of reported values. In the Beam model, metrics provide insight into the state of a pipeline, potentially while the pipeline is running. Because metrics are collected from many workers, the gauge value may not be the absolute last value, but it will be one of the latest values produced by one of the workers.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#types-of-metrics">Types of metrics</a></li> |
| </ul> |
| <h2 id="groupbykey">GroupByKey</h2> |
| <p>A PTransform for processing collections of key/value pairs. GroupByKey is a parallel reduction operation, similar to the shuffle of a map/shuffle/reduce algorithm. The input to GroupByKey is a collection of key/value pairs in which multiple pairs have the same key but different values (i.e. a multimap). You can use GroupByKey to collect all of the values associated with each unique key.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#groupbykey">GroupByKey</a></li> |
| <li><a href="/documentation/transforms/java/aggregation/groupbykey/">GroupByKey (Java)</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/groupbykey/">GroupByKey (Python)</a></li> |
| </ul> |
| <h2 id="io-connector">I/O connector</h2> |
| <p>A set of PTransforms for working with external data storage systems. When you create a pipeline, you often need to read from or write to external data systems such as files or databases. Beam provides read and write transforms for a number of common data storage types.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#pipeline-io">Pipeline I/O</a></li> |
| <li><a href="/documentation/io/built-in/">Built-in I/O Transforms</a></li> |
| </ul> |
| <h2 id="map">Map</h2> |
| <p>An element-wise PTransform that applies a user-defined function (UDF) to each element in a PCollection. Using Map, you can transform each individual element into a new element, but you can&rsquo;t change the number of elements.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/transforms/python/elementwise/map/">Map (Python)</a></li> |
| <li><a href="/documentation/transforms/java/elementwise/mapelements/">MapElements (Java)</a></li> |
| </ul> |
| <h2 id="metrics">Metrics</h2> |
| <p>Data on the state of a pipeline, potentially while the pipeline is running. You can use the built-in Beam metrics to gain insight into the functioning of your pipeline. For example, you might use Beam metrics to track errors, calls to a backend service, or the number of elements processed. Beam currently supports three types of metric: Counter, Distribution, and Gauge.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#metrics">Metrics</a></li> |
| </ul> |
| <h2 id="multi-language-pipeline">Multi-language pipeline</h2> |
| <p>A pipeline that uses cross-language transforms. You can combine transforms written in any supported SDK language (currently, Java and Python) and use them in one multi-language pipeline.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#multi-language-pipelines">Multi-language pipelines</a></li> |
| </ul> |
| <h2 id="pardo">ParDo</h2> |
| <p>The lowest-level element-wise PTransform. For each element in an input PCollection, ParDo applies a function and emits zero, one, or multiple elements to an output PCollection. “ParDo” is short for “Parallel Do.” It’s similar to the map operation in a <a href="https://en.wikipedia.org/wiki/MapReduce">MapReduce</a> algorithm and the reduce operation when following a GroupByKey. ParDo is also comparable to the <code>apply</code> method from a DataFrame, or the <code>UPDATE</code> keyword from SQL.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#pardo">ParDo</a></li> |
| <li><a href="/documentation/transforms/java/elementwise/pardo/">ParDo (Java)</a></li> |
| <li><a href="/documentation/transforms/python/elementwise/pardo/">ParDo (Python)</a></li> |
| </ul> |
| <h2 id="partition">Partition</h2> |
| <p>An element-wise PTransform that splits a single PCollection into a fixed number of smaller, disjoint PCollections. Partition requires a user-defined function (UDF) to determine how to split up the elements of the input collection into the resulting output collections. The number of partitions must be determined at graph construction time, meaning that you can’t determine the number of partitions using data calculated by the running pipeline.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#partition">Partition</a></li> |
| <li><a href="/documentation/transforms/java/elementwise/partition/">Partition (Java)</a></li> |
| <li><a href="/documentation/transforms/python/elementwise/partition/">Partition (Python)</a></li> |
| </ul> |
| <h2 id="pcollection">PCollection</h2> |
| <p>A potentially distributed, homogeneous dataset or data stream. PCollections represent data in a Beam pipeline, and Beam transforms (PTransforms) use PCollection objects as inputs and outputs. PCollections are intended to be immutable, meaning that once a PCollection is created, you can’t add, remove, or change individual elements. The “P” stands for “parallel.”</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/basics/#pcollection">Basics of the Beam model: PCollection</a></li> |
| <li><a href="/documentation/programming-guide/#pcollections">Programming guide: PCollections</a></li> |
| </ul> |
| <h2 id="pipe-operator-">Pipe operator (<code>|</code>)</h2> |
| <p>Delimits a step in a Python pipeline. For example: <code>[Final Output PCollection] = ([Initial Input PCollection] | [First Transform] | [Second Transform] | [Third Transform])</code>. The output of each transform is passed from left to right as input to the next transform. The pipe operator in Python is equivalent to the <code>apply</code> method in Java (in other words, the pipe applies a transform to a PCollection), and usage is similar to the pipe operator in shell scripts, which lets you pass the output of one program into the input of another.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#applying-transforms">Applying transforms</a></li> |
| </ul> |
| <h2 id="pipeline">Pipeline</h2> |
| <p>An encapsulation of your entire data processing task, including reading input data from a source, transforming that data, and writing output data to a sink. You can think of a pipeline as a Beam program that uses PTransforms to process PCollections. (Alternatively, you can think of it as a single, executable composite PTransform with no inputs or outputs.) The transforms in a pipeline can be represented as a directed acyclic graph (DAG). All Beam driver programs must create a pipeline.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/basics/#pipeline">Basics of the Beam model: Pipeline</a></li> |
| <li><a href="/documentation/programming-guide/#overview">Overview</a></li> |
| <li><a href="/documentation/programming-guide/#creating-a-pipeline">Creating a pipeline</a></li> |
| <li><a href="/documentation/pipelines/design-your-pipeline/">Design your pipeline</a></li> |
| <li><a href="/documentation/pipelines/create-your-pipeline/">Create your pipeline</a></li> |
| </ul> |
| <h2 id="processing-time">Processing time</h2> |
| <p>The real-world time at which an element is processed at some stage in a pipeline. Processing time is not the same as event time, which is the time at which a data event occurs. Processing time is determined by the clock on the system processing the element. There’s no guarantee that elements will be processed in order of event time.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#watermarks-and-late-data">Watermarks and late data</a></li> |
| <li><a href="/documentation/programming-guide/#triggers">Triggers</a></li> |
| </ul> |
| <h2 id="ptransform">PTransform</h2> |
| <p>A data processing operation, or a step, in your pipeline. A PTransform takes zero or more PCollections as input, applies a processing function to the elements of that PCollection, and produces zero or more output PCollections. Some PTransforms accept user-defined functions that apply custom logic. The “P” stands for “parallel.”</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/basics/#ptransform">Basics of the Beam model: PTransform</a></li> |
| <li><a href="/documentation/programming-guide/#overview">Overview</a></li> |
| <li><a href="/documentation/programming-guide/#transforms">Transforms</a></li> |
| </ul> |
| <h2 id="resource-hints">Resource hints</h2> |
| <p>A Beam feature that lets you provide information to a runner about the compute resource requirements of your pipeline. You can use resource hints to define requirements for specific transforms or for an entire pipeline. For example, you could use a resource hint to specify the minimum amount of memory to allocate to workers. The runner is responsible for interpreting resource hints, and runners can ignore unsupported hints.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/runtime/resource-hints">Resource hints</a></li> |
| </ul> |
| <h2 id="runner">Runner</h2> |
| <p>A runner runs a pipeline on a specific platform. Most runners are translators or adapters to massively parallel big data processing systems. Other runners exist for local testing and debugging. Among the supported runners are Google Cloud Dataflow, Apache Spark, Apache Samza, Apache Flink, the Interactive Runner, and the Direct Runner.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/basics/#runner">Basics of the Beam model: Runner</a></li> |
| <li><a href="/documentation/#choosing-a-runner">Choosing a Runner</a></li> |
| <li><a href="/documentation/runners/capability-matrix/">Beam Capability Matrix</a></li> |
| </ul> |
| <h2 id="schema">Schema</h2> |
| <p>A language-independent type definition for the elements of a PCollection. The schema for a PCollection defines elements of that PCollection as an ordered list of named fields. Each field has a name, a type, and possibly a set of user options. Schemas provide a way to reason about types across different programming-language APIs. They also let you describe data transformations more succinctly and at a higher level.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/basics/#schema">Basics of the Beam model: Schema</a></li> |
| <li><a href="/documentation/programming-guide/#schemas">Programming guide: Schemas</a></li> |
| <li><a href="/documentation/patterns/schema/">Schema Patterns</a></li> |
| </ul> |
| <h2 id="session">Session</h2> |
| <p>A time interval for grouping data events. A session is defined by some minimum gap duration between events. For example, a data stream representing user mouse activity may have periods with high concentrations of clicks followed by periods of inactivity. A session can represent such a pattern of activity delimited by inactivity.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#session-windows">Session windows</a></li> |
| <li><a href="/get-started/mobile-gaming-example/#analyzing-usage-patterns">Analyzing Usage Patterns</a></li> |
| </ul> |
| <h2 id="side-input">Side input</h2> |
| <p>Additional input to a PTransform that is provided in its entirety, rather than element-by-element. Side input is input that you provide in addition to the main input PCollection. A DoFn can access side input each time it processes an element in the PCollection.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#side-inputs">Side inputs</a></li> |
| <li><a href="/documentation/patterns/side-inputs/">Side input patterns</a></li> |
| </ul> |
| <h2 id="sink">Sink</h2> |
| <p>A transform that writes to an external data storage system, like a file or database.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/io/developing-io-overview/">Developing new I/O connectors</a></li> |
| <li><a href="/documentation/programming-guide/#pipeline-io">Pipeline I/O</a></li> |
| <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li> |
| </ul> |
| <h2 id="source">Source</h2> |
| <p>A transform that reads from an external storage system. A pipeline typically reads input data from a source. The source has a type, which may be different from the sink type, so you can change the format of data as it moves through your pipeline.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/io/developing-io-overview/">Developing new I/O connectors</a></li> |
| <li><a href="/documentation/programming-guide/#pipeline-io">Pipeline I/O</a></li> |
| <li><a href="/documentation/io/built-in/">Built-in I/O transforms</a></li> |
| </ul> |
| <h2 id="splittable-dofn">Splittable DoFn</h2> |
| <p>A generalization of DoFn that makes it easier to create complex, modular I/O connectors. A Splittable DoFn (SDF) can process elements in a non-monolithic way, meaning that the processing can be decomposed into smaller tasks. With SDF, you can check-point the processing of an element, and you can split the remaining work to yield additional parallelism. SDF is recommended for building new I/O connectors.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/basics/#splittable-dofn">Basics of the Beam model: Splittable DoFn</a></li> |
| <li><a href="/documentation/programming-guide/#splittable-dofns">Programming guide: Splittable DoFns</a></li> |
| <li><a href="/blog/splittable-do-fn-is-available/">Splittable DoFn in Apache Beam is Ready to Use</a></li> |
| </ul> |
| <h2 id="stage">Stage</h2> |
| <p>The unit of fused transforms in a pipeline. Runners can perform fusion optimization to make pipeline execution more efficient. In Dataflow, the pipeline is conceptualized as a graph of fused stages.</p> |
| <h2 id="state">State</h2> |
| <p>Persistent values that a PTransform can access. The state API lets you augment element-wise operations (for example, ParDo or Map) with mutable state. Using the state API, you can read from, and write to, state as you process each element of a PCollection. You can use the state API together with the timer API to create processing tasks that give you fine-grained control over the workflow. State is always local to a key and window.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/basics/#state-and-timers">Basics of the Beam model: State and timers</a></li> |
| <li><a href="/documentation/programming-guide/#state-and-timers">Programming guide: State and Timers</a></li> |
| <li><a href="/blog/stateful-processing/">Stateful processing with Apache Beam</a></li> |
| </ul> |
| <h2 id="streaming">Streaming</h2> |
| <p>A data processing paradigm for working with infinite, or unbounded, datasets. Reading from a streaming data source, such as Pub/Sub or Kafka, creates an unbounded PCollection. An unbounded PCollection must be processed using a job that runs continuously, because the entire collection can never be available for processing at any one time.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#size-and-boundedness">Size and boundedness</a></li> |
| <li><a href="/documentation/sdks/python-streaming/">Python Streaming Pipelines</a></li> |
| </ul> |
| <h1 id="timer">Timer</h1> |
| <p>A Beam feature that enables delayed processing of data stored using the state API. The timer API lets you set timers to call back at either an event-time or a processing-time timestamp. You can use the timer API together with the state API to create processing tasks that give you fine-grained control over the workflow.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/basics/#state-and-timers">Basics of the Beam model: State and timers</a></li> |
| <li><a href="/documentation/programming-guide/#state-and-timers">Programming guide: State and Timers</a></li> |
| <li><a href="/blog/stateful-processing/">Stateful processing with Apache Beam</a></li> |
| <li><a href="/blog/timely-processing/">Timely (and Stateful) Processing with Apache Beam</a></li> |
| </ul> |
| <h2 id="timestamp">Timestamp</h2> |
| <p>A point in event time associated with an element in a PCollection and used to assign a window to the element. The source that creates the PCollection assigns each element an initial timestamp, often corresponding to when the element was read or added. But you can also manually assign timestamps. This can be useful if elements have an inherent timestamp, but the timestamp is somewhere in the structure of the element itself (for example, a time field in a server log entry).</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/basics/#timestamp">Basics of the Beam model: Timestamp</a></li> |
| <li><a href="/documentation/programming-guide/#element-timestamps">Element timestamps</a></li> |
| <li><a href="/documentation/programming-guide/#adding-timestamps-to-a-pcollections-elements">Adding timestamps to a PCollection’s elements</a></li> |
| </ul> |
| <h2 id="transform">Transform</h2> |
| <p>See PTransform.</p> |
| <h2 id="trigger">Trigger</h2> |
| <p>Determines when to emit aggregated result data from a window. You can use triggers to refine the windowing strategy for your pipeline. If you use the default windowing configuration and default trigger, Beam outputs an aggregated result when it estimates that all data for a window has arrived, and it discards all subsequent data for that window. But you can also use triggers to emit early results, before all the data in a given window has arrived, or to process late data by triggering after the event time watermark passes the end of the window.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/basics/#trigger">Basics of the Beam model: Trigger</a></li> |
| <li><a href="/documentation/programming-guide/#triggers">Programming guide: Triggers</a></li> |
| </ul> |
| <h2 id="unbounded-data">Unbounded data</h2> |
| <p>A dataset that grows over time, with elements processed as they arrive. A PCollection can be bounded or unbounded, depending on the source of the data that it represents. Reading from a streaming or continuously-updating data source, such as Pub/Sub or Kafka, typically creates an unbounded PCollection.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/programming-guide/#size-and-boundedness">Size and boundedness</a></li> |
| </ul> |
| <h2 id="user-defined-function">User-defined function</h2> |
| <p>Custom logic that a PTransform applies to your data. Some PTransforms accept a user-defined function (UDF) as a way to configure the transform. For example, ParDo expects user code in the form of a DoFn object. Each language SDK has its own idiomatic way of expressing user-defined functions, but there are some common requirements, like serializability and thread compatibility.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/basics/#user-defined-functions-udfs">Basics of the Beam model: User-Defined Functions (UDFs)</a></li> |
| <li><a href="/documentation/programming-guide/#pardo">ParDo</a></li> |
| <li><a href="/documentation/programming-guide/#requirements-for-writing-user-code-for-beam-transforms">Requirements for writing user code for Beam transforms</a></li> |
| </ul> |
| <h2 id="watermark">Watermark</h2> |
| <p>An estimate on the lower bound of the timestamps that will be seen (in the future) at this point of the pipeline. Watermarks provide a way to estimate the completeness of input data. Every PCollection has an associated watermark. Once the watermark progresses past the end of a window, any element that arrives with a timestamp in that window is considered late data.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/basics/#watermark">Basics of the Beam model: Watermark</a></li> |
| <li><a href="/documentation/programming-guide/#watermarks-and-late-data">Programming guide: Watermarks and late data</a></li> |
| </ul> |
| <h2 id="windowing">Windowing</h2> |
| <p>Partitioning a PCollection into bounded subsets grouped by the timestamps of individual elements. In the Beam model, any PCollection – including unbounded PCollections – can be subdivided into logical windows. Each element in a PCollection is assigned to one or more windows according to the PCollection&rsquo;s windowing function, and each individual window contains a finite number of elements. Transforms that aggregate multiple elements, such as GroupByKey and Combine, work implicitly on a per-window basis.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/basics/#window">Basics of the Beam model: Window</a></li> |
| <li><a href="/documentation/programming-guide/#windowing">Programming guide: Windowing</a></li> |
| </ul> |
| <h2 id="worker">Worker</h2> |
| <p>A container, process, or virtual machine (VM) that handles some part of the parallel processing of a pipeline. Each worker node has its own independent copy of state. A Beam runner might serialize elements between machines for communication purposes and for other reasons such as persistence.</p> |
| <p>To learn more, see:</p> |
| <ul> |
| <li><a href="/documentation/runtime/model/">Execution model</a></li> |
| </ul></description></item><item><title>Documentation: Beam Programming Guide</title><link>/documentation/programming-guide/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/programming-guide/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="apache-beam-programming-guide">Apache Beam Programming Guide</h1> |
| <p>The <strong>Beam Programming Guide</strong> is intended for Beam users who want to use the |
| Beam SDKs to create data processing pipelines. It provides guidance for using |
| the Beam SDK classes to build and test your pipeline. The programming guide is |
| not intended as an exhaustive reference, but as a language-agnostic, high-level |
| guide to programmatically building your Beam pipeline. As the programming guide |
| is filled out, the text will include code samples in multiple languages to help |
| illustrate how to implement Beam concepts in your pipelines.</p> |
| <p>If you want a brief introduction to Beam&rsquo;s basic concepts before reading the |
| programming guide, take a look at the |
| <a href="/documentation/basics/">Basics of the Beam model</a> page.</p> |
| <nav class="language-switcher"> |
| <strong>Adapt for:</strong> |
| <ul> |
| <li data-value="java" class="active">Java SDK</li> |
| <li data-value="py">Python SDK</li> |
| <li data-value="go">Go SDK</li> |
| <li data-value="typescript">TypeScript SDK</li> |
| <li data-value="yaml">Yaml API</li> |
| </ul> |
| </nav> |
| <p class="language-py">The Python SDK supports Python 3.8, 3.9, 3.10, and 3.11.</p> |
| <p class="language-go">The <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam">Go SDK</a> supports Go v1.20+.</p> |
| <p class="language-typescript">The Typescript SDK supports Node v16+ and is still experimental.</p> |
| <p class="language-yaml">YAML is supported as of Beam 2.52, but is under active development and the most |
| recent SDK is advised.</p> |
| <h2 id="overview">1. Overview</h2> |
| <p>To use Beam, you need to first create a driver program using the classes in one |
| of the Beam SDKs. Your driver program <em>defines</em> your pipeline, including all of |
| the inputs, transforms, and outputs; it also sets execution options for your |
| pipeline (typically passed in using command-line options). These include the |
| Pipeline Runner, which, in turn, determines what back-end your pipeline will run |
| on.</p> |
| <p>The Beam SDKs provide a number of abstractions that simplify the mechanics of |
| large-scale distributed data processing. The same Beam abstractions work with |
| both batch and streaming data sources. When you create your Beam pipeline, you |
| can think about your data processing task in terms of these abstractions. They |
| include:</p> |
| <ul> |
| <li> |
| <p><code>Pipeline</code>: A <code>Pipeline</code> encapsulates your entire data processing task, from |
| start to finish. This includes reading input data, transforming that data, and |
| writing output data. All Beam driver programs must create a <code>Pipeline</code>. When |
| you create the <code>Pipeline</code>, you must also specify the execution options that |
| tell the <code>Pipeline</code> where and how to run.</p> |
| </li> |
| <li> |
| <p><code>PCollection</code>: A <code>PCollection</code> represents a distributed data set that your |
| Beam pipeline operates on. The data set can be <em>bounded</em>, meaning it comes |
| from a fixed source like a file, or <em>unbounded</em>, meaning it comes from a |
| continuously updating source via a subscription or other mechanism. Your |
| pipeline typically creates an initial <code>PCollection</code> by reading data from an |
| external data source, but you can also create a <code>PCollection</code> from in-memory |
| data within your driver program. From there, <code>PCollection</code>s are the inputs and |
| outputs for each step in your pipeline.</p> |
| </li> |
| <li> |
| <p><code>PTransform</code>: A <code>PTransform</code> represents a data processing operation, or a step, |
| in your pipeline. Every <code>PTransform</code> takes one or more <code>PCollection</code> objects as |
| input, performs a processing function that you provide on the elements of that |
| <code>PCollection</code>, and produces zero or more output <code>PCollection</code> objects.</p> |
| </li> |
| </ul> |
| <span class="language-go"> |
| <ul> |
| <li><code>Scope</code>: The Go SDK has an explicit scope variable used to build a <code>Pipeline</code>. |
| A <code>Pipeline</code> can return it&rsquo;s root scope with the <code>Root()</code> method. The scope |
| variable is passed to <code>PTransform</code> functions to place them in the <code>Pipeline</code> |
| that owns the <code>Scope</code>.</li> |
| </ul> |
| </span> |
| <ul> |
| <li>I/O transforms: Beam comes with a number of &ldquo;IOs&rdquo; - library <code>PTransform</code>s that |
| read or write data to various external storage systems.</li> |
| </ul> |
| <span class="language-yaml"> |
| Note that in Beam YAML, `PCollection`s are either implicit (e.g. when using |
| `chain`) or referenced by their producing `PTransform`. |
| </span> |
| <p>A typical Beam driver program works as follows:</p> |
| <ul> |
| <li><strong>Create</strong> a <code>Pipeline</code> object and set the pipeline execution options, including |
| the Pipeline Runner.</li> |
| <li>Create an initial <code>PCollection</code> for pipeline data, either using the IOs |
| to read data from an external storage system, or using a <code>Create</code> transform to |
| build a <code>PCollection</code> from in-memory data.</li> |
| <li><strong>Apply</strong> <code>PTransform</code>s to each <code>PCollection</code>. Transforms can change, filter, |
| group, analyze, or otherwise process the elements in a <code>PCollection</code>. A |
| transform creates a new output <code>PCollection</code> <em>without modifying the input |
| collection</em>. A typical pipeline applies subsequent transforms to each new |
| output <code>PCollection</code> in turn until processing is complete. However, note that |
| a pipeline does not have to be a single straight line of transforms applied |
| one after another: think of <code>PCollection</code>s as variables and <code>PTransform</code>s as |
| functions applied to these variables: the shape of the pipeline can be an |
| arbitrarily complex processing graph.</li> |
| <li>Use IOs to write the final, transformed <code>PCollection</code>(s) to an external source.</li> |
| <li><strong>Run</strong> the pipeline using the designated Pipeline Runner.</li> |
| </ul> |
| <p>When you run your Beam driver program, the Pipeline Runner that you designate |
| constructs a <strong>workflow graph</strong> of your pipeline based on the <code>PCollection</code> |
| objects you&rsquo;ve created and transforms that you&rsquo;ve applied. That graph is then |
| executed using the appropriate distributed processing back-end, becoming an |
| asynchronous &ldquo;job&rdquo; (or equivalent) on that back-end.</p> |
| <h2 id="creating-a-pipeline">2. Creating a pipeline</h2> |
| <p>The <code>Pipeline</code> abstraction encapsulates all the data and steps in your data |
| processing task. Your Beam driver program typically starts by constructing a |
| <span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.56.0/index.html?org/apache/beam/sdk/Pipeline.html">Pipeline</a></span> |
| <span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pipeline.py">Pipeline</a></span> |
| <span class="language-go"><a href="https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/pipeline.go#L62">Pipeline</a></span> |
| object, and then using that object as the basis for creating the pipeline&rsquo;s data |
| sets as <code>PCollection</code>s and its operations as <code>Transform</code>s.</p> |
| <p>To use Beam, your driver program must first create an instance of the Beam SDK |
| class <code>Pipeline</code> (typically in the <code>main()</code> function). When you create your |
| <code>Pipeline</code>, you&rsquo;ll also need to set some <strong>configuration options</strong>. You can set |
| your pipeline&rsquo;s configuration options programmatically, but it&rsquo;s often easier to |
| set the options ahead of time (or read them from the command line) and pass them |
| to the <code>Pipeline</code> object when you create the object.</p> |
| <span class="language-typescript"> |
| A Pipeline in the Typescript API is simply a function that will be called |
| with a single `root` object and is passed to a Runner's `run` method. |
| </span> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Start by defining the options for the pipeline. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PipelineOptions</span> <span class="n">options</span> <span class="o">=</span> <span class="n">PipelineOptionsFactory</span><span class="o">.</span><span class="na">create</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Then create the pipeline. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">Pipeline</span> <span class="n">p</span> <span class="o">=</span> <span class="n">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="k">as</span> <span class="nn">beam</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">pass</span> <span class="c1"># build your pipeline here</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// beam.Init() is an initialization hook that must be called |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// near the beginning of main(), before creating a pipeline. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">beam</span><span class="p">.</span><span class="nf">Init</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Create the Pipeline object and root scope. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">pipeline</span><span class="p">,</span> <span class="nx">scope</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">NewPipelineWithRoot</span><span class="p">()</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="k">await</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">createRunner</span><span class="p">().</span><span class="nx">run</span><span class="p">(</span><span class="kd">function</span> <span class="nx">pipeline</span><span class="p">(</span><span class="nx">root</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Use root to build a pipeline. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="p">});</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">pipeline</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="l">...</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">options</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="l">...</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>For a more in-depth tutorial on creating basic pipelines |
| in the Python SDK, please read and work through |
| <a href="https://colab.sandbox.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/learn_beam_basics_by_doing.ipynb">this colab notebook</a>.</p> |
| <h3 id="configuring-pipeline-options">2.1. Configuring pipeline options</h3> |
| <p>Use the pipeline options to configure different aspects of your pipeline, such |
| as the pipeline runner that will execute your pipeline and any runner-specific |
| configuration required by the chosen runner. Your pipeline options will |
| potentially include information such as your project ID or a location for |
| storing files.</p> |
| <p class="language-java">When you run the pipeline on a runner of your choice, a copy of the |
| PipelineOptions will be available to your code. For example, if you add a PipelineOptions parameter |
| to a DoFn&rsquo;s <code>@ProcessElement</code> method, it will be populated by the system.</p> |
| <h4 id="pipeline-options-cli">2.1.1. Setting PipelineOptions from command-line arguments</h4> |
| <p class="language-java language-py">While you can configure your pipeline by creating a <code>PipelineOptions</code> object and |
| setting the fields directly, the Beam SDKs include a command-line parser that |
| you can use to set fields in <code>PipelineOptions</code> using command-line arguments.</p> |
| <p class="language-java language-py">To read options from the command-line, construct your <code>PipelineOptions</code> object |
| as demonstrated in the following example code:</p> |
| <p class="language-go">Use Go flags to parse command line arguments to configure your pipeline. Flags must be parsed |
| before <code>beam.Init()</code> is called.</p> |
| <p class="language-typescript">Any Javascript object can be used as pipeline options. |
| One can either construct one manually, but it is also common to pass an object |
| created from command line options such as <code>yargs.argv</code>.</p> |
| <p class="language-yaml">Pipeline options are simply an optional YAML mapping property that is a sibling to |
| the pipeline definition itself. |
| It will be merged with whatever options are passed on the command line.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PipelineOptions</span> <span class="n">options</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">PipelineOptionsFactory</span><span class="o">.</span><span class="na">fromArgs</span><span class="o">(</span><span class="n">args</span><span class="o">).</span><span class="na">withValidation</span><span class="o">().</span><span class="na">create</span><span class="o">();</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.options.pipeline_options</span> <span class="kn">import</span> <span class="n">PipelineOptions</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">beam_options</span> <span class="o">=</span> <span class="n">PipelineOptions</span><span class="p">()</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// If beamx or Go flags are used, flags must be parsed first, |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// before beam.Init() is called. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">flag</span><span class="p">.</span><span class="nf">Parse</span><span class="p">()</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">pipeline_options</span> <span class="o">=</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">runner</span><span class="o">:</span> <span class="s2">&#34;default&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">project</span><span class="o">:</span> <span class="s2">&#34;my_project&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"><span class="p">};</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">runner</span> <span class="o">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">createRunner</span><span class="p">(</span><span class="nx">pipeline_options</span><span class="p">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">runnerFromCommandLineOptions</span> <span class="o">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">createRunner</span><span class="p">(</span><span class="nx">yargs</span><span class="p">.</span><span class="nx">argv</span><span class="p">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">pipeline</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="l">...</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">options</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">my_pipeline_option</span><span class="p">:</span><span class="w"> </span><span class="l">my_value</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="l">...</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>This interprets command-line arguments that follow the format:</p> |
| <pre tabindex="0"><code>--&lt;option&gt;=&lt;value&gt; |
| </code></pre><span class="language-java"> |
| <blockquote> |
| <p>Appending the method <code>.withValidation</code> will check for required |
| command-line arguments and validate argument values.</p> |
| </blockquote> |
| </span> |
| <p class="language-java language-py">Building your <code>PipelineOptions</code> this way lets you specify any of the options as |
| a command-line argument.</p> |
| <p class="language-go">Defining flag variables this way lets you specify any of the options as a command-line argument.</p> |
| <blockquote> |
| <p><strong>Note:</strong> The <a href="/get-started/wordcount-example">WordCount example pipeline</a> |
| demonstrates how to set pipeline options at runtime by using command-line |
| options.</p> |
| </blockquote> |
| <h4 id="creating-custom-options">2.1.2. Creating custom options</h4> |
| <p>You can add your own custom options in addition to the standard |
| <code>PipelineOptions</code>. |
| <p class="language-java">To add your own options, define an interface with getter and |
| setter methods for each option.</p> |
| The following example shows how to add <code>input</code> and <code>output</code> custom options:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">MyOptions</span> <span class="kd">extends</span> <span class="n">PipelineOptions</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">String</span> <span class="nf">getInput</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="kt">void</span> <span class="nf">setInput</span><span class="o">(</span><span class="n">String</span> <span class="n">input</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">String</span> <span class="nf">getOutput</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="kt">void</span> <span class="nf">setOutput</span><span class="o">(</span><span class="n">String</span> <span class="n">output</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.options.pipeline_options</span> <span class="kn">import</span> <span class="n">PipelineOptions</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">MyOptions</span><span class="p">(</span><span class="n">PipelineOptions</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@classmethod</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">_add_argparse_args</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">parser</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--input&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--output&#39;</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// Use standard Go flags to define pipeline options. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">var</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">input</span> <span class="p">=</span> <span class="nx">flag</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="s">&#34;input&#34;</span><span class="p">,</span> <span class="s">&#34;&#34;</span><span class="p">,</span> <span class="s">&#34;&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">output</span> <span class="p">=</span> <span class="nx">flag</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="s">&#34;output&#34;</span><span class="p">,</span> <span class="s">&#34;&#34;</span><span class="p">,</span> <span class="s">&#34;&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">options</span> <span class="o">=</span> <span class="nx">yargs</span><span class="p">.</span><span class="nx">argv</span><span class="p">;</span> <span class="c1">// Or an alternative command-line parsing library. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Use options.input and options.output during pipeline construction. |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>You can also specify a description, which appears when a user passes <code>--help</code> as |
| a command-line argument, and a default value.</p> |
| <p class="language-java language-py language-go">You set the description and default value using annotations, as follows:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">MyOptions</span> <span class="kd">extends</span> <span class="n">PipelineOptions</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Description</span><span class="o">(</span><span class="s">&#34;Input for the pipeline&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Default.String</span><span class="o">(</span><span class="s">&#34;gs://my-bucket/input&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">String</span> <span class="nf">getInput</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="kt">void</span> <span class="nf">setInput</span><span class="o">(</span><span class="n">String</span> <span class="n">input</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Description</span><span class="o">(</span><span class="s">&#34;Output for the pipeline&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Default.String</span><span class="o">(</span><span class="s">&#34;gs://my-bucket/output&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">String</span> <span class="nf">getOutput</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="kt">void</span> <span class="nf">setOutput</span><span class="o">(</span><span class="n">String</span> <span class="n">output</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.options.pipeline_options</span> <span class="kn">import</span> <span class="n">PipelineOptions</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">MyOptions</span><span class="p">(</span><span class="n">PipelineOptions</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@classmethod</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">_add_argparse_args</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">parser</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;--input&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">default</span><span class="o">=</span><span class="s1">&#39;gs://dataflow-samples/shakespeare/kinglear.txt&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;The file path for the input text to process.&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;--output&#39;</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;The path prefix for output files.&#39;</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">var</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">input</span> <span class="p">=</span> <span class="nx">flag</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="s">&#34;input&#34;</span><span class="p">,</span> <span class="s">&#34;gs://my-bucket/input&#34;</span><span class="p">,</span> <span class="s">&#34;Input for the pipeline&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">output</span> <span class="p">=</span> <span class="nx">flag</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="s">&#34;output&#34;</span><span class="p">,</span> <span class="s">&#34;gs://my-bucket/output&#34;</span><span class="p">,</span> <span class="s">&#34;Output for the pipeline&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-py">For Python, you can also simply parse your custom options with argparse; there |
| is no need to create a separate PipelineOptions subclass.</p> |
| <p class="language-java">It&rsquo;s recommended that you register your interface with <code>PipelineOptionsFactory</code> |
| and then pass the interface when creating the <code>PipelineOptions</code> object. When you |
| register your interface with <code>PipelineOptionsFactory</code>, the <code>--help</code> can find |
| your custom options interface and add it to the output of the <code>--help</code> command. |
| <code>PipelineOptionsFactory</code> will also validate that your custom options are |
| compatible with all other registered options.</p> |
| <p class="language-java">The following example code shows how to register your custom options interface |
| with <code>PipelineOptionsFactory</code>:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PipelineOptionsFactory</span><span class="o">.</span><span class="na">register</span><span class="o">(</span><span class="n">MyOptions</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">MyOptions</span> <span class="n">options</span> <span class="o">=</span> <span class="n">PipelineOptionsFactory</span><span class="o">.</span><span class="na">fromArgs</span><span class="o">(</span><span class="n">args</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValidation</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">as</span><span class="o">(</span><span class="n">MyOptions</span><span class="o">.</span><span class="na">class</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Now your pipeline can accept <code>--input=value</code> and <code>--output=value</code> as command-line arguments.</p> |
| <h2 id="pcollections">3. PCollections</h2> |
| <p>The <span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.56.0/index.html?org/apache/beam/sdk/values/PCollection.html">PCollection</a></span> |
| <span class="language-py"><code>PCollection</code></span> |
| <span class="language-go"><a href="https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/pcollection.go#L39">PCollection</a></span> |
| abstraction represents a |
| potentially distributed, multi-element data set. You can think of a |
| <code>PCollection</code> as &ldquo;pipeline&rdquo; data; Beam transforms use <code>PCollection</code> objects as |
| inputs and outputs. As such, if you want to work with data in your pipeline, it |
| must be in the form of a <code>PCollection</code>.</p> |
| <p>After you&rsquo;ve created your <code>Pipeline</code>, you&rsquo;ll need to begin by creating at least |
| one <code>PCollection</code> in some form. The <code>PCollection</code> you create serves as the input |
| for the first operation in your pipeline.</p> |
| <span class="language-yaml"> |
| In Beam YAML, `PCollection`s are either implicit (e.g. when using `chain`) |
| or referred to by their producing `PTransform`. |
| </span> |
| <h3 id="creating-a-pcollection">3.1. Creating a PCollection</h3> |
| <p>You create a <code>PCollection</code> by either reading data from an external source using |
| Beam&rsquo;s <a href="#pipeline-io">Source API</a>, or you can create a <code>PCollection</code> of data |
| stored in an in-memory collection class in your driver program. The former is |
| typically how a production pipeline would ingest data; Beam&rsquo;s Source APIs |
| contain adapters to help you read from external sources like large cloud-based |
| files, databases, or subscription services. The latter is primarily useful for |
| testing and debugging purposes.</p> |
| <h4 id="reading-external-source">3.1.1. Reading from an external source</h4> |
| <p>To read from an external source, you use one of the <a href="#pipeline-io">Beam-provided I/O |
| adapters</a>. The adapters vary in their exact usage, but all of them |
| read from some external data source and return a <code>PCollection</code> whose elements |
| represent the data records in that source.</p> |
| <p>Each data source adapter has a <code>Read</code> transform; to read, |
| <span class="language-java language-py language-go language-typescript"> |
| you must apply that transform to the <code>Pipeline</code> object itself. |
| </span> |
| <span class="language-yaml"> |
| place this transform in the <code>source</code> or <code>transforms</code> portion of the pipeline. |
| </span> |
| <span class="language-java"><code>TextIO.Read</code></span> |
| <span class="language-py"><code>io.TextFileSource</code></span> |
| <span class="language-go"><code>textio.Read</code></span> |
| <span class="language-typescript"><code>textio.ReadFromText</code></span>, |
| <span class="language-yaml"><code>ReadFromText</code></span>, |
| for example, reads from an |
| external text file and returns a <code>PCollection</code> whose elements |
| <span class="language-java language-py language-go language-typescript"> |
| are of type <code>String</code> where each <code>String</code> |
| </span> |
| represents one line from the text file. Here&rsquo;s how you |
| would apply <span class="language-java"><code>TextIO.Read</code></span> |
| <span class="language-py"><code>io.TextFileSource</code></span> |
| <span class="language-go"><code>textio.Read</code></span> |
| <span class="language-typescript"><code>textio.ReadFromText</code></span> |
| <span class="language-yaml"><code>ReadFromText</code></span> |
| to your <code>Pipeline</code> <span class="language-typescript">root</span> to create |
| a <code>PCollection</code>:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Create the pipeline. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">PipelineOptions</span> <span class="n">options</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">PipelineOptionsFactory</span><span class="o">.</span><span class="na">fromArgs</span><span class="o">(</span><span class="n">args</span><span class="o">).</span><span class="na">create</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Pipeline</span> <span class="n">p</span> <span class="o">=</span> <span class="n">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Create the PCollection &#39;lines&#39; by applying a &#39;Read&#39; transform. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">lines</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;ReadMyFile&#34;</span><span class="o">,</span> <span class="n">TextIO</span><span class="o">.</span><span class="na">read</span><span class="o">().</span><span class="na">from</span><span class="o">(</span><span class="s">&#34;gs://some/inputData.txt&#34;</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">lines</span> <span class="o">=</span> <span class="n">pipeline</span> <span class="o">|</span> <span class="s1">&#39;ReadMyFile&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ReadFromText</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;gs://some/inputData.txt&#39;</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// Read the file at the URI &#39;gs://some/inputData.txt&#39; and return |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// the lines as a PCollection&lt;string&gt;. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// Notice the scope as the first variable when calling |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// the method as is needed when calling all transforms. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">lines</span> <span class="o">:=</span> <span class="nx">textio</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="s">&#34;gs://some/inputData.txt&#34;</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">async</span> <span class="kd">function</span> <span class="nx">pipeline</span><span class="p">(</span><span class="nx">root</span>: <span class="kt">beam.Root</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Note that textio.ReadFromText is an AsyncPTransform. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kr">const</span> <span class="nx">pcoll</span>: <span class="kt">PCollection</span><span class="p">&lt;</span><span class="nt">string</span><span class="p">&gt;</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">root</span><span class="p">.</span><span class="nx">applyAsync</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">textio</span><span class="p">.</span><span class="nx">ReadFromText</span><span class="p">(</span><span class="s2">&#34;path/to/text_pattern&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">);</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">pipeline</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">source</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">ReadFromText</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">...</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>See the <a href="#pipeline-io">section on I/O</a> to learn more about how to read from the |
| various data sources supported by the Beam SDK.</p> |
| <h4 id="creating-pcollection-in-memory">3.1.2. Creating a PCollection from in-memory data</h4> |
| <p class="language-java">To create a <code>PCollection</code> from an in-memory Java <code>Collection</code>, you use the |
| Beam-provided <code>Create</code> transform. Much like a data adapter&rsquo;s <code>Read</code>, you apply |
| <code>Create</code> directly to your <code>Pipeline</code> object itself.</p> |
| <p class="language-java">As parameters, <code>Create</code> accepts the Java <code>Collection</code> and a <code>Coder</code> object. The |
| <code>Coder</code> specifies how the elements in the <code>Collection</code> should be |
| <a href="#element-type">encoded</a>.</p> |
| <p class="language-py">To create a <code>PCollection</code> from an in-memory <code>list</code>, you use the Beam-provided |
| <code>Create</code> transform. Apply this transform directly to your <code>Pipeline</code> object |
| itself.</p> |
| <p class="language-go">To create a <code>PCollection</code> from an in-memory <code>slice</code>, you use the Beam-provided |
| <code>beam.CreateList</code> transform. Pass the pipeline <code>scope</code>, and the <code>slice</code> to this transform.</p> |
| <p class="language-typescript">To create a <code>PCollection</code> from an in-memory <code>array</code>, you use the Beam-provided |
| <code>Create</code> transform. Apply this transform directly to your <code>Root</code> object.</p> |
| <p class="language-yaml">To create a <code>PCollection</code> from an in-memory <code>array</code>, you use the Beam-provided |
| <code>Create</code> transform. Specify the elements in the pipeline itself.</p> |
| <p>The following example code shows how to create a <code>PCollection</code> from an in-memory |
| <span class="language-java"><code>List</code></span> |
| <span class="language-py"><code>list</code></span> |
| <span class="language-go"><code>slice</code></span> |
| <span class="language-typescript language-yaml"><code>array</code></span>:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Create a Java Collection, in this case a List of Strings. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kd">final</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">LINES</span> <span class="o">=</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;To be, or not to be: that is the question: &#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;Whether &#39;tis nobler in the mind to suffer &#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;The slings and arrows of outrageous fortune, &#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;Or to take arms against a sea of troubles, &#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Create the pipeline. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">PipelineOptions</span> <span class="n">options</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">PipelineOptionsFactory</span><span class="o">.</span><span class="na">fromArgs</span><span class="o">(</span><span class="n">args</span><span class="o">).</span><span class="na">create</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Pipeline</span> <span class="n">p</span> <span class="o">=</span> <span class="n">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Apply Create, passing the list and the coder, to create the PCollection. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Create</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">LINES</span><span class="o">)).</span><span class="na">setCoder</span><span class="o">(</span><span class="n">StringUtf8Coder</span><span class="o">.</span><span class="na">of</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="k">as</span> <span class="nn">beam</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">lines</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;To be, or not to be: that is the question: &#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;Whether &#39;tis nobler in the mind to suffer &#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;The slings and arrows of outrageous fortune, &#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;Or to take arms against a sea of troubles, &#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">]))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">lines</span> <span class="o">:=</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;To be, or not to be: that is the question: &#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;Whether &#39;tis nobler in the mind to suffer &#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;The slings and arrows of outrageous fortune, &#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;Or to take arms against a sea of troubles, &#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Create the Pipeline object and root scope. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// It&#39;s conventional to use p as the Pipeline variable and |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// s as the scope variable. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">p</span><span class="p">,</span> <span class="nx">s</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">NewPipelineWithRoot</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Pass the slice to beam.CreateList, to create the pcollection. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// The scope variable s is used to add the CreateList transform |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// to the pipeline. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">linesPCol</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">CreateList</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">lines</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kd">function</span> <span class="nx">pipeline</span><span class="p">(</span><span class="nx">root</span>: <span class="kt">beam.Root</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kr">const</span> <span class="nx">pcoll</span> <span class="o">=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">([</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;To be, or not to be: that is the question: &#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;Whether &#39;tis nobler in the mind to suffer &#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;The slings and arrows of outrageous fortune, &#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;Or to take arms against a sea of troubles, &#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">])</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">);</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">pipeline</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">transforms</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">Create</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">elements</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">A</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">B</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">...</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="pcollection-characteristics">3.2. PCollection characteristics</h3> |
| <p>A <code>PCollection</code> is owned by the specific <code>Pipeline</code> object for which it is |
| created; multiple pipelines cannot share a <code>PCollection</code>. |
| <span class="language-java">In some respects, a <code>PCollection</code> functions like |
| a <code>Collection</code> class. However, a <code>PCollection</code> can differ in a few key ways:</span></p> |
| <h4 id="element-type">3.2.1. Element type</h4> |
| <p>The elements of a <code>PCollection</code> may be of any type, but must all be of the same |
| type. However, to support distributed processing, Beam needs to be able to |
| encode each individual element as a byte string (so elements can be passed |
| around to distributed workers). The Beam SDKs provide a data encoding mechanism |
| that includes built-in encoding for commonly-used types as well as support for |
| specifying custom encodings as needed.</p> |
| <h4 id="element-schema">3.2.2. Element schema</h4> |
| <p>In many cases, the element type in a <code>PCollection</code> has a structure that can be introspected. |
| Examples are JSON, Protocol Buffer, Avro, and database records. Schemas provide a way to |
| express types as a set of named fields, allowing for more-expressive aggregations.</p> |
| <h4 id="immutability">3.2.3. Immutability</h4> |
| <p>A <code>PCollection</code> is immutable. Once created, you cannot add, remove, or change |
| individual elements. A Beam Transform might process each element of a |
| <code>PCollection</code> and generate new pipeline data (as a new <code>PCollection</code>), <em>but it |
| does not consume or modify the original input collection</em>.</p> |
| <blockquote> |
| <p><strong>Note:</strong> Beam SDKs avoid unnecessary copying of elements, so <code>PCollection</code> |
| contents are logically immutable, not physically immutable. Changes to input |
| elements may be visible to other DoFns executing within the same bundle, and may |
| cause correctness issues. |
| As a rule, it&rsquo;s not safe to modify values provided to a DoFn.</p> |
| </blockquote> |
| <h4 id="random-access">3.2.4. Random access</h4> |
| <p>A <code>PCollection</code> does not support random access to individual elements. Instead, |
| Beam Transforms consider every element in a <code>PCollection</code> individually.</p> |
| <h4 id="size-and-boundedness">3.2.5. Size and boundedness</h4> |
| <p>A <code>PCollection</code> is a large, immutable &ldquo;bag&rdquo; of elements. There is no upper limit |
| on how many elements a <code>PCollection</code> can contain; any given <code>PCollection</code> might |
| fit in memory on a single machine, or it might represent a very large |
| distributed data set backed by a persistent data store.</p> |
| <p>A <code>PCollection</code> can be either <strong>bounded</strong> or <strong>unbounded</strong> in size. A |
| <strong>bounded</strong> <code>PCollection</code> represents a data set of a known, fixed size, while an |
| <strong>unbounded</strong> <code>PCollection</code> represents a data set of unlimited size. Whether a |
| <code>PCollection</code> is bounded or unbounded depends on the source of the data set that |
| it represents. Reading from a batch data source, such as a file or a database, |
| creates a bounded <code>PCollection</code>. Reading from a streaming or |
| continuously-updating data source, such as Pub/Sub or Kafka, creates an unbounded |
| <code>PCollection</code> (unless you explicitly tell it not to).</p> |
| <p>The bounded (or unbounded) nature of your <code>PCollection</code> affects how Beam |
| processes your data. A bounded <code>PCollection</code> can be processed using a batch job, |
| which might read the entire data set once, and perform processing in a job of |
| finite length. An unbounded <code>PCollection</code> must be processed using a streaming |
| job that runs continuously, as the entire collection can never be available for |
| processing at any one time.</p> |
| <p>Beam uses <a href="#windowing">windowing</a> to divide a continuously updating unbounded |
| <code>PCollection</code> into logical windows of finite size. These logical windows are |
| determined by some characteristic associated with a data element, such as a |
| <strong>timestamp</strong>. Aggregation transforms (such as <code>GroupByKey</code> and <code>Combine</code>) work |
| on a per-window basis — as the data set is generated, they process each |
| <code>PCollection</code> as a succession of these finite windows.</p> |
| <h4 id="element-timestamps">3.2.6. Element timestamps</h4> |
| <p>Each element in a <code>PCollection</code> has an associated intrinsic <strong>timestamp</strong>. The |
| timestamp for each element is initially assigned by the <a href="#pipeline-io">Source</a> |
| that creates the <code>PCollection</code>. Sources that create an unbounded <code>PCollection</code> |
| often assign each new element a timestamp that corresponds to when the element |
| was read or added.</p> |
| <blockquote> |
| <p><strong>Note</strong>: Sources that create a bounded <code>PCollection</code> for a fixed data set |
| also automatically assign timestamps, but the most common behavior is to |
| assign every element the same timestamp (<code>Long.MIN_VALUE</code>).</p> |
| </blockquote> |
| <p>Timestamps are useful for a <code>PCollection</code> that contains elements with an |
| inherent notion of time. If your pipeline is reading a stream of events, like |
| Tweets or other social media messages, each element might use the time the event |
| was posted as the element timestamp.</p> |
| <p>You can manually assign timestamps to the elements of a <code>PCollection</code> if the |
| source doesn&rsquo;t do it for you. You&rsquo;ll want to do this if the elements have an |
| inherent timestamp, but the timestamp is somewhere in the structure of the |
| element itself (such as a &ldquo;time&rdquo; field in a server log entry). Beam has |
| <a href="#transforms">Transforms</a> that take a <code>PCollection</code> as input and output an |
| identical <code>PCollection</code> with timestamps attached; see <a href="#adding-timestamps-to-a-pcollections-elements">Adding |
| Timestamps</a> for more information |
| about how to do so.</p> |
| <h2 id="transforms">4. Transforms</h2> |
| <p>Transforms are the operations in your pipeline, and provide a generic |
| processing framework. You provide processing logic in the form of a function |
| object (colloquially referred to as &ldquo;user code&rdquo;), and your user code is applied |
| to each element of an input <code>PCollection</code> (or more than one <code>PCollection</code>). |
| Depending on the pipeline runner and back-end that you choose, many different |
| workers across a cluster may execute instances of your user code in parallel. |
| The user code running on each worker generates the output elements that are |
| ultimately added to the final output <code>PCollection</code> that the transform produces.</p> |
| <blockquote> |
| <p>Aggregation is an important concept to understand when learning about Beam&rsquo;s |
| transforms. For an introduction to aggregation, see the Basics of the Beam |
| model <a href="/documentation/basics/#aggregation">Aggregation section</a>.</p> |
| </blockquote> |
| <p>The Beam SDKs contain a number of different transforms that you can apply to |
| your pipeline&rsquo;s <code>PCollection</code>s. These include general-purpose core transforms, |
| such as <a href="#pardo">ParDo</a> or <a href="#combine">Combine</a>. There are also pre-written |
| <a href="#composite-transforms">composite transforms</a> included in the SDKs, which |
| combine one or more of the core transforms in a useful processing pattern, such |
| as counting or combining elements in a collection. You can also define your own |
| more complex composite transforms to fit your pipeline&rsquo;s exact use case.</p> |
| <p>For a more in-depth tutorial of applying various transforms |
| in the Python SDK, please read and work through |
| <a href="https://colab.sandbox.google.com/github/liferoad/beam/blob/learn-transforms/examples/notebooks/get-started/learn_beam_transforms_by_doing.ipynb">this colab notebook</a>.</p> |
| <h3 id="applying-transforms">4.1. Applying transforms</h3> |
| <p>To invoke a transform, you must <strong>apply</strong> it to the input <code>PCollection</code>. Each |
| transform in the Beam SDKs has a generic <code>apply</code> method |
| <span class="language-py">(or pipe operator <code>|</code>)</span>. |
| Invoking multiple Beam transforms is similar to <em>method chaining</em>, but with one |
| slight difference: You apply the transform to the input <code>PCollection</code>, passing |
| the transform itself as an argument, and the operation returns the output |
| <code>PCollection</code>. |
| <span class="language-yaml"><code>array</code></span> |
| In YAML, transforms are applied by listing their inputs. |
| </span> |
| This takes the general form:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="o">[</span><span class="n">Output</span> <span class="n">PCollection</span><span class="o">]</span> <span class="o">=</span> <span class="o">[</span><span class="n">Input</span> <span class="n">PCollection</span><span class="o">].</span><span class="na">apply</span><span class="o">([</span><span class="n">Transform</span><span class="o">])</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="p">[</span><span class="n">Output</span> <span class="n">PCollection</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="n">Input</span> <span class="n">PCollection</span><span class="p">]</span> <span class="o">|</span> <span class="p">[</span><span class="n">Transform</span><span class="p">]</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">Output</span> <span class="nx">PCollection</span><span class="p">]</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="p">[</span><span class="nx">Transform</span><span class="p">],</span> <span class="p">[</span><span class="nx">Input</span> <span class="nx">PCollection</span><span class="p">])</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">Output</span> <span class="nx">PCollection</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="nx">Input</span> <span class="nx">PCollection</span><span class="p">].</span><span class="nx">apply</span><span class="p">([</span><span class="nx">Transform</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"><span class="p">[</span><span class="nx">Output</span> <span class="nx">PCollection</span><span class="p">]</span> <span class="o">=</span> <span class="k">await</span> <span class="p">[</span><span class="nx">Input</span> <span class="nx">PCollection</span><span class="p">].</span><span class="nx">applyAsync</span><span class="p">([</span><span class="nx">AsyncTransform</span><span class="p">])</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">pipeline</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">transforms</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="l">...</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">ProducingTransform</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">ProducingTransformType</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="l">...</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">MyTransform</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">MyTransformType</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">input</span><span class="p">:</span><span class="w"> </span><span class="l">ProducingTransform</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="l">...</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-yaml">If a transform has more than one |
| (<a href="https://beam.apache.org/documentation/sdks/yaml-errors/">non-error</a>) output, |
| the various outputs can be identified by explicitly giving the output name.</p> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">pipeline</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">transforms</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="l">...</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">ProducingTransform</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">ProducingTransformType</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="l">...</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">MyTransform</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">MyTransformType</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">input</span><span class="p">:</span><span class="w"> </span><span class="l">ProducingTransform.output_name</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="l">...</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">MyTransform</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">MyTransformType</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">input</span><span class="p">:</span><span class="w"> </span><span class="l">ProducingTransform.another_output_name</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="l">...</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p> |
| <p class="language-yaml">For linear pipelines, this can be further simplified by implicitly determining |
| the inputs based on by the ordering of the transforms by designating and setting |
| the type to <code>chain</code>. For example</p> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">pipeline</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">chain</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">transforms</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">ProducingTransform</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">ReadTransform</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> </span><span class="l">...</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">MyTransform</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">MyTransformType</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> </span><span class="l">...</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">ConsumingTransform</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">WriteTransform</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> </span><span class="l">...</span></span></span></code></pre></div> |
| </div> |
| </div> |
| </p> |
| <p class="language-java language-py language-typescript">Because Beam uses a generic <code>apply</code> method for <code>PCollection</code>, you can both chain |
| transforms sequentially and also apply transforms that contain other transforms |
| nested within (called <a href="#composite-transforms">composite transforms</a> in the Beam |
| SDKs).</p> |
| <p class="language-go">It&rsquo;s recommended to create a new variable for each new <code>PCollection</code> to |
| sequentially transform input data. <code>Scope</code>s can be used to create functions |
| that contain other transforms |
| (called <a href="#composite-transforms">composite transforms</a> in the Beam SDKs).</p> |
| <p>How you apply your pipeline&rsquo;s transforms determines the structure of your |
| pipeline. The best way to think of your pipeline is as a directed acyclic graph, |
| where <code>PTransform</code> nodes are subroutines that accept <code>PCollection</code> nodes as |
| inputs and emit <code>PCollection</code> nodes as outputs. |
| <span class="language-java language-py"> |
| For example, you can chain together transforms to create a pipeline that successively modifies input data: |
| </span> |
| <span class="language-go"> |
| For example, you can successively call transforms on PCollections to modify the input data: |
| </span></p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="o">[</span><span class="n">Final</span> <span class="n">Output</span> <span class="n">PCollection</span><span class="o">]</span> <span class="o">=</span> <span class="o">[</span><span class="n">Initial</span> <span class="n">Input</span> <span class="n">PCollection</span><span class="o">].</span><span class="na">apply</span><span class="o">([</span><span class="n">First</span> <span class="n">Transform</span><span class="o">])</span> |
| </span></span><span class="line"><span class="cl"><span class="o">.</span><span class="na">apply</span><span class="o">([</span><span class="n">Second</span> <span class="n">Transform</span><span class="o">])</span> |
| </span></span><span class="line"><span class="cl"><span class="o">.</span><span class="na">apply</span><span class="o">([</span><span class="n">Third</span> <span class="n">Transform</span><span class="o">])</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="p">[</span><span class="n">Final</span> <span class="n">Output</span> <span class="n">PCollection</span><span class="p">]</span> <span class="o">=</span> <span class="p">([</span><span class="n">Initial</span> <span class="n">Input</span> <span class="n">PCollection</span><span class="p">]</span> <span class="o">|</span> <span class="p">[</span><span class="n">First</span> <span class="n">Transform</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="p">[</span><span class="n">Second</span> <span class="n">Transform</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="p">[</span><span class="n">Third</span> <span class="n">Transform</span><span class="p">])</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">Second</span> <span class="nx">PCollection</span><span class="p">]</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="p">[</span><span class="nx">First</span> <span class="nx">Transform</span><span class="p">],</span> <span class="p">[</span><span class="nx">Initial</span> <span class="nx">Input</span> <span class="nx">PCollection</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"><span class="p">[</span><span class="nx">Third</span> <span class="nx">PCollection</span><span class="p">]</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="p">[</span><span class="nx">Second</span> <span class="nx">Transform</span><span class="p">],</span> <span class="p">[</span><span class="nx">Second</span> <span class="nx">PCollection</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"><span class="p">[</span><span class="nx">Final</span> <span class="nx">Output</span> <span class="nx">PCollection</span><span class="p">]</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="p">[</span><span class="nx">Third</span> <span class="nx">Transform</span><span class="p">],</span> <span class="p">[</span><span class="nx">Third</span> <span class="nx">PCollection</span><span class="p">])</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">Final</span> <span class="nx">Output</span> <span class="nx">PCollection</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="nx">Initial</span> <span class="nx">Input</span> <span class="nx">PCollection</span><span class="p">].</span><span class="nx">apply</span><span class="p">([</span><span class="nx">First</span> <span class="nx">Transform</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"><span class="p">.</span><span class="nx">apply</span><span class="p">([</span><span class="nx">Second</span> <span class="nx">Transform</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"><span class="p">.</span><span class="nx">apply</span><span class="p">([</span><span class="nx">Third</span> <span class="nx">Transform</span><span class="p">])</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>The graph of this pipeline looks like the following:</p> |
| <p><img src="/images/design-your-pipeline-linear.svg" alt="This linear pipeline starts with one input collection, sequentially appliesthree transforms, and ends with one output collection."></p> |
| <p><em>Figure 1: A linear pipeline with three sequential transforms.</em></p> |
| <p>However, note that a transform <em>does not consume or otherwise alter</em> the input |
| collection — remember that a <code>PCollection</code> is immutable by definition. This means |
| that you can apply multiple transforms to the same input <code>PCollection</code> to create |
| a branching pipeline, like so:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="o">[</span><span class="n">PCollection</span> <span class="n">of</span> <span class="n">database</span> <span class="n">table</span> <span class="n">rows</span><span class="o">]</span> <span class="o">=</span> <span class="o">[</span><span class="n">Database</span> <span class="n">Table</span> <span class="n">Reader</span><span class="o">].</span><span class="na">apply</span><span class="o">([</span><span class="n">Read</span> <span class="n">Transform</span><span class="o">])</span> |
| </span></span><span class="line"><span class="cl"><span class="o">[</span><span class="n">PCollection</span> <span class="n">of</span> <span class="sc">&#39;A&#39;</span> <span class="n">names</span><span class="o">]</span> <span class="o">=</span> <span class="o">[</span><span class="n">PCollection</span> <span class="n">of</span> <span class="n">database</span> <span class="n">table</span> <span class="n">rows</span><span class="o">].</span><span class="na">apply</span><span class="o">([</span><span class="n">Transform</span> <span class="n">A</span><span class="o">])</span> |
| </span></span><span class="line"><span class="cl"><span class="o">[</span><span class="n">PCollection</span> <span class="n">of</span> <span class="sc">&#39;B&#39;</span> <span class="n">names</span><span class="o">]</span> <span class="o">=</span> <span class="o">[</span><span class="n">PCollection</span> <span class="n">of</span> <span class="n">database</span> <span class="n">table</span> <span class="n">rows</span><span class="o">].</span><span class="na">apply</span><span class="o">([</span><span class="n">Transform</span> <span class="n">B</span><span class="o">])</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="p">[</span><span class="n">PCollection</span> <span class="n">of</span> <span class="n">database</span> <span class="n">table</span> <span class="n">rows</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="n">Database</span> <span class="n">Table</span> <span class="n">Reader</span><span class="p">]</span> <span class="o">|</span> <span class="p">[</span><span class="n">Read</span> <span class="n">Transform</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"><span class="p">[</span><span class="n">PCollection</span> <span class="n">of</span> <span class="s1">&#39;A&#39;</span> <span class="n">names</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="n">PCollection</span> <span class="n">of</span> <span class="n">database</span> <span class="n">table</span> <span class="n">rows</span><span class="p">]</span> <span class="o">|</span> <span class="p">[</span><span class="n">Transform</span> <span class="n">A</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"><span class="p">[</span><span class="n">PCollection</span> <span class="n">of</span> <span class="s1">&#39;B&#39;</span> <span class="n">names</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="n">PCollection</span> <span class="n">of</span> <span class="n">database</span> <span class="n">table</span> <span class="n">rows</span><span class="p">]</span> <span class="o">|</span> <span class="p">[</span><span class="n">Transform</span> <span class="n">B</span><span class="p">]</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">PCollection</span> <span class="nx">of</span> <span class="nx">database</span> <span class="nx">table</span> <span class="nx">rows</span><span class="p">]</span> <span class="p">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="p">[</span><span class="nx">Read</span> <span class="nx">Transform</span><span class="p">],</span> <span class="p">[</span><span class="nx">Database</span> <span class="nx">Table</span> <span class="nx">Reader</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"><span class="p">[</span><span class="nx">PCollection</span> <span class="nx">of</span> <span class="sc">&#39;A&#39;</span> <span class="nx">names</span><span class="p">]</span> <span class="p">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="p">[</span><span class="nx">Transform</span> <span class="nx">A</span><span class="p">],</span> <span class="p">[</span><span class="nx">PCollection</span> <span class="nx">of</span> <span class="nx">database</span> <span class="nx">table</span> <span class="nx">rows</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"><span class="p">[</span><span class="nx">PCollection</span> <span class="nx">of</span> <span class="sc">&#39;B&#39;</span> <span class="nx">names</span><span class="p">]</span> <span class="p">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="p">[</span><span class="nx">Transform</span> <span class="nx">B</span><span class="p">],</span> <span class="p">[</span><span class="nx">PCollection</span> <span class="nx">of</span> <span class="nx">database</span> <span class="nx">table</span> <span class="nx">rows</span><span class="p">])</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">PCollection</span> <span class="k">of</span> <span class="nx">database</span> <span class="nx">table</span> <span class="nx">rows</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="nx">Database</span> <span class="nx">Table</span> <span class="nx">Reader</span><span class="p">].</span><span class="nx">apply</span><span class="p">([</span><span class="nx">Read</span> <span class="nx">Transform</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"><span class="p">[</span><span class="nx">PCollection</span> <span class="k">of</span> <span class="s1">&#39;A&#39;</span> <span class="nx">names</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="nx">PCollection</span> <span class="k">of</span> <span class="nx">database</span> <span class="nx">table</span> <span class="nx">rows</span><span class="p">].</span><span class="nx">apply</span><span class="p">([</span><span class="nx">Transform</span> <span class="nx">A</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"><span class="p">[</span><span class="nx">PCollection</span> <span class="k">of</span> <span class="s1">&#39;B&#39;</span> <span class="nx">names</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="nx">PCollection</span> <span class="k">of</span> <span class="nx">database</span> <span class="nx">table</span> <span class="nx">rows</span><span class="p">].</span><span class="nx">apply</span><span class="p">([</span><span class="nx">Transform</span> <span class="nx">B</span><span class="p">])</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>The graph of this branching pipeline looks like the following:</p> |
| <p><img src="/images/design-your-pipeline-multiple-pcollections.svg" alt="This pipeline applies two transforms to a single input collection. Eachtransform produces an output collection."></p> |
| <p><em>Figure 2: A branching pipeline. Two transforms are applied to a single |
| PCollection of database table rows.</em></p> |
| <p>You can also build your own <a href="#composite-transforms">composite transforms</a> that |
| nest multiple transforms inside a single, larger transform. Composite transforms |
| are particularly useful for building a reusable sequence of simple steps that |
| get used in a lot of different places.</p> |
| <p class="language-python">The pipe syntax allows one to apply PTransforms to <code>tuple</code>s and <code>dict</code>s of |
| PCollections as well for those transforms accepting multiple inputs (such as |
| <code>Flatten</code> and <code>CoGroupByKey</code>).</p> |
| <p class="language-typescript">PTransforms can also be applied to any <code>PValue</code>, which include the Root object, |
| PCollections, arrays of <code>PValue</code>s, and objects with <code>PValue</code> values. |
| One can apply transforms to these composite types by wrapping them with |
| <code>beam.P</code>, e.g. |
| <code>beam.P({left: pcollA, right: pcollB}).apply(transformExpectingTwoPCollections)</code>.</p> |
| <p class="language-typescript">PTransforms come in two flavors, synchronous and asynchronous, depending on |
| whether their <em>application</em>* involves asynchronous invocations. |
| An <code>AsyncTransform</code> must be applied with <code>applyAsync</code> and returns a <code>Promise</code> |
| which must be awaited before further pipeline construction.</p> |
| <h3 id="core-beam-transforms">4.2. Core Beam transforms</h3> |
| <p>Beam provides the following core transforms, each of which represents a different |
| processing paradigm:</p> |
| <ul> |
| <li><code>ParDo</code></li> |
| <li><code>GroupByKey</code></li> |
| <li><code>CoGroupByKey</code></li> |
| <li><code>Combine</code></li> |
| <li><code>Flatten</code></li> |
| <li><code>Partition</code></li> |
| </ul> |
| <p class="language-typescript">The Typescript SDK provides some of the most basic of these transforms |
| as methods on <code>PCollection</code> itself.</p> |
| <h4 id="pardo">4.2.1. ParDo</h4> |
| <p><code>ParDo</code> is a Beam transform for generic parallel processing. The <code>ParDo</code> |
| processing paradigm is similar to the &ldquo;Map&rdquo; phase of a |
| <a href="https://en.wikipedia.org/wiki/MapReduce">Map/Shuffle/Reduce</a>-style |
| algorithm: a <code>ParDo</code> transform considers each element in the input |
| <code>PCollection</code>, performs some processing function (your user code) on that |
| element, and emits zero, one, or multiple elements to an output <code>PCollection</code>.</p> |
| <p><code>ParDo</code> is useful for a variety of common data processing operations, including:</p> |
| <ul> |
| <li><strong>Filtering a data set.</strong> You can use <code>ParDo</code> to consider each element in a |
| <code>PCollection</code> and either output that element to a new collection or discard |
| it.</li> |
| <li><strong>Formatting or type-converting each element in a data set.</strong> If your input |
| <code>PCollection</code> contains elements that are of a different type or format than |
| you want, you can use <code>ParDo</code> to perform a conversion on each element and |
| output the result to a new <code>PCollection</code>.</li> |
| <li><strong>Extracting parts of each element in a data set.</strong> If you have a |
| <code>PCollection</code> of records with multiple fields, for example, you can use a |
| <code>ParDo</code> to parse out just the fields you want to consider into a new |
| <code>PCollection</code>.</li> |
| <li><strong>Performing computations on each element in a data set.</strong> You can use <code>ParDo</code> |
| to perform simple or complex computations on every element, or certain |
| elements, of a <code>PCollection</code> and output the results as a new <code>PCollection</code>.</li> |
| </ul> |
| <p>In such roles, <code>ParDo</code> is a common intermediate step in a pipeline. You might |
| use it to extract certain fields from a set of raw input records, or convert raw |
| input into a different format; you might also use <code>ParDo</code> to convert processed |
| data into a format suitable for output, like database table rows or printable |
| strings.</p> |
| <p class="language-java language-go language-py language-typescript">When you apply a <code>ParDo</code> transform, you&rsquo;ll need to provide user code in the form |
| of a <code>DoFn</code> object. <code>DoFn</code> is a Beam SDK class that defines a distributed |
| processing function.</p> |
| <p class="language-yaml">In Beam YAML, <code>ParDo</code> operations are expressed by the <code>MapToFields</code>, <code>Filter</code>, |
| and <code>Explode</code> transform types. These types can take a UDF in the language of your |
| choice, rather than introducing the notion of a <code>DoFn</code>. |
| See <a href="https://beam.apache.org/documentation/sdks/yaml-udf/">the page on mapping fns</a> for more details.</p> |
| <span class="language-java language-py"> |
| <blockquote> |
| <p>When you create a subclass of <code>DoFn</code>, note that your subclass should adhere to |
| the <a href="#requirements-for-writing-user-code-for-beam-transforms">Requirements for writing user code for Beam transforms</a>.</p> |
| </blockquote> |
| </span> |
| <p class="language-go">All DoFns should be registered using a generic <code>register.DoFnXxY[...]</code> |
| function. This allows the Go SDK to infer an encoding from any inputs/outputs, |
| registers the DoFn for execution on remote runners, and optimizes the runtime |
| execution of the DoFns via reflection.</p> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// ComputeWordLengthFn is a DoFn that computes the word length of string elements. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">type</span> <span class="nx">ComputeWordLengthFn</span> <span class="kd">struct</span><span class="p">{}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// ProcessElement computes the length of word and emits the result. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// When creating structs as a DoFn, the ProcessElement method performs the |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// work of this step in the pipeline. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">ComputeWordLengthFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">word</span> <span class="kt">string</span><span class="p">)</span> <span class="kt">int</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// 2 inputs and 1 output =&gt; DoFn2x1 |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// Input/output types are included in order in the brackets |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">register</span><span class="p">.</span><span class="nx">DoFn2x1</span><span class="p">[</span><span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="kt">string</span><span class="p">,</span> <span class="kt">int</span><span class="p">](</span><span class="o">&amp;</span><span class="nx">ComputeWordLengthFn</span><span class="p">{})</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h5 id="applying-pardo">4.2.1.1. Applying ParDo</h5> |
| <p class="language-java">Like all Beam transforms, you apply <code>ParDo</code> by calling the <code>apply</code> method on the |
| input <code>PCollection</code> and passing <code>ParDo</code> as an argument, as shown in the |
| following example code:</p> |
| <p class="language-py">Like all Beam transforms, you apply <code>ParDo</code> by calling the <code>beam.ParDo</code> on the |
| input <code>PCollection</code> and passing the <code>DoFn</code> as an argument, as shown in the |
| following example code:</p> |
| <p class="language-go"><code>beam.ParDo</code> applies the passed in <code>DoFn</code> argument to the input <code>PCollection</code>, |
| as shown in the following example code:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// The input PCollection of Strings. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">words</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// The DoFn to perform on each element in the input PCollection. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">static</span> <span class="kd">class</span> <span class="nc">ComputeWordLengthFn</span> <span class="kd">extends</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;</span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Apply a ParDo to the PCollection &#34;words&#34; to compute lengths for each word. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">wordLengths</span> <span class="o">=</span> <span class="n">words</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ParDo</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">ComputeWordLengthFn</span><span class="o">()));</span> <span class="c1">// The DoFn to perform on each element, which |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// we define above. |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># The input PCollection of Strings.</span> |
| </span></span><span class="line"><span class="cl"><span class="n">words</span> <span class="o">=</span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># The DoFn to perform on each element in the input PCollection.</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">ComputeWordLengthFn</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="p">[</span><span class="nb">len</span><span class="p">(</span><span class="n">element</span><span class="p">)]</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Apply a ParDo to the PCollection &#34;words&#34; to compute lengths for each word.</span> |
| </span></span><span class="line"><span class="cl"><span class="n">word_lengths</span> <span class="o">=</span> <span class="n">words</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">ComputeWordLengthFn</span><span class="p">())</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// ComputeWordLengthFn is the DoFn to perform on each element in the input PCollection. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">type</span> <span class="nx">ComputeWordLengthFn</span> <span class="kd">struct</span><span class="p">{}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// ProcessElement is the method to execute for each element. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">ComputeWordLengthFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">word</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="kt">int</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emit</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="nx">word</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// DoFns must be registered with beam. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nf">RegisterType</span><span class="p">(</span><span class="nx">reflect</span><span class="p">.</span><span class="nf">TypeOf</span><span class="p">((</span><span class="o">*</span><span class="nx">ComputeWordLengthFn</span><span class="p">)(</span><span class="kc">nil</span><span class="p">)))</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// 2 inputs and 0 outputs =&gt; DoFn2x0 |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// 1 input =&gt; Emitter1 |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// Input/output types are included in order in the brackets |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">register</span><span class="p">.</span><span class="nx">DoFn2x0</span><span class="p">[</span><span class="kt">string</span><span class="p">,</span> <span class="kd">func</span><span class="p">(</span><span class="kt">int</span><span class="p">)](</span><span class="o">&amp;</span><span class="nx">ComputeWordLengthFn</span><span class="p">{})</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">register</span><span class="p">.</span><span class="nx">Emitter1</span><span class="p">[</span><span class="kt">int</span><span class="p">]()</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// words is an input PCollection of strings |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">var</span> <span class="nx">words</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">=</span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="nx">wordLengths</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">ComputeWordLengthFn</span><span class="p">{},</span> <span class="nx">words</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="err">#</span> <span class="nx">The</span> <span class="nx">input</span> <span class="nx">PCollection</span> <span class="k">of</span> <span class="nx">Strings</span><span class="p">.</span> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">words</span> : <span class="kt">PCollection</span><span class="p">&lt;</span><span class="nt">string</span><span class="p">&gt;</span> <span class="o">=</span> <span class="p">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="err">#</span> <span class="nx">The</span> <span class="nx">DoFn</span> <span class="nx">to</span> <span class="nx">perform</span> <span class="nx">on</span> <span class="nx">each</span> <span class="nx">element</span> <span class="k">in</span> <span class="nx">the</span> <span class="nx">input</span> <span class="nx">PCollection</span><span class="p">.</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">function</span> <span class="nx">computeWordLengthFn</span><span class="p">()</span><span class="o">:</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">DoFn</span><span class="p">&lt;</span><span class="nt">string</span><span class="err">,</span> <span class="na">number</span><span class="p">&gt;</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">process</span>: <span class="kt">function</span><span class="o">*</span> <span class="p">(</span><span class="nx">element</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="nx">element</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">};</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">words</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">parDo</span><span class="p">(</span><span class="nx">computeWordLengthFn</span><span class="p">()));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>In the example, our input <code>PCollection</code> contains <span class="language-java language-py"><code>String</code></span> |
| <span class="language-go"><code>string</code></span> values. We apply a |
| <code>ParDo</code> transform that specifies a function (<code>ComputeWordLengthFn</code>) to compute |
| the length of each string, and outputs the result to a new <code>PCollection</code> of |
| <span class="language-java language-py"><code>Integer</code></span> |
| <span class="language-go"><code>int</code></span> values that stores the length of each word.</p> |
| <h5 id="4212-creating-a-dofn">4.2.1.2. Creating a DoFn</h5> |
| <p>The <code>DoFn</code> object that you pass to <code>ParDo</code> contains the processing logic that |
| gets applied to the elements in the input collection. When you use Beam, often |
| the most important pieces of code you&rsquo;ll write are these <code>DoFn</code>s - they&rsquo;re what |
| define your pipeline&rsquo;s exact data processing tasks.</p> |
| <blockquote> |
| <p><strong>Note:</strong> When you create your <code>DoFn</code>, be mindful of the <a href="#requirements-for-writing-user-code-for-beam-transforms">Requirements |
| for writing user code for Beam transforms</a> |
| and ensure that your code follows them.</p> |
| </blockquote> |
| <p class="language-java">A <code>DoFn</code> processes one element at a time from the input <code>PCollection</code>. When you |
| create a subclass of <code>DoFn</code>, you&rsquo;ll need to provide type parameters that match |
| the types of the input and output elements. If your <code>DoFn</code> processes incoming |
| <code>String</code> elements and produces <code>Integer</code> elements for the output collection |
| (like our previous example, <code>ComputeWordLengthFn</code>), your class declaration would |
| look like this:</p> |
| <p class="language-go">A <code>DoFn</code> processes one element at a time from the input <code>PCollection</code>. When you |
| create a <code>DoFn</code> struct, you&rsquo;ll need to provide type parameters that match |
| the types of the input and output elements in a ProcessElement method. |
| If your <code>DoFn</code> processes incoming <code>string</code> elements and produces <code>int</code> elements |
| for the output collection (like our previous example, <code>ComputeWordLengthFn</code>), your dofn could |
| look like this:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">static</span> <span class="kd">class</span> <span class="nc">ComputeWordLengthFn</span> <span class="kd">extends</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;</span> <span class="o">{</span> <span class="o">...</span> <span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// ComputeWordLengthFn is a DoFn that computes the word length of string elements. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">type</span> <span class="nx">ComputeWordLengthFn</span> <span class="kd">struct</span><span class="p">{}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// ProcessElement computes the length of word and emits the result. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// When creating structs as a DoFn, the ProcessElement method performs the |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// work of this step in the pipeline. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">ComputeWordLengthFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">word</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="kt">int</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// 2 inputs and 0 outputs =&gt; DoFn2x0 |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// 1 input =&gt; Emitter1 |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// Input/output types are included in order in the brackets |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">register</span><span class="p">.</span><span class="nf">Function2x0</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">ComputeWordLengthFn</span><span class="p">{})</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">register</span><span class="p">.</span><span class="nx">Emitter1</span><span class="p">[</span><span class="kt">int</span><span class="p">]()</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">Inside your <code>DoFn</code> subclass, you&rsquo;ll write a method annotated with |
| <code>@ProcessElement</code> where you provide the actual processing logic. You don&rsquo;t need |
| to manually extract the elements from the input collection; the Beam SDKs handle |
| that for you. Your <code>@ProcessElement</code> method should accept a parameter tagged with |
| <code>@Element</code>, which will be populated with the input element. In order to output |
| elements, the method can also take a parameter of type <code>OutputReceiver</code> which |
| provides a method for emitting elements. The parameter types must match the input |
| and output types of your <code>DoFn</code> or the framework will raise an error. Note: <code>@Element</code> and |
| <code>OutputReceiver</code> were introduced in Beam 2.5.0; if using an earlier release of Beam, a |
| <code>ProcessContext</code> parameter should be used instead.</p> |
| <p class="language-py language-typescript">Inside your <code>DoFn</code> subclass, you&rsquo;ll write a method <code>process</code> where you provide |
| the actual processing logic. You don&rsquo;t need to manually extract the elements |
| from the input collection; the Beam SDKs handle that for you. Your <code>process</code> method |
| should accept an argument <code>element</code>, which is the input element, and return an |
| iterable with its output values. You can accomplish this by emitting individual |
| elements with <code>yield</code> statements, and use <code>yield from</code> to emit all elements from |
| an iterable, such as a list or a generator. Using <code>return</code> statement |
| with an iterable is also acceptable as long as you don&rsquo;t mix <code>yield</code> and |
| <code>return</code> statements in the same <code>process</code> method, since that leads to <a href="https://github.com/apache/beam/issues/22969">incorrect behavior</a>.</p> |
| <p class="language-go">For your <code>DoFn</code> type, you&rsquo;ll write a method <code>ProcessElement</code> where you provide |
| the actual processing logic. You don&rsquo;t need to manually extract the elements |
| from the input collection; the Beam SDKs handle that for you. Your <code>ProcessElement</code> method |
| should accept a parameter <code>element</code>, which is the input element. In order to output elements, |
| the method can also take a function parameter, which can be called to emit elements. |
| The parameter types must match the input and output types of your <code>DoFn</code> |
| or the framework will raise an error.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">static</span> <span class="kd">class</span> <span class="nc">ComputeWordLengthFn</span> <span class="kd">extends</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="nd">@Element</span> <span class="n">String</span> <span class="n">word</span><span class="o">,</span> <span class="n">OutputReceiver</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">out</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Use OutputReceiver.output to emit the output element. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">out</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">word</span><span class="o">.</span><span class="na">length</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">ComputeWordLengthFn</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="p">[</span><span class="nb">len</span><span class="p">(</span><span class="n">element</span><span class="p">)]</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// ComputeWordLengthFn is the DoFn to perform on each element in the input PCollection. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">type</span> <span class="nx">ComputeWordLengthFn</span> <span class="kd">struct</span><span class="p">{}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// ProcessElement is the method to execute for each element. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">ComputeWordLengthFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">word</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="kt">int</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emit</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="nx">word</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// DoFns must be registered with beam. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nf">RegisterType</span><span class="p">(</span><span class="nx">reflect</span><span class="p">.</span><span class="nf">TypeOf</span><span class="p">((</span><span class="o">*</span><span class="nx">ComputeWordLengthFn</span><span class="p">)(</span><span class="kc">nil</span><span class="p">)))</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// 2 inputs and 0 outputs =&gt; DoFn2x0 |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// 1 input =&gt; Emitter1 |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// Input/output types are included in order in the brackets |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">register</span><span class="p">.</span><span class="nx">DoFn2x0</span><span class="p">[</span><span class="kt">string</span><span class="p">,</span> <span class="kd">func</span><span class="p">(</span><span class="kt">int</span><span class="p">)](</span><span class="o">&amp;</span><span class="nx">ComputeWordLengthFn</span><span class="p">{})</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">register</span><span class="p">.</span><span class="nx">Emitter1</span><span class="p">[</span><span class="kt">int</span><span class="p">]()</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kd">function</span> <span class="nx">computeWordLengthFn</span><span class="p">()</span><span class="o">:</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">DoFn</span><span class="p">&lt;</span><span class="nt">string</span><span class="err">,</span> <span class="na">number</span><span class="p">&gt;</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">process</span>: <span class="kt">function</span><span class="o">*</span> <span class="p">(</span><span class="nx">element</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="nx">element</span><span class="p">.</span><span class="nx">length</span><span class="p">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">};</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-go">Simple DoFns can also be written as functions.</p> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">ComputeWordLengthFn</span><span class="p">(</span><span class="nx">word</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="kt">int</span><span class="p">))</span> <span class="p">{</span> <span class="o">...</span> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// 2 inputs and 0 outputs =&gt; DoFn2x0 |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// 1 input =&gt; Emitter1 |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// Input/output types are included in order in the brackets |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">register</span><span class="p">.</span><span class="nx">DoFn2x0</span><span class="p">[</span><span class="kt">string</span><span class="p">,</span> <span class="kd">func</span><span class="p">(</span><span class="kt">int</span><span class="p">)](</span><span class="o">&amp;</span><span class="nx">ComputeWordLengthFn</span><span class="p">{})</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">register</span><span class="p">.</span><span class="nx">Emitter1</span><span class="p">[</span><span class="kt">int</span><span class="p">]()</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <span class="language-go"> |
| <blockquote> |
| <p><strong>Note:</strong> Whether using a structural <code>DoFn</code> type or a functional <code>DoFn</code>, they should be registered with |
| beam in an <code>init</code> block. Otherwise they may not execute on distributed runners.</p> |
| </blockquote> |
| </span> |
| <span class="language-java"> |
| <blockquote> |
| <p><strong>Note:</strong> If the elements in your input <code>PCollection</code> are key/value pairs, you |
| can access the key or value by using <code>element.getKey()</code> or |
| <code>element.getValue()</code>, respectively.</p> |
| </blockquote> |
| </span> |
| <span class="language-go"> |
| <blockquote> |
| <p><strong>Note:</strong> If the elements in your input <code>PCollection</code> are key/value pairs, your |
| process element method must have two parameters, for each of the key and value, |
| respectively. Similarly, key/value pairs are also output as separate |
| parameters to a single <code>emitter function</code>.</p> |
| </blockquote> |
| </span> |
| <p>A given <code>DoFn</code> instance generally gets invoked one or more times to process some |
| arbitrary bundle of elements. However, Beam doesn&rsquo;t guarantee an exact number of |
| invocations; it may be invoked multiple times on a given worker node to account |
| for failures and retries. As such, you can cache information across multiple |
| calls to your processing method, but if you do so, make sure the implementation |
| <strong>does not depend on the number of invocations</strong>.</p> |
| <p>In your processing method, you&rsquo;ll also need to meet some immutability |
| requirements to ensure that Beam and the processing back-end can safely |
| serialize and cache the values in your pipeline. Your method should meet the |
| following requirements:</p> |
| <span class="language-java"> |
| <ul> |
| <li>You should not in any way modify an element returned by |
| the <code>@Element</code> annotation or <code>ProcessContext.sideInput()</code> (the incoming |
| elements from the input collection).</li> |
| <li>Once you output a value using <code>OutputReceiver.output()</code> you should not modify |
| that value in any way.</li> |
| </ul> |
| </span> |
| <span class="language-py language-typescript"> |
| <ul> |
| <li>You should not in any way modify the <code>element</code> argument provided to the |
| <code>process</code> method, or any side inputs.</li> |
| <li>Once you output a value using <code>yield</code> or <code>return</code>, you should not modify |
| that value in any way.</li> |
| </ul> |
| </span> |
| <span class="language-go"> |
| <ul> |
| <li>You should not in any way modify the parameters provided to the |
| <code>ProcessElement</code> method, or any side inputs.</li> |
| <li>Once you output a value using an <code>emitter function</code>, you should not modify |
| that value in any way.</li> |
| </ul> |
| </span> |
| <h5 id="lightweight-dofns">4.2.1.3. Lightweight DoFns and other abstractions</h5> |
| <p>If your function is relatively straightforward, you can simplify your use of |
| <code>ParDo</code> by providing a lightweight <code>DoFn</code> in-line, as |
| <span class="language-java">an anonymous inner class instance</span> |
| <span class="language-py">a lambda function</span> |
| <span class="language-go">an anonymous function</span> |
| <span class="language-typescript">a function passed to <code>PCollection.map</code> or <code>PCollection.flatMap</code></span>.</p> |
| <p>Here&rsquo;s the previous example, <code>ParDo</code> with <code>ComputeLengthWordsFn</code>, with the |
| <code>DoFn</code> specified as |
| <span class="language-java">an anonymous inner class instance</span> |
| <span class="language-py">a lambda function</span> |
| <span class="language-go">an anonymous function</span> |
| <span class="language-typescript">a function</span>:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// The input PCollection. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">words</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Apply a ParDo with an anonymous DoFn to the PCollection words. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// Save the result as the PCollection wordLengths. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">wordLengths</span> <span class="o">=</span> <span class="n">words</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;ComputeWordLengths&#34;</span><span class="o">,</span> <span class="c1">// the transform name |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;()</span> <span class="o">{</span> <span class="c1">// a DoFn as an anonymous inner class instance |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="nd">@Element</span> <span class="n">String</span> <span class="n">word</span><span class="o">,</span> <span class="n">OutputReceiver</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">out</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">out</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">word</span><span class="o">.</span><span class="na">length</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># The input PCollection of strings.</span> |
| </span></span><span class="line"><span class="cl"><span class="n">words</span> <span class="o">=</span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Apply a lambda function to the PCollection words.</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Save the result as the PCollection word_lengths.</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">word_lengths</span> <span class="o">=</span> <span class="n">words</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span class="k">lambda</span> <span class="n">word</span><span class="p">:</span> <span class="p">[</span><span class="nb">len</span><span class="p">(</span><span class="n">word</span><span class="p">)])</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">The</span> <span class="nx">Go</span> <span class="nx">SDK</span> <span class="nx">cannot</span> <span class="nx">support</span> <span class="nx">anonymous</span> <span class="nx">functions</span> <span class="nx">outside</span> <span class="nx">of</span> <span class="nx">the</span> <span class="nx">deprecated</span> <span class="nx">Go</span> <span class="nx">Direct</span> <span class="nx">runner</span><span class="p">.</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// words is the input PCollection of strings |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">var</span> <span class="nx">words</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">=</span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="nx">lengths</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="kd">func</span> <span class="p">(</span><span class="nx">word</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="kt">int</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emit</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="nx">word</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"><span class="p">},</span> <span class="nx">words</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="c1">// The input PCollection of strings. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">words</span> <span class="o">=</span> <span class="p">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">words</span><span class="p">.</span><span class="nx">flatMap</span><span class="p">((</span><span class="nx">word</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">[</span><span class="nx">word</span><span class="p">.</span><span class="nx">length</span><span class="p">]);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>If your <code>ParDo</code> performs a one-to-one mapping of input elements to output |
| elements&ndash;that is, for each input element, it applies a function that produces |
| <em>exactly one</em> output element, <span class="language-go">you can return that |
| element directly.</span><span class="language-java language-py">you can use the higher-level |
| <span class="language-java"><code>MapElements</code></span><span class="language-py language-py"><code>Map</code></span> |
| transform.</span><span class="language-java"><code>MapElements</code> can accept an anonymous |
| Java 8 lambda function for additional brevity.</span></p> |
| <p>Here&rsquo;s the previous example using <span class="language-java"><code>MapElements</code></span> |
| <span class="language-py language-typescript"><code>Map</code></span><span class="language-go">a direct return</span>:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// The input PCollection. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">words</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Apply a MapElements with an anonymous lambda function to the PCollection words. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// Save the result as the PCollection wordLengths. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">wordLengths</span> <span class="o">=</span> <span class="n">words</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">integers</span><span class="o">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">via</span><span class="o">((</span><span class="n">String</span> <span class="n">word</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">word</span><span class="o">.</span><span class="na">length</span><span class="o">()));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># The input PCollection of string.</span> |
| </span></span><span class="line"><span class="cl"><span class="n">words</span> <span class="o">=</span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Apply a Map with a lambda function to the PCollection words.</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Save the result as the PCollection word_lengths.</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">word_lengths</span> <span class="o">=</span> <span class="n">words</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="nb">len</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">The</span> <span class="nx">Go</span> <span class="nx">SDK</span> <span class="nx">cannot</span> <span class="nx">support</span> <span class="nx">anonymous</span> <span class="nx">functions</span> <span class="nx">outside</span> <span class="nx">of</span> <span class="nx">the</span> <span class="nx">deprecated</span> <span class="nx">Go</span> <span class="nx">Direct</span> <span class="nx">runner</span><span class="p">.</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">wordLengths</span><span class="p">(</span><span class="nx">word</span> <span class="kt">string</span><span class="p">)</span> <span class="kt">int</span> <span class="p">{</span> <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="nx">word</span><span class="p">)</span> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> <span class="nx">register</span><span class="p">.</span><span class="nf">Function1x1</span><span class="p">(</span><span class="nx">wordLengths</span><span class="p">)</span> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">applyWordLenAnon</span><span class="p">(</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">words</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span><span class="p">)</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">wordLengths</span><span class="p">,</span> <span class="nx">words</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="c1">// The input PCollection of string. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">words</span> <span class="o">=</span> <span class="p">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">words</span><span class="p">.</span><span class="nx">map</span><span class="p">((</span><span class="nx">word</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">word</span><span class="p">.</span><span class="nx">length</span><span class="p">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <span class="language-java"> |
| <blockquote> |
| <p><strong>Note:</strong> You can use Java 8 lambda functions with several other Beam |
| transforms, including <code>Filter</code>, <code>FlatMapElements</code>, and <code>Partition</code>.</p> |
| </blockquote> |
| </span> |
| <span class="language-go"> |
| <blockquote> |
| <p><strong>Note:</strong> Anonymous function DoFns do not work on distributed runners. |
| It&rsquo;s recommended to use named functions and register them with <code>register.FunctionXxY</code> in |
| an <code>init()</code> block.</p> |
| </blockquote> |
| </span> |
| <h5 id="dofn">4.2.1.4. DoFn lifecycle</h5> |
| <p>Here is a sequence diagram that shows the lifecycle of the DoFn during |
| the execution of the ParDo transform. The comments give useful |
| information to pipeline developers such as the constraints that |
| apply to the objects or particular cases such as failover or |
| instance reuse. They also give instantiation use cases. Three key points |
| to note are that:</p> |
| <ol> |
| <li>Teardown is done on a best effort basis and thus |
| isn&rsquo;t guaranteed.</li> |
| <li>The number of DoFn instances created at runtime is runner-dependent.</li> |
| <li>For the Python SDK, the pipeline contents such as DoFn user code, |
| is <a href="https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#pickling-and-managing-the-main-session">serialized into a bytecode</a>. Therefore, <code>DoFn</code>s should not reference objects that are not serializable, such as locks. To manage a single instance of an object across multiple <code>DoFn</code> instances in the same process, use utilities in the <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.utils.shared.html">shared.py</a> module.</li> |
| </ol> |
| <!-- The source for the sequence diagram can be found in the SVG resource. --> |
| <p><img src="/images/dofn-sequence-diagram.svg" alt="This is a sequence diagram that shows the lifecycle of the DoFn"></p> |
| <h4 id="groupbykey">4.2.2. GroupByKey</h4> |
| <p><code>GroupByKey</code> is a Beam transform for processing collections of key/value pairs. |
| It&rsquo;s a parallel reduction operation, analogous to the Shuffle phase of a |
| Map/Shuffle/Reduce-style algorithm. The input to <code>GroupByKey</code> is a collection of |
| key/value pairs that represents a <em>multimap</em>, where the collection contains |
| multiple pairs that have the same key, but different values. Given such a |
| collection, you use <code>GroupByKey</code> to collect all of the values associated with |
| each unique key.</p> |
| <p><code>GroupByKey</code> is a good way to aggregate data that has something in common. For |
| example, if you have a collection that stores records of customer orders, you |
| might want to group together all the orders from the same postal code (wherein |
| the &ldquo;key&rdquo; of the key/value pair is the postal code field, and the &ldquo;value&rdquo; is the |
| remainder of the record).</p> |
| <p>Let&rsquo;s examine the mechanics of <code>GroupByKey</code> with a simple example case, where |
| our data set consists of words from a text file and the line number on which |
| they appear. We want to group together all the line numbers (values) that share |
| the same word (key), letting us see all the places in the text where a |
| particular word appears.</p> |
| <p>Our input is a <code>PCollection</code> of key/value pairs where each word is a key, and |
| the value is a line number in the file where the word appears. Here&rsquo;s a list of |
| the key/value pairs in the input collection:</p> |
| <pre tabindex="0"><code>cat, 1 |
| dog, 5 |
| and, 1 |
| jump, 3 |
| tree, 2 |
| cat, 5 |
| dog, 2 |
| and, 2 |
| cat, 9 |
| and, 6 |
| ... |
| </code></pre><p><code>GroupByKey</code> gathers up all the values with the same key and outputs a new pair |
| consisting of the unique key and a collection of all of the values that were |
| associated with that key in the input collection. If we apply <code>GroupByKey</code> to |
| our input collection above, the output collection would look like this:</p> |
| <pre tabindex="0"><code>cat, [1,5,9] |
| dog, [5,2] |
| and, [1,2,6] |
| jump, [3] |
| tree, [2] |
| ... |
| </code></pre><p>Thus, <code>GroupByKey</code> represents a transform from a multimap (multiple keys to |
| individual values) to a uni-map (unique keys to collections of values).</p> |
| <p><span class="language-java">Using <code>GroupByKey</code> is straightforward:</span></p> |
| <p class="language-py language-typescript language-yaml">While all SDKs have a <code>GroupByKey</code> transform, using <code>GroupBy</code> is |
| generally more natural. |
| The <code>GroupBy</code> transform can be parameterized by the name(s) of properties |
| on which to group the elements of the PCollection, or a function taking |
| the each element as input that maps to a key on which to do grouping.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// The input PCollection. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;&gt;</span> <span class="n">mapped</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Apply GroupByKey to the PCollection mapped. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// Save the result as the PCollection reduced. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Iterable</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;&gt;&gt;</span> <span class="n">reduced</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">mapped</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">GroupByKey</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span><span class="n">create</span><span class="o">());</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># The input PCollection of (`string`, `int`) tuples.</span> |
| </span></span><span class="line"><span class="cl"><span class="n">words_and_counts</span> <span class="o">=</span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">grouped_words</span> <span class="o">=</span> <span class="n">words_and_counts</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">GroupByKey</span><span class="p">()</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// CreateAndSplit creates and returns a PCollection with &lt;K,V&gt; |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// from an input slice of stringPair (struct with K, V string fields). |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">pairs</span> <span class="o">:=</span> <span class="nf">CreateAndSplit</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">input</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="nx">keyed</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">GroupByKey</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">pairs</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="c1">// A PCollection of elements like |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// {word: &#34;cat&#34;, score: 1}, {word: &#34;dog&#34;, score: 5}, {word: &#34;cat&#34;, score: 5}, ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kr">const</span> <span class="nx">scores</span> : <span class="kt">PCollection</span><span class="o">&lt;</span><span class="p">{</span><span class="nx">word</span>: <span class="kt">string</span><span class="p">,</span> <span class="nx">score</span>: <span class="kt">number</span><span class="p">}</span><span class="o">&gt;</span> <span class="o">=</span> <span class="p">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// This will produce a PCollection with elements like |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// {key: &#34;cat&#34;, value: [{ word: &#34;cat&#34;, score: 1 }, |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// { word: &#34;cat&#34;, score: 5 }, ...]} |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// {key: &#34;dog&#34;, value: [{ word: &#34;dog&#34;, score: 5 }, ...]} |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kr">const</span> <span class="nx">grouped_by_word</span> <span class="o">=</span> <span class="nx">scores</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">groupBy</span><span class="p">(</span><span class="s2">&#34;word&#34;</span><span class="p">));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// This will produce a PCollection with elements like |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// {key: 3, value: [{ word: &#34;cat&#34;, score: 1 }, |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// { word: &#34;dog&#34;, score: 5 }, |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// { word: &#34;cat&#34;, score: 5 }, ...]} |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kr">const</span> <span class="nx">by_word_length</span> <span class="o">=</span> <span class="nx">scores</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">groupBy</span><span class="p">((</span><span class="nx">x</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">x</span><span class="p">.</span><span class="nx">word</span><span class="p">.</span><span class="nx">length</span><span class="p">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">Combine</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">group_by</span><span class="p">:</span><span class="w"> </span><span class="l">animal</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">combine</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">weight</span><span class="p">:</span><span class="w"> </span><span class="l">group</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h5 id="groupbykey-and-unbounded-pcollections">4.2.2.1 GroupByKey and unbounded PCollections</h5> |
| <p>If you are using unbounded <code>PCollection</code>s, you must use either <a href="#setting-your-pcollections-windowing-function">non-global |
| windowing</a> or an |
| <a href="#triggers">aggregation trigger</a> in order to perform a <code>GroupByKey</code> or |
| <a href="#cogroupbykey">CoGroupByKey</a>. This is because a bounded <code>GroupByKey</code> or |
| <code>CoGroupByKey</code> must wait for all the data with a certain key to be collected, |
| but with unbounded collections, the data is unlimited. Windowing and/or triggers |
| allow grouping to operate on logical, finite bundles of data within the |
| unbounded data streams.</p> |
| <p>If you do apply <code>GroupByKey</code> or <code>CoGroupByKey</code> to a group of unbounded |
| <code>PCollection</code>s without setting either a non-global windowing strategy, a trigger |
| strategy, or both for each collection, Beam generates an IllegalStateException |
| error at pipeline construction time.</p> |
| <p>When using <code>GroupByKey</code> or <code>CoGroupByKey</code> to group <code>PCollection</code>s that have a |
| <a href="#windowing">windowing strategy</a> applied, all of the <code>PCollection</code>s you want to |
| group <em>must use the same windowing strategy</em> and window sizing. For example, all |
| of the collections you are merging must use (hypothetically) identical 5-minute |
| fixed windows, or 4-minute sliding windows starting every 30 seconds.</p> |
| <p>If your pipeline attempts to use <code>GroupByKey</code> or <code>CoGroupByKey</code> to merge |
| <code>PCollection</code>s with incompatible windows, Beam generates an |
| IllegalStateException error at pipeline construction time.</p> |
| <h4 id="cogroupbykey">4.2.3. CoGroupByKey</h4> |
| <p><code>CoGroupByKey</code> performs a relational join of two or more key/value |
| <code>PCollection</code>s that have the same key type. |
| <a href="/documentation/pipelines/design-your-pipeline/#multiple-sources">Design Your Pipeline</a> |
| shows an example pipeline that uses a join.</p> |
| <p>Consider using <code>CoGroupByKey</code> if you have multiple data sets that provide |
| information about related things. For example, let&rsquo;s say you have two different |
| files with user data: one file has names and email addresses; the other file |
| has names and phone numbers. You can join those two data sets, using the user |
| name as a common key and the other data as the associated values. After the |
| join, you have one data set that contains all of the information (email |
| addresses and phone numbers) associated with each name.</p> |
| <p>One can also consider using SqlTransform to perform a join.</p> |
| <p>If you are using unbounded <code>PCollection</code>s, you must use either <a href="#setting-your-pcollections-windowing-function">non-global |
| windowing</a> or an |
| <a href="#triggers">aggregation trigger</a> in order to perform a <code>CoGroupByKey</code>. See |
| <a href="#groupbykey-and-unbounded-pcollections">GroupByKey and unbounded PCollections</a> |
| for more details.</p> |
| <p class="language-java">In the Beam SDK for Java, <code>CoGroupByKey</code> accepts a tuple of keyed |
| <code>PCollection</code>s (<code>PCollection&lt;KV&lt;K, V&gt;&gt;</code>) as input. For type safety, the SDK |
| requires you to pass each <code>PCollection</code> as part of a <code>KeyedPCollectionTuple</code>. |
| You must declare a <code>TupleTag</code> for each input <code>PCollection</code> in the |
| <code>KeyedPCollectionTuple</code> that you want to pass to <code>CoGroupByKey</code>. As output, |
| <code>CoGroupByKey</code> returns a <code>PCollection&lt;KV&lt;K, CoGbkResult&gt;&gt;</code>, which groups values |
| from all the input <code>PCollection</code>s by their common keys. Each key (all of type |
| <code>K</code>) will have a different <code>CoGbkResult</code>, which is a map from <code>TupleTag&lt;T&gt;</code> to |
| <code>Iterable&lt;T&gt;</code>. You can access a specific collection in an <code>CoGbkResult</code> object |
| by using the <code>TupleTag</code> that you supplied with the initial collection.</p> |
| <p class="language-py">In the Beam SDK for Python, <code>CoGroupByKey</code> accepts a dictionary of keyed |
| <code>PCollection</code>s as input. As output, <code>CoGroupByKey</code> creates a single output |
| <code>PCollection</code> that contains one key/value tuple for each key in the input |
| <code>PCollection</code>s. Each key&rsquo;s value is a dictionary that maps each tag to an |
| iterable of the values under they key in the corresponding <code>PCollection</code>.</p> |
| <p class="language-go">In the Beam Go SDK, <code>CoGroupByKey</code> accepts an arbitrary number of |
| <code>PCollection</code>s as input. As output, <code>CoGroupByKey</code> creates a single output |
| <code>PCollection</code> that groups each key with value iterator functions for each |
| input <code>PCollection</code>. The iterator functions map to input <code>PCollections</code> in |
| the same order they were provided to the <code>CoGroupByKey</code>.</p> |
| <p>The following conceptual examples use two input collections to show the mechanics of |
| <code>CoGroupByKey</code>.</p> |
| <p class="language-java">The first set of data has a <code>TupleTag&lt;String&gt;</code> called <code>emailsTag</code> and contains names |
| and email addresses. The second set of data has a <code>TupleTag&lt;String&gt;</code> called |
| <code>phonesTag</code> and contains names and phone numbers.</p> |
| <p class="language-py language-go">The first set of data contains names and email addresses. The second set of |
| data contains names and phone numbers.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">final</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;&gt;</span> <span class="n">emailsList</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;amy&#34;</span><span class="o">,</span> <span class="s">&#34;amy@example.com&#34;</span><span class="o">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;carl&#34;</span><span class="o">,</span> <span class="s">&#34;carl@example.com&#34;</span><span class="o">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;julia&#34;</span><span class="o">,</span> <span class="s">&#34;julia@example.com&#34;</span><span class="o">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;carl&#34;</span><span class="o">,</span> <span class="s">&#34;carl@email.com&#34;</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">final</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;&gt;</span> <span class="n">phonesList</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;amy&#34;</span><span class="o">,</span> <span class="s">&#34;111-222-3333&#34;</span><span class="o">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;james&#34;</span><span class="o">,</span> <span class="s">&#34;222-333-4444&#34;</span><span class="o">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;amy&#34;</span><span class="o">,</span> <span class="s">&#34;333-444-5555&#34;</span><span class="o">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;carl&#34;</span><span class="o">,</span> <span class="s">&#34;444-555-6666&#34;</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;&gt;</span> <span class="n">emails</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;CreateEmails&#34;</span><span class="o">,</span> <span class="n">Create</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">emailsList</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;&gt;</span> <span class="n">phones</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;CreatePhones&#34;</span><span class="o">,</span> <span class="n">Create</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">phonesList</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">emails_list</span> <span class="o">=</span> <span class="p">[</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;amy&#39;</span><span class="p">,</span> <span class="s1">&#39;amy@example.com&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;carl&#39;</span><span class="p">,</span> <span class="s1">&#39;carl@example.com&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;julia&#39;</span><span class="p">,</span> <span class="s1">&#39;julia@example.com&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;carl&#39;</span><span class="p">,</span> <span class="s1">&#39;carl@email.com&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"><span class="n">phones_list</span> <span class="o">=</span> <span class="p">[</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;amy&#39;</span><span class="p">,</span> <span class="s1">&#39;111-222-3333&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;james&#39;</span><span class="p">,</span> <span class="s1">&#39;222-333-4444&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;amy&#39;</span><span class="p">,</span> <span class="s1">&#39;333-444-5555&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;carl&#39;</span><span class="p">,</span> <span class="s1">&#39;444-555-6666&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">emails</span> <span class="o">=</span> <span class="n">p</span> <span class="o">|</span> <span class="s1">&#39;CreateEmails&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">(</span><span class="n">emails_list</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="n">phones</span> <span class="o">=</span> <span class="n">p</span> <span class="o">|</span> <span class="s1">&#39;CreatePhones&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">(</span><span class="n">phones_list</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">stringPair</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">K</span><span class="p">,</span> <span class="nx">V</span> <span class="kt">string</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">splitStringPair</span><span class="p">(</span><span class="nx">e</span> <span class="nx">stringPair</span><span class="p">)</span> <span class="p">(</span><span class="kt">string</span><span class="p">,</span> <span class="kt">string</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">e</span><span class="p">.</span><span class="nx">K</span><span class="p">,</span> <span class="nx">e</span><span class="p">.</span><span class="nx">V</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Register DoFn. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">register</span><span class="p">.</span><span class="nf">Function1x2</span><span class="p">(</span><span class="nx">splitStringPair</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// CreateAndSplit is a helper function that creates |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="nf">CreateAndSplit</span><span class="p">(</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">input</span> <span class="p">[]</span><span class="nx">stringPair</span><span class="p">)</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">initial</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">CreateList</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">input</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">splitStringPair</span><span class="p">,</span> <span class="nx">initial</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">var</span> <span class="nx">emailSlice</span> <span class="p">=</span> <span class="p">[]</span><span class="nx">stringPair</span><span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span><span class="s">&#34;amy&#34;</span><span class="p">,</span> <span class="s">&#34;amy@example.com&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span><span class="s">&#34;carl&#34;</span><span class="p">,</span> <span class="s">&#34;carl@example.com&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span><span class="s">&#34;julia&#34;</span><span class="p">,</span> <span class="s">&#34;julia@example.com&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span><span class="s">&#34;carl&#34;</span><span class="p">,</span> <span class="s">&#34;carl@email.com&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">var</span> <span class="nx">phoneSlice</span> <span class="p">=</span> <span class="p">[]</span><span class="nx">stringPair</span><span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span><span class="s">&#34;amy&#34;</span><span class="p">,</span> <span class="s">&#34;111-222-3333&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span><span class="s">&#34;james&#34;</span><span class="p">,</span> <span class="s">&#34;222-333-4444&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span><span class="s">&#34;amy&#34;</span><span class="p">,</span> <span class="s">&#34;333-444-5555&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span><span class="s">&#34;carl&#34;</span><span class="p">,</span> <span class="s">&#34;444-555-6666&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="nx">emails</span> <span class="o">:=</span> <span class="nf">CreateAndSplit</span><span class="p">(</span><span class="nx">s</span><span class="p">.</span><span class="nf">Scope</span><span class="p">(</span><span class="s">&#34;CreateEmails&#34;</span><span class="p">),</span> <span class="nx">emailSlice</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="nx">phones</span> <span class="o">:=</span> <span class="nf">CreateAndSplit</span><span class="p">(</span><span class="nx">s</span><span class="p">.</span><span class="nf">Scope</span><span class="p">(</span><span class="s">&#34;CreatePhones&#34;</span><span class="p">),</span> <span class="nx">phoneSlice</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">emails_list</span> <span class="o">=</span> <span class="p">[</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;amy&#34;</span><span class="p">,</span> <span class="nx">email</span><span class="o">:</span> <span class="s2">&#34;amy@example.com&#34;</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;carl&#34;</span><span class="p">,</span> <span class="nx">email</span><span class="o">:</span> <span class="s2">&#34;carl@example.com&#34;</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;julia&#34;</span><span class="p">,</span> <span class="nx">email</span><span class="o">:</span> <span class="s2">&#34;julia@example.com&#34;</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;carl&#34;</span><span class="p">,</span> <span class="nx">email</span><span class="o">:</span> <span class="s2">&#34;carl@email.com&#34;</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"><span class="p">];</span> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">phones_list</span> <span class="o">=</span> <span class="p">[</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;amy&#34;</span><span class="p">,</span> <span class="nx">phone</span><span class="o">:</span> <span class="s2">&#34;111-222-3333&#34;</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;james&#34;</span><span class="p">,</span> <span class="nx">phone</span><span class="o">:</span> <span class="s2">&#34;222-333-4444&#34;</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;amy&#34;</span><span class="p">,</span> <span class="nx">phone</span><span class="o">:</span> <span class="s2">&#34;333-444-5555&#34;</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;carl&#34;</span><span class="p">,</span> <span class="nx">phone</span><span class="o">:</span> <span class="s2">&#34;444-555-6666&#34;</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"><span class="p">];</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">emails</span> <span class="o">=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nx">withName</span><span class="p">(</span><span class="s2">&#34;createEmails&#34;</span><span class="p">,</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">(</span><span class="nx">emails_list</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"><span class="p">);</span> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">phones</span> <span class="o">=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nx">withName</span><span class="p">(</span><span class="s2">&#34;createPhones&#34;</span><span class="p">,</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">(</span><span class="nx">phones_list</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"><span class="p">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl">- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">Create</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">CreateEmails</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">elements</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- {<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;amy&#34;</span><span class="nt">, email</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;amy@example.com&#34;</span><span class="w"> </span>}<span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- {<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;carl&#34;</span><span class="nt">, email</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;carl@example.com&#34;</span><span class="w"> </span>}<span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- {<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;julia&#34;</span><span class="nt">, email</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;julia@example.com&#34;</span><span class="w"> </span>}<span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- {<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;carl&#34;</span><span class="nt">, email</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;carl@email.com&#34;</span><span class="w"> </span>}<span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">Create</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">CreatePhones</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">elements</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- {<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;amy&#34;</span><span class="nt">, phone</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;111-222-3333&#34;</span><span class="w"> </span>}<span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- {<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;james&#34;</span><span class="nt">, phone</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;222-333-4444&#34;</span><span class="w"> </span>}<span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- {<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;amy&#34;</span><span class="nt">, phone</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;333-444-5555&#34;</span><span class="w"> </span>}<span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- {<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;carl&#34;</span><span class="nt">, phone</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;444-555-6666&#34;</span><span class="w"> </span>}</span></span></code></pre></div> |
| </div> |
| </div> |
| <p>After <code>CoGroupByKey</code>, the resulting data contains all data associated with each |
| unique key from any of the input collections.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">final</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">emailsTag</span> <span class="o">=</span> <span class="k">new</span> <span class="n">TupleTag</span><span class="o">&lt;&gt;();</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">final</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">phonesTag</span> <span class="o">=</span> <span class="k">new</span> <span class="n">TupleTag</span><span class="o">&lt;&gt;();</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">final</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">CoGbkResult</span><span class="o">&gt;&gt;</span> <span class="n">expectedResults</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;amy&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CoGbkResult</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">emailsTag</span><span class="o">,</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="s">&#34;amy@example.com&#34;</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">and</span><span class="o">(</span><span class="n">phonesTag</span><span class="o">,</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="s">&#34;111-222-3333&#34;</span><span class="o">,</span> <span class="s">&#34;333-444-5555&#34;</span><span class="o">))),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;carl&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CoGbkResult</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">emailsTag</span><span class="o">,</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="s">&#34;carl@email.com&#34;</span><span class="o">,</span> <span class="s">&#34;carl@example.com&#34;</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">and</span><span class="o">(</span><span class="n">phonesTag</span><span class="o">,</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="s">&#34;444-555-6666&#34;</span><span class="o">))),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;james&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CoGbkResult</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">emailsTag</span><span class="o">,</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">and</span><span class="o">(</span><span class="n">phonesTag</span><span class="o">,</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="s">&#34;222-333-4444&#34;</span><span class="o">))),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;julia&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CoGbkResult</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">emailsTag</span><span class="o">,</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="s">&#34;julia@example.com&#34;</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">and</span><span class="o">(</span><span class="n">phonesTag</span><span class="o">,</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">())));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">results</span> <span class="o">=</span> <span class="p">[</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;amy&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;emails&#39;</span><span class="p">:</span> <span class="p">[</span><span class="s1">&#39;amy@example.com&#39;</span><span class="p">],</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;phones&#39;</span><span class="p">:</span> <span class="p">[</span><span class="s1">&#39;111-222-3333&#39;</span><span class="p">,</span> <span class="s1">&#39;333-444-5555&#39;</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;carl&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;emails&#39;</span><span class="p">:</span> <span class="p">[</span><span class="s1">&#39;carl@email.com&#39;</span><span class="p">,</span> <span class="s1">&#39;carl@example.com&#39;</span><span class="p">],</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;phones&#39;</span><span class="p">:</span> <span class="p">[</span><span class="s1">&#39;444-555-6666&#39;</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;james&#39;</span><span class="p">,</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;emails&#39;</span><span class="p">:</span> <span class="p">[],</span> <span class="s1">&#39;phones&#39;</span><span class="p">:</span> <span class="p">[</span><span class="s1">&#39;222-333-4444&#39;</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;julia&#39;</span><span class="p">,</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;emails&#39;</span><span class="p">:</span> <span class="p">[</span><span class="s1">&#39;julia@example.com&#39;</span><span class="p">],</span> <span class="s1">&#39;phones&#39;</span><span class="p">:</span> <span class="p">[]</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}),</span> |
| </span></span><span class="line"><span class="cl"><span class="p">]</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">results</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">CoGroupByKey</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">emails</span><span class="p">,</span> <span class="nx">phones</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="nx">contactLines</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">formatCoGBKResults</span><span class="p">,</span> <span class="nx">results</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Synthetic example results of a cogbk. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">results</span> <span class="o">:=</span> <span class="p">[]</span><span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Key</span> <span class="kt">string</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Emails</span><span class="p">,</span> <span class="nx">Phones</span> <span class="p">[]</span><span class="kt">string</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}{</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Key</span><span class="p">:</span> <span class="s">&#34;amy&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Emails</span><span class="p">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">&#34;amy@example.com&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Phones</span><span class="p">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">&#34;111-222-3333&#34;</span><span class="p">,</span> <span class="s">&#34;333-444-5555&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Key</span><span class="p">:</span> <span class="s">&#34;carl&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Emails</span><span class="p">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">&#34;carl@email.com&#34;</span><span class="p">,</span> <span class="s">&#34;carl@example.com&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Phones</span><span class="p">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">&#34;444-555-6666&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Key</span><span class="p">:</span> <span class="s">&#34;james&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Emails</span><span class="p">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{},</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Phones</span><span class="p">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">&#34;222-333-4444&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Key</span><span class="p">:</span> <span class="s">&#34;julia&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Emails</span><span class="p">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">&#34;julia@example.com&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Phones</span><span class="p">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">results</span> <span class="o">=</span> <span class="p">[</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;amy&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">values</span><span class="o">:</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">emails</span><span class="o">:</span> <span class="p">[{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;amy&#34;</span><span class="p">,</span> <span class="nx">email</span><span class="o">:</span> <span class="s2">&#34;amy@example.com&#34;</span> <span class="p">}],</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">phones</span><span class="o">:</span> <span class="p">[</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;amy&#34;</span><span class="p">,</span> <span class="nx">phone</span><span class="o">:</span> <span class="s2">&#34;111-222-3333&#34;</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;amy&#34;</span><span class="p">,</span> <span class="nx">phone</span><span class="o">:</span> <span class="s2">&#34;333-444-5555&#34;</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">],</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;carl&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">values</span><span class="o">:</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">emails</span><span class="o">:</span> <span class="p">[</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;carl&#34;</span><span class="p">,</span> <span class="nx">email</span><span class="o">:</span> <span class="s2">&#34;carl@example.com&#34;</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;carl&#34;</span><span class="p">,</span> <span class="nx">email</span><span class="o">:</span> <span class="s2">&#34;carl@email.com&#34;</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">],</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">phones</span><span class="o">:</span> <span class="p">[{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;carl&#34;</span><span class="p">,</span> <span class="nx">phone</span><span class="o">:</span> <span class="s2">&#34;444-555-6666&#34;</span> <span class="p">}],</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;james&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">values</span><span class="o">:</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">emails</span><span class="o">:</span> <span class="p">[],</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">phones</span><span class="o">:</span> <span class="p">[{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;james&#34;</span><span class="p">,</span> <span class="nx">phone</span><span class="o">:</span> <span class="s2">&#34;222-333-4444&#34;</span> <span class="p">}],</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;julia&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">values</span><span class="o">:</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">emails</span><span class="o">:</span> <span class="p">[{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">&#34;julia&#34;</span><span class="p">,</span> <span class="nx">email</span><span class="o">:</span> <span class="s2">&#34;julia@example.com&#34;</span> <span class="p">}],</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">phones</span><span class="o">:</span> <span class="p">[],</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"><span class="p">];</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java language-py language-typescript">The following code example joins the two <code>PCollection</code>s with <code>CoGroupByKey</code>, |
| followed by a <code>ParDo</code> to consume the result. Then, the code uses tags to look up |
| and format data from each collection.</p> |
| <p class="language-go">The following code example joins the two <code>PCollection</code>s with <code>CoGroupByKey</code>, |
| followed by a <code>ParDo</code> to consume the result. The ordering of the <code>DoFn</code> iterator |
| parameters maps to the ordering of the <code>CoGroupByKey</code> inputs.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">CoGbkResult</span><span class="o">&gt;&gt;</span> <span class="n">results</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KeyedPCollectionTuple</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">emailsTag</span><span class="o">,</span> <span class="n">emails</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">and</span><span class="o">(</span><span class="n">phonesTag</span><span class="o">,</span> <span class="n">phones</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">CoGroupByKey</span><span class="o">.</span><span class="na">create</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">contactLines</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">results</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">CoGbkResult</span><span class="o">&gt;,</span> <span class="n">String</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">CoGbkResult</span><span class="o">&gt;</span> <span class="n">e</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">String</span> <span class="n">name</span> <span class="o">=</span> <span class="n">e</span><span class="o">.</span><span class="na">getKey</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Iterable</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">emailsIter</span> <span class="o">=</span> <span class="n">e</span><span class="o">.</span><span class="na">getValue</span><span class="o">().</span><span class="na">getAll</span><span class="o">(</span><span class="n">emailsTag</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Iterable</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">phonesIter</span> <span class="o">=</span> <span class="n">e</span><span class="o">.</span><span class="na">getValue</span><span class="o">().</span><span class="na">getAll</span><span class="o">(</span><span class="n">phonesTag</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">String</span> <span class="n">formattedResult</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Snippets</span><span class="o">.</span><span class="na">formatCoGbkResults</span><span class="o">(</span><span class="n">name</span><span class="o">,</span> <span class="n">emailsIter</span><span class="o">,</span> <span class="n">phonesIter</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">c</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">formattedResult</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># The result PCollection contains one key-value element for each key in the</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># input PCollections. The key of the pair will be the key from the input and</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># the value will be a dictionary with two entries: &#39;emails&#39; - an iterable of</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># all values for the current key in the emails PCollection and &#39;phones&#39;: an</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># iterable of all values for the current key in the phones PCollection.</span> |
| </span></span><span class="line"><span class="cl"><span class="n">results</span> <span class="o">=</span> <span class="p">({</span><span class="s1">&#39;emails&#39;</span><span class="p">:</span> <span class="n">emails</span><span class="p">,</span> <span class="s1">&#39;phones&#39;</span><span class="p">:</span> <span class="n">phones</span><span class="p">}</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CoGroupByKey</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">join_info</span><span class="p">(</span><span class="n">name_info</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">info</span><span class="p">)</span> <span class="o">=</span> <span class="n">name_info</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="s1">&#39;</span><span class="si">%s</span><span class="s1">; </span><span class="si">%s</span><span class="s1">; </span><span class="si">%s</span><span class="s1">&#39;</span> <span class="o">%</span>\ |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">info</span><span class="p">[</span><span class="s1">&#39;emails&#39;</span><span class="p">]),</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">info</span><span class="p">[</span><span class="s1">&#39;phones&#39;</span><span class="p">]))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">contact_lines</span> <span class="o">=</span> <span class="n">results</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="n">join_info</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">formatCoGBKResults</span><span class="p">(</span><span class="nx">key</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emailIter</span><span class="p">,</span> <span class="nx">phoneIter</span> <span class="kd">func</span><span class="p">(</span><span class="o">*</span><span class="kt">string</span><span class="p">)</span> <span class="kt">bool</span><span class="p">)</span> <span class="kt">string</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">var</span> <span class="nx">s</span> <span class="kt">string</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">var</span> <span class="nx">emails</span><span class="p">,</span> <span class="nx">phones</span> <span class="p">[]</span><span class="kt">string</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="nf">emailIter</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">s</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">emails</span> <span class="p">=</span> <span class="nb">append</span><span class="p">(</span><span class="nx">emails</span><span class="p">,</span> <span class="nx">s</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="nf">phoneIter</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">s</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">phones</span> <span class="p">=</span> <span class="nb">append</span><span class="p">(</span><span class="nx">phones</span><span class="p">,</span> <span class="nx">s</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Values have no guaranteed order, sort for deterministic output. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">sort</span><span class="p">.</span><span class="nf">Strings</span><span class="p">(</span><span class="nx">emails</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">sort</span><span class="p">.</span><span class="nf">Strings</span><span class="p">(</span><span class="nx">phones</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">fmt</span><span class="p">.</span><span class="nf">Sprintf</span><span class="p">(</span><span class="s">&#34;%s; %s; %s&#34;</span><span class="p">,</span> <span class="nx">key</span><span class="p">,</span> <span class="nf">formatStringIter</span><span class="p">(</span><span class="nx">emails</span><span class="p">),</span> <span class="nf">formatStringIter</span><span class="p">(</span><span class="nx">phones</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">register</span><span class="p">.</span><span class="nf">Function3x1</span><span class="p">(</span><span class="nx">formatCoGBKResults</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// 1 input of type string =&gt; Iter1[string] |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">register</span><span class="p">.</span><span class="nx">Iter1</span><span class="p">[</span><span class="kt">string</span><span class="p">]()</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Synthetic example results of a cogbk. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">results</span> <span class="o">:=</span> <span class="p">[]</span><span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Key</span> <span class="kt">string</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Emails</span><span class="p">,</span> <span class="nx">Phones</span> <span class="p">[]</span><span class="kt">string</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}{</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Key</span><span class="p">:</span> <span class="s">&#34;amy&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Emails</span><span class="p">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">&#34;amy@example.com&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Phones</span><span class="p">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">&#34;111-222-3333&#34;</span><span class="p">,</span> <span class="s">&#34;333-444-5555&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Key</span><span class="p">:</span> <span class="s">&#34;carl&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Emails</span><span class="p">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">&#34;carl@email.com&#34;</span><span class="p">,</span> <span class="s">&#34;carl@example.com&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Phones</span><span class="p">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">&#34;444-555-6666&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Key</span><span class="p">:</span> <span class="s">&#34;james&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Emails</span><span class="p">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{},</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Phones</span><span class="p">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">&#34;222-333-4444&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Key</span><span class="p">:</span> <span class="s">&#34;julia&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Emails</span><span class="p">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">&#34;julia@example.com&#34;</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Phones</span><span class="p">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">formatted_results_pcoll</span> <span class="o">=</span> <span class="nx">beam</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">P</span><span class="p">({</span> <span class="nx">emails</span><span class="p">,</span> <span class="nx">phones</span> <span class="p">})</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">coGroupBy</span><span class="p">(</span><span class="s2">&#34;name&#34;</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="kd">function</span> <span class="nx">formatResults</span><span class="p">({</span> <span class="nx">key</span><span class="p">,</span> <span class="nx">values</span> <span class="p">})</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kr">const</span> <span class="nx">emails</span> <span class="o">=</span> <span class="nx">values</span><span class="p">.</span><span class="nx">emails</span><span class="p">.</span><span class="nx">map</span><span class="p">((</span><span class="nx">x</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">x</span><span class="p">.</span><span class="nx">email</span><span class="p">).</span><span class="nx">sort</span><span class="p">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="kr">const</span> <span class="nx">phones</span> <span class="o">=</span> <span class="nx">values</span><span class="p">.</span><span class="nx">phones</span><span class="p">.</span><span class="nx">map</span><span class="p">((</span><span class="nx">x</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">x</span><span class="p">.</span><span class="nx">phone</span><span class="p">).</span><span class="nx">sort</span><span class="p">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="sb">`</span><span class="si">${</span><span class="nx">key</span><span class="si">}</span><span class="sb">; [</span><span class="si">${</span><span class="nx">emails</span><span class="si">}</span><span class="sb">]; [</span><span class="si">${</span><span class="nx">phones</span><span class="si">}</span><span class="sb">]`</span><span class="p">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">});</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl">- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">MapToFields</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">PrepareEmails</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">input</span><span class="p">:</span><span class="w"> </span><span class="l">CreateEmails</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">language</span><span class="p">:</span><span class="w"> </span><span class="l">python</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">fields</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">name</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">email</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;[email]&#34;</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">phone</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;[]&#34;</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">MapToFields</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">PreparePhones</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">input</span><span class="p">:</span><span class="w"> </span><span class="l">CreatePhones</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">language</span><span class="p">:</span><span class="w"> </span><span class="l">python</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">fields</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">name</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">email</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;[]&#34;</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">phone</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;[phone]&#34;</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">Combine</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">CoGropuBy</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">input</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">PrepareEmails, PreparePhones]</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">group_by</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">name]</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">combine</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">email</span><span class="p">:</span><span class="w"> </span><span class="l">concat</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">phone</span><span class="p">:</span><span class="w"> </span><span class="l">concat</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">MapToFields</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">FormatResults</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">input</span><span class="p">:</span><span class="w"> </span><span class="l">CoGropuBy</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">language</span><span class="p">:</span><span class="w"> </span><span class="l">python</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">fields</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">formatted</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="s2">&#34;&#39;%s; %s; %s&#39; % (name, sorted(email), sorted(phone))&#34;</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>The formatted data looks like this:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">final</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">formattedResults</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;amy; [&#39;amy@example.com&#39;]; [&#39;111-222-3333&#39;, &#39;333-444-5555&#39;]&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;carl; [&#39;carl@email.com&#39;, &#39;carl@example.com&#39;]; [&#39;444-555-6666&#39;]&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;james; []; [&#39;222-333-4444&#39;]&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;julia; [&#39;julia@example.com&#39;]; []&#34;</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">formatted_results</span> <span class="o">=</span> <span class="p">[</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;amy; [&#39;amy@example.com&#39;]; [&#39;111-222-3333&#39;, &#39;333-444-5555&#39;]&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;carl; [&#39;carl@email.com&#39;, &#39;carl@example.com&#39;]; [&#39;444-555-6666&#39;]&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;james; []; [&#39;222-333-4444&#39;]&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;julia; [&#39;julia@example.com&#39;]; []&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"><span class="p">]</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">formattedResults</span> <span class="o">:=</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;amy; [&#39;amy@example.com&#39;]; [&#39;111-222-3333&#39;, &#39;333-444-5555&#39;]&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;carl; [&#39;carl@email.com&#39;, &#39;carl@example.com&#39;]; [&#39;444-555-6666&#39;]&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;james; []; [&#39;222-333-4444&#39;]&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;julia; [&#39;julia@example.com&#39;]; []&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">formatted_results</span> <span class="o">=</span> <span class="p">[</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;amy; [amy@example.com]; [111-222-3333,333-444-5555]&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;carl; [carl@email.com,carl@example.com]; [444-555-6666]&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;james; []; [222-333-4444]&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;julia; [julia@example.com]; []&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"><span class="p">];</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="s2">&#34;amy; [&#39;amy@example.com&#39;]; [&#39;111-222-3333&#39;, &#39;333-444-5555&#39;]&#34;</span><span class="p">,</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="s2">&#34;carl; [&#39;carl@email.com&#39;, &#39;carl@example.com&#39;]; [&#39;444-555-6666&#39;]&#34;</span><span class="p">,</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="s2">&#34;james; []; [&#39;222-333-4444&#39;]&#34;</span><span class="p">,</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="s2">&#34;julia; [&#39;julia@example.com&#39;]; []&#34;</span><span class="p">,</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="combine">4.2.4. Combine</h4> |
| <p><span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.56.0/index.html?org/apache/beam/sdk/transforms/Combine.html"><code>Combine</code></a></span> |
| <span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py"><code>Combine</code></a></span> |
| <span class="language-go"><a href="https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/combine.go#L27"><code>Combine</code></a></span> |
| <span class="language-typescript"><a href="https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/transforms/group_and_combine.ts"><code>Combine</code></a></span> |
| is a Beam transform for combining collections of elements or values in your |
| data. <code>Combine</code> has variants that work on entire <code>PCollection</code>s, and some that |
| combine the values for each key in <code>PCollection</code>s of key/value pairs.</p> |
| <p>When you apply a <code>Combine</code> transform, you must provide the function that |
| contains the logic for combining the elements or values. The combining function |
| should be commutative and associative, as the function is not necessarily |
| invoked exactly once on all values with a given key. Because the input data |
| (including the value collection) may be distributed across multiple workers, the |
| combining function might be called multiple times to perform partial combining |
| on subsets of the value collection. The Beam SDK also provides some pre-built |
| combine functions for common numeric combination operations such as sum, min, |
| and max.</p> |
| <p>Simple combine operations, such as sums, can usually be implemented as a simple |
| function. More complex combination operations might require you to create a |
| <span class="language-java language-py">subclass of</span> <code>CombineFn</code> |
| that has an accumulation type distinct from the input/output type.</p> |
| <p>The associativity and commutativity of a <code>CombineFn</code> allows runners to |
| automatically apply some optimizations:</p> |
| <ul> |
| <li><strong>Combiner lifting</strong>: This is the most significant optimization. Input |
| elements are combined per key and window before they are shuffled, so the |
| volume of data shuffled might be reduced by many orders of magnitude. Another |
| term for this optimization is &ldquo;mapper-side combine.&rdquo;</li> |
| <li><strong>Incremental combining</strong>: When you have a <code>CombineFn</code> that reduces the data |
| size by a lot, it is useful to combine elements as they emerge from a |
| streaming shuffle. This spreads out the cost of doing combines over the time |
| that your streaming computation might be idle. Incremental combining also |
| reduces the storage of intermediate accumulators.</li> |
| </ul> |
| <h5 id="simple-combines">4.2.4.1. Simple combinations using simple functions</h5> |
| <span class="language-yaml"> |
| Beam YAML has the following buit-in CombineFns: count, sum, min, max, |
| mean, any, all, group, and concat. |
| CombineFns from other languages can also be referenced |
| as described in the (full docs on aggregation)[https://beam.apache.org/documentation/sdks/yaml-combine/]. |
| </span> |
| The following example code shows a simple combine function. |
| <span class="language-typescript"> |
| Combining is done by modifying a grouping transform with the `combining` method. |
| This method takes three parameters: the value to combine (either as a named |
| property of the input elements, or a function of the entire input), |
| the combining operation (either a binary function or a `CombineFn`), |
| and finally a name for the combined value in the output object. |
| </span> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Sum a collection of Integer values. The function SumInts implements the interface SerializableFunction. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">SumInts</span> <span class="kd">implements</span> <span class="n">SerializableFunction</span><span class="o">&lt;</span><span class="n">Iterable</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;,</span> <span class="n">Integer</span><span class="o">&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">Integer</span> <span class="nf">apply</span><span class="o">(</span><span class="n">Iterable</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">input</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kt">int</span> <span class="n">sum</span> <span class="o">=</span> <span class="n">0</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">item</span> <span class="o">:</span> <span class="n">input</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">sum</span> <span class="o">+=</span> <span class="n">item</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">sum</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">pc</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">100</span><span class="p">,</span> <span class="mi">1000</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">bounded_sum</span><span class="p">(</span><span class="n">values</span><span class="p">,</span> <span class="n">bound</span><span class="o">=</span><span class="mi">500</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nb">min</span><span class="p">(</span><span class="nb">sum</span><span class="p">(</span><span class="n">values</span><span class="p">),</span> <span class="n">bound</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">small_sum</span> <span class="o">=</span> <span class="n">pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span><span class="n">bounded_sum</span><span class="p">)</span> <span class="c1"># [500]</span> |
| </span></span><span class="line"><span class="cl"><span class="n">large_sum</span> <span class="o">=</span> <span class="n">pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span><span class="n">bounded_sum</span><span class="p">,</span> <span class="n">bound</span><span class="o">=</span><span class="mi">5000</span><span class="p">)</span> <span class="c1"># [1111]</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">sumInts</span><span class="p">(</span><span class="nx">a</span><span class="p">,</span> <span class="nx">v</span> <span class="kt">int</span><span class="p">)</span> <span class="kt">int</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">a</span> <span class="o">+</span> <span class="nx">v</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">register</span><span class="p">.</span><span class="nf">Function2x1</span><span class="p">(</span><span class="nx">sumInts</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">globallySumInts</span><span class="p">(</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">ints</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span><span class="p">)</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">Combine</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">sumInts</span><span class="p">,</span> <span class="nx">ints</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">boundedSum</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Bound</span> <span class="kt">int</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">boundedSum</span><span class="p">)</span> <span class="nf">MergeAccumulators</span><span class="p">(</span><span class="nx">a</span><span class="p">,</span> <span class="nx">v</span> <span class="kt">int</span><span class="p">)</span> <span class="kt">int</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">sum</span> <span class="o">:=</span> <span class="nx">a</span> <span class="o">+</span> <span class="nx">v</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">Bound</span> <span class="p">&gt;</span> <span class="mi">0</span> <span class="o">&amp;&amp;</span> <span class="nx">sum</span> <span class="p">&gt;</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">Bound</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">Bound</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">sum</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">register</span><span class="p">.</span><span class="nx">Combiner1</span><span class="p">[</span><span class="kt">int</span><span class="p">](</span><span class="o">&amp;</span><span class="nx">boundedSum</span><span class="p">{})</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">globallyBoundedSumInts</span><span class="p">(</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">bound</span> <span class="kt">int</span><span class="p">,</span> <span class="nx">ints</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span><span class="p">)</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">Combine</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">boundedSum</span><span class="p">{</span><span class="nx">Bound</span><span class="p">:</span> <span class="nx">bound</span><span class="p">},</span> <span class="nx">ints</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">pcoll</span> <span class="o">=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">100</span><span class="p">,</span> <span class="mi">1000</span><span class="p">]));</span> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">pcoll</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">groupGlobally</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">combining</span><span class="p">((</span><span class="nx">c</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">c</span><span class="p">,</span> <span class="p">(</span><span class="nx">x</span><span class="p">,</span> <span class="nx">y</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">x</span> <span class="o">+</span> <span class="nx">y</span><span class="p">,</span> <span class="s2">&#34;sum&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">combining</span><span class="p">((</span><span class="nx">c</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">c</span><span class="p">,</span> <span class="p">(</span><span class="nx">x</span><span class="p">,</span> <span class="nx">y</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">x</span> <span class="o">*</span> <span class="nx">y</span><span class="p">,</span> <span class="s2">&#34;product&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">);</span> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">expected</span> <span class="o">=</span> <span class="p">{</span> <span class="nx">sum</span>: <span class="kt">1111</span><span class="p">,</span> <span class="nx">product</span>: <span class="kt">1000000</span> <span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">Combine</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">language</span><span class="p">:</span><span class="w"> </span><span class="l">python</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">group_by</span><span class="p">:</span><span class="w"> </span><span class="l">animal</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">combine</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">biggest</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">fn</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;apache_beam.transforms.combiners.TopCombineFn&#39;</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">n</span><span class="p">:</span><span class="w"> </span><span class="m">2</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">value</span><span class="p">:</span><span class="w"> </span><span class="l">weight</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-go">All Combiners should be registered using a generic <code>register.CombinerX[...]</code> |
| function. This allows the Go SDK to infer an encoding from any inputs/outputs, |
| registers the Combiner for execution on remote runners, and optimizes the runtime |
| execution of the Combiner via reflection.</p> |
| <p class="language-go">Combiner1 should be used when your accumulator, input, and output are all of the |
| same type. It can be called with <code>register.Combiner1[T](&amp;CustomCombiner{})</code> where <code>T</code> |
| is the type of the input/accumulator/output.</p> |
| <p class="language-go">Combiner2 should be used when your accumulator, input, and output are 2 distinct |
| types. It can be called with <code>register.Combiner2[T1, T2](&amp;CustomCombiner{})</code> where |
| <code>T1</code> is the type of the accumulator and <code>T2</code> is the other type.</p> |
| <p class="language-go">Combiner3 should be used when your accumulator, input, and output are 3 distinct |
| types. It can be called with <code>register.Combiner3[T1, T2, T3](&amp;CustomCombiner{})</code> |
| where <code>T1</code> is the type of the accumulator, <code>T2</code> is the type of the input, and <code>T3</code> is |
| the type of the output.</p> |
| <h5 id="advanced-combines">4.2.4.2. Advanced combinations using CombineFn</h5> |
| <p>For more complex combine functions, you can define a |
| <span class="language-java language-py">subclass of</span><code>CombineFn</code>. |
| You should use a <code>CombineFn</code> if the combine function requires a more sophisticated |
| accumulator, must perform additional pre- or post-processing, might change the |
| output type, or takes the key into account.</p> |
| <p>A general combining operation consists of five operations. When you create a |
| <span class="language-java language-py">subclass of</span> |
| <code>CombineFn</code>, you must provide five operations by overriding the |
| corresponding methods. Only <code>MergeAccumulators</code> is a required method. The |
| others will have a default interpretation based on the accumulator type. The |
| lifecycle methods are:</p> |
| <ol> |
| <li> |
| <p><strong>Create Accumulator</strong> creates a new &ldquo;local&rdquo; accumulator. In the example |
| case, taking a mean average, a local accumulator tracks the running sum of |
| values (the numerator value for our final average division) and the number of |
| values summed so far (the denominator value). It may be called any number of |
| times in a distributed fashion.</p> |
| </li> |
| <li> |
| <p><strong>Add Input</strong> adds an input element to an accumulator, returning the |
| accumulator value. In our example, it would update the sum and increment the |
| count. It may also be invoked in parallel.</p> |
| </li> |
| <li> |
| <p><strong>Merge Accumulators</strong> merges several accumulators into a single accumulator; |
| this is how data in multiple accumulators is combined before the final |
| calculation. In the case of the mean average computation, the accumulators |
| representing each portion of the division are merged together. It may be |
| called again on its outputs any number of times.</p> |
| </li> |
| <li> |
| <p><strong>Extract Output</strong> performs the final computation. In the case of computing a |
| mean average, this means dividing the combined sum of all the values by the |
| number of values summed. It is called once on the final, merged accumulator.</p> |
| </li> |
| <li> |
| <p><strong>Compact</strong> returns a more compact represenation of the accumulator. This is |
| called before an accumulator is sent across the wire, and can be useful in |
| cases where values are buffered or otherwise lazily kept unprocessed when |
| added to the accumulator. Compact should return an equivalent, though |
| possibly modified, accumulator. In most cases, Compact is not necessary. For |
| a real world example of using Compact, see the Python SDK implementation of |
| <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/combiners.py#L523">TopCombineFn</a></p> |
| </li> |
| </ol> |
| <p>The following example code shows how to define a <code>CombineFn</code> that computes a |
| mean average:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">AverageFn</span> <span class="kd">extends</span> <span class="n">CombineFn</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">,</span> <span class="n">AverageFn</span><span class="o">.</span><span class="na">Accum</span><span class="o">,</span> <span class="n">Double</span><span class="o">&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">Accum</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kt">int</span> <span class="n">sum</span> <span class="o">=</span> <span class="n">0</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="kt">int</span> <span class="n">count</span> <span class="o">=</span> <span class="n">0</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">Accum</span> <span class="nf">createAccumulator</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="k">new</span> <span class="n">Accum</span><span class="o">();</span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">Accum</span> <span class="nf">addInput</span><span class="o">(</span><span class="n">Accum</span> <span class="n">accum</span><span class="o">,</span> <span class="n">Integer</span> <span class="n">input</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">accum</span><span class="o">.</span><span class="na">sum</span> <span class="o">+=</span> <span class="n">input</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">accum</span><span class="o">.</span><span class="na">count</span><span class="o">++;</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">accum</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">Accum</span> <span class="nf">mergeAccumulators</span><span class="o">(</span><span class="n">Iterable</span><span class="o">&lt;</span><span class="n">Accum</span><span class="o">&gt;</span> <span class="n">accums</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Accum</span> <span class="n">merged</span> <span class="o">=</span> <span class="n">createAccumulator</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="o">(</span><span class="n">Accum</span> <span class="n">accum</span> <span class="o">:</span> <span class="n">accums</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">merged</span><span class="o">.</span><span class="na">sum</span> <span class="o">+=</span> <span class="n">accum</span><span class="o">.</span><span class="na">sum</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">merged</span><span class="o">.</span><span class="na">count</span> <span class="o">+=</span> <span class="n">accum</span><span class="o">.</span><span class="na">count</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">merged</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">Double</span> <span class="nf">extractOutput</span><span class="o">(</span><span class="n">Accum</span> <span class="n">accum</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="o">((</span><span class="kt">double</span><span class="o">)</span> <span class="n">accum</span><span class="o">.</span><span class="na">sum</span><span class="o">)</span> <span class="o">/</span> <span class="n">accum</span><span class="o">.</span><span class="na">count</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// No-op |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">Accum</span> <span class="nf">compact</span><span class="o">(</span><span class="n">Accum</span> <span class="n">accum</span><span class="o">)</span> <span class="o">{</span> <span class="k">return</span> <span class="n">accum</span><span class="o">;</span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">pc</span> <span class="o">=</span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">AverageFn</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">CombineFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">create_accumulator</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">add_input</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sum_count</span><span class="p">,</span> <span class="nb">input</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="nb">sum</span><span class="p">,</span> <span class="n">count</span><span class="p">)</span> <span class="o">=</span> <span class="n">sum_count</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nb">sum</span> <span class="o">+</span> <span class="nb">input</span><span class="p">,</span> <span class="n">count</span> <span class="o">+</span> <span class="mi">1</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">merge_accumulators</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">accumulators</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">sums</span><span class="p">,</span> <span class="n">counts</span> <span class="o">=</span> <span class="nb">zip</span><span class="p">(</span><span class="o">*</span><span class="n">accumulators</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nb">sum</span><span class="p">(</span><span class="n">sums</span><span class="p">),</span> <span class="nb">sum</span><span class="p">(</span><span class="n">counts</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">extract_output</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sum_count</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="nb">sum</span><span class="p">,</span> <span class="n">count</span><span class="p">)</span> <span class="o">=</span> <span class="n">sum_count</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nb">sum</span> <span class="o">/</span> <span class="n">count</span> <span class="k">if</span> <span class="n">count</span> <span class="k">else</span> <span class="nb">float</span><span class="p">(</span><span class="s1">&#39;NaN&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">compact</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">accumulator</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># No-op</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">accumulator</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">averageFn</span> <span class="kd">struct</span><span class="p">{}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">averageAccum</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Count</span><span class="p">,</span> <span class="nx">Sum</span> <span class="kt">int</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">averageFn</span><span class="p">)</span> <span class="nf">CreateAccumulator</span><span class="p">()</span> <span class="nx">averageAccum</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">averageAccum</span><span class="p">{</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">averageFn</span><span class="p">)</span> <span class="nf">AddInput</span><span class="p">(</span><span class="nx">a</span> <span class="nx">averageAccum</span><span class="p">,</span> <span class="nx">v</span> <span class="kt">int</span><span class="p">)</span> <span class="nx">averageAccum</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">averageAccum</span><span class="p">{</span><span class="nx">Count</span><span class="p">:</span> <span class="nx">a</span><span class="p">.</span><span class="nx">Count</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span> <span class="nx">Sum</span><span class="p">:</span> <span class="nx">a</span><span class="p">.</span><span class="nx">Sum</span> <span class="o">+</span> <span class="nx">v</span><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">averageFn</span><span class="p">)</span> <span class="nf">MergeAccumulators</span><span class="p">(</span><span class="nx">a</span><span class="p">,</span> <span class="nx">v</span> <span class="nx">averageAccum</span><span class="p">)</span> <span class="nx">averageAccum</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">averageAccum</span><span class="p">{</span><span class="nx">Count</span><span class="p">:</span> <span class="nx">a</span><span class="p">.</span><span class="nx">Count</span> <span class="o">+</span> <span class="nx">v</span><span class="p">.</span><span class="nx">Count</span><span class="p">,</span> <span class="nx">Sum</span><span class="p">:</span> <span class="nx">a</span><span class="p">.</span><span class="nx">Sum</span> <span class="o">+</span> <span class="nx">v</span><span class="p">.</span><span class="nx">Sum</span><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">averageFn</span><span class="p">)</span> <span class="nf">ExtractOutput</span><span class="p">(</span><span class="nx">a</span> <span class="nx">averageAccum</span><span class="p">)</span> <span class="kt">float64</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">a</span><span class="p">.</span><span class="nx">Count</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">math</span><span class="p">.</span><span class="nf">NaN</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nb">float64</span><span class="p">(</span><span class="nx">a</span><span class="p">.</span><span class="nx">Sum</span><span class="p">)</span> <span class="o">/</span> <span class="nb">float64</span><span class="p">(</span><span class="nx">a</span><span class="p">.</span><span class="nx">Count</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">averageFn</span><span class="p">)</span> <span class="nf">Compact</span><span class="p">(</span><span class="nx">a</span> <span class="nx">averageAccum</span><span class="p">)</span> <span class="nx">averageAccum</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// No-op |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="nx">a</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">register</span><span class="p">.</span><span class="nx">Combiner3</span><span class="p">[</span><span class="nx">averageAccum</span><span class="p">,</span> <span class="kt">int</span><span class="p">,</span> <span class="kt">float64</span><span class="p">](</span><span class="o">&amp;</span><span class="nx">averageFn</span><span class="p">{})</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">meanCombineFn</span>: <span class="kt">beam.CombineFn</span><span class="p">&lt;</span><span class="nt">number</span><span class="err">,</span> <span class="err">[</span><span class="na">number</span><span class="err">,</span> <span class="na">number</span><span class="err">],</span> <span class="na">number</span><span class="p">&gt;</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">createAccumulator</span><span class="o">:</span> <span class="p">()</span> <span class="o">=&gt;</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">addInput</span><span class="o">:</span> <span class="p">([</span><span class="nx">sum</span><span class="p">,</span> <span class="nx">count</span><span class="p">]</span><span class="o">:</span> <span class="p">[</span><span class="kt">number</span><span class="p">,</span> <span class="kt">number</span><span class="p">],</span> <span class="nx">i</span>: <span class="kt">number</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">[</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">sum</span> <span class="o">+</span> <span class="nx">i</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">count</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">],</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">mergeAccumulators</span><span class="o">:</span> <span class="p">(</span><span class="nx">accumulators</span><span class="o">:</span> <span class="p">[</span><span class="kt">number</span><span class="p">,</span> <span class="kt">number</span><span class="p">][])</span> <span class="o">=&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">accumulators</span><span class="p">.</span><span class="nx">reduce</span><span class="p">(([</span><span class="nx">sum0</span><span class="p">,</span> <span class="nx">count0</span><span class="p">],</span> <span class="p">[</span><span class="nx">sum1</span><span class="p">,</span> <span class="nx">count1</span><span class="p">])</span> <span class="o">=&gt;</span> <span class="p">[</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">sum0</span> <span class="o">+</span> <span class="nx">sum1</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">count0</span> <span class="o">+</span> <span class="nx">count1</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">]),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">extractOutput</span><span class="o">:</span> <span class="p">([</span><span class="nx">sum</span><span class="p">,</span> <span class="nx">count</span><span class="p">]</span><span class="o">:</span> <span class="p">[</span><span class="kt">number</span><span class="p">,</span> <span class="kt">number</span><span class="p">])</span> <span class="o">=&gt;</span> <span class="nx">sum</span> <span class="o">/</span> <span class="nx">count</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">};</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <span class="language-go"> |
| </span> |
| <h5 id="combining-pcollection">4.2.4.3. Combining a PCollection into a single value</h5> |
| <p>Use the global combine to transform all of the elements in a given <code>PCollection</code> |
| into a single value, represented in your pipeline as a new <code>PCollection</code> |
| containing one element. The following example code shows how to apply the Beam |
| provided sum combine function to produce a single sum value for a <code>PCollection</code> |
| of integers.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Sum.SumIntegerFn() combines the elements in the input PCollection. The resulting PCollection, called sum, |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// contains one value: the sum of all the elements in the input PCollection. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">pc</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">sum</span> <span class="o">=</span> <span class="n">pc</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Combine</span><span class="o">.</span><span class="na">globally</span><span class="o">(</span><span class="k">new</span> <span class="n">Sum</span><span class="o">.</span><span class="na">SumIntegerFn</span><span class="o">()));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># sum combines the elements in the input PCollection.</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># The resulting PCollection, called result, contains one value: the sum of all</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># the elements in the input PCollection.</span> |
| </span></span><span class="line"><span class="cl"><span class="n">pc</span> <span class="o">=</span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">average</span> <span class="o">=</span> <span class="n">pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span><span class="n">AverageFn</span><span class="p">())</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">average</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">Combine</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">averageFn</span><span class="p">{},</span> <span class="nx">ints</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">pcoll</span> <span class="o">=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">([</span><span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">6</span><span class="p">]));</span> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">pcoll</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nx">groupGlobally</span><span class="p">().</span><span class="nx">combining</span><span class="p">((</span><span class="nx">c</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">c</span><span class="p">,</span> <span class="nx">meanCombineFn</span><span class="p">,</span> <span class="s2">&#34;mean&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">Combine</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">group_by</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">combine</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">weight</span><span class="p">:</span><span class="w"> </span><span class="l">sum</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h5 id="combine-global-windowing">4.2.4.4. Combine and global windowing</h5> |
| <p>If your input <code>PCollection</code> uses the default global windowing, the default |
| behavior is to return a <code>PCollection</code> containing one item. That item&rsquo;s value |
| comes from the accumulator in the combine function that you specified when |
| applying <code>Combine</code>. For example, the Beam provided sum combine function returns |
| a zero value (the sum of an empty input), while the min combine function returns |
| a maximal or infinite value.</p> |
| <p>To have <code>Combine</code> instead return an empty <code>PCollection</code> if the input is empty, |
| specify <code>.withoutDefaults</code> when you apply your <code>Combine</code> transform, as in the |
| following code example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">pc</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">sum</span> <span class="o">=</span> <span class="n">pc</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Combine</span><span class="o">.</span><span class="na">globally</span><span class="o">(</span><span class="k">new</span> <span class="n">Sum</span><span class="o">.</span><span class="na">SumIntegerFn</span><span class="o">()).</span><span class="na">withoutDefaults</span><span class="o">());</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">pc</span> <span class="o">=</span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="nb">sum</span> <span class="o">=</span> <span class="n">pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span><span class="nb">sum</span><span class="p">)</span><span class="o">.</span><span class="n">without_defaults</span><span class="p">()</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">returnSideOrDefault</span><span class="p">(</span><span class="nx">d</span> <span class="kt">float64</span><span class="p">,</span> <span class="nx">iter</span> <span class="kd">func</span><span class="p">(</span><span class="o">*</span><span class="kt">float64</span><span class="p">)</span> <span class="kt">bool</span><span class="p">)</span> <span class="kt">float64</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">var</span> <span class="nx">c</span> <span class="kt">float64</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nf">iter</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">c</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Side input has a value, so return it. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="nx">c</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Otherwise, return the default |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="nx">d</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> <span class="nx">register</span><span class="p">.</span><span class="nf">Function2x1</span><span class="p">(</span><span class="nx">returnSideOrDefault</span><span class="p">)</span> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">globallyAverageWithDefault</span><span class="p">(</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">ints</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span><span class="p">)</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Setting combine defaults has requires no helper function in the Go SDK. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">average</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">Combine</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">averageFn</span><span class="p">{},</span> <span class="nx">ints</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// To add a default value: |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">defaultValue</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">Create</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nb">float64</span><span class="p">(</span><span class="mi">0</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">returnSideOrDefault</span><span class="p">,</span> <span class="nx">defaultValue</span><span class="p">,</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">SideInput</span><span class="p">{</span><span class="nx">Input</span><span class="p">:</span> <span class="nx">average</span><span class="p">})</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">pcoll</span> <span class="o">=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">([</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;alice&#34;</span><span class="p">,</span> <span class="nx">accuracy</span>: <span class="kt">1.0</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;bob&#34;</span><span class="p">,</span> <span class="nx">accuracy</span>: <span class="kt">0.99</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;eve&#34;</span><span class="p">,</span> <span class="nx">accuracy</span>: <span class="kt">0.5</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;eve&#34;</span><span class="p">,</span> <span class="nx">accuracy</span>: <span class="kt">0.25</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">])</span> |
| </span></span><span class="line"><span class="cl"><span class="p">);</span> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">pcoll</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">groupGlobally</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">combining</span><span class="p">(</span><span class="s2">&#34;accuracy&#34;</span><span class="p">,</span> <span class="nx">combiners</span><span class="p">.</span><span class="nx">mean</span><span class="p">,</span> <span class="s2">&#34;mean&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">combining</span><span class="p">(</span><span class="s2">&#34;accuracy&#34;</span><span class="p">,</span> <span class="nx">combiners</span><span class="p">.</span><span class="nx">max</span><span class="p">,</span> <span class="s2">&#34;max&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">);</span> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">expected</span> <span class="o">=</span> <span class="p">[{</span> <span class="nx">max</span>: <span class="kt">1.0</span><span class="p">,</span> <span class="nx">mean</span>: <span class="kt">0.685</span> <span class="p">}];</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h5 id="combine-non-global-windowing">4.2.4.5. Combine and non-global windowing</h5> |
| <span class="language-java language-py"> |
| <p>If your <code>PCollection</code> uses any non-global windowing function, Beam does not |
| provide the default behavior. You must specify one of the following options when |
| applying <code>Combine</code>:</p> |
| <ul> |
| <li>Specify <code>.withoutDefaults</code>, where windows that are empty in the input |
| <code>PCollection</code> will likewise be empty in the output collection.</li> |
| <li>Specify <code>.asSingletonView</code>, in which the output is immediately converted to a |
| <code>PCollectionView</code>, which will provide a default value for each empty window |
| when used as a side input. You&rsquo;ll generally only need to use this option if |
| the result of your pipeline&rsquo;s <code>Combine</code> is to be used as a side input later in |
| the pipeline.</li> |
| </ul> |
| </span> |
| <p class="language-go">If your <code>PCollection</code> uses any non-global windowing function, the Beam Go SDK |
| behaves the same way as with global windowing. Windows that are empty in the input |
| <code>PCollection</code> will likewise be empty in the output collection.</p> |
| <h5 id="combining-values-in-a-keyed-pcollection">4.2.4.6. Combining values in a keyed PCollection</h5> |
| <p>After creating a keyed PCollection (for example, by using a <code>GroupByKey</code> |
| transform), a common pattern is to combine the collection of values associated |
| with each key into a single, merged value. Drawing on the previous example from |
| <code>GroupByKey</code>, a key-grouped <code>PCollection</code> called <code>groupedWords</code> looks like this:</p> |
| <pre tabindex="0"><code> cat, [1,5,9] |
| dog, [5,2] |
| and, [1,2,6] |
| jump, [3] |
| tree, [2] |
| ... |
| </code></pre><p>In the above <code>PCollection</code>, each element has a string key (for example, &ldquo;cat&rdquo;) |
| and an iterable of integers for its value (in the first element, containing [1, |
| 5, 9]). If our pipeline&rsquo;s next processing step combines the values (rather than |
| considering them individually), you can combine the iterable of integers to |
| create a single, merged value to be paired with each key. This pattern of a |
| <code>GroupByKey</code> followed by merging the collection of values is equivalent to |
| Beam&rsquo;s Combine PerKey transform. The combine function you supply to Combine |
| PerKey must be an associative reduction function or a |
| <span class="language-java language-py">subclass of</span> <code>CombineFn</code>.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// PCollection is grouped by key and the Double values associated with each key are combined into a Double. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Double</span><span class="o">&gt;&gt;</span> <span class="n">salesRecords</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Double</span><span class="o">&gt;&gt;</span> <span class="n">totalSalesPerPerson</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">salesRecords</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Combine</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Double</span><span class="o">,</span> <span class="n">Double</span><span class="o">&gt;</span><span class="n">perKey</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">Sum</span><span class="o">.</span><span class="na">SumDoubleFn</span><span class="o">()));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// The combined value is of a different type than the original collection of values per key. PCollection has |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// keys of type String and values of type Integer, and the combined value is a Double. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">playerAccuracy</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Double</span><span class="o">&gt;&gt;</span> <span class="n">avgAccuracyPerPlayer</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">playerAccuracy</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Combine</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">,</span> <span class="n">Double</span><span class="o">&gt;</span><span class="n">perKey</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">MeanInts</span><span class="o">())));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># PCollection is grouped by key and the numeric values associated with each key</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># are averaged into a float.</span> |
| </span></span><span class="line"><span class="cl"><span class="n">player_accuracies</span> <span class="o">=</span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">avg_accuracy_per_player</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">player_accuracies</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombinePerKey</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">combiners</span><span class="o">.</span><span class="n">MeanCombineFn</span><span class="p">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// PCollection is grouped by key and the numeric values associated with each key |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// are averaged into a float64. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">playerAccuracies</span> <span class="o">:=</span> <span class="o">...</span> <span class="c1">// PCollection&lt;string,int&gt; |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"><span class="nx">avgAccuracyPerPlayer</span> <span class="o">:=</span> <span class="nx">stats</span><span class="p">.</span><span class="nf">MeanPerKey</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">playerAccuracies</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// avgAccuracyPerPlayer is a PCollection&lt;string,float64&gt; |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">pcoll</span> <span class="o">=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">([</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;alice&#34;</span><span class="p">,</span> <span class="nx">accuracy</span>: <span class="kt">1.0</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;bob&#34;</span><span class="p">,</span> <span class="nx">accuracy</span>: <span class="kt">0.99</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;eve&#34;</span><span class="p">,</span> <span class="nx">accuracy</span>: <span class="kt">0.5</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;eve&#34;</span><span class="p">,</span> <span class="nx">accuracy</span>: <span class="kt">0.25</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">])</span> |
| </span></span><span class="line"><span class="cl"><span class="p">);</span> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">pcoll</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">groupBy</span><span class="p">(</span><span class="s2">&#34;player&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">combining</span><span class="p">(</span><span class="s2">&#34;accuracy&#34;</span><span class="p">,</span> <span class="nx">combiners</span><span class="p">.</span><span class="nx">mean</span><span class="p">,</span> <span class="s2">&#34;mean&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">combining</span><span class="p">(</span><span class="s2">&#34;accuracy&#34;</span><span class="p">,</span> <span class="nx">combiners</span><span class="p">.</span><span class="nx">max</span><span class="p">,</span> <span class="s2">&#34;max&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">);</span> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">expected</span> <span class="o">=</span> <span class="p">[</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;alice&#34;</span><span class="p">,</span> <span class="nx">mean</span>: <span class="kt">1.0</span><span class="p">,</span> <span class="nx">max</span>: <span class="kt">1.0</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;bob&#34;</span><span class="p">,</span> <span class="nx">mean</span>: <span class="kt">0.99</span><span class="p">,</span> <span class="nx">max</span>: <span class="kt">0.99</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">player</span><span class="o">:</span> <span class="s2">&#34;eve&#34;</span><span class="p">,</span> <span class="nx">mean</span>: <span class="kt">0.375</span><span class="p">,</span> <span class="nx">max</span>: <span class="kt">0.5</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"><span class="p">];</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">Combine</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">group_by</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">animal]</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">combine</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">total_weight</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">fn</span><span class="p">:</span><span class="w"> </span><span class="l">sum</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">value</span><span class="p">:</span><span class="w"> </span><span class="l">weight</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">average_weight</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">fn</span><span class="p">:</span><span class="w"> </span><span class="l">mean</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">value</span><span class="p">:</span><span class="w"> </span><span class="l">weight</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="flatten">4.2.5. Flatten</h4> |
| <p><span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.56.0/index.html?org/apache/beam/sdk/transforms/Flatten.html"><code>Flatten</code></a></span> |
| <span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py"><code>Flatten</code></a></span> |
| <span class="language-go"><a href="https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/flatten.go"><code>Flatten</code></a></span> |
| <span class="language-typescript"><code>Flatten</code></span> |
| is a Beam transform for <code>PCollection</code> objects that store the same data type. |
| <code>Flatten</code> merges multiple <code>PCollection</code> objects into a single logical |
| <code>PCollection</code>.</p> |
| <p>The following example shows how to apply a <code>Flatten</code> transform to merge multiple |
| <code>PCollection</code> objects.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Flatten takes a PCollectionList of PCollection objects of a given type. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// Returns a single PCollection that contains all of the elements in the PCollection objects in that list. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">pc1</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">pc2</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">pc3</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollectionList</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">collections</span> <span class="o">=</span> <span class="n">PCollectionList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">pc1</span><span class="o">).</span><span class="na">and</span><span class="o">(</span><span class="n">pc2</span><span class="o">).</span><span class="na">and</span><span class="o">(</span><span class="n">pc3</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">merged</span> <span class="o">=</span> <span class="n">collections</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Flatten</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">pCollections</span><span class="o">());</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># Flatten takes a tuple of PCollection objects.</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Returns a single PCollection that contains all of the elements in the PCollection objects in that tuple.</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">merged</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="n">pcoll1</span><span class="p">,</span> <span class="n">pcoll2</span><span class="p">,</span> <span class="n">pcoll3</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># A list of tuples can be &#34;piped&#34; directly into a Flatten transform.</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Flatten</span><span class="p">())</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// Flatten accepts any number of PCollections of the same element type. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// Returns a single PCollection that contains all of the elements in input PCollections. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"><span class="nx">merged</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">Flatten</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">pcol1</span><span class="p">,</span> <span class="nx">pcol2</span><span class="p">,</span> <span class="nx">pcol3</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="c1">// Flatten taken an array of PCollection objects, wrapped in beam.P(...) |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// Returns a single PCollection that contains a union of all of the elements in all input PCollections. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">fib</span> <span class="o">=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nx">withName</span><span class="p">(</span><span class="s2">&#34;createFib&#34;</span><span class="p">,</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">8</span><span class="p">]))</span> |
| </span></span><span class="line"><span class="cl"><span class="p">);</span> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">pow</span> <span class="o">=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nx">withName</span><span class="p">(</span><span class="s2">&#34;createPow&#34;</span><span class="p">,</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">16</span><span class="p">,</span> <span class="mi">32</span><span class="p">]))</span> |
| </span></span><span class="line"><span class="cl"><span class="p">);</span> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">P</span><span class="p">([</span><span class="nx">fib</span><span class="p">,</span> <span class="nx">pow</span><span class="p">]).</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">flatten</span><span class="p">());</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl">- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">Flatten</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">input</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="l">SomeProducingTransform, AnotherProducingTransform]</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-yaml">In Beam YAML explicit flattens are not usually needed as one can list |
| multiple inputs for any transform which will be implicitly flattened.</p> |
| <h5 id="data-encoding-merged-collections">4.2.5.1. Data encoding in merged collections</h5> |
| <p>By default, the coder for the output <code>PCollection</code> is the same as the coder for |
| the first <code>PCollection</code> in the input <code>PCollectionList</code>. However, the input |
| <code>PCollection</code> objects can each use different coders, as long as they all contain |
| the same data type in your chosen language.</p> |
| <h5 id="merging-windowed-collections">4.2.5.2. Merging windowed collections</h5> |
| <p>When using <code>Flatten</code> to merge <code>PCollection</code> objects that have a windowing |
| strategy applied, all of the <code>PCollection</code> objects you want to merge must use a |
| compatible windowing strategy and window sizing. For example, all the |
| collections you&rsquo;re merging must all use (hypothetically) identical 5-minute |
| fixed windows or 4-minute sliding windows starting every 30 seconds.</p> |
| <p>If your pipeline attempts to use <code>Flatten</code> to merge <code>PCollection</code> objects with |
| incompatible windows, Beam generates an <code>IllegalStateException</code> error when your |
| pipeline is constructed.</p> |
| <h4 id="partition">4.2.6. Partition</h4> |
| <p><span class="language-java"><a href="https://beam.apache.org/releases/javadoc/2.56.0/index.html?org/apache/beam/sdk/transforms/Partition.html"><code>Partition</code></a></span> |
| <span class="language-py"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py"><code>Partition</code></a></span> |
| <span class="language-go"><a href="https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/partition.go"><code>Partition</code></a></span> |
| <span class="language-typescript"><code>Partition</code></span> |
| is a Beam transform for <code>PCollection</code> objects that store the same data |
| type. <code>Partition</code> splits a single <code>PCollection</code> into a fixed number of smaller |
| collections.</p> |
| <p class="language-typescript">Often in the Typescript SDK the <code>Split</code> transform is more natural to use.</p> |
| <p><code>Partition</code> divides the elements of a <code>PCollection</code> according to a partitioning |
| function that you provide. The partitioning function contains the logic that |
| determines how to split up the elements of the input <code>PCollection</code> into each |
| resulting partition <code>PCollection</code>. The number of partitions must be determined |
| at graph construction time. You can, for example, pass the number of partitions |
| as a command-line option at runtime (which will then be used to build your |
| pipeline graph), but you cannot determine the number of partitions in |
| mid-pipeline (based on data calculated after your pipeline graph is constructed, |
| for instance).</p> |
| <p>The following example divides a <code>PCollection</code> into percentile groups.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Provide an int value with the desired number of result partitions, and a PartitionFn that represents the |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// partitioning function. In this example, we define the PartitionFn in-line. Returns a PCollectionList |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// containing each of the resulting partitions as individual PCollection objects. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Student</span><span class="o">&gt;</span> <span class="n">students</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Split students up into 10 partitions, by percentile: |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollectionList</span><span class="o">&lt;</span><span class="n">Student</span><span class="o">&gt;</span> <span class="n">studentsByPercentile</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">students</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Partition</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">10</span><span class="o">,</span> <span class="k">new</span> <span class="n">PartitionFn</span><span class="o">&lt;</span><span class="n">Student</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">int</span> <span class="nf">partitionFor</span><span class="o">(</span><span class="n">Student</span> <span class="n">student</span><span class="o">,</span> <span class="kt">int</span> <span class="n">numPartitions</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">student</span><span class="o">.</span><span class="na">getPercentile</span><span class="o">()</span> <span class="c1">// 0..99 |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">*</span> <span class="n">numPartitions</span> <span class="o">/</span> <span class="n">100</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}}));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// You can extract each partition from the PCollectionList using the get method, as follows: |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Student</span><span class="o">&gt;</span> <span class="n">fortiethPercentile</span> <span class="o">=</span> <span class="n">studentsByPercentile</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">4</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># Provide an int value with the desired number of result partitions, and a partitioning function (partition_fn in this example).</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Returns a tuple of PCollection objects containing each of the resulting partitions as individual PCollection objects.</span> |
| </span></span><span class="line"><span class="cl"><span class="n">students</span> <span class="o">=</span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">partition_fn</span><span class="p">(</span><span class="n">student</span><span class="p">,</span> <span class="n">num_partitions</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nb">int</span><span class="p">(</span><span class="n">get_percentile</span><span class="p">(</span><span class="n">student</span><span class="p">)</span> <span class="o">*</span> <span class="n">num_partitions</span> <span class="o">/</span> <span class="mi">100</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">by_decile</span> <span class="o">=</span> <span class="n">students</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Partition</span><span class="p">(</span><span class="n">partition_fn</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># You can extract each partition from the tuple of PCollection objects as follows:</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">fortieth_percentile</span> <span class="o">=</span> <span class="n">by_decile</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">decileFn</span><span class="p">(</span><span class="nx">student</span> <span class="nx">Student</span><span class="p">)</span> <span class="kt">int</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nb">int</span><span class="p">(</span><span class="nb">float64</span><span class="p">(</span><span class="nx">student</span><span class="p">.</span><span class="nx">Percentile</span><span class="p">)</span> <span class="o">/</span> <span class="nb">float64</span><span class="p">(</span><span class="mi">10</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">register</span><span class="p">.</span><span class="nf">Function1x1</span><span class="p">(</span><span class="nx">decileFn</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Partition returns a slice of PCollections |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">studentsByPercentile</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">Partition</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="nx">decileFn</span><span class="p">,</span> <span class="nx">students</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Each partition can be extracted by indexing into the slice. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">fortiethPercentile</span> <span class="o">:=</span> <span class="nx">studentsByPercentile</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">deciles</span>: <span class="kt">PCollection</span><span class="p">&lt;</span><span class="nt">Student</span><span class="p">&gt;[]</span> <span class="o">=</span> <span class="nx">students</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nx">partition</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="nx">student</span><span class="p">,</span> <span class="nx">numPartitions</span><span class="p">)</span> <span class="o">=&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="nb">Math</span><span class="p">.</span><span class="nx">floor</span><span class="p">((</span><span class="nx">getPercentile</span><span class="p">(</span><span class="nx">student</span><span class="p">)</span> <span class="o">/</span> <span class="mi">100</span><span class="p">)</span> <span class="o">*</span> <span class="nx">numPartitions</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="mi">10</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">);</span> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">topDecile</span>: <span class="kt">PCollection</span><span class="p">&lt;</span><span class="nt">Student</span><span class="p">&gt;</span> <span class="o">=</span> <span class="nx">deciles</span><span class="p">[</span><span class="mi">9</span><span class="p">];</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">Partition</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">by</span><span class="p">:</span><span class="w"> </span><span class="l">str(percentile // 10)</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">language</span><span class="p">:</span><span class="w"> </span><span class="l">python</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">outputs</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&#34;0&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;1&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;2&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;3&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;4&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;5&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;6&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;7&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;8&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;9&#34;</span><span class="p">,</span><span class="w"> </span><span class="s2">&#34;10&#34;</span><span class="p">]</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-yaml">Note that in Beam YAML, <code>PCollections</code> are partitioned via string rather than integer values.</p> |
| <h3 id="requirements-for-writing-user-code-for-beam-transforms">4.3. Requirements for writing user code for Beam transforms</h3> |
| <p>When you build user code for a Beam transform, you should keep in mind the |
| distributed nature of execution. For example, there might be many copies of your |
| function running on a lot of different machines in parallel, and those copies |
| function independently, without communicating or sharing state with any of the |
| other copies. Depending on the Pipeline Runner and processing back-end you |
| choose for your pipeline, each copy of your user code function may be retried or |
| run multiple times. As such, you should be cautious about including things like |
| state dependency in your user code.</p> |
| <p>In general, your user code must fulfill at least these requirements:</p> |
| <ul> |
| <li>Your function object must be <strong>serializable</strong>.</li> |
| <li>Your function object must be <strong>thread-compatible</strong>, and be aware that <em>the |
| Beam SDKs are not thread-safe</em>.</li> |
| </ul> |
| <p>In addition, it&rsquo;s recommended that you make your function object <strong>idempotent</strong>. |
| Non-idempotent functions are supported by Beam, but require additional |
| thought to ensure correctness when there are external side effects.</p> |
| <span class="language-java language-py"> |
| <blockquote> |
| <p><strong>Note:</strong> These requirements apply to subclasses of <code>DoFn</code>(a function object |
| used with the <a href="#pardo">ParDo</a> transform), <code>CombineFn</code> (a function object used |
| with the <a href="#combine">Combine</a> transform), and <code>WindowFn</code> (a function object |
| used with the <a href="#windowing">Window</a> transform).</p> |
| </blockquote> |
| </span> |
| <span class="language-go"> |
| <blockquote> |
| <p><strong>Note:</strong> These requirements apply to <code>DoFn</code>s (a function object |
| used with the <a href="#pardo">ParDo</a> transform), <code>CombineFn</code>s (a function object used |
| with the <a href="#combine">Combine</a> transform), and <code>WindowFn</code>s (a function object |
| used with the <a href="#windowing">Window</a> transform).</p> |
| </blockquote> |
| </span> |
| <h4 id="user-code-serializability">4.3.1. Serializability</h4> |
| <p>Any function object you provide to a transform must be <strong>fully serializable</strong>. |
| This is because a copy of the function needs to be serialized and transmitted to |
| a remote worker in your processing cluster. |
| <span class="language-java language-py">The base classes for user code, such |
| as <code>DoFn</code>, <code>CombineFn</code>, and <code>WindowFn</code>, already implement <code>Serializable</code>; |
| however, your subclass must not add any non-serializable members.</span> |
| <span class="language-go">Funcs are serializable as long as |
| they are registered with <code>register.FunctionXxY</code> (for simple functions) or |
| <code>register.DoFnXxY</code> (for structural DoFns), and are not closures. Structural |
| <code>DoFn</code>s will have all exported fields serialized. Unexported fields are unable to |
| be serialized, and will be silently ignored.</span> |
| <span class="language-typescript"> |
| The Typescript SDK use <a href="https://github.com/nokia/ts-serialize-closures">ts-serialize-closures</a> |
| to serialize functions (and other objects). |
| This works out of the box for functions that are not closures, and also works |
| for closures as long as the function in question (and any closures it references) |
| are compiled with the |
| <a href="https://github.com/apache/beam/blob/master/sdks/typescript/tsconfig.json"><code>ts-closure-transform</code> hooks</a> |
| (e.g. by using <code>ttsc</code> in place of <code>tsc</code>). |
| One can alternatively call |
| <code>requireForSerialization(&quot;importableModuleDefiningFunc&quot;, {func})</code> |
| to <a href="https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/serialization.ts">register a function directly</a> by name which can be less error-prone. |
| Note that if, as is often the case in Javascript, <code>func</code> returns objects that |
| contain closures, it is not sufficient to register <code>func</code> alone&ndash;its return |
| value must be registered if used.</span></p> |
| <p>Some other serializability factors you should keep in mind are:</p> |
| <ul> |
| <li><span class="language-java language-py">Transient</span><span class="language-go">Unexported</span> |
| fields in your function object are <em>not</em> transmitted to worker |
| instances, because they are not automatically serialized.</li> |
| <li>Avoid loading a field with a large amount of data before serialization.</li> |
| <li>Individual instances of your function object cannot share data.</li> |
| <li>Mutating a function object after it gets applied will have no effect.</li> |
| </ul> |
| <span class="language-java"> |
| <blockquote> |
| <p><strong>Note:</strong> Take care when declaring your function object inline by using an anonymous |
| inner class instance. In a non-static context, your inner class instance will |
| implicitly contain a pointer to the enclosing class and that class&rsquo; state. |
| That enclosing class will also be serialized, and thus the same considerations |
| that apply to the function object itself also apply to this outer class.</p> |
| </blockquote> |
| </span> |
| <span class="language-go"> |
| <blockquote> |
| <p><strong>Note:</strong> There&rsquo;s no way to detect if a function is a closure. Closures will cause |
| runtime errors and pipeline failures. Avoid using anonymous functions when possible.</p> |
| </blockquote> |
| </span> |
| <h4 id="user-code-thread-compatibility">4.3.2. Thread-compatibility</h4> |
| <p>Your function object should be thread-compatible. Each instance of your function |
| object is accessed by a single thread at a time on a worker instance, unless you |
| explicitly create your own threads. Note, however, that <strong>the Beam SDKs are not |
| thread-safe</strong>. If you create your own threads in your user code, you must |
| provide your own synchronization. <span class="language-java"> Note that static members in your function |
| object are not passed to worker instances and that multiple instances of your |
| function may be accessed from different threads.</span></p> |
| <h4 id="user-code-idempotence">4.3.3. Idempotence</h4> |
| <p>It&rsquo;s recommended that you make your function object idempotent&ndash;that is, that it |
| can be repeated or retried as often as necessary without causing unintended side |
| effects. Non-idempotent functions are supported, however the Beam model provides |
| no guarantees as to the number of times your user code might be invoked or retried; |
| as such, keeping your function object idempotent keeps your pipeline&rsquo;s output |
| deterministic, and your transforms&rsquo; behavior more predictable and easier to debug.</p> |
| <h3 id="side-inputs">4.4. Side inputs</h3> |
| <p>In addition to the main input <code>PCollection</code>, you can provide additional inputs |
| to a <code>ParDo</code> transform in the form of side inputs. A side input is an additional |
| input that your <code>DoFn</code> can access each time it processes an element in the input |
| <code>PCollection</code>. When you specify a side input, you create a view of some other |
| data that can be read from within the <code>ParDo</code> transform&rsquo;s <code>DoFn</code> while processing |
| each element.</p> |
| <p>Side inputs are useful if your <code>ParDo</code> needs to inject additional data when |
| processing each element in the input <code>PCollection</code>, but the additional data |
| needs to be determined at runtime (and not hard-coded). Such values might be |
| determined by the input data, or depend on a different branch of your pipeline.</p> |
| <p class="language-go">All side input iterables should be registered using a generic <code>register.IterX[...]</code> |
| function. This optimizes runtime execution of the iterable.</p> |
| <h4 id="side-inputs-pardo">4.4.1. Passing side inputs to ParDo</h4> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="c1">// Pass side inputs to your ParDo transform by invoking .withSideInputs. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// Inside your DoFn, access the side input by using the method DoFn.ProcessContext.sideInput. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The input PCollection to ParDo. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">words</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// A PCollection of word lengths that we&#39;ll combine into a single value. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">wordLengths</span> <span class="o">=</span> <span class="o">...;</span> <span class="c1">// Singleton PCollection |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Create a singleton PCollectionView from wordLengths using Combine.globally and View.asSingleton. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kd">final</span> <span class="n">PCollectionView</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">maxWordLengthCutOffView</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">wordLengths</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Combine</span><span class="o">.</span><span class="na">globally</span><span class="o">(</span><span class="k">new</span> <span class="n">Max</span><span class="o">.</span><span class="na">MaxIntFn</span><span class="o">()).</span><span class="na">asSingletonView</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Apply a ParDo that takes maxWordLengthCutOffView as a side input. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">wordsBelowCutOff</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">words</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="nd">@Element</span> <span class="n">String</span> <span class="n">word</span><span class="o">,</span> <span class="n">OutputReceiver</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">out</span><span class="o">,</span> <span class="n">ProcessContext</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// In our DoFn, access the side input. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kt">int</span> <span class="n">lengthCutOff</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="na">sideInput</span><span class="o">(</span><span class="n">maxWordLengthCutOffView</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="o">(</span><span class="n">word</span><span class="o">.</span><span class="na">length</span><span class="o">()</span> <span class="o">&lt;=</span> <span class="n">lengthCutOff</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">out</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">word</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}).</span><span class="na">withSideInputs</span><span class="o">(</span><span class="n">maxWordLengthCutOffView</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># Side inputs are available as extra arguments in the DoFn&#39;s process method or Map / FlatMap&#39;s callable.</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Optional, positional, and keyword arguments are all supported. Deferred arguments are unwrapped into their</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># actual values. For example, using pvalue.AsIteor(pcoll) at pipeline construction time results in an iterable</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># of the actual elements of pcoll being passed into each process invocation. In this example, side inputs are</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># passed to a FlatMap transform as extra arguments and consumed by filter_using_length.</span> |
| </span></span><span class="line"><span class="cl"><span class="n">words</span> <span class="o">=</span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Callable takes additional arguments.</span> |
| </span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">filter_using_length</span><span class="p">(</span><span class="n">word</span><span class="p">,</span> <span class="n">lower_bound</span><span class="p">,</span> <span class="n">upper_bound</span><span class="o">=</span><span class="nb">float</span><span class="p">(</span><span class="s1">&#39;inf&#39;</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">lower_bound</span> <span class="o">&lt;=</span> <span class="nb">len</span><span class="p">(</span><span class="n">word</span><span class="p">)</span> <span class="o">&lt;=</span> <span class="n">upper_bound</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">word</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Construct a deferred side input.</span> |
| </span></span><span class="line"><span class="cl"><span class="n">avg_word_len</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">words</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="nb">len</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">combiners</span><span class="o">.</span><span class="n">MeanCombineFn</span><span class="p">()))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Call with explicit side inputs.</span> |
| </span></span><span class="line"><span class="cl"><span class="n">small_words</span> <span class="o">=</span> <span class="n">words</span> <span class="o">|</span> <span class="s1">&#39;small&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span class="n">filter_using_length</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># A single deferred side input.</span> |
| </span></span><span class="line"><span class="cl"><span class="n">larger_than_average</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">words</span> <span class="o">|</span> <span class="s1">&#39;large&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMap</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">filter_using_length</span><span class="p">,</span> <span class="n">lower_bound</span><span class="o">=</span><span class="n">pvalue</span><span class="o">.</span><span class="n">AsSingleton</span><span class="p">(</span><span class="n">avg_word_len</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Mix and match.</span> |
| </span></span><span class="line"><span class="cl"><span class="n">small_but_nontrivial</span> <span class="o">=</span> <span class="n">words</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMap</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">filter_using_length</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">lower_bound</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">upper_bound</span><span class="o">=</span><span class="n">pvalue</span><span class="o">.</span><span class="n">AsSingleton</span><span class="p">(</span><span class="n">avg_word_len</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># We can also pass side inputs to a ParDo transform, which will get passed to its process method.</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># The first two arguments for the process method would be self and element.</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">FilterUsingLength</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">,</span> <span class="n">lower_bound</span><span class="p">,</span> <span class="n">upper_bound</span><span class="o">=</span><span class="nb">float</span><span class="p">(</span><span class="s1">&#39;inf&#39;</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">lower_bound</span> <span class="o">&lt;=</span> <span class="nb">len</span><span class="p">(</span><span class="n">element</span><span class="p">)</span> <span class="o">&lt;=</span> <span class="n">upper_bound</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">element</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">small_words</span> <span class="o">=</span> <span class="n">words</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">FilterUsingLength</span><span class="p">(),</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="o">...</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// Side inputs are provided using `beam.SideInput` in the DoFn&#39;s ProcessElement method. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// Side inputs can be arbitrary PCollections, which can then be iterated over per element |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// in a DoFn. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// Side input parameters appear after main input elements, and before any output emitters. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">words</span> <span class="p">=</span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// avgWordLength is a PCollection containing a single element, a singleton. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">avgWordLength</span> <span class="o">:=</span> <span class="nx">stats</span><span class="p">.</span><span class="nf">Mean</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">wordLengths</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Side inputs are added as with the beam.SideInput option to beam.ParDo. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">wordsAboveCutOff</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">filterWordsAbove</span><span class="p">,</span> <span class="nx">words</span><span class="p">,</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">SideInput</span><span class="p">{</span><span class="nx">Input</span><span class="p">:</span> <span class="nx">avgWordLength</span><span class="p">})</span> |
| </span></span><span class="line"><span class="cl"><span class="nx">wordsBelowCutOff</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">filterWordsBelow</span><span class="p">,</span> <span class="nx">words</span><span class="p">,</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">SideInput</span><span class="p">{</span><span class="nx">Input</span><span class="p">:</span> <span class="nx">avgWordLength</span><span class="p">})</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// filterWordsAbove is a DoFn that takes in a word, |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// and a singleton side input iterator as of a length cut off |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// and only emits words that are beneath that cut off. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// If the iterator has no elements, an error is returned, aborting processing. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="nf">filterWordsAbove</span><span class="p">(</span><span class="nx">word</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">lengthCutOffIter</span> <span class="kd">func</span><span class="p">(</span><span class="o">*</span><span class="kt">float64</span><span class="p">)</span> <span class="kt">bool</span><span class="p">,</span> <span class="nx">emitAboveCutoff</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="kt">error</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">var</span> <span class="nx">cutOff</span> <span class="kt">float64</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">ok</span> <span class="o">:=</span> <span class="nf">lengthCutOffIter</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">cutOff</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="p">!</span><span class="nx">ok</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">fmt</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;no length cutoff provided&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nb">float64</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="nx">word</span><span class="p">))</span> <span class="p">&gt;</span> <span class="nx">cutOff</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emitAboveCutoff</span><span class="p">(</span><span class="nx">word</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// filterWordsBelow is a DoFn that takes in a word, |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// and a singleton side input of a length cut off |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// and only emits words that are beneath that cut off. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// If the side input isn&#39;t a singleton, a runtime panic will occur. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="nf">filterWordsBelow</span><span class="p">(</span><span class="nx">word</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">lengthCutOff</span> <span class="kt">float64</span><span class="p">,</span> <span class="nx">emitBelowCutoff</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nb">float64</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="nx">word</span><span class="p">))</span> <span class="o">&lt;=</span> <span class="nx">lengthCutOff</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emitBelowCutoff</span><span class="p">(</span><span class="nx">word</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">register</span><span class="p">.</span><span class="nf">Function3x1</span><span class="p">(</span><span class="nx">filterWordsAbove</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">register</span><span class="p">.</span><span class="nf">Function3x0</span><span class="p">(</span><span class="nx">filterWordsBelow</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// 1 input of type string =&gt; Emitter1[string] |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">register</span><span class="p">.</span><span class="nx">Emitter1</span><span class="p">[</span><span class="kt">string</span><span class="p">]()</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// 1 input of type float64 =&gt; Iter1[float64] |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">register</span><span class="p">.</span><span class="nx">Iter1</span><span class="p">[</span><span class="kt">float64</span><span class="p">]()</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// The Go SDK doesn&#39;t support custom ViewFns. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// See https://github.com/apache/beam/issues/18602 for details |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// on how to contribute them! |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="c1">// Side inputs are provided by passing an extra context object to |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// `map`, `flatMap`, or `parDo` transforms. This object will get passed as an |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// extra argument to the provided function (or `process` method of the `DoFn`). |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// `SideInputParam` properties (generally created with `pardo.xxxSideInput(...)`) |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// have a `lookup` method that can be invoked from within the process method. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Let words be a PCollection of strings. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kr">const</span> <span class="nx">words</span> : <span class="kt">PCollection</span><span class="p">&lt;</span><span class="nt">string</span><span class="p">&gt;</span> <span class="o">=</span> <span class="p">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// meanLengthPColl will contain a single number whose value is the |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// average length of the words |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kr">const</span> <span class="nx">meanLengthPColl</span>: <span class="kt">PCollection</span><span class="p">&lt;</span><span class="nt">number</span><span class="p">&gt;</span> <span class="o">=</span> <span class="nx">words</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">groupGlobally</span><span class="p">&lt;</span><span class="nt">string</span><span class="p">&gt;()</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">combining</span><span class="p">((</span><span class="nx">word</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">word</span><span class="p">.</span><span class="nx">length</span><span class="p">,</span> <span class="nx">combiners</span><span class="p">.</span><span class="nx">mean</span><span class="p">,</span> <span class="s2">&#34;mean&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">map</span><span class="p">(({</span> <span class="nx">mean</span> <span class="p">})</span> <span class="o">=&gt;</span> <span class="nx">mean</span><span class="p">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Now we use this as a side input to yield only words that are |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// smaller than average. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kr">const</span> <span class="nx">smallWords</span> <span class="o">=</span> <span class="nx">words</span><span class="p">.</span><span class="nx">flatMap</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// This is the function, taking context as a second argument. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kd">function</span><span class="o">*</span> <span class="nx">keepSmall</span><span class="p">(</span><span class="nx">word</span><span class="p">,</span> <span class="nx">context</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="p">(</span><span class="nx">word</span><span class="p">.</span><span class="nx">length</span> <span class="o">&lt;</span> <span class="nx">context</span><span class="p">.</span><span class="nx">meanLength</span><span class="p">.</span><span class="nx">lookup</span><span class="p">())</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="nx">word</span><span class="p">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// This is the context that will be passed as a second argument. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="p">{</span> <span class="nx">meanLength</span>: <span class="kt">pardo.singletonSideInput</span><span class="p">(</span><span class="nx">meanLengthPColl</span><span class="p">)</span> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="side-inputs-windowing">4.4.2. Side inputs and windowing</h4> |
| <p>A windowed <code>PCollection</code> may be infinite and thus cannot be compressed into a |
| single value (or single collection class). When you create a <code>PCollectionView</code> |
| of a windowed <code>PCollection</code>, the <code>PCollectionView</code> represents a single entity |
| per window (one singleton per window, one list per window, etc.).</p> |
| <p>Beam uses the window(s) for the main input element to look up the appropriate |
| window for the side input element. Beam projects the main input element&rsquo;s window |
| into the side input&rsquo;s window set, and then uses the side input from the |
| resulting window. If the main input and side inputs have identical windows, the |
| projection provides the exact corresponding window. However, if the inputs have |
| different windows, Beam uses the projection to choose the most appropriate side |
| input window.</p> |
| <p>For example, if the main input is windowed using fixed-time windows of one |
| minute, and the side input is windowed using fixed-time windows of one hour, |
| Beam projects the main input window against the side input window set and |
| selects the side input value from the appropriate hour-long side input window.</p> |
| <p>If the main input element exists in more than one window, then <code>processElement</code> |
| gets called multiple times, once for each window. Each call to <code>processElement</code> |
| projects the &ldquo;current&rdquo; window for the main input element, and thus might provide |
| a different view of the side input each time.</p> |
| <p>If the side input has multiple trigger firings, Beam uses the value from the |
| latest trigger firing. This is particularly useful if you use a side input with |
| a single global window and specify a trigger.</p> |
| <h3 id="additional-outputs">4.5. Additional outputs</h3> |
| <p class="language-java language-py">While <code>ParDo</code> always produces a main output <code>PCollection</code> (as the return value |
| from <code>apply</code>), you can also have your <code>ParDo</code> produce any number of additional |
| output <code>PCollection</code>s. If you choose to have multiple outputs, your <code>ParDo</code> |
| returns all of the output <code>PCollection</code>s (including the main output) bundled |
| together.</p> |
| <p class="language-go">While <code>beam.ParDo</code> always produces an output <code>PCollection</code>, your <code>DoFn</code> can produce any |
| number of additional output <code>PCollections</code>s, or even none at all. |
| If you choose to have multiple outputs, your <code>DoFn</code> needs to be called with the <code>ParDo</code> |
| function that matches the number of outputs. <code>beam.ParDo2</code> for two output <code>PCollection</code>s, |
| <code>beam.ParDo3</code> for three and so on until <code>beam.ParDo7</code>. If you need more, you can |
| use <code>beam.ParDoN</code> which will return a <code>[]beam.PCollection</code>.</p> |
| <p class="language-typescript">While <code>ParDo</code> always produces a main output <code>PCollection</code> (as the return value |
| from <code>apply</code>). If you want to have multiple outputs, emit an object with distinct |
| properties in your <code>ParDo</code> operation and follow this operation with a <code>Split</code> |
| to break it into multiple <code>PCollection</code>s.</p> |
| <p class="language-yaml">In Beam YAML, one obtains multiple outputs by emitting all outputs to a single |
| <code>PCollection</code>, possibly with an extra field, and then using <code>Partition</code> to |
| split this single <code>PCollection</code> into multiple distinct <code>PCollection</code> |
| outputs.</p> |
| <h4 id="output-tags">4.5.1. Tags for multiple outputs</h4> |
| <p class="language-typescript">The <code>Split</code> PTransform will take a PCollection of elements of the form |
| <code>{tagA?: A, tagB?: B, ...}</code> and return a object |
| <code>{tagA: PCollection&lt;A&gt;, tagB: PCollection&lt;B&gt;, ...}</code>. |
| The set of expected tags is passed to the operation; how multiple or |
| unknown tags are handled can be specified by passing a non-default |
| <code>SplitOptions</code> instance.</p> |
| <p class="language-go">The Go SDK doesn&rsquo;t use output tags, and instead uses positional ordering for |
| multiple output PCollections.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// To emit elements to multiple output PCollections, create a TupleTag object to identify each collection |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// that your ParDo produces. For example, if your ParDo produces three output PCollections (the main output |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// and two additional outputs), you must create three TupleTags. The following example code shows how to |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// create TupleTags for a ParDo with three output PCollections. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Input PCollection to our ParDo. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">words</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The ParDo will filter words whose length is below a cutoff and add them to |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// the main output PCollection&lt;String&gt;. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// If a word is above the cutoff, the ParDo will add the word length to an |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// output PCollection&lt;Integer&gt;. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// If a word starts with the string &#34;MARKER&#34;, the ParDo will add that word to an |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// output PCollection&lt;String&gt;. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">wordLengthCutOff</span> <span class="o">=</span> <span class="n">10</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Create three TupleTags, one for each output PCollection. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// Output that contains words below the length cutoff. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kd">final</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">wordsBelowCutOffTag</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;(){};</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Output that contains word lengths. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kd">final</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">wordLengthsAboveCutOffTag</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;(){};</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Output that contains &#34;MARKER&#34; words. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kd">final</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">markedWordsTag</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;(){};</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Passing Output Tags to ParDo: |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// After you specify the TupleTags for each of your ParDo outputs, pass the tags to your ParDo by invoking |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// .withOutputTags. You pass the tag for the main output first, and then the tags for any additional outputs |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// in a TupleTagList. Building on our previous example, we pass the three TupleTags for our three output |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// PCollections to our ParDo. Note that all of the outputs (including the main output PCollection) are |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// bundled into the returned PCollectionTuple. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="n">PCollectionTuple</span> <span class="n">results</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">words</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// DoFn continues here. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">})</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Specify the tag for the main output. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">withOutputTags</span><span class="o">(</span><span class="n">wordsBelowCutOffTag</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Specify the tags for the two additional outputs as a TupleTagList. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">TupleTagList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">wordLengthsAboveCutOffTag</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">and</span><span class="o">(</span><span class="n">markedWordsTag</span><span class="o">)));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># To emit elements to multiple output PCollections, invoke with_outputs() on the ParDo, and specify the</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># expected tags for the outputs. with_outputs() returns a DoOutputsTuple object. Tags specified in</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># with_outputs are attributes on the returned DoOutputsTuple object. The tags give access to the</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># corresponding output PCollections.</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">results</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">words</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">ProcessWords</span><span class="p">(),</span> <span class="n">cutoff_length</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">marker</span><span class="o">=</span><span class="s1">&#39;x&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">with_outputs</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;above_cutoff_lengths&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;marked strings&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">main</span><span class="o">=</span><span class="s1">&#39;below_cutoff_strings&#39;</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"><span class="n">below</span> <span class="o">=</span> <span class="n">results</span><span class="o">.</span><span class="n">below_cutoff_strings</span> |
| </span></span><span class="line"><span class="cl"><span class="n">above</span> <span class="o">=</span> <span class="n">results</span><span class="o">.</span><span class="n">above_cutoff_lengths</span> |
| </span></span><span class="line"><span class="cl"><span class="n">marked</span> <span class="o">=</span> <span class="n">results</span><span class="p">[</span><span class="s1">&#39;marked strings&#39;</span><span class="p">]</span> <span class="c1"># indexing works as well</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># The result is also iterable, ordered in the same order that the tags were passed to with_outputs(),</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># the main tag (if specified) first.</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">below</span><span class="p">,</span> <span class="n">above</span><span class="p">,</span> <span class="n">marked</span> <span class="o">=</span> <span class="p">(</span><span class="n">words</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ProcessWords</span><span class="p">(),</span> <span class="n">cutoff_length</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">marker</span><span class="o">=</span><span class="s1">&#39;x&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="n">with_outputs</span><span class="p">(</span><span class="s1">&#39;above_cutoff_lengths&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;marked strings&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">main</span><span class="o">=</span><span class="s1">&#39;below_cutoff_strings&#39;</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// beam.ParDo3 returns PCollections in the same order as |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// the emit function parameters in processWords. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">below</span><span class="p">,</span> <span class="nx">above</span><span class="p">,</span> <span class="nx">marked</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo3</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">processWords</span><span class="p">,</span> <span class="nx">words</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// processWordsMixed uses both a standard return and an emitter function. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// The standard return produces the first PCollection from beam.ParDo2, |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// and the emitter produces the second PCollection. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">length</span><span class="p">,</span> <span class="nx">mixedMarked</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo2</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">processWordsMixed</span><span class="p">,</span> <span class="nx">words</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="err">#</span> <span class="nx">Create</span> <span class="nx">three</span> <span class="nx">PCollections</span> <span class="kr">from</span> <span class="nx">a</span> <span class="nx">single</span> <span class="nx">input</span> <span class="nx">PCollection</span><span class="p">.</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="p">{</span> <span class="nx">below</span><span class="p">,</span> <span class="nx">above</span><span class="p">,</span> <span class="nx">marked</span> <span class="p">}</span> <span class="o">=</span> <span class="nx">to_split</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nx">split</span><span class="p">([</span><span class="s2">&#34;below&#34;</span><span class="p">,</span> <span class="s2">&#34;above&#34;</span><span class="p">,</span> <span class="s2">&#34;marked&#34;</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"><span class="p">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="multiple-outputs-dofn">4.5.2. Emitting to multiple outputs in your DoFn</h4> |
| <p class="language-go">Call emitter functions as needed to produce 0 or more elements for its matching |
| <code>PCollection</code>. The same value can be emitted with multiple emitters. |
| As normal, do not mutate values after emitting them from any emitter.</p> |
| <p class="language-go">All emitters should be registered using a generic <code>register.EmitterX[...]</code> |
| function. This optimizes runtime execution of the emitter.</p> |
| <p class="language-go">DoFns can also return a single element via the standard return. |
| The standard return is always the first PCollection returned from beam.ParDo. |
| Other emitters output to their own PCollections in their defined parameter order.</p> |
| <p class="language-yaml"><code>MapToFields</code> is always one-to-one. To perform a one-to-many mapping one can |
| first map a field to an iterable type and then follow this transform with an |
| <code>Explode</code> transform that will emit multiple values, one per value of the |
| exploded field.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Inside your ParDo&#39;s DoFn, you can emit an element to a specific output PCollection by providing a |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// MultiOutputReceiver to your process method, and passing in the appropriate TupleTag to obtain an OutputReceiver. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// After your ParDo, extract the resulting output PCollections from the returned PCollectionTuple. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// Based on the previous example, this shows the DoFn emitting to the main output and two additional outputs. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="nd">@Element</span> <span class="n">String</span> <span class="n">word</span><span class="o">,</span> <span class="n">MultiOutputReceiver</span> <span class="n">out</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="o">(</span><span class="n">word</span><span class="o">.</span><span class="na">length</span><span class="o">()</span> <span class="o">&lt;=</span> <span class="n">wordLengthCutOff</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Emit short word to the main output. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// In this example, it is the output with tag wordsBelowCutOffTag. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">out</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">wordsBelowCutOffTag</span><span class="o">).</span><span class="na">output</span><span class="o">(</span><span class="n">word</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> <span class="k">else</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Emit long word length to the output with tag wordLengthsAboveCutOffTag. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">out</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">wordLengthsAboveCutOffTag</span><span class="o">).</span><span class="na">output</span><span class="o">(</span><span class="n">word</span><span class="o">.</span><span class="na">length</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="o">(</span><span class="n">word</span><span class="o">.</span><span class="na">startsWith</span><span class="o">(</span><span class="s">&#34;MARKER&#34;</span><span class="o">))</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Emit word to the output with tag markedWordsTag. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">out</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">markedWordsTag</span><span class="o">).</span><span class="na">output</span><span class="o">(</span><span class="n">word</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># Inside your ParDo&#39;s DoFn, you can emit an element to a specific output by wrapping the value and the output tag (str).</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># using the pvalue.OutputValue wrapper class.</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Based on the previous example, this shows the DoFn emitting to the main output and two additional outputs.</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">ProcessWords</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">,</span> <span class="n">cutoff_length</span><span class="p">,</span> <span class="n">marker</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">element</span><span class="p">)</span> <span class="o">&lt;=</span> <span class="n">cutoff_length</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Emit this short word to the main output.</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">element</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">else</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Emit this word&#39;s long length to the &#39;above_cutoff_lengths&#39; output.</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">pvalue</span><span class="o">.</span><span class="n">TaggedOutput</span><span class="p">(</span><span class="s1">&#39;above_cutoff_lengths&#39;</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">element</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">element</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="n">marker</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Emit this word to a different output with the &#39;marked strings&#39; tag.</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">pvalue</span><span class="o">.</span><span class="n">TaggedOutput</span><span class="p">(</span><span class="s1">&#39;marked strings&#39;</span><span class="p">,</span> <span class="n">element</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Producing multiple outputs is also available in Map and FlatMap.</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Here is an example that uses FlatMap and shows that the tags do not need to be specified ahead of time.</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">even_odd</span><span class="p">(</span><span class="n">x</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">pvalue</span><span class="o">.</span><span class="n">TaggedOutput</span><span class="p">(</span><span class="s1">&#39;odd&#39;</span> <span class="k">if</span> <span class="n">x</span> <span class="o">%</span> <span class="mi">2</span> <span class="k">else</span> <span class="s1">&#39;even&#39;</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">x</span> <span class="o">%</span> <span class="mi">10</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">x</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">results</span> <span class="o">=</span> <span class="n">numbers</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span class="n">even_odd</span><span class="p">)</span><span class="o">.</span><span class="n">with_outputs</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">evens</span> <span class="o">=</span> <span class="n">results</span><span class="o">.</span><span class="n">even</span> |
| </span></span><span class="line"><span class="cl"><span class="n">odds</span> <span class="o">=</span> <span class="n">results</span><span class="o">.</span><span class="n">odd</span> |
| </span></span><span class="line"><span class="cl"><span class="n">tens</span> <span class="o">=</span> <span class="n">results</span><span class="p">[</span><span class="kc">None</span><span class="p">]</span> <span class="c1"># the undeclared main output</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// processWords is a DoFn that has 3 output PCollections. The emitter functions |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// are matched in positional order to the PCollections returned by beam.ParDo3. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="nf">processWords</span><span class="p">(</span><span class="nx">word</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emitBelowCutoff</span><span class="p">,</span> <span class="nx">emitAboveCutoff</span><span class="p">,</span> <span class="nx">emitMarked</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">const</span> <span class="nx">cutOff</span> <span class="p">=</span> <span class="mi">5</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="nx">word</span><span class="p">)</span> <span class="p">&lt;</span> <span class="nx">cutOff</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emitBelowCutoff</span><span class="p">(</span><span class="nx">word</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emitAboveCutoff</span><span class="p">(</span><span class="nx">word</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nf">isMarkedWord</span><span class="p">(</span><span class="nx">word</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emitMarked</span><span class="p">(</span><span class="nx">word</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// processWordsMixed demonstrates mixing an emitter, with a standard return. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// If a standard return is used, it will always be the first returned PCollection, |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// followed in positional order by the emitter functions. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="nf">processWordsMixed</span><span class="p">(</span><span class="nx">word</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emitMarked</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="kt">int</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nf">isMarkedWord</span><span class="p">(</span><span class="nx">word</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emitMarked</span><span class="p">(</span><span class="nx">word</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="nx">word</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">register</span><span class="p">.</span><span class="nf">Function4x0</span><span class="p">(</span><span class="nx">processWords</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">register</span><span class="p">.</span><span class="nf">Function2x1</span><span class="p">(</span><span class="nx">processWordsMixed</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// 1 input of type string =&gt; Emitter1[string] |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">register</span><span class="p">.</span><span class="nx">Emitter1</span><span class="p">[</span><span class="kt">string</span><span class="p">]()</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">to_split</span> <span class="o">=</span> <span class="nx">words</span><span class="p">.</span><span class="nx">flatMap</span><span class="p">(</span><span class="kd">function</span><span class="o">*</span> <span class="p">(</span><span class="nx">word</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="p">(</span><span class="nx">word</span><span class="p">.</span><span class="nx">length</span> <span class="o">&lt;</span> <span class="mi">5</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="p">{</span> <span class="nx">below</span>: <span class="kt">word</span> <span class="p">};</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="p">{</span> <span class="nx">above</span>: <span class="kt">word</span> <span class="p">};</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="p">(</span><span class="nx">isMarkedWord</span><span class="p">(</span><span class="nx">word</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="p">{</span> <span class="nx">marked</span>: <span class="kt">word</span> <span class="p">};</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">});</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl">- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">MapToFields</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">input</span><span class="p">:</span><span class="w"> </span><span class="l">SomeProducingTransform</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">language</span><span class="p">:</span><span class="w"> </span><span class="l">python</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">fields</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">word</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;line.split()&#34;</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">Explode</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">input</span><span class="p">:</span><span class="w"> </span><span class="l">MapToFields</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">fields</span><span class="p">:</span><span class="w"> </span><span class="l">word</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="other-dofn-parameters">4.5.3. Accessing additional parameters in your DoFn</h4> |
| <p class="language-java">In addition to the element and the <code>OutputReceiver</code>, Beam will populate other parameters to your DoFn&rsquo;s <code>@ProcessElement</code> method. |
| Any combination of these parameters can be added to your process method in any order.</p> |
| <p class="language-py">In addition to the element, Beam will populate other parameters to your DoFn&rsquo;s <code>process</code> method. |
| Any combination of these parameters can be added to your process method in any order.</p> |
| <p class="language-typescript">In addition to the element, Beam will populate other parameters to your DoFn&rsquo;s <code>process</code> method. |
| These are available by placing accessors in the context argument, just as for side inputs.</p> |
| <p class="language-go">In addition to the element, Beam will populate other parameters to your DoFn&rsquo;s <code>ProcessElement</code> method. |
| Any combination of these parameters can be added to your process method in a standard order.</p> |
| <p class="language-go"><strong>context.Context:</strong> |
| To support consolidated logging and user defined metrics, a <code>context.Context</code> parameter can be requested. |
| Per Go conventions, if present it&rsquo;s required to be the first parameter of the <code>DoFn</code> method.</p> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">MyDoFn</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">word</span> <span class="kt">string</span><span class="p">)</span> <span class="kt">string</span> <span class="p">{</span> <span class="o">...</span> <span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java"><strong>Timestamp:</strong> |
| To access the timestamp of an input element, add a parameter annotated with <code>@Timestamp</code> of type <code>Instant</code>. For example:</p> |
| <p class="language-py"><strong>Timestamp:</strong> |
| To access the timestamp of an input element, add a keyword parameter default to <code>DoFn.TimestampParam</code>. For example:</p> |
| <p class="language-go"><strong>Timestamp:</strong> |
| To access the timestamp of an input element, add a <code>beam.EventTime</code> parameter before the element. For example:</p> |
| <p class="language-typescript"><strong>Timestamp:</strong> |
| To access the window an input element falls into, add a <code>pardo.windowParam()</code> to the context argument.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="nd">@Element</span> <span class="n">String</span> <span class="n">word</span><span class="o">,</span> <span class="nd">@Timestamp</span> <span class="n">Instant</span> <span class="n">timestamp</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}})</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="k">as</span> <span class="nn">beam</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">ProcessRecord</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">,</span> <span class="n">timestamp</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">TimestampParam</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># access timestamp of element.</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">pass</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">MyDoFn</span><span class="p">(</span><span class="nx">ts</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span><span class="p">,</span> <span class="nx">word</span> <span class="kt">string</span><span class="p">)</span> <span class="kt">string</span> <span class="p">{</span> <span class="o">...</span> <span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kd">function</span> <span class="nx">processFn</span><span class="p">(</span><span class="nx">element</span><span class="p">,</span> <span class="nx">context</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">context</span><span class="p">.</span><span class="nx">timestamp</span><span class="p">.</span><span class="nx">lookup</span><span class="p">();</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="nx">pcoll</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">processFn</span><span class="p">,</span> <span class="p">{</span> <span class="nx">timestamp</span>: <span class="kt">pardo.timestampParam</span><span class="p">()</span> <span class="p">});</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java"><strong>Window:</strong> |
| To access the window an input element falls into, add a parameter of the type of the window used for the input <code>PCollection</code>. |
| If the parameter is a window type (a subclass of <code>BoundedWindow</code>) that does not match the input <code>PCollection</code>, then an error |
| will be raised. If an element falls in multiple windows (for example, this will happen when using <code>SlidingWindows</code>), then the |
| <code>@ProcessElement</code> method will be invoked multiple time for the element, once for each window. For example, when fixed windows |
| are being used, the window is of type <code>IntervalWindow</code>.</p> |
| <p class="language-py"><strong>Window:</strong> |
| To access the window an input element falls into, add a keyword parameter default to <code>DoFn.WindowParam</code>. |
| If an element falls in multiple windows (for example, this will happen when using <code>SlidingWindows</code>), then the |
| <code>process</code> method will be invoked multiple time for the element, once for each window.</p> |
| <p class="language-go"><strong>Window:</strong> |
| To access the window an input element falls into, add a <code>beam.Window</code> parameter before the element. |
| If an element falls in multiple windows (for example, this will happen when using SlidingWindows), |
| then the <code>ProcessElement</code> method will be invoked multiple time for the element, once for each window. |
| Since <code>beam.Window</code> is an interface it&rsquo;s possible to type assert to the concrete implementation of the window. |
| For example, when fixed windows are being used, the window is of type <code>window.IntervalWindow</code>.</p> |
| <p class="language-typescript"><strong>Window:</strong> |
| To access the window an input element falls into, add a <code>pardo.windowParam()</code> to the context argument. |
| If an element falls in multiple windows (for example, this will happen when using <code>SlidingWindows</code>), then the |
| function will be invoked multiple time for the element, once for each window.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="nd">@Element</span> <span class="n">String</span> <span class="n">word</span><span class="o">,</span> <span class="n">IntervalWindow</span> <span class="n">window</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}})</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="k">as</span> <span class="nn">beam</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">ProcessRecord</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">,</span> <span class="n">window</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">WindowParam</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># access window e.g. window.end.micros</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">pass</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">MyDoFn</span><span class="p">(</span><span class="nx">w</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Window</span><span class="p">,</span> <span class="nx">word</span> <span class="kt">string</span><span class="p">)</span> <span class="kt">string</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">iw</span> <span class="o">:=</span> <span class="nx">w</span><span class="p">.(</span><span class="nx">window</span><span class="p">.</span><span class="nx">IntervalWindow</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="nx">pcoll</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">processWithWindow</span><span class="p">,</span> <span class="p">{</span> <span class="nx">timestamp</span>: <span class="kt">pardo.windowParam</span><span class="p">()</span> <span class="p">});</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java"><strong>PaneInfo:</strong> |
| When triggers are used, Beam provides a <code>PaneInfo</code> object that contains information about the current firing. Using <code>PaneInfo</code> |
| you can determine whether this is an early or a late firing, and how many times this window has already fired for this key.</p> |
| <p class="language-py"><strong>PaneInfo:</strong> |
| When triggers are used, Beam provides a <code>DoFn.PaneInfoParam</code> object that contains information about the current firing. Using <code>DoFn.PaneInfoParam</code> |
| you can determine whether this is an early or a late firing, and how many times this window has already fired for this key. |
| This feature implementation in Python SDK is not fully completed; see more at <a href="https://github.com/apache/beam/issues/18721">Issue 17821</a>.</p> |
| <p class="language-go"><strong>PaneInfo:</strong> |
| When triggers are used, Beam provides <code>beam.PaneInfo</code> object that contains information about the current firing. Using <code>beam.PaneInfo</code> |
| you can determine whether this is an early or a late firing, and how many times this window has already fired for this key.</p> |
| <p class="language-typescript"><strong>Window:</strong> |
| To access the window an input element falls into, add a <code>pardo.paneInfoParam()</code> to the context argument. |
| Using <code>beam.PaneInfo</code> you can determine whether this is an early or a late firing, |
| and how many times this window has already fired for this key.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="nd">@Element</span> <span class="n">String</span> <span class="n">word</span><span class="o">,</span> <span class="n">PaneInfo</span> <span class="n">paneInfo</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}})</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="k">as</span> <span class="nn">beam</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">ProcessRecord</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">,</span> <span class="n">pane_info</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">PaneInfoParam</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># access pane info, e.g. pane_info.is_first, pane_info.is_last, pane_info.timing</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">pass</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">extractWordsFn</span><span class="p">(</span><span class="nx">pn</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PaneInfo</span><span class="p">,</span> <span class="nx">line</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emitWords</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">pn</span><span class="p">.</span><span class="nx">Timing</span> <span class="o">==</span> <span class="nx">typex</span><span class="p">.</span><span class="nx">PaneEarly</span> <span class="o">||</span> <span class="nx">pn</span><span class="p">.</span><span class="nx">Timing</span> <span class="o">==</span> <span class="nx">typex</span><span class="p">.</span><span class="nx">PaneOnTime</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... perform operation ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">pn</span><span class="p">.</span><span class="nx">Timing</span> <span class="o">==</span> <span class="nx">typex</span><span class="p">.</span><span class="nx">PaneLate</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... perform operation ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">pn</span><span class="p">.</span><span class="nx">IsFirst</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... perform operation ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">pn</span><span class="p">.</span><span class="nx">IsLast</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... perform operation ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nx">words</span> <span class="o">:=</span> <span class="nx">strings</span><span class="p">.</span><span class="nf">Split</span><span class="p">(</span><span class="nx">line</span><span class="p">,</span> <span class="s">&#34; &#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">w</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">words</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emitWords</span><span class="p">(</span><span class="nx">w</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="nx">pcoll</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">processWithPaneInfo</span><span class="p">,</span> <span class="p">{</span> <span class="nx">timestamp</span>: <span class="kt">pardo.paneInfoParam</span><span class="p">()</span> <span class="p">});</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java"><strong>PipelineOptions:</strong> |
| The <code>PipelineOptions</code> for the current pipeline can always be accessed in a process method by adding it |
| as a parameter:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="nd">@Element</span> <span class="n">String</span> <span class="n">word</span><span class="o">,</span> <span class="n">PipelineOptions</span> <span class="n">options</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}})</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java"><code>@OnTimer</code> methods can also access many of these parameters. Timestamp, Window, key, <code>PipelineOptions</code>, <code>OutputReceiver</code>, and |
| <code>MultiOutputReceiver</code> parameters can all be accessed in an <code>@OnTimer</code> method. In addition, an <code>@OnTimer</code> method can take |
| a parameter of type <code>TimeDomain</code> which tells whether the timer is based on event time or processing time. |
| Timers are explained in more detail in the |
| <a href="/blog/2017/08/28/timely-processing.html">Timely (and Stateful) Processing with Apache Beam</a> blog post.</p> |
| <p class="language-py"><strong>Timer and State:</strong> |
| In addition to aforementioned parameters, user defined Timer and State parameters can be used in a stateful DoFn. |
| Timers and States are explained in more detail in the |
| <a href="/blog/2017/08/28/timely-processing.html">Timely (and Stateful) Processing with Apache Beam</a> blog post.</p> |
| <p class="language-go"><strong>Timer and State:</strong> |
| User defined State and Timer parameters can be used in a stateful DoFn. |
| Timers and States are explained in more detail in the |
| <a href="/blog/2017/08/28/timely-processing.html">Timely (and Stateful) Processing with Apache Beam</a> blog post.</p> |
| <p class="language-typescript"><strong>Timer and State:</strong> |
| This feature isn&rsquo;t yet implemented in the Typescript SDK, |
| but we welcome <a href="/contribute/">contributions</a>. |
| In the meantime, Typescript pipelines wishing to use state and timers can do so |
| using <a href="#use-x-lang-transforms">cross-language transforms</a>.</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">StatefulDoFn</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;&#34;&#34;An example stateful DoFn with state and timer&#34;&#34;&#34;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">BUFFER_STATE_1</span> <span class="o">=</span> <span class="n">BagStateSpec</span><span class="p">(</span><span class="s1">&#39;buffer1&#39;</span><span class="p">,</span> <span class="n">beam</span><span class="o">.</span><span class="n">BytesCoder</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">BUFFER_STATE_2</span> <span class="o">=</span> <span class="n">BagStateSpec</span><span class="p">(</span><span class="s1">&#39;buffer2&#39;</span><span class="p">,</span> <span class="n">beam</span><span class="o">.</span><span class="n">VarIntCoder</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">WATERMARK_TIMER</span> <span class="o">=</span> <span class="n">TimerSpec</span><span class="p">(</span><span class="s1">&#39;watermark_timer&#39;</span><span class="p">,</span> <span class="n">TimeDomain</span><span class="o">.</span><span class="n">WATERMARK</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">element</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timestamp</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">TimestampParam</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">window</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">WindowParam</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer_1</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">BUFFER_STATE_1</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer_2</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">BUFFER_STATE_2</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">watermark_timer</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">TimerParam</span><span class="p">(</span><span class="n">WATERMARK_TIMER</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Do your processing here</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">key</span><span class="p">,</span> <span class="n">value</span> <span class="o">=</span> <span class="n">element</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Read all the data from buffer1</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">all_values_in_buffer_1</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">buffer_1</span><span class="o">.</span><span class="n">read</span><span class="p">()]</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">StatefulDoFn</span><span class="o">.</span><span class="n">_is_clear_buffer_1_required</span><span class="p">(</span><span class="n">all_values_in_buffer_1</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># clear the buffer data if required conditions are met.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer_1</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># add the value to buffer 2</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer_2</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">value</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">StatefulDoFn</span><span class="o">.</span><span class="n">_all_condition_met</span><span class="p">():</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Clear the timer if certain condition met and you don&#39;t want to trigger</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># the callback method.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">watermark_timer</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">element</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@on_timer</span><span class="p">(</span><span class="n">WATERMARK_TIMER</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">on_expiry_1</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timestamp</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">TimestampParam</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">window</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">WindowParam</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">key</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">KeyParam</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer_1</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">BUFFER_STATE_1</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer_2</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">BUFFER_STATE_2</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Window and key parameters are really useful especially for debugging issues.</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="s1">&#39;expired1&#39;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@staticmethod</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">_all_condition_met</span><span class="p">():</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># some logic</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">True</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@staticmethod</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">_is_clear_buffer_1_required</span><span class="p">(</span><span class="n">buffer_1_data</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Some business logic</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">True</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// stateAndTimersFn is an example stateful DoFn with state and a timer. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">type</span> <span class="nx">stateAndTimersFn</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Buffer1</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Bag</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Buffer2</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Bag</span><span class="p">[</span><span class="kt">int64</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Watermark</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">EventTime</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">s</span> <span class="o">*</span><span class="nx">stateAndTimersFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">sp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">w</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Window</span><span class="p">,</span> <span class="nx">key</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">value</span> <span class="kt">int64</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">,</span> <span class="kt">int64</span><span class="p">))</span> <span class="kt">error</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... handle processing elements here, set a callback timer... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Read all the data from Buffer1 in this window. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">vals</span><span class="p">,</span> <span class="nx">ok</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">s</span><span class="p">.</span><span class="nx">Buffer1</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">ok</span> <span class="o">&amp;&amp;</span> <span class="nx">s</span><span class="p">.</span><span class="nf">shouldClearBuffer</span><span class="p">(</span><span class="nx">vals</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// clear the buffer data if required conditions are met. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">s</span><span class="p">.</span><span class="nx">Buffer1</span><span class="p">.</span><span class="nf">Clear</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Add the value to Buffer2. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">s</span><span class="p">.</span><span class="nx">Buffer2</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="nx">sp</span><span class="p">,</span> <span class="nx">value</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">s</span><span class="p">.</span><span class="nf">allConditionsMet</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Clear the timer if certain condition met and you don&#39;t want to trigger |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// the callback method. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">s</span><span class="p">.</span><span class="nx">Watermark</span><span class="p">.</span><span class="nf">Clear</span><span class="p">(</span><span class="nx">tp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emit</span><span class="p">(</span><span class="nx">key</span><span class="p">,</span> <span class="nx">value</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">s</span> <span class="o">*</span><span class="nx">stateAndTimersFn</span><span class="p">)</span> <span class="nf">OnTimer</span><span class="p">(</span><span class="nx">sp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">w</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Window</span><span class="p">,</span> <span class="nx">key</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">timer</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">,</span> <span class="kt">int64</span><span class="p">))</span> <span class="kt">error</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Window and key parameters are really useful especially for debugging issues. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">switch</span> <span class="nx">timer</span><span class="p">.</span><span class="nx">Family</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">case</span> <span class="nx">s</span><span class="p">.</span><span class="nx">Watermark</span><span class="p">.</span><span class="nx">Family</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// timer expired, emit a different signal |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nf">emit</span><span class="p">(</span><span class="nx">key</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">s</span> <span class="o">*</span><span class="nx">stateAndTimersFn</span><span class="p">)</span> <span class="nf">shouldClearBuffer</span><span class="p">([]</span><span class="kt">string</span><span class="p">)</span> <span class="kt">bool</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// some business logic |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="kc">false</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">s</span> <span class="o">*</span><span class="nx">stateAndTimersFn</span><span class="p">)</span> <span class="nf">allConditionsMet</span><span class="p">()</span> <span class="kt">bool</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// other business logic |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="kc">true</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="c1">// Not yet implemented. |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="composite-transforms">4.6. Composite transforms</h3> |
| <p>Transforms can have a nested structure, where a complex transform performs |
| multiple simpler transforms (such as more than one <code>ParDo</code>, <code>Combine</code>, |
| <code>GroupByKey</code>, or even other composite transforms). These transforms are called |
| composite transforms. Nesting multiple transforms inside a single composite |
| transform can make your code more modular and easier to understand.</p> |
| <p>The Beam SDK comes packed with many useful composite transforms. See the API |
| reference pages for a list of transforms:</p> |
| <ul> |
| <li><a href="https://beam.apache.org/releases/javadoc/2.56.0/index.html?org/apache/beam/sdk/transforms/package-summary.html">Pre-written Beam transforms for Java</a></li> |
| <li><a href="https://beam.apache.org/releases/pydoc/2.56.0/apache_beam.transforms.html">Pre-written Beam transforms for Python</a></li> |
| <li><a href="https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/transforms">Pre-written Beam transforms for Go</a></li> |
| </ul> |
| <h4 id="composite-transform-example">4.6.1. An example composite transform</h4> |
| <p>The <code>CountWords</code> transform in the <a href="/get-started/wordcount-example/">WordCount example program</a> |
| is an example of a composite transform. <code>CountWords</code> is a <code>PTransform</code> |
| <span class="language-java language-py">subclass</span> that consists |
| of multiple nested transforms.</p> |
| <p><span class="language-java language-py">In its <code>expand</code> method, the</span> |
| <span class="language-go">The</span> <code>CountWords</code> transform applies the following |
| transform operations:</p> |
| <ol> |
| <li>It applies a <code>ParDo</code> on the input <code>PCollection</code> of text lines, producing |
| an output <code>PCollection</code> of individual words.</li> |
| <li>It applies the Beam SDK library transform <code>Count</code> on the <code>PCollection</code> of |
| words, producing a <code>PCollection</code> of key/value pairs. Each key represents a |
| word in the text, and each value represents the number of times that word |
| appeared in the original data.</li> |
| </ol> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">CountWords</span> <span class="kd">extends</span> <span class="n">PTransform</span><span class="o">&lt;</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;&gt;&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;&gt;</span> <span class="nf">expand</span><span class="o">(</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">lines</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Convert lines of text into individual words. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">words</span> <span class="o">=</span> <span class="n">lines</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">ExtractWordsFn</span><span class="o">()));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Count the number of times each word occurs. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;&gt;</span> <span class="n">wordCounts</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">words</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Count</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">perElement</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">wordCounts</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># The CountWords Composite Transform inside the WordCount pipeline.</span> |
| </span></span><span class="line"><span class="cl"><span class="nd">@beam.ptransform_fn</span> |
| </span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">CountWords</span><span class="p">(</span><span class="n">pcoll</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pcoll</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Convert lines of text into individual words.</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;ExtractWords&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">ExtractWordsFn</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Count the number of times each word occurs.</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">combiners</span><span class="o">.</span><span class="n">Count</span><span class="o">.</span><span class="n">PerElement</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Format each word and count into a printable string.</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;FormatCounts&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">FormatCountsFn</span><span class="p">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// CountWords is a function that builds a composite PTransform |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// to count the number of times each word appears. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="nf">CountWords</span><span class="p">(</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">lines</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span><span class="p">)</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// A subscope is required for a function to become a composite transform. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// We assign it to the original scope variable s to shadow the original |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// for the rest of the CountWords function. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">s</span> <span class="p">=</span> <span class="nx">s</span><span class="p">.</span><span class="nf">Scope</span><span class="p">(</span><span class="s">&#34;CountWords&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Since the same subscope is used for the following transforms, |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// they are in the same composite PTransform. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Convert lines of text into individual words. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">words</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">extractWordsFn</span><span class="p">,</span> <span class="nx">lines</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Count the number of times each word occurs. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">wordCounts</span> <span class="o">:=</span> <span class="nx">stats</span><span class="p">.</span><span class="nf">Count</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">words</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Return any PCollections that should be available after |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// the composite transform. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="nx">wordCounts</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kd">function</span> <span class="nx">countWords</span><span class="p">(</span><span class="nx">lines</span>: <span class="kt">PCollection</span><span class="p">&lt;</span><span class="nt">string</span><span class="p">&gt;)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">lines</span> <span class="c1">// |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="p">.</span><span class="nx">map</span><span class="p">((</span><span class="nx">s</span>: <span class="kt">string</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">s</span><span class="p">.</span><span class="nx">toLowerCase</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">flatMap</span><span class="p">(</span><span class="kd">function</span><span class="o">*</span> <span class="nx">splitWords</span><span class="p">(</span><span class="nx">line</span>: <span class="kt">string</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span><span class="o">*</span> <span class="nx">line</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="sr">/[^a-z]+/</span><span class="p">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">})</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">countPerElement</span><span class="p">());</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">counted</span> <span class="o">=</span> <span class="nx">lines</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">countWords</span><span class="p">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <blockquote> |
| <p><strong>Note:</strong> Because <code>Count</code> is itself a composite transform, |
| <code>CountWords</code> is also a nested composite transform.</p> |
| </blockquote> |
| <h4 id="composite-transform-creation">4.6.2. Creating a composite transform</h4> |
| <p class="language-typescript">A PTransform in the Typescript SDK is simply a function that accepts and |
| returns <code>PValue</code>s such as <code>PCollection</code>s.</p> |
| <p class="language-java language-py">To create your own composite transform, create a subclass of the <code>PTransform</code> |
| class and override the <code>expand</code> method to specify the actual processing logic. |
| You can then use this transform just as you would a built-in transform from the |
| Beam SDK.</p> |
| <p class="language-java">For the <code>PTransform</code> class type parameters, you pass the <code>PCollection</code> types |
| that your transform takes as input, and produces as output. To take multiple |
| <code>PCollection</code>s as input, or produce multiple <code>PCollection</code>s as output, use one |
| of the multi-collection types for the relevant type parameter.</p> |
| <p class="language-go">To create your own composite <code>PTransform</code> call the <code>Scope</code> method on the current |
| pipeline scope variable. Transforms passed this new sub-<code>Scope</code> will be a part of |
| the same composite <code>PTransform</code>.</p> |
| <p class="language-go">To be able to re-use your Composite, build it inside a normal Go function or method. |
| This function is passed a scope and input PCollections, and returns any |
| output PCollections it produces. <strong>Note:</strong> Such functions cannot be passed directly to |
| <code>ParDo</code> functions.</p> |
| <p class="language-java language-py">The following code sample shows how to declare a <code>PTransform</code> that accepts a |
| <code>PCollection</code> of <code>String</code>s for input, and outputs a <code>PCollection</code> of <code>Integer</code>s:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">ComputeWordLengths</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">extends</span> <span class="n">PTransform</span><span class="o">&lt;</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;,</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">ComputeWordLengths</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">PTransform</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">expand</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pcoll</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Transform logic goes here.</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">pcoll</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// CountWords is a function that builds a composite PTransform |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// to count the number of times each word appears. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="nf">CountWords</span><span class="p">(</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">lines</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span><span class="p">)</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// A subscope is required for a function to become a composite transform. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// We assign it to the original scope variable s to shadow the original |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// for the rest of the CountWords function. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">s</span> <span class="p">=</span> <span class="nx">s</span><span class="p">.</span><span class="nf">Scope</span><span class="p">(</span><span class="s">&#34;CountWords&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Since the same subscope is used for the following transforms, |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// they are in the same composite PTransform. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Convert lines of text into individual words. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">words</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">extractWordsFn</span><span class="p">,</span> <span class="nx">lines</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Count the number of times each word occurs. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">wordCounts</span> <span class="o">:=</span> <span class="nx">stats</span><span class="p">.</span><span class="nf">Count</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">words</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Return any PCollections that should be available after |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// the composite transform. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="nx">wordCounts</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java language-py">Within your <code>PTransform</code> subclass, you&rsquo;ll need to override the <code>expand</code> method. |
| The <code>expand</code> method is where you add the processing logic for the <code>PTransform</code>. |
| Your override of <code>expand</code> must accept the appropriate type of input |
| <code>PCollection</code> as a parameter, and specify the output <code>PCollection</code> as the return |
| value.</p> |
| <p class="language-java language-py">The following code sample shows how to override <code>expand</code> for the |
| <code>ComputeWordLengths</code> class declared in the previous example:</p> |
| <p class="language-go">The following code sample shows how to call the <code>CountWords</code> composite PTransform, |
| adding it to your pipeline:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">ComputeWordLengths</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">extends</span> <span class="n">PTransform</span><span class="o">&lt;</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;,</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="nf">expand</span><span class="o">(</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// transform logic goes here |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">ComputeWordLengths</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">PTransform</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">expand</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pcoll</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Transform logic goes here.</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">pcoll</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">lines</span> <span class="o">:=</span> <span class="o">...</span> <span class="c1">// a PCollection of strings. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"><span class="c1">// A Composite PTransform function is called like any other function. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">wordCounts</span> <span class="o">:=</span> <span class="nf">CountWords</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">lines</span><span class="p">)</span> <span class="c1">// returns a PCollection&lt;KV&lt;string,int&gt;&gt; |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java language-py">As long as you override the <code>expand</code> method in your <code>PTransform</code> subclass to |
| accept the appropriate input <code>PCollection</code>(s) and return the corresponding |
| output <code>PCollection</code>(s), you can include as many transforms as you want. These |
| transforms can include core transforms, composite transforms, or the transforms |
| included in the Beam SDK libraries.</p> |
| <p class="language-go">Your composite <code>PTransform</code>s can include as many transforms as you want. These |
| transforms can include core transforms, other composite transforms, or the transforms |
| included in the Beam SDK libraries. They can also consume and return as many |
| <code>PCollection</code>s as are necessary.</p> |
| <p class="language-java language-py">Your composite transform&rsquo;s parameters and return value must match the initial |
| input type and final return type for the entire transform, even if the |
| transform&rsquo;s intermediate data changes type multiple times.</p> |
| <p class="language-java language-py"><strong>Note:</strong> The <code>expand</code> method of a <code>PTransform</code> is not meant to be invoked |
| directly by the user of a transform. Instead, you should call the <code>apply</code> method |
| on the <code>PCollection</code> itself, with the transform as an argument. This allows |
| transforms to be nested within the structure of your pipeline.</p> |
| <h4 id="ptransform-style-guide">4.6.3. PTransform Style Guide</h4> |
| <p>The <a href="/contribute/ptransform-style-guide/">PTransform Style Guide</a> |
| contains additional information not included here, such as style guidelines, |
| logging and testing guidance, and language-specific considerations. The guide |
| is a useful starting point when you want to write new composite PTransforms.</p> |
| <h2 id="pipeline-io">5. Pipeline I/O</h2> |
| <p>When you create a pipeline, you often need to read data from some external |
| source, such as a file or a database. Likewise, you may |
| want your pipeline to output its result data to an external storage system. |
| Beam provides read and write transforms for a <a href="/documentation/io/built-in/">number of common data storage |
| types</a>. If you want your pipeline |
| to read from or write to a data storage format that isn&rsquo;t supported by the |
| built-in transforms, you can <a href="/documentation/io/developing-io-overview/">implement your own read and write |
| transforms</a>.</p> |
| <h3 id="pipeline-io-reading-data">5.1. Reading input data</h3> |
| <p>Read transforms read data from an external source and return a <code>PCollection</code> |
| representation of the data for use by your pipeline. You can use a read |
| transform at any point while constructing your pipeline to create a new |
| <code>PCollection</code>, though it will be most common at the start of your pipeline.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">lines</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">TextIO</span><span class="o">.</span><span class="na">read</span><span class="o">().</span><span class="na">from</span><span class="o">(</span><span class="s">&#34;gs://some/inputData.txt&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">lines</span> <span class="o">=</span> <span class="n">pipeline</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ReadFromText</span><span class="p">(</span><span class="s1">&#39;gs://some/inputData.txt&#39;</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">lines</span> <span class="o">:=</span> <span class="nx">textio</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="err">&#39;</span><span class="nx">gs</span><span class="p">:</span><span class="c1">//some/inputData.txt&#39;) |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="pipeline-io-writing-data">5.2. Writing output data</h3> |
| <p>Write transforms write the data in a <code>PCollection</code> to an external data source. |
| You will most often use write transforms at the end of your pipeline to output |
| your pipeline&rsquo;s final results. However, you can use a write transform to output |
| a <code>PCollection</code>&rsquo;s data at any point in your pipeline.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">output</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">TextIO</span><span class="o">.</span><span class="na">write</span><span class="o">().</span><span class="na">to</span><span class="o">(</span><span class="s">&#34;gs://some/outputData&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">output</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">WriteToText</span><span class="p">(</span><span class="s1">&#39;gs://some/outputData&#39;</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">textio</span><span class="p">.</span><span class="nf">Write</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="err">&#39;</span><span class="nx">gs</span><span class="p">:</span><span class="c1">//some/inputData.txt&#39;, output) |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="file-based-data">5.3. File-based input and output data</h3> |
| <h4 id="file-based-reading-multiple-locations">5.3.1. Reading from multiple locations</h4> |
| <p>Many read transforms support reading from multiple input files matching a glob |
| operator you provide. Note that glob operators are filesystem-specific and obey |
| filesystem-specific consistency models. The following TextIO example uses a glob |
| operator (<code>*</code>) to read all matching input files that have prefix &ldquo;input-&rdquo; and the |
| suffix &ldquo;.csv&rdquo; in the given location:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;ReadFromText&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TextIO</span><span class="o">.</span><span class="na">read</span><span class="o">().</span><span class="na">from</span><span class="o">(</span><span class="s">&#34;protocol://my_bucket/path/to/input-*.csv&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">lines</span> <span class="o">=</span> <span class="n">pipeline</span> <span class="o">|</span> <span class="s1">&#39;ReadFromText&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ReadFromText</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;path/to/input-*.csv&#39;</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">lines</span> <span class="o">:=</span> <span class="nx">textio</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="s">&#34;path/to/input-*.csv&#34;</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>To read data from disparate sources into a single <code>PCollection</code>, read each one |
| independently and then use the <a href="#flatten">Flatten</a> transform to create a single |
| <code>PCollection</code>.</p> |
| <h4 id="file-based-writing-multiple-files">5.3.2. Writing to multiple output files</h4> |
| <p>For file-based output data, write transforms write to multiple output files by |
| default. When you pass an output file name to a write transform, the file name |
| is used as the prefix for all output files that the write transform produces. |
| You can append a suffix to each output file by specifying a suffix.</p> |
| <p>The following write transform example writes multiple output files to a |
| location. Each file has the prefix &ldquo;numbers&rdquo;, a numeric tag, and the suffix |
| &ldquo;.csv&rdquo;.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">records</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;WriteToText&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TextIO</span><span class="o">.</span><span class="na">write</span><span class="o">().</span><span class="na">to</span><span class="o">(</span><span class="s">&#34;protocol://my_bucket/path/to/numbers&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withSuffix</span><span class="o">(</span><span class="s">&#34;.csv&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">filtered_words</span> <span class="o">|</span> <span class="s1">&#39;WriteToText&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">WriteToText</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;/path/to/numbers&#39;</span><span class="p">,</span> <span class="n">file_name_suffix</span><span class="o">=</span><span class="s1">&#39;.csv&#39;</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// The Go SDK textio doesn&#39;t support sharding on writes yet. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// See https://github.com/apache/beam/issues/21031 for ways |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// to contribute a solution. |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="provided-io-transforms">5.4. Beam-provided I/O transforms</h3> |
| <p>See the <a href="/documentation/io/built-in/">Beam-provided I/O Transforms</a> |
| page for a list of the currently available I/O transforms.</p> |
| <h2 id="schemas">6. Schemas</h2> |
| <p>Often, the types of the records being processed have an obvious structure. Common Beam sources produce |
| JSON, Avro, Protocol Buffer, or database row objects; all of these types have well defined structures, |
| structures that can often be determined by examining the type. Even within a SDK pipeline, Simple Java POJOs |
| (or equivalent structures in other languages) are often used as intermediate types, and these also have a |
| clear structure that can be inferred by inspecting the class. By understanding the structure of a pipeline’s |
| records, we can provide much more concise APIs for data processing.</p> |
| <h3 id="what-is-a-schema">6.1. What is a schema?</h3> |
| <p>Most structured records share some common characteristics:</p> |
| <ul> |
| <li>They can be subdivided into separate named fields. Fields usually have string names, but sometimes - as in the case of indexed |
| tuples - have numerical indices instead.</li> |
| <li>There is a confined list of primitive types that a field can have. These often match primitive types in most programming |
| languages: int, long, string, etc.</li> |
| <li>Often a field type can be marked as optional (sometimes referred to as nullable) or required.</li> |
| </ul> |
| <p>Often records have a nested structure. A nested structure occurs when a field itself has subfields so the |
| type of the field itself has a schema. Fields that are array or map types is also a common feature of these structured |
| records.</p> |
| <p>For example, consider the following schema, representing actions in a fictitious e-commerce company:</p> |
| <p><strong>Purchase</strong></p> |
| <div class="table-container-wrapper"> |
| <table class="table-wrapper--pr"> |
| <thead> |
| <tr class="header"> |
| <th><b>Field Name</b></th> |
| <th><b>Field Type</b></th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>userId</td> |
| <td>STRING</td> |
| </tr> |
| <tr> |
| <td>itemId</td> |
| <td>INT64</td> |
| </tr> |
| <tr> |
| <td>shippingAddress</td> |
| <td>ROW(ShippingAddress)</td> |
| </tr> |
| <tr> |
| <td>cost</td> |
| <td>INT64</td> |
| </tr> |
| <tr> |
| <td>transactions</td> |
| <td>ARRAY[ROW(Transaction)]</td> |
| </tr> |
| </tbody> |
| </table> |
| <br/> |
| </div> |
| <p><strong>ShippingAddress</strong></p> |
| <table class="table-wrapper--pr"> |
| <thead> |
| <tr class="header"> |
| <th><b>Field Name</b></th> |
| <th><b>Field Type</b></th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>streetAddress</td> |
| <td>STRING</td> |
| </tr> |
| <tr> |
| <td>city</td> |
| <td>STRING</td> |
| </tr> |
| <tr> |
| <td>state</td> |
| <td>nullable STRING</td> |
| </tr> |
| <tr> |
| <td>country</td> |
| <td>STRING</td> |
| </tr> |
| <tr> |
| <td>postCode</td> |
| <td>STRING</td> |
| </tr> |
| </tbody> |
| </table> |
| <br/> |
| <p><strong>Transaction</strong></p> |
| <table class="table-wrapper--pr"> |
| <thead> |
| <tr class="header"> |
| <th><b>Field Name</b></th> |
| <th><b>Field Type</b></th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>bank</td> |
| <td>STRING</td> |
| </tr> |
| <tr> |
| <td>purchaseAmount</td> |
| <td>DOUBLE</td> |
| </tr> |
| </tbody> |
| </table> |
| <br/> |
| <p>Purchase event records are represented by the above purchase schema. Each purchase event contains a shipping address, which |
| is a nested row containing its own schema. Each purchase also contains an array of credit-card transactions |
| (a list, because a purchase might be split across multiple credit cards); each item in the transaction list is a row |
| with its own schema.</p> |
| <p>This provides an abstract description of the types involved, one that is abstracted away from any specific programming |
| language.</p> |
| <p>Schemas provide us a type-system for Beam records that is independent of any specific programming-language type. There |
| might be multiple Java classes that all have the same schema (for example a Protocol-Buffer class or a POJO class), |
| and Beam will allow us to seamlessly convert between these types. Schemas also provide a simple way to reason about |
| types across different programming-language APIs.</p> |
| <p>A <code>PCollection</code> with a schema does not need to have a <code>Coder</code> specified, as Beam knows how to encode and decode |
| Schema rows; Beam uses a special coder to encode schema types.</p> |
| <h3 id="schemas-for-pl-types">6.2. Schemas for programming language types</h3> |
| <p>While schemas themselves are language independent, they are designed to embed naturally into the programming languages |
| of the Beam SDK being used. This allows Beam users to continue using native types while reaping the advantage of |
| having Beam understand their element schemas.</p> |
| <p class="language-java">In Java you could use the following set of classes to represent the purchase schema. Beam will automatically |
| infer the correct schema based on the members of the class.</p> |
| <p class="language-py">In Python you can use the following set of classes to represent the purchase schema. Beam will automatically infer the correct schema based on the members of the class.</p> |
| <p class="language-go">In Go, schema encoding is used by default for struct types, with Exported fields becoming part of the schema. |
| Beam will automatically infer the schema based on the fields and field tags of the struct, and their order.</p> |
| <p class="language-typescript">In Typescript, JSON objects are used to represent schema&rsquo;d data. |
| Unfortunately type information in Typescript is not propagated to the runtime layer, |
| so it needs to be manually specified in some places (e.g. when using cross-language pipelines).</p> |
| <p class="language-yaml">In Beam YAML, all transforms produce and accept schema&rsquo;d data which is used to validate the pipeline.</p> |
| <p class="language-yaml">In some cases, Beam is unable to figure out the output type of a mapping function. |
| In this case, you can specify it manually using |
| <a href="https://json-schema.org/understanding-json-schema/reference/type">JSON schema syntax</a>.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@DefaultSchema</span><span class="o">(</span><span class="n">JavaBeanSchema</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">Purchase</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">String</span> <span class="nf">getUserId</span><span class="o">();</span> <span class="c1">// Returns the id of the user who made the purchase. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kd">public</span> <span class="kt">long</span> <span class="nf">getItemId</span><span class="o">();</span> <span class="c1">// Returns the identifier of the item that was purchased. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kd">public</span> <span class="n">ShippingAddress</span> <span class="nf">getShippingAddress</span><span class="o">();</span> <span class="c1">// Returns the shipping address, a nested type. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kd">public</span> <span class="kt">long</span> <span class="nf">getCostCents</span><span class="o">();</span> <span class="c1">// Returns the cost of the item. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kd">public</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">&gt;</span> <span class="nf">getTransactions</span><span class="o">();</span> <span class="c1">// Returns the transactions that paid for this purchase (returns a list, since the purchase might be spread out over multiple credit cards). |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@SchemaCreate</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="nf">Purchase</span><span class="o">(</span><span class="n">String</span> <span class="n">userId</span><span class="o">,</span> <span class="kt">long</span> <span class="n">itemId</span><span class="o">,</span> <span class="n">ShippingAddress</span> <span class="n">shippingAddress</span><span class="o">,</span> <span class="kt">long</span> <span class="n">costCents</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">List</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">&gt;</span> <span class="n">transactions</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="nd">@DefaultSchema</span><span class="o">(</span><span class="n">JavaBeanSchema</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">ShippingAddress</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">String</span> <span class="nf">getStreetAddress</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">String</span> <span class="nf">getCity</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Nullable</span> <span class="kd">public</span> <span class="n">String</span> <span class="nf">getState</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">String</span> <span class="nf">getCountry</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">String</span> <span class="nf">getPostCode</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@SchemaCreate</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="nf">ShippingAddress</span><span class="o">(</span><span class="n">String</span> <span class="n">streetAddress</span><span class="o">,</span> <span class="n">String</span> <span class="n">city</span><span class="o">,</span> <span class="nd">@Nullable</span> <span class="n">String</span> <span class="n">state</span><span class="o">,</span> <span class="n">String</span> <span class="n">country</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">String</span> <span class="n">postCode</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="nd">@DefaultSchema</span><span class="o">(</span><span class="n">JavaBeanSchema</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">Transaction</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">String</span> <span class="nf">getBank</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">double</span> <span class="nf">getPurchaseAmount</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@SchemaCreate</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="nf">Transaction</span><span class="o">(</span><span class="n">String</span> <span class="n">bank</span><span class="o">,</span> <span class="kt">double</span> <span class="n">purchaseAmount</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">typing</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">Purchase</span><span class="p">(</span><span class="n">typing</span><span class="o">.</span><span class="n">NamedTuple</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">user_id</span><span class="p">:</span> <span class="nb">str</span> <span class="c1"># The id of the user who made the purchase.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">item_id</span><span class="p">:</span> <span class="nb">int</span> <span class="c1"># The identifier of the item that was purchased.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">shipping_address</span><span class="p">:</span> <span class="n">ShippingAddress</span> <span class="c1"># The shipping address, a nested type.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">cost_cents</span><span class="p">:</span> <span class="nb">int</span> <span class="c1"># The cost of the item</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">transactions</span><span class="p">:</span> <span class="n">typing</span><span class="o">.</span><span class="n">Sequence</span><span class="p">[</span><span class="n">Transaction</span><span class="p">]</span> <span class="c1"># The transactions that paid for this purchase (a list, since the purchase might be spread out over multiple credit cards).</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">ShippingAddress</span><span class="p">(</span><span class="n">typing</span><span class="o">.</span><span class="n">NamedTuple</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">street_address</span><span class="p">:</span> <span class="nb">str</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">city</span><span class="p">:</span> <span class="nb">str</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state</span><span class="p">:</span> <span class="n">typing</span><span class="o">.</span><span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">country</span><span class="p">:</span> <span class="nb">str</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">postal_code</span><span class="p">:</span> <span class="nb">str</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">Transaction</span><span class="p">(</span><span class="n">typing</span><span class="o">.</span><span class="n">NamedTuple</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">bank</span><span class="p">:</span> <span class="nb">str</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">purchase_amount</span><span class="p">:</span> <span class="nb">float</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">Purchase</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ID of the user who made the purchase. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">UserID</span> <span class="kt">string</span> <span class="s">`beam:&#34;userId&#34;`</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Identifier of the item that was purchased. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">ItemID</span> <span class="kt">int64</span> <span class="s">`beam:&#34;itemId&#34;`</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The shipping address, a nested type. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">ShippingAddress</span> <span class="nx">ShippingAddress</span> <span class="s">`beam:&#34;shippingAddress&#34;`</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The cost of the item in cents. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">Cost</span> <span class="kt">int64</span> <span class="s">`beam:&#34;cost&#34;`</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The transactions that paid for this purchase. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// A slice since the purchase might be spread out over multiple |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// credit cards. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">Transactions</span> <span class="p">[]</span><span class="nx">Transaction</span> <span class="s">`beam:&#34;transactions&#34;`</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">ShippingAddress</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">StreetAddress</span> <span class="kt">string</span> <span class="s">`beam:&#34;streetAddress&#34;`</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">City</span> <span class="kt">string</span> <span class="s">`beam:&#34;city&#34;`</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">State</span> <span class="o">*</span><span class="kt">string</span> <span class="s">`beam:&#34;state&#34;`</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Country</span> <span class="kt">string</span> <span class="s">`beam:&#34;country&#34;`</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">PostCode</span> <span class="kt">string</span> <span class="s">`beam:&#34;postCode&#34;`</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">Transaction</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Bank</span> <span class="kt">string</span> <span class="s">`beam:&#34;bank&#34;`</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">PurchaseAmount</span> <span class="kt">float64</span> <span class="s">`beam:&#34;purchaseAmount&#34;`</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">pcoll</span> <span class="o">=</span> <span class="nx">root</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">([</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">intField</span>: <span class="kt">1</span><span class="p">,</span> <span class="nx">stringField</span><span class="o">:</span> <span class="s2">&#34;a&#34;</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">intField</span>: <span class="kt">2</span><span class="p">,</span> <span class="nx">stringField</span><span class="o">:</span> <span class="s2">&#34;b&#34;</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">])</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Let beam know the type of the elements by providing an exemplar. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">withRowCoder</span><span class="p">({</span> <span class="nx">intField</span>: <span class="kt">0</span><span class="p">,</span> <span class="nx">stringField</span><span class="o">:</span> <span class="s2">&#34;&#34;</span> <span class="p">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">MapToFields</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">language</span><span class="p">:</span><span class="w"> </span><span class="l">python</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">fields</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">new_field</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">expression</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;hex(weight)&#34;</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">output_type</span><span class="p">:</span><span class="w"> </span>{<span class="w"> </span><span class="nt">&#34;type&#34;: </span><span class="s2">&#34;string&#34;</span><span class="w"> </span>}</span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">Using JavaBean classes as above is one way to map a schema to Java classes. However multiple Java classes might have |
| the same schema, in which case the different Java types can often be used interchangeably. Beam will add implicit |
| conversions between types that have matching schemas. For example, the above |
| <code>Transaction</code> class has the same schema as the following class:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@DefaultSchema</span><span class="o">(</span><span class="n">JavaFieldSchema</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">TransactionPojo</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">String</span> <span class="n">bank</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">double</span> <span class="n">purchaseAmount</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">So if we had two <code>PCollection</code>s as follows</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">&gt;</span> <span class="n">transactionBeans</span> <span class="o">=</span> <span class="n">readTransactionsAsJavaBean</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">TransactionPojos</span><span class="o">&gt;</span> <span class="n">transactionPojos</span> <span class="o">=</span> <span class="n">readTransactionsAsPojo</span><span class="o">();</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">Then these two <code>PCollection</code>s would have the same schema, even though their Java types would be different. This means |
| for example the following two code snippets are valid:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">transactionBeans</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;...&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span><span class="nd">@Element</span> <span class="n">TransactionPojo</span> <span class="n">pojo</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">and</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">transactionPojos</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;...&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span><span class="nd">@Element</span> <span class="n">Transaction</span> <span class="n">row</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">Even though the in both cases the <code>@Element</code> parameter differs from the <code>PCollection</code>&rsquo;s Java type, since the |
| schemas are the same Beam will automatically make the conversion. The built-in <code>Convert</code> transform can also be used |
| to translate between Java types of equivalent schemas, as detailed below.</p> |
| <h3 id="schema-definition">6.3. Schema definition</h3> |
| <p>The schema for a <code>PCollection</code> defines elements of that <code>PCollection</code> as an ordered list of named fields. Each field |
| has a name, a type, and possibly a set of user options. The type of a field can be primitive or composite. The following |
| are the primitive types currently supported by Beam:</p> |
| <table class="table-wrapper--pr"> |
| <thead> |
| <tr class="header"> |
| <th>Type</th> |
| <th>Description</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>BYTE</td> |
| <td>An 8-bit signed value</td> |
| </tr> |
| <tr> |
| <td>INT16</td> |
| <td>A 16-bit signed value</td> |
| </tr> |
| <tr> |
| <td>INT32</td> |
| <td>A 32-bit signed value</td> |
| </tr> |
| <tr> |
| <td>INT64</td> |
| <td>A 64-bit signed value</td> |
| </tr> |
| <tr> |
| <td>DECIMAL</td> |
| <td>An arbitrary-precision decimal type</td> |
| </tr> |
| <tr> |
| <td>FLOAT</td> |
| <td>A 32-bit IEEE 754 floating point number</td> |
| </tr> |
| <tr> |
| <td>DOUBLE</td> |
| <td>A 64-bit IEEE 754 floating point number</td> |
| </tr> |
| <tr> |
| <td>STRING</td> |
| <td>A string</td> |
| </tr> |
| <tr> |
| <td>DATETIME</td> |
| <td>A timestamp represented as milliseconds since the epoch</td> |
| </tr> |
| <tr> |
| <td>BOOLEAN</td> |
| <td>A boolean value</td> |
| </tr> |
| <tr> |
| <td>BYTES</td> |
| <td>A raw byte array</td> |
| </tr> |
| </tbody> |
| </table> |
| <br/> |
| <p>A field can also reference a nested schema. In this case, the field will have type ROW, and the nested schema will |
| be an attribute of this field type.</p> |
| <p>Three collection types are supported as field types: ARRAY, ITERABLE and MAP:</p> |
| <ul> |
| <li><strong>ARRAY</strong> This represents a repeated value type, where the repeated elements can have any supported type. Arrays of |
| nested rows are supported, as are arrays of arrays.</li> |
| <li><strong>ITERABLE</strong> This is very similar to the array type, it represents a repeated value, but one in which the full list of |
| items is not known until iterated over. This is intended for the case where an iterable might be larger than the |
| available memory, and backed by external storage (for example, this can happen with the iterable returned by a |
| <code>GroupByKey</code>). The repeated elements can have any supported type.</li> |
| <li><strong>MAP</strong> This represents an associative map from keys to values. All schema types are supported for both keys and values. |
| Values that contain map types cannot be used as keys in any grouping operation.</li> |
| </ul> |
| <h3 id="logical-types">6.4. Logical types</h3> |
| <p>Users can extend the schema type system to add custom logical types that can be used as a field. A logical type is |
| identified by a unique identifier and an argument. A logical type also specifies an underlying schema type to be used |
| for storage, along with conversions to and from that type. As an example, a logical union can always be represented as |
| a row with nullable fields, where the user ensures that only one of those fields is ever set at a time. However this can |
| be tedious and complex to manage. The OneOf logical type provides a value class that makes it easier to manage the type |
| as a union, while still using a row with nullable fields as its underlying storage. Each logical type also has a |
| unique identifier, so they can be interpreted by other languages as well. More examples of logical types are listed |
| below.</p> |
| <h4 id="defining-a-logical-type">6.4.1. Defining a logical type</h4> |
| <p>To define a logical type you must specify a Schema type to be used to represent the underlying type as well as a unique |
| identifier for that type. A logical type imposes additional semantics on top a schema type. For example, a logical |
| type to represent nanosecond timestamps is represented as a schema containing an INT64 and an INT32 field. This schema |
| alone does not say anything about how to interpret this type, however the logical type tells you that this represents |
| a nanosecond timestamp, with the INT64 field representing seconds and the INT32 field representing nanoseconds.</p> |
| <p>Logical types are also specified by an argument, which allows creating a class of related types. For example, a |
| limited-precision decimal type would have an integer argument indicating how many digits of precision are represented. |
| The argument is represented by a schema type, so can itself be a complex type.</p> |
| <p class="language-java">In Java, a logical type is specified as a subclass of the <code>LogicalType</code> class. A custom Java class can be specified to represent the logical type and conversion functions must be supplied to convert back and forth between this Java class and the underlying Schema type representation. For example, the logical type representing nanosecond timestamp might be implemented as follows</p> |
| <p class="language-go">In Go, a logical type is specified with a custom implementation of the <code>beam.SchemaProvider</code> interface. |
| For example, the logical type provider representing nanosecond timestamps |
| might be implemented as follows</p> |
| <p class="language-typescript">In Typescript, a logical type defined by the <a href="https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/coders/row_coder.ts">LogicalTypeInfo</a> |
| interface which associates a logical type&rsquo;s URN with its representation |
| and its conversion to and from this representation.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// A Logical type using java.time.Instant to represent the logical type. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">public</span> <span class="kd">class</span> <span class="nc">TimestampNanos</span> <span class="kd">implements</span> <span class="n">LogicalType</span><span class="o">&lt;</span><span class="n">Instant</span><span class="o">,</span> <span class="n">Row</span><span class="o">&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The underlying schema used to represent rows. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">Schema</span> <span class="n">SCHEMA</span> <span class="o">=</span> <span class="n">Schema</span><span class="o">.</span><span class="na">builder</span><span class="o">().</span><span class="na">addInt64Field</span><span class="o">(</span><span class="s">&#34;seconds&#34;</span><span class="o">).</span><span class="na">addInt32Field</span><span class="o">(</span><span class="s">&#34;nanos&#34;</span><span class="o">).</span><span class="na">build</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> <span class="kd">public</span> <span class="n">String</span> <span class="nf">getIdentifier</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="s">&#34;timestampNanos&#34;</span><span class="o">;</span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> <span class="kd">public</span> <span class="n">FieldType</span> <span class="nf">getBaseType</span><span class="o">()</span> <span class="o">{</span> <span class="k">return</span> <span class="n">schema</span><span class="o">;</span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Convert the representation type to the underlying Row type. Called by Beam when necessary. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@Override</span> <span class="kd">public</span> <span class="n">Row</span> <span class="nf">toBaseType</span><span class="o">(</span><span class="n">Instant</span> <span class="n">instant</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">Row</span><span class="o">.</span><span class="na">withSchema</span><span class="o">(</span><span class="n">schema</span><span class="o">).</span><span class="na">addValues</span><span class="o">(</span><span class="n">instant</span><span class="o">.</span><span class="na">getEpochSecond</span><span class="o">(),</span> <span class="n">instant</span><span class="o">.</span><span class="na">getNano</span><span class="o">()).</span><span class="na">build</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Convert the underlying Row type to an Instant. Called by Beam when necessary. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@Override</span> <span class="kd">public</span> <span class="n">Instant</span> <span class="nf">toInputType</span><span class="o">(</span><span class="n">Row</span> <span class="n">base</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">Instant</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">row</span><span class="o">.</span><span class="na">getInt64</span><span class="o">(</span><span class="s">&#34;seconds&#34;</span><span class="o">),</span> <span class="n">row</span><span class="o">.</span><span class="na">getInt32</span><span class="o">(</span><span class="s">&#34;nanos&#34;</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// Define a logical provider like so: |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// TimestampNanos is a logical type using time.Time, but |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// encodes as a schema type. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">type</span> <span class="nx">TimestampNanos</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Time</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">tn</span> <span class="nx">TimestampNanos</span><span class="p">)</span> <span class="nf">Seconds</span><span class="p">()</span> <span class="kt">int64</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">time</span><span class="p">.</span><span class="nf">Time</span><span class="p">(</span><span class="nx">tn</span><span class="p">).</span><span class="nf">Unix</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">tn</span> <span class="nx">TimestampNanos</span><span class="p">)</span> <span class="nf">Nanos</span><span class="p">()</span> <span class="kt">int32</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nb">int32</span><span class="p">(</span><span class="nx">time</span><span class="p">.</span><span class="nf">Time</span><span class="p">(</span><span class="nx">tn</span><span class="p">).</span><span class="nf">UnixNano</span><span class="p">()</span> <span class="o">%</span> <span class="mi">1000000000</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// tnStorage is the storage schema for TimestampNanos. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">type</span> <span class="nx">tnStorage</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Seconds</span> <span class="kt">int64</span> <span class="s">`beam:&#34;seconds&#34;`</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Nanos</span> <span class="kt">int32</span> <span class="s">`beam:&#34;nanos&#34;`</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">var</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// reflect.Type of the Value type of TimestampNanos |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">tnType</span> <span class="p">=</span> <span class="nx">reflect</span><span class="p">.</span><span class="nf">TypeOf</span><span class="p">((</span><span class="o">*</span><span class="nx">TimestampNanos</span><span class="p">)(</span><span class="kc">nil</span><span class="p">)).</span><span class="nf">Elem</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">tnStorageType</span> <span class="p">=</span> <span class="nx">reflect</span><span class="p">.</span><span class="nf">TypeOf</span><span class="p">((</span><span class="o">*</span><span class="nx">tnStorage</span><span class="p">)(</span><span class="kc">nil</span><span class="p">)).</span><span class="nf">Elem</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// TimestampNanosProvider implements the beam.SchemaProvider interface. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">type</span> <span class="nx">TimestampNanosProvider</span> <span class="kd">struct</span><span class="p">{}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// FromLogicalType converts checks if the given type is TimestampNanos, and if so |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// returns the storage type. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">p</span> <span class="o">*</span><span class="nx">TimestampNanosProvider</span><span class="p">)</span> <span class="nf">FromLogicalType</span><span class="p">(</span><span class="nx">rt</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">Type</span><span class="p">)</span> <span class="p">(</span><span class="nx">reflect</span><span class="p">.</span><span class="nx">Type</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">rt</span> <span class="o">!=</span> <span class="nx">tnType</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">fmt</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;unable to provide schema.LogicalType for type %v, want %v&#34;</span><span class="p">,</span> <span class="nx">rt</span><span class="p">,</span> <span class="nx">tnType</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">tnStorageType</span><span class="p">,</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// BuildEncoder builds a Beam schema encoder for the TimestampNanos type. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">p</span> <span class="o">*</span><span class="nx">TimestampNanosProvider</span><span class="p">)</span> <span class="nf">BuildEncoder</span><span class="p">(</span><span class="nx">rt</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">Type</span><span class="p">)</span> <span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">any</span><span class="p">,</span> <span class="nx">io</span><span class="p">.</span><span class="nx">Writer</span><span class="p">)</span> <span class="kt">error</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">p</span><span class="p">.</span><span class="nf">FromLogicalType</span><span class="p">(</span><span class="nx">rt</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">enc</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">coder</span><span class="p">.</span><span class="nf">RowEncoderForStruct</span><span class="p">(</span><span class="nx">tnStorageType</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kd">func</span><span class="p">(</span><span class="nx">iface</span> <span class="nx">any</span><span class="p">,</span> <span class="nx">w</span> <span class="nx">io</span><span class="p">.</span><span class="nx">Writer</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">v</span> <span class="o">:=</span> <span class="nx">iface</span><span class="p">.(</span><span class="nx">TimestampNanos</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nf">enc</span><span class="p">(</span><span class="nx">tnStorage</span><span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Seconds</span><span class="p">:</span> <span class="nx">v</span><span class="p">.</span><span class="nf">Seconds</span><span class="p">(),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Nanos</span><span class="p">:</span> <span class="nx">v</span><span class="p">.</span><span class="nf">Nanos</span><span class="p">(),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="nx">w</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// BuildDecoder builds a Beam schema decoder for the TimestampNanos type. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">p</span> <span class="o">*</span><span class="nx">TimestampNanosProvider</span><span class="p">)</span> <span class="nf">BuildDecoder</span><span class="p">(</span><span class="nx">rt</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">Type</span><span class="p">)</span> <span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">io</span><span class="p">.</span><span class="nx">Reader</span><span class="p">)</span> <span class="p">(</span><span class="nx">any</span><span class="p">,</span> <span class="kt">error</span><span class="p">),</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">p</span><span class="p">.</span><span class="nf">FromLogicalType</span><span class="p">(</span><span class="nx">rt</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">dec</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">coder</span><span class="p">.</span><span class="nf">RowDecoderForStruct</span><span class="p">(</span><span class="nx">tnStorageType</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kd">func</span><span class="p">(</span><span class="nx">r</span> <span class="nx">io</span><span class="p">.</span><span class="nx">Reader</span><span class="p">)</span> <span class="p">(</span><span class="nx">any</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">s</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nf">dec</span><span class="p">(</span><span class="nx">r</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">tn</span> <span class="o">:=</span> <span class="nx">s</span><span class="p">.(</span><span class="nx">tnStorage</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nf">TimestampNanos</span><span class="p">(</span><span class="nx">time</span><span class="p">.</span><span class="nf">Unix</span><span class="p">(</span><span class="nx">tn</span><span class="p">.</span><span class="nx">Seconds</span><span class="p">,</span> <span class="nb">int64</span><span class="p">(</span><span class="nx">tn</span><span class="p">.</span><span class="nx">Nanos</span><span class="p">))),</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Register it like so: |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"><span class="nx">beam</span><span class="p">.</span><span class="nf">RegisterSchemaProvider</span><span class="p">(</span><span class="nx">tnType</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">TimestampNanosProvider</span><span class="p">{})</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="c1">// Register a logical type: |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"><span class="kr">class</span> <span class="nx">Foo</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kr">constructor</span><span class="p">(</span><span class="kr">public</span> <span class="nx">value</span>: <span class="kt">string</span><span class="p">)</span> <span class="p">{}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="nx">requireForSerialization</span><span class="p">(</span><span class="s2">&#34;apache-beam&#34;</span><span class="p">,</span> <span class="p">{</span> <span class="nx">Foo</span> <span class="p">});</span> |
| </span></span><span class="line"><span class="cl"><span class="nx">row_coder</span><span class="p">.</span><span class="nx">registerLogicalType</span><span class="p">({</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">urn</span><span class="o">:</span> <span class="s2">&#34;beam:logical_type:typescript_foo:v1&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">reprType</span>: <span class="kt">row_coder.RowCoder.inferTypeFromJSON</span><span class="p">(</span><span class="s2">&#34;string&#34;</span><span class="p">,</span> <span class="kc">false</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">toRepr</span><span class="o">:</span> <span class="p">(</span><span class="nx">foo</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="nx">foo</span><span class="p">.</span><span class="nx">value</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fromRepr</span><span class="o">:</span> <span class="p">(</span><span class="nx">value</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="k">new</span> <span class="nx">Foo</span><span class="p">(</span><span class="nx">value</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"><span class="p">});</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// And use it as follows: |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">pcoll</span> <span class="o">=</span> <span class="nx">root</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">([</span><span class="k">new</span> <span class="nx">Foo</span><span class="p">(</span><span class="s2">&#34;a&#34;</span><span class="p">),</span> <span class="k">new</span> <span class="nx">Foo</span><span class="p">(</span><span class="s2">&#34;b&#34;</span><span class="p">)]))</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Use beamLogicalType in the exemplar to indicate its use. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nx">withRowCoder</span><span class="p">({</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beamLogicalType</span><span class="o">:</span> <span class="s2">&#34;beam:logical_type:typescript_foo:v1&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> <span class="kr">as</span> <span class="kt">any</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="built-in-logical-types">6.4.2. Useful logical types</h4> |
| <p class="language-py">Currently the Python SDK provides minimal convenience logical types, |
| other than to handle <code>MicrosInstant</code>.</p> |
| <p class="language-go">Currently the Go SDK provides minimal convenience logical types, |
| other than to handle additional integer primitives, and <code>time.Time</code>.</p> |
| <h5 id="enumerationtype"><strong>EnumerationType</strong></h5> |
| <p class="language-py">This convenience builder doesn&rsquo;t yet exist for the Python SDK.</p> |
| <p class="language-go">This convenience builder doesn&rsquo;t yet exist for the Go SDK.</p> |
| <p class="language-java">This logical type allows creating an enumeration type consisting of a set of named constants.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">Schema</span> <span class="n">schema</span> <span class="o">=</span> <span class="n">Schema</span><span class="o">.</span><span class="na">builder</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="err">…</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">addLogicalTypeField</span><span class="o">(</span><span class="s">&#34;color&#34;</span><span class="o">,</span> <span class="n">EnumerationType</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="s">&#34;RED&#34;</span><span class="o">,</span> <span class="s">&#34;GREEN&#34;</span><span class="o">,</span> <span class="s">&#34;BLUE&#34;</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">build</span><span class="o">();</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">The value of this field is stored in the row as an INT32 type, however the logical type defines a value type that lets |
| you access the enumeration either as a string or a value. For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">EnumerationType</span><span class="o">.</span><span class="na">Value</span> <span class="n">enumValue</span> <span class="o">=</span> <span class="n">enumType</span><span class="o">.</span><span class="na">valueOf</span><span class="o">(</span><span class="s">&#34;RED&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">enumValue</span><span class="o">.</span><span class="na">getValue</span><span class="o">();</span> <span class="c1">// Returns 0, the integer value of the constant. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">enumValue</span><span class="o">.</span><span class="na">toString</span><span class="o">();</span> <span class="c1">// Returns &#34;RED&#34;, the string value of the constant |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">Given a row object with an enumeration field, you can also extract the field as the enumeration value.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">EnumerationType</span><span class="o">.</span><span class="na">Value</span> <span class="n">enumValue</span> <span class="o">=</span> <span class="n">row</span><span class="o">.</span><span class="na">getLogicalTypeValue</span><span class="o">(</span><span class="s">&#34;color&#34;</span><span class="o">,</span> <span class="n">EnumerationType</span><span class="o">.</span><span class="na">Value</span><span class="o">.</span><span class="na">class</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">Automatic schema inference from Java POJOs and JavaBeans automatically converts Java enums to EnumerationType logical |
| types.</p> |
| <h5 id="oneoftype"><strong>OneOfType</strong></h5> |
| <p class="language-py">This convenience builder doesn&rsquo;t yet exist for the Python SDK.</p> |
| <p class="language-go">This convenience builder doesn&rsquo;t yet exist for the Go SDK.</p> |
| <p class="language-java">OneOfType allows creating a disjoint union type over a set of schema fields. For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">Schema</span> <span class="n">schema</span> <span class="o">=</span> <span class="n">Schema</span><span class="o">.</span><span class="na">builder</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="err">…</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">addLogicalTypeField</span><span class="o">(</span><span class="s">&#34;oneOfField&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">OneOfType</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">Field</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;intField&#34;</span><span class="o">,</span> <span class="n">FieldType</span><span class="o">.</span><span class="na">INT32</span><span class="o">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Field</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;stringField&#34;</span><span class="o">,</span> <span class="n">FieldType</span><span class="o">.</span><span class="na">STRING</span><span class="o">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Field</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;bytesField&#34;</span><span class="o">,</span> <span class="n">FieldType</span><span class="o">.</span><span class="na">BYTES</span><span class="o">)))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">build</span><span class="o">();</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">The value of this field is stored in the row as another Row type, where all the fields are marked as nullable. The |
| logical type however defines a Value object that contains an enumeration value indicating which field was set and allows |
| getting just that field:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Returns an enumeration indicating all possible case values for the enum. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// For the above example, this will be |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// EnumerationType.create(&#34;intField&#34;, &#34;stringField&#34;, &#34;bytesField&#34;); |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">EnumerationType</span> <span class="n">oneOfEnum</span> <span class="o">=</span> <span class="n">onOfType</span><span class="o">.</span><span class="na">getCaseEnumType</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Creates an instance of the union with the string field set. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">OneOfType</span><span class="o">.</span><span class="na">Value</span> <span class="n">oneOfValue</span> <span class="o">=</span> <span class="n">oneOfType</span><span class="o">.</span><span class="na">createValue</span><span class="o">(</span><span class="s">&#34;stringField&#34;</span><span class="o">,</span> <span class="s">&#34;foobar&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Handle the oneof |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="k">switch</span> <span class="o">(</span><span class="n">oneOfValue</span><span class="o">.</span><span class="na">getCaseEnumType</span><span class="o">().</span><span class="na">toString</span><span class="o">())</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">case</span> <span class="s">&#34;intField&#34;</span><span class="o">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">processInt</span><span class="o">(</span><span class="n">oneOfValue</span><span class="o">.</span><span class="na">getValue</span><span class="o">(</span><span class="n">Integer</span><span class="o">.</span><span class="na">class</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">case</span> <span class="s">&#34;stringField&#34;</span><span class="o">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">processString</span><span class="o">(</span><span class="n">oneOfValue</span><span class="o">.</span><span class="na">getValue</span><span class="o">(</span><span class="n">String</span><span class="o">.</span><span class="na">class</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">case</span> <span class="s">&#34;bytesField&#34;</span><span class="o">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">processBytes</span><span class="o">(</span><span class="n">oneOfValue</span><span class="o">.</span><span class="na">getValue</span><span class="o">(</span><span class="n">bytes</span><span class="o">[].</span><span class="na">class</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">In the above example we used the field names in the switch statement for clarity, however the enum integer values could |
| also be used.</p> |
| <h3 id="creating-schemas">6.5. Creating Schemas</h3> |
| <p>In order to take advantage of schemas, your <code>PCollection</code>s must have a schema attached to it. |
| Often, the source itself will attach a schema to the PCollection. |
| For example, when using <code>AvroIO</code> to read Avro files, the source can automatically infer a Beam schema from the Avro schema and attach that to the Beam <code>PCollection</code>. |
| However not all sources produce schemas. |
| In addition, often Beam pipelines have intermediate stages and types, and those also can benefit from the expressiveness of schemas.</p> |
| <h4 id="inferring-schemas">6.5.1. Inferring schemas</h4> |
| <nav class="language-switcher"> |
| <strong>Adapt for:</strong> |
| <ul> |
| <li data-value="java" class="active">Java SDK</li> |
| <li data-value="py">Python SDK</li> |
| <li data-value="go">Go SDK</li> |
| <li data-value="typescript">TypeScript SDK</li> |
| </ul> |
| </nav> |
| <p class="language-typescript">Unfortunately, Beam is unable to access Typescript&rsquo;s type information at runtime. |
| Schemas must be manually declared with <code>beam.withRowCoder</code>. |
| On the other hand, schema-aware operations such as <code>GroupBy</code> can be used |
| without an explicit schema declared.</p> |
| <p class="language-java">Beam is able to infer schemas from a variety of common Java types. |
| The <code>@DefaultSchema</code> annotation can be used to tell Beam to infer schemas from a specific type. |
| The annotation takes a <code>SchemaProvider</code> as an argument, and <code>SchemaProvider</code> classes are already built in for common Java types. |
| The <code>SchemaRegistry</code> can also be invoked programmatically for cases where it is not practical to annotate the Java type itself.</p> |
| <p class="language-java"><strong>Java POJOs</strong></p> |
| <p class="language-java">A POJO (Plain Old Java Object) is a Java object that is not bound by any restriction other than the Java Language |
| Specification. A POJO can contain member variables that are primitives, that are other POJOs, or are collections maps or |
| arrays thereof. POJOs do not have to extend prespecified classes or extend any specific interfaces.</p> |
| <p class="language-java">If a POJO class is annotated with <code>@DefaultSchema(JavaFieldSchema.class)</code>, Beam will automatically infer a schema for |
| this class. Nested classes are supported as are classes with <code>List</code>, array, and <code>Map</code> fields.</p> |
| <p class="language-java">For example, annotating the following class tells Beam to infer a schema from this POJO class and apply it to any |
| <code>PCollection&lt;TransactionPojo&gt;</code>.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@DefaultSchema</span><span class="o">(</span><span class="n">JavaFieldSchema</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">TransactionPojo</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kd">final</span> <span class="n">String</span> <span class="n">bank</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kd">final</span> <span class="kt">double</span> <span class="n">purchaseAmount</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@SchemaCreate</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="nf">TransactionPojo</span><span class="o">(</span><span class="n">String</span> <span class="n">bank</span><span class="o">,</span> <span class="kt">double</span> <span class="n">purchaseAmount</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">this</span><span class="o">.</span><span class="na">bank</span> <span class="o">=</span> <span class="n">bank</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">this</span><span class="o">.</span><span class="na">purchaseAmount</span> <span class="o">=</span> <span class="n">purchaseAmount</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Beam will automatically infer the correct schema for this PCollection. No coder is needed as a result. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">TransactionPojo</span><span class="o">&gt;</span> <span class="n">pojos</span> <span class="o">=</span> <span class="n">readPojos</span><span class="o">();</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">The <code>@SchemaCreate</code> annotation tells Beam that this constructor can be used to create instances of TransactionPojo, |
| assuming that constructor parameters have the same names as the field names. <code>@SchemaCreate</code> can also be used to annotate |
| static factory methods on the class, allowing the constructor to remain private. If there is no <code>@SchemaCreate</code> |
| annotation then all the fields must be non-final and the class must have a zero-argument constructor.</p> |
| <p class="language-java">There are a couple of other useful annotations that affect how Beam infers schemas. By default the schema field names |
| inferred will match that of the class field names. However <code>@SchemaFieldName</code> can be used to specify a different name to |
| be used for the schema field. <code>@SchemaIgnore</code> can be used to mark specific class fields as excluded from the inferred |
| schema. For example, it’s common to have ephemeral fields in a class that should not be included in a schema |
| (e.g. caching the hash value to prevent expensive recomputation of the hash), and <code>@SchemaIgnore</code> can be used to |
| exclude these fields. Note that ignored fields will not be included in the encoding of these records.</p> |
| <p class="language-java">In some cases it is not convenient to annotate the POJO class, for example if the POJO is in a different package that is |
| not owned by the Beam pipeline author. In these cases the schema inference can be triggered programmatically in |
| pipeline’s main function as follows:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="n">pipeline</span><span class="o">.</span><span class="na">getSchemaRegistry</span><span class="o">().</span><span class="na">registerPOJO</span><span class="o">(</span><span class="n">TransactionPOJO</span><span class="o">.</span><span class="na">class</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java"><strong>Java Beans</strong></p> |
| <p class="language-java">Java Beans are a de-facto standard for creating reusable property classes in Java. While the full |
| standard has many characteristics, the key ones are that all properties are accessed via getter and setter classes, and |
| the name format for these getters and setters is standardized. A Java Bean class can be annotated with |
| <code>@DefaultSchema(JavaBeanSchema.class)</code> and Beam will automatically infer a schema for this class. For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@DefaultSchema</span><span class="o">(</span><span class="n">JavaBeanSchema</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">TransactionBean</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="nf">TransactionBean</span><span class="o">()</span> <span class="o">{</span> <span class="err">…</span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">String</span> <span class="nf">getBank</span><span class="o">()</span> <span class="o">{</span> <span class="err">…</span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">setBank</span><span class="o">(</span><span class="n">String</span> <span class="n">bank</span><span class="o">)</span> <span class="o">{</span> <span class="err">…</span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">double</span> <span class="nf">getPurchaseAmount</span><span class="o">()</span> <span class="o">{</span> <span class="err">…</span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">setPurchaseAmount</span><span class="o">(</span><span class="kt">double</span> <span class="n">purchaseAmount</span><span class="o">)</span> <span class="o">{</span> <span class="err">…</span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Beam will automatically infer the correct schema for this PCollection. No coder is needed as a result. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">TransactionBean</span><span class="o">&gt;</span> <span class="n">beans</span> <span class="o">=</span> <span class="n">readBeans</span><span class="o">();</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">The <code>@SchemaCreate</code> annotation can be used to specify a constructor or a static factory method, in which case the |
| setters and zero-argument constructor can be omitted.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@DefaultSchema</span><span class="o">(</span><span class="n">JavaBeanSchema</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">TransactionBean</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@SchemaCreate</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Public</span> <span class="nf">TransactionBean</span><span class="o">(</span><span class="n">String</span> <span class="n">bank</span><span class="o">,</span> <span class="kt">double</span> <span class="n">purchaseAmount</span><span class="o">)</span> <span class="o">{</span> <span class="err">…</span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">String</span> <span class="nf">getBank</span><span class="o">()</span> <span class="o">{</span> <span class="err">…</span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">double</span> <span class="nf">getPurchaseAmount</span><span class="o">()</span> <span class="o">{</span> <span class="err">…</span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java"><code>@SchemaFieldName</code> and <code>@SchemaIgnore</code> can be used to alter the schema inferred, just like with POJO classes.</p> |
| <p class="language-java"><strong>AutoValue</strong></p> |
| <p class="language-java">Java value classes are notoriously difficult to generate correctly. There is a lot of boilerplate you must create in |
| order to properly implement a value class. AutoValue is a popular library for easily generating such classes by |
| implementing a simple abstract base class.</p> |
| <p class="language-java">Beam can infer a schema from an AutoValue class. For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@DefaultSchema</span><span class="o">(</span><span class="n">AutoValueSchema</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"><span class="nd">@AutoValue</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">abstract</span> <span class="kd">class</span> <span class="nc">TransactionValue</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kd">abstract</span> <span class="n">String</span> <span class="nf">getBank</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kd">abstract</span> <span class="kt">double</span> <span class="nf">getPurchaseAmount</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">This is all that’s needed to generate a simple AutoValue class, and the above <code>@DefaultSchema</code> annotation tells Beam to |
| infer a schema from it. This also allows AutoValue elements to be used inside of <code>PCollection</code>s.</p> |
| <p class="language-java"><code>@SchemaFieldName</code> and <code>@SchemaIgnore</code> can be used to alter the schema inferred.</p> |
| <p class="language-py">Beam has a few different mechanisms for inferring schemas from Python code.</p> |
| <p class="language-py"><strong>NamedTuple classes</strong></p> |
| <p class="language-py">A <a href="https://docs.python.org/3/library/typing.html#typing.NamedTuple">NamedTuple</a> |
| class is a Python class that wraps a <code>tuple</code>, assigning a name to each element |
| and restricting it to a particular type. Beam will automatically infer the |
| schema for PCollections with <code>NamedTuple</code> output types. For example:</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">Transaction</span><span class="p">(</span><span class="n">typing</span><span class="o">.</span><span class="n">NamedTuple</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">bank</span><span class="p">:</span> <span class="nb">str</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">purchase_amount</span><span class="p">:</span> <span class="nb">float</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">pc</span> <span class="o">=</span> <span class="nb">input</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="o">...</span><span class="p">)</span><span class="o">.</span><span class="n">with_output_types</span><span class="p">(</span><span class="n">Transaction</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-py"><strong>beam.Row and Select</strong></p> |
| <p class="language-py">There are also methods for creating ad-hoc schema declarations. First, you can |
| use a lambda that returns instances of <code>beam.Row</code>:</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">input_pc</span> <span class="o">=</span> <span class="o">...</span> <span class="c1"># {&#34;bank&#34;: ..., &#34;purchase_amount&#34;: ...}</span> |
| </span></span><span class="line"><span class="cl"><span class="n">output_pc</span> <span class="o">=</span> <span class="n">input_pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">item</span><span class="p">:</span> <span class="n">beam</span><span class="o">.</span><span class="n">Row</span><span class="p">(</span><span class="n">bank</span><span class="o">=</span><span class="n">item</span><span class="p">[</span><span class="s2">&#34;bank&#34;</span><span class="p">],</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">purchase_amount</span><span class="o">=</span><span class="n">item</span><span class="p">[</span><span class="s2">&#34;purchase_amount&#34;</span><span class="p">])</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-py">Sometimes it can be more concise to express the same logic with the |
| <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Select"><code>Select</code></a> transform:</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">input_pc</span> <span class="o">=</span> <span class="o">...</span> <span class="c1"># {&#34;bank&#34;: ..., &#34;purchase_amount&#34;: ...}</span> |
| </span></span><span class="line"><span class="cl"><span class="n">output_pc</span> <span class="o">=</span> <span class="n">input_pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Select</span><span class="p">(</span><span class="n">bank</span><span class="o">=</span><span class="k">lambda</span> <span class="n">item</span><span class="p">:</span> <span class="n">item</span><span class="p">[</span><span class="s2">&#34;bank&#34;</span><span class="p">],</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">purchase_amount</span><span class="o">=</span><span class="k">lambda</span> <span class="n">item</span><span class="p">:</span> <span class="n">item</span><span class="p">[</span><span class="s2">&#34;purchase_amount&#34;</span><span class="p">])</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-py">Note that these declaration don&rsquo;t include any specific information about the |
| types of the <code>bank</code> and <code>purchase_amount</code> fields, so Beam will attempt to infer |
| type information. If it&rsquo;s unable to it will fall back to the generic type |
| <code>Any</code>. Sometimes this is not ideal, you can use casts to make sure Beam |
| correctly infers types with <code>beam.Row</code> or with <code>Select</code>:</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">input_pc</span> <span class="o">=</span> <span class="o">...</span> <span class="c1"># {&#34;bank&#34;: ..., &#34;purchase_amount&#34;: ...}</span> |
| </span></span><span class="line"><span class="cl"><span class="n">output_pc</span> <span class="o">=</span> <span class="n">input_pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">item</span><span class="p">:</span> <span class="n">beam</span><span class="o">.</span><span class="n">Row</span><span class="p">(</span><span class="n">bank</span><span class="o">=</span><span class="nb">str</span><span class="p">(</span><span class="n">item</span><span class="p">[</span><span class="s2">&#34;bank&#34;</span><span class="p">]),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">purchase_amount</span><span class="o">=</span><span class="nb">float</span><span class="p">(</span><span class="n">item</span><span class="p">[</span><span class="s2">&#34;purchase_amount&#34;</span><span class="p">])))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-go">Beam currently only infers schemas for exported fields in Go structs.</p> |
| <p class="language-go"><strong>Structs</strong></p> |
| <p class="language-go">Beam will automatically infer schemas for all Go structs used |
| as PCollection elements, and default to encoding them using |
| schema encoding.</p> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">Transaction</span> <span class="kd">struct</span><span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Bank</span> <span class="kt">string</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">PurchaseAmount</span> <span class="kt">float64</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nx">checksum</span> <span class="p">[]</span><span class="kt">byte</span> <span class="c1">// ignored |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-go">Unexported fields are ignored, and cannot be automatically inferred as part of the schema. |
| Fields of type func, channel, unsafe.Pointer, or uintptr will be ignored by inference. |
| Fields of interface types are ignored, unless a schema provider |
| is registered for them.</p> |
| <p class="language-go">By default, schema field names will match the exported struct field names. |
| In the above example, &ldquo;Bank&rdquo; and &ldquo;PurchaseAmount&rdquo; are the schema field names. |
| A schema field name can be overridden with a struct tag for the field.</p> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">Transaction</span> <span class="kd">struct</span><span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Bank</span> <span class="kt">string</span> <span class="s">`beam:&#34;bank&#34;`</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">PurchaseAmount</span> <span class="kt">float64</span> <span class="s">`beam:&#34;purchase_amount&#34;`</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-go">Overriding schema field names is useful for compatibility cross language transforms, |
| as schema fields may have different requirements or restrictions from Go exported fields.</p> |
| <h3 id="using-schemas">6.6. Using Schema Transforms</h3> |
| <p>A schema on a <code>PCollection</code> enables a rich variety of relational transforms. The fact that each record is composed of |
| named fields allows for simple and readable aggregations that reference fields by name, similar to the aggregations in |
| a SQL expression.</p> |
| <p class="language-go">Beam does not yet support Schema transforms natively in Go. However, it will be implemented with the following behavior.</p> |
| <h4 id="661-field-selection-syntax">6.6.1. Field selection syntax</h4> |
| <p>The advantage of schemas is that they allow referencing of element fields by name. Beam provides a selection syntax for |
| referencing fields, including nested and repeated fields. This syntax is used by all of the schema transforms when |
| referencing the fields they operate on. The syntax can also be used inside of a DoFn to specify which schema fields to |
| process.</p> |
| <p>Addressing fields by name still retains type safety as Beam will check that schemas match at the time the pipeline graph |
| is constructed. If a field is specified that does not exist in the schema, the pipeline will fail to launch. In addition, |
| if a field is specified with a type that does not match the type of that field in the schema, the pipeline will fail to |
| launch.</p> |
| <p>The following characters are not allowed in field names: . * [ ] { }</p> |
| <h5 id="top-level-fields"><strong>Top-level fields</strong></h5> |
| <p>In order to select a field at the top level of a schema, the name of the field is specified. For example, to select just |
| the user ids from a <code>PCollection</code> of purchases one would write (using the <code>Select</code> transform)</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Select</span><span class="o">.</span><span class="na">fieldNames</span><span class="o">(</span><span class="s">&#34;userId&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">input_pc</span> <span class="o">=</span> <span class="o">...</span> <span class="c1"># {&#34;user_id&#34;: ...,&#34;bank&#34;: ..., &#34;purchase_amount&#34;: ...}</span> |
| </span></span><span class="line"><span class="cl"><span class="n">output_pc</span> <span class="o">=</span> <span class="n">input_pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Select</span><span class="p">(</span><span class="s2">&#34;user_id&#34;</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h5 id="nested-fields"><strong>Nested fields</strong></h5> |
| <p class="language-py">Support for Nested fields hasn&rsquo;t been developed for the Python SDK yet.</p> |
| <p class="language-go">Support for Nested fields hasn&rsquo;t been developed for the Go SDK yet.</p> |
| <p class="language-java">Individual nested fields can be specified using the dot operator. For example, to select just the postal code from the |
| shipping address one would write</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Select</span><span class="o">.</span><span class="na">fieldNames</span><span class="o">(</span><span class="s">&#34;shippingAddress.postCode&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <!-- |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">input_pc</span> <span class="o">=</span> <span class="o">...</span> <span class="c1"># {&#34;user_id&#34;: ..., &#34;shipping_address&#34;: &#34;post_code&#34;: ..., &#34;bank&#34;: ..., &#34;purchase_amount&#34;: ...}</span> |
| </span></span><span class="line"><span class="cl"><span class="n">output_pc</span> <span class="o">=</span> <span class="n">input_pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Select</span><span class="p">(</span><span class="n">post_code</span><span class="o">=</span><span class="k">lambda</span> <span class="n">item</span><span class="p">:</span> <span class="nb">str</span><span class="p">(</span><span class="n">item</span><span class="p">[</span><span class="s2">&#34;shipping_address.post_code&#34;</span><span class="p">]))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| --> |
| <h5 id="wildcards"><strong>Wildcards</strong></h5> |
| <p class="language-py">Support for wildcards hasn&rsquo;t been developed for the Python SDK yet.</p> |
| <p class="language-go">Support for wildcards hasn&rsquo;t been developed for the Go SDK yet.</p> |
| <p class="language-java">The * operator can be specified at any nesting level to represent all fields at that level. For example, to select all |
| shipping-address fields one would write</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Select</span><span class="o">.</span><span class="na">fieldNames</span><span class="o">(</span><span class="s">&#34;shippingAddress.*&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <!-- |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1">#TODO(https://github.com/apache/beam/issues/23275): Add support for projecting nested fields</span> |
| </span></span><span class="line"><span class="cl"><span class="n">input_pc</span> <span class="o">=</span> <span class="o">...</span> <span class="c1"># {&#34;user_id&#34;: ..., &#34;shipping_address&#34;: &#34;post_code&#34;: ..., &#34;bank&#34;: ..., &#34;purchase_amount&#34;: ...}</span> |
| </span></span><span class="line"><span class="cl"><span class="n">output_pc</span> <span class="o">=</span> <span class="n">input_pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Select</span><span class="p">(</span><span class="s2">&#34;shipping_address.*&#34;</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| --> |
| <h5 id="arrays"><strong>Arrays</strong></h5> |
| <p class="language-java">An array field, where the array element type is a row, can also have subfields of the element type addressed. When |
| selected, the result is an array of the selected subfield type. For example</p> |
| <p class="language-py">Support for Array fields hasn&rsquo;t been developed for the Python SDK yet.</p> |
| <p class="language-go">Support for Array fields hasn&rsquo;t been developed for the Go SDK yet.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Select</span><span class="o">.</span><span class="na">fieldNames</span><span class="o">(</span><span class="s">&#34;transactions[].bank&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">Will result in a row containing an array field with element-type string, containing the list of banks for each |
| transaction.</p> |
| <p class="language-java">While the use of [] brackets in the selector is recommended, to make it clear that array elements are being selected, |
| they can be omitted for brevity. In the future, array slicing will be supported, allowing selection of portions of the |
| array.</p> |
| <h5 id="maps"><strong>Maps</strong></h5> |
| <p>A map field, where the value type is a row, can also have subfields of the value type addressed. When selected, the |
| result is a map where the keys are the same as in the original map but the value is the specified type. Similar to |
| arrays, the use of {} curly brackets in the selector is recommended, to make it clear that map value elements are being |
| selected, they can be omitted for brevity. In the future, map key selectors will be supported, allowing selection of |
| specific keys from the map. For example, given the following schema:</p> |
| <p><strong>PurchasesByType</strong></p> |
| <table class="table-wrapper--pr"> |
| <thead> |
| <tr class="header"> |
| <th><b>Field Name</b></th> |
| <th><b>Field Type</b></th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>purchases</td> |
| <td>MAP{STRING, ROW{PURCHASE}</td> |
| </tr> |
| </tbody> |
| </table> |
| <br/> |
| <p>The following</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchasesByType</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Select</span><span class="o">.</span><span class="na">fieldNames</span><span class="o">(</span><span class="s">&#34;purchases{}.userId&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-py">Support for Map fields hasn&rsquo;t been developed for the Python SDK yet.</p> |
| <p class="language-go">Support for Map fields hasn&rsquo;t been developed for the Go SDK yet.</p> |
| <p>Will result in a row containing a map field with key-type string and value-type string. The selected map will contain |
| all of the keys from the original map, and the values will be the userId contained in the purchase record.</p> |
| <p>While the use of {} brackets in the selector is recommended, to make it clear that map value elements are being selected, |
| they can be omitted for brevity. In the future, map slicing will be supported, allowing selection of specific keys from |
| the map.</p> |
| <h4 id="662-schema-transforms">6.6.2. Schema transforms</h4> |
| <p>Beam provides a collection of transforms that operate natively on schemas. These transforms are very expressive, |
| allowing selections and aggregations in terms of named schema fields. Following are some examples of useful |
| schema transforms.</p> |
| <h5 id="selecting-input"><strong>Selecting input</strong></h5> |
| <p>Often a computation is only interested in a subset of the fields in an input <code>PCollection</code>. The <code>Select</code> transform allows |
| one to easily project out only the fields of interest. The resulting <code>PCollection</code> has a schema containing each selected |
| field as a top-level field. Both top-level and nested fields can be selected. For example, in the Purchase schema, one |
| could select only the userId and streetAddress fields as follows</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Select</span><span class="o">.</span><span class="na">fieldNames</span><span class="o">(</span><span class="s">&#34;userId&#34;</span><span class="o">,</span> <span class="s">&#34;shippingAddress.streetAddress&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-py">Support for Nested fields hasn&rsquo;t been developed for the Python SDK yet.</p> |
| <p class="language-go">Support for Nested fields hasn&rsquo;t been developed for the Go SDK yet.</p> |
| <p>The resulting <code>PCollection</code> will have the following schema</p> |
| <table class="table-wrapper--pr"> |
| <thead> |
| <tr class="header"> |
| <th><b>Field Name</b></th> |
| <th><b>Field Type</b></th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>userId</td> |
| <td>STRING</td> |
| </tr> |
| <tr> |
| <td>streetAddress</td> |
| <td>STRING</td> |
| </tr> |
| </tbody> |
| </table> |
| <br/> |
| <p>The same is true for wildcard selections. The following</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Select</span><span class="o">.</span><span class="na">fieldNames</span><span class="o">(</span><span class="s">&#34;userId&#34;</span><span class="o">,</span> <span class="s">&#34;shippingAddress.*&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-py">Support for Wildcards hasn&rsquo;t been developed for the Python SDK yet.</p> |
| <p class="language-go">Support for Wildcards hasn&rsquo;t been developed for the Go SDK yet.</p> |
| <p>Will result in the following schema</p> |
| <table class="table-wrapper--pr"> |
| <thead> |
| <tr class="header"> |
| <th><b>Field Name</b></th> |
| <th><b>Field Type</b></th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>userId</td> |
| <td>STRING</td> |
| </tr> |
| <tr> |
| <td>streetAddress</td> |
| <td>STRING</td> |
| </tr> |
| <tr> |
| <td>city</td> |
| <td>STRING</td> |
| </tr> |
| <tr> |
| <td>state</td> |
| <td>nullable STRING</td> |
| </tr> |
| <tr> |
| <td>country</td> |
| <td>STRING</td> |
| </tr> |
| <tr> |
| <td>postCode</td> |
| <td>STRING</td> |
| </tr> |
| </tbody> |
| </table> |
| <br/> |
| <p>When selecting fields nested inside of an array, the same rule applies that each selected field appears separately as a |
| top-level field in the resulting row. This means that if multiple fields are selected from the same nested row, each |
| selected field will appear as its own array field. For example</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Select</span><span class="o">.</span><span class="na">fieldNames</span><span class="o">(</span> <span class="s">&#34;transactions.bank&#34;</span><span class="o">,</span> <span class="s">&#34;transactions.purchaseAmount&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-py">Support for nested fields hasn&rsquo;t been developed for the Python SDK yet.</p> |
| <p class="language-go">Support for nested fields hasn&rsquo;t been developed for the Go SDK yet.</p> |
| <p class="language-java"><p>Will result in the following schema</p> |
| <table class="table-wrapper--pr"> |
| <thead> |
| <tr class="header"> |
| <th><b>Field Name</b></th> |
| <th><b>Field Type</b></th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>bank</td> |
| <td>ARRAY[STRING]</td> |
| </tr> |
| <tr> |
| <td>purchaseAmount</td> |
| <td>ARRAY[DOUBLE]</td> |
| </tr> |
| </tbody> |
| </table> |
| <br/> |
| </p> |
| <p>Wildcard selections are equivalent to separately selecting each field.</p> |
| <p>Selecting fields nested inside of maps have the same semantics as arrays. If you select multiple fields from a map |
| , then each selected field will be expanded to its own map at the top level. This means that the set of map keys will |
| be copied, once for each selected field.</p> |
| <p>Sometimes different nested rows will have fields with the same name. Selecting multiple of these fields would result in |
| a name conflict, as all selected fields are put in the same row schema. When this situation arises, the |
| <code>Select.withFieldNameAs</code> builder method can be used to provide an alternate name for the selected field.</p> |
| <p>Another use of the Select transform is to flatten a nested schema into a single flat schema. For example</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Select</span><span class="o">.</span><span class="na">flattenedSchema</span><span class="o">());</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-py">Support for nested fields hasn&rsquo;t been developed for the Python SDK yet.</p> |
| <p class="language-go">Support for nested fields hasn&rsquo;t been developed for the Go SDK yet.</p> |
| <p class="language-java"><p>Will result in the following schema</p> |
| <div class="table-container-wrapper"> |
| <table class="table-wrapper--pr"> |
| <thead> |
| <tr class="header"> |
| <th><b>Field Name</b></th> |
| <th><b>Field Type</b></th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>userId</td> |
| <td>STRING</td> |
| </tr> |
| <tr> |
| <td>itemId</td> |
| <td>STRING</td> |
| </tr> |
| <tr> |
| <td>shippingAddress_streetAddress</td> |
| <td>STRING</td> |
| </tr> |
| <tr> |
| <td>shippingAddress_city</td> |
| <td>nullable STRING</td> |
| </tr> |
| <tr> |
| <td>shippingAddress_state</td> |
| <td>STRING</td> |
| </tr> |
| <tr> |
| <td>shippingAddress_country</td> |
| <td>STRING</td> |
| </tr> |
| <tr> |
| <td>shippingAddress_postCode</td> |
| <td>STRING</td> |
| </tr> |
| <tr> |
| <td>costCents</td> |
| <td>INT64</td> |
| </tr> |
| <tr> |
| <td>transactions_bank</td> |
| <td>ARRAY[STRING]</td> |
| </tr> |
| <tr> |
| <td>transactions_purchaseAmount</td> |
| <td>ARRAY[DOUBLE]</td> |
| </tr> |
| </tbody> |
| </table> |
| <br/> |
| </div> |
| </p> |
| <h5 id="grouping-aggregations"><strong>Grouping aggregations</strong></h5> |
| <p class="language-java">The <code>Group</code> transform allows simply grouping data by any number of fields in the input schema, applying aggregations to |
| those groupings, and storing the result of those aggregations in a new schema field. The output of the <code>Group</code> transform |
| has a schema with one field corresponding to each aggregation performed.</p> |
| <p class="language-py">The <code>GroupBy</code> transform allows simply grouping data by any number of fields in the input schema, applying aggregations to |
| those groupings, and storing the result of those aggregations in a new schema field. The output of the <code>GroupBy</code> transform |
| has a schema with one field corresponding to each aggregation performed.</p> |
| <p class="language-java">The simplest usage of <code>Group</code> specifies no aggregations, in which case all inputs matching the provided set of fields |
| are grouped together into an <code>ITERABLE</code> field. For example</p> |
| <p class="language-py">The simplest usage of <code>GroupBy</code> specifies no aggregations, in which case all inputs matching the provided set of fields |
| are grouped together into an <code>ITERABLE</code> field. For example</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Group</span><span class="o">.</span><span class="na">byFieldNames</span><span class="o">(</span><span class="s">&#34;userId&#34;</span><span class="o">,</span> <span class="s">&#34;bank&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">input_pc</span> <span class="o">=</span> <span class="o">...</span> <span class="c1"># {&#34;user_id&#34;: ...,&#34;bank&#34;: ..., &#34;purchase_amount&#34;: ...}</span> |
| </span></span><span class="line"><span class="cl"><span class="n">output_pc</span> <span class="o">=</span> <span class="n">input_pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">GroupBy</span><span class="p">(</span><span class="s1">&#39;user_id&#39;</span><span class="p">,</span><span class="s1">&#39;bank&#39;</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-go">Support for schema-aware grouping hasn&rsquo;t been developed for the Go SDK yet.</p> |
| <p class="lanuage-java">The output schema of this is:</p> |
| <table class="table-wrapper--pr"> |
| <thead> |
| <tr class="header"> |
| <th><b>Field Name</b></th> |
| <th><b>Field Type</b></th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>key</td> |
| <td>ROW{userId:STRING, bank:STRING}</td> |
| </tr> |
| <tr> |
| <td>values</td> |
| <td>ITERABLE[ROW[Purchase]]</td> |
| </tr> |
| </tbody> |
| </table> |
| <br/> |
| <p>The key field contains the grouping key and the values field contains a list of all the values that matched that key.</p> |
| <p>The names of the key and values fields in the output schema can be controlled using this withKeyField and withValueField |
| builders, as follows:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Group</span><span class="o">.</span><span class="na">byFieldNames</span><span class="o">(</span><span class="s">&#34;userId&#34;</span><span class="o">,</span> <span class="s">&#34;shippingAddress.streetAddress&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyField</span><span class="o">(</span><span class="s">&#34;userAndStreet&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueField</span><span class="o">(</span><span class="s">&#34;matchingPurchases&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>It is quite common to apply one or more aggregations to the grouped result. Each aggregation can specify one or more fields |
| to aggregate, an aggregation function, and the name of the resulting field in the output schema. For example, the |
| following application computes three aggregations grouped by userId, with all aggregations represented in a single |
| output schema:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Group</span><span class="o">.</span><span class="na">byFieldNames</span><span class="o">(</span><span class="s">&#34;userId&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">aggregateField</span><span class="o">(</span><span class="s">&#34;itemId&#34;</span><span class="o">,</span> <span class="n">Count</span><span class="o">.</span><span class="na">combineFn</span><span class="o">(),</span> <span class="s">&#34;numPurchases&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">aggregateField</span><span class="o">(</span><span class="s">&#34;costCents&#34;</span><span class="o">,</span> <span class="n">Sum</span><span class="o">.</span><span class="na">ofLongs</span><span class="o">(),</span> <span class="s">&#34;totalSpendCents&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">aggregateField</span><span class="o">(</span><span class="s">&#34;costCents&#34;</span><span class="o">,</span> <span class="n">Top</span><span class="o">.&lt;</span><span class="n">Long</span><span class="o">&gt;</span><span class="n">largestLongsFn</span><span class="o">(</span><span class="n">10</span><span class="o">),</span> <span class="s">&#34;topPurchases&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">input_pc</span> <span class="o">=</span> <span class="o">...</span> <span class="c1"># {&#34;user_id&#34;: ..., &#34;item_Id&#34;: ..., &#34;cost_cents&#34;: ...}</span> |
| </span></span><span class="line"><span class="cl"><span class="n">output_pc</span> <span class="o">=</span> <span class="n">input_pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">GroupBy</span><span class="p">(</span><span class="s2">&#34;user_id&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="n">aggregate_field</span><span class="p">(</span><span class="s2">&#34;item_id&#34;</span><span class="p">,</span> <span class="n">CountCombineFn</span><span class="p">,</span> <span class="s2">&#34;num_purchases&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="n">aggregate_field</span><span class="p">(</span><span class="s2">&#34;cost_cents&#34;</span><span class="p">,</span> <span class="nb">sum</span><span class="p">,</span> <span class="s2">&#34;total_spendcents&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="n">aggregate_field</span><span class="p">(</span><span class="s2">&#34;cost_cents&#34;</span><span class="p">,</span> <span class="n">TopCombineFn</span><span class="p">,</span> <span class="s2">&#34;top_purchases&#34;</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-go">Support for schema-aware grouping hasn&rsquo;t been developed for the Go SDK yet.</p> |
| <p>The result of this aggregation will have the following schema:</p> |
| <table class="table-wrapper--pr"> |
| <thead> |
| <tr class="header"> |
| <th><b>Field Name</b></th> |
| <th><b>Field Type</b></th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>key</td> |
| <td>ROW{userId:STRING}</td> |
| </tr> |
| <tr> |
| <td>value</td> |
| <td>ROW{numPurchases: INT64, totalSpendCents: INT64, topPurchases: ARRAY[INT64]}</td> |
| </tr> |
| </tbody> |
| </table> |
| <br/> |
| <p>Often <code>Selected.flattenedSchema</code> will be use to flatten the result into a non-nested, flat schema.</p> |
| <h5 id="joins"><strong>Joins</strong></h5> |
| <p>Beam supports equijoins on schema <code>PCollections</code> - namely joins where the join condition depends on the equality of a |
| subset of fields. For example, the following examples uses the Purchases schema to join transactions with the reviews |
| that are likely associated with that transaction (both the user and product match that in the transaction). This is a |
| &ldquo;natural join&rdquo; - one in which the same field names are used on both the left-hand and right-hand sides of the join - |
| and is specified with the <code>using</code> keyword:</p> |
| <p class="language-py">Support for joins hasn&rsquo;t been developed for the Python SDK yet.</p> |
| <p class="language-go">Support for joins hasn&rsquo;t been developed for the Go SDK yet.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Transaction</span><span class="o">&gt;</span> <span class="n">transactions</span> <span class="o">=</span> <span class="n">readTransactions</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Review</span><span class="o">&gt;</span> <span class="n">reviews</span> <span class="o">=</span> <span class="n">readReviews</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Row</span><span class="o">&gt;</span> <span class="n">joined</span> <span class="o">=</span> <span class="n">transactions</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Join</span><span class="o">.</span><span class="na">innerJoin</span><span class="o">(</span><span class="n">reviews</span><span class="o">).</span><span class="na">using</span><span class="o">(</span><span class="s">&#34;userId&#34;</span><span class="o">,</span> <span class="s">&#34;productId&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java"><p>The resulting schema is the following:</p> |
| <table class="table-wrapper--pr"> |
| <thead> |
| <tr class="header"> |
| <th><b>Field Name</b></th> |
| <th><b>Field Type</b></th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>lhs</td> |
| <td>ROW{Transaction}</td> |
| </tr> |
| <tr> |
| <td>rhs</td> |
| <td>ROW{Review}</td> |
| </tr> |
| </tbody> |
| </table> |
| <br/> |
| </p> |
| <p>Each resulting row contains one Transaction and one Review that matched the join condition.</p> |
| <p>If the fields to match in the two schemas have different names, then the on function can be used. For example, if the |
| Review schema named those fields differently than the Transaction schema, then we could write the following:</p> |
| <p class="language-py">Support for joins hasn&rsquo;t been developed for the Python SDK yet.</p> |
| <p class="language-go">Support for joins hasn&rsquo;t been developed for the Go SDK yet.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Row</span><span class="o">&gt;</span> <span class="n">joined</span> <span class="o">=</span> <span class="n">transactions</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Join</span><span class="o">.</span><span class="na">innerJoin</span><span class="o">(</span><span class="n">reviews</span><span class="o">).</span><span class="na">on</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">FieldsEqual</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">left</span><span class="o">(</span><span class="s">&#34;userId&#34;</span><span class="o">,</span> <span class="s">&#34;productId&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">right</span><span class="o">(</span><span class="s">&#34;reviewUserId&#34;</span><span class="o">,</span> <span class="s">&#34;reviewProductId&#34;</span><span class="o">)));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>In addition to inner joins, the Join transform supports full outer joins, left outer joins, and right outer joins.</p> |
| <h5 id="complex-joins"><strong>Complex joins</strong></h5> |
| <p>While most joins tend to be binary joins - joining two inputs together - sometimes you have more than two input |
| streams that all need to be joined on a common key. The <code>CoGroup</code> transform allows joining multiple <code>PCollections</code> |
| together based on equality of schema fields. Each <code>PCollection</code> can be marked as required or optional in the final |
| join record, providing a generalization of outer joins to joins with greater than two input <code>PCollection</code>s. The output |
| can optionally be expanded - providing individual joined records, as in the <code>Join</code> transform. The output can also be |
| processed in unexpanded format - providing the join key along with Iterables of all records from each input that matched |
| that key.</p> |
| <p class="language-py">Support for joins hasn&rsquo;t been developed for the Python SDK yet.</p> |
| <p class="language-go">Support for joins hasn&rsquo;t been developed for the Go SDK yet.</p> |
| <h5 id="filtering-events"><strong>Filtering events</strong></h5> |
| <p>The <code>Filter</code> transform can be configured with a set of predicates, each one based one specified fields. Only records for |
| which all predicates return true will pass the filter. For example the following</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Filter</span><span class="o">.</span><span class="na">create</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">whereFieldName</span><span class="o">(</span><span class="s">&#34;costCents&#34;</span><span class="o">,</span> <span class="n">c</span> <span class="o">-&gt;</span> <span class="n">c</span> <span class="o">&gt;</span> <span class="n">100</span> <span class="o">*</span> <span class="n">20</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">whereFieldName</span><span class="o">(</span><span class="s">&#34;shippingAddress.country&#34;</span><span class="o">,</span> <span class="n">c</span> <span class="o">-&gt;</span> <span class="n">c</span><span class="o">.</span><span class="na">equals</span><span class="o">(</span><span class="s">&#34;de&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Will produce all purchases made from Germany with a purchase price of greater than twenty cents.</p> |
| <h5 id="adding-fields-to-a-schema"><strong>Adding fields to a schema</strong></h5> |
| <p>The AddFields transform can be used to extend a schema with new fields. Input rows will be extended to the new schema by |
| inserting null values for the new fields, though alternate default values can be specified; if the default null value |
| is used then the new field type will be marked as nullable. Nested subfields can be added using the field selection |
| syntax, including nested fields inside arrays or map values.</p> |
| <p>For example, the following application</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">AddFields</span><span class="o">.&lt;</span><span class="n">PurchasePojo</span><span class="o">&gt;</span><span class="n">create</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">field</span><span class="o">(</span><span class="s">&#34;timeOfDaySeconds&#34;</span><span class="o">,</span> <span class="n">FieldType</span><span class="o">.</span><span class="na">INT32</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">field</span><span class="o">(</span><span class="s">&#34;shippingAddress.deliveryNotes&#34;</span><span class="o">,</span> <span class="n">FieldType</span><span class="o">.</span><span class="na">STRING</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">field</span><span class="o">(</span><span class="s">&#34;transactions.isFlagged&#34;</span><span class="o">,</span> <span class="n">FieldType</span><span class="o">.</span><span class="na">BOOLEAN</span><span class="o">,</span> <span class="kc">false</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Results in a <code>PCollection</code> with an expanded schema. All of the rows and fields of the input, but also with the specified |
| fields added to the schema. All resulting rows will have null values filled in for the <strong>timeOfDaySeconds</strong> and the |
| <strong>shippingAddress.deliveryNotes</strong> fields, and a false value filled in for the <strong>transactions.isFlagged</strong> field.</p> |
| <h5 id="removing-fields-from-a-schema"><strong>Removing fields from a schema</strong></h5> |
| <p><code>DropFields</code> allows specific fields to be dropped from a schema. Input rows will have their schemas truncated, and any |
| values for dropped fields will be removed from the output. Nested fields can also be dropped using the field selection |
| syntax.</p> |
| <p>For example, the following snippet</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">DropFields</span><span class="o">.</span><span class="na">fields</span><span class="o">(</span><span class="s">&#34;userId&#34;</span><span class="o">,</span> <span class="s">&#34;shippingAddress.streetAddress&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Results in a copy of the input with those two fields and their corresponding values removed.</p> |
| <h5 id="renaming-schema-fields"><strong>Renaming schema fields</strong></h5> |
| <p><code>RenameFields</code> allows specific fields in a schema to be renamed. The field values in input rows are left unchanged, only |
| the schema is modified. This transform is often used to prepare records for output to a schema-aware sink, such as an |
| RDBMS, to make sure that the <code>PCollection</code> schema field names match that of the output. It can also be used to rename |
| fields generated by other transforms to make them more usable (similar to SELECT AS in SQL). Nested fields can also be |
| renamed using the field-selection syntax.</p> |
| <p>For example, the following snippet</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">RenameFields</span><span class="o">.&lt;</span><span class="n">PurchasePojo</span><span class="o">&gt;</span><span class="n">create</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">rename</span><span class="o">(</span><span class="s">&#34;userId&#34;</span><span class="o">,</span> <span class="s">&#34;userIdentifier&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">rename</span><span class="o">(</span><span class="s">&#34;shippingAddress.streetAddress&#34;</span><span class="o">,</span> <span class="s">&#34;shippingAddress.street&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Results in the same set of unmodified input elements, however the schema on the PCollection has been changed to rename |
| <strong>userId</strong> to <strong>userIdentifier</strong> and <strong>shippingAddress.streetAddress</strong> to <strong>shippingAddress.street</strong>.</p> |
| <h5 id="converting-between-types"><strong>Converting between types</strong></h5> |
| <p>As mentioned, Beam can automatically convert between different Java types, as long as those types have equivalent |
| schemas. One way to do this is by using the <code>Convert</code> transform, as follows.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">PurchaseBean</span><span class="o">&gt;</span> <span class="n">purchaseBeans</span> <span class="o">=</span> <span class="n">readPurchasesAsBeans</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">PurchasePojo</span><span class="o">&gt;</span> <span class="n">pojoPurchases</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">purchaseBeans</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Convert</span><span class="o">.</span><span class="na">to</span><span class="o">(</span><span class="n">PurchasePojo</span><span class="o">.</span><span class="na">class</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Beam will validate that the inferred schema for <code>PurchasePojo</code> matches that of the input <code>PCollection</code>, and will |
| then cast to a <code>PCollection&lt;PurchasePojo&gt;</code>.</p> |
| <p>Since the <code>Row</code> class can support any schema, any <code>PCollection</code> with schema can be cast to a <code>PCollection</code> of rows, as |
| follows.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Row</span><span class="o">&gt;</span> <span class="n">purchaseRows</span> <span class="o">=</span> <span class="n">purchaseBeans</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Convert</span><span class="o">.</span><span class="na">toRows</span><span class="o">());</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>If the source type is a single-field schema, Convert will also convert to the type of the field if asked, effectively |
| unboxing the row. For example, give a schema with a single INT64 field, the following will convert it to a |
| <code>PCollection&lt;Long&gt;</code></p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">&gt;</span> <span class="n">longs</span> <span class="o">=</span> <span class="n">rows</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Convert</span><span class="o">.</span><span class="na">to</span><span class="o">(</span><span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">longs</span><span class="o">()));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>In all cases, type checking is done at pipeline graph construction, and if the types do not match the schema then the |
| pipeline will fail to launch.</p> |
| <h4 id="663-schemas-in-pardo">6.6.3. Schemas in ParDo</h4> |
| <p>A <code>PCollection</code> with a schema can apply a <code>ParDo</code>, just like any other <code>PCollection</code>. However the Beam runner is aware |
| of schemas when applying a <code>ParDo</code>, which enables additional functionality.</p> |
| <h5 id="input-conversion"><strong>Input conversion</strong></h5> |
| <p class="language-go">Beam does not yet support input conversion in Go.</p> |
| <p>Since Beam knows the schema of the source <code>PCollection</code>, it can automatically convert the elements to any Java type for |
| which a matching schema is known. For example, using the above-mentioned Transaction schema, say we have the following |
| <code>PCollection</code>:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">PurchasePojo</span><span class="o">&gt;</span> <span class="n">purchases</span> <span class="o">=</span> <span class="n">readPurchases</span><span class="o">();</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>If there were no schema, then the applied <code>DoFn</code> would have to accept an element of type <code>TransactionPojo</code>. However |
| since there is a schema, you could apply the following DoFn:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">PurchasePojo</span><span class="o">,</span> <span class="n">PurchasePojo</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span><span class="nd">@Element</span> <span class="n">PurchaseBean</span> <span class="n">purchase</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Even though the <code>@Element</code> parameter does not match the Java type of the <code>PCollection</code>, since it has a matching schema |
| Beam will automatically convert elements. If the schema does not match, Beam will detect this at graph-construction time |
| and will fail the job with a type error.</p> |
| <p>Since every schema can be represented by a Row type, Row can also be used here:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">appy</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">PurchasePojo</span><span class="o">,</span> <span class="n">PurchasePojo</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span><span class="nd">@Element</span> <span class="n">Row</span> <span class="n">purchase</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h5 id="input-selection"><strong>Input selection</strong></h5> |
| <p>Since the input has a schema, you can also automatically select specific fields to process in the DoFn.</p> |
| <p>Given the above purchases <code>PCollection</code>, say you want to process just the userId and the itemId fields. You can do these |
| using the above-described selection expressions, as follows:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">appy</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">PurchasePojo</span><span class="o">,</span> <span class="n">PurchasePojo</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@FieldAccess</span><span class="o">(</span><span class="s">&#34;userId&#34;</span><span class="o">)</span> <span class="n">String</span> <span class="n">userId</span><span class="o">,</span> <span class="nd">@FieldAccess</span><span class="o">(</span><span class="s">&#34;itemId&#34;</span><span class="o">)</span> <span class="kt">long</span> <span class="n">itemId</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>You can also select nested fields, as follows.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">purchases</span><span class="o">.</span><span class="na">appy</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">PurchasePojo</span><span class="o">,</span> <span class="n">PurchasePojo</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@FieldAccess</span><span class="o">(</span><span class="s">&#34;shippingAddress.street&#34;</span><span class="o">)</span> <span class="n">String</span> <span class="n">street</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>For more information, see the section on field-selection expressions. When selecting subschemas, Beam will |
| automatically convert to any matching schema type, just like when reading the entire row.</p> |
| <h2 id="data-encoding-and-type-safety">7. Data encoding and type safety</h2> |
| <nav class="language-switcher"> |
| <strong>Adapt for:</strong> |
| <ul> |
| <li data-value="java" class="active">Java SDK</li> |
| <li data-value="py">Python SDK</li> |
| <li data-value="go">Go SDK</li> |
| <li data-value="typescript">TypeScript SDK</li> |
| </ul> |
| </nav> |
| <p>When Beam runners execute your pipeline, they often need to materialize the |
| intermediate data in your <code>PCollection</code>s, which requires converting elements to |
| and from byte strings. The Beam SDKs use objects called <code>Coder</code>s to describe how |
| the elements of a given <code>PCollection</code> may be encoded and decoded.</p> |
| <blockquote> |
| <p>Note that coders are unrelated to parsing or formatting data when interacting |
| with external data sources or sinks. Such parsing or formatting should |
| typically be done explicitly, using transforms such as <code>ParDo</code> or |
| <code>MapElements</code>.</p> |
| </blockquote> |
| <p class="language-java">In the Beam SDK for Java, the type <code>Coder</code> provides the methods required for |
| encoding and decoding data. The SDK for Java provides a number of Coder |
| subclasses that work with a variety of standard Java types, such as Integer, |
| Long, Double, StringUtf8 and more. You can find all of the available Coder |
| subclasses in the <a href="https://github.com/apache/beam/tree/master/sdks/java/core/src/main/java/org/apache/beam/sdk/coders">Coder package</a>.</p> |
| <p class="language-py">In the Beam SDK for Python, the type <code>Coder</code> provides the methods required for |
| encoding and decoding data. The SDK for Python provides a number of Coder |
| subclasses that work with a variety of standard Python types, such as primitive |
| types, Tuple, Iterable, StringUtf8 and more. You can find all of the available |
| Coder subclasses in the |
| <a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/coders">apache_beam.coders</a> |
| package.</p> |
| <p class="language-go">Standard Go types like <code>int</code>, <code>int64</code> <code>float64</code>, <code>[]byte</code>, and <code>string</code> and more are coded using builtin coders. |
| Structs and pointers to structs default using Beam Schema Row encoding. |
| However, users can build and register custom coders with <code>beam.RegisterCoder</code>. |
| You can find available Coder functions in the |
| <a href="https://pkg.go.dev/github.com/apache/beam/sdks/go/pkg/beam/core/graph/coder">coder</a> |
| package.</p> |
| <p class="language-typescript">Standard Typescript types like <code>number</code>, <code>UInt8Array</code> and <code>string</code> and more are coded using builtin coders. |
| Json objects and arrays are encoded via a BSON encoding. |
| For these types, coders need not be specified unless interacting with cross-language transforms. |
| Users can build custom coders by extending <code>beam.coders.Coder</code> |
| for use with <code>withCoderInternal</code>, but generally logical types are preferred for this case.</p> |
| <blockquote> |
| <p>Note that coders do not necessarily have a 1:1 relationship with types. For |
| example, the Integer type can have multiple valid coders, and input and output |
| data can use different Integer coders. A transform might have Integer-typed |
| input data that uses BigEndianIntegerCoder, and Integer-typed output data that |
| uses VarIntCoder.</p> |
| </blockquote> |
| <h3 id="specifying-coders">7.1. Specifying coders</h3> |
| <p>The Beam SDKs require a coder for every <code>PCollection</code> in your pipeline. In most |
| cases, the Beam SDK is able to automatically infer a <code>Coder</code> for a <code>PCollection</code> |
| based on its element type or the transform that produces it, however, in some |
| cases the pipeline author will need to specify a <code>Coder</code> explicitly, or develop |
| a <code>Coder</code> for their custom type.</p> |
| <p class="language-java">You can explicitly set the coder for an existing <code>PCollection</code> by using the |
| method <code>PCollection.setCoder</code>. Note that you cannot call <code>setCoder</code> on a |
| <code>PCollection</code> that has been finalized (e.g. by calling <code>.apply</code> on it).</p> |
| <p class="language-java">You can get the coder for an existing <code>PCollection</code> by using the method |
| <code>getCoder</code>. This method will fail with an <code>IllegalStateException</code> if a coder has |
| not been set and cannot be inferred for the given <code>PCollection</code>.</p> |
| <p>Beam SDKs use a variety of mechanisms when attempting to automatically infer the |
| <code>Coder</code> for a <code>PCollection</code>.</p> |
| <p class="language-java">Each pipeline object has a <code>CoderRegistry</code>. The <code>CoderRegistry</code> represents a |
| mapping of Java types to the default coders that the pipeline should use for |
| <code>PCollection</code>s of each type.</p> |
| <p class="language-py">The Beam SDK for Python has a <code>CoderRegistry</code> that represents a mapping of |
| Python types to the default coder that should be used for <code>PCollection</code>s of each |
| type.</p> |
| <p class="language-go">The Beam SDK for Go allows users to register default coder |
| implementations with <code>beam.RegisterCoder</code>.</p> |
| <p class="language-java">By default, the Beam SDK for Java automatically infers the <code>Coder</code> for the |
| elements of a <code>PCollection</code> produced by a <code>PTransform</code> using the type parameter |
| from the transform&rsquo;s function object, such as <code>DoFn</code>. In the case of <code>ParDo</code>, |
| for example, a <code>DoFn&lt;Integer, String&gt;</code> function object accepts an input element |
| of type <code>Integer</code> and produces an output element of type <code>String</code>. In such a |
| case, the SDK for Java will automatically infer the default <code>Coder</code> for the |
| output <code>PCollection&lt;String&gt;</code> (in the default pipeline <code>CoderRegistry</code>, this is |
| <code>StringUtf8Coder</code>).</p> |
| <p class="language-py">By default, the Beam SDK for Python automatically infers the <code>Coder</code> for the |
| elements of an output <code>PCollection</code> using the typehints from the transform&rsquo;s |
| function object, such as <code>DoFn</code>. In the case of <code>ParDo</code>, for example a <code>DoFn</code> |
| with the typehints <code>@beam.typehints.with_input_types(int)</code> and |
| <code>@beam.typehints.with_output_types(str)</code> accepts an input element of type int |
| and produces an output element of type str. In such a case, the Beam SDK for |
| Python will automatically infer the default <code>Coder</code> for the output <code>PCollection</code> |
| (in the default pipeline <code>CoderRegistry</code>, this is <code>BytesCoder</code>).</p> |
| <p class="language-go">By default, the Beam SDK for Go automatically infers the <code>Coder</code> for the elements of an output <code>PCollection</code> by the output of the transform&rsquo;s function object, such as a <code>DoFn</code>. |
| In the case of <code>ParDo</code>, for example a <code>DoFn</code> |
| with the parameters of <code>v int, emit func(string)</code> accepts an input element of type <code>int</code> |
| and produces an output element of type <code>string</code>. |
| In such a case, the Beam SDK for Go will automatically infer the default <code>Coder</code> for the output <code>PCollection</code> to be the <code>string_utf8</code> coder.</p> |
| <span class="language-java"> |
| <blockquote> |
| <p><strong>Note:</strong> If you create your <code>PCollection</code> from in-memory data by using the |
| <code>Create</code> transform, you cannot rely on coder inference and default coders. |
| <code>Create</code> does not have access to any typing information for its arguments, and |
| may not be able to infer a coder if the argument list contains a value whose |
| exact run-time class doesn&rsquo;t have a default coder registered.</p> |
| </blockquote> |
| </span> |
| <p class="language-java">When using <code>Create</code>, the simplest way to ensure that you have the correct coder |
| is by invoking <code>withCoder</code> when you apply the <code>Create</code> transform.</p> |
| <h3 id="default-coders-and-the-coderregistry">7.2. Default coders and the CoderRegistry</h3> |
| <p>Each Pipeline object has a <code>CoderRegistry</code> object, which maps language types to |
| the default coder the pipeline should use for those types. You can use the |
| <code>CoderRegistry</code> yourself to look up the default coder for a given type, or to |
| register a new default coder for a given type.</p> |
| <p><code>CoderRegistry</code> contains a default mapping of coders to standard |
| <span class="language-java">Java</span><span class="language-py">Python</span> |
| types for any pipeline you create using the Beam SDK for |
| <span class="language-java">Java</span><span class="language-py">Python</span>. |
| The following table shows the standard mapping:</p> |
| <p class="language-java"><div class="table-container-wrapper"> |
| <table class="table-wrapper--pr"> |
| <thead> |
| <tr class="header"> |
| <th>Java Type</th> |
| <th>Default Coder</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr class="odd"> |
| <td>Double</td> |
| <td>DoubleCoder</td> |
| </tr> |
| <tr class="even"> |
| <td>Instant</td> |
| <td>InstantCoder</td> |
| </tr> |
| <tr class="odd"> |
| <td>Integer</td> |
| <td>VarIntCoder</td> |
| </tr> |
| <tr class="even"> |
| <td>Iterable</td> |
| <td>IterableCoder</td> |
| </tr> |
| <tr class="odd"> |
| <td>KV</td> |
| <td>KvCoder</td> |
| </tr> |
| <tr class="even"> |
| <td>List</td> |
| <td>ListCoder</td> |
| </tr> |
| <tr class="odd"> |
| <td>Map</td> |
| <td>MapCoder</td> |
| </tr> |
| <tr class="even"> |
| <td>Long</td> |
| <td>VarLongCoder</td> |
| </tr> |
| <tr class="odd"> |
| <td>String</td> |
| <td>StringUtf8Coder</td> |
| </tr> |
| <tr class="even"> |
| <td>TableRow</td> |
| <td>TableRowJsonCoder</td> |
| </tr> |
| <tr class="odd"> |
| <td>Void</td> |
| <td>VoidCoder</td> |
| </tr> |
| <tr class="even"> |
| <td>byte[ ]</td> |
| <td>ByteArrayCoder</td> |
| </tr> |
| <tr class="odd"> |
| <td>TimestampedValue</td> |
| <td>TimestampedValueCoder</td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
| </p> |
| <p class="language-py"><table class="table-wrapper--pr"> |
| <thead> |
| <tr class="header"> |
| <th>Python Type</th> |
| <th>Default Coder</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr class="odd"> |
| <td>int</td> |
| <td>VarIntCoder</td> |
| </tr> |
| <tr class="even"> |
| <td>float</td> |
| <td>FloatCoder</td> |
| </tr> |
| <tr class="odd"> |
| <td>str</td> |
| <td>BytesCoder</td> |
| </tr> |
| <tr class="even"> |
| <td>bytes</td> |
| <td>StrUtf8Coder</td> |
| </tr> |
| <tr class="odd"> |
| <td>Tuple</td> |
| <td>TupleCoder</td> |
| </tr> |
| </tbody> |
| </table> |
| </p> |
| <h4 id="default-coder-lookup">7.2.1. Looking up a default coder</h4> |
| <p class="language-java">You can use the method <code>CoderRegistry.getCoder</code> to determine the default |
| Coder for a Java type. You can access the <code>CoderRegistry</code> for a given pipeline |
| by using the method <code>Pipeline.getCoderRegistry</code>. This allows you to determine |
| (or set) the default Coder for a Java type on a per-pipeline basis: i.e. &ldquo;for |
| this pipeline, verify that Integer values are encoded using |
| <code>BigEndianIntegerCoder</code>.&rdquo;</p> |
| <p class="language-py">You can use the method <code>CoderRegistry.get_coder</code> to determine the default Coder |
| for a Python type. You can use <code>coders.registry</code> to access the <code>CoderRegistry</code>. |
| This allows you to determine (or set) the default Coder for a Python type.</p> |
| <p class="language-go">You can use the <code>beam.NewCoder</code> function to determine the default Coder for a Go type.</p> |
| <h4 id="setting-default-coder">7.2.2. Setting the default coder for a type</h4> |
| <p class="language-java language-py">To set the default Coder for a |
| <span class="language-java">Java</span><span class="language-py">Python</span> |
| type for a particular pipeline, you obtain and modify the pipeline&rsquo;s |
| <code>CoderRegistry</code>. You use the method |
| <span class="language-java"><code>Pipeline.getCoderRegistry</code></span> |
| <span class="language-py"><code>coders.registry</code></span> |
| to get the <code>CoderRegistry</code> object, and then use the method |
| <span class="language-java"><code>CoderRegistry.registerCoder</code></span> |
| <span class="language-py"><code>CoderRegistry.register_coder</code></span> |
| to register a new <code>Coder</code> for the target type.</p> |
| <p class="language-go">To set the default Coder for a Go type you use the function <code>beam.RegisterCoder</code> to register a encoder and decoder functions for the target type. |
| However, built in types like <code>int</code>, <code>string</code>, <code>float64</code>, etc cannot have their coders override.</p> |
| <p class="language-java language-py">The following example code demonstrates how to set a default Coder, in this case |
| <code>BigEndianIntegerCoder</code>, for |
| <span class="language-java">Integer</span><span class="language-py">int</span> |
| values for a pipeline.</p> |
| <p class="language-go">The following example code demonstrates how to set a custom Coder for <code>MyCustomType</code> elements.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PipelineOptions</span> <span class="n">options</span> <span class="o">=</span> <span class="n">PipelineOptionsFactory</span><span class="o">.</span><span class="na">create</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">Pipeline</span> <span class="n">p</span> <span class="o">=</span> <span class="n">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">CoderRegistry</span> <span class="n">cr</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="na">getCoderRegistry</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">cr</span><span class="o">.</span><span class="na">registerCoder</span><span class="o">(</span><span class="n">Integer</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> <span class="n">BigEndianIntegerCoder</span><span class="o">.</span><span class="na">class</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">apache_beam</span><span class="o">.</span><span class="n">coders</span><span class="o">.</span><span class="n">registry</span><span class="o">.</span><span class="n">register_coder</span><span class="p">(</span><span class="nb">int</span><span class="p">,</span> <span class="n">BigEndianIntegerCoder</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">MyCustomType</span> <span class="kd">struct</span><span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// See documentation on beam.RegisterCoder for other supported coder forms. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">encode</span><span class="p">(</span><span class="nx">MyCustomType</span><span class="p">)</span> <span class="p">[]</span><span class="kt">byte</span> <span class="p">{</span> <span class="o">...</span> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">decode</span><span class="p">(</span><span class="nx">b</span> <span class="p">[]</span><span class="kt">byte</span><span class="p">)</span> <span class="nx">MyCustomType</span> <span class="p">{</span> <span class="o">...</span> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nf">RegisterCoder</span><span class="p">(</span><span class="nx">reflect</span><span class="p">.</span><span class="nf">TypeOf</span><span class="p">((</span><span class="o">*</span><span class="nx">MyCustomType</span><span class="p">)(</span><span class="kc">nil</span><span class="p">)).</span><span class="nf">Elem</span><span class="p">(),</span> <span class="nx">encode</span><span class="p">,</span> <span class="nx">decode</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="annotating-custom-type-default-coder">7.2.3. Annotating a custom data type with a default coder</h4> |
| <span class="language-java"> |
| <p>If your pipeline program defines a custom data type, you can use the |
| <code>@DefaultCoder</code> annotation to specify the coder to use with that type. |
| By default, Beam will use <code>SerializableCoder</code> which uses Java serialization, |
| but it has drawbacks:</p> |
| <ol> |
| <li> |
| <p>It is inefficient in encoding size and speed. |
| See this <a href="https://blog.softwaremill.com/the-best-serialization-strategy-for-event-sourcing-9321c299632b">comparison of Java serialization methods.</a></p> |
| </li> |
| <li> |
| <p>It is non-deterministic: it may produce different binary encodings for two |
| equivalent objects.</p> |
| <p>For key/value pairs, the correctness of key-based operations |
| (GroupByKey, Combine) and per-key State depends on having a deterministic |
| coder for the key</p> |
| </li> |
| </ol> |
| <p>You can use the <code>@DefaultCoder</code> annotation to set a new default as follows:</p> |
| </span> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@DefaultCoder</span><span class="o">(</span><span class="n">AvroCoder</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyCustomDataType</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">If you&rsquo;ve created a custom coder to match your data type, and you want to use |
| the <code>@DefaultCoder</code> annotation, your coder class must implement a static |
| <code>Coder.of(Class&lt;T&gt;)</code> factory method.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyCustomCoder</span> <span class="kd">implements</span> <span class="n">Coder</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kd">static</span> <span class="n">Coder</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="nf">of</span><span class="o">(</span><span class="n">Class</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="n">clazz</span><span class="o">)</span> <span class="o">{...}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="nd">@DefaultCoder</span><span class="o">(</span><span class="n">MyCustomCoder</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyCustomDataType</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-py language-go">The Beam SDK for <span class="language-py">Python</span><span class="language-go">Go</span> |
| does not support annotating data types with a default coder. |
| If you would like to set a default coder, use the method described in the |
| previous section, <em>Setting the default coder for a type</em>.</p> |
| <h2 id="windowing">8. Windowing</h2> |
| <p>Windowing subdivides a <code>PCollection</code> according to the timestamps of its |
| individual elements. Transforms that aggregate multiple elements, such as |
| <code>GroupByKey</code> and <code>Combine</code>, work implicitly on a per-window basis — they process |
| each <code>PCollection</code> as a succession of multiple, finite windows, though the |
| entire collection itself may be of unbounded size.</p> |
| <p>A related concept, called <strong>triggers</strong>, determines when to emit the results of |
| aggregation as unbounded data arrives. You can use triggers to refine the |
| windowing strategy for your <code>PCollection</code>. Triggers allow you to deal with |
| late-arriving data or to provide early results. See the <a href="#triggers">triggers</a> |
| section for more information.</p> |
| <h3 id="windowing-basics">8.1. Windowing basics</h3> |
| <p>Some Beam transforms, such as <code>GroupByKey</code> and <code>Combine</code>, group multiple |
| elements by a common key. Ordinarily, that grouping operation groups all of the |
| elements that have the same key within the entire data set. With an unbounded |
| data set, it is impossible to collect all of the elements, since new elements |
| are constantly being added and may be infinitely many (e.g. streaming data). If |
| you are working with unbounded <code>PCollection</code>s, windowing is especially useful.</p> |
| <p>In the Beam model, any <code>PCollection</code> (including unbounded <code>PCollection</code>s) can be |
| subdivided into logical windows. Each element in a <code>PCollection</code> is assigned to |
| one or more windows according to the <code>PCollection</code>&rsquo;s windowing function, and |
| each individual window contains a finite number of elements. Grouping transforms |
| then consider each <code>PCollection</code>&rsquo;s elements on a per-window basis. <code>GroupByKey</code>, |
| for example, implicitly groups the elements of a <code>PCollection</code> by <em>key and |
| window</em>.</p> |
| <p><strong>Caution:</strong> Beam&rsquo;s default windowing behavior is to assign all elements of a |
| <code>PCollection</code> to a single, global window and discard late data, <em>even for |
| unbounded <code>PCollection</code>s</em>. Before you use a grouping transform such as |
| <code>GroupByKey</code> on an unbounded <code>PCollection</code>, you must do at least one of the |
| following:</p> |
| <ul> |
| <li>Set a non-global windowing function. See <a href="#setting-your-pcollections-windowing-function">Setting your PCollection&rsquo;s |
| windowing function</a>.</li> |
| <li>Set a non-default <a href="#triggers">trigger</a>. This allows the global window to emit |
| results under other conditions, since the default windowing behavior (waiting |
| for all data to arrive) will never occur.</li> |
| </ul> |
| <p>If you don&rsquo;t set a non-global windowing function or a non-default trigger for |
| your unbounded <code>PCollection</code> and subsequently use a grouping transform such as |
| <code>GroupByKey</code> or <code>Combine</code>, your pipeline will generate an error upon |
| construction and your job will fail.</p> |
| <h4 id="windowing-constraints">8.1.1. Windowing constraints</h4> |
| <p>After you set the windowing function for a <code>PCollection</code>, the elements&rsquo; windows |
| are used the next time you apply a grouping transform to that <code>PCollection</code>. |
| Window grouping occurs on an as-needed basis. If you set a windowing function |
| using the <code>Window</code> transform, each element is assigned to a window, but the |
| windows are not considered until <code>GroupByKey</code> or <code>Combine</code> aggregates across a |
| window and key. This can have different effects on your pipeline. Consider the |
| example pipeline in the figure below:</p> |
| <p><img src="/images/windowing-pipeline-unbounded.svg" alt="Diagram of pipeline applying windowing"></p> |
| <p><strong>Figure 3:</strong> Pipeline applying windowing</p> |
| <p>In the above pipeline, we create an unbounded <code>PCollection</code> by reading a set of |
| key/value pairs using <code>KafkaIO</code>, and then apply a windowing function to that |
| collection using the <code>Window</code> transform. We then apply a <code>ParDo</code> to the |
| collection, and then later group the result of that <code>ParDo</code> using <code>GroupByKey</code>. |
| The windowing function has no effect on the <code>ParDo</code> transform, because the |
| windows are not actually used until they&rsquo;re needed for the <code>GroupByKey</code>. |
| Subsequent transforms, however, are applied to the result of the <code>GroupByKey</code> &ndash; |
| data is grouped by both key and window.</p> |
| <h4 id="windowing-bounded-collections">8.1.2. Windowing with bounded PCollections</h4> |
| <p>You can use windowing with fixed-size data sets in <strong>bounded</strong> <code>PCollection</code>s. |
| However, note that windowing considers only the implicit timestamps attached to |
| each element of a <code>PCollection</code>, and data sources that create fixed data sets |
| (such as <code>TextIO</code>) assign the same timestamp to every element. This means that |
| all the elements are by default part of a single, global window.</p> |
| <p>To use windowing with fixed data sets, you can assign your own timestamps to |
| each element. To assign timestamps to elements, use a <code>ParDo</code> transform with a |
| <code>DoFn</code> that outputs each element with a new timestamp (for example, the |
| <a href="https://beam.apache.org/releases/javadoc/2.56.0/index.html?org/apache/beam/sdk/transforms/WithTimestamps.html">WithTimestamps</a> |
| transform in the Beam SDK for Java).</p> |
| <p>To illustrate how windowing with a bounded <code>PCollection</code> can affect how your |
| pipeline processes data, consider the following pipeline:</p> |
| <p><img src="/images/unwindowed-pipeline-bounded.svg" alt="Diagram of GroupByKey and ParDo without windowing, on a bounded collection"></p> |
| <p><strong>Figure 4:</strong> <code>GroupByKey</code> and <code>ParDo</code> without windowing, on a bounded collection.</p> |
| <p>In the above pipeline, we create a bounded <code>PCollection</code> by reading lines from a |
| file using <code>TextIO</code>. We then group the collection using <code>GroupByKey</code>, |
| and apply a <code>ParDo</code> transform to the grouped <code>PCollection</code>. In this example, the |
| <code>GroupByKey</code> creates a collection of unique keys, and then <code>ParDo</code> gets applied |
| exactly once per key.</p> |
| <p>Note that even if you don’t set a windowing function, there is still a window &ndash; |
| all elements in your <code>PCollection</code> are assigned to a single global window.</p> |
| <p>Now, consider the same pipeline, but using a windowing function:</p> |
| <p><img src="/images/windowing-pipeline-bounded.svg" alt="Diagram of GroupByKey and ParDo with windowing, on a bounded collection"></p> |
| <p><strong>Figure 5:</strong> <code>GroupByKey</code> and <code>ParDo</code> with windowing, on a bounded collection.</p> |
| <p>As before, the pipeline creates a bounded <code>PCollection</code> by reading lines from a |
| file. We then set a <a href="#setting-your-pcollections-windowing-function">windowing function</a> |
| for that <code>PCollection</code>. The <code>GroupByKey</code> transform groups the elements of the |
| <code>PCollection</code> by both key and window, based on the windowing function. The |
| subsequent <code>ParDo</code> transform gets applied multiple times per key, once for each |
| window.</p> |
| <h3 id="provided-windowing-functions">8.2. Provided windowing functions</h3> |
| <p>You can define different kinds of windows to divide the elements of your |
| <code>PCollection</code>. Beam provides several windowing functions, including:</p> |
| <ul> |
| <li>Fixed Time Windows</li> |
| <li>Sliding Time Windows</li> |
| <li>Per-Session Windows</li> |
| <li>Single Global Window</li> |
| <li>Calendar-based Windows (not supported by the Beam SDK for Python or Go)</li> |
| </ul> |
| <p>You can also define your own <code>WindowFn</code> if you have a more complex need.</p> |
| <p>Note that each element can logically belong to more than one window, depending |
| on the windowing function you use. Sliding time windowing, for example, can |
| create overlapping windows wherein a single element can be assigned to multiple |
| windows. However, each element in a <code>PCollection</code> can only be in one window, so |
| if an element is assigned to multiple windows, the element is conceptually |
| duplicated into each of the windows and each element is identical except for its |
| window.</p> |
| <h4 id="fixed-time-windows">8.2.1. Fixed time windows</h4> |
| <p>The simplest form of windowing is using <strong>fixed time windows</strong>: given a |
| timestamped <code>PCollection</code> which might be continuously updating, each window |
| might capture (for example) all elements with timestamps that fall into a 30 |
| second interval.</p> |
| <p>A fixed time window represents a consistent duration, non overlapping time |
| interval in the data stream. Consider windows with a 30 second duration: all |
| of the elements in your unbounded <code>PCollection</code> with timestamp values from |
| 0:00:00 up to (but not including) 0:00:30 belong to the first window, elements |
| with timestamp values from 0:00:30 up to (but not including) 0:01:00 belong to |
| the second window, and so on.</p> |
| <p><img src="/images/fixed-time-windows.png" alt="Diagram of fixed time windows, 30s in duration"></p> |
| <p><strong>Figure 6:</strong> Fixed time windows, 30s in duration.</p> |
| <h4 id="sliding-time-windows">8.2.2. Sliding time windows</h4> |
| <p>A <strong>sliding time window</strong> also represents time intervals in the data stream; |
| however, sliding time windows can overlap. For example, each window might |
| capture 60 seconds worth of data, but a new window starts every 30 seconds. |
| The frequency with which sliding windows begin is called the <em>period</em>. |
| Therefore, our example would have a window <em>duration</em> of 60 seconds and a |
| <em>period</em> of 30 seconds.</p> |
| <p>Because multiple windows overlap, most elements in a data set will belong to |
| more than one window. This kind of windowing is useful for taking running |
| averages of data; using sliding time windows, you can compute a running average |
| of the past 60 seconds&rsquo; worth of data, updated every 30 seconds, in our |
| example.</p> |
| <p><img src="/images/sliding-time-windows.png" alt="Diagram of sliding time windows, with 1 minute window duration and 30s window period"></p> |
| <p><strong>Figure 7:</strong> Sliding time windows, with 1 minute window duration and 30s window |
| period.</p> |
| <h4 id="session-windows">8.2.3. Session windows</h4> |
| <p>A <strong>session window</strong> function defines windows that contain elements that are |
| within a certain gap duration of another element. Session windowing applies on a |
| per-key basis and is useful for data that is irregularly distributed with |
| respect to time. For example, a data stream representing user mouse activity may |
| have long periods of idle time interspersed with high concentrations of clicks. |
| If data arrives after the minimum specified gap duration time, this initiates |
| the start of a new window.</p> |
| <p><img src="/images/session-windows.png" alt="Diagram of session windows with a minimum gap duration"></p> |
| <p><strong>Figure 8:</strong> Session windows, with a minimum gap duration. Note how each data key |
| has different windows, according to its data distribution.</p> |
| <h4 id="single-global-window">8.2.4. The single global window</h4> |
| <p>By default, all data in a <code>PCollection</code> is assigned to the single global window, |
| and late data is discarded. If your data set is of a fixed size, you can use the |
| global window default for your <code>PCollection</code>.</p> |
| <p>You can use the single global window if you are working with an unbounded data set |
| (e.g. from a streaming data source) but use caution when applying aggregating |
| transforms such as <code>GroupByKey</code> and <code>Combine</code>. The single global window with a |
| default trigger generally requires the entire data set to be available before |
| processing, which is not possible with continuously updating data. To perform |
| aggregations on an unbounded <code>PCollection</code> that uses global windowing, you |
| should specify a non-default trigger for that <code>PCollection</code>.</p> |
| <h3 id="setting-your-pcollections-windowing-function">8.3. Setting your PCollection&rsquo;s windowing function</h3> |
| <p>You can set the windowing function for a <code>PCollection</code> by applying the <code>Window</code> |
| transform. When you apply the <code>Window</code> transform, you must provide a <code>WindowFn</code>. |
| The <code>WindowFn</code> determines the windowing function your <code>PCollection</code> will use for |
| subsequent grouping transforms, such as a fixed or sliding time window.</p> |
| <p>When you set a windowing function, you may also want to set a trigger for your |
| <code>PCollection</code>. The trigger determines when each individual window is aggregated |
| and emitted, and helps refine how the windowing function performs with respect |
| to late data and computing early results. See the <a href="#triggers">triggers</a> section |
| for more information.</p> |
| <p class="language-yaml">In Beam YAML windowing specifications can also be placed directly on any |
| transform rather than requiring an explicit <code>WindowInto</code> transform.</p> |
| <h4 id="using-fixed-time-windows">8.3.1. Fixed-time windows</h4> |
| <p>The following example code shows how to apply <code>Window</code> to divide a <code>PCollection</code> |
| into fixed windows, each 60 seconds in length:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">items</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">fixedWindowedItems</span> <span class="o">=</span> <span class="n">items</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Window</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">into</span><span class="o">(</span><span class="n">FixedWindows</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="n">60</span><span class="o">))));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam</span> <span class="kn">import</span> <span class="n">window</span> |
| </span></span><span class="line"><span class="cl"><span class="n">fixed_windowed_items</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">items</span> <span class="o">|</span> <span class="s1">&#39;window&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span><span class="n">window</span><span class="o">.</span><span class="n">FixedWindows</span><span class="p">(</span><span class="mi">60</span><span class="p">)))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">fixedWindowedItems</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">WindowInto</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">window</span><span class="p">.</span><span class="nf">NewFixedWindows</span><span class="p">(</span><span class="mi">60</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">items</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="nx">pcoll</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">windowInto</span><span class="p">(</span><span class="nx">windowings</span><span class="p">.</span><span class="nx">fixedWindows</span><span class="p">(</span><span class="mi">60</span><span class="p">)))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">WindowInto</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">windowing</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">fixed</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">size</span><span class="p">:</span><span class="w"> </span><span class="l">60s</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="using-sliding-time-windows">8.3.2. Sliding time windows</h4> |
| <p>The following example code shows how to apply <code>Window</code> to divide a <code>PCollection</code> |
| into sliding time windows. Each window is 30 seconds in length, and a new window |
| begins every five seconds:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">items</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">slidingWindowedItems</span> <span class="o">=</span> <span class="n">items</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Window</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">into</span><span class="o">(</span><span class="n">SlidingWindows</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="n">30</span><span class="o">)).</span><span class="na">every</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="n">5</span><span class="o">))));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam</span> <span class="kn">import</span> <span class="n">window</span> |
| </span></span><span class="line"><span class="cl"><span class="n">sliding_windowed_items</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">items</span> <span class="o">|</span> <span class="s1">&#39;window&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span><span class="n">window</span><span class="o">.</span><span class="n">SlidingWindows</span><span class="p">(</span><span class="mi">30</span><span class="p">,</span> <span class="mi">5</span><span class="p">)))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">slidingWindowedItems</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">WindowInto</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">window</span><span class="p">.</span><span class="nf">NewSlidingWindows</span><span class="p">(</span><span class="mi">5</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">,</span> <span class="mi">30</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">items</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="nx">pcoll</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">windowInto</span><span class="p">(</span><span class="nx">windowings</span><span class="p">.</span><span class="nx">slidingWindows</span><span class="p">(</span><span class="mi">30</span><span class="p">,</span> <span class="mi">5</span><span class="p">)))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">WindowInto</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">windowing</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">sliding</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">size</span><span class="p">:</span><span class="w"> </span><span class="l">5m</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">period</span><span class="p">:</span><span class="w"> </span><span class="l">30s</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="using-session-windows">8.3.3. Session windows</h4> |
| <p>The following example code shows how to apply <code>Window</code> to divide a <code>PCollection</code> |
| into session windows, where each session must be separated by a time gap of at |
| least 10 minutes (600 seconds):</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">items</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">sessionWindowedItems</span> <span class="o">=</span> <span class="n">items</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Window</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">into</span><span class="o">(</span><span class="n">Sessions</span><span class="o">.</span><span class="na">withGapDuration</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="n">600</span><span class="o">))));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam</span> <span class="kn">import</span> <span class="n">window</span> |
| </span></span><span class="line"><span class="cl"><span class="n">session_windowed_items</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">items</span> <span class="o">|</span> <span class="s1">&#39;window&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span><span class="n">window</span><span class="o">.</span><span class="n">Sessions</span><span class="p">(</span><span class="mi">10</span> <span class="o">*</span> <span class="mi">60</span><span class="p">)))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">sessionWindowedItems</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">WindowInto</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">window</span><span class="p">.</span><span class="nf">NewSessions</span><span class="p">(</span><span class="mi">600</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">items</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="nx">pcoll</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">windowInto</span><span class="p">(</span><span class="nx">windowings</span><span class="p">.</span><span class="nx">sessions</span><span class="p">(</span><span class="mi">10</span> <span class="o">*</span> <span class="mi">60</span><span class="p">)))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">WindowInto</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">windowing</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">sessions</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">gap</span><span class="p">:</span><span class="w"> </span><span class="l">60s</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Note that the sessions are per-key — each key in the collection will have its |
| own session groupings depending on the data distribution.</p> |
| <h4 id="using-single-global-window">8.3.4. Single global window</h4> |
| <p>If your <code>PCollection</code> is bounded (the size is fixed), you can assign all the |
| elements to a single global window. The following example code shows how to set |
| a single global window for a <code>PCollection</code>:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">items</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">batchItems</span> <span class="o">=</span> <span class="n">items</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Window</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">into</span><span class="o">(</span><span class="k">new</span> <span class="n">GlobalWindows</span><span class="o">()));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam</span> <span class="kn">import</span> <span class="n">window</span> |
| </span></span><span class="line"><span class="cl"><span class="n">global_windowed_items</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">items</span> <span class="o">|</span> <span class="s1">&#39;window&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span><span class="n">window</span><span class="o">.</span><span class="n">GlobalWindows</span><span class="p">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">globalWindowedItems</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">WindowInto</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">window</span><span class="p">.</span><span class="nf">NewGlobalWindows</span><span class="p">(),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">items</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="nx">pcoll</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">windowInto</span><span class="p">(</span><span class="nx">windowings</span><span class="p">.</span><span class="nx">globalWindows</span><span class="p">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">WindowInto</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">windowing</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">global</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="watermarks-and-late-data">8.4. Watermarks and late data</h3> |
| <p>In any data processing system, there is a certain amount of lag between the time |
| a data event occurs (the &ldquo;event time&rdquo;, determined by the timestamp on the data |
| element itself) and the time the actual data element gets processed at any stage |
| in your pipeline (the &ldquo;processing time&rdquo;, determined by the clock on the system |
| processing the element). In addition, there are no guarantees that data events |
| will appear in your pipeline in the same order that they were generated.</p> |
| <p>For example, let&rsquo;s say we have a <code>PCollection</code> that&rsquo;s using fixed-time |
| windowing, with windows that are five minutes long. For each window, Beam must |
| collect all the data with an <em>event time</em> timestamp in the given window range |
| (between 0:00 and 4:59 in the first window, for instance). Data with timestamps |
| outside that range (data from 5:00 or later) belong to a different window.</p> |
| <p>However, data isn&rsquo;t always guaranteed to arrive in a pipeline in time order, or |
| to always arrive at predictable intervals. Beam tracks a <em>watermark</em>, which is |
| the system&rsquo;s notion of when all data in a certain window can be expected to have |
| arrived in the pipeline. Once the watermark progresses past the end of a window, |
| any further element that arrives with a timestamp in that window is considered |
| <strong>late data</strong>.</p> |
| <p>From our example, suppose we have a simple watermark that assumes approximately |
| 30s of lag time between the data timestamps (the event time) and the time the |
| data appears in the pipeline (the processing time), then Beam would close the |
| first window at 5:30. If a data record arrives at 5:34, but with a timestamp |
| that would put it in the 0:00-4:59 window (say, 3:38), then that record is late |
| data.</p> |
| <p>Note: For simplicity, we&rsquo;ve assumed that we&rsquo;re using a very straightforward |
| watermark that estimates the lag time. In practice, your <code>PCollection</code>&rsquo;s data |
| source determines the watermark, and watermarks can be more precise or complex.</p> |
| <p>Beam&rsquo;s default windowing configuration tries to determine when all data has |
| arrived (based on the type of data source) and then advances the watermark past |
| the end of the window. This default configuration does <em>not</em> allow late data. |
| <a href="#triggers">Triggers</a> allow you to modify and refine the windowing strategy for |
| a <code>PCollection</code>. You can use triggers to decide when each individual window |
| aggregates and reports its results, including how the window emits late |
| elements.</p> |
| <h4 id="managing-late-data">8.4.1. Managing late data</h4> |
| <p>You can allow late data by invoking the <code>.withAllowedLateness</code> operation when |
| you set your <code>PCollection</code>&rsquo;s windowing strategy. The following code example |
| demonstrates a windowing strategy that will allow late data up to two days after |
| the end of a window.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">items</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">fixedWindowedItems</span> <span class="o">=</span> <span class="n">items</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Window</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">into</span><span class="o">(</span><span class="n">FixedWindows</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="n">1</span><span class="o">)))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withAllowedLateness</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardDays</span><span class="o">(</span><span class="n">2</span><span class="o">)));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="n">pc</span> <span class="o">=</span> <span class="p">[</span><span class="n">Initial</span> <span class="n">PCollection</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">FixedWindows</span><span class="p">(</span><span class="mi">60</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">trigger</span><span class="o">=</span><span class="n">trigger_fn</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">accumulation_mode</span><span class="o">=</span><span class="n">accumulation_mode</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timestamp_combiner</span><span class="o">=</span><span class="n">timestamp_combiner</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">allowed_lateness</span><span class="o">=</span><span class="n">Duration</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="mi">2</span><span class="o">*</span><span class="mi">24</span><span class="o">*</span><span class="mi">60</span><span class="o">*</span><span class="mi">60</span><span class="p">))</span> <span class="c1"># 2 days</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">windowedItems</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">WindowInto</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">window</span><span class="p">.</span><span class="nf">NewFixedWindows</span><span class="p">(</span><span class="mi">1</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Minute</span><span class="p">),</span> <span class="nx">items</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nf">AllowedLateness</span><span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="mi">24</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Hour</span><span class="p">),</span> <span class="c1">// 2 days |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>When you set <code>.withAllowedLateness</code> on a <code>PCollection</code>, that allowed lateness |
| propagates forward to any subsequent <code>PCollection</code> derived from the first |
| <code>PCollection</code> you applied allowed lateness to. If you want to change the allowed |
| lateness later in your pipeline, you must do so explicitly by applying |
| <code>Window.configure().withAllowedLateness()</code>.</p> |
| <h3 id="adding-timestamps-to-a-pcollections-elements">8.5. Adding timestamps to a PCollection&rsquo;s elements</h3> |
| <p>An unbounded source provides a timestamp for each element. Depending on your |
| unbounded source, you may need to configure how the timestamp is extracted from |
| the raw data stream.</p> |
| <p>However, bounded sources (such as a file from <code>TextIO</code>) do not provide |
| timestamps. If you need timestamps, you must add them to your <code>PCollection</code>’s |
| elements.</p> |
| <p>You can assign new timestamps to the elements of a <code>PCollection</code> by applying a |
| <a href="#pardo">ParDo</a> transform that outputs new elements with timestamps that you |
| set.</p> |
| <p>An example might be if your pipeline reads log records from an input file, and |
| each log record includes a timestamp field; since your pipeline reads the |
| records in from a file, the file source doesn&rsquo;t assign timestamps automatically. |
| You can parse the timestamp field from each record and use a <code>ParDo</code> transform |
| with a <code>DoFn</code> to attach the timestamps to each element in your <code>PCollection</code>.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">LogEntry</span><span class="o">&gt;</span> <span class="n">unstampedLogs</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">LogEntry</span><span class="o">&gt;</span> <span class="n">stampedLogs</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">unstampedLogs</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">LogEntry</span><span class="o">,</span> <span class="n">LogEntry</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="nd">@Element</span> <span class="n">LogEntry</span> <span class="n">element</span><span class="o">,</span> <span class="n">OutputReceiver</span><span class="o">&lt;</span><span class="n">LogEntry</span><span class="o">&gt;</span> <span class="n">out</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Extract the timestamp from log entry we&#39;re currently processing. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">Instant</span> <span class="n">logTimeStamp</span> <span class="o">=</span> <span class="n">extractTimeStampFromLogEntry</span><span class="o">(</span><span class="n">element</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Use OutputReceiver.outputWithTimestamp (rather than |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// OutputReceiver.output) to emit the entry with timestamp attached. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">out</span><span class="o">.</span><span class="na">outputWithTimestamp</span><span class="o">(</span><span class="n">element</span><span class="o">,</span> <span class="n">logTimeStamp</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">AddTimestampDoFn</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Extract the numeric Unix seconds-since-epoch timestamp to be</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># associated with the current log entry.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">unix_timestamp</span> <span class="o">=</span> <span class="n">extract_timestamp_from_log_entry</span><span class="p">(</span><span class="n">element</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Wrap and emit the current entry and new timestamp in a</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># TimestampedValue.</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">beam</span><span class="o">.</span><span class="n">window</span><span class="o">.</span><span class="n">TimestampedValue</span><span class="p">(</span><span class="n">element</span><span class="p">,</span> <span class="n">unix_timestamp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">timestamped_items</span> <span class="o">=</span> <span class="n">items</span> <span class="o">|</span> <span class="s1">&#39;timestamp&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">AddTimestampDoFn</span><span class="p">())</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// AddTimestampDoFn extracts an event time from a LogEntry. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="nf">AddTimestampDoFn</span><span class="p">(</span><span class="nx">element</span> <span class="nx">LogEntry</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span><span class="p">,</span> <span class="nx">LogEntry</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">et</span> <span class="o">:=</span> <span class="nf">extractEventTime</span><span class="p">(</span><span class="nx">element</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Defining an emitter with beam.EventTime as the first parameter |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// allows the DoFn to set the event time for the emitted element. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nf">emit</span><span class="p">(</span><span class="nx">mtime</span><span class="p">.</span><span class="nf">FromTime</span><span class="p">(</span><span class="nx">et</span><span class="p">),</span> <span class="nx">element</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Use the DoFn with ParDo as normal. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"><span class="nx">stampedLogs</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">AddTimestampDoFn</span><span class="p">,</span> <span class="nx">unstampedLogs</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-yaml snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">AssignTimestamps</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">language</span><span class="p">:</span><span class="w"> </span><span class="l">python</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">timestamp</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">callable</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd"> |
| </span></span></span><span class="line"><span class="cl"><span class="sd"> import datetime |
| </span></span></span><span class="line"><span class="cl"><span class="sd"> |
| </span></span></span><span class="line"><span class="cl"><span class="sd"> def extract_timestamp(x): |
| </span></span></span><span class="line"><span class="cl"><span class="sd"> raw = datetime.datetime.strptime( |
| </span></span></span><span class="line"><span class="cl"><span class="sd"> x.external_timestamp_field, &#34;%Y-%m-%d&#34;) |
| </span></span></span><span class="line"><span class="cl"><span class="sd"> return raw.astimezone(datetime.timezone.utc)</span><span class="w"> </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h2 id="triggers">9. Triggers</h2> |
| <span class="language-go"> |
| <blockquote> |
| <p><strong>Note:</strong> The Trigger API in the Beam SDK for Go is currently experimental and subject to change.</p> |
| </blockquote> |
| </span> |
| <p>When collecting and grouping data into windows, Beam uses <strong>triggers</strong> to |
| determine when to emit the aggregated results of each window (referred to as a |
| <em>pane</em>). If you use Beam&rsquo;s default windowing configuration and <a href="#default-trigger">default |
| trigger</a>, Beam outputs the aggregated result when it |
| <a href="#watermarks-and-late-data">estimates all data has arrived</a>, and discards all |
| subsequent data for that window.</p> |
| <p>You can set triggers for your <code>PCollection</code>s to change this default behavior. |
| Beam provides a number of pre-built triggers that you can set:</p> |
| <ul> |
| <li><strong>Event time triggers</strong>. These triggers operate on the event time, as |
| indicated by the timestamp on each data element. Beam&rsquo;s default trigger is |
| event time-based.</li> |
| <li><strong>Processing time triggers</strong>. These triggers operate on the processing time |
| &ndash; the time when the data element is processed at any given stage in the |
| pipeline.</li> |
| <li><strong>Data-driven triggers</strong>. These triggers operate by examining the data as it |
| arrives in each window, and firing when that data meets a certain property. |
| Currently, data-driven triggers only support firing after a certain number |
| of data elements.</li> |
| <li><strong>Composite triggers</strong>. These triggers combine multiple triggers in various |
| ways.</li> |
| </ul> |
| <p>At a high level, triggers provide two additional capabilities compared to simply |
| outputting at the end of a window:</p> |
| <ul> |
| <li>Triggers allow Beam to emit early results, before all the data in a given |
| window has arrived. For example, emitting after a certain amount of time |
| elapses, or after a certain number of elements arrives.</li> |
| <li>Triggers allow processing of late data by triggering after the event time |
| watermark passes the end of the window.</li> |
| </ul> |
| <p>These capabilities allow you to control the flow of your data and balance |
| between different factors depending on your use case:</p> |
| <ul> |
| <li><strong>Completeness:</strong> How important is it to have all of your data before you |
| compute your result?</li> |
| <li><strong>Latency:</strong> How long do you want to wait for data? For example, do you wait |
| until you think you have all data? Do you process data as it arrives?</li> |
| <li><strong>Cost:</strong> How much compute power/money are you willing to spend to lower the |
| latency?</li> |
| </ul> |
| <p>For example, a system that requires time-sensitive updates might use a strict |
| time-based trigger that emits a window every <em>N</em> seconds, valuing promptness |
| over data completeness. A system that values data completeness more than the |
| exact timing of results might choose to use Beam&rsquo;s default trigger, which fires |
| at the end of the window.</p> |
| <p>You can also set a trigger for an unbounded <code>PCollection</code> that uses a <a href="#windowing">single |
| global window for its windowing function</a>. This can be useful when |
| you want your pipeline to provide periodic updates on an unbounded data set — |
| for example, a running average of all data provided to the present time, updated |
| every N seconds or every N elements.</p> |
| <h3 id="event-time-triggers">9.1. Event time triggers</h3> |
| <p>The <code>AfterWatermark</code> trigger operates on <em>event time</em>. The <code>AfterWatermark</code> |
| trigger emits the contents of a window after the |
| <a href="#watermarks-and-late-data">watermark</a> passes the end of the window, based on the |
| timestamps attached to the data elements. The watermark is a global progress |
| metric, and is Beam&rsquo;s notion of input completeness within your pipeline at any |
| given point. <span class="language-java"><code>AfterWatermark.pastEndOfWindow()</code></span> |
| <span class="language-py"><code>AfterWatermark</code></span> |
| <span class="language-go"><code>trigger.AfterEndOfWindow</code></span> <em>only</em> fires when the |
| watermark passes the end of the window.</p> |
| <p>In addition, you can configure triggers that fire if your pipeline receives data |
| before or after the end of the window.</p> |
| <p>The following example shows a billing scenario, and uses both early and late |
| firings:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="c1">// Create a bill at the end of the month. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">AfterWatermark</span><span class="o">.</span><span class="na">pastEndOfWindow</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// During the month, get near real-time estimates. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">withEarlyFirings</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">AfterProcessingTime</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">pastFirstElementInPane</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">plusDuration</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="n">1</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Fire on any late data so the bill can be corrected. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">withLateFirings</span><span class="o">(</span><span class="n">AfterPane</span><span class="o">.</span><span class="na">elementCountAtLeast</span><span class="o">(</span><span class="n">1</span><span class="o">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">AfterWatermark</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">early</span><span class="o">=</span><span class="n">AfterProcessingTime</span><span class="p">(</span><span class="n">delay</span><span class="o">=</span><span class="mi">1</span> <span class="o">*</span> <span class="mi">60</span><span class="p">),</span> <span class="n">late</span><span class="o">=</span><span class="n">AfterCount</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">trigger</span> <span class="o">:=</span> <span class="nx">trigger</span><span class="p">.</span><span class="nf">AfterEndOfWindow</span><span class="p">().</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">EarlyFiring</span><span class="p">(</span><span class="nx">trigger</span><span class="p">.</span><span class="nf">AfterProcessingTime</span><span class="p">().</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">PlusDelay</span><span class="p">(</span><span class="mi">60</span> <span class="o">*</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">)).</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">LateFiring</span><span class="p">(</span><span class="nx">trigger</span><span class="p">.</span><span class="nf">Repeat</span><span class="p">(</span><span class="nx">trigger</span><span class="p">.</span><span class="nf">AfterCount</span><span class="p">(</span><span class="mi">1</span><span class="p">)))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="default-trigger">9.1.1. Default trigger</h4> |
| <p>The default trigger for a <code>PCollection</code> is based on event time, and emits the |
| results of the window when the Beam&rsquo;s watermark passes the end of the window, |
| and then fires each time late data arrives.</p> |
| <p>However, if you are using both the default windowing configuration and the |
| default trigger, the default trigger emits exactly once, and late data is |
| discarded. This is because the default windowing configuration has an allowed |
| lateness value of 0. See the Handling Late Data section for information about |
| modifying this behavior.</p> |
| <h3 id="processing-time-triggers">9.2. Processing time triggers</h3> |
| <p>The <code>AfterProcessingTime</code> trigger operates on <em>processing time</em>. For example, |
| the <span class="language-java"><code>AfterProcessingTime.pastFirstElementInPane()</code></span> |
| <span class="language-py"><code>AfterProcessingTime</code></span> |
| <span class="language-go"><code>trigger.AfterProcessingTime()</code></span> trigger emits a window |
| after a certain amount of processing time has passed since data was received. |
| The processing time is determined by the system clock, rather than the data |
| element&rsquo;s timestamp.</p> |
| <p>The <code>AfterProcessingTime</code> trigger is useful for triggering early results from a |
| window, particularly a window with a large time frame such as a single global |
| window.</p> |
| <h3 id="data-driven-triggers">9.3. Data-driven triggers</h3> |
| <p>Beam provides one data-driven trigger, |
| <span class="language-java"><code>AfterPane.elementCountAtLeast()</code></span> |
| <span class="language-py"><code>AfterCount</code></span> |
| <span class="language-go"><code>trigger.AfterCount()</code></span>. This trigger works on an element |
| count; it fires after the current pane has collected at least <em>N</em> elements. This |
| allows a window to emit early results (before all the data has accumulated), |
| which can be particularly useful if you are using a single global window.</p> |
| <p>It is important to note that if, for example, you specify |
| <span class="language-java"><code>.elementCountAtLeast(50)</code></span> |
| <span class="language-py">AfterCount(50)</span> |
| <span class="language-go"><code>trigger.AfterCount(50)</code></span> and only 32 elements arrive, |
| those 32 elements sit around forever. If the 32 elements are important to you, |
| consider using <a href="#composite-triggers">composite triggers</a> to combine multiple |
| conditions. This allows you to specify multiple firing conditions such as &ldquo;fire |
| either when I receive 50 elements, or every 1 second&rdquo;.</p> |
| <h3 id="setting-a-trigger">9.4. Setting a trigger</h3> |
| <p>When you set a windowing function for a <code>PCollection</code> by using the |
| <span class="language-java"><code>Window</code></span><span class="language-py"><code>WindowInto</code></span><span class="language-go"><code>beam.WindowInto</code></span> |
| transform, you can also specify a trigger.</p> |
| <p class="language-java">You set the trigger(s) for a <code>PCollection</code> by invoking the method |
| <code>.triggering()</code> on the result of your <code>Window.into()</code> transform. This code |
| sample sets a time-based trigger for a <code>PCollection</code>, which emits results one |
| minute after the first element in that window has been processed. The last line |
| in the code sample, <code>.discardingFiredPanes()</code>, sets the window&rsquo;s <strong>accumulation |
| mode</strong>.</p> |
| <p class="language-py">You set the trigger(s) for a <code>PCollection</code> by setting the <code>trigger</code> parameter |
| when you use the <code>WindowInto</code> transform. This code sample sets a time-based |
| trigger for a <code>PCollection</code>, which emits results one minute after the first |
| element in that window has been processed. The <code>accumulation_mode</code> parameter |
| sets the window&rsquo;s <strong>accumulation mode</strong>.</p> |
| <p class="language-go">You set the trigger(s) for a <code>PCollection</code> by passing in the <code>beam.Trigger</code> parameter |
| when you use the <code>beam.WindowInto</code> transform. This code sample sets a time-based |
| trigger for a <code>PCollection</code>, which emits results one minute after the first |
| element in that window has been processed. |
| The <code>beam.AccumulationMode</code> parameter sets the window&rsquo;s <strong>accumulation mode</strong>.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">pc</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pc</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Window</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">into</span><span class="o">(</span><span class="n">FixedWindows</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">1</span><span class="o">,</span> <span class="n">TimeUnit</span><span class="o">.</span><span class="na">MINUTES</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">triggering</span><span class="o">(</span><span class="n">AfterProcessingTime</span><span class="o">.</span><span class="na">pastFirstElementInPane</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">plusDelayOf</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="n">1</span><span class="o">)))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">discardingFiredPanes</span><span class="o">());</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="n">pcollection</span> <span class="o">|</span> <span class="n">WindowInto</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">FixedWindows</span><span class="p">(</span><span class="mi">1</span> <span class="o">*</span> <span class="mi">60</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">trigger</span><span class="o">=</span><span class="n">AfterProcessingTime</span><span class="p">(</span><span class="mi">1</span> <span class="o">*</span> <span class="mi">60</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">accumulation_mode</span><span class="o">=</span><span class="n">AccumulationMode</span><span class="o">.</span><span class="n">DISCARDING</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">windowedItems</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">WindowInto</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">window</span><span class="p">.</span><span class="nf">NewFixedWindows</span><span class="p">(</span><span class="mi">1</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Minute</span><span class="p">),</span> <span class="nx">pcollection</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nf">Trigger</span><span class="p">(</span><span class="nx">trigger</span><span class="p">.</span><span class="nf">AfterProcessingTime</span><span class="p">().</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">PlusDelay</span><span class="p">(</span><span class="mi">1</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Minute</span><span class="p">)),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nf">AllowedLateness</span><span class="p">(</span><span class="mi">30</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Minute</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nf">PanesDiscard</span><span class="p">(),</span> |
| </span></span><span class="line"><span class="cl"><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="window-accumulation-modes">9.4.1. Window accumulation modes</h4> |
| <p>When you specify a trigger, you must also set the window&rsquo;s <strong>accumulation |
| mode</strong>. When a trigger fires, it emits the current contents of the window as a |
| pane. Since a trigger can fire multiple times, the accumulation mode determines |
| whether the system <em>accumulates</em> the window panes as the trigger fires, or |
| <em>discards</em> them.</p> |
| <p class="language-java">To set a window to accumulate the panes that are produced when the trigger |
| fires, invoke<code>.accumulatingFiredPanes()</code> when you set the trigger. To set a |
| window to discard fired panes, invoke <code>.discardingFiredPanes()</code>.</p> |
| <p class="language-py">To set a window to accumulate the panes that are produced when the trigger |
| fires, set the <code>accumulation_mode</code> parameter to <code>ACCUMULATING</code> when you set the |
| trigger. To set a window to discard fired panes, set <code>accumulation_mode</code> to |
| <code>DISCARDING</code>.</p> |
| <p class="language-go">To set a window to accumulate the panes that are produced when the trigger |
| fires, set the <code>beam.AccumulationMode</code> parameter to <code>beam.PanesAccumulate()</code> when you set the |
| trigger. To set a window to discard fired panes, set <code>beam.AccumulationMode</code> to |
| <code>beam.PanesDiscard()</code>.</p> |
| <p>Let&rsquo;s look an example that uses a <code>PCollection</code> with fixed-time windowing and a |
| data-based trigger. This is something you might do if, for example, each window |
| represented a ten-minute running average, but you wanted to display the current |
| value of the average in a UI more frequently than every ten minutes. We&rsquo;ll |
| assume the following conditions:</p> |
| <ul> |
| <li>The <code>PCollection</code> uses 10-minute fixed-time windows.</li> |
| <li>The <code>PCollection</code> has a repeating trigger that fires every time 3 elements |
| arrive.</li> |
| </ul> |
| <p>The following diagram shows data events for key X as they arrive in the |
| PCollection and are assigned to windows. To keep the diagram a bit simpler, |
| we&rsquo;ll assume that the events all arrive in the pipeline in order.</p> |
| <p><img src="/images/trigger-accumulation.png" alt="Diagram of data events for accumulating mode example"></p> |
| <h5 id="accumulating-mode">9.4.1.1. Accumulating mode</h5> |
| <p>If our trigger is set to accumulating mode, the trigger emits the following |
| values each time it fires. Keep in mind that the trigger fires every time three |
| elements arrive:</p> |
| <pre tabindex="0"><code> First trigger firing: [5, 8, 3] |
| Second trigger firing: [5, 8, 3, 15, 19, 23] |
| Third trigger firing: [5, 8, 3, 15, 19, 23, 9, 13, 10] |
| </code></pre><h5 id="discarding-mode">9.4.1.2. Discarding mode</h5> |
| <p>If our trigger is set to discarding mode, the trigger emits the following values |
| on each firing:</p> |
| <pre tabindex="0"><code> First trigger firing: [5, 8, 3] |
| Second trigger firing: [15, 19, 23] |
| Third trigger firing: [9, 13, 10] |
| </code></pre><h4 id="handling-late-data">9.4.2. Handling late data</h4> |
| <p>If you want your pipeline to process data that arrives after the watermark |
| passes the end of the window, you can apply an <em>allowed lateness</em> when you set |
| your windowing configuration. This gives your trigger the opportunity to react |
| to the late data. If allowed lateness is set, the default trigger will emit new |
| results immediately whenever late data arrives.</p> |
| <p>You set the allowed lateness by using <span class="language-java"><code>.withAllowedLateness()</code></span> |
| <span class="language-py"><code>allowed_lateness</code></span> |
| <span class="language-go"><code>beam.AllowedLateness()</code></span> |
| when you set your windowing function:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">pc</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pc</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Window</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">into</span><span class="o">(</span><span class="n">FixedWindows</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">1</span><span class="o">,</span> <span class="n">TimeUnit</span><span class="o">.</span><span class="na">MINUTES</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">triggering</span><span class="o">(</span><span class="n">AfterProcessingTime</span><span class="o">.</span><span class="na">pastFirstElementInPane</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">plusDelayOf</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="n">1</span><span class="o">)))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withAllowedLateness</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="n">30</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="n">pc</span> <span class="o">=</span> <span class="p">[</span><span class="n">Initial</span> <span class="n">PCollection</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pc</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">FixedWindows</span><span class="p">(</span><span class="mi">60</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">trigger</span><span class="o">=</span><span class="n">AfterProcessingTime</span><span class="p">(</span><span class="mi">60</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">allowed_lateness</span><span class="o">=</span><span class="mi">1800</span><span class="p">)</span> <span class="c1"># 30 minutes</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="o">...</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">allowedToBeLateItems</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">WindowInto</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">window</span><span class="p">.</span><span class="nf">NewFixedWindows</span><span class="p">(</span><span class="mi">1</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Minute</span><span class="p">),</span> <span class="nx">pcollection</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nf">Trigger</span><span class="p">(</span><span class="nx">trigger</span><span class="p">.</span><span class="nf">AfterProcessingTime</span><span class="p">().</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">PlusDelay</span><span class="p">(</span><span class="mi">1</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Minute</span><span class="p">)),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nf">AllowedLateness</span><span class="p">(</span><span class="mi">30</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Minute</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>This allowed lateness propagates to all <code>PCollection</code>s derived as a result of |
| applying transforms to the original <code>PCollection</code>. If you want to change the |
| allowed lateness later in your pipeline, you can apply |
| <span class="language-java"><code>Window.configure().withAllowedLateness()</code></span> |
| <span class="language-py"><code>allowed_lateness</code></span> |
| <span class="language-go"><code>beam.AllowedLateness()</code></span> |
| again, explicitly.</p> |
| <h3 id="composite-triggers">9.5. Composite triggers</h3> |
| <p>You can combine multiple triggers to form <strong>composite triggers</strong>, and can |
| specify a trigger to emit results repeatedly, at most once, or under other |
| custom conditions.</p> |
| <h4 id="composite-trigger-types">9.5.1. Composite trigger types</h4> |
| <p>Beam includes the following composite triggers:</p> |
| <ul> |
| <li>You can add additional early firings or late firings to |
| <code>AfterWatermark.pastEndOfWindow</code> via <code>.withEarlyFirings</code> and |
| <code>.withLateFirings</code>.</li> |
| <li><code>Repeatedly.forever</code> specifies a trigger that executes forever. Any time the |
| trigger&rsquo;s conditions are met, it causes a window to emit results and then |
| resets and starts over. It can be useful to combine <code>Repeatedly.forever</code> |
| with <code>.orFinally</code> to specify a condition that causes the repeating trigger |
| to stop.</li> |
| <li><code>AfterEach.inOrder</code> combines multiple triggers to fire in a specific |
| sequence. Each time a trigger in the sequence emits a window, the sequence |
| advances to the next trigger.</li> |
| <li><code>AfterFirst</code> takes multiple triggers and emits the first time <em>any</em> of its |
| argument triggers is satisfied. This is equivalent to a logical OR operation |
| for multiple triggers.</li> |
| <li><code>AfterAll</code> takes multiple triggers and emits when <em>all</em> of its argument |
| triggers are satisfied. This is equivalent to a logical AND operation for |
| multiple triggers.</li> |
| <li><code>orFinally</code> can serve as a final condition to cause any trigger to fire one |
| final time and never fire again.</li> |
| </ul> |
| <h4 id="composite-afterwatermark">9.5.2. Composition with AfterWatermark</h4> |
| <p>Some of the most useful composite triggers fire a single time when Beam |
| estimates that all the data has arrived (i.e. when the watermark passes the end |
| of the window) combined with either, or both, of the following:</p> |
| <ul> |
| <li> |
| <p>Speculative firings that precede the watermark passing the end of the window |
| to allow faster processing of partial results.</p> |
| </li> |
| <li> |
| <p>Late firings that happen after the watermark passes the end of the window, |
| to allow for handling late-arriving data</p> |
| </li> |
| </ul> |
| <p>You can express this pattern using <code>AfterWatermark</code>. For example, the following |
| example trigger code fires on the following conditions:</p> |
| <ul> |
| <li> |
| <p>On Beam&rsquo;s estimate that all the data has arrived (the watermark passes the |
| end of the window)</p> |
| </li> |
| <li> |
| <p>Any time late data arrives, after a ten-minute delay</p> |
| </li> |
| </ul> |
| <p class="language-java"><ul> |
| <li>After two days, we assume no more data of interest will arrive, and the |
| trigger stops executing</li> |
| </ul> |
| </p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Window</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">configure</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">triggering</span><span class="o">(</span><span class="n">AfterWatermark</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">pastEndOfWindow</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withLateFirings</span><span class="o">(</span><span class="n">AfterProcessingTime</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">pastFirstElementInPane</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">plusDelayOf</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="n">10</span><span class="o">))))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withAllowedLateness</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardDays</span><span class="o">(</span><span class="n">2</span><span class="o">)));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">pcollection</span> <span class="o">|</span> <span class="n">WindowInto</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">FixedWindows</span><span class="p">(</span><span class="mi">1</span> <span class="o">*</span> <span class="mi">60</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">trigger</span><span class="o">=</span><span class="n">AfterWatermark</span><span class="p">(</span><span class="n">late</span><span class="o">=</span><span class="n">AfterProcessingTime</span><span class="p">(</span><span class="mi">10</span> <span class="o">*</span> <span class="mi">60</span><span class="p">)),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">allowed_lateness</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">accumulation_mode</span><span class="o">=</span><span class="n">AccumulationMode</span><span class="o">.</span><span class="n">DISCARDING</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">compositeTriggerItems</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">WindowInto</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">window</span><span class="p">.</span><span class="nf">NewFixedWindows</span><span class="p">(</span><span class="mi">1</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Minute</span><span class="p">),</span> <span class="nx">pcollection</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nf">Trigger</span><span class="p">(</span><span class="nx">trigger</span><span class="p">.</span><span class="nf">AfterEndOfWindow</span><span class="p">().</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">LateFiring</span><span class="p">(</span><span class="nx">trigger</span><span class="p">.</span><span class="nf">AfterProcessingTime</span><span class="p">().</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">PlusDelay</span><span class="p">(</span><span class="mi">10</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Minute</span><span class="p">))),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nf">AllowedLateness</span><span class="p">(</span><span class="mi">2</span><span class="o">*</span><span class="mi">24</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Hour</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="other-composite-triggers">9.5.3. Other composite triggers</h4> |
| <p>You can also build other sorts of composite triggers. The following example code |
| shows a simple composite trigger that fires whenever the pane has at least 100 |
| elements, or after a minute.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="n">Repeatedly</span><span class="o">.</span><span class="na">forever</span><span class="o">(</span><span class="n">AfterFirst</span><span class="o">.</span><span class="na">of</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">AfterPane</span><span class="o">.</span><span class="na">elementCountAtLeast</span><span class="o">(</span><span class="n">100</span><span class="o">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">AfterProcessingTime</span><span class="o">.</span><span class="na">pastFirstElementInPane</span><span class="o">().</span><span class="na">plusDelayOf</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="n">1</span><span class="o">))))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">pcollection</span> <span class="o">|</span> <span class="n">WindowInto</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">FixedWindows</span><span class="p">(</span><span class="mi">1</span> <span class="o">*</span> <span class="mi">60</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">trigger</span><span class="o">=</span><span class="n">Repeatedly</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">AfterAny</span><span class="p">(</span><span class="n">AfterCount</span><span class="p">(</span><span class="mi">100</span><span class="p">),</span> <span class="n">AfterProcessingTime</span><span class="p">(</span><span class="mi">1</span> <span class="o">*</span> <span class="mi">60</span><span class="p">))),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">accumulation_mode</span><span class="o">=</span><span class="n">AccumulationMode</span><span class="o">.</span><span class="n">DISCARDING</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h2 id="metrics">10. Metrics</h2> |
| <p>In the Beam model, metrics provide some insight into the current state of a user pipeline, |
| potentially while the pipeline is running. There could be different reasons for that, for instance:</p> |
| <ul> |
| <li>Check the number of errors encountered while running a specific step in the pipeline;</li> |
| <li>Monitor the number of RPCs made to backend service;</li> |
| <li>Retrieve an accurate count of the number of elements that have been processed;</li> |
| <li>&hellip;and so on.</li> |
| </ul> |
| <h3 id="101-the-main-concepts-of-beam-metrics">10.1. The main concepts of Beam metrics</h3> |
| <ul> |
| <li><strong>Named</strong>. Each metric has a name which consists of a namespace and an actual name. The |
| namespace can be used to differentiate between multiple metrics with the same name and also |
| allows querying for all metrics within a specific namespace.</li> |
| <li><strong>Scoped</strong>. Each metric is reported against a specific step in the pipeline, indicating what |
| code was running when the metric was incremented.</li> |
| <li><strong>Dynamically Created</strong>. Metrics may be created during runtime without pre-declaring them, in |
| much the same way a logger could be created. This makes it easier to produce metrics in utility |
| code and have them usefully reported.</li> |
| <li><strong>Degrade Gracefully</strong>. If a runner doesn’t support some part of reporting metrics, the |
| fallback behavior is to drop the metric updates rather than failing the pipeline. If a runner |
| doesn’t support some part of querying metrics, the runner will not return the associated data.</li> |
| </ul> |
| <p>Reported metrics are implicitly scoped to the transform within the pipeline that reported them. |
| This allows reporting the same metric name in multiple places and identifying the value each |
| transform reported, as well as aggregating the metric across the entire pipeline.</p> |
| <blockquote> |
| <p><strong>Note:</strong> It is runner-dependent whether metrics are accessible during pipeline execution or only |
| after jobs have completed.</p> |
| </blockquote> |
| <h3 id="types-of-metrics">10.2. Types of metrics</h3> |
| <p>There are three types of metrics that are supported for the moment: <code>Counter</code>, <code>Distribution</code> and |
| <code>Gauge</code>.</p> |
| <p class="language-go">In the Beam SDK for Go, a <code>context.Context</code> provided by the framework must be passed to the metric |
| or the metric value will not be recorded. The framework will automatically provide a valid |
| <code>context.Context</code> to <code>ProcessElement</code> and similar methods when it&rsquo;s the first parameter.</p> |
| <p><strong>Counter</strong>: A metric that reports a single long value and can be incremented or decremented.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">Counter</span> <span class="n">counter</span> <span class="o">=</span> <span class="n">Metrics</span><span class="o">.</span><span class="na">counter</span><span class="o">(</span> <span class="s">&#34;namespace&#34;</span><span class="o">,</span> <span class="s">&#34;counter1&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// count the elements |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">counter</span><span class="o">.</span><span class="na">inc</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">var</span> <span class="nx">counter</span> <span class="p">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">NewCounter</span><span class="p">(</span><span class="s">&#34;namespace&#34;</span><span class="p">,</span> <span class="s">&#34;counter1&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">MyDoFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="o">...</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// count the elements |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">counter</span><span class="p">.</span><span class="nf">Inc</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p><strong>Distribution</strong>: A metric that reports information about the distribution of reported values.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">Distribution</span> <span class="n">distribution</span> <span class="o">=</span> <span class="n">Metrics</span><span class="o">.</span><span class="na">distribution</span><span class="o">(</span> <span class="s">&#34;namespace&#34;</span><span class="o">,</span> <span class="s">&#34;distribution1&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Integer</span> <span class="n">element</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="na">element</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// create a distribution (histogram) of the values |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">distribution</span><span class="o">.</span><span class="na">update</span><span class="o">(</span><span class="n">element</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">var</span> <span class="nx">distribution</span> <span class="p">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">NewDistribution</span><span class="p">(</span><span class="s">&#34;namespace&#34;</span><span class="p">,</span> <span class="s">&#34;distribution1&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">MyDoFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">v</span> <span class="kt">int64</span><span class="p">,</span> <span class="o">...</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// create a distribution (histogram) of the values |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">distribution</span><span class="p">.</span><span class="nf">Update</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">v</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p><strong>Gauge</strong>: A metric that reports the latest value out of reported values. Since metrics are |
| collected from many workers the value may not be the absolute last, but one of the latest values.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">Gauge</span> <span class="n">gauge</span> <span class="o">=</span> <span class="n">Metrics</span><span class="o">.</span><span class="na">gauge</span><span class="o">(</span> <span class="s">&#34;namespace&#34;</span><span class="o">,</span> <span class="s">&#34;gauge1&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Integer</span> <span class="n">element</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="na">element</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// create a gauge (latest value received) of the values |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">gauge</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">element</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">var</span> <span class="nx">gauge</span> <span class="p">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">NewGauge</span><span class="p">(</span><span class="s">&#34;namespace&#34;</span><span class="p">,</span> <span class="s">&#34;gauge1&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">MyDoFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">v</span> <span class="kt">int64</span><span class="p">,</span> <span class="o">...</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// create a gauge (latest value received) of the values |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">gauge</span><span class="p">.</span><span class="nf">Set</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">v</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="querying-metrics">10.3. Querying metrics</h3> |
| <p class="language-java language-python"><code>PipelineResult</code> has a method <code>metrics()</code> which returns a <code>MetricResults</code> object that allows |
| accessing metrics. The main method available in <code>MetricResults</code> allows querying for all metrics |
| matching a given filter.</p> |
| <p class="language-go"><code>beam.PipelineResult</code> has a method <code>Metrics()</code> which returns a <code>metrics.Results</code> object that allows |
| accessing metrics. The main method available in <code>metrics.Results</code> allows querying for all metrics |
| matching a given filter. It takes in a predicate with a <code>SingleResult</code> parameter type, which can |
| be used for custom filters.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">PipelineResult</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MetricResults</span> <span class="nf">metrics</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">abstract</span> <span class="kd">class</span> <span class="nc">MetricResults</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kd">abstract</span> <span class="n">MetricQueryResults</span> <span class="nf">queryMetrics</span><span class="o">(</span><span class="nd">@Nullable</span> <span class="n">MetricsFilter</span> <span class="n">filter</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">MetricQueryResults</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Iterable</span><span class="o">&lt;</span><span class="n">MetricResult</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">&gt;&gt;</span> <span class="nf">getCounters</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Iterable</span><span class="o">&lt;</span><span class="n">MetricResult</span><span class="o">&lt;</span><span class="n">DistributionResult</span><span class="o">&gt;&gt;</span> <span class="nf">getDistributions</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Iterable</span><span class="o">&lt;</span><span class="n">MetricResult</span><span class="o">&lt;</span><span class="n">GaugeResult</span><span class="o">&gt;&gt;</span> <span class="nf">getGauges</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">MetricResult</span><span class="o">&lt;</span><span class="n">T</span><span class="o">&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MetricName</span> <span class="nf">getName</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">String</span> <span class="nf">getStep</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">T</span> <span class="nf">getCommitted</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">T</span> <span class="nf">getAttempted</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">queryMetrics</span><span class="p">(</span><span class="nx">pr</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PipelineResult</span><span class="p">,</span> <span class="nx">ns</span><span class="p">,</span> <span class="nx">n</span> <span class="kt">string</span><span class="p">)</span> <span class="nx">metrics</span><span class="p">.</span><span class="nx">QueryResults</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">pr</span><span class="p">.</span><span class="nf">Metrics</span><span class="p">().</span><span class="nf">Query</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">r</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">MetricResult</span><span class="p">)</span> <span class="kt">bool</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">r</span><span class="p">.</span><span class="nf">Namespace</span><span class="p">()</span> <span class="o">==</span> <span class="nx">ns</span> <span class="o">&amp;&amp;</span> <span class="nx">r</span><span class="p">.</span><span class="nf">Name</span><span class="p">()</span> <span class="o">==</span> <span class="nx">n</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">})</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="using-metrics">10.4. Using metrics in pipeline</h3> |
| <p>Below, there is a simple example of how to use a <code>Counter</code> metric in a user pipeline.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// creating a pipeline with custom metrics DoFn |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(...)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">MyMetricsDoFn</span><span class="o">()));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">pipelineResult</span> <span class="o">=</span> <span class="n">pipeline</span><span class="o">.</span><span class="na">run</span><span class="o">().</span><span class="na">waitUntilFinish</span><span class="o">(...);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// request the metric called &#34;counter1&#34; in namespace called &#34;namespace&#34; |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">MetricQueryResults</span> <span class="n">metrics</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pipelineResult</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">metrics</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">queryMetrics</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MetricsFilter</span><span class="o">.</span><span class="na">builder</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">addNameFilter</span><span class="o">(</span><span class="n">MetricNameFilter</span><span class="o">.</span><span class="na">named</span><span class="o">(</span><span class="s">&#34;namespace&#34;</span><span class="o">,</span> <span class="s">&#34;counter1&#34;</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">build</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// print the metric value - there should be only one line because there is only one metric |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// called &#34;counter1&#34; in the namespace called &#34;namespace&#34; |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="k">for</span> <span class="o">(</span><span class="n">MetricResult</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">&gt;</span> <span class="n">counter</span><span class="o">:</span> <span class="n">metrics</span><span class="o">.</span><span class="na">getCounters</span><span class="o">())</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">counter</span><span class="o">.</span><span class="na">getName</span><span class="o">()</span> <span class="o">+</span> <span class="s">&#34;:&#34;</span> <span class="o">+</span> <span class="n">counter</span><span class="o">.</span><span class="na">getAttempted</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyMetricsDoFn</span> <span class="kd">extends</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">private</span> <span class="kd">final</span> <span class="n">Counter</span> <span class="n">counter</span> <span class="o">=</span> <span class="n">Metrics</span><span class="o">.</span><span class="na">counter</span><span class="o">(</span> <span class="s">&#34;namespace&#34;</span><span class="o">,</span> <span class="s">&#34;counter1&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// count the elements |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">counter</span><span class="o">.</span><span class="na">inc</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">context</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">context</span><span class="o">.</span><span class="na">element</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">addMetricDoFnToPipeline</span><span class="p">(</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">input</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span><span class="p">)</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">MyMetricsDoFn</span><span class="p">{},</span> <span class="nx">input</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">executePipelineAndGetMetrics</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">p</span> <span class="o">*</span><span class="nx">beam</span><span class="p">.</span><span class="nx">Pipeline</span><span class="p">)</span> <span class="p">(</span><span class="nx">metrics</span><span class="p">.</span><span class="nx">QueryResults</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">pr</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">Run</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">runner</span><span class="p">,</span> <span class="nx">p</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">metrics</span><span class="p">.</span><span class="nx">QueryResults</span><span class="p">{},</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Request the metric called &#34;counter1&#34; in namespace called &#34;namespace&#34; |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">ms</span> <span class="o">:=</span> <span class="nx">pr</span><span class="p">.</span><span class="nf">Metrics</span><span class="p">().</span><span class="nf">Query</span><span class="p">(</span><span class="kd">func</span><span class="p">(</span><span class="nx">r</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">MetricResult</span><span class="p">)</span> <span class="kt">bool</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">r</span><span class="p">.</span><span class="nf">Namespace</span><span class="p">()</span> <span class="o">==</span> <span class="s">&#34;namespace&#34;</span> <span class="o">&amp;&amp;</span> <span class="nx">r</span><span class="p">.</span><span class="nf">Name</span><span class="p">()</span> <span class="o">==</span> <span class="s">&#34;counter1&#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">})</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Print the metric value - there should be only one line because there is |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// only one metric called &#34;counter1&#34; in the namespace called &#34;namespace&#34; |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">c</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">ms</span><span class="p">.</span><span class="nf">Counters</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fmt</span><span class="p">.</span><span class="nf">Println</span><span class="p">(</span><span class="nx">c</span><span class="p">.</span><span class="nf">Namespace</span><span class="p">(),</span> <span class="s">&#34;-&#34;</span><span class="p">,</span> <span class="nx">c</span><span class="p">.</span><span class="nf">Name</span><span class="p">(),</span> <span class="s">&#34;:&#34;</span><span class="p">,</span> <span class="nx">c</span><span class="p">.</span><span class="nx">Committed</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">ms</span><span class="p">,</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">MyMetricsDoFn</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">counter</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Counter</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nf">RegisterType</span><span class="p">(</span><span class="nx">reflect</span><span class="p">.</span><span class="nf">TypeOf</span><span class="p">((</span><span class="o">*</span><span class="nx">MyMetricsDoFn</span><span class="p">)(</span><span class="kc">nil</span><span class="p">)))</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">MyMetricsDoFn</span><span class="p">)</span> <span class="nf">Setup</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// While metrics can be defined in package scope or dynamically |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// it&#39;s most efficient to include them in the DoFn. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">fn</span><span class="p">.</span><span class="nx">counter</span> <span class="p">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">NewCounter</span><span class="p">(</span><span class="s">&#34;namespace&#34;</span><span class="p">,</span> <span class="s">&#34;counter1&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">MyMetricsDoFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">v</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">V</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">V</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// count the elements |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">fn</span><span class="p">.</span><span class="nx">counter</span><span class="p">.</span><span class="nf">Inc</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emit</span><span class="p">(</span><span class="nx">v</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="export-metrics">10.5. Export metrics</h3> |
| <p>Beam metrics can be exported to external sinks. If a metrics sink is set up in the configuration, the runner will push metrics to it at a default 5s period. |
| The configuration is held in the <a href="https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/metrics/MetricsOptions.html">MetricsOptions</a> class. |
| It contains push period configuration and also sink specific options such as type and URL. As for now only the REST HTTP and the Graphite sinks are supported and only |
| Flink and Spark runners support metrics export.</p> |
| <p>Also Beam metrics are exported to inner Spark and Flink dashboards to be consulted in their respective UI.</p> |
| <h2 id="state-and-timers">11. State and Timers</h2> |
| <p>Beam&rsquo;s windowing and triggering facilities provide a powerful abstraction for grouping and aggregating unbounded input |
| data based on timestamps. However there are aggregation use cases for which developers may require a higher degree of |
| control than provided by windows and triggers. Beam provides an API for manually managing per-key state, allowing for |
| fine-grained control over aggregations.</p> |
| <p>Beam&rsquo;s state API models state per key. To use the state API, you start out with a keyed <code>PCollection</code>, which in Java |
| is modeled as a <code>PCollection&lt;KV&lt;K, V&gt;&gt;</code>. A <code>ParDo</code> processing this <code>PCollection</code> can now declare state variables. Inside |
| the <code>ParDo</code> these state variables can be used to write or update state for the current key or to read previous state |
| written for that key. State is always fully scoped only to the current processing key.</p> |
| <p>Windowing can still be used together with stateful processing. All state for a key is scoped to the current window. This |
| means that the first time a key is seen for a given window any state reads will return empty, and that a runner can |
| garbage collect state when a window is completed. It&rsquo;s also often useful to use Beam&rsquo;s windowed aggregations prior to |
| the stateful operator. For example, using a combiner to preaggregate data, and then storing aggregated data inside of |
| state. Merging windows are not currently supported when using state and timers.</p> |
| <p>Sometimes stateful processing is used to implement state-machine style processing inside a <code>DoFn</code>. When doing this, |
| care must be taken to remember that the elements in input PCollection have no guaranteed order and to ensure that the |
| program logic is resilient to this. Unit tests written using the DirectRunner will shuffle the order of element |
| processing, and are recommended to test for correctness.</p> |
| <p class="language-java">In Java, DoFn declares states to be accessed by creating final <code>StateSpec</code> member variables representing each state. Each |
| state must be named using the <code>StateId</code> annotation; this name is unique to a ParDo in the graph and has no relation |
| to other nodes in the graph. A <code>DoFn</code> can declare multiple state variables.</p> |
| <p class="language-py">In Python, DoFn declares states to be accessed by creating <code>StateSpec</code> class member variables representing each state. Each |
| <code>StateSpec</code> is initialized with a name, this name is unique to a ParDo in the graph and has no relation |
| to other nodes in the graph. A <code>DoFn</code> can declare multiple state variables.</p> |
| <p class="language-go">In Go, DoFn declares states to be accessed by creating state struct member variables representing each state. Each |
| state variable is initialized with a key, this key is unique to a ParDo in the graph and has no relation |
| to other nodes in the graph. If no name is supplied, the key defaults to the member variable&rsquo;s name. |
| A <code>DoFn</code> can declare multiple state variables.</p> |
| <span class="language-typescript"> |
| <blockquote> |
| <p><strong>Note:</strong> The Beam SDK for Typescript does not yet support a State and Timer API, |
| but it is possible to use these features from cross-language pipelines (see below).</p> |
| </blockquote> |
| </span> |
| <h3 id="types-of-state">11.1. Types of state</h3> |
| <p>Beam provides several types of state:</p> |
| <h4 id="valuestate">ValueState</h4> |
| <p>A ValueState is a scalar state value. For each key in the input, a ValueState will store a typed value that can be |
| read and modified inside the DoFn&rsquo;s <code>@ProcessElement</code> or <code>@OnTimer</code> methods. If the type of the ValueState has a coder |
| registered, then Beam will automatically infer the coder for the state value. Otherwise, a coder can be explicitly |
| specified when creating the ValueState. For example, the following ParDo creates a single state variable that |
| accumulates the number of elements seen.</p> |
| <p>Note: <code>ValueState</code> is called <code>ReadModifyWriteState</code> in the Python SDK.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">perUser</span> <span class="o">=</span> <span class="n">readPerUser</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">perUser</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;,</span> <span class="n">OutputT</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">numElements</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span><span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">state</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Read the number element seen so far for this user key. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// state.read() returns null if it was never set. The below code allows us to have a default value of 0. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kt">int</span> <span class="n">currentValue</span> <span class="o">=</span> <span class="n">MoreObjects</span><span class="o">.</span><span class="na">firstNonNull</span><span class="o">(</span><span class="n">state</span><span class="o">.</span><span class="na">read</span><span class="o">(),</span> <span class="n">0</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Update the state. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">state</span><span class="o">.</span><span class="na">write</span><span class="o">(</span><span class="n">currentValue</span> <span class="o">+</span> <span class="n">1</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// valueStateFn keeps track of the number of elements seen. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">type</span> <span class="nx">valueStateFn</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Val</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Value</span><span class="p">[</span><span class="kt">int</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">s</span> <span class="o">*</span><span class="nx">valueStateFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">p</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">book</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">word</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emitWords</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="kt">error</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Get the value stored in our state |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">val</span><span class="p">,</span> <span class="nx">ok</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">s</span><span class="p">.</span><span class="nx">Val</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">p</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="p">!</span><span class="nx">ok</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">s</span><span class="p">.</span><span class="nx">Val</span><span class="p">.</span><span class="nf">Write</span><span class="p">(</span><span class="nx">p</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">s</span><span class="p">.</span><span class="nx">Val</span><span class="p">.</span><span class="nf">Write</span><span class="p">(</span><span class="nx">p</span><span class="p">,</span> <span class="nx">val</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">val</span> <span class="p">&gt;</span> <span class="mi">10000</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Example of clearing and starting again with an empty bag |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">s</span><span class="p">.</span><span class="nx">Val</span><span class="p">.</span><span class="nf">Clear</span><span class="p">(</span><span class="nx">p</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Beam also allows explicitly specifying a coder for <code>ValueState</code> values. For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">perUser</span> <span class="o">=</span> <span class="n">readPerUser</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">perUser</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;,</span> <span class="n">OutputT</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">ValueState</span><span class="o">&lt;</span><span class="n">MyType</span><span class="o">&gt;&gt;</span> <span class="n">numElements</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">(</span><span class="k">new</span> <span class="n">MyTypeCoder</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">ReadModifyWriteStateDoFn</span><span class="p">(</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">STATE_SPEC</span> <span class="o">=</span> <span class="n">ReadModifyWriteStateSpec</span><span class="p">(</span><span class="s1">&#39;num_elements&#39;</span><span class="p">,</span> <span class="n">VarIntCoder</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">,</span> <span class="n">state</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">STATE_SPEC</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Read the number element seen so far for this user key.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">current_value</span> <span class="o">=</span> <span class="n">state</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> <span class="ow">or</span> <span class="mi">0</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">current_value</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">_</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span> <span class="o">|</span> <span class="s1">&#39;Read per user&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ReadPerUser</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;state pardo&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">ReadModifyWriteStateDoFn</span><span class="p">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">valueStateDoFn</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Val</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Value</span><span class="p">[</span><span class="nx">MyCustomType</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">encode</span><span class="p">(</span><span class="nx">m</span> <span class="nx">MyCustomType</span><span class="p">)</span> <span class="p">[]</span><span class="kt">byte</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">m</span><span class="p">.</span><span class="nf">Bytes</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">decode</span><span class="p">(</span><span class="nx">b</span> <span class="p">[]</span><span class="kt">byte</span><span class="p">)</span> <span class="nx">MyCustomType</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">MyCustomType</span><span class="p">{}.</span><span class="nf">FromBytes</span><span class="p">(</span><span class="nx">b</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nf">RegisterCoder</span><span class="p">(</span><span class="nx">reflect</span><span class="p">.</span><span class="nf">TypeOf</span><span class="p">((</span><span class="o">*</span><span class="nx">MyCustomType</span><span class="p">)(</span><span class="kc">nil</span><span class="p">)).</span><span class="nf">Elem</span><span class="p">(),</span> <span class="nx">encode</span><span class="p">,</span> <span class="nx">decode</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-typescript snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">pcoll</span> <span class="o">=</span> <span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">([</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">key</span><span class="o">:</span> <span class="s2">&#34;a&#34;</span><span class="p">,</span> <span class="nx">value</span>: <span class="kt">1</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">key</span><span class="o">:</span> <span class="s2">&#34;b&#34;</span><span class="p">,</span> <span class="nx">value</span>: <span class="kt">10</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">key</span><span class="o">:</span> <span class="s2">&#34;a&#34;</span><span class="p">,</span> <span class="nx">value</span>: <span class="kt">100</span> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">])</span> |
| </span></span><span class="line"><span class="cl"><span class="p">);</span> |
| </span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">result</span>: <span class="kt">PCollection</span><span class="p">&lt;</span><span class="nt">number</span><span class="p">&gt;</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">pcoll</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">apply</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">withCoderInternal</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="nx">KVCoder</span><span class="p">(</span><span class="k">new</span> <span class="nx">StrUtf8Coder</span><span class="p">(),</span> <span class="k">new</span> <span class="nx">VarIntCoder</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">.</span><span class="nx">applyAsync</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">pythonTransform</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Construct a new Transform from source. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="s2">&#34;__constructor__&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">[</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">pythonCallable</span><span class="p">(</span><span class="sb">` |
| </span></span></span><span class="line"><span class="cl"><span class="sb"> # Define a DoFn to be used below. |
| </span></span></span><span class="line"><span class="cl"><span class="sb"> class ReadModifyWriteStateDoFn(beam.DoFn): |
| </span></span></span><span class="line"><span class="cl"><span class="sb"> STATE_SPEC = beam.transforms.userstate.ReadModifyWriteStateSpec( |
| </span></span></span><span class="line"><span class="cl"><span class="sb"> &#39;num_elements&#39;, beam.coders.VarIntCoder()) |
| </span></span></span><span class="line"><span class="cl"><span class="sb"> |
| </span></span></span><span class="line"><span class="cl"><span class="sb"> def process(self, element, state=beam.DoFn.StateParam(STATE_SPEC)): |
| </span></span></span><span class="line"><span class="cl"><span class="sb"> current_value = state.read() or 0 |
| </span></span></span><span class="line"><span class="cl"><span class="sb"> state.write(current_value + 1) |
| </span></span></span><span class="line"><span class="cl"><span class="sb"> yield current_value + 1 |
| </span></span></span><span class="line"><span class="cl"><span class="sb"> |
| </span></span></span><span class="line"><span class="cl"><span class="sb"> class MyPythonTransform(beam.PTransform): |
| </span></span></span><span class="line"><span class="cl"><span class="sb"> def expand(self, pcoll): |
| </span></span></span><span class="line"><span class="cl"><span class="sb"> return pcoll | beam.ParDo(ReadModifyWriteStateDoFn()) |
| </span></span></span><span class="line"><span class="cl"><span class="sb"> `</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">],</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Keyword arguments to pass to the transform, if any. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="p">{},</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Output type if it cannot be inferred |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="p">{</span> <span class="nx">requestedOutputCoders</span><span class="o">:</span> <span class="p">{</span> <span class="nx">output</span>: <span class="kt">new</span> <span class="nx">VarIntCoder</span><span class="p">()</span> <span class="p">}</span> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="combiningstate">CombiningState</h4> |
| <p><code>CombiningState</code> allows you to create a state object that is updated using a Beam combiner. For example, the previous |
| <code>ValueState</code> example could be rewritten to use <code>CombiningState</code></p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">perUser</span> <span class="o">=</span> <span class="n">readPerUser</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">perUser</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;,</span> <span class="n">OutputT</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">CombiningState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">,</span> <span class="kt">int</span><span class="o">[],</span> <span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">numElements</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">combining</span><span class="o">(</span><span class="n">Sum</span><span class="o">.</span><span class="na">ofIntegers</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span><span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">state</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">1</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">CombiningStateDoFn</span><span class="p">(</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">SUM_TOTAL</span> <span class="o">=</span> <span class="n">CombiningValueStateSpec</span><span class="p">(</span><span class="s1">&#39;total&#39;</span><span class="p">,</span> <span class="nb">sum</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">,</span> <span class="n">state</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">SUM_TOTAL</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">_</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span> <span class="o">|</span> <span class="s1">&#39;Read per user&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ReadPerUser</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Combine state pardo&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">CombiningStateDofn</span><span class="p">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// combiningStateFn keeps track of the number of elements seen. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">type</span> <span class="nx">combiningStateFn</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// types are the types of the accumulator, input, and output respectively |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">Val</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Combining</span><span class="p">[</span><span class="kt">int</span><span class="p">,</span> <span class="kt">int</span><span class="p">,</span> <span class="kt">int</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">s</span> <span class="o">*</span><span class="nx">combiningStateFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">p</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">book</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">word</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emitWords</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="kt">error</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Get the value stored in our state |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">val</span><span class="p">,</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">s</span><span class="p">.</span><span class="nx">Val</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">p</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">s</span><span class="p">.</span><span class="nx">Val</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="nx">p</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">val</span> <span class="p">&gt;</span> <span class="mi">10000</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Example of clearing and starting again with an empty bag |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">s</span><span class="p">.</span><span class="nx">Val</span><span class="p">.</span><span class="nf">Clear</span><span class="p">(</span><span class="nx">p</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">combineState</span><span class="p">(</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">input</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span><span class="p">)</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// CombineFn param can be a simple fn like this or a structural CombineFn |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">cFn</span> <span class="o">:=</span> <span class="nx">state</span><span class="p">.</span><span class="nx">MakeCombiningState</span><span class="p">[</span><span class="kt">int</span><span class="p">,</span> <span class="kt">int</span><span class="p">,</span> <span class="kt">int</span><span class="p">](</span><span class="s">&#34;stateKey&#34;</span><span class="p">,</span> <span class="kd">func</span><span class="p">(</span><span class="nx">a</span><span class="p">,</span> <span class="nx">b</span> <span class="kt">int</span><span class="p">)</span> <span class="kt">int</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">a</span> <span class="o">+</span> <span class="nx">b</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">})</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">combined</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">combiningStateFn</span><span class="p">{</span><span class="nx">Val</span><span class="p">:</span> <span class="nx">cFn</span><span class="p">},</span> <span class="nx">input</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="bagstate">BagState</h4> |
| <p>A common use case for state is to accumulate multiple elements. <code>BagState</code> allows for accumulating an unordered set |
| of elements. This allows for addition of elements to the collection without requiring the reading of the entire |
| collection first, which is an efficiency gain. In addition, runners that support paged reads can allow individual |
| bags larger than available memory.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">perUser</span> <span class="o">=</span> <span class="n">readPerUser</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">perUser</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;,</span> <span class="n">OutputT</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">BagState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">numElements</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">bag</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Element</span> <span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">element</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state&#34;</span><span class="o">)</span> <span class="n">BagState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">state</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Add the current element to the bag for this key. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">state</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">element</span><span class="o">.</span><span class="na">getValue</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="o">(</span><span class="n">shouldFetch</span><span class="o">())</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Occasionally we fetch and process the values. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">Iterable</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">values</span> <span class="o">=</span> <span class="n">state</span><span class="o">.</span><span class="na">read</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">processValues</span><span class="o">(</span><span class="n">values</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state</span><span class="o">.</span><span class="na">clear</span><span class="o">();</span> <span class="c1">// Clear the state for this key. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">BagStateDoFn</span><span class="p">(</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ALL_ELEMENTS</span> <span class="o">=</span> <span class="n">BagStateSpec</span><span class="p">(</span><span class="s1">&#39;buffer&#39;</span><span class="p">,</span> <span class="n">coders</span><span class="o">.</span><span class="n">VarIntCoder</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element_pair</span><span class="p">,</span> <span class="n">state</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">ALL_ELEMENTS</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">element_pair</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">should_fetch</span><span class="p">():</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">all_elements</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">state</span><span class="o">.</span><span class="n">read</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">process_values</span><span class="p">(</span><span class="n">all_elements</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">_</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span> <span class="o">|</span> <span class="s1">&#39;Read per user&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ReadPerUser</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Bag state pardo&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">BagStateDoFn</span><span class="p">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// bagStateFn only emits words that haven&#39;t been seen |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">type</span> <span class="nx">bagStateFn</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Bag</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Bag</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">s</span> <span class="o">*</span><span class="nx">bagStateFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">p</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">book</span><span class="p">,</span> <span class="nx">word</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emitWords</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="kt">error</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Get all values we&#39;ve written to this bag state in this window. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">vals</span><span class="p">,</span> <span class="nx">ok</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">s</span><span class="p">.</span><span class="nx">Bag</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">p</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="p">!</span><span class="nx">ok</span> <span class="o">||</span> <span class="p">!</span><span class="nf">contains</span><span class="p">(</span><span class="nx">vals</span><span class="p">,</span> <span class="nx">word</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emitWords</span><span class="p">(</span><span class="nx">word</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">s</span><span class="p">.</span><span class="nx">Bag</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="nx">p</span><span class="p">,</span> <span class="nx">word</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="nx">vals</span><span class="p">)</span> <span class="p">&gt;</span> <span class="mi">10000</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Example of clearing and starting again with an empty bag |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">s</span><span class="p">.</span><span class="nx">Bag</span><span class="p">.</span><span class="nf">Clear</span><span class="p">(</span><span class="nx">p</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="deferred-state-reads">11.2. Deferred state reads</h3> |
| <p>When a <code>DoFn</code> contains multiple state specifications, reading each one in order can be slow. Calling the <code>read()</code> function |
| on a state can cause the runner to perform a blocking read. Performing multiple blocking reads in sequence adds latency |
| to element processing. If you know that a state will always be read, you can annotate it as @AlwaysFetched, and then the |
| runner can prefetch all of the states necessary. For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">perUser</span> <span class="o">=</span> <span class="n">readPerUser</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">perUser</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;,</span> <span class="n">OutputT</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state1&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">state1</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state2&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">ValueState</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;&gt;</span> <span class="n">state2</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state3&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">BagState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">state3</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">bag</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@AlwaysFetched</span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state1&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">state1</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@AlwaysFetched</span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state2&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">state2</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@AlwaysFetched</span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state3&#34;</span><span class="o">)</span> <span class="n">BagState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">state3</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state1</span><span class="o">.</span><span class="na">read</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state2</span><span class="o">.</span><span class="na">read</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state3</span><span class="o">.</span><span class="na">read</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">This</span> <span class="ow">is</span> <span class="ow">not</span> <span class="n">supported</span> <span class="n">yet</span><span class="p">,</span> <span class="n">see</span> <span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">github</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">apache</span><span class="o">/</span><span class="n">beam</span><span class="o">/</span><span class="n">issues</span><span class="o">/</span><span class="mf">20739.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">This</span> <span class="nx">is</span> <span class="nx">not</span> <span class="nx">supported</span> <span class="nx">yet</span><span class="p">,</span> <span class="nx">see</span> <span class="nx">https</span><span class="p">:</span><span class="c1">//github.com/apache/beam/issues/22964. |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>If however there are code paths in which the states are not fetched, then annotating with @AlwaysFetched will add |
| unnecessary fetching for those paths. In this case, the readLater method allows the runner to know that the state will |
| be read in the future, allowing multiple state reads to be batched together.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">perUser</span> <span class="o">=</span> <span class="n">readPerUser</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">perUser</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;,</span> <span class="n">OutputT</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state1&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">state1</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state2&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">ValueState</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;&gt;</span> <span class="n">state2</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state3&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">BagState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">state3</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">bag</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state1&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">state1</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state2&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">state2</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state3&#34;</span><span class="o">)</span> <span class="n">BagState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">state3</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="o">(</span><span class="cm">/* should read state */</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state1</span><span class="o">.</span><span class="na">readLater</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state2</span><span class="o">.</span><span class="na">readLater</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state3</span><span class="o">.</span><span class="na">readLater</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The runner can now batch all three states into a single read, reducing latency. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">processState1</span><span class="o">(</span><span class="n">state1</span><span class="o">.</span><span class="na">read</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">processState2</span><span class="o">(</span><span class="n">state2</span><span class="o">.</span><span class="na">read</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">processState3</span><span class="o">(</span><span class="n">state3</span><span class="o">.</span><span class="na">read</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="timers">11.3. Timers</h3> |
| <p>Beam provides a per-key timer callback API. This allows for delayed processing of data stored using the state API. |
| Timers can be set to callback at either an event-time or a processing-time timestamp. Every timer is identified with a |
| TimerId. A given timer for a key can only be set for a single timestamp. Calling set on a timer overwrites the previous |
| firing time for that key&rsquo;s timer.</p> |
| <h4 id="event-time-timers">11.3.1. Event-time timers</h4> |
| <p>Event-time timers fire when the input watermark for the DoFn passes the time at which the timer is set, meaning that |
| the runner believes that there are no more elements to be processed with timestamps before the timer timestamp. This |
| allows for event-time aggregations.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">perUser</span> <span class="o">=</span> <span class="n">readPerUser</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">perUser</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;,</span> <span class="n">OutputT</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">state</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@TimerId</span><span class="o">(</span><span class="s">&#34;timer&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">TimerSpec</span> <span class="n">timer</span> <span class="o">=</span> <span class="n">TimerSpecs</span><span class="o">.</span><span class="na">timer</span><span class="o">(</span><span class="n">TimeDomain</span><span class="o">.</span><span class="na">EVENT_TIME</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Element</span> <span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">element</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Timestamp</span> <span class="n">Instant</span> <span class="n">elementTs</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">state</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@TimerId</span><span class="o">(</span><span class="s">&#34;timer&#34;</span><span class="o">)</span> <span class="n">Timer</span> <span class="n">timer</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Set an event-time timer to the element timestamp. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">timer</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">elementTs</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@OnTimer</span><span class="o">(</span><span class="s">&#34;timer&#34;</span><span class="o">)</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onTimer</span><span class="o">()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">//Process timer. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">EventTimerDoFn</span><span class="p">(</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ALL_ELEMENTS</span> <span class="o">=</span> <span class="n">BagStateSpec</span><span class="p">(</span><span class="s1">&#39;buffer&#39;</span><span class="p">,</span> <span class="n">coders</span><span class="o">.</span><span class="n">VarIntCoder</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TIMER</span> <span class="o">=</span> <span class="n">TimerSpec</span><span class="p">(</span><span class="s1">&#39;timer&#39;</span><span class="p">,</span> <span class="n">TimeDomain</span><span class="o">.</span><span class="n">WATERMARK</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">element_pair</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">t</span> <span class="o">=</span> <span class="n">DoFn</span><span class="o">.</span><span class="n">TimestampParam</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer</span> <span class="o">=</span> <span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">ALL_ELEMENTS</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timer</span> <span class="o">=</span> <span class="n">DoFn</span><span class="o">.</span><span class="n">TimerParam</span><span class="p">(</span><span class="n">TIMER</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">element_pair</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Set an event-time timer to the element timestamp.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timer</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@on_timer</span><span class="p">(</span><span class="n">TIMER</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">expiry_callback</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">buffer</span> <span class="o">=</span> <span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">ALL_ELEMENTS</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">_</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span> <span class="o">|</span> <span class="s1">&#39;Read per user&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ReadPerUser</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;EventTime timer pardo&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">EventTimerDoFn</span><span class="p">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">eventTimerDoFn</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">State</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Value</span><span class="p">[</span><span class="kt">int64</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Timer</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">EventTime</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">eventTimerDoFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">ts</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span><span class="p">,</span> <span class="nx">sp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">book</span><span class="p">,</span> <span class="nx">word</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emitWords</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Set an event-time timer to the element timestamp. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">fn</span><span class="p">.</span><span class="nx">Timer</span><span class="p">.</span><span class="nf">Set</span><span class="p">(</span><span class="nx">tp</span><span class="p">,</span> <span class="nx">ts</span><span class="p">.</span><span class="nf">ToTime</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">eventTimerDoFn</span><span class="p">)</span> <span class="nf">OnTimer</span><span class="p">(</span><span class="nx">sp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">w</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Window</span><span class="p">,</span> <span class="nx">key</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">timer</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">emitWords</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">switch</span> <span class="nx">timer</span><span class="p">.</span><span class="nx">Family</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">case</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">Timer</span><span class="p">.</span><span class="nx">Family</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// process callback for this timer |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">AddEventTimeDoFn</span><span class="p">(</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">in</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span><span class="p">)</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">eventTimerDoFn</span><span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Timers are given family names so their callbacks can be handled independantly. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">Timer</span><span class="p">:</span> <span class="nx">timers</span><span class="p">.</span><span class="nf">InEventTime</span><span class="p">(</span><span class="s">&#34;processWatermark&#34;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">State</span><span class="p">:</span> <span class="nx">state</span><span class="p">.</span><span class="nx">MakeValueState</span><span class="p">[</span><span class="kt">int64</span><span class="p">](</span><span class="s">&#34;latest&#34;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="nx">in</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="processing-time-timers">11.3.2. Processing-time timers</h4> |
| <p>Processing-time timers fire when the real wall-clock time passes. This is often used to create larger batches of data |
| before processing. It can also be used to schedule events that should occur at a specific time. Just like with |
| event-time timers, processing-time timers are per key - each key has a separate copy of the timer.</p> |
| <p>While processing-time timers can be set to an absolute timestamp, it is very common to set them to an offset relative |
| to the current time. In Java, the <code>Timer.offset</code> and <code>Timer.setRelative</code> methods can be used to accomplish this.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">perUser</span> <span class="o">=</span> <span class="n">readPerUser</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">perUser</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;,</span> <span class="n">OutputT</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@TimerId</span><span class="o">(</span><span class="s">&#34;timer&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">TimerSpec</span> <span class="n">timer</span> <span class="o">=</span> <span class="n">TimerSpecs</span><span class="o">.</span><span class="na">timer</span><span class="o">(</span><span class="n">TimeDomain</span><span class="o">.</span><span class="na">PROCESSING_TIME</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span><span class="nd">@TimerId</span><span class="o">(</span><span class="s">&#34;timer&#34;</span><span class="o">)</span> <span class="n">Timer</span> <span class="n">timer</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Set a timer to go off 30 seconds in the future. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">timer</span><span class="o">.</span><span class="na">offset</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="n">30</span><span class="o">)).</span><span class="na">setRelative</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@OnTimer</span><span class="o">(</span><span class="s">&#34;timer&#34;</span><span class="o">)</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onTimer</span><span class="o">()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">//Process timer. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">ProcessingTimerDoFn</span><span class="p">(</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ALL_ELEMENTS</span> <span class="o">=</span> <span class="n">BagStateSpec</span><span class="p">(</span><span class="s1">&#39;buffer&#39;</span><span class="p">,</span> <span class="n">coders</span><span class="o">.</span><span class="n">VarIntCoder</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TIMER</span> <span class="o">=</span> <span class="n">TimerSpec</span><span class="p">(</span><span class="s1">&#39;timer&#39;</span><span class="p">,</span> <span class="n">TimeDomain</span><span class="o">.</span><span class="n">REAL_TIME</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">element_pair</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer</span> <span class="o">=</span> <span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">ALL_ELEMENTS</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timer</span> <span class="o">=</span> <span class="n">DoFn</span><span class="o">.</span><span class="n">TimerParam</span><span class="p">(</span><span class="n">TIMER</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">element_pair</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Set a timer to go off 30 seconds in the future.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timer</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">Timestamp</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">+</span> <span class="n">Duration</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="mi">30</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@on_timer</span><span class="p">(</span><span class="n">TIMER</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">expiry_callback</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">buffer</span> <span class="o">=</span> <span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">ALL_ELEMENTS</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Process timer.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">_</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span> <span class="o">|</span> <span class="s1">&#39;Read per user&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ReadPerUser</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;ProcessingTime timer pardo&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">ProcessingTimerDoFn</span><span class="p">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">processingTimerDoFn</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Timer</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">ProcessingTime</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">processingTimerDoFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">sp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">book</span><span class="p">,</span> <span class="nx">word</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emitWords</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Set a timer to go off 30 seconds in the future. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">fn</span><span class="p">.</span><span class="nx">Timer</span><span class="p">.</span><span class="nf">Set</span><span class="p">(</span><span class="nx">tp</span><span class="p">,</span> <span class="nx">time</span><span class="p">.</span><span class="nf">Now</span><span class="p">().</span><span class="nf">Add</span><span class="p">(</span><span class="mi">30</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">processingTimerDoFn</span><span class="p">)</span> <span class="nf">OnTimer</span><span class="p">(</span><span class="nx">sp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">w</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Window</span><span class="p">,</span> <span class="nx">key</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">timer</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">emitWords</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">switch</span> <span class="nx">timer</span><span class="p">.</span><span class="nx">Family</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">case</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">Timer</span><span class="p">.</span><span class="nx">Family</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// process callback for this timer |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">AddProcessingTimeDoFn</span><span class="p">(</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">in</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span><span class="p">)</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">processingTimerDoFn</span><span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Timers are given family names so their callbacks can be handled independantly. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">Timer</span><span class="p">:</span> <span class="nx">timers</span><span class="p">.</span><span class="nf">InProcessingTime</span><span class="p">(</span><span class="s">&#34;timer&#34;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="nx">in</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="dynamic-timer-tags">11.3.3. Dynamic timer tags</h4> |
| <p>Beam also supports dynamically setting a timer tag using <code>TimerMap</code> in the Java SDK. This allows for setting multiple different timers |
| in a <code>DoFn</code> and allowing for the timer tags to be dynamically chosen - e.g. based on data in the input elements. A |
| timer with a specific tag can only be set to a single timestamp, so setting the timer again has the effect of |
| overwriting the previous expiration time for the timer with that tag. Each <code>TimerMap</code> is identified with a timer family |
| id, and timers in different timer families are independent.</p> |
| <p>In the Python SDK, a dynamic timer tag can be specified while calling <code>set()</code> or <code>clear()</code>. By default, the timer |
| tag is an empty string if not specified.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">perUser</span> <span class="o">=</span> <span class="n">readPerUser</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">perUser</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;,</span> <span class="n">OutputT</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@TimerFamily</span><span class="o">(</span><span class="s">&#34;actionTimers&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">TimerSpec</span> <span class="n">timer</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TimerSpecs</span><span class="o">.</span><span class="na">timerMap</span><span class="o">(</span><span class="n">TimeDomain</span><span class="o">.</span><span class="na">EVENT_TIME</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Element</span> <span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">element</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Timestamp</span> <span class="n">Instant</span> <span class="n">elementTs</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@TimerFamily</span><span class="o">(</span><span class="s">&#34;actionTimers&#34;</span><span class="o">)</span> <span class="n">TimerMap</span> <span class="n">timers</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timers</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">element</span><span class="o">.</span><span class="na">getValue</span><span class="o">().</span><span class="na">getActionType</span><span class="o">(),</span> <span class="n">elementTs</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@OnTimerFamily</span><span class="o">(</span><span class="s">&#34;actionTimers&#34;</span><span class="o">)</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onTimer</span><span class="o">(</span><span class="nd">@TimerId</span> <span class="n">String</span> <span class="n">timerId</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">LOG</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">&#34;Timer fired with id &#34;</span> <span class="o">+</span> <span class="n">timerId</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">TimerDoFn</span><span class="p">(</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ALL_ELEMENTS</span> <span class="o">=</span> <span class="n">BagStateSpec</span><span class="p">(</span><span class="s1">&#39;buffer&#39;</span><span class="p">,</span> <span class="n">coders</span><span class="o">.</span><span class="n">VarIntCoder</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TIMER</span> <span class="o">=</span> <span class="n">TimerSpec</span><span class="p">(</span><span class="s1">&#39;timer&#39;</span><span class="p">,</span> <span class="n">TimeDomain</span><span class="o">.</span><span class="n">REAL_TIME</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">element_pair</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer</span> <span class="o">=</span> <span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">ALL_ELEMENTS</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timer</span> <span class="o">=</span> <span class="n">DoFn</span><span class="o">.</span><span class="n">TimerParam</span><span class="p">(</span><span class="n">TIMER</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">element_pair</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Set a timer to go off 30 seconds in the future with dynamic timer tag &#39;first_timer&#39;.</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># And set a timer to go off 60 seconds in the future with dynamic timer tag &#39;second_timer&#39;.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timer</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">Timestamp</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">+</span> <span class="n">Duration</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="mi">30</span><span class="p">),</span> <span class="n">dynamic_timer_tag</span><span class="o">=</span><span class="s1">&#39;first_timer&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timer</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">Timestamp</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">+</span> <span class="n">Duration</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="mi">60</span><span class="p">),</span> <span class="n">dynamic_timer_tag</span><span class="o">=</span><span class="s1">&#39;second_timer&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Note that a timer can also be explicitly cleared if previously set with a dynamic timer tag:</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># timer.clear(dynamic_timer_tag=...)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@on_timer</span><span class="p">(</span><span class="n">TIMER</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">expiry_callback</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">buffer</span> <span class="o">=</span> <span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">ALL_ELEMENTS</span><span class="p">),</span> <span class="n">timer_tag</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">DynamicTimerTagParam</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Process timer, the dynamic timer tag associated with expiring timer can be read back with DoFn.DynamicTimerTagParam.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="p">(</span><span class="n">timer_tag</span><span class="p">,</span> <span class="s1">&#39;fired&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">_</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span> <span class="o">|</span> <span class="s1">&#39;Read per user&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ReadPerUser</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;ProcessingTime timer pardo&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">TimerDoFn</span><span class="p">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">hasAction</span> <span class="kd">interface</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">Action</span><span class="p">()</span> <span class="kt">string</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">dynamicTagsDoFn</span><span class="p">[</span><span class="nx">V</span> <span class="nx">hasAction</span><span class="p">]</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Timer</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">EventTime</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">dynamicTagsDoFn</span><span class="p">[</span><span class="nx">V</span><span class="p">])</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">ts</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span><span class="p">,</span> <span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">key</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">value</span> <span class="nx">V</span><span class="p">,</span> <span class="nx">emitWords</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Set a timer to go off 30 seconds in the future. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">fn</span><span class="p">.</span><span class="nx">Timer</span><span class="p">.</span><span class="nf">Set</span><span class="p">(</span><span class="nx">tp</span><span class="p">,</span> <span class="nx">ts</span><span class="p">.</span><span class="nf">ToTime</span><span class="p">(),</span> <span class="nx">timers</span><span class="p">.</span><span class="nf">WithTag</span><span class="p">(</span><span class="nx">value</span><span class="p">.</span><span class="nf">Action</span><span class="p">()))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">dynamicTagsDoFn</span><span class="p">[</span><span class="nx">V</span><span class="p">])</span> <span class="nf">OnTimer</span><span class="p">(</span><span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">w</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Window</span><span class="p">,</span> <span class="nx">key</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">timer</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">emitWords</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">switch</span> <span class="nx">timer</span><span class="p">.</span><span class="nx">Family</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">case</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">Timer</span><span class="p">.</span><span class="nx">Family</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">tag</span> <span class="o">:=</span> <span class="nx">timer</span><span class="p">.</span><span class="nx">Tag</span> <span class="c1">// Do something with fired tag |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">_</span> <span class="p">=</span> <span class="nx">tag</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nx">AddDynamicTimerTagsDoFn</span><span class="p">[</span><span class="nx">V</span> <span class="nx">hasAction</span><span class="p">](</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">in</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span><span class="p">)</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">dynamicTagsDoFn</span><span class="p">[</span><span class="nx">V</span><span class="p">]{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Timer</span><span class="p">:</span> <span class="nx">timers</span><span class="p">.</span><span class="nf">InEventTime</span><span class="p">(</span><span class="s">&#34;actionTimers&#34;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="nx">in</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="timer-output-timestamps">11.3.4. Timer output timestamps</h4> |
| <p>By default, event-time timers will hold the output watermark of the <code>ParDo</code> to the timestamp of the timer. This means |
| that if a timer is set to 12pm, any windowed aggregations or event-time timers later in the pipeline graph that finish |
| after 12pm will not expire. The timestamp of the timer is also the default output timestamp for the timer callback. This |
| means that any elements output from the onTimer method will have a timestamp equal to the timestamp of the timer firing. |
| For processing-time timers, the default output timestamp and watermark hold is the value of the input watermark at the |
| time the timer was set.</p> |
| <p>In some cases, a DoFn needs to output timestamps earlier than the timer expiration time, and therefore also needs to |
| hold its output watermark to those timestamps. For example, consider the following pipeline that temporarily batches |
| records into state, and sets a timer to drain the state. This code may appear correct, but will not work properly.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">perUser</span> <span class="o">=</span> <span class="n">readPerUser</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">perUser</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;,</span> <span class="n">OutputT</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;elementBag&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">BagState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">elementBag</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">bag</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;timerSet&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Boolean</span><span class="o">&gt;&gt;</span> <span class="n">timerSet</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@TimerId</span><span class="o">(</span><span class="s">&#34;outputState&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">TimerSpec</span> <span class="n">timer</span> <span class="o">=</span> <span class="n">TimerSpecs</span><span class="o">.</span><span class="na">timer</span><span class="o">(</span><span class="n">TimeDomain</span><span class="o">.</span><span class="na">PROCESSING_TIME</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Element</span> <span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">element</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;elementBag&#34;</span><span class="o">)</span> <span class="n">BagState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">elementBag</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;timerSet&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Boolean</span><span class="o">&gt;</span> <span class="n">timerSet</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@TimerId</span><span class="o">(</span><span class="s">&#34;outputState&#34;</span><span class="o">)</span> <span class="n">Timer</span> <span class="n">timer</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Add the current element to the bag for this key. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">elementBag</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">element</span><span class="o">.</span><span class="na">getValue</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="o">(!</span><span class="n">MoreObjects</span><span class="o">.</span><span class="na">firstNonNull</span><span class="o">(</span><span class="n">timerSet</span><span class="o">.</span><span class="na">read</span><span class="o">(),</span> <span class="kc">false</span><span class="o">))</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// If the timer is not current set, then set it to go off in a minute. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">timer</span><span class="o">.</span><span class="na">offset</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="n">1</span><span class="o">)).</span><span class="na">setRelative</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timerSet</span><span class="o">.</span><span class="na">write</span><span class="o">(</span><span class="kc">true</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@OnTimer</span><span class="o">(</span><span class="s">&#34;outputState&#34;</span><span class="o">)</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onTimer</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;elementBag&#34;</span><span class="o">)</span> <span class="n">BagState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">elementBag</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;timerSet&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Boolean</span><span class="o">&gt;</span> <span class="n">timerSet</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">OutputReceiver</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">output</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="o">(</span><span class="n">ValueT</span> <span class="n">bufferedElement</span> <span class="o">:</span> <span class="n">elementBag</span><span class="o">.</span><span class="na">read</span><span class="o">())</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Output each element. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">output</span><span class="o">.</span><span class="na">outputWithTimestamp</span><span class="o">(</span><span class="n">bufferedElement</span><span class="o">,</span> <span class="n">bufferedElement</span><span class="o">.</span><span class="na">timestamp</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">elementBag</span><span class="o">.</span><span class="na">clear</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Note that the timer has now fired. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">timerSet</span><span class="o">.</span><span class="na">clear</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">badTimerOutputTimestampsFn</span><span class="p">[</span><span class="nx">V</span> <span class="nx">any</span><span class="p">]</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">ElementBag</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Bag</span><span class="p">[</span><span class="nx">V</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">TimerSet</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Value</span><span class="p">[</span><span class="kt">bool</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">OutputState</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">ProcessingTime</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">badTimerOutputTimestampsFn</span><span class="p">[</span><span class="nx">V</span><span class="p">])</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">sp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">key</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">value</span> <span class="nx">V</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="kt">error</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Add the current element to the bag for this key. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">ElementBag</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="nx">sp</span><span class="p">,</span> <span class="nx">value</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">set</span><span class="p">,</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">TimerSet</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="p">!</span><span class="nx">set</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">OutputState</span><span class="p">.</span><span class="nf">Set</span><span class="p">(</span><span class="nx">tp</span><span class="p">,</span> <span class="nx">time</span><span class="p">.</span><span class="nf">Now</span><span class="p">().</span><span class="nf">Add</span><span class="p">(</span><span class="mi">1</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Minute</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">TimerSet</span><span class="p">.</span><span class="nf">Write</span><span class="p">(</span><span class="nx">sp</span><span class="p">,</span> <span class="kc">true</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">badTimerOutputTimestampsFn</span><span class="p">[</span><span class="nx">V</span><span class="p">])</span> <span class="nf">OnTimer</span><span class="p">(</span><span class="nx">sp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">w</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Window</span><span class="p">,</span> <span class="nx">key</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">timer</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="kt">error</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">switch</span> <span class="nx">timer</span><span class="p">.</span><span class="nx">Family</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">case</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">OutputState</span><span class="p">.</span><span class="nx">Family</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">vs</span><span class="p">,</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">ElementBag</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">v</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">vs</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Output each element |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nf">emit</span><span class="p">(</span><span class="nx">fmt</span><span class="p">.</span><span class="nf">Sprintf</span><span class="p">(</span><span class="s">&#34;%v&#34;</span><span class="p">,</span> <span class="nx">v</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">ElementBag</span><span class="p">.</span><span class="nf">Clear</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Note that the timer has now fired. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">fn</span><span class="p">.</span><span class="nx">TimerSet</span><span class="p">.</span><span class="nf">Clear</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>The problem with this code is that the ParDo is buffering elements, however nothing is preventing the watermark |
| from advancing past the timestamp of those elements, so all those elements might be dropped as late data. In order |
| to prevent this from happening, an output timestamp needs to be set on the timer to prevent the watermark from advancing |
| past the timestamp of the minimum element. The following code demonstrates this.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">perUser</span> <span class="o">=</span> <span class="n">readPerUser</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">perUser</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;,</span> <span class="n">OutputT</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The bag of elements accumulated. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;elementBag&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">BagState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">elementBag</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">bag</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The timestamp of the timer set. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;timerTimestamp&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">&gt;&gt;</span> <span class="n">timerTimestamp</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The minimum timestamp stored in the bag. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;minTimestampInBag&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">CombiningState</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="kt">long</span><span class="o">[],</span> <span class="n">Long</span><span class="o">&gt;&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">minTimestampInBag</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">combining</span><span class="o">(</span><span class="n">Min</span><span class="o">.</span><span class="na">ofLongs</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@TimerId</span><span class="o">(</span><span class="s">&#34;outputState&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">TimerSpec</span> <span class="n">timer</span> <span class="o">=</span> <span class="n">TimerSpecs</span><span class="o">.</span><span class="na">timer</span><span class="o">(</span><span class="n">TimeDomain</span><span class="o">.</span><span class="na">PROCESSING_TIME</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Element</span> <span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">element</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;elementBag&#34;</span><span class="o">)</span> <span class="n">BagState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">elementBag</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@AlwaysFetched</span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;timerTimestamp&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">&gt;</span> <span class="n">timerTimestamp</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@AlwaysFetched</span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;minTimestampInBag&#34;</span><span class="o">)</span> <span class="n">CombiningState</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="kt">long</span><span class="o">[],</span> <span class="n">Long</span><span class="o">&gt;</span> <span class="n">minTimestamp</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@TimerId</span><span class="o">(</span><span class="s">&#34;outputState&#34;</span><span class="o">)</span> <span class="n">Timer</span> <span class="n">timer</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Add the current element to the bag for this key. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">elementBag</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">element</span><span class="o">.</span><span class="na">getValue</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Keep track of the minimum element timestamp currently stored in the bag. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">minTimestamp</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">element</span><span class="o">.</span><span class="na">getValue</span><span class="o">().</span><span class="na">timestamp</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// If the timer is already set, then reset it at the same time but with an updated output timestamp (otherwise |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// we would keep resetting the timer to the future). If there is no timer set, then set one to expire in a minute. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">Long</span> <span class="n">timerTimestampMs</span> <span class="o">=</span> <span class="n">timerTimestamp</span><span class="o">.</span><span class="na">read</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Instant</span> <span class="n">timerToSet</span> <span class="o">=</span> <span class="o">(</span><span class="n">timerTimestamp</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">().</span><span class="na">read</span><span class="o">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">?</span> <span class="n">Instant</span><span class="o">.</span><span class="na">now</span><span class="o">().</span><span class="na">plus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="n">1</span><span class="o">))</span> <span class="o">:</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="n">timerTimestampMs</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Setting the outputTimestamp to the minimum timestamp in the bag holds the watermark to that timestamp until the |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// timer fires. This allows outputting all the elements with their timestamp. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">timer</span><span class="o">.</span><span class="na">withOutputTimestamp</span><span class="o">(</span><span class="n">minTimestamp</span><span class="o">.</span><span class="na">read</span><span class="o">()).</span><span class="na">s</span> <span class="n">et</span><span class="o">(</span><span class="n">timerToSet</span><span class="o">).</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timerTimestamp</span><span class="o">.</span><span class="na">write</span><span class="o">(</span><span class="n">timerToSet</span><span class="o">.</span><span class="na">getMillis</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@OnTimer</span><span class="o">(</span><span class="s">&#34;outputState&#34;</span><span class="o">)</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onTimer</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;elementBag&#34;</span><span class="o">)</span> <span class="n">BagState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">elementBag</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;timerTimestamp&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">&gt;</span> <span class="n">timerTimestamp</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">OutputReceiver</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">output</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="o">(</span><span class="n">ValueT</span> <span class="n">bufferedElement</span> <span class="o">:</span> <span class="n">elementBag</span><span class="o">.</span><span class="na">read</span><span class="o">())</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Output each element. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">output</span><span class="o">.</span><span class="na">outputWithTimestamp</span><span class="o">(</span><span class="n">bufferedElement</span><span class="o">,</span> <span class="n">bufferedElement</span><span class="o">.</span><span class="na">timestamp</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Note that the timer has now fired. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">timerTimestamp</span><span class="o">.</span><span class="na">clear</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">Timer</span> <span class="n">output</span> <span class="n">timestamps</span> <span class="ow">is</span> <span class="ow">not</span> <span class="n">yet</span> <span class="n">supported</span> <span class="ow">in</span> <span class="n">Python</span> <span class="n">SDK</span><span class="o">.</span> <span class="n">See</span> <span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">github</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">apache</span><span class="o">/</span><span class="n">beam</span><span class="o">/</span><span class="n">issues</span><span class="o">/</span><span class="mf">20705.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">element</span><span class="p">[</span><span class="nx">V</span> <span class="nx">any</span><span class="p">]</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Timestamp</span> <span class="kt">int64</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Value</span> <span class="nx">V</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">goodTimerOutputTimestampsFn</span><span class="p">[</span><span class="nx">V</span> <span class="nx">any</span><span class="p">]</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">ElementBag</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Bag</span><span class="p">[</span><span class="nx">element</span><span class="p">[</span><span class="nx">V</span><span class="p">]]</span> <span class="c1">// The bag of elements accumulated. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">TimerTimerstamp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Value</span><span class="p">[</span><span class="kt">int64</span><span class="p">]</span> <span class="c1">// The timestamp of the timer set. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">MinTimestampInBag</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Combining</span><span class="p">[</span><span class="kt">int64</span><span class="p">,</span> <span class="kt">int64</span><span class="p">,</span> <span class="kt">int64</span><span class="p">]</span> <span class="c1">// The minimum timestamp stored in the bag. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">OutputState</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">ProcessingTime</span> <span class="c1">// The timestamp of the timer. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">goodTimerOutputTimestampsFn</span><span class="p">[</span><span class="nx">V</span><span class="p">])</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">et</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span><span class="p">,</span> <span class="nx">sp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">key</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">value</span> <span class="nx">V</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span><span class="p">,</span> <span class="kt">string</span><span class="p">))</span> <span class="kt">error</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// Add the current element to the bag for this key, and preserve the event time. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">if</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">ElementBag</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="nx">sp</span><span class="p">,</span> <span class="nx">element</span><span class="p">[</span><span class="nx">V</span><span class="p">]{</span><span class="nx">Timestamp</span><span class="p">:</span> <span class="nx">et</span><span class="p">.</span><span class="nf">Milliseconds</span><span class="p">(),</span> <span class="nx">Value</span><span class="p">:</span> <span class="nx">value</span><span class="p">});</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Keep track of the minimum element timestamp currently stored in the bag. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">fn</span><span class="p">.</span><span class="nx">MinTimestampInBag</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="nx">sp</span><span class="p">,</span> <span class="nx">et</span><span class="p">.</span><span class="nf">Milliseconds</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// If the timer is already set, then reset it at the same time but with an updated output timestamp (otherwise |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// we would keep resetting the timer to the future). If there is no timer set, then set one to expire in a minute. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">ts</span><span class="p">,</span> <span class="nx">ok</span><span class="p">,</span> <span class="nx">_</span> <span class="o">:=</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">TimerTimerstamp</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">var</span> <span class="nx">tsToSet</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Time</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">ok</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">tsToSet</span> <span class="p">=</span> <span class="nx">time</span><span class="p">.</span><span class="nf">UnixMilli</span><span class="p">(</span><span class="nx">ts</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">tsToSet</span> <span class="p">=</span> <span class="nx">time</span><span class="p">.</span><span class="nf">Now</span><span class="p">().</span><span class="nf">Add</span><span class="p">(</span><span class="mi">1</span> <span class="o">*</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Minute</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nx">minTs</span><span class="p">,</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">_</span> <span class="o">:=</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">MinTimestampInBag</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">outputTs</span> <span class="o">:=</span> <span class="nx">time</span><span class="p">.</span><span class="nf">UnixMilli</span><span class="p">(</span><span class="nx">minTs</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Setting the outputTimestamp to the minimum timestamp in the bag holds the watermark to that timestamp until the |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// timer fires. This allows outputting all the elements with their timestamp. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">fn</span><span class="p">.</span><span class="nx">OutputState</span><span class="p">.</span><span class="nf">Set</span><span class="p">(</span><span class="nx">tp</span><span class="p">,</span> <span class="nx">tsToSet</span><span class="p">,</span> <span class="nx">timers</span><span class="p">.</span><span class="nf">WithOutputTimestamp</span><span class="p">(</span><span class="nx">outputTs</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">TimerTimerstamp</span><span class="p">.</span><span class="nf">Write</span><span class="p">(</span><span class="nx">sp</span><span class="p">,</span> <span class="nx">tsToSet</span><span class="p">.</span><span class="nf">UnixMilli</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">goodTimerOutputTimestampsFn</span><span class="p">[</span><span class="nx">V</span><span class="p">])</span> <span class="nf">OnTimer</span><span class="p">(</span><span class="nx">sp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">w</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Window</span><span class="p">,</span> <span class="nx">key</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">timer</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span><span class="p">,</span> <span class="kt">string</span><span class="p">))</span> <span class="kt">error</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">switch</span> <span class="nx">timer</span><span class="p">.</span><span class="nx">Family</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">case</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">OutputState</span><span class="p">.</span><span class="nx">Family</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">vs</span><span class="p">,</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">ElementBag</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">v</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">vs</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Output each element with their timestamp |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nf">emit</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nf">EventTime</span><span class="p">(</span><span class="nx">v</span><span class="p">.</span><span class="nx">Timestamp</span><span class="p">),</span> <span class="nx">fmt</span><span class="p">.</span><span class="nf">Sprintf</span><span class="p">(</span><span class="s">&#34;%v&#34;</span><span class="p">,</span> <span class="nx">v</span><span class="p">.</span><span class="nx">Value</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">ElementBag</span><span class="p">.</span><span class="nf">Clear</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Note that the timer has now fired. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">fn</span><span class="p">.</span><span class="nx">TimerTimerstamp</span><span class="p">.</span><span class="nf">Clear</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nx">AddTimedOutputBatching</span><span class="p">[</span><span class="nx">V</span> <span class="nx">any</span><span class="p">](</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">in</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span><span class="p">)</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">goodTimerOutputTimestampsFn</span><span class="p">[</span><span class="nx">V</span><span class="p">]{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">ElementBag</span><span class="p">:</span> <span class="nx">state</span><span class="p">.</span><span class="nx">MakeBagState</span><span class="p">[</span><span class="nx">element</span><span class="p">[</span><span class="nx">V</span><span class="p">]](</span><span class="s">&#34;elementBag&#34;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">TimerTimerstamp</span><span class="p">:</span> <span class="nx">state</span><span class="p">.</span><span class="nx">MakeValueState</span><span class="p">[</span><span class="kt">int64</span><span class="p">](</span><span class="s">&#34;timerTimestamp&#34;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">MinTimestampInBag</span><span class="p">:</span> <span class="nx">state</span><span class="p">.</span><span class="nx">MakeCombiningState</span><span class="p">[</span><span class="kt">int64</span><span class="p">,</span> <span class="kt">int64</span><span class="p">,</span> <span class="kt">int64</span><span class="p">](</span><span class="s">&#34;minTimestampInBag&#34;</span><span class="p">,</span> <span class="kd">func</span><span class="p">(</span><span class="nx">a</span><span class="p">,</span> <span class="nx">b</span> <span class="kt">int64</span><span class="p">)</span> <span class="kt">int64</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">a</span> <span class="p">&lt;</span> <span class="nx">b</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">a</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">b</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">OutputState</span><span class="p">:</span> <span class="nx">timers</span><span class="p">.</span><span class="nf">InProcessingTime</span><span class="p">(</span><span class="s">&#34;outputState&#34;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="nx">in</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="garbage-collecting-state">11.4. Garbage collecting state</h3> |
| <p>Per-key state needs to be garbage collected, or eventually the increasing size of state may negatively impact |
| performance. There are two common strategies for garbage collecting state.</p> |
| <h5 id="using-windows-for-garbage-collection">11.4.1. <strong>Using windows for garbage collection</strong></h5> |
| <p>All state and timers for a key is scoped to the window it is in. This means that depending on the timestamp of the |
| input element the ParDo will see different values for the state depending on the window that element falls into. In |
| addition, once the input watermark passes the end of the window, the runner should garbage collect all state for that |
| window. (note: if allowed lateness is set to a positive value for the window, the runner must wait for the watermark to |
| pass the end of the window plus the allowed lateness before garbage collecting state). This can be used as a |
| garbage-collection strategy.</p> |
| <p>For example, given the following:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">perUser</span> <span class="o">=</span> <span class="n">readPerUser</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">perUser</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Window</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">CalendarWindows</span><span class="o">.</span><span class="na">days</span><span class="o">(</span><span class="n">1</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withTimeZone</span><span class="o">(</span><span class="n">DateTimeZone</span><span class="o">.</span><span class="na">forID</span><span class="o">(</span><span class="s">&#34;America/Los_Angeles&#34;</span><span class="o">))));</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;,</span> <span class="n">OutputT</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">state</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span><span class="nd">@Timestamp</span> <span class="n">Instant</span> <span class="n">ts</span><span class="o">,</span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">state</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The state is scoped to a calendar day window. That means that if the input timestamp ts is after |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// midnight PST, then a new copy of the state will be seen for the next day. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">StateDoFn</span><span class="p">(</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ALL_ELEMENTS</span> <span class="o">=</span> <span class="n">BagStateSpec</span><span class="p">(</span><span class="s1">&#39;buffer&#39;</span><span class="p">,</span> <span class="n">coders</span><span class="o">.</span><span class="n">VarIntCoder</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">element_pair</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer</span> <span class="o">=</span> <span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">ALL_ELEMENTS</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">_</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span> <span class="o">|</span> <span class="s1">&#39;Read per user&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ReadPerUser</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Windowing&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span><span class="n">FixedWindows</span><span class="p">(</span><span class="mi">60</span> <span class="o">*</span> <span class="mi">60</span> <span class="o">*</span> <span class="mi">24</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;DoFn&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">StateDoFn</span><span class="p">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"> <span class="nx">items</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">statefulDoFn</span><span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">S</span><span class="p">:</span> <span class="nx">state</span><span class="p">.</span><span class="nx">MakeValueState</span><span class="p">[</span><span class="kt">int</span><span class="p">](</span><span class="s">&#34;S&#34;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="nx">elements</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">out</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">WindowInto</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">window</span><span class="p">.</span><span class="nf">NewFixedWindows</span><span class="p">(</span><span class="mi">24</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Hour</span><span class="p">),</span> <span class="nx">items</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>This <code>ParDo</code> stores state per day. Once the pipeline is done processing data for a given day, all the state for that |
| day is garbage collected.</p> |
| <h5 id="using-timers-for-garbage-collection">11.4.1. <strong>Using timers For garbage collection</strong></h5> |
| <p>In some cases, it is difficult to find a windowing strategy that models the desired garbage-collection strategy. For |
| example, a common desire is to garbage collect state for a key once no activity has been seen on the key for some time. |
| This can be done by updating a timer that garbage collects state. For example</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">perUser</span> <span class="o">=</span> <span class="n">readPerUser</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">perUser</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;,</span> <span class="n">OutputT</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The state for the key. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">ValueState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">state</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The maximum element timestamp seen so far. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;maxTimestampSeen&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">CombiningState</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="kt">long</span><span class="o">[],</span> <span class="n">Long</span><span class="o">&gt;&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">maxTimestamp</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">combining</span><span class="o">(</span><span class="n">Max</span><span class="o">.</span><span class="na">ofLongs</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@TimerId</span><span class="o">(</span><span class="s">&#34;gcTimer&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">TimerSpec</span> <span class="n">gcTimer</span> <span class="o">=</span> <span class="n">TimerSpecs</span><span class="o">.</span><span class="na">timer</span><span class="o">(</span><span class="n">TimeDomain</span><span class="o">.</span><span class="na">EVENT_TIME</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Element</span> <span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">element</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Timestamp</span> <span class="n">Instant</span> <span class="n">ts</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">state</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;maxTimestampSeen&#34;</span><span class="o">)</span> <span class="n">CombiningState</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="kt">long</span><span class="o">[],</span> <span class="n">Long</span><span class="o">&gt;</span> <span class="n">maxTimestamp</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@TimerId</span><span class="o">(</span><span class="s">&#34;gcTimer&#34;</span><span class="o">)</span> <span class="n">gcTimer</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">updateState</span><span class="o">(</span><span class="n">state</span><span class="o">,</span> <span class="n">element</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">maxTimestamp</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">ts</span><span class="o">.</span><span class="na">getMillis</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Set the timer to be one hour after the maximum timestamp seen. This will keep overwriting the same timer, so |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// as long as there is activity on this key the state will stay active. Once the key goes inactive for one hour&#39;s |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// worth of event time (as measured by the watermark), then the gc timer will fire. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">Instant</span> <span class="n">expirationTime</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="n">maxTimestamp</span><span class="o">.</span><span class="na">read</span><span class="o">()).</span><span class="na">plus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardHours</span><span class="o">(</span><span class="n">1</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timer</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">expirationTime</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@OnTimer</span><span class="o">(</span><span class="s">&#34;gcTimer&#34;</span><span class="o">)</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onTimer</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">state</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;maxTimestampSeen&#34;</span><span class="o">)</span> <span class="n">CombiningState</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="kt">long</span><span class="o">[],</span> <span class="n">Long</span><span class="o">&gt;</span> <span class="n">maxTimestamp</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Clear all state for the key. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">state</span><span class="o">.</span><span class="na">clear</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">maxTimestamp</span><span class="o">.</span><span class="na">clear</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">UserDoFn</span><span class="p">(</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ALL_ELEMENTS</span> <span class="o">=</span> <span class="n">BagStateSpec</span><span class="p">(</span><span class="s1">&#39;state&#39;</span><span class="p">,</span> <span class="n">coders</span><span class="o">.</span><span class="n">VarIntCoder</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MAX_TIMESTAMP</span> <span class="o">=</span> <span class="n">CombiningValueStateSpec</span><span class="p">(</span><span class="s1">&#39;max_timestamp_seen&#39;</span><span class="p">,</span> <span class="nb">max</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TIMER</span> <span class="o">=</span> <span class="n">TimerSpec</span><span class="p">(</span><span class="s1">&#39;gc-timer&#39;</span><span class="p">,</span> <span class="n">TimeDomain</span><span class="o">.</span><span class="n">WATERMARK</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">element</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">t</span> <span class="o">=</span> <span class="n">DoFn</span><span class="o">.</span><span class="n">TimestampParam</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state</span> <span class="o">=</span> <span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">ALL_ELEMENTS</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">max_timestamp</span> <span class="o">=</span> <span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">MAX_TIMESTAMP</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timer</span> <span class="o">=</span> <span class="n">DoFn</span><span class="o">.</span><span class="n">TimerParam</span><span class="p">(</span><span class="n">TIMER</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">update_state</span><span class="p">(</span><span class="n">state</span><span class="p">,</span> <span class="n">element</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">max_timestamp</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">t</span><span class="o">.</span><span class="n">micros</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Set the timer to be one hour after the maximum timestamp seen. This will keep overwriting the same timer, so</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># as long as there is activity on this key the state will stay active. Once the key goes inactive for one hour&#39;s</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># worth of event time (as measured by the watermark), then the gc timer will fire.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">expiration_time</span> <span class="o">=</span> <span class="n">Timestamp</span><span class="p">(</span><span class="n">micros</span><span class="o">=</span><span class="n">max_timestamp</span><span class="o">.</span><span class="n">read</span><span class="p">())</span> <span class="o">+</span> <span class="n">Duration</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="mi">60</span><span class="o">*</span><span class="mi">60</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timer</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">expiration_time</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@on_timer</span><span class="p">(</span><span class="n">TIMER</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">expiry_callback</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state</span> <span class="o">=</span> <span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">ALL_ELEMENTS</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">max_timestamp</span> <span class="o">=</span> <span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">MAX_TIMESTAMP</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">max_timestamp</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">_</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span> <span class="o">|</span> <span class="s1">&#39;Read per user&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ReadPerUser</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;User DoFn&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">UserDoFn</span><span class="p">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">timerGarbageCollectionFn</span><span class="p">[</span><span class="nx">V</span> <span class="nx">any</span><span class="p">]</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">State</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Value</span><span class="p">[</span><span class="nx">V</span><span class="p">]</span> <span class="c1">// The state for the key. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">MaxTimestampInBag</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Combining</span><span class="p">[</span><span class="kt">int64</span><span class="p">,</span> <span class="kt">int64</span><span class="p">,</span> <span class="kt">int64</span><span class="p">]</span> <span class="c1">// The maximum element timestamp seen so far. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">GcTimer</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">EventTime</span> <span class="c1">// The timestamp of the timer. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">timerGarbageCollectionFn</span><span class="p">[</span><span class="nx">V</span><span class="p">])</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">et</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span><span class="p">,</span> <span class="nx">sp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">key</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">value</span> <span class="nx">V</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span><span class="p">,</span> <span class="kt">string</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">updateState</span><span class="p">(</span><span class="nx">sp</span><span class="p">,</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">State</span><span class="p">,</span> <span class="nx">key</span><span class="p">,</span> <span class="nx">value</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">MaxTimestampInBag</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="nx">sp</span><span class="p">,</span> <span class="nx">et</span><span class="p">.</span><span class="nf">Milliseconds</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Set the timer to be one hour after the maximum timestamp seen. This will keep overwriting the same timer, so |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// as long as there is activity on this key the state will stay active. Once the key goes inactive for one hour&#39;s |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// worth of event time (as measured by the watermark), then the gc timer will fire. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">maxTs</span><span class="p">,</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">_</span> <span class="o">:=</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">MaxTimestampInBag</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">expirationTime</span> <span class="o">:=</span> <span class="nx">time</span><span class="p">.</span><span class="nf">UnixMilli</span><span class="p">(</span><span class="nx">maxTs</span><span class="p">).</span><span class="nf">Add</span><span class="p">(</span><span class="mi">1</span> <span class="o">*</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Hour</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">GcTimer</span><span class="p">.</span><span class="nf">Set</span><span class="p">(</span><span class="nx">tp</span><span class="p">,</span> <span class="nx">expirationTime</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">timerGarbageCollectionFn</span><span class="p">[</span><span class="nx">V</span><span class="p">])</span> <span class="nf">OnTimer</span><span class="p">(</span><span class="nx">sp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">w</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Window</span><span class="p">,</span> <span class="nx">key</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">timer</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span><span class="p">,</span> <span class="kt">string</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">switch</span> <span class="nx">timer</span><span class="p">.</span><span class="nx">Family</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">case</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">GcTimer</span><span class="p">.</span><span class="nx">Family</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Clear all the state for the key |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">fn</span><span class="p">.</span><span class="nx">State</span><span class="p">.</span><span class="nf">Clear</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">MaxTimestampInBag</span><span class="p">.</span><span class="nf">Clear</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nx">AddTimerGarbageCollection</span><span class="p">[</span><span class="nx">V</span> <span class="nx">any</span><span class="p">](</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">in</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span><span class="p">)</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">timerGarbageCollectionFn</span><span class="p">[</span><span class="nx">V</span><span class="p">]{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">State</span><span class="p">:</span> <span class="nx">state</span><span class="p">.</span><span class="nx">MakeValueState</span><span class="p">[</span><span class="nx">V</span><span class="p">](</span><span class="s">&#34;timerTimestamp&#34;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">MaxTimestampInBag</span><span class="p">:</span> <span class="nx">state</span><span class="p">.</span><span class="nx">MakeCombiningState</span><span class="p">[</span><span class="kt">int64</span><span class="p">,</span> <span class="kt">int64</span><span class="p">,</span> <span class="kt">int64</span><span class="p">](</span><span class="s">&#34;maxTimestampInBag&#34;</span><span class="p">,</span> <span class="kd">func</span><span class="p">(</span><span class="nx">a</span><span class="p">,</span> <span class="nx">b</span> <span class="kt">int64</span><span class="p">)</span> <span class="kt">int64</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">a</span> <span class="p">&gt;</span> <span class="nx">b</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">a</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">b</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">GcTimer</span><span class="p">:</span> <span class="nx">timers</span><span class="p">.</span><span class="nf">InEventTime</span><span class="p">(</span><span class="s">&#34;gcTimer&#34;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="nx">in</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="state-timers-examples">11.5. State and timers examples</h3> |
| <p>Following are some example uses of state and timers</p> |
| <h4 id="joining-clicks-and-views">11.5.1. Joining clicks and views</h4> |
| <p>In this example, the pipeline is processing data from an e-commerce site&rsquo;s home page. There are two input streams: |
| a stream of views, representing suggested product links displayed to the user on the home page, and a stream of |
| clicks, representing actual user clicks on these links. The goal of the pipeline is to join click events with view |
| events, outputting a new joined event that contains information from both events. Each link has a unique identifier |
| that is present in both the view event and the join event.</p> |
| <p>Many view events will never be followed up with clicks. This pipeline will wait one hour for a click, after which it |
| will give up on this join. While every click event should have a view event, some small number of view events may be |
| lost and never make it to the Beam pipeline; the pipeline will similarly wait one hour after seeing a click event, and |
| give up if the view event does not arrive in that time. Input events are not ordered - it is possible to see the click |
| event before the view event. The one hour join timeout should be based on event time, not on processing time.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Read the event stream and key it by the link id. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Event</span><span class="o">&gt;&gt;</span> <span class="n">eventsPerLinkId</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">readEvents</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">WithKeys</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">Event</span><span class="o">::</span><span class="n">getLinkId</span><span class="o">).</span><span class="na">withKeyType</span><span class="o">(</span><span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">()));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">eventsPerLinkId</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Event</span><span class="o">&gt;,</span> <span class="n">JoinedEvent</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Store the view event. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;view&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Event</span><span class="o">&gt;&gt;</span> <span class="n">viewState</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Store the click event. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;click&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Event</span><span class="o">&gt;&gt;</span> <span class="n">clickState</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The maximum element timestamp seen so far. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;maxTimestampSeen&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">CombiningState</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="kt">long</span><span class="o">[],</span> <span class="n">Long</span><span class="o">&gt;&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">maxTimestamp</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">combining</span><span class="o">(</span><span class="n">Max</span><span class="o">.</span><span class="na">ofLongs</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Timer that fires when an hour goes by with an incomplete join. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@TimerId</span><span class="o">(</span><span class="s">&#34;gcTimer&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">TimerSpec</span> <span class="n">gcTimer</span> <span class="o">=</span> <span class="n">TimerSpecs</span><span class="o">.</span><span class="na">timer</span><span class="o">(</span><span class="n">TimeDomain</span><span class="o">.</span><span class="na">EVENT_TIME</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Element</span> <span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Event</span><span class="o">&gt;</span> <span class="n">element</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Timestamp</span> <span class="n">Instant</span> <span class="n">ts</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@AlwaysFetched</span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;view&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Event</span><span class="o">&gt;</span> <span class="n">viewState</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@AlwaysFetched</span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;click&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Event</span><span class="o">&gt;</span> <span class="n">clickState</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@AlwaysFetched</span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;maxTimestampSeen&#34;</span><span class="o">)</span> <span class="n">CombiningState</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="kt">long</span><span class="o">[],</span> <span class="n">Long</span><span class="o">&gt;</span> <span class="n">maxTimestampState</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@TimerId</span><span class="o">(</span><span class="s">&#34;gcTimer&#34;</span><span class="o">)</span> <span class="n">gcTimer</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">OutputReceiver</span><span class="o">&lt;</span><span class="n">JoinedEvent</span><span class="o">&gt;</span> <span class="n">output</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Store the event into the correct state variable. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">Event</span> <span class="n">event</span> <span class="o">=</span> <span class="n">element</span><span class="o">.</span><span class="na">getValue</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Event</span><span class="o">&gt;</span> <span class="n">valueState</span> <span class="o">=</span> <span class="n">event</span><span class="o">.</span><span class="na">getType</span><span class="o">().</span><span class="na">equals</span><span class="o">(</span><span class="n">VIEW</span><span class="o">)</span> <span class="o">?</span> <span class="n">viewState</span> <span class="o">:</span> <span class="n">clickState</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">valueState</span><span class="o">.</span><span class="na">write</span><span class="o">(</span><span class="n">event</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">Event</span> <span class="n">view</span> <span class="o">=</span> <span class="n">viewState</span><span class="o">.</span><span class="na">read</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Event</span> <span class="n">click</span> <span class="o">=</span> <span class="n">clickState</span><span class="o">.</span><span class="na">read</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">(</span><span class="k">if</span> <span class="n">view</span> <span class="o">!=</span> <span class="kc">null</span> <span class="o">&amp;&amp;</span> <span class="n">click</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// We&#39;ve seen both a view and a click. Output a joined event and clear state. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">output</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">JoinedEvent</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">view</span><span class="o">,</span> <span class="n">click</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">clearState</span><span class="o">(</span><span class="n">viewState</span><span class="o">,</span> <span class="n">clickState</span><span class="o">,</span> <span class="n">maxTimestampState</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> <span class="k">else</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// We&#39;ve only seen on half of the join. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// Set the timer to be one hour after the maximum timestamp seen. This will keep overwriting the same timer, so |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// as long as there is activity on this key the state will stay active. Once the key goes inactive for one hour&#39;s |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// worth of event time (as measured by the watermark), then the gc timer will fire. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">maxTimestampState</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">ts</span><span class="o">.</span><span class="na">getMillis</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Instant</span> <span class="n">expirationTime</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="n">maxTimestampState</span><span class="o">.</span><span class="na">read</span><span class="o">()).</span><span class="na">plus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardHours</span><span class="o">(</span><span class="n">1</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">gcTimer</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">expirationTime</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@OnTimer</span><span class="o">(</span><span class="s">&#34;gcTimer&#34;</span><span class="o">)</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onTimer</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;view&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Event</span><span class="o">&gt;</span> <span class="n">viewState</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;click&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Event</span><span class="o">&gt;</span> <span class="n">clickState</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;maxTimestampSeen&#34;</span><span class="o">)</span> <span class="n">CombiningState</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="kt">long</span><span class="o">[],</span> <span class="n">Long</span><span class="o">&gt;</span> <span class="n">maxTimestampState</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// An hour has gone by with an incomplete join. Give up and clear the state. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">clearState</span><span class="o">(</span><span class="n">viewState</span><span class="o">,</span> <span class="n">clickState</span><span class="o">,</span> <span class="n">maxTimestampState</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="kd">private</span> <span class="kt">void</span> <span class="nf">clearState</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;view&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Event</span><span class="o">&gt;</span> <span class="n">viewState</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;click&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Event</span><span class="o">&gt;</span> <span class="n">clickState</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;maxTimestampSeen&#34;</span><span class="o">)</span> <span class="n">CombiningState</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">,</span> <span class="kt">long</span><span class="o">[],</span> <span class="n">Long</span><span class="o">&gt;</span> <span class="n">maxTimestampState</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">viewState</span><span class="o">.</span><span class="na">clear</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">clickState</span><span class="o">.</span><span class="na">clear</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">maxTimestampState</span><span class="o">.</span><span class="na">clear</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">JoinDoFn</span><span class="p">(</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># stores the view event.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">VIEW_STATE_SPEC</span> <span class="o">=</span> <span class="n">ReadModifyWriteStateSpec</span><span class="p">(</span><span class="s1">&#39;view&#39;</span><span class="p">,</span> <span class="n">EventCoder</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># stores the click event.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CLICK_STATE_SPEC</span> <span class="o">=</span> <span class="n">ReadModifyWriteStateSpec</span><span class="p">(</span><span class="s1">&#39;click&#39;</span><span class="p">,</span> <span class="n">EventCoder</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># The maximum element timestamp value seen so far.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MAX_TIMESTAMP</span> <span class="o">=</span> <span class="n">CombiningValueStateSpec</span><span class="p">(</span><span class="s1">&#39;max_timestamp_seen&#39;</span><span class="p">,</span> <span class="nb">max</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Timer that fires when an hour goes by with an incomplete join.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">GC_TIMER</span> <span class="o">=</span> <span class="n">TimerSpec</span><span class="p">(</span><span class="s1">&#39;gc&#39;</span><span class="p">,</span> <span class="n">TimeDomain</span><span class="o">.</span><span class="n">WATERMARK</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">element</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">view</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">VIEW_STATE_SPEC</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">click</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">CLICK_STATE_SPEC</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">max_timestamp_seen</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">MAX_TIMESTAMP</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ts</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">TimestampParam</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">gc</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">TimerParam</span><span class="p">(</span><span class="n">GC_TIMER</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">event</span> <span class="o">=</span> <span class="n">element</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">event</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="s1">&#39;view&#39;</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">view</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">event</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">else</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">click</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">event</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">previous_view</span> <span class="o">=</span> <span class="n">view</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">previous_click</span> <span class="o">=</span> <span class="n">click</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># We&#39;ve seen both a view and a click. Output a joined event and clear state.</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">previous_view</span> <span class="ow">and</span> <span class="n">previous_click</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="p">(</span><span class="n">previous_view</span><span class="p">,</span> <span class="n">previous_click</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">view</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">click</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">max_timestamp_seen</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">else</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">max_timestamp_seen</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">ts</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">gc</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">max_timestamp_seen</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> <span class="o">+</span> <span class="n">Duration</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="mi">3600</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@on_timer</span><span class="p">(</span><span class="n">GC_TIMER</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">gc_callback</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">view</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">VIEW_STATE_SPEC</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">click</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">CLICK_STATE_SPEC</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">max_timestamp_seen</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">MAX_TIMESTAMP</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">view</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">click</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">max_timestamp_seen</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">_</span> <span class="o">=</span> <span class="p">(</span><span class="n">p</span> <span class="o">|</span> <span class="s1">&#39;EventsPerLinkId&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ReadPerLinkEvents</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Join DoFn&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">JoinDoFn</span><span class="p">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">JoinedEvent</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">View</span><span class="p">,</span> <span class="nx">Click</span> <span class="o">*</span><span class="nx">Event</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">joinDoFn</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">View</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Value</span><span class="p">[</span><span class="o">*</span><span class="nx">Event</span><span class="p">]</span> <span class="c1">// Store the view event. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">Click</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Value</span><span class="p">[</span><span class="o">*</span><span class="nx">Event</span><span class="p">]</span> <span class="c1">// Store the click event. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">MaxTimestampSeen</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Combining</span><span class="p">[</span><span class="kt">int64</span><span class="p">,</span> <span class="kt">int64</span><span class="p">,</span> <span class="kt">int64</span><span class="p">]</span> <span class="c1">// The maximum element timestamp seen so far. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">GcTimer</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">EventTime</span> <span class="c1">// The timestamp of the timer. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">joinDoFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">et</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span><span class="p">,</span> <span class="nx">sp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">key</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">event</span> <span class="o">*</span><span class="nx">Event</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="nx">JoinedEvent</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">valueState</span> <span class="o">:=</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">View</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">event</span><span class="p">.</span><span class="nf">isClick</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">valueState</span> <span class="p">=</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">Click</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">valueState</span><span class="p">.</span><span class="nf">Write</span><span class="p">(</span><span class="nx">sp</span><span class="p">,</span> <span class="nx">event</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nx">view</span><span class="p">,</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">_</span> <span class="o">:=</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">View</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">click</span><span class="p">,</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">_</span> <span class="o">:=</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">Click</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">view</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="o">&amp;&amp;</span> <span class="nx">click</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emit</span><span class="p">(</span><span class="nx">JoinedEvent</span><span class="p">{</span><span class="nx">View</span><span class="p">:</span> <span class="nx">view</span><span class="p">,</span> <span class="nx">Click</span><span class="p">:</span> <span class="nx">click</span><span class="p">})</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nf">clearState</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">MaxTimestampSeen</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="nx">sp</span><span class="p">,</span> <span class="nx">et</span><span class="p">.</span><span class="nf">Milliseconds</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">expTs</span><span class="p">,</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">_</span> <span class="o">:=</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">MaxTimestampSeen</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">GcTimer</span><span class="p">.</span><span class="nf">Set</span><span class="p">(</span><span class="nx">tp</span><span class="p">,</span> <span class="nx">time</span><span class="p">.</span><span class="nf">UnixMilli</span><span class="p">(</span><span class="nx">expTs</span><span class="p">).</span><span class="nf">Add</span><span class="p">(</span><span class="mi">1</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Hour</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">joinDoFn</span><span class="p">)</span> <span class="nf">OnTimer</span><span class="p">(</span><span class="nx">sp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">w</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Window</span><span class="p">,</span> <span class="nx">key</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">timer</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span><span class="p">,</span> <span class="kt">string</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">switch</span> <span class="nx">timer</span><span class="p">.</span><span class="nx">Family</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">case</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">GcTimer</span><span class="p">.</span><span class="nx">Family</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nf">clearState</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">joinDoFn</span><span class="p">)</span> <span class="nf">clearState</span><span class="p">(</span><span class="nx">sp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">View</span><span class="p">.</span><span class="nf">Clear</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">Click</span><span class="p">.</span><span class="nf">Clear</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">MaxTimestampSeen</span><span class="p">.</span><span class="nf">Clear</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">AddJoinDoFn</span><span class="p">(</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">in</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span><span class="p">)</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">joinDoFn</span><span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">View</span><span class="p">:</span> <span class="nx">state</span><span class="p">.</span><span class="nx">MakeValueState</span><span class="p">[</span><span class="o">*</span><span class="nx">Event</span><span class="p">](</span><span class="s">&#34;view&#34;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Click</span><span class="p">:</span> <span class="nx">state</span><span class="p">.</span><span class="nx">MakeValueState</span><span class="p">[</span><span class="o">*</span><span class="nx">Event</span><span class="p">](</span><span class="s">&#34;click&#34;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">MaxTimestampSeen</span><span class="p">:</span> <span class="nx">state</span><span class="p">.</span><span class="nx">MakeCombiningState</span><span class="p">[</span><span class="kt">int64</span><span class="p">,</span> <span class="kt">int64</span><span class="p">,</span> <span class="kt">int64</span><span class="p">](</span><span class="s">&#34;maxTimestampSeen&#34;</span><span class="p">,</span> <span class="kd">func</span><span class="p">(</span><span class="nx">a</span><span class="p">,</span> <span class="nx">b</span> <span class="kt">int64</span><span class="p">)</span> <span class="kt">int64</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">a</span> <span class="p">&gt;</span> <span class="nx">b</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">a</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">b</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">GcTimer</span><span class="p">:</span> <span class="nx">timers</span><span class="p">.</span><span class="nf">InEventTime</span><span class="p">(</span><span class="s">&#34;gcTimer&#34;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="nx">in</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="batching-rpcs">11.5.2. Batching RPCs</h4> |
| <p>In this example, input elements are being forwarded to an external RPC service. The RPC accepts batch requests - |
| multiple events for the same user can be batched in a single RPC call. Since this RPC service also imposes rate limits, |
| we want to batch ten seconds worth of events together in order to reduce the number of calls.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">perUser</span> <span class="o">=</span> <span class="n">readPerUser</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">perUser</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;,</span> <span class="n">OutputT</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Store the elements buffered so far. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">BagState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;&gt;</span> <span class="n">elements</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">bag</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Keep track of whether a timer is currently set or not. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;isTimerSet&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">StateSpec</span><span class="o">&lt;</span><span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Boolean</span><span class="o">&gt;&gt;</span> <span class="n">isTimerSet</span> <span class="o">=</span> <span class="n">StateSpecs</span><span class="o">.</span><span class="na">value</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The processing-time timer user to publish the RPC. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@TimerId</span><span class="o">(</span><span class="s">&#34;outputState&#34;</span><span class="o">)</span> <span class="kd">private</span> <span class="kd">final</span> <span class="n">TimerSpec</span> <span class="n">timer</span> <span class="o">=</span> <span class="n">TimerSpecs</span><span class="o">.</span><span class="na">timer</span><span class="o">(</span><span class="n">TimeDomain</span><span class="o">.</span><span class="na">PROCESSING_TIME</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Element</span> <span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">element</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state&#34;</span><span class="o">)</span> <span class="n">BagState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">elementsState</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;isTimerSet&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Boolean</span><span class="o">&gt;</span> <span class="n">isTimerSetState</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@TimerId</span><span class="o">(</span><span class="s">&#34;outputState&#34;</span><span class="o">)</span> <span class="n">Timer</span> <span class="n">timer</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Add the current element to the bag for this key. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">state</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">element</span><span class="o">.</span><span class="na">getValue</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="o">(!</span><span class="n">MoreObjects</span><span class="o">.</span><span class="na">firstNonNull</span><span class="o">(</span><span class="n">isTimerSetState</span><span class="o">.</span><span class="na">read</span><span class="o">(),</span> <span class="kc">false</span><span class="o">))</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// If there is no timer currently set, then set one to go off in 10 seconds. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">timer</span><span class="o">.</span><span class="na">offset</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="n">10</span><span class="o">)).</span><span class="na">setRelative</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">isTimerSetState</span><span class="o">.</span><span class="na">write</span><span class="o">(</span><span class="kc">true</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@OnTimer</span><span class="o">(</span><span class="s">&#34;outputState&#34;</span><span class="o">)</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">onTimer</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;state&#34;</span><span class="o">)</span> <span class="n">BagState</span><span class="o">&lt;</span><span class="n">ValueT</span><span class="o">&gt;</span> <span class="n">elementsState</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@StateId</span><span class="o">(</span><span class="s">&#34;isTimerSet&#34;</span><span class="o">)</span> <span class="n">ValueState</span><span class="o">&lt;</span><span class="n">Boolean</span><span class="o">&gt;</span> <span class="n">isTimerSetState</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Send an RPC containing the batched elements and clear state. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">sendRPC</span><span class="o">(</span><span class="n">elementsState</span><span class="o">.</span><span class="na">read</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">elementsState</span><span class="o">.</span><span class="na">clear</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">isTimerSetState</span><span class="o">.</span><span class="na">clear</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">BufferDoFn</span><span class="p">(</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">BUFFER</span> <span class="o">=</span> <span class="n">BagStateSpec</span><span class="p">(</span><span class="s1">&#39;buffer&#39;</span><span class="p">,</span> <span class="n">EventCoder</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">IS_TIMER_SET</span> <span class="o">=</span> <span class="n">ReadModifyWriteStateSpec</span><span class="p">(</span><span class="s1">&#39;is_timer_set&#39;</span><span class="p">,</span> <span class="n">BooleanCoder</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">OUTPUT</span> <span class="o">=</span> <span class="n">TimerSpec</span><span class="p">(</span><span class="s1">&#39;output&#39;</span><span class="p">,</span> <span class="n">TimeDomain</span><span class="o">.</span><span class="n">REAL_TIME</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">BUFFER</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">is_timer_set</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">IS_TIMER_SET</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timer</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">TimerParam</span><span class="p">(</span><span class="n">OUTPUT</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">element</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="ow">not</span> <span class="n">is_timer_set</span><span class="o">.</span><span class="n">read</span><span class="p">():</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timer</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">Timestamp</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">+</span> <span class="n">Duration</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="mi">10</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">is_timer_set</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="kc">True</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@on_timer</span><span class="p">(</span><span class="n">OUTPUT</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">output_callback</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">BUFFER</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">is_timer_set</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">StateParam</span><span class="p">(</span><span class="n">IS_TIMER_SET</span><span class="p">)):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">send_rpc</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">buffer</span><span class="o">.</span><span class="n">read</span><span class="p">()))</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">buffer</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">is_timer_set</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">bufferDoFn</span><span class="p">[</span><span class="nx">V</span> <span class="nx">any</span><span class="p">]</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Elements</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Bag</span><span class="p">[</span><span class="nx">V</span><span class="p">]</span> <span class="c1">// Store the elements buffered so far. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">IsTimerSet</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Value</span><span class="p">[</span><span class="kt">bool</span><span class="p">]</span> <span class="c1">// Keep track of whether a timer is currently set or not. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">OutputElements</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">ProcessingTime</span> <span class="c1">// The processing-time timer user to publish the RPC. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">bufferDoFn</span><span class="p">[</span><span class="nx">V</span><span class="p">])</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">et</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span><span class="p">,</span> <span class="nx">sp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">key</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">value</span> <span class="nx">V</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">Elements</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="nx">sp</span><span class="p">,</span> <span class="nx">value</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nx">isSet</span><span class="p">,</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">_</span> <span class="o">:=</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">IsTimerSet</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="p">!</span><span class="nx">isSet</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">OutputElements</span><span class="p">.</span><span class="nf">Set</span><span class="p">(</span><span class="nx">tp</span><span class="p">,</span> <span class="nx">time</span><span class="p">.</span><span class="nf">Now</span><span class="p">().</span><span class="nf">Add</span><span class="p">(</span><span class="mi">10</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">IsTimerSet</span><span class="p">.</span><span class="nf">Write</span><span class="p">(</span><span class="nx">sp</span><span class="p">,</span> <span class="kc">true</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">bufferDoFn</span><span class="p">[</span><span class="nx">V</span><span class="p">])</span> <span class="nf">OnTimer</span><span class="p">(</span><span class="nx">sp</span> <span class="nx">state</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">tp</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Provider</span><span class="p">,</span> <span class="nx">w</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Window</span><span class="p">,</span> <span class="nx">key</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">timer</span> <span class="nx">timers</span><span class="p">.</span><span class="nx">Context</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">switch</span> <span class="nx">timer</span><span class="p">.</span><span class="nx">Family</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">case</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">OutputElements</span><span class="p">.</span><span class="nx">Family</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">elements</span><span class="p">,</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">_</span> <span class="o">:=</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">Elements</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">sendRpc</span><span class="p">(</span><span class="nx">elements</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">Elements</span><span class="p">.</span><span class="nf">Clear</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">fn</span><span class="p">.</span><span class="nx">IsTimerSet</span><span class="p">.</span><span class="nf">Clear</span><span class="p">(</span><span class="nx">sp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nx">AddBufferDoFn</span><span class="p">[</span><span class="nx">V</span> <span class="nx">any</span><span class="p">](</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">in</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span><span class="p">)</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">bufferDoFn</span><span class="p">[</span><span class="nx">V</span><span class="p">]{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Elements</span><span class="p">:</span> <span class="nx">state</span><span class="p">.</span><span class="nx">MakeBagState</span><span class="p">[</span><span class="nx">V</span><span class="p">](</span><span class="s">&#34;elements&#34;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">IsTimerSet</span><span class="p">:</span> <span class="nx">state</span><span class="p">.</span><span class="nx">MakeValueState</span><span class="p">[</span><span class="kt">bool</span><span class="p">](</span><span class="s">&#34;isTimerSet&#34;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nx">OutputElements</span><span class="p">:</span> <span class="nx">timers</span><span class="p">.</span><span class="nf">InProcessingTime</span><span class="p">(</span><span class="s">&#34;outputElements&#34;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="nx">in</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h2 id="splittable-dofns">12. Splittable <code>DoFns</code></h2> |
| <p>A Splittable <code>DoFn</code> (SDF) enables users to create modular components containing I/Os (and some advanced |
| <a href="https://s.apache.org/splittable-do-fn#heading=h.5cep9s8k4fxv">non I/O use cases</a>). Having modular |
| I/O components that can be connected to each other simplify typical patterns that users want. |
| For example, a popular use case is to read filenames from a message queue followed by parsing those |
| files. Traditionally, users were required to either write a single I/O connector that contained the |
| logic for the message queue and the file reader (increased complexity) or choose to reuse a message |
| queue I/O followed by a regular <code>DoFn</code> that read the file (decreased performance). With SDF, |
| we bring the richness of Apache Beam’s I/O APIs to a <code>DoFn</code> enabling modularity while maintaining the |
| performance of traditional I/O connectors.</p> |
| <h3 id="sdf-basics">12.1. SDF basics</h3> |
| <p>At a high level, an SDF is responsible for processing element and restriction pairs. A |
| restriction represents a subset of work that would have been necessary to have been done when |
| processing the element.</p> |
| <p>Executing an SDF follows the following steps:</p> |
| <ol> |
| <li>Each element is paired with a restriction (e.g. filename is paired with offset range representing the whole file).</li> |
| <li>Each element and restriction pair is split (e.g. offset ranges are broken up into smaller pieces).</li> |
| <li>The runner redistributes the element and restriction pairs to several workers.</li> |
| <li>Element and restriction pairs are processed in parallel (e.g. the file is read). Within this last step, |
| the element and restriction pair can pause its own processing and/or be split into further element and |
| restriction pairs.</li> |
| </ol> |
| <p><img src="/images/sdf_high_level_overview.svg" alt="Diagram of steps that an SDF is composed of"></p> |
| <h4 id="a-basic-sdf">12.1.1. A basic SDF</h4> |
| <p>A basic SDF is composed of three parts: a restriction, a restriction provider, and a |
| restriction tracker. If you want to control the watermark, especially in a streaming |
| pipeline, two more components are needed: a watermark estimator provider and a watermark estimator.</p> |
| <p>The restriction is a user-defined object that is used to represent a subset of |
| work for a given element. For example, we defined <code>OffsetRange</code> as a restriction to represent offset |
| positions in <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/range/OffsetRange.html">Java</a> |
| and <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.io.restriction_trackers.html#apache_beam.io.restriction_trackers.OffsetRange">Python</a>.</p> |
| <p>The restriction provider lets SDF authors override default implementations, including the ones for |
| splitting and sizing. In <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.ProcessElement.html">Java</a> |
| and <a href="https://github.com/apache/beam/blob/0f466e6bcd4ac8677c2bd9ecc8e6af3836b7f3b8/sdks/go/pkg/beam/pardo.go#L226">Go</a>, |
| this is the <code>DoFn</code>. <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.RestrictionProvider">Python</a> |
| has a dedicated <code>RestrictionProvider</code> type.</p> |
| <p>The restriction tracker is responsible for tracking which subset of the restriction has been |
| completed during processing. For APIs details, read the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/splittabledofn/RestrictionTracker.html">Java</a> |
| and <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.io.iobase.html#apache_beam.io.iobase.RestrictionTracker">Python</a> |
| reference documentation.</p> |
| <p>There are some built-in <code>RestrictionTracker</code> implementations defined in Java:</p> |
| <ol> |
| <li><a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/splittabledofn/OffsetRangeTracker.html">OffsetRangeTracker</a></li> |
| <li><a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/splittabledofn/GrowableOffsetRangeTracker.html">GrowableOffsetRangeTracker</a></li> |
| <li><a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/splittabledofn/ByteKeyRangeTracker.html">ByteKeyRangeTracker</a></li> |
| </ol> |
| <p>The SDF also has a built-in <code>RestrictionTracker</code> implementation in Python:</p> |
| <ol> |
| <li><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.io.restriction_trackers.html#apache_beam.io.restriction_trackers.OffsetRestrictionTracker">OffsetRangeTracker</a></li> |
| </ol> |
| <p>Go also has a built-in <code>RestrictionTracker</code> type:</p> |
| <ol> |
| <li><a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/rtrackers/offsetrange">OffsetRangeTracker</a></li> |
| </ol> |
| <p>The watermark state is a user-defined object which is used to create a <code>WatermarkEstimator</code> from a |
| <code>WatermarkEstimatorProvider</code>. The simplest watermark state could be a <code>timestamp</code>.</p> |
| <p>The watermark estimator provider lets SDF authors define how to initialize the watermark state and |
| create a watermark estimator. In <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.ProcessElement.html">Java</a> and <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam#ParDo">Go</a> |
| this is the <code>DoFn</code>. <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.WatermarkEstimatorProvider">Python</a> |
| has a dedicated <code>WatermarkEstimatorProvider</code> type.</p> |
| <p>The watermark estimator tracks the watermark when an element-restriction pair is in progress. |
| For APIs details, read the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/splittabledofn/WatermarkEstimator.html">Java</a>, <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.io.iobase.html#apache_beam.io.iobase.WatermarkEstimator">Python</a>, and <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf#WatermarkEstimator">Go</a> |
| reference documentation.</p> |
| <p>There are some built-in <code>WatermarkEstimator</code> implementations in Java:</p> |
| <ol> |
| <li><a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/splittabledofn/WatermarkEstimators.Manual.html">Manual</a></li> |
| <li><a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/splittabledofn/WatermarkEstimators.MonotonicallyIncreasing.html">MonotonicallyIncreasing</a></li> |
| <li><a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/splittabledofn/WatermarkEstimators.WallTime.html">WallTime</a></li> |
| </ol> |
| <p>Along with the default <code>WatermarkEstimatorProvider</code>, there are the same set of built-in |
| <code>WatermarkEstimator</code> implementations in Python:</p> |
| <ol> |
| <li><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.io.watermark_estimators.html#apache_beam.io.watermark_estimators.ManualWatermarkEstimator">ManualWatermarkEstimator</a></li> |
| <li><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.io.watermark_estimators.html#apache_beam.io.watermark_estimators.MonotonicWatermarkEstimator">MonotonicWatermarkEstimator</a></li> |
| <li><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.io.watermark_estimators.html#apache_beam.io.watermark_estimators.WalltimeWatermarkEstimator">WalltimeWatermarkEstimator</a></li> |
| </ol> |
| <p>The following <code>WatermarkEstimator</code> types are implemented in Go:</p> |
| <ol> |
| <li><a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf#TimestampObservingWatermarkEstimator">TimestampObservingEstimator</a></li> |
| <li><a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/core/sdf#WallTimeWatermarkEstimator">WalltimeWatermarkEstimator</a></li> |
| </ol> |
| <p>To define an SDF, you must choose whether the SDF is bounded (default) or |
| unbounded and define a way to initialize an initial restriction for an element. The distinction is |
| based on how the amount of work is represented:</p> |
| <ul> |
| <li>Bounded DoFns are those where the work represented by an element is well-known beforehand and has |
| an end. Examples of bounded elements include a file or group of files.</li> |
| <li>Unbounded DoFns are those where the amount of work does not have a specific end or the |
| amount of work is not known beforehand. Examples of unbounded elements include a Kafka or a PubSub |
| topic.</li> |
| </ul> |
| <p>In Java, you can use <a href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/DoFn.UnboundedPerElement.html">@UnboundedPerElement</a> |
| or <a href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/DoFn.BoundedPerElement.html">@BoundedPerElement</a> |
| to annotate your <code>DoFn</code>. In Python, you can use <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.unbounded_per_element">@unbounded_per_element</a> |
| to annotate the <code>DoFn</code>.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@BoundedPerElement</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">private</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">FileToWordsFn</span> <span class="kd">extends</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@GetInitialRestriction</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">OffsetRange</span> <span class="nf">getInitialRestriction</span><span class="o">(</span><span class="nd">@Element</span> <span class="n">String</span> <span class="n">fileName</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="k">new</span> <span class="n">OffsetRange</span><span class="o">(</span><span class="n">0</span><span class="o">,</span> <span class="k">new</span> <span class="n">File</span><span class="o">(</span><span class="n">fileName</span><span class="o">).</span><span class="na">length</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Element</span> <span class="n">String</span> <span class="n">fileName</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">RestrictionTracker</span><span class="o">&lt;</span><span class="n">OffsetRange</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;</span> <span class="n">tracker</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">OutputReceiver</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">outputReceiver</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">throws</span> <span class="n">IOException</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">RandomAccessFile</span> <span class="n">file</span> <span class="o">=</span> <span class="k">new</span> <span class="n">RandomAccessFile</span><span class="o">(</span><span class="n">fileName</span><span class="o">,</span> <span class="s">&#34;r&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">seekToNextRecordBoundaryInFile</span><span class="o">(</span><span class="n">file</span><span class="o">,</span> <span class="n">tracker</span><span class="o">.</span><span class="na">currentRestriction</span><span class="o">().</span><span class="na">getFrom</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">while</span> <span class="o">(</span><span class="n">tracker</span><span class="o">.</span><span class="na">tryClaim</span><span class="o">(</span><span class="n">file</span><span class="o">.</span><span class="na">getFilePointer</span><span class="o">()))</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">outputReceiver</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">readNextRecord</span><span class="o">(</span><span class="n">file</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Providing the coder is only necessary if it can not be inferred at runtime. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@GetRestrictionCoder</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">Coder</span><span class="o">&lt;</span><span class="n">OffsetRange</span><span class="o">&gt;</span> <span class="nf">getRestrictionCoder</span><span class="o">()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">OffsetRange</span><span class="o">.</span><span class="na">Coder</span><span class="o">.</span><span class="na">of</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">FileToWordsRestrictionProvider</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">transforms</span><span class="o">.</span><span class="n">core</span><span class="o">.</span><span class="n">RestrictionProvider</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">initial_restriction</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">file_name</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">OffsetRange</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">stat</span><span class="p">(</span><span class="n">file_name</span><span class="p">)</span><span class="o">.</span><span class="n">st_size</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">create_tracker</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">restriction</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">restriction_trackers</span><span class="o">.</span><span class="n">OffsetRestrictionTracker</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">FileToWordsFn</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">file_name</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Alternatively, we can let FileToWordsFn itself inherit from</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># RestrictionProvider, implement the required methods and let</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># tracker=beam.DoFn.RestrictionParam() which will use self as</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># the provider.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">tracker</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">RestrictionParam</span><span class="p">(</span><span class="n">FileToWordsRestrictionProvider</span><span class="p">())):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">file_name</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_handle</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">file_handle</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="n">tracker</span><span class="o">.</span><span class="n">current_restriction</span><span class="o">.</span><span class="n">start</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">while</span> <span class="n">tracker</span><span class="o">.</span><span class="n">try_claim</span><span class="p">(</span><span class="n">file_handle</span><span class="o">.</span><span class="n">tell</span><span class="p">()):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">read_next_record</span><span class="p">(</span><span class="n">file_handle</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Providing the coder is only necessary if it can not be inferred at</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># runtime.</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">restriction_coder</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="o">...</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">splittableDoFn</span><span class="p">)</span> <span class="nf">CreateInitialRestriction</span><span class="p">(</span><span class="nx">filename</span> <span class="kt">string</span><span class="p">)</span> <span class="nx">offsetrange</span><span class="p">.</span><span class="nx">Restriction</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">offsetrange</span><span class="p">.</span><span class="nx">Restriction</span><span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Start</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">End</span><span class="p">:</span> <span class="nf">getFileLength</span><span class="p">(</span><span class="nx">filename</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">splittableDoFn</span><span class="p">)</span> <span class="nf">CreateTracker</span><span class="p">(</span><span class="nx">rest</span> <span class="nx">offsetrange</span><span class="p">.</span><span class="nx">Restriction</span><span class="p">)</span> <span class="o">*</span><span class="nx">sdf</span><span class="p">.</span><span class="nx">LockRTracker</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">sdf</span><span class="p">.</span><span class="nf">NewLockRTracker</span><span class="p">(</span><span class="nx">offsetrange</span><span class="p">.</span><span class="nf">NewTracker</span><span class="p">(</span><span class="nx">rest</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">splittableDoFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">rt</span> <span class="o">*</span><span class="nx">sdf</span><span class="p">.</span><span class="nx">LockRTracker</span><span class="p">,</span> <span class="nx">filename</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="kt">int</span><span class="p">))</span> <span class="kt">error</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">file</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">os</span><span class="p">.</span><span class="nf">Open</span><span class="p">(</span><span class="nx">filename</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">offset</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nf">seekToNextRecordBoundaryInFile</span><span class="p">(</span><span class="nx">file</span><span class="p">,</span> <span class="nx">rt</span><span class="p">.</span><span class="nf">GetRestriction</span><span class="p">().(</span><span class="nx">offsetrange</span><span class="p">.</span><span class="nx">Restriction</span><span class="p">).</span><span class="nx">Start</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="nx">rt</span><span class="p">.</span><span class="nf">TryClaim</span><span class="p">(</span><span class="nx">offset</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">record</span><span class="p">,</span> <span class="nx">newOffset</span> <span class="o">:=</span> <span class="nf">readNextRecord</span><span class="p">(</span><span class="nx">file</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emit</span><span class="p">(</span><span class="nx">record</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">offset</span> <span class="p">=</span> <span class="nx">newOffset</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>At this point, we have an SDF that supports <a href="#runner-initiated-split">runner-initiated splits</a> |
| enabling dynamic work rebalancing. To increase the rate at which initial parallelization of work occurs |
| or for those runners that do not support runner-initiated splitting, we recommend providing |
| a set of initial splits:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kt">void</span> <span class="nf">splitRestriction</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Restriction</span> <span class="n">OffsetRange</span> <span class="n">restriction</span><span class="o">,</span> <span class="n">OutputReceiver</span><span class="o">&lt;</span><span class="n">OffsetRange</span><span class="o">&gt;</span> <span class="n">splitReceiver</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kt">long</span> <span class="n">splitSize</span> <span class="o">=</span> <span class="n">64</span> <span class="o">*</span> <span class="o">(</span><span class="n">1</span> <span class="o">&lt;&lt;</span> <span class="n">20</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="kt">long</span> <span class="n">i</span> <span class="o">=</span> <span class="n">restriction</span><span class="o">.</span><span class="na">getFrom</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">while</span> <span class="o">(</span><span class="n">i</span> <span class="o">&lt;</span> <span class="n">restriction</span><span class="o">.</span><span class="na">getTo</span><span class="o">()</span> <span class="o">-</span> <span class="n">splitSize</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Compute and output 64 MiB size ranges to process in parallel |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kt">long</span> <span class="n">end</span> <span class="o">=</span> <span class="n">i</span> <span class="o">+</span> <span class="n">splitSize</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">splitReceiver</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="k">new</span> <span class="n">OffsetRange</span><span class="o">(</span><span class="n">i</span><span class="o">,</span> <span class="n">end</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">i</span> <span class="o">=</span> <span class="n">end</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Output the last range |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">splitReceiver</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="k">new</span> <span class="n">OffsetRange</span><span class="o">(</span><span class="n">i</span><span class="o">,</span> <span class="n">restriction</span><span class="o">.</span><span class="na">getTo</span><span class="o">()));</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">FileToWordsRestrictionProvider</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">transforms</span><span class="o">.</span><span class="n">core</span><span class="o">.</span><span class="n">RestrictionProvider</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">split</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">file_name</span><span class="p">,</span> <span class="n">restriction</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Compute and output 64 MiB size ranges to process in parallel</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">split_size</span> <span class="o">=</span> <span class="mi">64</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">&lt;&lt;</span> <span class="mi">20</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">i</span> <span class="o">=</span> <span class="n">restriction</span><span class="o">.</span><span class="n">start</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">while</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">restriction</span><span class="o">.</span><span class="n">end</span> <span class="o">-</span> <span class="n">split_size</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">OffsetRange</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">i</span> <span class="o">+</span> <span class="n">split_size</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">i</span> <span class="o">+=</span> <span class="n">split_size</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">OffsetRange</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">restriction</span><span class="o">.</span><span class="n">end</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">splittableDoFn</span><span class="p">)</span> <span class="nf">SplitRestriction</span><span class="p">(</span><span class="nx">filename</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">rest</span> <span class="nx">offsetrange</span><span class="p">.</span><span class="nx">Restriction</span><span class="p">)</span> <span class="p">(</span><span class="nx">splits</span> <span class="p">[]</span><span class="nx">offsetrange</span><span class="p">.</span><span class="nx">Restriction</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">size</span> <span class="o">:=</span> <span class="mi">64</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">&lt;&lt;</span> <span class="mi">20</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">i</span> <span class="o">:=</span> <span class="nx">rest</span><span class="p">.</span><span class="nx">Start</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="nx">i</span> <span class="p">&lt;</span> <span class="nx">rest</span><span class="p">.</span><span class="nx">End</span> <span class="o">-</span> <span class="nx">size</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Compute and output 64 MiB size ranges to process in parallel |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">end</span> <span class="o">:=</span> <span class="nx">i</span> <span class="o">+</span> <span class="nx">size</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">splits</span> <span class="p">=</span> <span class="nb">append</span><span class="p">(</span><span class="nx">splits</span><span class="p">,</span> <span class="nx">offsetrange</span><span class="p">.</span><span class="nx">Restriction</span><span class="p">{</span><span class="nx">i</span><span class="p">,</span> <span class="nx">end</span><span class="p">})</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">i</span> <span class="p">=</span> <span class="nx">end</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Output the last range |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">splits</span> <span class="p">=</span> <span class="nb">append</span><span class="p">(</span><span class="nx">splits</span><span class="p">,</span> <span class="nx">offsetrange</span><span class="p">.</span><span class="nx">Restriction</span><span class="p">{</span><span class="nx">i</span><span class="p">,</span> <span class="nx">rest</span><span class="p">.</span><span class="nx">End</span><span class="p">})</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">splits</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="sizing-and-progress">12.2. Sizing and progress</h3> |
| <p>Sizing and progress are used during execution of an SDF to inform runners so that they may |
| perform intelligent decisions about which restrictions to split and how to parallelize work.</p> |
| <p>Before processing an element and restriction, an initial size may be used by a runner to choose |
| how and who processes the restrictions attempting to improve initial balancing and parallelization |
| of work. During the processing of an element and restriction, sizing and progress are used to choose |
| which restrictions to split and who should process them.</p> |
| <p>By default, we use the restriction tracker’s estimate for work remaining falling back to assuming |
| that all restrictions have an equal cost. To override the default, SDF authors can provide the |
| appropriate method within the restriction provider. SDF authors need to be aware that the |
| sizing method will be invoked concurrently during bundle processing due to runner initiated splitting |
| and progress estimation.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@GetSize</span> |
| </span></span><span class="line"><span class="cl"><span class="kt">double</span> <span class="nf">getSize</span><span class="o">(</span><span class="nd">@Element</span> <span class="n">String</span> <span class="n">fileName</span><span class="o">,</span> <span class="nd">@Restriction</span> <span class="n">OffsetRange</span> <span class="n">restriction</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="o">(</span><span class="n">fileName</span><span class="o">.</span><span class="na">contains</span><span class="o">(</span><span class="s">&#34;expensiveRecords&#34;</span><span class="o">)</span> <span class="o">?</span> <span class="n">2</span> <span class="o">:</span> <span class="n">1</span><span class="o">)</span> <span class="o">*</span> <span class="n">restriction</span><span class="o">.</span><span class="na">getTo</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">-</span> <span class="n">restriction</span><span class="o">.</span><span class="na">getFrom</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># The RestrictionProvider is responsible for calculating the size of given</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># restriction.</span> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">MyRestrictionProvider</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">transforms</span><span class="o">.</span><span class="n">core</span><span class="o">.</span><span class="n">RestrictionProvider</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">restriction_size</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">file_name</span><span class="p">,</span> <span class="n">restriction</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">weight</span> <span class="o">=</span> <span class="mi">2</span> <span class="k">if</span> <span class="s2">&#34;expensiveRecords&#34;</span> <span class="ow">in</span> <span class="n">file_name</span> <span class="k">else</span> <span class="mi">1</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">restriction</span><span class="o">.</span><span class="n">size</span><span class="p">()</span> <span class="o">*</span> <span class="n">weight</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">splittableDoFn</span><span class="p">)</span> <span class="nf">RestrictionSize</span><span class="p">(</span><span class="nx">filename</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">rest</span> <span class="nx">offsetrange</span><span class="p">.</span><span class="nx">Restriction</span><span class="p">)</span> <span class="kt">float64</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">weight</span> <span class="o">:=</span> <span class="nb">float64</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">strings</span><span class="p">.</span><span class="nf">Contains</span><span class="p">(</span><span class="nx">filename</span><span class="p">,</span> <span class="err">“</span><span class="nx">expensiveRecords</span><span class="err">”</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">weight</span> <span class="p">=</span> <span class="mi">2</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">weight</span> <span class="o">*</span> <span class="p">(</span><span class="nx">rest</span><span class="p">.</span><span class="nx">End</span> <span class="o">-</span> <span class="nx">rest</span><span class="p">.</span><span class="nx">Start</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="user-initiated-checkpoint">12.3. User-initiated checkpoint</h3> |
| <p>Some I/Os cannot produce all of the data necessary to complete a restriction within the lifetime of a |
| single bundle. This typically happens with unbounded restrictions, but can also happen with bounded |
| restrictions. For example, there could be more data that needs to be ingested but is not available yet. |
| Another cause of this scenario is the source system throttling your data.</p> |
| <p>Your SDF can signal to you that you are not done processing the current restriction. This |
| signal can suggest a time to resume at. While the runner tries to honor the resume time, this is not |
| guaranteed. This allows execution to continue on a restriction that has available work improving |
| resource utilization.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="n">ProcessContinuation</span> <span class="nf">processElement</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">RestrictionTracker</span><span class="o">&lt;</span><span class="n">OffsetRange</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;</span> <span class="n">tracker</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">OutputReceiver</span><span class="o">&lt;</span><span class="n">RecordPosition</span><span class="o">&gt;</span> <span class="n">outputReceiver</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kt">long</span> <span class="n">currentPosition</span> <span class="o">=</span> <span class="n">tracker</span><span class="o">.</span><span class="na">currentRestriction</span><span class="o">().</span><span class="na">getFrom</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Service</span> <span class="n">service</span> <span class="o">=</span> <span class="n">initializeService</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">try</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">while</span> <span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">List</span><span class="o">&lt;</span><span class="n">RecordPosition</span><span class="o">&gt;</span> <span class="n">records</span> <span class="o">=</span> <span class="n">service</span><span class="o">.</span><span class="na">readNextRecords</span><span class="o">(</span><span class="n">currentPosition</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="o">(</span><span class="n">records</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">())</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Return a short delay if there is no data to process at the moment. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="n">ProcessContinuation</span><span class="o">.</span><span class="na">resume</span><span class="o">().</span><span class="na">withResumeDelay</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="n">10</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="o">(</span><span class="n">RecordPosition</span> <span class="n">record</span> <span class="o">:</span> <span class="n">records</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="o">(!</span><span class="n">tracker</span><span class="o">.</span><span class="na">tryClaim</span><span class="o">(</span><span class="n">record</span><span class="o">.</span><span class="na">getPosition</span><span class="o">()))</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">ProcessContinuation</span><span class="o">.</span><span class="na">stop</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">currentPosition</span> <span class="o">=</span> <span class="n">record</span><span class="o">.</span><span class="na">getPosition</span><span class="o">()</span> <span class="o">+</span> <span class="n">1</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">outputReceiver</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">record</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">ThrottlingException</span> <span class="n">exception</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Return a longer delay in case we are being throttled. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="n">ProcessContinuation</span><span class="o">.</span><span class="na">resume</span><span class="o">().</span><span class="na">withResumeDelay</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="n">60</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">MySplittableDoFn</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">element</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">restriction_tracker</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">RestrictionParam</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MyRestrictionProvider</span><span class="p">())):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">current_position</span> <span class="o">=</span> <span class="n">restriction_tracker</span><span class="o">.</span><span class="n">current_restriction</span><span class="o">.</span><span class="n">start</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">while</span> <span class="kc">True</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Pull records from an external service.</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">try</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">records</span> <span class="o">=</span> <span class="n">external_service</span><span class="o">.</span><span class="n">fetch</span><span class="p">(</span><span class="n">current_position</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">records</span><span class="o">.</span><span class="n">empty</span><span class="p">():</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Set a shorter delay in case we are being throttled.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">restriction_tracker</span><span class="o">.</span><span class="n">defer_remainder</span><span class="p">(</span><span class="n">timestamp</span><span class="o">.</span><span class="n">Duration</span><span class="p">(</span><span class="n">second</span><span class="o">=</span><span class="mi">10</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="n">record</span> <span class="ow">in</span> <span class="n">records</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">restriction_tracker</span><span class="o">.</span><span class="n">try_claim</span><span class="p">(</span><span class="n">record</span><span class="o">.</span><span class="n">position</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">current_position</span> <span class="o">=</span> <span class="n">record</span><span class="o">.</span><span class="n">position</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">record</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">else</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">except</span> <span class="ne">TimeoutError</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Set a longer delay in case we are being throttled.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">restriction_tracker</span><span class="o">.</span><span class="n">defer_remainder</span><span class="p">(</span><span class="n">timestamp</span><span class="o">.</span><span class="n">Duration</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="mi">60</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">checkpointingSplittableDoFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">rt</span> <span class="o">*</span><span class="nx">sdf</span><span class="p">.</span><span class="nx">LockRTracker</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="nx">Record</span><span class="p">))</span> <span class="p">(</span><span class="nx">sdf</span><span class="p">.</span><span class="nx">ProcessContinuation</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">position</span> <span class="o">:=</span> <span class="nx">rt</span><span class="p">.</span><span class="nf">GetRestriction</span><span class="p">().(</span><span class="nx">offsetrange</span><span class="p">.</span><span class="nx">Restriction</span><span class="p">).</span><span class="nx">Start</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">records</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">ExternalService</span><span class="p">.</span><span class="nf">readNextRecords</span><span class="p">(</span><span class="nx">position</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">err</span> <span class="o">==</span> <span class="nx">fn</span><span class="p">.</span><span class="nx">ExternalService</span><span class="p">.</span><span class="nx">ThrottlingErr</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Resume at a later time to avoid throttling. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="nx">sdf</span><span class="p">.</span><span class="nf">ResumeProcessingIn</span><span class="p">(</span><span class="mi">60</span> <span class="o">*</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">),</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">sdf</span><span class="p">.</span><span class="nf">StopProcessing</span><span class="p">(),</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="nx">records</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Wait for data to be available. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="nx">sdf</span><span class="p">.</span><span class="nf">ResumeProcessingIn</span><span class="p">(</span><span class="mi">10</span> <span class="o">*</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">),</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">record</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">records</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="p">!</span><span class="nx">rt</span><span class="p">.</span><span class="nf">TryClaim</span><span class="p">(</span><span class="nx">position</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Records have been claimed, finish processing. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="nx">sdf</span><span class="p">.</span><span class="nf">StopProcessing</span><span class="p">(),</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">position</span> <span class="o">+=</span> <span class="mi">1</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emit</span><span class="p">(</span><span class="nx">record</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="runner-initiated-split">12.4. Runner-initiated split</h3> |
| <p>A runner at any time may attempt to split a restriction while it is being processed. This allows the |
| runner to either pause processing of the restriction so that other work may be done (common for |
| unbounded restrictions to limit the amount of output and/or improve latency) or split the restriction |
| into two pieces, increasing the available parallelism within the system. Different runners (e.g., |
| Dataflow, Flink, Spark) have different strategies to issue splits under batch and streaming |
| execution.</p> |
| <p>Author an SDF with this in mind since the end of the restriction may change. When writing the |
| processing loop, use the result from trying to claim a piece of the restriction instead of assuming |
| you can process until the end.</p> |
| <p>One incorrect example could be:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kt">void</span> <span class="nf">badTryClaimLoop</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Element</span> <span class="n">String</span> <span class="n">fileName</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">RestrictionTracker</span><span class="o">&lt;</span><span class="n">OffsetRange</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;</span> <span class="n">tracker</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">OutputReceiver</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">outputReceiver</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">throws</span> <span class="n">IOException</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">RandomAccessFile</span> <span class="n">file</span> <span class="o">=</span> <span class="k">new</span> <span class="n">RandomAccessFile</span><span class="o">(</span><span class="n">fileName</span><span class="o">,</span> <span class="s">&#34;r&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">seekToNextRecordBoundaryInFile</span><span class="o">(</span><span class="n">file</span><span class="o">,</span> <span class="n">tracker</span><span class="o">.</span><span class="na">currentRestriction</span><span class="o">().</span><span class="na">getFrom</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The restriction tracker can be modified by another thread in parallel |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// so storing state locally is ill advised. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kt">long</span> <span class="n">end</span> <span class="o">=</span> <span class="n">tracker</span><span class="o">.</span><span class="na">currentRestriction</span><span class="o">().</span><span class="na">getTo</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">while</span> <span class="o">(</span><span class="n">file</span><span class="o">.</span><span class="na">getFilePointer</span><span class="o">()</span> <span class="o">&lt;</span> <span class="n">end</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Only after successfully claiming should we produce any output and/or |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// perform side effects. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">tracker</span><span class="o">.</span><span class="na">tryClaim</span><span class="o">(</span><span class="n">file</span><span class="o">.</span><span class="na">getFilePointer</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">outputReceiver</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">readNextRecord</span><span class="o">(</span><span class="n">file</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">BadTryClaimLoop</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">file_name</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">tracker</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">RestrictionParam</span><span class="p">(</span><span class="n">FileToWordsRestrictionProvider</span><span class="p">())):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">file_name</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_handle</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">file_handle</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="n">tracker</span><span class="o">.</span><span class="n">current_restriction</span><span class="o">.</span><span class="n">start</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># The restriction tracker can be modified by another thread in parallel</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># so storing state locally is ill advised.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">end</span> <span class="o">=</span> <span class="n">tracker</span><span class="o">.</span><span class="n">current_restriction</span><span class="o">.</span><span class="n">end</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">while</span> <span class="n">file_handle</span><span class="o">.</span><span class="n">tell</span><span class="p">()</span> <span class="o">&lt;</span> <span class="n">end</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Only after successfully claiming should we produce any output and/or</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># perform side effects.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">tracker</span><span class="o">.</span><span class="n">try_claim</span><span class="p">(</span><span class="n">file_handle</span><span class="o">.</span><span class="n">tell</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">read_next_record</span><span class="p">(</span><span class="n">file_handle</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">badTryClaimLoop</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">rt</span> <span class="o">*</span><span class="nx">sdf</span><span class="p">.</span><span class="nx">LockRTracker</span><span class="p">,</span> <span class="nx">filename</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="kt">int</span><span class="p">))</span> <span class="kt">error</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">file</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">os</span><span class="p">.</span><span class="nf">Open</span><span class="p">(</span><span class="nx">filename</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">offset</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nf">seekToNextRecordBoundaryInFile</span><span class="p">(</span><span class="nx">file</span><span class="p">,</span> <span class="nx">rt</span><span class="p">.</span><span class="nf">GetRestriction</span><span class="p">().(</span><span class="nx">offsetrange</span><span class="p">.</span><span class="nx">Restriction</span><span class="p">).</span><span class="nx">Start</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">err</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The restriction tracker can be modified by another thread in parallel |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// so storing state locally is ill advised. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">end</span> <span class="p">=</span> <span class="nx">rt</span><span class="p">.</span><span class="nf">GetRestriction</span><span class="p">().(</span><span class="nx">offsetrange</span><span class="p">.</span><span class="nx">Restriction</span><span class="p">).</span><span class="nx">End</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="nx">offset</span> <span class="p">&lt;</span> <span class="nx">end</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Only after successfully claiming should we produce any output and/or |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// perform side effects. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">rt</span><span class="p">.</span><span class="nf">TryClaim</span><span class="p">(</span><span class="nx">offset</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">record</span><span class="p">,</span> <span class="nx">newOffset</span> <span class="o">:=</span> <span class="nf">readNextRecord</span><span class="p">(</span><span class="nx">file</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emit</span><span class="p">(</span><span class="nx">record</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">offset</span> <span class="p">=</span> <span class="nx">newOffset</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="watermark-estimation">12.5. Watermark estimation</h3> |
| <p>The default watermark estimator does not produce a watermark estimate. Therefore, the output watermark |
| is solely computed by the minimum of upstream watermarks.</p> |
| <p>An SDF can advance the output watermark by specifying a lower bound for all future output |
| that this element and restriction pair will produce. The runner computes the minimum output watermark |
| by taking the minimum over all upstream watermarks and the minimum reported by each element and |
| restriction pair. The reported watermark must monotonically increase for each element and restriction |
| pair across bundle boundaries. When an element and restriction pair stops processing its watermark, |
| it is no longer considered part of the above calculation.</p> |
| <p>Tips:</p> |
| <ul> |
| <li>If you author an SDF that outputs records with timestamps, you should expose ways to allow users of |
| this SDF to configure which watermark estimator to use.</li> |
| <li>Any data produced before the watermark may be considered late. See |
| <a href="#watermarks-and-late-data">watermarks and late data</a> for more details.</li> |
| </ul> |
| <h4 id="controlling-the-watermark">12.5.1. Controlling the watermark</h4> |
| <p>There are two general types of watermark estimators: timestamp observing and external clock observing. |
| Timestamp observing watermark estimators use the output timestamp of each record to compute the watermark |
| estimate while external clock observing watermark estimators control the watermark by using a clock that |
| is not associated to any individual output, such as the local clock of the machine or a clock exposed |
| through an external service.</p> |
| <p>The watermark estimator provider lets you override the default watermark estimation logic and use an existing |
| watermark estimator implementation. You can also provide your own watermark estimator implementation.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="c1">// (Optional) Define a custom watermark state type to save information between bundle |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// processing rounds. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">MyCustomWatermarkState</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="nf">MyCustomWatermarkState</span><span class="o">(</span><span class="n">String</span> <span class="n">element</span><span class="o">,</span> <span class="n">OffsetRange</span> <span class="n">restriction</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Store data necessary for future watermark computations |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// (Optional) Choose which coder to use to encode the watermark estimator state. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nd">@GetWatermarkEstimatorStateCoder</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">Coder</span><span class="o">&lt;</span><span class="n">MyCustomWatermarkState</span><span class="o">&gt;</span> <span class="nf">getWatermarkEstimatorStateCoder</span><span class="o">()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">AvroCoder</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">MyCustomWatermarkState</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Define a WatermarkEstimator |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">MyCustomWatermarkEstimator</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">implements</span> <span class="n">TimestampObservingWatermarkEstimator</span><span class="o">&lt;</span><span class="n">MyCustomWatermarkState</span><span class="o">&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="nf">MyCustomWatermarkEstimator</span><span class="o">(</span><span class="n">MyCustomWatermarkState</span> <span class="n">type</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Initialize watermark estimator state |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">observeTimestamp</span><span class="o">(</span><span class="n">Instant</span> <span class="n">timestamp</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Will be invoked on each output from the SDF |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">Instant</span> <span class="nf">currentWatermark</span><span class="o">()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Return a monotonically increasing value |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="n">currentWatermark</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">MyCustomWatermarkState</span> <span class="nf">getState</span><span class="o">()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Return state to resume future watermark estimation after a checkpoint/split |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="kc">null</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Then, update the DoFn to generate the initial watermark estimator state for all new element |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// and restriction pairs and to create a new instance given watermark estimator state. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@GetInitialWatermarkEstimatorState</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">MyCustomWatermarkState</span> <span class="nf">getInitialWatermarkEstimatorState</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Element</span> <span class="n">String</span> <span class="n">element</span><span class="o">,</span> <span class="nd">@Restriction</span> <span class="n">OffsetRange</span> <span class="n">restriction</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Compute and return the initial watermark estimator state for each element and |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// restriction. All subsequent processing of an element and restriction will be restored |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// from the existing state. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="k">new</span> <span class="n">MyCustomWatermarkState</span><span class="o">(</span><span class="n">element</span><span class="o">,</span> <span class="n">restriction</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@NewWatermarkEstimator</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">WatermarkEstimator</span><span class="o">&lt;</span><span class="n">MyCustomWatermarkState</span><span class="o">&gt;</span> <span class="nf">newWatermarkEstimator</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@WatermarkEstimatorState</span> <span class="n">MyCustomWatermarkState</span> <span class="n">oldState</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="k">new</span> <span class="n">MyCustomWatermarkEstimator</span><span class="o">(</span><span class="n">oldState</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># (Optional) Define a custom watermark state type to save information between</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># bundle processing rounds.</span> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">MyCustomerWatermarkEstimatorState</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">,</span> <span class="n">restriction</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Store data necessary for future watermark computations</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">pass</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Define a WatermarkEstimator</span> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">MyCustomWatermarkEstimator</span><span class="p">(</span><span class="n">WatermarkEstimator</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">estimator_state</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">state</span> <span class="o">=</span> <span class="n">estimator_state</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">observe_timestamp</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">timestamp</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Will be invoked on each output from the SDF</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">pass</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">current_watermark</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Return a monotonically increasing value</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">current_watermark</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">get_estimator_state</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Return state to resume future watermark estimation after a</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># checkpoint/split</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">state</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Then, a WatermarkEstimatorProvider needs to be created for this</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># WatermarkEstimator</span> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">MyWatermarkEstimatorProvider</span><span class="p">(</span><span class="n">WatermarkEstimatorProvider</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">initial_estimator_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">,</span> <span class="n">restriction</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">MyCustomerWatermarkEstimatorState</span><span class="p">(</span><span class="n">element</span><span class="p">,</span> <span class="n">restriction</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">create_watermark_estimator</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">estimator_state</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">MyCustomWatermarkEstimator</span><span class="p">(</span><span class="n">estimator_state</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Finally, define the SDF using your estimator.</span> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">MySplittableDoFn</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">element</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">restriction_tracker</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">RestrictionParam</span><span class="p">(</span><span class="n">MyRestrictionProvider</span><span class="p">()),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">watermark_estimator</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">WatermarkEstimatorParam</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MyWatermarkEstimatorProvider</span><span class="p">())):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># The current watermark can be inspected.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">watermark_estimator</span><span class="o">.</span><span class="n">current_watermark</span><span class="p">()</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// WatermarkState is a custom type.` |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// It is optional to write your own state type when making a custom estimator. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">type</span> <span class="nx">WatermarkState</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Watermark</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Time</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// CustomWatermarkEstimator is a custom watermark estimator. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// You may use any type here, including some of Beam&#39;s built in watermark estimator types, |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// e.g. sdf.WallTimeWatermarkEstimator, sdf.TimestampObservingWatermarkEstimator, and sdf.ManualWatermarkEstimator |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">type</span> <span class="nx">CustomWatermarkEstimator</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">state</span> <span class="nx">WatermarkState</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// CurrentWatermark returns the current watermark and is invoked on DoFn splits and self-checkpoints. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// Watermark estimators must implement CurrentWatermark() time.Time |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">e</span> <span class="o">*</span><span class="nx">CustomWatermarkEstimator</span><span class="p">)</span> <span class="nf">CurrentWatermark</span><span class="p">()</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Time</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">e</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">Watermark</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// ObserveTimestamp is called on the output timestamps of all |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// emitted elements to update the watermark. It is optional |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">e</span> <span class="o">*</span><span class="nx">CustomWatermarkEstimator</span><span class="p">)</span> <span class="nf">ObserveTimestamp</span><span class="p">(</span><span class="nx">ts</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Time</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">e</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">Watermark</span> <span class="p">=</span> <span class="nx">ts</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// InitialWatermarkEstimatorState defines an initial state used to initialize the watermark |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// estimator. It is optional. If this is not defined, WatermarkEstimatorState may not be |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// defined and CreateWatermarkEstimator must not take in parameters. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">weDoFn</span><span class="p">)</span> <span class="nf">InitialWatermarkEstimatorState</span><span class="p">(</span><span class="nx">et</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span><span class="p">,</span> <span class="nx">rest</span> <span class="nx">offsetrange</span><span class="p">.</span><span class="nx">Restriction</span><span class="p">,</span> <span class="nx">element</span> <span class="kt">string</span><span class="p">)</span> <span class="nx">WatermarkState</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Return some watermark state |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="nx">WatermarkState</span><span class="p">{</span><span class="nx">Watermark</span><span class="p">:</span> <span class="nx">time</span><span class="p">.</span><span class="nf">Now</span><span class="p">()}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// CreateWatermarkEstimator creates the watermark estimator used by this Splittable DoFn. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// Must take in a state parameter if InitialWatermarkEstimatorState is defined, otherwise takes no parameters. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">weDoFn</span><span class="p">)</span> <span class="nf">CreateWatermarkEstimator</span><span class="p">(</span><span class="nx">initialState</span> <span class="nx">WatermarkState</span><span class="p">)</span> <span class="o">*</span><span class="nx">CustomWatermarkEstimator</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="o">&amp;</span><span class="nx">CustomWatermarkEstimator</span><span class="p">{</span><span class="nx">state</span><span class="p">:</span> <span class="nx">initialState</span><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// WatermarkEstimatorState returns the state used to resume future watermark estimation |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// after a checkpoint/split. It is required if InitialWatermarkEstimatorState is defined, |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// otherwise it must not be defined. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">weDoFn</span><span class="p">)</span> <span class="nf">WatermarkEstimatorState</span><span class="p">(</span><span class="nx">e</span> <span class="o">*</span><span class="nx">CustomWatermarkEstimator</span><span class="p">)</span> <span class="nx">WatermarkState</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">e</span><span class="p">.</span><span class="nx">state</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// ProcessElement is the method to execute for each element. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// It can optionally take in a watermark estimator. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">weDoFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">e</span> <span class="o">*</span><span class="nx">CustomWatermarkEstimator</span><span class="p">,</span> <span class="nx">element</span> <span class="kt">string</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">e</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">Watermark</span> <span class="p">=</span> <span class="nx">time</span><span class="p">.</span><span class="nf">Now</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="truncating-during-drain">12.6. Truncating during drain</h3> |
| <p>Runners which support draining pipelines need the ability to drain SDFs; otherwise, the |
| pipeline may never stop. By default, bounded restrictions process the remainder of the restriction while |
| unbounded restrictions finish processing at the next SDF-initiated checkpoint or runner-initiated split. |
| You are able to override this default behavior by defining the appropriate method on the restriction |
| provider.</p> |
| <p class="language-go">Note: Once the pipeline drain starts and truncate restriction transform is triggered, the <code>sdf.ProcessContinuation</code> |
| will not be rescheduled.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@TruncateRestriction</span> |
| </span></span><span class="line"><span class="cl"><span class="nd">@Nullable</span> |
| </span></span><span class="line"><span class="cl"><span class="n">TruncateResult</span><span class="o">&lt;</span><span class="n">OffsetRange</span><span class="o">&gt;</span> <span class="nf">truncateRestriction</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Element</span> <span class="n">String</span> <span class="n">fileName</span><span class="o">,</span> <span class="nd">@Restriction</span> <span class="n">OffsetRange</span> <span class="n">restriction</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="o">(</span><span class="n">fileName</span><span class="o">.</span><span class="na">contains</span><span class="o">(</span><span class="s">&#34;optional&#34;</span><span class="o">))</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Skip optional files |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="kc">null</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">TruncateResult</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">restriction</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">MyRestrictionProvider</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">transforms</span><span class="o">.</span><span class="n">core</span><span class="o">.</span><span class="n">RestrictionProvider</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">truncate</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">file_name</span><span class="p">,</span> <span class="n">restriction</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="s2">&#34;optional&#34;</span> <span class="ow">in</span> <span class="n">file_name</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Skip optional files</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">None</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">restriction</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// TruncateRestriction is a transform that is triggered when pipeline starts to drain. It helps to finish a |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// pipeline quicker by truncating the restriction. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">splittableDoFn</span><span class="p">)</span> <span class="nf">TruncateRestriction</span><span class="p">(</span><span class="nx">rt</span> <span class="o">*</span><span class="nx">sdf</span><span class="p">.</span><span class="nx">LockRTracker</span><span class="p">,</span> <span class="nx">element</span> <span class="kt">string</span><span class="p">)</span> <span class="nx">offsetrange</span><span class="p">.</span><span class="nx">Restriction</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">start</span> <span class="o">:=</span> <span class="nx">rt</span><span class="p">.</span><span class="nf">GetRestriction</span><span class="p">().(</span><span class="nx">offsetrange</span><span class="p">.</span><span class="nx">Restriction</span><span class="p">).</span><span class="nx">Start</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">prevEnd</span> <span class="o">:=</span> <span class="nx">rt</span><span class="p">.</span><span class="nf">GetRestriction</span><span class="p">().(</span><span class="nx">offsetrange</span><span class="p">.</span><span class="nx">Restriction</span><span class="p">).</span><span class="nx">End</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// truncate the restriction by half. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">newEnd</span> <span class="o">:=</span> <span class="nx">prevEnd</span> <span class="o">/</span> <span class="mi">2</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">offsetrange</span><span class="p">.</span><span class="nx">Restriction</span><span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Start</span><span class="p">:</span> <span class="nx">start</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">End</span><span class="p">:</span> <span class="nx">newEnd</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="bundle-finalization">12.7. Bundle finalization</h3> |
| <p>Bundle finalization enables a <code>DoFn</code> to perform side effects by registering a callback. |
| The callback is invoked once the runner has acknowledged that it has durably persisted the output. |
| For example, a message queue might need to acknowledge messages that it has ingested into the pipeline. |
| Bundle finalization is not limited to SDFs but is called out here since this is the primary |
| use case.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">c</span><span class="o">,</span> <span class="n">BundleFinalizer</span> <span class="n">bundleFinalizer</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... produce output ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="n">bundleFinalizer</span><span class="o">.</span><span class="na">afterBundleCommit</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Instant</span><span class="o">.</span><span class="na">now</span><span class="o">().</span><span class="na">plus</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="n">5</span><span class="o">)),</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">()</span> <span class="o">-&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... perform a side effect ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">});</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">MySplittableDoFn</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">,</span> <span class="n">bundle_finalizer</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">BundleFinalizerParam</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># ... produce output ...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Register callback function for this bundle that performs the side</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># effect.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">bundle_finalizer</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">my_callback_func</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">fn</span> <span class="o">*</span><span class="nx">splittableDoFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">bf</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">BundleFinalization</span><span class="p">,</span> <span class="nx">rt</span> <span class="o">*</span><span class="nx">sdf</span><span class="p">.</span><span class="nx">LockRTracker</span><span class="p">,</span> <span class="nx">element</span> <span class="kt">string</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... produce output ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">bf</span><span class="p">.</span><span class="nf">RegisterCallback</span><span class="p">(</span><span class="mi">5</span><span class="o">*</span><span class="nx">time</span><span class="p">.</span><span class="nx">Minute</span><span class="p">,</span> <span class="kd">func</span><span class="p">()</span> <span class="kt">error</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// ... perform a side effect ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="kc">nil</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">})</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h2 id="multi-language-pipelines">13. Multi-language pipelines</h2> |
| <p>This section provides comprehensive documentation of multi-language pipelines. To get started creating a multi-language pipeline, see:</p> |
| <ul> |
| <li><a href="/documentation/sdks/python-multi-language-pipelines">Python multi-language pipelines quickstart</a></li> |
| <li><a href="/documentation/sdks/java-multi-language-pipelines">Java multi-language pipelines quickstart</a></li> |
| </ul> |
| <p>Beam lets you combine transforms written in any supported SDK language (currently, Java and Python) and use them in one multi-language pipeline. This capability makes it easy to provide new functionality simultaneously in different Apache Beam SDKs through a single cross-language transform. For example, the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/kafka.py">Apache Kafka connector</a> and <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/sql.py">SQL transform</a> from the Java SDK can be used in Python pipelines.</p> |
| <p>Pipelines that use transforms from more than one SDK-language are known as <em>multi-language pipelines</em>.</p> |
| <p class="language-yaml">Beam YAML is built entirely on top of cross-language transforms. |
| In addition to the built in transforms, you can author your own transforms |
| (using the full expressivity of the Beam API) and surface them via a concept |
| called <a href="https://beam.apache.org/documentation/sdks/yaml/#providers">providers</a>.</p> |
| <h3 id="create-x-lang-transforms">13.1. Creating cross-language transforms</h3> |
| <p>To make transforms written in one language available to pipelines written in another language, Beam uses an <em>expansion service</em>, which creates and injects the appropriate language-specific pipeline fragments into the pipeline.</p> |
| <p>In the following example, a Beam Python pipeline starts up a local Java expansion service to create and inject the appropriate Java pipeline fragments for executing the Java Kafka cross-language transform into the Python pipeline. The SDK then downloads and stages the necessary Java dependencies needed to execute these transforms.</p> |
| <p><img src="/images/multi-language-pipelines-diagram.svg" alt="Diagram of multi-language pipeline execution flow."></p> |
| <p>At runtime, the Beam runner will execute both Python and Java transforms to run the pipeline.</p> |
| <p>In this section, we will use <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kafka/KafkaIO.Read.html">KafkaIO.Read</a> to illustrate how to create a cross-language transform for Java and a test example for Python.</p> |
| <h4 id="1311-creating-cross-language-java-transforms">13.1.1. Creating cross-language Java transforms</h4> |
| <p>There are two ways to make Java transforms available to other SDKs.</p> |
| <ul> |
| <li>Option 1: In some cases, you can use existing Java transforms from other SDKs without writing any additional Java code.</li> |
| <li>Option 2: You can use arbitrary Java transforms from other SDKs by adding a few Java classes.</li> |
| </ul> |
| <h5 id="13111-using-existing-java-transforms-without-writing-more-java-code">13.1.1.1 Using existing Java transforms without writing more Java code</h5> |
| <p>Starting with Beam 2.34.0, Python SDK users can use some Java transforms without writing additional Java code. This can be useful in many cases. For example:</p> |
| <ul> |
| <li>A developer not familiar with Java may need to use an existing Java transform from a Python pipeline.</li> |
| <li>A developer may need to make an existing Java transform available to a Python pipeline without writing/releasing more Java code.</li> |
| </ul> |
| <blockquote> |
| <p><strong>Note:</strong> This feature is currently only available when using Java transforms from a Python pipeline.</p> |
| </blockquote> |
| <p>To be eligible for direct usage, the API of the Java transform has to meet the following requirements:</p> |
| <ol> |
| <li>The Java transform can be constructed using an available public constructor or a public static method (a constructor method) in the same Java class.</li> |
| <li>The Java transform can be configured using one or more builder methods. Each builder method should be public and should return an instance of the Java transform.</li> |
| </ol> |
| <p>Here&rsquo;s an example Java class that can be directly used from the Python API.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">JavaDataGenerator</span> <span class="kd">extends</span> <span class="n">PTransform</span><span class="o">&lt;</span><span class="n">PBegin</span><span class="o">,</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span> <span class="o">.</span> <span class="o">.</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The following method satisfies requirement 1. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// Note that you could use a class constructor instead of a static method. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kd">public</span> <span class="kd">static</span> <span class="n">JavaDataGenerator</span> <span class="nf">create</span><span class="o">(</span><span class="n">Integer</span> <span class="n">size</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="k">new</span> <span class="n">JavaDataGenerator</span><span class="o">(</span><span class="n">size</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">JavaDataGeneratorConfig</span> <span class="kd">implements</span> <span class="n">Serializable</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">String</span> <span class="n">prefix</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">long</span> <span class="n">length</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">String</span> <span class="n">suffix</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span> <span class="o">.</span> <span class="o">.</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The following method conforms to requirement 2. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kd">public</span> <span class="n">JavaDataGenerator</span> <span class="nf">withJavaDataGeneratorConfig</span><span class="o">(</span><span class="n">JavaDataGeneratorConfig</span> <span class="n">dataConfig</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="k">new</span> <span class="n">JavaDataGenerator</span><span class="o">(</span><span class="k">this</span><span class="o">.</span><span class="na">size</span><span class="o">,</span> <span class="n">javaDataGeneratorConfig</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span> <span class="o">.</span> <span class="o">.</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span> |
| </span></span></code></pre></div><p>For a complete example, see <a href="https://github.com/apache/beam/blob/master/examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/JavaDataGenerator.java">JavaDataGenerator</a>.</p> |
| <p>To use a Java class that conforms to the above requirements from a Python SDK pipeline, follow these steps:</p> |
| <ol> |
| <li>Create a <em>yaml</em> allowlist that describes the Java transform classes and methods that will be directly accessed from Python.</li> |
| <li>Start an expansion service, using the <code>javaClassLookupAllowlistFile</code> option to pass the path to the allowlist.</li> |
| <li>Use the Python <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/external.py">JavaExternalTransform</a> API to directly access Java transforms defined in the allowlist from the Python side.</li> |
| </ol> |
| <p>Starting with Beam 2.36.0, steps 1 and 2 can be skipped, as described in the corresponding sections below.</p> |
| <p><strong>Step 1</strong></p> |
| <p>To use an eligible Java transform from Python, define a <em>yaml</em> allowlist. This allowlist lists the class names, |
| constructor methods, and builder methods that are directly available to be used from the Python side.</p> |
| <p>Starting with Beam 2.35.0, you have the option to pass <code>*</code> to the <code>javaClassLookupAllowlistFile</code> option instead of defining an actual allowlist. The <code>*</code> specifies that all supported transforms in the classpath of the expansion service can be accessed through the API. We encourage using an actual allowlist for production, because allowing clients to access arbitrary Java classes can pose a security risk.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>version: v1 |
| allowedClasses: |
| - className: my.beam.transforms.JavaDataGenerator |
| allowedConstructorMethods: |
| - create |
| allowedBuilderMethods: |
| - withJavaDataGeneratorConfig</code></pre> |
| </div> |
| </div> |
| <p><strong>Step 2</strong></p> |
| <p>Provide the allowlist as an argument when starting up the Java expansion service. For example, you can start the expansion service |
| as a local Java process using the following command:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>java -jar &lt;jar file&gt; &lt;port&gt; --javaClassLookupAllowlistFile=&lt;path to the allowlist file&gt;</code></pre> |
| </div> |
| </div> |
| <p>Starting with Beam 2.36.0, the <code>JavaExternalTransform</code> API will automatically start up an expansion service with a given <code>jar</code> file dependency if an expansion service address was not provided.</p> |
| <p><strong>Step 3</strong></p> |
| <p>You can use the Java class directly from your Python pipeline using a stub transform created from the <code>JavaExternalTransform</code> API. This API allows you to construct the transform using the Java class name and allows you to invoke builder methods to configure the class.</p> |
| <p>Constructor and method parameter types are mapped between Python and Java using a Beam schema. The schema is auto-generated using the object types |
| provided on the Python side. If the Java class constructor method or builder method accepts any complex object types, make sure that the Beam schema |
| for these objects is registered and available for the Java expansion service. If a schema has not been registered, the Java expansion service will |
| try to register a schema using <a href="/documentation/programming-guide/#creating-schemas">JavaFieldSchema</a>. In Python, arbitrary objects |
| can be represented using <code>NamedTuple</code>s, which will be represented as Beam rows in the schema. Here is a Python stub transform that represents the above |
| mentioned Java transform:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">JavaDataGeneratorConfig</span> <span class="o">=</span> <span class="n">typing</span><span class="o">.</span><span class="n">NamedTuple</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"><span class="s1">&#39;JavaDataGeneratorConfig&#39;</span><span class="p">,</span> <span class="p">[(</span><span class="s1">&#39;prefix&#39;</span><span class="p">,</span> <span class="nb">str</span><span class="p">),</span> <span class="p">(</span><span class="s1">&#39;length&#39;</span><span class="p">,</span> <span class="nb">int</span><span class="p">),</span> <span class="p">(</span><span class="s1">&#39;suffix&#39;</span><span class="p">,</span> <span class="nb">str</span><span class="p">)])</span> |
| </span></span><span class="line"><span class="cl"><span class="n">data_config</span> <span class="o">=</span> <span class="n">JavaDataGeneratorConfig</span><span class="p">(</span><span class="n">prefix</span><span class="o">=</span><span class="s1">&#39;start&#39;</span><span class="p">,</span> <span class="n">length</span><span class="o">=</span><span class="mi">20</span><span class="p">,</span> <span class="n">suffix</span><span class="o">=</span><span class="s1">&#39;end&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">java_transform</span> <span class="o">=</span> <span class="n">JavaExternalTransform</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"><span class="s1">&#39;my.beam.transforms.JavaDataGenerator&#39;</span><span class="p">,</span> <span class="n">expansion_service</span><span class="o">=</span><span class="s1">&#39;localhost:&lt;port&gt;&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="n">numpy</span><span class="o">.</span><span class="n">int32</span><span class="p">(</span><span class="mi">100</span><span class="p">))</span><span class="o">.</span><span class="n">withJavaDataGeneratorConfig</span><span class="p">(</span><span class="n">data_config</span><span class="p">)</span> |
| </span></span></code></pre></div><p>You can use this transform in a Python pipeline along with other Python transforms. For a complete example, see <a href="https://github.com/apache/beam/blob/master/examples/multi-language/python/javadatagenerator.py">javadatagenerator.py</a>.</p> |
| <h5 id="13112-using-the-api-to-make-existing-java-transforms-available-to-other-sdks">13.1.1.2 Using the API to make existing Java transforms available to other SDKs</h5> |
| <p>To make your Beam Java SDK transform portable across SDK languages, you must implement two interfaces: <a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ExternalTransformBuilder.java">ExternalTransformBuilder</a> and <a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/expansion/ExternalTransformRegistrar.java">ExternalTransformRegistrar</a>. The <code>ExternalTransformBuilder</code> interface constructs the cross-language transform using configuration values passed in from the pipeline, and the <code>ExternalTransformRegistrar</code> interface registers the cross-language transform for use with the expansion service.</p> |
| <p><strong>Implementing the interfaces</strong></p> |
| <ol> |
| <li> |
| <p>Define a Builder class for your transform that implements the <code>ExternalTransformBuilder</code> interface and overrides the <code>buildExternal</code> method that will be used to build your transform object. Initial configuration values for your transform should be defined in the <code>buildExternal</code> method. In most cases, it&rsquo;s convenient to make the Java transform builder class implement <code>ExternalTransformBuilder</code>.</p> |
| <blockquote> |
| <p><strong>Note:</strong> <code>ExternalTransformBuilder</code> requires you to define a configuration object (a simple POJO) to capture a set of parameters sent by external SDKs to initiate the Java transform. Usually these parameters directly map to constructor parameters of the Java transform.</p> |
| </blockquote> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@AutoValue.Builder</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">abstract</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">Builder</span><span class="o">&lt;</span><span class="n">K</span><span class="o">,</span> <span class="n">V</span><span class="o">&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">implements</span> <span class="n">ExternalTransformBuilder</span><span class="o">&lt;</span><span class="n">External</span><span class="o">.</span><span class="na">Configuration</span><span class="o">,</span> <span class="n">PBegin</span><span class="o">,</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">K</span><span class="o">,</span> <span class="n">V</span><span class="o">&gt;&gt;&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">abstract</span> <span class="n">Builder</span><span class="o">&lt;</span><span class="n">K</span><span class="o">,</span> <span class="n">V</span><span class="o">&gt;</span> <span class="nf">setConsumerConfig</span><span class="o">(</span><span class="n">Map</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Object</span><span class="o">&gt;</span> <span class="n">config</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="kd">abstract</span> <span class="n">Builder</span><span class="o">&lt;</span><span class="n">K</span><span class="o">,</span> <span class="n">V</span><span class="o">&gt;</span> <span class="nf">setTopics</span><span class="o">(</span><span class="n">List</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">topics</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="cm">/** Remaining property declarations omitted for clarity. */</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="kd">abstract</span> <span class="n">Read</span><span class="o">&lt;</span><span class="n">K</span><span class="o">,</span> <span class="n">V</span><span class="o">&gt;</span> <span class="nf">build</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">PTransform</span><span class="o">&lt;</span><span class="n">PBegin</span><span class="o">,</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">K</span><span class="o">,</span> <span class="n">V</span><span class="o">&gt;&gt;&gt;</span> <span class="nf">buildExternal</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">External</span><span class="o">.</span><span class="na">Configuration</span> <span class="n">config</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">setTopics</span><span class="o">(</span><span class="n">ImmutableList</span><span class="o">.</span><span class="na">copyOf</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">topics</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="cm">/** Remaining property defaults omitted for clarity. */</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span> |
| </span></span></code></pre></div><p>For complete examples, see <a href="https://github.com/apache/beam/blob/master/examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/JavaCountBuilder.java">JavaCountBuilder</a> and <a href="https://github.com/apache/beam/blob/master/examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/JavaPrefixBuilder.java">JavaPrefixBuilder</a>.</p> |
| <p>Note that the <code>buildExternal</code> method can perform additional operations before setting properties received from external SDKs in the transform. For example, <code>buildExternal</code> can validate properties available in the configuration object before setting them in the transform.</p> |
| </li> |
| <li> |
| <p>Register the transform as an external cross-language transform by defining a class that implements <code>ExternalTransformRegistrar</code>. You must annotate your class with the <code>AutoService</code> annotation to ensure that your transform is registered and instantiated properly by the expansion service.</p> |
| </li> |
| <li> |
| <p>In your registrar class, define a Uniform Resource Name (URN) for your transform. The URN must be a unique string that identifies your transform with the expansion service.</p> |
| </li> |
| <li> |
| <p>From within your registrar class, define a configuration class for the parameters used during the initialization of your transform by the external SDK.</p> |
| <p>The following example from the KafkaIO transform shows how to implement steps two through four:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@AutoService</span><span class="o">(</span><span class="n">ExternalTransformRegistrar</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">External</span> <span class="kd">implements</span> <span class="n">ExternalTransformRegistrar</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">String</span> <span class="n">URN</span> <span class="o">=</span> <span class="s">&#34;beam:external:java:kafka:read:v1&#34;</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">Map</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Class</span><span class="o">&lt;?</span> <span class="kd">extends</span> <span class="n">ExternalTransformBuilder</span><span class="o">&lt;?,</span> <span class="o">?,</span> <span class="o">?&gt;&gt;&gt;</span> <span class="n">knownBuilders</span><span class="o">()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">ImmutableMap</span><span class="o">.</span><span class="na">of</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">URN</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">(</span><span class="n">Class</span><span class="o">&lt;?</span> <span class="kd">extends</span> <span class="n">ExternalTransformBuilder</span><span class="o">&lt;?,</span> <span class="o">?,</span> <span class="o">?&gt;&gt;)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">(</span><span class="n">Class</span><span class="o">&lt;?&gt;)</span> <span class="n">AutoValue_KafkaIO_Read</span><span class="o">.</span><span class="na">Builder</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="cm">/** Parameters class to expose the Read transform to an external SDK. */</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">Configuration</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">private</span> <span class="n">Map</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">consumerConfig</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">private</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">topics</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">setConsumerConfig</span><span class="o">(</span><span class="n">Map</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">consumerConfig</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">this</span><span class="o">.</span><span class="na">consumerConfig</span> <span class="o">=</span> <span class="n">consumerConfig</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">setTopics</span><span class="o">(</span><span class="n">List</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">topics</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">this</span><span class="o">.</span><span class="na">topics</span> <span class="o">=</span> <span class="n">topics</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="cm">/** Remaining properties omitted for clarity. */</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span> |
| </span></span></code></pre></div><p>For additional examples, see <a href="https://github.com/apache/beam/blob/master/examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/JavaCountRegistrar.java">JavaCountRegistrar</a> and <a href="https://github.com/apache/beam/blob/master/examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/JavaPrefixRegistrar.java">JavaPrefixRegistrar</a>.</p> |
| </li> |
| </ol> |
| <p>After you have implemented the <code>ExternalTransformBuilder</code> and <code>ExternalTransformRegistrar</code> interfaces, your transform can be registered and created successfully by the default Java expansion service.</p> |
| <p><strong>Starting the expansion service</strong></p> |
| <p>You can use an expansion service with multiple transforms in the same pipeline. The Beam Java SDK provides a default expansion service for Java transforms. You can also write your own expansion service, but that&rsquo;s generally not needed, so it&rsquo;s not covered in this section.</p> |
| <p>Perform the following to start up a Java expansion service directly:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code># Build a JAR with both your transform and the expansion service |
| # Start the expansion service at the specified port. |
| $ jar -jar /path/to/expansion_service.jar &lt;PORT_NUMBER&gt;</code></pre> |
| </div> |
| </div> |
| <p>The expansion service is now ready to serve transforms on the specified port.</p> |
| <p>When creating SDK-specific wrappers for your transform, you may be able to use SDK-provided utilities to start up an expansion service. For example, the Python SDK provides the utilities <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.JavaJarExpansionService"><code>JavaJarExpansionService</code></a> and <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.BeamJarExpansionService"><code>BeamJarExpansionService</code></a> for starting up a Java expansion service using a JAR file.</p> |
| <p><strong>Including dependencies</strong></p> |
| <p>If your transform requires external libraries, you can include them by adding them to the classpath of the expansion service. After they are included in the classpath, they will be staged when your transform is expanded by the expansion service.</p> |
| <p><strong>Writing SDK-specific wrappers</strong></p> |
| <p>Your cross-language Java transform can be called through the lower-level <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.ExternalTransform"><code>ExternalTransform</code></a> class in a multi-language pipeline (as described in the next section); however, if possible, you should write an SDK-specific wrapper in the language of the pipeline (such as Python) to access the transform instead. This higher-level abstraction will make it easier for pipeline authors to use your transform.</p> |
| <p>To create an SDK wrapper for use in a Python pipeline, do the following:</p> |
| <ol> |
| <li> |
| <p>Create a Python module for your cross-language transform(s).</p> |
| </li> |
| <li> |
| <p>In the module, use one of the <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.PayloadBuilder"><code>PayloadBuilder</code></a> classes to build the payload for the initial cross-language transform expansion request.</p> |
| <p>The parameter names and types of the payload should map to parameter names and types of the configuration POJO provided to the Java <code>ExternalTransformBuilder</code>. Parameter types are mapped across SDKs using a <a href="https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema.proto">Beam schema</a>. Parameter names are mapped by simply converting Python underscore-separated variable names to camel-case (Java standard).</p> |
| <p>In the following example, <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/kafka.py">kafka.py</a> uses <code>NamedTupleBasedPayloadBuilder</code> to build the payload. The parameters map to the Java <a href="https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java">KafkaIO.External.Configuration</a> config object defined in the previous section.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">ReadFromKafkaSchema</span><span class="p">(</span><span class="n">typing</span><span class="o">.</span><span class="n">NamedTuple</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">consumer_config</span><span class="p">:</span> <span class="n">typing</span><span class="o">.</span><span class="n">Mapping</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">topics</span><span class="p">:</span> <span class="n">typing</span><span class="o">.</span><span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Other properties omitted for clarity.</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">payload</span> <span class="o">=</span> <span class="n">NamedTupleBasedPayloadBuilder</span><span class="p">(</span><span class="n">ReadFromKafkaSchema</span><span class="p">(</span><span class="o">...</span><span class="p">))</span> |
| </span></span></code></pre></div></li> |
| <li> |
| <p>Start an expansion service, unless one is specified by the pipeline creator. The Beam Python SDK provides the utilities <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.JavaJarExpansionService"><code>JavaJarExpansionService</code></a> and <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.BeamJarExpansionService"><code>BeamJarExpansionService</code></a> for starting up an expansion service using a JAR file. <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.JavaJarExpansionService"><code>JavaJarExpansionService</code></a> can be used to start up an expansion service using the path (a local path or a URL) to a given JAR file. <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.BeamJarExpansionService"><code>BeamJarExpansionService</code></a> can be used to start an expansion service from a JAR released with Beam.</p> |
| <p>For transforms released with Beam, do the following:</p> |
| <ol> |
| <li> |
| <p>Add a Gradle target to Beam that can be used to build a shaded expansion service JAR for the target Java transform. This target should produce a Beam JAR that contains all dependencies needed for expanding the Java transform, and the JAR should be released with Beam. You might be able to use an existing Gradle target that offers an aggregated version of an expansion service JAR (for example, for all GCP IO).</p> |
| </li> |
| <li> |
| <p>In your Python module, instantiate <code>BeamJarExpansionService</code> with the Gradle target.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">expansion_service</span> <span class="o">=</span> <span class="n">BeamJarExpansionService</span><span class="p">(</span><span class="s1">&#39;sdks:java:io:expansion-service:shadowJar&#39;</span><span class="p">)</span> |
| </span></span></code></pre></div></li> |
| </ol> |
| </li> |
| <li> |
| <p>Add a Python wrapper transform class that extends <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.ExternalTransform"><code>ExternalTransform</code></a>. Pass the payload and expansion service defined above as parameters to the constructor of the <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.ExternalTransform"><code>ExternalTransform</code></a> parent class.</p> |
| </li> |
| </ol> |
| <h4 id="1312-creating-cross-language-python-transforms">13.1.2. Creating cross-language Python transforms</h4> |
| <p>Any Python transforms defined in the scope of the expansion service should be accessible by specifying their fully qualified names. For example, you could use Python&rsquo;s <code>ReadFromText</code> transform in a Java pipeline with its fully qualified name <code>apache_beam.io.ReadFromText</code>:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;Read&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">PythonExternalTransform</span><span class="o">.&lt;</span><span class="n">PBegin</span><span class="o">,</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;&gt;</span><span class="n">from</span><span class="o">(</span><span class="s">&#34;apache_beam.io.ReadFromText&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKwarg</span><span class="o">(</span><span class="s">&#34;file_pattern&#34;</span><span class="o">,</span> <span class="n">options</span><span class="o">.</span><span class="na">getInputFile</span><span class="o">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKwarg</span><span class="o">(</span><span class="s">&#34;validate&#34;</span><span class="o">,</span> <span class="kc">false</span><span class="o">))</span> |
| </span></span></code></pre></div><p><code>PythonExternalTransform</code> has other useful methods such as <code>withExtraPackages</code> for staging PyPI package dependencies and <code>withOutputCoder</code> for setting an output coder. If your transform exists in an external package, make sure to specify that package using <code>withExtraPackages</code>, for example:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;Read&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">PythonExternalTransform</span><span class="o">.&lt;</span><span class="n">PBegin</span><span class="o">,</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;&gt;</span><span class="n">from</span><span class="o">(</span><span class="s">&#34;my_python_package.BeamReadPTransform&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withExtraPackages</span><span class="o">(</span><span class="n">ImmutableList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;my_python_package&#34;</span><span class="o">)))</span> |
| </span></span></code></pre></div><p>Alternatively, you may want to create a Python module that registers an existing Python transform as a cross-language transform for use with the Python expansion service and calls into that existing transform to perform its intended operation. A registered URN can be used later in an expansion request for indicating an expansion target.</p> |
| <p><strong>Defining the Python module</strong></p> |
| <ol> |
| <li> |
| <p>Define a Uniform Resource Name (URN) for your transform. The URN must be a unique string that identifies your transform with the expansion service.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">TEST_COMPK_URN</span> <span class="o">=</span> <span class="s2">&#34;beam:transforms:xlang:test:compk&#34;</span> |
| </span></span></code></pre></div></li> |
| <li> |
| <p>For an existing Python transform, create a new class to register the URN with the Python expansion service.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="nd">@ptransform.PTransform.register_urn</span><span class="p">(</span><span class="n">TEST_COMPK_URN</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">CombinePerKeyTransform</span><span class="p">(</span><span class="n">ptransform</span><span class="o">.</span><span class="n">PTransform</span><span class="p">):</span> |
| </span></span></code></pre></div></li> |
| <li> |
| <p>From within the class, define an expand method that takes an input PCollection, runs the Python transform, and then returns the output PCollection.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">expand</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pcoll</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">pcoll</span> \ |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombinePerKey</span><span class="p">(</span><span class="nb">sum</span><span class="p">)</span><span class="o">.</span><span class="n">with_output_types</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">typing</span><span class="o">.</span><span class="n">Tuple</span><span class="p">[</span><span class="n">unicode</span><span class="p">,</span> <span class="nb">int</span><span class="p">])</span> |
| </span></span></code></pre></div></li> |
| <li> |
| <p>As with other Python transforms, define a <code>to_runner_api_parameter</code> method that returns the URN.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">to_runner_api_parameter</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">unused_context</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">TEST_COMPK_URN</span><span class="p">,</span> <span class="kc">None</span> |
| </span></span></code></pre></div></li> |
| <li> |
| <p>Define a static <code>from_runner_api_parameter</code> method that returns an instantiation of the cross-language Python transform.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="nd">@staticmethod</span> |
| </span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">from_runner_api_parameter</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">unused_ptransform</span><span class="p">,</span> <span class="n">unused_parameter</span><span class="p">,</span> <span class="n">unused_context</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">CombinePerKeyTransform</span><span class="p">()</span> |
| </span></span></code></pre></div></li> |
| </ol> |
| <p><strong>Starting the expansion service</strong></p> |
| <p>An expansion service can be used with multiple transforms in the same pipeline. The Beam Python SDK provides a default expansion service for you to use with your Python transforms. You are free to write your own expansion service, but that is generally not needed, so it is not covered in this section.</p> |
| <p>Perform the following steps to start up the default Python expansion service directly:</p> |
| <ol> |
| <li> |
| <p>Create a virtual environment and <a href="/get-started/quickstart-py/">install the Apache Beam SDK</a>.</p> |
| </li> |
| <li> |
| <p>Start the Python SDK’s expansion service with a specified port.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>$ export PORT_FOR_EXPANSION_SERVICE=12345 |
| </code></pre> |
| </div> |
| </div> |
| </li> |
| <li> |
| <p>Import any modules that contain transforms to be made available using the expansion service.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>$ python -m apache_beam.runners.portability.expansion_service_test -p $PORT_FOR_EXPANSION_SERVICE --pickle_library=cloudpickle |
| </code></pre> |
| </div> |
| </div> |
| </li> |
| <li> |
| <p>This expansion service is now ready to serve up transforms on the address `localhost:$PORT_FOR_EXPANSION_SERVICE</p> |
| </li> |
| </ol> |
| <h4 id="1313-creating-cross-language-go-transforms">13.1.3. Creating cross-language Go transforms</h4> |
| <p>Go currently does not support creating cross-language transforms, only using cross-language |
| transforms from other languages; see more at <a href="https://github.com/apache/beam/issues/21767">Issue 21767</a>.</p> |
| <h4 id="1314-defining-a-urn">13.1.4. Defining a URN</h4> |
| <p>Developing a cross-language transform involves defining a URN for registering the transform with an expansion service. In this section |
| we provide a convention for defining such URNs. Following this convention is optional but it will ensure that your transform |
| will not run into conflicts when registering in an expansion service along with transforms developed by other developers.</p> |
| <h5 id="13141-schema">13.1.4.1. Schema</h5> |
| <p>A URN should consist of the following components:</p> |
| <ul> |
| <li><strong>ns-id</strong>: A namespace identifier. Default recommendation is <code>beam:transform</code>.</li> |
| <li><strong>org-identifier</strong>: Identifies the organization where the transform was defined. Transforms defined in Apache Beam use <code>org.apache.beam</code> for this.</li> |
| <li><strong>functionality-identifier</strong>: Identifies the functionality of the cross-language transform.</li> |
| <li><strong>version</strong>: a version number for the transform.</li> |
| </ul> |
| <p>We provide the schema from the URN convention in <a href="https://en.wikipedia.org/wiki/Augmented_Backus%E2%80%93Naur_form">augmented Backus–Naur</a> form. |
| Keywords in upper case are from the <a href="https://datatracker.ietf.org/doc/html/rfc8141">URN spec</a>.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>transform-urn = ns-id “:” org-identifier “:” functionality-identifier “:” version |
| ns-id = (“beam” / NID) “:” “transform” |
| id-char = ALPHA / DIGIT / &#34;-&#34; / &#34;.&#34; / &#34;_&#34; / &#34;~&#34; ; A subset of characters allowed in a URN |
| org-identifier = 1*id-char |
| functionality-identifier = 1*id-char |
| version = “v” 1*(DIGIT / “.”) ; For example, ‘v1.2’</code></pre> |
| </div> |
| </div> |
| <h5 id="13142-examples">13.1.4.2. Examples</h5> |
| <p>Below we’ve given some example transform classes and corresponding URNs to be used.</p> |
| <ul> |
| <li>A transform offered with Apache Beam that writes Parquet files. |
| <ul> |
| <li><code>beam:transform:org.apache.beam:parquet_write:v1</code></li> |
| </ul> |
| </li> |
| <li>A transform offered with Apache Beam that reads from Kafka with metadata. |
| <ul> |
| <li><code>beam:transform:org.apache.beam:kafka_read_with_metadata:v1</code></li> |
| </ul> |
| </li> |
| <li>A transform developed by organization abc.org that reads from data store MyDatastore. |
| <ul> |
| <li><code>beam:transform:org.abc:mydatastore_read:v1</code></li> |
| </ul> |
| </li> |
| </ul> |
| <h3 id="use-x-lang-transforms">13.2. Using cross-language transforms</h3> |
| <p>Depending on the SDK language of the pipeline, you can use a high-level SDK-wrapper class, or a low-level transform class to access a cross-language transform.</p> |
| <h4 id="1321-using-cross-language-transforms-in-a-java-pipeline">13.2.1. Using cross-language transforms in a Java pipeline</h4> |
| <p>Users have three options to use cross-language transforms in a Java pipeline. At the highest level of abstraction, some popular Python transforms are accessible through dedicated Java wrapper transforms. For example, the Java SDK has the <code>DataframeTransform</code> class, which uses the Python SDK&rsquo;s <code>DataframeTransform</code>, and it has the <code>RunInference</code> class, which uses the Python SDK&rsquo;s <code>RunInference</code>, and so on. When an SDK-specific wrapper transform is not available for a target Python transform, you can use the lower-level <a href="https://github.com/apache/beam/blob/master/sdks/java/extensions/python/src/main/java/org/apache/beam/sdk/extensions/python/PythonExternalTransform.java">PythonExternalTransform</a> class instead by specifying the fully qualified name of the Python transform. If you want to try external transforms from SDKs other than Python (including Java SDK itself), you can also use the lowest-level <a href="https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/External.java">External</a> class.</p> |
| <p><strong>Using an SDK wrapper</strong></p> |
| <p>To use a cross-language transform through an SDK wrapper, import the module for the SDK wrapper and call it from your pipeline, as shown in the example:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">org.apache.beam.sdk.extensions.python.transforms.DataframeTransform</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">input</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">DataframeTransform</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;lambda df: df.groupby(&#39;a&#39;).sum()&#34;</span><span class="o">).</span><span class="na">withIndexes</span><span class="o">())</span> |
| </span></span></code></pre></div><p><strong>Using the PythonExternalTransform class</strong></p> |
| <p>When an SDK-specific wrapper is not available, you can access the Python cross-language transform through the <code>PythonExternalTransform</code> class by specifying the fully qualified name and the constructor arguments of the target Python transform.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">input</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">PythonExternalTransform</span><span class="o">.&lt;</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Row</span><span class="o">&gt;,</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Row</span><span class="o">&gt;&gt;</span><span class="n">from</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;apache_beam.dataframe.transforms.DataframeTransform&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKwarg</span><span class="o">(</span><span class="s">&#34;func&#34;</span><span class="o">,</span> <span class="n">PythonCallableSource</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;lambda df: df.groupby(&#39;a&#39;).sum()&#34;</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKwarg</span><span class="o">(</span><span class="s">&#34;include_indexes&#34;</span><span class="o">,</span> <span class="kc">true</span><span class="o">))</span> |
| </span></span></code></pre></div><p><strong>Using the External class</strong></p> |
| <ol> |
| <li> |
| <p>Make sure you have any runtime environment dependencies (like the JRE) installed on your local machine (either directly on the local machine or available through a container). See the expansion service section for more details.</p> |
| <blockquote> |
| <p><strong>Note:</strong> When including Python transforms from within a Java pipeline, all Python dependencies have to be included in the SDK harness container.</p> |
| </blockquote> |
| </li> |
| <li> |
| <p>Start up the expansion service for the SDK that is in the language of the transform you&rsquo;re trying to consume, if not available.</p> |
| <p>Make sure the transform you are trying to use is available and can be used by the expansion service.</p> |
| </li> |
| <li> |
| <p>Include <a href="https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/External.java">External.of(&hellip;)</a> when instantiating your pipeline. Reference the URN, payload, and expansion service. For examples, see the <a href="https://github.com/apache/beam/blob/master/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java">cross-language transform test suite</a>.</p> |
| </li> |
| <li> |
| <p>After the job has been submitted to the Beam runner, shutdown the expansion service by terminating the expansion service process.</p> |
| </li> |
| </ol> |
| <h4 id="1322-using-cross-language-transforms-in-a-python-pipeline">13.2.2. Using cross-language transforms in a Python pipeline</h4> |
| <p>If a Python-specific wrapper for a cross-language transform is available, use that. Otherwise, you have to use the lower-level <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.ExternalTransform"><code>ExternalTransform</code></a> class to access the transform.</p> |
| <p><strong>Using an SDK wrapper</strong></p> |
| <p>To use a cross-language transform through an SDK wrapper, import the module for the SDK wrapper and call it from your pipeline, as shown in the example:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.io.kafka</span> <span class="kn">import</span> <span class="n">ReadFromKafka</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">kafka_records</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;ReadFromKafka&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ReadFromKafka</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">consumer_config</span><span class="o">=</span><span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;bootstrap.servers&#39;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">bootstrap_servers</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;auto.offset.reset&#39;</span><span class="p">:</span> <span class="s1">&#39;earliest&#39;</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">topics</span><span class="o">=</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">topic</span><span class="p">],</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">max_num_records</span><span class="o">=</span><span class="n">max_num_records</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">expansion_service</span><span class="o">=&lt;</span><span class="n">Address</span> <span class="n">of</span> <span class="n">expansion</span> <span class="n">service</span><span class="o">&gt;</span><span class="p">))</span> |
| </span></span></code></pre></div><p><strong>Using the ExternalTransform class</strong></p> |
| <p>When an SDK-specific wrapper isn&rsquo;t available, you will have to access the cross-language transform through the <code>ExternalTransform</code> class.</p> |
| <ol> |
| <li> |
| <p>Make sure you have any runtime environment dependencies (like the JRE) installed on your local machine. See the expansion service section for more details.</p> |
| </li> |
| <li> |
| <p>Start up the expansion service for the SDK that is in the language of the transform you&rsquo;re trying to consume, if not available. |
| Python provides several classes for automatically starting expansion java services such as |
| <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.JavaJarExpansionService">JavaJarExpansionService</a> |
| and <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.BeamJarExpansionService">BeamJarExpansionService</a> |
| which can be passed directly as an expansion service to <code>beam.ExternalTransform</code>. |
| Make sure the transform you&rsquo;re trying to use is available and can be used by the expansion service.</p> |
| <p>For Java, make sure the builder and registrar for the transform are available in the classpath of the expansion service.</p> |
| </li> |
| <li> |
| <p>Include <code>ExternalTransform</code> when instantiating your pipeline. Reference the URN, payload, and expansion service. |
| You can use one of the available <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.PayloadBuilder"><code>PayloadBuilder</code></a> classes to build the payload for <code>ExternalTransform</code>.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">with</span> <span class="n">pipeline</span> <span class="k">as</span> <span class="n">p</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">res</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">p</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span><span class="s1">&#39;a&#39;</span><span class="p">,</span> <span class="s1">&#39;b&#39;</span><span class="p">])</span><span class="o">.</span><span class="n">with_output_types</span><span class="p">(</span><span class="n">unicode</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">ExternalTransform</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TEST_PREFIX_URN</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ImplicitSchemaPayloadBuilder</span><span class="p">({</span><span class="s1">&#39;data&#39;</span><span class="p">:</span> <span class="s1">&#39;0&#39;</span><span class="p">}),</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">&lt;</span><span class="n">expansion</span> <span class="n">service</span><span class="o">&gt;</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">assert_that</span><span class="p">(</span><span class="n">res</span><span class="p">,</span> <span class="n">equal_to</span><span class="p">([</span><span class="s1">&#39;0a&#39;</span><span class="p">,</span> <span class="s1">&#39;0b&#39;</span><span class="p">]))</span> |
| </span></span></code></pre></div><p>For additional examples, see <a href="https://github.com/apache/beam/blob/master/examples/multi-language/python/addprefix.py">addprefix.py</a> and <a href="https://github.com/apache/beam/blob/master/examples/multi-language/python/javacount.py">javacount.py</a>.</p> |
| </li> |
| <li> |
| <p>After the job has been submitted to the Beam runner, shut down any manually started expansion services by terminating the expansion service process.</p> |
| </li> |
| </ol> |
| <p><strong>Using the JavaExternalTransform class</strong></p> |
| <p>Python has the ability to invoke Java-defined transforms via <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.external.html#apache_beam.transforms.external.JavaExternalTransform">proxy objects</a> |
| as if they were Python transforms. |
| These are invoked as follows</p> |
| <pre><code>```py |
| MyJavaTransform = beam.JavaExternalTransform('fully.qualified.ClassName', classpath=[jars]) |
| with pipeline as p: |
| res = ( |
| p |
| | beam.Create(['a', 'b']).with_output_types(unicode) |
| | MyJavaTransform(javaConstructorArg, ...).builderMethod(...) |
| assert_that(res, equal_to(['0a', '0b'])) |
| ``` |
| </code></pre> |
| <p>Python&rsquo;s <code>getattr</code> method can be used if the method names in java are reserved |
| Python keywords such as <code>from</code>.</p> |
| <p>As with other external transforms, either a pre-started expansion service can |
| be provided, or jar files that include the transform, its dependencies, and |
| Beam&rsquo;s expansion service in which case an expansion service will be auto-started.</p> |
| <h4 id="1323-using-cross-language-transforms-in-a-go-pipeline">13.2.3. Using cross-language transforms in a Go pipeline</h4> |
| <p>If a Go-specific wrapper for a cross-language is available, use that. Otherwise, you have to use the |
| lower-level <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam#CrossLanguage">CrossLanguage</a> |
| function to access the transform.</p> |
| <p><strong>Expansion Services</strong></p> |
| <p>The Go SDK supports automatically starting Java expansion services if an expansion address is not provided, although this is slower than |
| providing a persistent expansion service. Many wrapped Java transforms manage perform this automatically; if you wish to do this manually, use the <code>xlangx</code> package&rsquo;s |
| <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.40.0/go/pkg/beam/core/runtime/xlangx#UseAutomatedJavaExpansionService">UseAutomatedJavaExpansionService()</a> function. In order to use Python cross-language transforms, you must manually start any necessary expansion |
| services on your local machine and ensure they are accessible to your code during pipeline construction.</p> |
| <p><strong>Using an SDK wrapper</strong></p> |
| <p>To use a cross-language transform through an SDK wrapper, import the package for the SDK wrapper |
| and call it from your pipeline as shown in the example:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">import</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;github.com/apache/beam/sdks/v2/go/pkg/beam/io/xlang/kafkaio&#34;</span> |
| </span></span><span class="line"><span class="cl"><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Kafka Read using previously defined values. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nx">kafkaRecords</span> <span class="o">:=</span> <span class="nx">kafkaio</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">s</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">expansionAddr</span><span class="p">,</span> <span class="c1">// Address of expansion service. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">bootstrapAddr</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="nx">topicName</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">kafkaio</span><span class="p">.</span><span class="nf">MaxNumRecords</span><span class="p">(</span><span class="nx">numRecords</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">kafkaio</span><span class="p">.</span><span class="nf">ConsumerConfigs</span><span class="p">(</span><span class="kd">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kt">string</span><span class="p">{</span><span class="s">&#34;auto.offset.reset&#34;</span><span class="p">:</span> <span class="s">&#34;earliest&#34;</span><span class="p">}))</span> |
| </span></span></code></pre></div><p><strong>Using the CrossLanguage function</strong></p> |
| <p>When an SDK-specific wrapper isn&rsquo;t available, you will have to access the cross-language transform through the <code>beam.CrossLanguage</code> function.</p> |
| <ol> |
| <li> |
| <p>Make sure you have the appropriate expansion service running. See the expansion service section for details.</p> |
| </li> |
| <li> |
| <p>Make sure the transform you&rsquo;re trying to use is available and can be used by the expansion service. |
| Refer to <a href="#create-x-lang-transforms">Creating cross-language transforms</a> for details.</p> |
| </li> |
| <li> |
| <p>Use the <code>beam.CrossLanguage</code> function in your pipeline as appropriate. Reference the URN, payload, |
| expansion service address, and define inputs and outputs. You can use the |
| <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam#CrossLanguagePayload">beam.CrossLanguagePayload</a> |
| function as a helper for encoding a payload. You can use the |
| <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam#UnnamedInput">beam.UnnamedInput</a> and |
| <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam#UnnamedOutput">beam.UnnamedOutput</a> |
| functions as shortcuts for single, unnamed inputs/outputs or define a map for named ones.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">prefixPayload</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Data</span> <span class="kt">string</span> <span class="s">`beam:&#34;data&#34;`</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="nx">urn</span> <span class="o">:=</span> <span class="s">&#34;beam:transforms:xlang:test:prefix&#34;</span> |
| </span></span><span class="line"><span class="cl"><span class="nx">payload</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">CrossLanguagePayload</span><span class="p">(</span><span class="nx">prefixPayload</span><span class="p">{</span><span class="nx">Data</span><span class="p">:</span> <span class="nx">prefix</span><span class="p">})</span> |
| </span></span><span class="line"><span class="cl"><span class="nx">expansionAddr</span> <span class="o">:=</span> <span class="s">&#34;localhost:8097&#34;</span> |
| </span></span><span class="line"><span class="cl"><span class="nx">outT</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">UnnamedOutput</span><span class="p">(</span><span class="nx">typex</span><span class="p">.</span><span class="nf">New</span><span class="p">(</span><span class="nx">reflectx</span><span class="p">.</span><span class="nx">String</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"><span class="nx">res</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">CrossLanguage</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">urn</span><span class="p">,</span> <span class="nx">payload</span><span class="p">,</span> <span class="nx">expansionAddr</span><span class="p">,</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">UnnamedInput</span><span class="p">(</span><span class="nx">inputPCol</span><span class="p">),</span> <span class="nx">outT</span><span class="p">)</span> |
| </span></span></code></pre></div></li> |
| <li> |
| <p>After the job has been submitted to the Beam runner, shutdown the expansion service by |
| terminating the expansion service process.</p> |
| </li> |
| </ol> |
| <h4 id="1324-using-cross-language-transforms-in-a-typescript-pipeline">13.2.4. Using cross-language transforms in a Typescript pipeline</h4> |
| <p>Using a Typescript wrapper for a cross-language pipeline is similar to using any |
| other transform, provided the dependencies (e.g. a recent Python interpreter or |
| a Java JRE) is available. For example, most of the Typescript IOs are simply |
| wrappers around Beam transforms from other languages.</p> |
| <p class="paragraph-wrap">If a wrapper is not already available, one can use it explicitly using |
| <a href="https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/transforms/external.ts" target="_blank" rel="noopener noreferrer">apache_beam.transforms.external.rawExternalTransform</a>. |
| which takes a `urn` (a string identifying the transform), |
| a `payload` (a binary or json object parameterizing the transform), |
| and a `expansionService` which can either be an address of a pre-started service |
| or a callable returning an auto-started expansion service object.</p> |
| <p>For example, one could write</p> |
| <pre tabindex="0"><code>pcoll.applyAsync( |
| rawExternalTransform( |
| &#34;beam:registered:urn&#34;, |
| {arg: value}, |
| &#34;localhost:expansion_service_port&#34; |
| ) |
| ); |
| </code></pre><p>Note that <code>pcoll</code> must have a cross-language compatible coder coder such as <code>SchemaCoder</code>. |
| This can be ensured with the <a href="https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/transforms/internal.ts">withCoderInternal</a> |
| or <a href="https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/transforms/internal.ts">withRowCoder</a> |
| transforms, e.g.</p> |
| <pre tabindex="0"><code> |
| const result = pcoll.apply( |
| beam.withRowCoder({ intFieldName: 0, stringFieldName: &#34;&#34; }) |
| ); |
| </code></pre><p>Coder can also be specified on the output if it cannot be inferred, e.g.</p> |
| <p>In addition, there are several utilities such as <a href="https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/transforms/python.ts">pythonTransform</a> |
| that make it easier to invoke transforms from specific languages:</p> |
| <pre tabindex="0"><code> |
| const result: PCollection&lt;number&gt; = await pcoll |
| .apply( |
| beam.withName(&#34;UpdateCoder1&#34;, beam.withRowCoder({ a: 0, b: 0 })) |
| ) |
| .applyAsync( |
| pythonTransform( |
| // Fully qualified name |
| &#34;apache_beam.transforms.Map&#34;, |
| // Positional arguments |
| [pythonCallable(&#34;lambda x: x.a &#43; x.b&#34;)], |
| // Keyword arguments |
| {}, |
| // Output type if it cannot be inferred |
| { requestedOutputCoders: { output: new VarIntCoder() } } |
| ) |
| ); |
| </code></pre><p>Cross-language transforms can also be defined in line, which can be useful |
| for accessing features or libraries not available in the calling SDK</p> |
| <pre tabindex="0"><code> |
| const result: PCollection&lt;string&gt; = await pcoll |
| .apply(withCoderInternal(new StrUtf8Coder())) |
| .applyAsync( |
| pythonTransform( |
| // Define an arbitrary transform from a callable. |
| &#34;__callable__&#34;, |
| [ |
| pythonCallable(` |
| def apply(pcoll, prefix, postfix): |
| return pcoll | beam.Map(lambda s: prefix &#43; s &#43; postfix) |
| `), |
| ], |
| // Keyword arguments to pass above, if any. |
| { prefix: &#34;x&#34;, postfix: &#34;y&#34; }, |
| // Output type if it cannot be inferred |
| { requestedOutputCoders: { output: new StrUtf8Coder() } } |
| ) |
| ); |
| </code></pre><h3 id="x-lang-transform-runner-support">13.3. Runner Support</h3> |
| <p>Currently, portable runners such as Flink, Spark, and the direct runner can be used with multi-language pipelines.</p> |
| <p>Dataflow supports multi-language pipelines through the Dataflow Runner v2 backend architecture.</p> |
| <h3 id="x-lang-transform-tips-troubleshooting">13.4 Tips and Troubleshooting</h3> |
| <p>For additional tips and troubleshooting information, see <a href="https://cwiki.apache.org/confluence/display/BEAM/Multi-language+Pipelines+Tips">here</a>.</p> |
| <h2 id="batched-dofns">14 Batched DoFns</h2> |
| <nav class="language-switcher"> |
| <strong>Adapt for:</strong> |
| <ul> |
| <li data-value="java" class="active">Java SDK</li> |
| <li data-value="py">Python SDK</li> |
| <li data-value="go">Go SDK</li> |
| <li data-value="typescript">TypeScript SDK</li> |
| </ul> |
| </nav> |
| <p class="language-go language-java language-typescript">Batched DoFns are currently a Python-only feature.</p> |
| <p class="language-py">Batched DoFns enable users to create modular, composable components that |
| operate on batches of multiple logical elements. These DoFns can leverage |
| vectorized Python libraries, like numpy, scipy, and pandas, which operate on |
| batches of data for efficiency.</p> |
| <h3 id="batched-dofn-basics">14.1 Basics</h3> |
| <p class="language-go language-java language-typescript">Batched DoFns are currently a Python-only feature.</p> |
| <p class="language-py">A trivial Batched DoFn might look like this:</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">MultiplyByTwo</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Type</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process_batch</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">batch</span><span class="p">:</span> <span class="n">np</span><span class="o">.</span><span class="n">ndarray</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Iterator</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ndarray</span><span class="p">]:</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">batch</span> <span class="o">*</span> <span class="mi">2</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Declare what the element-wise output type is</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">infer_output_type</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">input_element_type</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">input_element_type</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-py">This DoFn can be used in a Beam pipeline that otherwise operates on individual |
| elements. Beam will implicitly buffer elements and create numpy arrays on the |
| input side, and on the output side it will explode the numpy arrays back into |
| individual elements:</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="p">(</span><span class="n">p</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">])</span><span class="o">.</span><span class="n">with_output_types</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">int64</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">MultiplyByTwo</span><span class="p">())</span> <span class="c1"># Implicit buffering and batch creation</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="o">/</span><span class="mi">3</span><span class="p">))</span> <span class="c1"># Implicit batch explosion</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-py">Note that we use |
| <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform.with_output_types"><code>PTransform.with_output_types</code></a> to |
| set the <em>element-wise</em> typehint for the output of <code>beam.Create</code>. Then, when |
| <code>MultiplyByTwo</code> is applied to this <code>PCollection</code>, Beam recognizes that |
| <code>np.ndarray</code> is an acceptable batch type to use in conjunction with <code>np.int64</code> |
| elements. We will use numpy typehints like these throughout this guide, but |
| Beam supports typehints from other libraries as well, see <a href="#batched-dofn-types">Supported Batch |
| Types</a>.</p> |
| <p class="language-py">In the previous case, Beam will implicitly create and explode batches at the |
| input and output boundaries. However, if Batched DoFns with equivalent types are |
| chained together, this batch creation and explosion will be elided. The batches |
| will be passed straight through! This makes it much simpler to efficiently |
| compose transforms that operate on batches.</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="p">(</span><span class="n">p</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">])</span><span class="o">.</span><span class="n">with_output_types</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">int64</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">MultiplyByTwo</span><span class="p">())</span> <span class="c1"># Implicit buffering and batch creation</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">MultiplyByTwo</span><span class="p">())</span> <span class="c1"># Batches passed through</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">MultiplyByTwo</span><span class="p">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="batched-dofn-elementwise">14.2 Element-wise Fallback</h3> |
| <p class="language-go language-java language-typescript">Batched DoFns are currently a Python-only feature.</p> |
| <p class="language-py">For some DoFns you may be able to provide both a batched and an element-wise |
| implementation of your desired logic. You can do this by simply defining both |
| <code>process</code> and <code>process_batch</code>:</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">MultiplyByTwo</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">:</span> <span class="n">np</span><span class="o">.</span><span class="n">int64</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Iterator</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">int64</span><span class="p">]:</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Multiply an individual int64 by 2</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">element</span> <span class="o">*</span> <span class="mi">2</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process_batch</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">batch</span><span class="p">:</span> <span class="n">np</span><span class="o">.</span><span class="n">ndarray</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Iterator</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ndarray</span><span class="p">]:</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Multiply a _batch_ of int64s by 2</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">batch</span> <span class="o">*</span> <span class="mi">2</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-py">When executing this DoFn, Beam will select the best implementation to use given |
| the context. Generally, if the inputs to a DoFn are already batched Beam will |
| use the batched implementation; otherwise it will use the element-wise |
| implementation defined in the <code>process</code> method.</p> |
| <p class="language-py">Note that, in this case, there is no need to define <code>infer_output_type</code>. This is |
| because Beam can get the output type from the typehint on <code>process</code>.</p> |
| <h3 id="batched-dofn-batch-production">14.3 Batch Production vs. Batch Consumption</h3> |
| <p class="language-go language-java language-typescript">Batched DoFns are currently a Python-only feature.</p> |
| <p class="language-py">By convention, Beam assumes that the <code>process_batch</code> method, which consumes |
| batched inputs, will also produce batched outputs. Similarly, Beam assumes the |
| <code>process</code> method will produce individual elements. This can be overridden with |
| the <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.yields_elements"><code>@beam.DoFn.yields_elements</code></a> and |
| <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.yields_batches"><code>@beam.DoFn.yields_batches</code></a> decorators. For example:</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># Consumes elements, produces batches</span> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">ReadFromFile</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@beam.DoFn.yields_batches</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Iterator</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ndarray</span><span class="p">]:</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">array</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Declare what the element-wise output type is</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">infer_output_type</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">int64</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Consumes batches, produces elements</span> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">WriteToFile</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@beam.DoFn.yields_elements</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process_batch</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">batch</span><span class="p">:</span> <span class="n">np</span><span class="o">.</span><span class="n">ndarray</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Iterator</span><span class="p">[</span><span class="nb">str</span><span class="p">]:</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">output_path</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="batched-dofn-types">14.4 Supported Batch Types</h3> |
| <p class="language-go language-java language-typescript">Batched DoFns are currently a Python-only feature.</p> |
| <p class="language-py">We’ve used numpy types in the Batched DoFn implementations in this guide – |
| <code>np.int64 </code> as the element typehint and <code>np.ndarray</code> as the corresponding |
| batch typehint – but Beam supports typehints from other libraries as well.</p> |
| <h4 id="numpyhttpsgithubcomapachebeamblobmastersdkspythonapache_beamtypehintsbatchpy"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/typehints/batch.py">numpy</a></h4> |
| <table> |
| <thead> |
| <tr> |
| <th>Element Typehint</th> |
| <th>Batch Typehint</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>Numeric types (<code>int</code>, <code>np.int32</code>, <code>bool</code>, &hellip;)</td> |
| <td>np.ndarray (or NumpyArray)</td> |
| </tr> |
| </tbody> |
| </table> |
| <h4 id="pandashttpsgithubcomapachebeamblobmastersdkspythonapache_beamtypehintspandas_type_compatibilitypy"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/typehints/pandas_type_compatibility.py">pandas</a></h4> |
| <table> |
| <thead> |
| <tr> |
| <th>Element Typehint</th> |
| <th>Batch Typehint</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>Numeric types (<code>int</code>, <code>np.int32</code>, <code>bool</code>, &hellip;)</td> |
| <td><code>pd.Series</code></td> |
| </tr> |
| <tr> |
| <td><code>bytes</code></td> |
| <td></td> |
| </tr> |
| <tr> |
| <td><code>Any</code></td> |
| <td></td> |
| </tr> |
| <tr> |
| <td><a href="#schemas">Beam Schema Types</a></td> |
| <td><code>pd.DataFrame</code></td> |
| </tr> |
| </tbody> |
| </table> |
| <h4 id="pyarrowhttpsgithubcomapachebeamblobmastersdkspythonapache_beamtypehintsarrow_type_compatibilitypy"><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/typehints/arrow_type_compatibility.py">pyarrow</a></h4> |
| <table> |
| <thead> |
| <tr> |
| <th>Element Typehint</th> |
| <th>Batch Typehint</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>Numeric types (<code>int</code>, <code>np.int32</code>, <code>bool</code>, &hellip;)</td> |
| <td><code>pd.Series</code></td> |
| </tr> |
| <tr> |
| <td><code>Any</code></td> |
| <td></td> |
| </tr> |
| <tr> |
| <td><code>List</code></td> |
| <td></td> |
| </tr> |
| <tr> |
| <td><code>Mapping</code></td> |
| <td></td> |
| </tr> |
| <tr> |
| <td><a href="#schemas">Beam Schema Types</a></td> |
| <td><code>pa.Table</code></td> |
| </tr> |
| </tbody> |
| </table> |
| <h4 id="other-types">Other types?</h4> |
| <p>If there are other batch types you would like to use with Batched DoFns, please |
| <a href="https://github.com/apache/beam/issues/new/choose">file an issue</a>.</p> |
| <h3 id="batched-dofn-dynamic-types">14.5 Dynamic Batch Input and Output Types</h3> |
| <p class="language-go language-java language-typescript">Batched DoFns are currently a Python-only feature.</p> |
| <p class="language-py">For some Batched DoFns, it may not be sufficient to declare batch types |
| statically, with typehints on <code>process</code> and/or <code>process_batch</code>. You may need to |
| declare these types dynamically. You can do this by overriding the |
| <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.get_input_batch_type"><code>get_input_batch_type</code></a> |
| and |
| <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn.get_output_batch_type"><code>get_output_batch_type</code></a> |
| methods on your DoFn:</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># Utilize Beam&#39;s parameterized NumpyArray typehint</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.typehints.batch</span> <span class="kn">import</span> <span class="n">NumpyArray</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">MultipyByTwo</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># No typehints needed</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process_batch</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">batch</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">batch</span> <span class="o">*</span> <span class="mi">2</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">get_input_batch_type</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">input_element_type</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">NumpyArray</span><span class="p">[</span><span class="n">input_element_type</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">get_output_batch_type</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">input_element_type</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">NumpyArray</span><span class="p">[</span><span class="n">input_element_type</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">infer_output_type</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">input_element_type</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">input_element_type</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="batched-dofn-event-time">14.6 Batches and Event-time Semantics</h3> |
| <p class="language-go language-java language-typescript">Batched DoFns are currently a Python-only feature.</p> |
| <p class="language-py">Currently, batches must have a single set of timing information (event time, |
| windows, etc&hellip;) that applies to every logical element in the batch. There is |
| currently no mechanism to create batches that span multiple timestamps. However, |
| it is possible to retrieve this timing information in Batched DoFn |
| implementations. This information can be accessed by using the conventional |
| <code>DoFn.*Param</code> attributes:</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">RetrieveTimingDoFn</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process_batch</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">batch</span><span class="p">:</span> <span class="n">np</span><span class="o">.</span><span class="n">ndarray</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timestamp</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">TimestampParam</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pane_info</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="o">.</span><span class="n">PaneInfoParam</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Iterator</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ndarray</span><span class="p">]:</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">infer_output_type</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">input_type</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">input_type</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h2 id="transform-service">15 Transform service</h2> |
| <p>The Apache Beam SDK versions 2.49.0 and later include a <a href="https://docs.docker.com/compose/">Docker Compose</a> |
| service named <em>Transform service</em>.</p> |
| <p>The following diagram illustrates the basic architecture of the Transform service.</p> |
| <p><img src="/images/transform_service.png" alt="Diagram of the Transform service architecture"></p> |
| <p>To use the Transform service, Docker must be available on the machine that starts the service.</p> |
| <p>The Transform service has several primary use cases.</p> |
| <h3 id="transform-service-usage-upgrade">15.1 Using the transform service to upgrade transforms</h3> |
| <p>Transform service can be used to upgrade (or downgrade) the Beam SDK versions of supported individual transforms used by Beam pipelines without changing the Beam version of the pipelines. |
| This feature is currently only available for Beam Java SDK 2.53.0 and later. Currently, the following transforms are available for upgrading:</p> |
| <ul> |
| <li>BigQuery read transform (URN: <em>beam:transform:org.apache.beam:bigquery_read:v1</em>)</li> |
| <li>BigQuery write transform (URN: <em>beam:transform:org.apache.beam:bigquery_write:v1</em>)</li> |
| <li>Kafka read transform (URN: <em>beam:transform:org.apache.beam:kafka_read_with_metadata:v2</em>)</li> |
| <li>Kafka write transform (URN: <em>beam:transform:org.apache.beam:kafka_write:v2</em>)</li> |
| </ul> |
| <p>To use this feature, you can simply execute a Java pipeline with additional pipeline options that specify the URNs of the transforms you would like to upgrade and the Beam version you would like to upgrade the transforms to. All transforms in the pipeline with matching URNs will be upgraded.</p> |
| <p>For example, to upgrade the BigQuery read transform for a pipeline run using Beam <code>2.53.0</code> to a future Beam version <code>2.xy.z</code>, you can specify the following additional pipelines options.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="o">--</span><span class="n">transformsToOverride</span><span class="o">=</span><span class="n">beam</span><span class="o">:</span><span class="n">transform</span><span class="o">:</span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">beam</span><span class="o">:</span><span class="n">bigquery_read</span><span class="o">:</span><span class="n">v1</span> <span class="o">--</span><span class="n">transformServiceBeamVersion</span><span class="o">=</span><span class="n">2</span><span class="o">.</span><span class="na">xy</span><span class="o">.</span><span class="na">z</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">This</span> <span class="n">feature</span> <span class="ow">is</span> <span class="n">currently</span> <span class="ow">not</span> <span class="n">available</span> <span class="k">for</span> <span class="n">Python</span> <span class="n">SDK</span><span class="o">.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">This</span> <span class="nx">feature</span> <span class="nx">is</span> <span class="nx">currently</span> <span class="nx">not</span> <span class="nx">available</span> <span class="k">for</span> <span class="nx">Go</span> <span class="nx">SDK</span><span class="p">.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Note that the framework will automatically download the relevant Docker containers and startup the transform service for you.</p> |
| <p>Please see <a href="https://cwiki.apache.org/confluence/display/BEAM/Transform+Service#TransformService-Upgradetransformswithoutupgradingthepipeline">here</a> for a full example that uses this feature to upgrade BigQuery read and write transforms.</p> |
| <h3 id="transform-service-usage-multi-language">15.2 Using the Transform service for multi-language pipelines</h3> |
| <p>Transform service implements the Beam expansion API. This allows Beam multi-language pipelines to use the transform service when expanding transforms available within the transform service. |
| The main advantage here is that multi-language pipelines will be able to operate without installing support for additional language runtimes. For example, Beam Python pipelines that use Java transforms such as |
| <code>KafkaIO</code> can operate without installing Java locally during job submission as long as Docker is available in the system.</p> |
| <p>In some cases, Apache Beam SDKs can automatically start the Transform service.</p> |
| <ul> |
| <li> |
| <p>The Java <a href="https://github.com/apache/beam/blob/master/sdks/java/extensions/python/src/main/java/org/apache/beam/sdk/extensions/python/PythonExternalTransform.java"><code>PythonExternalTransform</code> API</a> automatically |
| starts the Transform service when a Python runtime isn&rsquo;t available locally, but Docker is.</p> |
| </li> |
| <li> |
| <p>The Apache Beam Python multi-language wrappers might automatically start the Transform service when you&rsquo;re using Java transforms, a Java language runtime isn&rsquo;t available locally, and Docker is available locally.</p> |
| </li> |
| </ul> |
| <p>Beam users also have the option to <a href="/documentation/programming-guide/#transform-service-usage-muanual">manually start</a> a transform service and use that as the expansion service used by multi-language pipelines.</p> |
| <h3 id="transform-service-usage-muanual">15.3 Manually starting the transform service</h3> |
| <p>A Beam Transform service instance can be manually started by using utilities provided with Apache Beam SDKs.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">java</span> <span class="o">-</span><span class="n">jar</span> <span class="n">beam</span><span class="o">-</span><span class="n">sdks</span><span class="o">-</span><span class="n">java</span><span class="o">-</span><span class="n">transform</span><span class="o">-</span><span class="n">service</span><span class="o">-</span><span class="n">app</span><span class="o">-&lt;</span><span class="n">Beam</span> <span class="n">version</span> <span class="k">for</span> <span class="n">the</span> <span class="n">jar</span><span class="o">&gt;.</span><span class="na">jar</span> <span class="o">--</span><span class="n">port</span> <span class="o">&lt;</span><span class="n">port</span><span class="o">&gt;</span> <span class="o">--</span><span class="n">beam_version</span> <span class="o">&lt;</span><span class="n">Beam</span> <span class="n">version</span> <span class="k">for</span> <span class="n">the</span> <span class="n">transform</span> <span class="n">service</span><span class="o">&gt;</span> <span class="o">--</span><span class="n">project_name</span> <span class="o">&lt;</span><span class="n">a</span> <span class="n">unique</span> <span class="n">ID</span> <span class="k">for</span> <span class="n">the</span> <span class="n">transform</span> <span class="n">service</span><span class="o">&gt;</span> <span class="o">--</span><span class="n">command</span> <span class="n">up</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">apache_beam</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">transform_service_launcher</span> <span class="o">--</span><span class="n">port</span> <span class="o">&lt;</span><span class="n">port</span><span class="o">&gt;</span> <span class="o">--</span><span class="n">beam_version</span> <span class="o">&lt;</span><span class="n">Beam</span> <span class="n">version</span> <span class="k">for</span> <span class="n">the</span> <span class="n">transform</span> <span class="n">service</span><span class="o">&gt;</span> <span class="o">--</span><span class="n">project_name</span> <span class="o">&lt;</span><span class="n">a</span> <span class="n">unique</span> <span class="n">ID</span> <span class="k">for</span> <span class="n">the</span> <span class="n">transform</span> <span class="n">service</span><span class="o">&gt;</span> <span class="o">--</span><span class="n">command</span> <span class="n">up</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">This</span> <span class="nx">feature</span> <span class="nx">is</span> <span class="nx">currently</span> <span class="nx">in</span> <span class="nx">development</span><span class="p">.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>To stop the transform service, use the following commands.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">java</span> <span class="o">-</span><span class="n">jar</span> <span class="n">beam</span><span class="o">-</span><span class="n">sdks</span><span class="o">-</span><span class="n">java</span><span class="o">-</span><span class="n">transform</span><span class="o">-</span><span class="n">service</span><span class="o">-</span><span class="n">app</span><span class="o">-&lt;</span><span class="n">Beam</span> <span class="n">version</span> <span class="k">for</span> <span class="n">the</span> <span class="n">jar</span><span class="o">&gt;.</span><span class="na">jar</span> <span class="o">--</span><span class="n">port</span> <span class="o">&lt;</span><span class="n">port</span><span class="o">&gt;</span> <span class="o">--</span><span class="n">beam_version</span> <span class="o">&lt;</span><span class="n">Beam</span> <span class="n">version</span> <span class="k">for</span> <span class="n">the</span> <span class="n">transform</span> <span class="n">service</span><span class="o">&gt;</span> <span class="o">--</span><span class="n">project_name</span> <span class="o">&lt;</span><span class="n">a</span> <span class="n">unique</span> <span class="n">ID</span> <span class="k">for</span> <span class="n">the</span> <span class="n">transform</span> <span class="n">service</span><span class="o">&gt;</span> <span class="o">--</span><span class="n">command</span> <span class="n">down</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">apache_beam</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">transform_service_launcher</span> <span class="o">--</span><span class="n">port</span> <span class="o">&lt;</span><span class="n">port</span><span class="o">&gt;</span> <span class="o">--</span><span class="n">beam_version</span> <span class="o">&lt;</span><span class="n">Beam</span> <span class="n">version</span> <span class="k">for</span> <span class="n">the</span> <span class="n">transform</span> <span class="n">service</span><span class="o">&gt;</span> <span class="o">--</span><span class="n">project_name</span> <span class="o">&lt;</span><span class="n">a</span> <span class="n">unique</span> <span class="n">ID</span> <span class="k">for</span> <span class="n">the</span> <span class="n">transform</span> <span class="n">service</span><span class="o">&gt;</span> <span class="o">--</span><span class="n">command</span> <span class="n">down</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">This</span> <span class="nx">feature</span> <span class="nx">is</span> <span class="nx">currently</span> <span class="nx">in</span> <span class="nx">development</span><span class="p">.</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="transform-service-included-transforms">15.4 Portable transforms included in the Transform service</h3> |
| <p>Beam Transform service includes a number of transforms implemented in the Apache Beam Java and Python SDKs.</p> |
| <p>Currently, the following transforms are included in the Transform service:</p> |
| <ul> |
| <li> |
| <p>Java transforms: Google Cloud I/O connectors, the Kafka I/O connector, and the JDBC I/O connector</p> |
| </li> |
| <li> |
| <p>Python transforms: all portable transforms implemented within the Apache Beam Python SDK, such as |
| <a href="/documentation/transforms/python/elementwise/runinference/">RunInference</a> and |
| <a href="/documentation/dsls/dataframes/overview/">DataFrame</a> transforms.</p> |
| </li> |
| </ul> |
| <p>For a more comprehensive list of available transforms, see the |
| <a href="https://cwiki.apache.org/confluence/display/BEAM/Transform+Service#TransformService-TransformsincludedintheTransformservice">Transform service</a> developer guide.</p></description></item><item><title>Documentation: BigQuery ML integration</title><link>/documentation/patterns/bqml/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/patterns/bqml/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="bigquery-ml-integration">BigQuery ML integration</h1> |
| <p>With the samples on this page we will demonstrate how to integrate models exported from <a href="https://cloud.google.com/bigquery-ml/docs">BigQuery ML (BQML)</a> into your Apache Beam pipeline using <a href="https://github.com/tensorflow/tfx-bsl">TFX Basic Shared Libraries (tfx_bsl)</a>.</p> |
| <p>Roughly, the sections below will go through the following steps in more detail:</p> |
| <ol> |
| <li>Create and train your BigQuery ML model</li> |
| <li>Export your BigQuery ML model</li> |
| <li>Create a transform that uses the brand-new BigQuery ML model</li> |
| </ol> |
| <nav class="language-switcher"> |
| <strong>Adapt for:</strong> |
| <ul> |
| <li data-value="java" class="active">Java SDK</li> |
| <li data-value="py">Python SDK</li> |
| </ul> |
| </nav> |
| <h2 id="create-and-train-your-bigquery-ml-model">Create and train your BigQuery ML model</h2> |
| <p>To be able to incorporate your BQML model into an Apache Beam pipeline using tfx_bsl, it has to be in the <a href="https://www.tensorflow.org/guide/saved_model">TensorFlow SavedModel</a> format. An overview that maps different model types to their export model format for BQML can be found <a href="https://cloud.google.com/bigquery-ml/docs/exporting-models#export_model_formats_and_samples">here</a>.</p> |
| <p>For the sake of simplicity, we&rsquo;ll be training a (simplified version of the) logistic regression model in the <a href="https://cloud.google.com/bigquery-ml/docs/bigqueryml-web-ui-start">BQML quickstart guide</a>, using the publicly available Google Analytics sample dataset (which is a <a href="https://cloud.google.com/bigquery/docs/partitioned-tables#dt_partition_shard">date-sharded table</a> - alternatively, you might encounter <a href="https://cloud.google.com/bigquery/docs/partitioned-tables">partitioned tables</a>). An overview of all models you can create using BQML can be found <a href="https://cloud.google.com/bigquery-ml/docs/introduction#supported_models_in">here</a>.</p> |
| <p>After creating a BigQuery dataset, you continue to create the model, which is fully defined in SQL:</p> |
| <pre tabindex="0"><code>CREATE MODEL IF NOT EXISTS `bqml_tutorial.sample_model` |
| OPTIONS(model_type=&#39;logistic_reg&#39;, input_label_cols=[&#34;label&#34;]) AS |
| SELECT |
| IF(totals.transactions IS NULL, 0, 1) AS label, |
| IFNULL(geoNetwork.country, &#34;&#34;) AS country |
| FROM |
| `bigquery-public-data.google_analytics_sample.ga_sessions_*` |
| WHERE |
| _TABLE_SUFFIX BETWEEN &#39;20160801&#39; AND &#39;20170630&#39; |
| </code></pre><p>The model will predict if a purchase will be made given the country of the visitor on data gathered between 2016-08-01 and 2017-06-30.</p> |
| <h2 id="export-your-bigquery-ml-model">Export your BigQuery ML model</h2> |
| <p>In order to incorporate your model in an Apache Beam pipeline, you will need to export it. Prerequisites to do so are <a href="https://cloud.google.com/bigquery/docs/bq-command-line-tool">installing the <code>bq</code> command-line tool</a> and <a href="https://cloud.google.com/storage/docs/creating-buckets">creating a Google Cloud Storage bucket</a> to store your exported model.</p> |
| <p>Export the model using the following command:</p> |
| <pre tabindex="0"><code>bq extract -m bqml_tutorial.sample_model gs://some/gcs/path |
| </code></pre><h2 id="create-an-apache-beam-transform-that-uses-your-bigquery-ml-model">Create an Apache Beam transform that uses your BigQuery ML model</h2> |
| <p>In this section we will construct an Apache Beam pipeline that will use the BigQuery ML model we just created and exported. The model can be served using Google Cloud AI Platform Prediction - for this please refer to the <a href="/documentation/patterns/ai-platform/">AI Platform patterns</a>. In this case, we&rsquo;ll be illustrating how to use the tfx_bsl library to do local predictions (on your Apache Beam workers).</p> |
| <p>First, the model needs to be downloaded to a local directory where you will be developing the rest of your pipeline (e.g. to <code>serving_dir/sample_model/1</code>).</p> |
| <p>Then, you can start developing your pipeline like you would normally do. We will be using the <code>RunInference</code> PTransform from the <a href="https://github.com/tensorflow/tfx-bsl">tfx_bsl</a> library, and we will point it to our local directory where the model is stored (see the <code>model_path</code> variable in the code example). The transform takes elements of the type <code>tf.train.Example</code> as inputs and outputs elements of the type <a href="https://github.com/tensorflow/serving/blob/master/tensorflow_serving/apis/prediction_log.proto"><code>tensorflow_serving.apis.prediction_log_pb2.PredictionLog</code></a>. Depending on the signature of your model, you can extract values from the output; in our case we extract <code>label_probs</code>, <code>label_values</code> and the <code>predicted_label</code> as per the <a href="https://cloud.google.com/bigquery-ml/docs/exporting-models#logistic_reg">docs on the logistic regression model</a> in the <code>extract_prediction</code> function.</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">apache_beam</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">tensorflow</span> <span class="k">as</span> <span class="nn">tf</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">google.protobuf</span> <span class="kn">import</span> <span class="n">text_format</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">tensorflow.python.framework</span> <span class="kn">import</span> <span class="n">tensor_util</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">tfx_bsl.beam</span> <span class="kn">import</span> <span class="n">run_inference</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">tfx_bsl.public.beam</span> <span class="kn">import</span> <span class="n">RunInference</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">tfx_bsl.public.proto</span> <span class="kn">import</span> <span class="n">model_spec_pb2</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">inputs</span> <span class="o">=</span> <span class="n">tf</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">Example</span><span class="p">(</span><span class="n">features</span><span class="o">=</span><span class="n">tf</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">Features</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">feature</span><span class="o">=</span><span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;os&#39;</span><span class="p">:</span> <span class="n">tf</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">Feature</span><span class="p">(</span><span class="n">bytes_list</span><span class="o">=</span><span class="n">tf</span><span class="o">.</span><span class="n">train</span><span class="o">.</span><span class="n">BytesList</span><span class="p">(</span><span class="sa">b</span><span class="s2">&#34;Microsoft&#34;</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">})</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">model_path</span> <span class="o">=</span> <span class="s2">&#34;serving_dir/sample_model/1&#34;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">extract_prediction</span><span class="p">(</span><span class="n">response</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">response</span><span class="o">.</span><span class="n">predict_log</span><span class="o">.</span><span class="n">response</span><span class="o">.</span><span class="n">outputs</span><span class="p">[</span><span class="s1">&#39;label_values&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">string_val</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">tensor_util</span><span class="o">.</span><span class="n">MakeNdarray</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">predict_log</span><span class="o">.</span><span class="n">response</span><span class="o">.</span><span class="n">outputs</span><span class="p">[</span><span class="s1">&#39;label_probs&#39;</span><span class="p">]),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">response</span><span class="o">.</span><span class="n">predict_log</span><span class="o">.</span><span class="n">response</span><span class="o">.</span><span class="n">outputs</span><span class="p">[</span><span class="s1">&#39;predicted_label&#39;</span><span class="p">]</span><span class="o">.</span><span class="n">string_val</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">p</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">res</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">p</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span><span class="n">inputs</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">RunInference</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">model_spec_pb2</span><span class="o">.</span><span class="n">InferenceSpecType</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">saved_model_spec</span><span class="o">=</span><span class="n">model_spec_pb2</span><span class="o">.</span><span class="n">SavedModelSpec</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">model_path</span><span class="o">=</span><span class="n">model_path</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">signature_name</span><span class="o">=</span><span class="p">[</span><span class="s1">&#39;serving_default&#39;</span><span class="p">])))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">extract_prediction</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">Implemented</span> <span class="n">in</span> <span class="n">Python</span><span class="o">.</span></span></span></code></pre></div> |
| </div> |
| </div></description></item><item><title>Documentation: BigQuery patterns</title><link>/documentation/patterns/bigqueryio/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/patterns/bigqueryio/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="google-bigquery-patterns">Google BigQuery patterns</h1> |
| <p>The samples on this page show you common patterns for use with BigQueryIO.</p> |
| <nav class="language-switcher"> |
| <strong>Adapt for:</strong> |
| <ul> |
| <li data-value="java" class="active">Java SDK</li> |
| <li data-value="py">Python SDK</li> |
| </ul> |
| </nav> |
| <h2 id="bigqueryio-deadletter-pattern">BigQueryIO deadletter pattern</h2> |
| <p>In production systems, it is useful to implement the deadletter pattern with BigQueryIO outputting any elements which had errors during processing by BigQueryIO into another PCollection for further processing. |
| The samples below print the errors, but in a production system they can be sent to a deadletter table for later correction.</p> |
| <p class="language-java">When using <code>STREAMING_INSERTS</code> you can use the <code>WriteResult</code> object to access a <code>PCollection</code> with the <code>TableRows</code> that failed to be inserted into BigQuery. |
| If you also set the <code>withExtendedErrorInfo</code> property , you will be able to access a <code>PCollection&lt;BigQueryInsertError&gt;</code> from the <code>WriteResult</code>. The <code>PCollection</code> will then include a reference to the table, the data row and the <code>InsertErrors</code>. Which errors are added to the deadletter queue is determined via the <code>InsertRetryPolicy</code>.</p> |
| <p class="language-py">In the result tuple you can access <code>FailedRows</code> to access the failed inserts.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="n">PipelineOptions</span> <span class="n">options</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">PipelineOptionsFactory</span><span class="o">.</span><span class="na">fromArgs</span><span class="o">(</span><span class="n">args</span><span class="o">).</span><span class="na">withValidation</span><span class="o">().</span><span class="na">as</span><span class="o">(</span><span class="n">BigQueryOptions</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">Pipeline</span> <span class="n">p</span> <span class="o">=</span> <span class="n">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Create a bug by writing the 2nd value as null. The API will correctly |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// throw an error when trying to insert a null value into a REQUIRED field. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">WriteResult</span> <span class="n">result</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Create</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">1</span><span class="o">,</span> <span class="n">2</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">BigQueryIO</span><span class="o">.&lt;</span><span class="n">Integer</span><span class="o">&gt;</span><span class="n">write</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withSchema</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">TableSchema</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">setFields</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ImmutableList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">TableFieldSchema</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">setName</span><span class="o">(</span><span class="s">&#34;num&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">setType</span><span class="o">(</span><span class="s">&#34;INTEGER&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">setMode</span><span class="o">(</span><span class="s">&#34;REQUIRED&#34;</span><span class="o">))))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">to</span><span class="o">(</span><span class="s">&#34;Test.dummyTable&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withFormatFunction</span><span class="o">(</span><span class="n">x</span> <span class="o">-&gt;</span> <span class="k">new</span> <span class="n">TableRow</span><span class="o">().</span><span class="na">set</span><span class="o">(</span><span class="s">&#34;num&#34;</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span> <span class="o">==</span> <span class="n">2</span><span class="o">)</span> <span class="o">?</span> <span class="kc">null</span> <span class="o">:</span> <span class="n">x</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withFailedInsertRetryPolicy</span><span class="o">(</span><span class="n">InsertRetryPolicy</span><span class="o">.</span><span class="na">retryTransientErrors</span><span class="o">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Forcing the bounded pipeline to use streaming inserts |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">withMethod</span><span class="o">(</span><span class="n">BigQueryIO</span><span class="o">.</span><span class="na">Write</span><span class="o">.</span><span class="na">Method</span><span class="o">.</span><span class="na">STREAMING_INSERTS</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// set the withExtendedErrorInfo property. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">withExtendedErrorInfo</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withCreateDisposition</span><span class="o">(</span><span class="n">BigQueryIO</span><span class="o">.</span><span class="na">Write</span><span class="o">.</span><span class="na">CreateDisposition</span><span class="o">.</span><span class="na">CREATE_IF_NEEDED</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withWriteDisposition</span><span class="o">(</span><span class="n">BigQueryIO</span><span class="o">.</span><span class="na">Write</span><span class="o">.</span><span class="na">WriteDisposition</span><span class="o">.</span><span class="na">WRITE_APPEND</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">result</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">getFailedInsertsWithErr</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">via</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">x</span> <span class="o">-&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">&#34; The table was &#34;</span> <span class="o">+</span> <span class="n">x</span><span class="o">.</span><span class="na">getTable</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">&#34; The row was &#34;</span> <span class="o">+</span> <span class="n">x</span><span class="o">.</span><span class="na">getRow</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">&#34; The error was &#34;</span> <span class="o">+</span> <span class="n">x</span><span class="o">.</span><span class="na">getError</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="s">&#34;&#34;</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}));</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">p</span><span class="o">.</span><span class="na">run</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="cm">/* Sample Output From the pipeline: |
| </span></span></span><span class="line"><span class="cl"><span class="cm"> &lt;p&gt;The table was GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=Test,projectId=&lt;&gt;, tableId=dummyTable}} |
| </span></span></span><span class="line"><span class="cl"><span class="cm"> &lt;p&gt;The row was GenericData{classInfo=[f], {num=null}} |
| </span></span></span><span class="line"><span class="cl"><span class="cm"> &lt;p&gt;The error was GenericData{classInfo=[errors, index],{errors=[GenericData{classInfo=[debugInfo, location, message, reason], {debugInfo=,location=, message=Missing required field: Msg_0_CLOUD_QUERY_TABLE.num., reason=invalid}}],index=0}} |
| </span></span></span><span class="line"><span class="cl"><span class="cm"> */</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># Create pipeline.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">schema</span> <span class="o">=</span> <span class="p">({</span><span class="s1">&#39;fields&#39;</span><span class="p">:</span> <span class="p">[{</span><span class="s1">&#39;name&#39;</span><span class="p">:</span> <span class="s1">&#39;a&#39;</span><span class="p">,</span> <span class="s1">&#39;type&#39;</span><span class="p">:</span> <span class="s1">&#39;STRING&#39;</span><span class="p">,</span> <span class="s1">&#39;mode&#39;</span><span class="p">:</span> <span class="s1">&#39;REQUIRED&#39;</span><span class="p">}]})</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">pipeline</span> <span class="o">=</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">errors</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pipeline</span> <span class="o">|</span> <span class="s1">&#39;Data&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;CreateBrokenData&#39;</span> <span class="o">&gt;&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">src</span><span class="p">:</span> <span class="p">{</span><span class="s1">&#39;a&#39;</span><span class="p">:</span> <span class="n">src</span><span class="p">}</span> <span class="k">if</span> <span class="n">src</span> <span class="o">==</span> <span class="mi">2</span> <span class="k">else</span> <span class="p">{</span><span class="s1">&#39;a&#39;</span><span class="p">:</span> <span class="kc">None</span><span class="p">})</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;WriteToBigQuery&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">WriteToBigQuery</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;&lt;Your Project:Test.dummy_a_table&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">schema</span><span class="o">=</span><span class="n">schema</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">insert_retry_strategy</span><span class="o">=</span><span class="s1">&#39;RETRY_ON_TRANSIENT_ERROR&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">create_disposition</span><span class="o">=</span><span class="s1">&#39;CREATE_IF_NEEDED&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">write_disposition</span><span class="o">=</span><span class="s1">&#39;WRITE_APPEND&#39;</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">result</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">errors</span><span class="p">[</span><span class="s1">&#39;FailedRows&#39;</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;PrintErrors&#39;</span> <span class="o">&gt;&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span class="k">lambda</span> <span class="n">err</span><span class="p">:</span> <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;Error Found </span><span class="si">{}</span><span class="s2">&#34;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">err</span><span class="p">))))</span></span></span></code></pre></div> |
| </div> |
| </div></description></item><item><title>Documentation: Cdap IO</title><link>/documentation/io/built-in/cdap/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/io/built-in/cdap/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="cdap-io">Cdap IO</h1> |
| <p>A <code>CdapIO</code> is a transform for reading data from source or writing data to sink CDAP plugin.</p> |
| <h2 id="batch-plugins-support">Batch plugins support</h2> |
| <p><code>CdapIO</code> currently supports the following CDAP Batch plugins by referencing <code>CDAP plugin</code> class name:</p> |
| <ul> |
| <li><a href="https://github.com/data-integrations/hubspot/blob/develop/src/main/java/io/cdap/plugin/hubspot/source/batch/HubspotBatchSource.java">Hubspot Batch Source</a></li> |
| <li><a href="https://github.com/data-integrations/hubspot/blob/develop/src/main/java/io/cdap/plugin/hubspot/sink/batch/HubspotBatchSink.java">Hubspot Batch Sink</a></li> |
| <li><a href="https://github.com/data-integrations/salesforce/blob/develop/src/main/java/io/cdap/plugin/salesforce/plugin/source/batch/SalesforceBatchSource.java">Salesforce Batch Source</a></li> |
| <li><a href="https://github.com/data-integrations/salesforce/blob/develop/src/main/java/io/cdap/plugin/salesforce/plugin/sink/batch/SalesforceBatchSink.java">Salesforce Batch Sink</a></li> |
| <li><a href="https://github.com/data-integrations/servicenow-plugins/blob/develop/src/main/java/io/cdap/plugin/servicenow/source/ServiceNowSource.java">ServiceNow Batch Source</a></li> |
| <li><a href="https://github.com/data-integrations/zendesk/blob/develop/src/main/java/io/cdap/plugin/zendesk/source/batch/ZendeskBatchSource.java">Zendesk Batch Source</a></li> |
| </ul> |
| <p>Also, any other CDAP Batch plugin based on Hadoop&rsquo;s <code>InputFormat</code> or <code>OutputFormat</code> can be used. They can be easily added to the list of supported by class name plugins, for more details please see <a href="https://github.com/apache/beam/blob/master/sdks/java/io/cdap/README.md">CdapIO readme</a>.</p> |
| <h2 id="streaming-plugins-support">Streaming plugins support</h2> |
| <p><code>CdapIO</code> currently supports CDAP Streaming plugins based on <a href="https://spark.apache.org/docs/2.4.0/streaming-custom-receivers.html">Apache Spark Receiver</a>.</p> |
| <p>Requirements for CDAP Streaming plugins:</p> |
| <ul> |
| <li>CDAP Streaming plugin should be based on <code>Spark Receiver</code> (Spark 2.4).</li> |
| <li>CDAP Streaming plugin should support work with offsets.</li> |
| <li>Corresponding Spark Receiver should implement <a href="https://github.com/apache/beam/blob/master/sdks/java/io/sparkreceiver/2/src/main/java/org/apache/beam/sdk/io/sparkreceiver/HasOffset.java">HasOffset</a> interface.</li> |
| <li>Records should have the numeric field that represents record offset.</li> |
| </ul> |
| <h2 id="batch-reading-using-cdapio">Batch reading using CdapIO</h2> |
| <p>In order to read from CDAP plugin you will need to pass:</p> |
| <ul> |
| <li><code>Key</code> and <code>Value</code> classes. You will need to check if these classes have a Beam Coder available.</li> |
| <li><code>PluginConfig</code> object with parameters for certain CDAP plugin.</li> |
| </ul> |
| <p>You can easily build <code>PluginConfig</code> object using <code>ConfigWrapper</code> class by specifying:</p> |
| <ul> |
| <li>Class of the needed <code>PluginConfig</code>.</li> |
| <li><code>Map&lt;String, Object&gt;</code> parameters map for corresponding CDAP plugin.</li> |
| </ul> |
| <p>For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">Map</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Object</span><span class="o">&gt;</span> <span class="n">myPluginConfigParams</span> <span class="o">=</span> <span class="k">new</span> <span class="n">HashMap</span><span class="o">&lt;&gt;();</span> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Read plugin parameters (e.g. from PipelineOptions) and put them into &#39;myPluginConfigParams&#39; map. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">myPluginConfigParams</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="n">MyPluginConstants</span><span class="o">.</span><span class="na">USERNAME_PARAMETER_NAME</span><span class="o">,</span> <span class="n">pipelineOptions</span><span class="o">.</span><span class="na">getUsername</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"><span class="c1">// ... |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">MyPluginConfig</span> <span class="n">pluginConfig</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">ConfigWrapper</span><span class="o">&lt;&gt;(</span><span class="n">MyPluginConfig</span><span class="o">.</span><span class="na">class</span><span class="o">).</span><span class="na">withParams</span><span class="o">(</span><span class="n">myPluginConfigParams</span><span class="o">).</span><span class="na">build</span><span class="o">();</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="read-data-by-plugin-class-name">Read data by plugin class name</h3> |
| <p>Some CDAP plugins are already supported and can be used just by plugin class name.</p> |
| <p>For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">CdapIO</span><span class="o">.</span><span class="na">Read</span><span class="o">&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">JsonElement</span><span class="o">&gt;</span> <span class="n">readTransform</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CdapIO</span><span class="o">.&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">JsonElement</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withCdapPluginClass</span><span class="o">(</span><span class="n">HubspotBatchSource</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPluginConfig</span><span class="o">(</span><span class="n">pluginConfig</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyClass</span><span class="o">(</span><span class="n">NullWritable</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueClass</span><span class="o">(</span><span class="n">JsonElement</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;read&#34;</span><span class="o">,</span> <span class="n">readTransform</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="read-data-with-building-batch-plugin">Read data with building Batch Plugin</h3> |
| <p>If CDAP plugin is not supported by plugin class name, you can easily build <code>Plugin</code> object by passing the following parameters:</p> |
| <ul> |
| <li>Class of CDAP Batch plugin.</li> |
| <li>The <code>InputFormat</code> class used to connect to your CDAP plugin of choice.</li> |
| <li>The <code>InputFormatProvider</code> class used to provide <code>InputFormat</code>.</li> |
| </ul> |
| <p>Then you will be able to pass this <code>Plugin</code> object to <code>CdapIO</code>.</p> |
| <p>For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">CdapIO</span><span class="o">.</span><span class="na">Read</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">readTransform</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CdapIO</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withCdapPlugin</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Plugin</span><span class="o">.</span><span class="na">createBatch</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MyCdapPlugin</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MyInputFormat</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MyInputFormatProvider</span><span class="o">.</span><span class="na">class</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPluginConfig</span><span class="o">(</span><span class="n">pluginConfig</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyClass</span><span class="o">(</span><span class="n">String</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueClass</span><span class="o">(</span><span class="n">String</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;read&#34;</span><span class="o">,</span> <span class="n">readTransform</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="examples-for-specific-cdap-plugins">Examples for specific CDAP plugins</h3> |
| <h4 id="cdap-hubspot-batch-source-plugin">CDAP Hubspot Batch Source plugin</h4> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">SourceHubspotConfig</span> <span class="n">pluginConfig</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">ConfigWrapper</span><span class="o">&lt;&gt;(</span><span class="n">SourceHubspotConfig</span><span class="o">.</span><span class="na">class</span><span class="o">).</span><span class="na">withParams</span><span class="o">(</span><span class="n">pluginConfigParams</span><span class="o">).</span><span class="na">build</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">CdapIO</span><span class="o">&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">JsonElement</span><span class="o">&gt;</span> <span class="n">readTransform</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CdapIO</span><span class="o">.&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">JsonElement</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withCdapPluginClass</span><span class="o">(</span><span class="n">HubspotBatchSource</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPluginConfig</span><span class="o">(</span><span class="n">pluginConfig</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyClass</span><span class="o">(</span><span class="n">NullWritable</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueClass</span><span class="o">(</span><span class="n">JsonElement</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;readFromHubspotPlugin&#34;</span><span class="o">,</span> <span class="n">readTransform</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="cdap-salesforce-batch-source-plugin">CDAP Salesforce Batch Source plugin</h4> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">SalesforceSourceConfig</span> <span class="n">pluginConfig</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">ConfigWrapper</span><span class="o">&lt;&gt;(</span><span class="n">SalesforceSourceConfig</span><span class="o">.</span><span class="na">class</span><span class="o">).</span><span class="na">withParams</span><span class="o">(</span><span class="n">pluginConfigParams</span><span class="o">).</span><span class="na">build</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">CdapIO</span><span class="o">&lt;</span><span class="n">Schema</span><span class="o">,</span> <span class="n">LinkedHashMap</span><span class="o">&gt;</span> <span class="n">readTransform</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CdapIO</span><span class="o">.&lt;</span><span class="n">Schema</span><span class="o">,</span> <span class="n">LinkedHashMap</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withCdapPluginClass</span><span class="o">(</span><span class="n">SalesforceBatchSource</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPluginConfig</span><span class="o">(</span><span class="n">pluginConfig</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyClass</span><span class="o">(</span><span class="n">Schema</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueClass</span><span class="o">(</span><span class="n">LinkedHashMap</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;readFromSalesforcePlugin&#34;</span><span class="o">,</span> <span class="n">readTransform</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="cdap-servicenow-batch-source-plugin">CDAP ServiceNow Batch Source plugin</h4> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">ServiceNowSourceConfig</span> <span class="n">pluginConfig</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">ConfigWrapper</span><span class="o">&lt;&gt;(</span><span class="n">ServiceNowSourceConfig</span><span class="o">.</span><span class="na">class</span><span class="o">).</span><span class="na">withParams</span><span class="o">(</span><span class="n">pluginConfigParams</span><span class="o">).</span><span class="na">build</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">CdapIO</span><span class="o">&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">StructuredRecord</span><span class="o">&gt;</span> <span class="n">readTransform</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CdapIO</span><span class="o">.&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">StructuredRecord</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withCdapPluginClass</span><span class="o">(</span><span class="n">ServiceNowSource</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPluginConfig</span><span class="o">(</span><span class="n">pluginConfig</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyClass</span><span class="o">(</span><span class="n">NullWritable</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueClass</span><span class="o">(</span><span class="n">StructuredRecord</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;readFromServiceNowPlugin&#34;</span><span class="o">,</span> <span class="n">readTransform</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="cdap-zendesk-batch-source-plugin">CDAP Zendesk Batch Source plugin</h4> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">ZendeskBatchSourceConfig</span> <span class="n">pluginConfig</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">ConfigWrapper</span><span class="o">&lt;&gt;(</span><span class="n">ZendeskBatchSourceConfig</span><span class="o">.</span><span class="na">class</span><span class="o">).</span><span class="na">withParams</span><span class="o">(</span><span class="n">pluginConfigParams</span><span class="o">).</span><span class="na">build</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">CdapIO</span><span class="o">&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">StructuredRecord</span><span class="o">&gt;</span> <span class="n">readTransform</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CdapIO</span><span class="o">.&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">StructuredRecord</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withCdapPluginClass</span><span class="o">(</span><span class="n">ZendeskBatchSource</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPluginConfig</span><span class="o">(</span><span class="n">pluginConfig</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyClass</span><span class="o">(</span><span class="n">NullWritable</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueClass</span><span class="o">(</span><span class="n">StructuredRecord</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;readFromZendeskPlugin&#34;</span><span class="o">,</span> <span class="n">readTransform</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>To learn more please check out <a href="https://github.com/apache/beam/tree/master/examples/java/cdap">complete examples</a>.</p> |
| <h2 id="batch-writing-using-cdapio">Batch writing using CdapIO</h2> |
| <p>In order to write to CDAP plugin you will need to pass:</p> |
| <ul> |
| <li><code>Key</code> and <code>Value</code> classes. You will need to check if these classes have a Beam Coder available.</li> |
| <li><code>locksDirPath</code>, which is locks directory path where locks will be stored. This parameter is needed for Hadoop External Synchronization (mechanism for acquiring locks related to the write job).</li> |
| <li><code>PluginConfig</code> object with parameters for certain CDAP plugin.</li> |
| </ul> |
| <p>You can easily build <code>PluginConfig</code> object using <code>ConfigWrapper</code> class by specifying:</p> |
| <ul> |
| <li>Class of the needed <code>PluginConfig</code>.</li> |
| <li><code>Map&lt;String, Object&gt;</code> parameters map for corresponding CDAP plugin.</li> |
| </ul> |
| <p>For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">MyPluginConfig</span> <span class="n">pluginConfig</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">ConfigWrapper</span><span class="o">&lt;&gt;(</span><span class="n">MyPluginConfig</span><span class="o">.</span><span class="na">class</span><span class="o">).</span><span class="na">withParams</span><span class="o">(</span><span class="n">pluginConfigParams</span><span class="o">).</span><span class="na">build</span><span class="o">();</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="write-data-by-plugin-class-name">Write data by plugin class name</h3> |
| <p>Some CDAP plugins are already supported and can be used just by plugin class name.</p> |
| <p>For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">CdapIO</span><span class="o">.</span><span class="na">Write</span><span class="o">&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">readTransform</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CdapIO</span><span class="o">.&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span><span class="n">write</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withCdapPluginClass</span><span class="o">(</span><span class="n">HubspotBatchSink</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPluginConfig</span><span class="o">(</span><span class="n">pluginConfig</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyClass</span><span class="o">(</span><span class="n">NullWritable</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueClass</span><span class="o">(</span><span class="n">String</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withLocksDirPath</span><span class="o">(</span><span class="n">locksDirPath</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;write&#34;</span><span class="o">,</span> <span class="n">writeTransform</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="write-data-with-building-batch-plugin">Write data with building Batch Plugin</h3> |
| <p>If CDAP plugin is not supported by plugin class name, you can easily build <code>Plugin</code> object by passing the following parameters:</p> |
| <ul> |
| <li>Class of CDAP plugin.</li> |
| <li>The <code>OutputFormat</code> class used to connect to your CDAP plugin of choice.</li> |
| <li>The <code>OutputFormatProvider</code> class used to provide <code>OutputFormat</code>.</li> |
| </ul> |
| <p>Then you will be able to pass this <code>Plugin</code> object to <code>CdapIO</code>.</p> |
| <p>For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">CdapIO</span><span class="o">.</span><span class="na">Write</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">writeTransform</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CdapIO</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span><span class="n">write</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withCdapPlugin</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Plugin</span><span class="o">.</span><span class="na">createBatch</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MyCdapPlugin</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MyOutputFormat</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MyOutputFormatProvider</span><span class="o">.</span><span class="na">class</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPluginConfig</span><span class="o">(</span><span class="n">pluginConfig</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyClass</span><span class="o">(</span><span class="n">String</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueClass</span><span class="o">(</span><span class="n">String</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withLocksDirPath</span><span class="o">(</span><span class="n">locksDirPath</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;write&#34;</span><span class="o">,</span> <span class="n">writeTransform</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="examples-for-specific-cdap-plugins-1">Examples for specific CDAP plugins</h3> |
| <h4 id="cdap-hubspot-batch-sink-plugin">CDAP Hubspot Batch Sink plugin</h4> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">SinkHubspotConfig</span> <span class="n">pluginConfig</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">ConfigWrapper</span><span class="o">&lt;&gt;(</span><span class="n">SinkHubspotConfig</span><span class="o">.</span><span class="na">class</span><span class="o">).</span><span class="na">withParams</span><span class="o">(</span><span class="n">pluginConfigParams</span><span class="o">).</span><span class="na">build</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">CdapIO</span><span class="o">&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">writeTransform</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CdapIO</span><span class="o">.&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span><span class="n">write</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withCdapPluginClass</span><span class="o">(</span><span class="n">pluginClass</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPluginConfig</span><span class="o">(</span><span class="n">pluginConfig</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyClass</span><span class="o">(</span><span class="n">NullWritable</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueClass</span><span class="o">(</span><span class="n">String</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withLocksDirPath</span><span class="o">(</span><span class="n">locksDirPath</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;writeToHubspotPlugin&#34;</span><span class="o">,</span> <span class="n">writeTransform</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="cdap-salesforce-batch-sink-plugin">CDAP Salesforce Batch Sink plugin</h4> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">SalesforceSinkConfig</span> <span class="n">pluginConfig</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">ConfigWrapper</span><span class="o">&lt;&gt;(</span><span class="n">SalesforceSinkConfig</span><span class="o">.</span><span class="na">class</span><span class="o">).</span><span class="na">withParams</span><span class="o">(</span><span class="n">pluginConfigParams</span><span class="o">).</span><span class="na">build</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">CdapIO</span><span class="o">&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">CSVRecord</span><span class="o">&gt;</span> <span class="n">writeTransform</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CdapIO</span><span class="o">.&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">CSVRecord</span><span class="o">&gt;</span><span class="n">write</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withCdapPluginClass</span><span class="o">(</span><span class="n">pluginClass</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPluginConfig</span><span class="o">(</span><span class="n">pluginConfig</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyClass</span><span class="o">(</span><span class="n">NullWritable</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueClass</span><span class="o">(</span><span class="n">CSVRecord</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withLocksDirPath</span><span class="o">(</span><span class="n">locksDirPath</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;writeToSalesforcePlugin&#34;</span><span class="o">,</span> <span class="n">writeTransform</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>To learn more please check out <a href="https://github.com/apache/beam/tree/master/examples/java/cdap/src/main/java/org/apache/beam/examples/complete/cdap">complete examples</a>.</p> |
| <h2 id="streaming-reading-using-cdapio">Streaming reading using CdapIO</h2> |
| <p>In order to read from CDAP plugin you will need to pass:</p> |
| <ul> |
| <li><code>Key</code> and <code>Value</code> classes. You will need to check if these classes have a Beam Coder available.</li> |
| <li><code>PluginConfig</code> object with parameters for certain CDAP plugin.</li> |
| </ul> |
| <p>You can easily build <code>PluginConfig</code> object using <code>ConfigWrapper</code> class by specifying:</p> |
| <ul> |
| <li>Class of the needed <code>PluginConfig</code>.</li> |
| <li><code>Map&lt;String, Object&gt;</code> parameters map for corresponding CDAP plugin.</li> |
| </ul> |
| <p>For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">MyPluginConfig</span> <span class="n">pluginConfig</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">ConfigWrapper</span><span class="o">&lt;&gt;(</span><span class="n">MyPluginConfig</span><span class="o">.</span><span class="na">class</span><span class="o">).</span><span class="na">withParams</span><span class="o">(</span><span class="n">pluginConfigParams</span><span class="o">).</span><span class="na">build</span><span class="o">();</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="read-data-by-plugin-class-name-1">Read data by plugin class name</h3> |
| <p>Some CDAP plugins are already supported and can be used just by plugin class name.</p> |
| <p>For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">CdapIO</span><span class="o">.</span><span class="na">Read</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">readTransform</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CdapIO</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withCdapPluginClass</span><span class="o">(</span><span class="n">MyStreamingPlugin</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPluginConfig</span><span class="o">(</span><span class="n">pluginConfig</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyClass</span><span class="o">(</span><span class="n">NullWritable</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueClass</span><span class="o">(</span><span class="n">String</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;read&#34;</span><span class="o">,</span> <span class="n">readTransform</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="read-data-with-building-streaming-plugin">Read data with building Streaming Plugin</h3> |
| <p>If CDAP plugin is not supported by plugin class name, you can easily build <code>Plugin</code> object by passing the following parameters:</p> |
| <ul> |
| <li>Class of CDAP Streaming plugin.</li> |
| <li><code>getOffsetFn</code>, which is <code>SerializableFunction</code> that defines how to get <code>Long</code> record offset from a record.</li> |
| <li><code>receiverClass</code>, which is Spark (v 2.4) <code>Receiver</code> class associated with CDAP plugin.</li> |
| <li>(Optionally) <code>getReceiverArgsFromConfigFn</code>, which is <code>SerializableFunction</code> that defines how to get constructor arguments for Spark <code>Receiver</code> using <code>PluginConfig</code> object.</li> |
| </ul> |
| <p>Then you will be able to pass this <code>Plugin</code> object to <code>CdapIO</code>.</p> |
| <p>For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">CdapIO</span><span class="o">.</span><span class="na">Read</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">readTransform</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CdapIO</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withCdapPlugin</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Plugin</span><span class="o">.</span><span class="na">createStreaming</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MyStreamingPlugin</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">myGetOffsetFn</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MyReceiver</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">myGetReceiverArgsFromConfigFn</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPluginConfig</span><span class="o">(</span><span class="n">pluginConfig</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyClass</span><span class="o">(</span><span class="n">NullWritable</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueClass</span><span class="o">(</span><span class="n">String</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;read&#34;</span><span class="o">,</span> <span class="n">readTransform</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="read-data-with-optional-parameters">Read data with optional parameters</h3> |
| <p>Optionally you can pass the following optional parameters:</p> |
| <ul> |
| <li><code>pullFrequencySec</code>, which is delay in seconds between polling for new records updates.</li> |
| <li><code>startOffset</code>, which is inclusive start offset from which the reading should be started.</li> |
| </ul> |
| <p>For example:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">CdapIO</span><span class="o">.</span><span class="na">Read</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">readTransform</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CdapIO</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withCdapPluginClass</span><span class="o">(</span><span class="n">MyStreamingPlugin</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPluginConfig</span><span class="o">(</span><span class="n">pluginConfig</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyClass</span><span class="o">(</span><span class="n">NullWritable</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueClass</span><span class="o">(</span><span class="n">String</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPullFrequencySec</span><span class="o">(</span><span class="n">1L</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withStartOffset</span><span class="o">(</span><span class="n">1L</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;read&#34;</span><span class="o">,</span> <span class="n">readTransform</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="examples-for-specific-cdap-plugins-2">Examples for specific CDAP plugins</h3> |
| <h4 id="cdap-hubspot-streaming-source-plugin">CDAP Hubspot Streaming Source plugin</h4> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">HubspotStreamingSourceConfig</span> <span class="n">pluginConfig</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">ConfigWrapper</span><span class="o">&lt;&gt;(</span><span class="n">HubspotStreamingSourceConfig</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withParams</span><span class="o">(</span><span class="n">pluginConfigParams</span><span class="o">).</span><span class="na">build</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">CdapIO</span><span class="o">.</span><span class="na">Read</span><span class="o">&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">readTransform</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CdapIO</span><span class="o">.&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withCdapPlugin</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Plugin</span><span class="o">.</span><span class="na">createStreaming</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">HubspotStreamingSource</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">GetOffsetUtils</span><span class="o">.</span><span class="na">getOffsetFnForHubspot</span><span class="o">(),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">HubspotReceiver</span><span class="o">.</span><span class="na">class</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPluginConfig</span><span class="o">(</span><span class="n">pluginConfig</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyClass</span><span class="o">(</span><span class="n">NullWritable</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueClass</span><span class="o">(</span><span class="n">String</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;readFromHubspotPlugin&#34;</span><span class="o">,</span> <span class="n">readTransform</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="cdap-salesforce-streaming-source-plugin">CDAP Salesforce Streaming Source plugin</h4> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">SalesforceStreamingSourceConfig</span> <span class="n">pluginConfig</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">ConfigWrapper</span><span class="o">&lt;&gt;(</span><span class="n">SalesforceStreamingSourceConfig</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withParams</span><span class="o">(</span><span class="n">pluginConfigParams</span><span class="o">).</span><span class="na">build</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">CdapIO</span><span class="o">.</span><span class="na">Read</span><span class="o">&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="n">readTransform</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CdapIO</span><span class="o">.&lt;</span><span class="n">NullWritable</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withCdapPlugin</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Plugin</span><span class="o">.</span><span class="na">createStreaming</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">SalesforceStreamingSource</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">GetOffsetUtils</span><span class="o">.</span><span class="na">getOffsetFnForSalesforce</span><span class="o">(),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">SalesforceReceiver</span><span class="o">.</span><span class="na">class</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">config</span> <span class="o">-&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">SalesforceStreamingSourceConfig</span> <span class="n">salesforceConfig</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">SalesforceStreamingSourceConfig</span><span class="o">)</span> <span class="n">config</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="k">new</span> <span class="n">Object</span><span class="o">[]</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">salesforceConfig</span><span class="o">.</span><span class="na">getAuthenticatorCredentials</span><span class="o">(),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">salesforceConfig</span><span class="o">.</span><span class="na">getPushTopicName</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">};</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withPluginConfig</span><span class="o">(</span><span class="n">pluginConfig</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withKeyClass</span><span class="o">(</span><span class="n">NullWritable</span><span class="o">.</span><span class="na">class</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueClass</span><span class="o">(</span><span class="n">String</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;readFromSalesforcePlugin&#34;</span><span class="o">,</span> <span class="n">readTransform</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>To learn more please check out <a href="https://github.com/apache/beam/tree/master/examples/java/cdap/src/main/java/org/apache/beam/examples/complete/cdap">complete examples</a>.</p></description></item><item><title>Documentation: CoGroupByKey</title><link>/documentation/transforms/java/aggregation/cogroupbykey/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/java/aggregation/cogroupbykey/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="cogroupbykey">CoGroupByKey</h1> |
| <table align="left"> |
| <a target="_blank" class="button" |
| href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/join/CoGroupByKey.html"> |
| <img src="/images/logos/sdks/java.png" width="20px" height="20px" |
| alt="Javadoc" /> |
| Javadoc |
| </a> |
| </table> |
| <br><br> |
| <p>Aggregates all input elements by their key and allows downstream processing |
| to consume all values associated with the key. While <code>GroupByKey</code> performs |
| this operation over a single input collection and thus a single type of |
| input values, <code>CoGroupByKey</code> operates over multiple input collections. As |
| a result, the result for each key is a tuple of the values associated with |
| that key in each input collection.</p> |
| <p>See more information in the <a href="/documentation/programming-guide/#cogroupbykey">Beam Programming Guide</a>.</p> |
| <h2 id="examples">Examples</h2> |
| <p><strong>Example 1</strong>: Say you have two different files with user data; one file has |
| names and email addresses and the other file has names and phone numbers.</p> |
| <p>You can join those two data sets, using the username as a common key and the |
| other data as the associated values. After the join, you have one data set |
| that contains all of the information (email addresses and phone numbers) |
| associated with each name.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">UID</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">pt1</span> <span class="o">=</span> <span class="cm">/* ... */</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">UID</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;&gt;</span> <span class="n">pt2</span> <span class="o">=</span> <span class="cm">/* ... */</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">final</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">t1</span> <span class="o">=</span> <span class="k">new</span> <span class="n">TupleTag</span><span class="o">&lt;&gt;();</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">final</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">t2</span> <span class="o">=</span> <span class="k">new</span> <span class="n">TupleTag</span><span class="o">&lt;&gt;();</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">UID</span><span class="o">,</span> <span class="n">CoGBKResult</span><span class="o">&gt;&gt;</span> <span class="n">result</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KeyedPCollectionTuple</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">t1</span><span class="o">,</span> <span class="n">pt1</span><span class="o">).</span><span class="na">and</span><span class="o">(</span><span class="n">t2</span><span class="o">,</span> <span class="n">pt2</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">CoGroupByKey</span><span class="o">.</span><span class="na">create</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"><span class="n">result</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">K</span><span class="o">,</span> <span class="n">CoGbkResult</span><span class="o">&gt;,</span> <span class="cm">/* some result */</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KV</span><span class="o">&lt;</span><span class="n">K</span><span class="o">,</span> <span class="n">CoGbkResult</span><span class="o">&gt;</span> <span class="n">e</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">CoGbkResult</span> <span class="n">result</span> <span class="o">=</span> <span class="n">e</span><span class="o">.</span><span class="na">getValue</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Retrieve all integers associated with this key from pt1 |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">Iterable</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">allIntegers</span> <span class="o">=</span> <span class="n">result</span><span class="o">.</span><span class="na">getAll</span><span class="o">(</span><span class="n">t1</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Retrieve the string associated with this key from pt2. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// Note: This will fail if multiple values had the same key in pt2. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">String</span> <span class="n">string</span> <span class="o">=</span> <span class="n">e</span><span class="o">.</span><span class="na">getOnly</span><span class="o">(</span><span class="n">t2</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p><strong>Example 2:</strong></p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-java playground-snippet" |
| data-sdk="java" |
| data-path="SDK_JAVA_GroupByKey" |
| data-show="main_section" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_JAVA_GroupByKey%22%2c%22sdk%22%3a%22java%22%2c%22show%22%3a%22main_section%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2> |
| <ul> |
| <li><a href="/documentation/transforms/java/aggregation/groupbykey">GroupByKey</a> |
| takes one input collection.</li> |
| </ul></description></item><item><title>Documentation: CoGroupByKey</title><link>/documentation/transforms/python/aggregation/cogroupbykey/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/python/aggregation/cogroupbykey/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="cogroupbykey">CoGroupByKey</h1> |
| <script type="text/javascript"> |
| localStorage.setItem("language", "language-py") |
| </script> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p> |
| <p>Aggregates all input elements by their key and allows downstream processing |
| to consume all values associated with the key. While <code>GroupByKey</code> performs |
| this operation over a single input collection and thus a single type of input |
| values, <code>CoGroupByKey</code> operates over multiple input collections. As a result, |
| the result for each key is a tuple of the values associated with that key in |
| each input collection.</p> |
| <p>See more information in the <a href="/documentation/programming-guide/#cogroupbykey">Beam Programming Guide</a>.</p> |
| <h2 id="examples">Examples</h2> |
| <p>In the following example, we create a pipeline with two <code>PCollection</code>s of produce, one with icons and one with durations, both with a common key of the produce name. |
| Then, we apply <code>CoGroupByKey</code> to join both <code>PCollection</code>s using their keys.</p> |
| <p><code>CoGroupByKey</code> expects a dictionary of named keyed <code>PCollection</code>s, and produces elements joined by their keys. |
| The values of each output element are dictionaries where the names correspond to the input dictionary, with lists of all the values found for that key.</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="k">as</span> <span class="nn">beam</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">icon_pairs</span> <span class="o">=</span> <span class="n">pipeline</span> <span class="o">|</span> <span class="s1">&#39;Create icons&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;Apple&#39;</span><span class="p">,</span> <span class="s1">&#39;🍎&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;Apple&#39;</span><span class="p">,</span> <span class="s1">&#39;🍏&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;Eggplant&#39;</span><span class="p">,</span> <span class="s1">&#39;🍆&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;Tomato&#39;</span><span class="p">,</span> <span class="s1">&#39;🍅&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">])</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">duration_pairs</span> <span class="o">=</span> <span class="n">pipeline</span> <span class="o">|</span> <span class="s1">&#39;Create durations&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;Apple&#39;</span><span class="p">,</span> <span class="s1">&#39;perennial&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;Carrot&#39;</span><span class="p">,</span> <span class="s1">&#39;biennial&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;Tomato&#39;</span><span class="p">,</span> <span class="s1">&#39;perennial&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="s1">&#39;Tomato&#39;</span><span class="p">,</span> <span class="s1">&#39;annual&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">])</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">plants</span> <span class="o">=</span> <span class="p">(({</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;icons&#39;</span><span class="p">:</span> <span class="n">icon_pairs</span><span class="p">,</span> <span class="s1">&#39;durations&#39;</span><span class="p">:</span> <span class="n">duration_pairs</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">})</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Merge&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">CoGroupByKey</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="nb">print</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="notebook-skip">Output:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>(&#39;Apple&#39;, {&#39;icons&#39;: [&#39;🍎&#39;, &#39;🍏&#39;], &#39;durations&#39;: [&#39;perennial&#39;]}) |
| (&#39;Carrot&#39;, {&#39;icons&#39;: [], &#39;durations&#39;: [&#39;biennial&#39;]}) |
| (&#39;Tomato&#39;, {&#39;icons&#39;: [&#39;🍅&#39;], &#39;durations&#39;: [&#39;perennial&#39;, &#39;annual&#39;]}) |
| (&#39;Eggplant&#39;, {&#39;icons&#39;: [&#39;🍆&#39;], &#39;durations&#39;: []})</code></pre> |
| </div> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2> |
| <ul> |
| <li><a href="/documentation/transforms/python/aggregation/combineglobally">CombineGlobally</a> to combine elements.</li> |
| <li><a href="/documentation/transforms/python/aggregation/groupbykey">GroupByKey</a> takes one input collection.</li> |
| </ul> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.CoGroupByKey" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p></description></item><item><title>Documentation: Combine</title><link>/documentation/transforms/java/aggregation/combine/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/java/aggregation/combine/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="combine">Combine</h1> |
| <table align="left"> |
| <a target="_blank" class="button" |
| href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Combine.html"> |
| <img src="/images/logos/sdks/java.png" width="20px" height="20px" |
| alt="Javadoc" /> |
| Javadoc |
| </a> |
| </table> |
| <br><br> |
| <p>A user-defined <code>CombineFn</code> may be applied to combine all elements in a |
| <code>PCollection</code> (global combine) or to combine all elements associated |
| with each key.</p> |
| <p>While the result is similar to applying a <code>GroupByKey</code> followed by |
| aggregating values in each <code>Iterable</code>, there is an impact |
| on the code you must write as well as the performance of the pipeline. |
| Writing a <code>ParDo</code> that counts the number of elements in each value |
| would be very straightforward. However, as described in the execution |
| model, it would also require all values associated with each key to be |
| processed by a single worker. This introduces a lot of communication overhead. |
| Using a <code>CombineFn</code> requires the code be structured as an associative and |
| commumative operation. But, it allows the use of partial sums to be precomputed.</p> |
| <p>See more information in the <a href="/documentation/programming-guide/#combine">Beam Programming Guide</a>.</p> |
| <h2 id="examples">Examples</h2> |
| <p><strong>Example 1</strong>: Global combine</p> |
| <p>Use the global combine to combine all of the elements in a given <code>PCollection</code> |
| into a single value, represented in your pipeline as a new <code>PCollection</code> containing |
| one element. The following example code shows how to apply the Beam-provided |
| sum combine function to produce a single sum value for a <code>PCollection</code> of integers.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Sum.SumIntegerFn() combines the elements in the input PCollection. The resulting PCollection, called sum, |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// contains one value: the sum of all the elements in the input PCollection. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">pc</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Integer</span><span class="o">&gt;</span> <span class="n">sum</span> <span class="o">=</span> <span class="n">pc</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Combine</span><span class="o">.</span><span class="na">globally</span><span class="o">(</span><span class="k">new</span> <span class="n">Sum</span><span class="o">.</span><span class="na">SumIntegerFn</span><span class="o">()));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p><strong>Example 2</strong>: Keyed combine</p> |
| <p>Use a keyed combine to combine all of the values associated with each key |
| into a single output value for each key. As with the global combine, the |
| function passed to a keyed combine must be associative and commutative.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// PCollection is grouped by key and the Double values associated with each key are combined into a Double. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Double</span><span class="o">&gt;&gt;</span> <span class="n">salesRecords</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Double</span><span class="o">&gt;&gt;</span> <span class="n">totalSalesPerPerson</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">salesRecords</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Combine</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Double</span><span class="o">,</span> <span class="n">Double</span><span class="o">&gt;</span><span class="n">perKey</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">Sum</span><span class="o">.</span><span class="na">SumDoubleFn</span><span class="o">()));</span> |
| </span></span><span class="line"><span class="cl"><span class="c1">// The combined value is of a different type than the original collection of values per key. PCollection has |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// keys of type String and values of type Integer, and the combined value is a Double. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">playerAccuracy</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Double</span><span class="o">&gt;&gt;</span> <span class="n">avgAccuracyPerPlayer</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">playerAccuracy</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Combine</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">,</span> <span class="n">Double</span><span class="o">&gt;</span><span class="n">perKey</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">MeanInts</span><span class="o">())));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p><strong>Example 3</strong>:</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-java playground-snippet" |
| data-sdk="java" |
| data-path="SDK_JAVA_Combine" |
| data-show="main_section" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_JAVA_Combine%22%2c%22sdk%22%3a%22java%22%2c%22show%22%3a%22main_section%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2> |
| <ul> |
| <li><a href="/documentation/transforms/java/aggregation/combinewithcontext">CombineWithContext</a></li> |
| <li><a href="/documentation/transforms/java/aggregation/groupbykey">GroupByKey</a></li> |
| </ul></description></item><item><title>Documentation: CombineGlobally</title><link>/documentation/transforms/python/aggregation/combineglobally/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/python/aggregation/combineglobally/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="combineglobally">CombineGlobally</h1> |
| <script type="text/javascript"> |
| localStorage.setItem("language", "language-py") |
| </script> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p> |
| <p>Combines all elements in a collection.</p> |
| <p>See more information in the <a href="/documentation/programming-guide/#combine">Beam Programming Guide</a>.</p> |
| <h2 id="examples">Examples</h2> |
| <p>In the following examples, we create a pipeline with a <code>PCollection</code> of produce. |
| Then, we apply <code>CombineGlobally</code> in multiple ways to combine all the elements in the <code>PCollection</code>.</p> |
| <p><code>CombineGlobally</code> accepts a function that takes an <code>iterable</code> of elements as an input, and combines them to return a single element.</p> |
| <h3 id="example-1-combining-with-a-function">Example 1: Combining with a function</h3> |
| <p>We define a function <code>get_common_items</code> which takes an <code>iterable</code> of sets as an input, and calculates the intersection (common items) of those sets.</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_CombineGloballyFunction" |
| data-show="combineglobally_function" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_CombineGloballyFunction%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22combineglobally_function%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h3 id="example-2-combining-with-a-lambda-function">Example 2: Combining with a lambda function</h3> |
| <p>We can also use lambda functions to simplify <strong>Example 1</strong>.</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_CombineGloballyLambda" |
| data-show="combineglobally_lambda" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_CombineGloballyLambda%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22combineglobally_lambda%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h3 id="example-3-combining-with-multiple-arguments">Example 3: Combining with multiple arguments</h3> |
| <p>You can pass functions with multiple arguments to <code>CombineGlobally</code>. |
| They are passed as additional positional arguments or keyword arguments to the function.</p> |
| <p>In this example, the lambda function takes <code>sets</code> and <code>exclude</code> as arguments.</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_CombineGloballyMultipleArguments" |
| data-show="combineglobally_multiple_arguments" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_CombineGloballyMultipleArguments%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22combineglobally_multiple_arguments%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h3 id="example-4-combining-with-a-combinefn">Example 4: Combining with a <code>CombineFn</code></h3> |
| <p>The more general way to combine elements, and the most flexible, is with a class that inherits from <code>CombineFn</code>.</p> |
| <ul> |
| <li> |
| <p><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.create_accumulator"><code>CombineFn.create_accumulator()</code></a>: |
| This creates an empty accumulator. |
| For example, an empty accumulator for a sum would be <code>0</code>, while an empty accumulator for a product (multiplication) would be <code>1</code>.</p> |
| </li> |
| <li> |
| <p><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.add_input"><code>CombineFn.add_input()</code></a>: |
| Called <em>once per element</em>. |
| Takes an accumulator and an input element, combines them and returns the updated accumulator.</p> |
| </li> |
| <li> |
| <p><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.merge_accumulators"><code>CombineFn.merge_accumulators()</code></a>: |
| Multiple accumulators could be processed in parallel, so this function helps merging them into a single accumulator.</p> |
| </li> |
| <li> |
| <p><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.extract_output"><code>CombineFn.extract_output()</code></a>: |
| It allows to do additional calculations before extracting a result.</p> |
| </li> |
| </ul> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_CombineGloballyCombineFn" |
| data-show="combineglobally_combinefn" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_CombineGloballyCombineFn%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22combineglobally_combinefn%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2> |
| <p>You can use the following combiner transforms:</p> |
| <ul> |
| <li><a href="/documentation/transforms/python/aggregation/combineperkey">CombinePerKey</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/combinevalues">CombineValues</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/mean">Mean</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/count">Count</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/top">Top</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/sample">Sample</a></li> |
| </ul> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineGlobally" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p></description></item><item><title>Documentation: CombinePerKey</title><link>/documentation/transforms/python/aggregation/combineperkey/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/python/aggregation/combineperkey/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="combineperkey">CombinePerKey</h1> |
| <script type="text/javascript"> |
| localStorage.setItem("language", "language-py") |
| </script> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombinePerKey" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p> |
| <p>Combines all elements for each key in a collection.</p> |
| <p>See more information in the <a href="/documentation/programming-guide/#combine">Beam Programming Guide</a>.</p> |
| <h2 id="examples">Examples</h2> |
| <p>In the following examples, we create a pipeline with a <code>PCollection</code> of produce. |
| Then, we apply <code>CombinePerKey</code> in multiple ways to combine all the elements in the <code>PCollection</code>.</p> |
| <p><code>CombinePerKey</code> accepts a function that takes a list of values as an input, and combines them for each key.</p> |
| <h3 id="example-1-combining-with-a-predefined-function">Example 1: Combining with a predefined function</h3> |
| <p>We use the function |
| <a href="https://docs.python.org/3/library/functions.html#sum"><code>sum</code></a> |
| which takes an <code>iterable</code> of numbers and adds them together.</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_CombinePerKeySimple" |
| data-show="combineperkey_simple" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_CombinePerKeySimple%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22combineperkey_simple%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h3 id="example-2-combining-with-a-function">Example 2: Combining with a function</h3> |
| <p>We define a function <code>saturated_sum</code> which takes an <code>iterable</code> of numbers and adds them together, up to a predefined maximum number.</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_CombinePerKeyFunction" |
| data-show="combineperkey_function" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_CombinePerKeyFunction%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22combineperkey_function%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h3 id="example-3-combining-with-a-lambda-function">Example 3: Combining with a lambda function</h3> |
| <p>We can also use lambda functions to simplify <strong>Example 2</strong>.</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_CombinePerKeyLambda" |
| data-show="combineperkey_lambda" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_CombinePerKeyLambda%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22combineperkey_lambda%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h3 id="example-4-combining-with-multiple-arguments">Example 4: Combining with multiple arguments</h3> |
| <p>You can pass functions with multiple arguments to <code>CombinePerKey</code>. |
| They are passed as additional positional arguments or keyword arguments to the function.</p> |
| <p>In this example, the lambda function takes <code>values</code> and <code>max_value</code> as arguments.</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_CombinePerKeyMultipleArguments" |
| data-show="combineperkey_multiple_arguments" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_CombinePerKeyMultipleArguments%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22combineperkey_multiple_arguments%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h3 id="example-5-combining-with-a-combinefn">Example 5: Combining with a <code>CombineFn</code></h3> |
| <p>The more general way to combine elements, and the most flexible, is with a class that inherits from <code>CombineFn</code>.</p> |
| <ul> |
| <li> |
| <p><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.create_accumulator"><code>CombineFn.create_accumulator()</code></a>: |
| This creates an empty accumulator. |
| For example, an empty accumulator for a sum would be <code>0</code>, while an empty accumulator for a product (multiplication) would be <code>1</code>.</p> |
| </li> |
| <li> |
| <p><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.add_input"><code>CombineFn.add_input()</code></a>: |
| Called <em>once per element</em>. |
| Takes an accumulator and an input element, combines them and returns the updated accumulator.</p> |
| </li> |
| <li> |
| <p><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.merge_accumulators"><code>CombineFn.merge_accumulators()</code></a>: |
| Multiple accumulators could be processed in parallel, so this function helps merging them into a single accumulator.</p> |
| </li> |
| <li> |
| <p><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.extract_output"><code>CombineFn.extract_output()</code></a>: |
| It allows to do additional calculations before extracting a result.</p> |
| </li> |
| </ul> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_CombinePerKeyCombineFn" |
| data-show="combineperkey_combinefn" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_CombinePerKeyCombineFn%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22combineperkey_combinefn%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2> |
| <p>You can use the following combiner transforms:</p> |
| <ul> |
| <li><a href="/documentation/transforms/python/aggregation/combineglobally">CombineGlobally</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/combinevalues">CombineValues</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/mean">Mean</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/count">Count</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/top">Top</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/sample">Sample</a></li> |
| </ul> |
| <p>See also <a href="/documentation/transforms/python/aggregation/groupby">GroupBy</a> which allows you to combine more than one field at once.</p> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombinePerKey" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p></description></item><item><title>Documentation: CombineValues</title><link>/documentation/transforms/python/aggregation/combinevalues/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/python/aggregation/combinevalues/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="combinevalues">CombineValues</h1> |
| <script type="text/javascript"> |
| localStorage.setItem("language", "language-py") |
| </script> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineValues" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p> |
| <p>Combines an iterable of values in a keyed collection of elements.</p> |
| <p>See more information in the <a href="/documentation/programming-guide/#combine">Beam Programming Guide</a>.</p> |
| <h2 id="examples">Examples</h2> |
| <p>In the following examples, we create a pipeline with a <code>PCollection</code> of produce. |
| Then, we apply <code>CombineValues</code> in multiple ways to combine the keyed values in the <code>PCollection</code>.</p> |
| <p><code>CombineValues</code> accepts a function that takes an <code>iterable</code> of elements as an input, and combines them to return a single element. |
| <code>CombineValues</code> expects a keyed <code>PCollection</code> of elements, where the value is an iterable of elements to be combined.</p> |
| <h3 id="example-1-combining-with-a-predefined-function">Example 1: Combining with a predefined function</h3> |
| <p>We use the function |
| <a href="https://docs.python.org/3/library/functions.html#sum"><code>sum</code></a> |
| which takes an <code>iterable</code> of numbers and adds them together.</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_CombineValuesSimple" |
| data-show="combinevalues_simple" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_CombineValuesSimple%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22combinevalues_simple%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h3 id="example-2-combining-with-a-function">Example 2: Combining with a function</h3> |
| <p>We want the sum to be bounded up to a maximum value, so we use |
| <a href="https://en.wikipedia.org/wiki/Saturation_arithmetic">saturated arithmetic</a>.</p> |
| <p>We define a function <code>saturated_sum</code> which takes an <code>iterable</code> of numbers and adds them together, up to a predefined maximum number.</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_CombineValuesFunction" |
| data-show="combinevalues_function" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_CombineValuesFunction%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22combinevalues_function%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h3 id="example-3-combining-with-a-lambda-function">Example 3: Combining with a lambda function</h3> |
| <p>We can also use lambda functions to simplify <strong>Example 2</strong>.</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_CombineValuesLambda" |
| data-show="combinevalues_lambda" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_CombineValuesLambda%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22combinevalues_lambda%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h3 id="example-4-combining-with-multiple-arguments">Example 4: Combining with multiple arguments</h3> |
| <p>You can pass functions with multiple arguments to <code>CombineValues</code>. |
| They are passed as additional positional arguments or keyword arguments to the function.</p> |
| <p>In this example, the lambda function takes <code>values</code> and <code>max_value</code> as arguments.</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_CombineValuesMultipleArguments" |
| data-show="combinevalues_multiple_arguments" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_CombineValuesMultipleArguments%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22combinevalues_multiple_arguments%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h3 id="example-5-combining-with-a-combinefn">Example 5: Combining with a <code>CombineFn</code></h3> |
| <p>The more general way to combine elements, and the most flexible, is with a class that inherits from <code>CombineFn</code>.</p> |
| <ul> |
| <li> |
| <p><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.create_accumulator"><code>CombineFn.create_accumulator()</code></a>: |
| This creates an empty accumulator. |
| For example, an empty accumulator for a sum would be <code>0</code>, while an empty accumulator for a product (multiplication) would be <code>1</code>.</p> |
| </li> |
| <li> |
| <p><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.add_input"><code>CombineFn.add_input()</code></a>: |
| Called <em>once per element</em>. |
| Takes an accumulator and an input element, combines them and returns the updated accumulator.</p> |
| </li> |
| <li> |
| <p><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.merge_accumulators"><code>CombineFn.merge_accumulators()</code></a>: |
| Multiple accumulators could be processed in parallel, so this function helps merging them into a single accumulator.</p> |
| </li> |
| <li> |
| <p><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.extract_output"><code>CombineFn.extract_output()</code></a>: |
| It allows to do additional calculations before extracting a result.</p> |
| </li> |
| </ul> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_CombineValuesCombineFn" |
| data-show="combinevalues_combinefn" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_CombineValuesCombineFn%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22combinevalues_combinefn%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2> |
| <p>You can use the following combiner transforms:</p> |
| <ul> |
| <li><a href="/documentation/transforms/python/aggregation/combineglobally">CombineGlobally</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/combineperkey">CombinePerKey</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/mean">Mean</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/count">Count</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/top">Top</a></li> |
| <li><a href="/documentation/transforms/python/aggregation/sample">Sample</a></li> |
| </ul> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineValues" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p></description></item><item><title>Documentation: CombineWithContext</title><link>/documentation/transforms/java/aggregation/combinewithcontext/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/java/aggregation/combinewithcontext/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="combinewithcontext">CombineWithContext</h1> |
| <table align="left"> |
| <a target="_blank" class="button" |
| href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/CombineWithContext.html"> |
| <img src="/images/logos/sdks/java.png" width="20px" height="20px" |
| alt="Javadoc" /> |
| Javadoc |
| </a> |
| </table> |
| <br><br> |
| <p>A class of transforms that contains combine functions that have access to <code>PipelineOptions</code> and side inputs through <code>CombineWithContext.Context</code>.</p> |
| <h2 id="examples">Examples</h2> |
| <p>See <a href="https://issues.apache.org/jira/browse/BEAM-7703">BEAM-7703</a> for updates.</p> |
| <h2 id="related-transforms">Related transforms</h2> |
| <ul> |
| <li><a href="/documentation/transforms/java/aggregation/combine">Combine</a> |
| for combining all values associated with a key to a single result</li> |
| </ul></description></item><item><title>Documentation: CombineWithContext</title><link>/documentation/transforms/python/aggregation/combinewithcontext/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/python/aggregation/combinewithcontext/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="combinewithcontext">CombineWithContext</h1> |
| <h2 id="examples">Examples</h2> |
| <p>See <a href="https://github.com/apache/beam/issues/19547">Issue 19547</a> for updates.</p> |
| <h2 id="related-transforms">Related transforms</h2></description></item><item><title>Documentation: Container environments</title><link>/documentation/runtime/environments/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/runtime/environments/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="container-environments">Container environments</h1> |
| <p>The Beam SDK runtime environment can be <a href="https://www.docker.com/resources/what-container">containerized</a> with <a href="https://www.docker.com/">Docker</a> to isolate it from other runtime systems. To learn more about the container environment, read the Beam <a href="https://s.apache.org/beam-fn-api-container-contract">SDK Harness container contract</a>.</p> |
| <p>Prebuilt SDK container images are released per supported language during Beam releases and pushed to <a href="https://hub.docker.com/search?q=apache%2Fbeam&amp;type=image">Docker Hub</a>.</p> |
| <h2 id="custom-containers">Custom containers</h2> |
| <p>You may want to customize container images for many reasons, including:</p> |
| <ul> |
| <li>Pre-installing additional dependencies</li> |
| <li>Launching third-party software in the worker environment</li> |
| <li>Further customizing the execution environment</li> |
| </ul> |
| <p>This guide describes how to create and use customized containers for the Beam SDKs.</p> |
| <h3 id="prerequisites">Prerequisites</h3> |
| <ul> |
| <li>This guide requires building images using Docker. <a href="https://docs.docker.com/get-docker/">Install Docker locally</a>. Some CI/CD platforms like <a href="https://cloud.google.com/cloud-build/docs/building/build-containers">Google Cloud Build</a> also provide the ability to build images using Docker.</li> |
| <li>For remote execution engines/runners, have a container registry to host your custom container image. Options include <a href="https://hub.docker.com/">Docker Hub</a> or a &ldquo;self-hosted&rdquo; repository, including cloud-specific container registries like <a href="https://cloud.google.com/container-registry">Google Container Registry</a> (GCR) or <a href="https://aws.amazon.com/ecr/">Amazon Elastic Container Registry</a> (ECR). Make sure your registry can be accessed by your execution engine or runner.</li> |
| </ul> |
| <blockquote> |
| <p><strong>NOTE</strong>: On Nov 20, 2020, Docker Hub put <a href="https://www.docker.com/increase-rate-limits">rate limits</a> into effect for anonymous and free authenticated use, which may impact larger pipelines that pull containers several times.</p> |
| </blockquote> |
| <p>For optimal user experience, we also recommend you use the latest released version of Beam.</p> |
| <h3 id="building-and-pushing-custom-containers">Building and pushing custom containers</h3> |
| <p>Beam <a href="https://hub.docker.com/search?q=apache%2Fbeam&amp;type=image">SDK container images</a> are built from Dockerfiles checked into the <a href="https://github.com/apache/beam">Github</a> repository and published to Docker Hub for every release. You can build customized containers in one of three ways:</p> |
| <ol> |
| <li><strong><a href="#writing-new-dockerfiles">Writing a new</a> Dockerfile based on a released container image</strong>. This is sufficient for simple additions to the image, such as adding artifacts or environment variables.</li> |
| <li><strong><a href="#modifying-dockerfiles">Modifying</a> a source Dockerfile in <a href="https://github.com/apache/beam">Beam</a></strong>. This method requires building from Beam source but allows for greater customization of the container (including replacement of artifacts or base OS/language versions).</li> |
| <li><strong><a href="#modify-existing-base-image">Modifying</a> an existing container image to make it compatible with Apache Beam Runners</strong>. This method is used when users start from an existing image, and configure the image to be compatible with Apache Beam Runners.</li> |
| </ol> |
| <h4 id="writing-new-dockerfiles">Writing a new Dockerfile based on an existing published container image</h4> |
| <ol> |
| <li>Create a new Dockerfile that designates a base image using the <a href="https://docs.docker.com/engine/reference/builder/#from">FROM instruction</a>.</li> |
| </ol> |
| <pre tabindex="0"><code>FROM apache/beam_python3.7_sdk:2.25.0 |
| ENV FOO=bar |
| COPY /src/path/to/file /dest/path/to/file/ |
| </code></pre><p>This <code>Dockerfile</code> uses the prebuilt Python 3.7 SDK container image <a href="https://hub.docker.com/r/apache/beam_python3.7_sdk"><code>beam_python3.7_sdk</code></a> tagged at (SDK version) <code>2.25.0</code>, and adds an additional environment variable and file to the image.</p> |
| <ol start="2"> |
| <li><a href="https://docs.docker.com/engine/reference/commandline/build/">Build</a> and <a href="https://docs.docker.com/engine/reference/commandline/push/">push</a> the image using Docker.</li> |
| </ol> |
| <pre tabindex="0"><code>export BASE_IMAGE=&#34;apache/beam_python3.7_sdk:2.25.0&#34; |
| export IMAGE_NAME=&#34;myremoterepo/mybeamsdk&#34; |
| # Avoid using `latest` with custom containers to make reproducing failures easier. |
| export TAG=&#34;mybeamsdk-versioned-tag&#34; |
| # Optional - pull the base image into your local Docker daemon to ensure |
| # you have the most up-to-date version of the base image locally. |
| docker pull &#34;${BASE_IMAGE}&#34; |
| docker build -f Dockerfile -t &#34;${IMAGE_NAME}:${TAG}&#34; . |
| </code></pre><ol start="3"> |
| <li>If your runner is running remotely, retag and <a href="https://docs.docker.com/engine/reference/commandline/push/">push</a> the image to the appropriate repository.</li> |
| </ol> |
| <pre tabindex="0"><code>docker push &#34;${IMAGE_NAME}:${TAG}&#34; |
| </code></pre><ol start="4"> |
| <li>After pushing a container image, verify the remote image ID and digest matches the local image ID and digest, output from <code>docker build</code> or <code>docker images</code>.</li> |
| </ol> |
| <h4 id="modifying-dockerfiles">Modifying a source Dockerfile in Beam</h4> |
| <p>This method requires building image artifacts from Beam source. For additional instructions on setting up your development environment, see the <a href="/contribute/#development-setup">Contribution guide</a>.</p> |
| <blockquote> |
| <p><strong>NOTE</strong>: It is recommended that you start from a stable release branch (<code>release-X.XX.X</code>) corresponding to the same version of the SDK to run your pipeline. Differences in SDK version may result in unexpected errors.</p> |
| </blockquote> |
| <ol> |
| <li>Clone the <code>beam</code> repository.</li> |
| </ol> |
| <pre tabindex="0"><code>export BEAM_SDK_VERSION=&#34;2.26.0&#34; |
| git clone https://github.com/apache/beam.git |
| cd beam |
| # Save current directory as working directory |
| export BEAM_WORKDIR=$PWD |
| git checkout origin/release-$BEAM_SDK_VERSION |
| </code></pre><ol start="2"> |
| <li> |
| <p>Customize the <code>Dockerfile</code> for a given language, typically <code>sdks/&lt;language&gt;/container/Dockerfile</code> directory (e.g. the <a href="https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile">Dockerfile for Python</a>.</p> |
| </li> |
| <li> |
| <p>Return to the root Beam directory and run the Gradle <code>docker</code> target for your image.</p> |
| </li> |
| </ol> |
| <pre tabindex="0"><code>cd $BEAM_WORKDIR |
| # The default repository of each SDK |
| ./gradlew :sdks:java:container:java8:docker |
| ./gradlew :sdks:java:container:java11:docker |
| ./gradlew :sdks:java:container:java17:docker |
| ./gradlew :sdks:go:container:docker |
| ./gradlew :sdks:python:container:py38:docker |
| ./gradlew :sdks:python:container:py39:docker |
| ./gradlew :sdks:python:container:py310:docker |
| ./gradlew :sdks:python:container:py311:docker |
| # Shortcut for building all Python SDKs |
| ./gradlew :sdks:python:container:buildAll |
| </code></pre><ol start="4"> |
| <li>Verify the images you built were created by running <code>docker images</code>.</li> |
| </ol> |
| <pre tabindex="0"><code>$&gt; docker images --digests |
| REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE |
| apache/beam_java8_sdk latest sha256:... ... 1 min ago ... |
| apache/beam_java11_sdk latest sha256:... ... 1 min ago ... |
| apache/beam_java17_sdk latest sha256:... ... 1 min ago ... |
| apache/beam_python3.6_sdk latest sha256:... ... 1 min ago ... |
| apache/beam_python3.7_sdk latest sha256:... ... 1 min ago ... |
| apache/beam_python3.8_sdk latest sha256:... ... 1 min ago ... |
| apache/beam_python3.9_sdk latest sha256:... ... 1 min ago ... |
| apache/beam_python3.10_sdk latest sha256:... ... 1 min ago ... |
| apache/beam_go_sdk latest sha256:... ... 1 min ago ... |
| </code></pre><ol start="5"> |
| <li>If your runner is running remotely, retag the image and <a href="https://docs.docker.com/engine/reference/commandline/push/">push</a> the image to your repository. You can skip this step if you provide a custom repo/tag as <a href="#additional-build-parameters">additional parameters</a>.</li> |
| </ol> |
| <pre tabindex="0"><code>export BEAM_SDK_VERSION=&#34;2.26.0&#34; |
| export IMAGE_NAME=&#34;gcr.io/my-gcp-project/beam_python3.7_sdk&#34; |
| export TAG=&#34;${BEAM_SDK_VERSION}-custom&#34; |
| docker tag apache/beam_python3.7_sdk &#34;${IMAGE_NAME}:${TAG}&#34; |
| docker push &#34;${IMAGE_NAME}:${TAG}&#34; |
| </code></pre><ol start="6"> |
| <li>After pushing a container image, verify the remote image ID and digest matches the local image ID and digest output from <code>docker_images --digests</code>.</li> |
| </ol> |
| <h4 id="additional-build-parameters">Additional build parameters</h4> |
| <p>The docker Gradle task defines a default image repository and <a href="https://docs.docker.com/engine/reference/commandline/tag/">tag</a> is the SDK version defined at <a href="https://github.com/apache/beam/blob/master/gradle.properties">gradle.properties</a>. The default repository is the Docker Hub <code>apache</code> namespace, and the default tag is the <a href="https://github.com/apache/beam/blob/master/gradle.properties">SDK version</a> defined at gradle.properties.</p> |
| <p>You can specify a different repository or tag for built images by providing parameters to the build task. For example:</p> |
| <pre tabindex="0"><code>./gradlew :sdks:python:container:py36:docker -Pdocker-repository-root=&#34;example-repo&#34; -Pdocker-tag=&#34;2.26.0-custom&#34; |
| </code></pre><p>builds the Python 3.6 container and tags it as <code>example-repo/beam_python3.6_sdk:2.26.0-custom</code>.</p> |
| <p>From Beam 2.21.0 and later, a <code>docker-pull-licenses</code> flag was introduced to add licenses/notices for third party dependencies to the docker images. For example:</p> |
| <pre tabindex="0"><code>./gradlew :sdks:java:container:java8:docker -Pdocker-pull-licenses |
| </code></pre><p>creates a Java 8 SDK image with appropriate licenses in <code>/opt/apache/beam/third_party_licenses/</code>.</p> |
| <p>By default, no licenses/notices are added to the docker images.</p> |
| <h4 id="modify-existing-base-image">Modifying an existing container image to make it compatible with Apache Beam Runners</h4> |
| <p>Beam offers a way to provide your own custom container image. The easiest way to build a new custom image that is compatible with Apache Beam Runners is to use a <a href="https://docs.docker.com/develop/develop-images/multistage-build/">multi-stage build</a> process. This copies over the necessary artifacts from a default Apache Beam base image to build your custom container image.</p> |
| <ol> |
| <li>Copy necessary artifacts from Apache Beam base image to your image.</li> |
| </ol> |
| <pre tabindex="0"><code># This can be any container image, |
| FROM python:3.8-bookworm |
| # Install SDK. (needed for Python SDK) |
| RUN pip install --no-cache-dir apache-beam[gcp]==2.52.0 |
| # Copy files from official SDK image, including script/dependencies. |
| COPY --from=apache/beam_python3.8_sdk:2.52.0 /opt/apache/beam /opt/apache/beam |
| # Perform any additional customizations if desired |
| # Set the entrypoint to Apache Beam SDK launcher. |
| ENTRYPOINT [&#34;/opt/apache/beam/boot&#34;] |
| </code></pre><blockquote> |
| <p><strong>NOTE</strong>: This example assumes necessary dependencies (in this case, Python 3.8 and pip) have been installed on the existing base image. Installing the Apache Beam SDK into the image will ensure that the image has the necessary SDK dependencies and reduce the worker startup time. |
| The version specified in the <code>RUN</code> instruction must match the version used to launch the pipeline.<br> |
| <strong>Make sure that the Python or Java runtime version specified in the base image is the same as the version used to run the pipeline.</strong></p> |
| </blockquote> |
| <blockquote> |
| <p><strong>NOTE</strong>: Any additional Python dependenices should be installed in the global Python environment in the custom image.</p> |
| </blockquote> |
| <ol start="2"> |
| <li><a href="https://docs.docker.com/engine/reference/commandline/build/">Build</a> and <a href="https://docs.docker.com/engine/reference/commandline/push/">push</a> the image using Docker.</li> |
| </ol> |
| <pre tabindex="0"><code> export BASE_IMAGE=&#34;apache/beam_python3.8_sdk:2.52.0&#34; |
| export IMAGE_NAME=&#34;myremoterepo/mybeamsdk&#34; |
| export TAG=&#34;latest&#34; |
| # Optional - pull the base image into your local Docker daemon to ensure |
| # you have the most up-to-date version of the base image locally. |
| docker pull &#34;${BASE_IMAGE}&#34; |
| docker build -f Dockerfile -t &#34;${IMAGE_NAME}:${TAG}&#34; . |
| </code></pre><ol start="3"> |
| <li>If your runner is running remotely, retag the image and <a href="https://docs.docker.com/engine/reference/commandline/push/">push</a> the image to your repository.</li> |
| </ol> |
| <pre tabindex="0"><code>docker push &#34;${IMAGE_NAME}:${TAG}&#34; |
| </code></pre><h4 id="from-scratch-go">Building a compatible container image from scratch (Go)</h4> |
| <p>From the 2.55.0 release, the Beam Go SDK has moved to using <a href="https://github.com/GoogleContainerTools/distroless">distroless images</a> as a base. |
| These images have a reduced security attack surface by not including common tools and utilities. |
| This may cause difficulties customizing the image with using one of the above approaches. |
| As a fallback, it&rsquo;s possible to build a custom image from scratch, by building a matching boot loader, and setting |
| that as the container&rsquo;s entry point.</p> |
| <p>For example, if it&rsquo;s preferable to use alpine as the container OS your multi-stage docker file might |
| look like the following:</p> |
| <pre tabindex="0"><code>FROM golang:latest-alpine AS build_base |
| # Set the Current Working Directory inside the container |
| WORKDIR /tmp/beam |
| # Build the Beam Go bootloader, to the local directory, matching your Beam version. |
| # Similar go targets exist for other SDK languages. |
| RUN GOBIN=`pwd` go install github.com/apache/beam/sdks/v2/go/container@v2.53.0 |
| # Set the real base image. |
| FROM alpine:3.9 |
| RUN apk add ca-certificates |
| # The following are required for the container to operate correctly. |
| # Copy the boot loader `container` to the image. |
| COPY --from=build_base /tmp/beam/container /opt/apache/beam/boot |
| # Set the container to use the newly built boot loader. |
| ENTRYPOINT [&#34;/opt/apache/beam/boot&#34;] |
| </code></pre><p>Build and push the new image as when <a href="#modify-existing-base-image">modifying an existing base image</a> above.</p> |
| <blockquote> |
| <p><strong>NOTE</strong>: Java and Python require additional dependencies, such as their runtimes, and SDK packages for |
| a valid container image. The bootloader isn&rsquo;t sufficient for creating a custom container for these SDKs.</p> |
| </blockquote> |
| <h2 id="running-pipelines">Running pipelines with custom container images</h2> |
| <p>The common method for providing a container image requires using the |
| PortableRunner flag <code>--environment_config</code> as supported by the Portable |
| Runner or by runners supported PortableRunner flags. |
| Other runners, such as Dataflow, support specifying containers with different flags.</p> |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">export IMAGE=&#34;my-repo/beam_python_sdk_custom&#34; |
| export TAG=&#34;X.Y.Z&#34; |
| export IMAGE_URL=&#34;${IMAGE}:${TAG}&#34; |
| python -m apache_beam.examples.wordcount \ |
| --input=/path/to/inputfile \ |
| --output /path/to/write/counts \ |
| --runner=PortableRunner \ |
| --job_endpoint=embed \ |
| --environment_type=&#34;DOCKER&#34; \ |
| --environment_config=&#34;${IMAGE_URL}&#34;</code></pre> |
| </div> |
| </div> |
| <div class='runner-flink snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flink" data-lang="flink">export IMAGE=&#34;my-repo/beam_python_sdk_custom&#34; |
| export TAG=&#34;X.Y.Z&#34; |
| export IMAGE_URL = &#34;${IMAGE}:${TAG}&#34; |
| # Run a pipeline using the FlinkRunner which starts a Flink job server. |
| python -m apache_beam.examples.wordcount \ |
| --input=/path/to/inputfile \ |
| --output=path/to/write/counts \ |
| --runner=FlinkRunner \ |
| # When running batch jobs locally, we need to reuse the container. |
| --environment_cache_millis=10000 \ |
| --environment_type=&#34;DOCKER&#34; \ |
| --environment_config=&#34;${IMAGE_URL}&#34;</code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark">export IMAGE=&#34;my-repo/beam_python_sdk_custom&#34; |
| export TAG=&#34;X.Y.Z&#34; |
| export IMAGE_URL = &#34;${IMAGE}:${TAG}&#34; |
| # Run a pipeline using the SparkRunner which starts the Spark job server |
| python -m apache_beam.examples.wordcount \ |
| --input=/path/to/inputfile \ |
| --output=path/to/write/counts \ |
| --runner=SparkRunner \ |
| # When running batch jobs locally, we need to reuse the container. |
| --environment_cache_millis=10000 \ |
| --environment_type=&#34;DOCKER&#34; \ |
| --environment_config=&#34;${IMAGE_URL}&#34;</code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow">export GCS_PATH=&#34;gs://my-gcs-bucket&#34; |
| export GCP_PROJECT=&#34;my-gcp-project&#34; |
| export REGION=&#34;us-central1&#34; |
| # By default, the Dataflow runner has access to the GCR images |
| # under the same project. |
| export IMAGE=&#34;my-repo/beam_python_sdk_custom&#34; |
| export TAG=&#34;X.Y.Z&#34; |
| export IMAGE_URL = &#34;${IMAGE}:${TAG}&#34; |
| # Run a pipeline on Dataflow. |
| # This is a Python batch pipeline, so to run on Dataflow Runner V2 |
| # you must specify the experiment &#34;use_runner_v2&#34; |
| python -m apache_beam.examples.wordcount \ |
| --input gs://dataflow-samples/shakespeare/kinglear.txt \ |
| --output &#34;${GCS_PATH}/counts&#34; \ |
| --runner DataflowRunner \ |
| --project $GCP_PROJECT \ |
| --region $REGION \ |
| --temp_location &#34;${GCS_PATH}/tmp/&#34; \ |
| --experiment=use_runner_v2 \ |
| --sdk_container_image=$IMAGE_URL</code></pre> |
| </div> |
| </div> |
| <p>Avoid using the tag <code>:latest</code> with your custom images. Tag your builds with a date |
| or a unique identifier. If something goes wrong, using this type of tag might make |
| it possible to revert the pipeline execution to a previously known working |
| configuration and allow for an inspection of changes.</p> |
| <h3 id="troubleshooting">Troubleshooting</h3> |
| <p>The following section describes some common issues to consider |
| when you encounter unexpected errors running Beam pipelines with |
| custom containers.</p> |
| <ul> |
| <li>Differences in language and SDK version between the container SDK and |
| pipeline SDK may result in unexpected errors due to incompatibility. For best |
| results, make sure to use the same stable SDK version for your base container |
| and when running your pipeline.</li> |
| <li>If you are running into unexpected errors when using remote containers, |
| make sure that your container exists in the remote repository and can be |
| accessed by any third-party service, if needed.</li> |
| <li>Local runners attempt to pull remote images and default to local |
| images. If an image cannot be pulled locally (by the docker daemon), |
| you may see an log message like: |
| <pre tabindex="0"><code>Error response from daemon: manifest for remote.repo/beam_python3.7_sdk:2.25.0-custom not found: manifest unknown: ... |
| INFO:apache_beam.runners.portability.fn_api_runner.worker_handlers:Unable to pull image... |
| </code></pre></li> |
| </ul></description></item><item><title>Documentation: Count</title><link>/documentation/transforms/java/aggregation/count/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/java/aggregation/count/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="count">Count</h1> |
| <table align="left"> |
| <a target="_blank" class="button" |
| href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Count.html"> |
| <img src="/images/logos/sdks/java.png" width="20px" height="20px" |
| alt="Javadoc" /> |
| Javadoc |
| </a> |
| </table> |
| <br><br> |
| <p>Counts the number of elements within each aggregation. The <code>Count</code> |
| transform has three varieties:</p> |
| <ul> |
| <li><code>Count.globally()</code> counts the number of elements in the entire |
| <code>PCollection</code>. The result is a collection with a single element.</li> |
| <li><code>Count.perKey()</code> counts how many elements are associated with each |
| key. It ignores the values. The resulting collection has one |
| output for every key in the input collection.</li> |
| <li><code>Count.perElement()</code> counts how many times each element appears |
| in the input collection. The output collection is a key-value |
| pair, containing each unique element and the number of times it |
| appeared in the original collection.</li> |
| </ul> |
| <h2 id="examples">Examples</h2> |
| <p><strong>Example 1</strong>: Count.globally</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-java playground-snippet" |
| data-sdk="java" |
| data-path="SDK_JAVA_Count" |
| data-show="main_section" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_JAVA_Count%22%2c%22sdk%22%3a%22java%22%2c%22show%22%3a%22main_section%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <p><strong>Example 2</strong>: Count.perKey</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-java playground-snippet" |
| data-sdk="java" |
| data-path="SDK_JAVA_CountPerKey" |
| data-show="main_section" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_JAVA_CountPerKey%22%2c%22sdk%22%3a%22java%22%2c%22show%22%3a%22main_section%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2> |
| <ul> |
| <li><a href="/documentation/transforms/java/aggregation/approximateunique">ApproximateUnique</a> |
| estimates the number of distinct elements or distinct values in key-value pairs</li> |
| <li><a href="/documentation/transforms/java/aggregation/sum">Sum</a> computes |
| the sum of elements in a collection</li> |
| </ul></description></item><item><title>Documentation: Count</title><link>/documentation/transforms/python/aggregation/count/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/python/aggregation/count/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="count">Count</h1> |
| <script type="text/javascript"> |
| localStorage.setItem("language", "language-py") |
| </script> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Count" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p> |
| <p>Counts the number of elements within each aggregation.</p> |
| <h2 id="examples">Examples</h2> |
| <p>In the following example, we create a pipeline with two <code>PCollection</code>s of produce. |
| Then, we apply <code>Count</code> to get the total number of elements in different ways.</p> |
| <h3 id="example-1-counting-all-elements-in-a-pcollection">Example 1: Counting all elements in a PCollection</h3> |
| <p>We use <code>Count.Globally()</code> to count <em>all</em> elements in a <code>PCollection</code>, even if there are duplicate elements.</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_CountGlobally" |
| data-show="count_globally" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_CountGlobally%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22count_globally%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h3 id="example-2-counting-elements-for-each-key">Example 2: Counting elements for each key</h3> |
| <p>We use <code>Count.PerKey()</code> to count the elements for each unique key in a <code>PCollection</code> of key-values.</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_CountPerKey" |
| data-show="count_per_key" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_CountPerKey%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22count_per_key%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h3 id="example-3-counting-all-unique-elements">Example 3: Counting all unique elements</h3> |
| <p>We use <code>Count.PerElement()</code> to count the only the unique elements in a <code>PCollection</code>.</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_CountPerElement" |
| data-show="count_per_element" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_CountPerElement%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22count_per_element%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2> |
| <p>N/A</p> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.combiners.html#apache_beam.transforms.combiners.Count" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p></description></item><item><title>Documentation: Create</title><link>/documentation/transforms/java/other/create/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/java/other/create/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="create">Create</h1> |
| <table align="left"> |
| <a target="_blank" class="button" |
| href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Create.html"> |
| <img src="/images/logos/sdks/java.png" width="20px" height="20px" |
| alt="Javadoc" /> |
| Javadoc |
| </a> |
| </table> |
| <br><br> |
| <p>Creates a collection containing a specified set of elements. This is useful |
| for testing, as well as creating an initial input to process in parallel. |
| For example, a single element to execute a one-time <code>ParDo</code> or a list of filenames to be read.</p> |
| <h2 id="examples">Examples</h2> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-java playground-snippet" |
| data-sdk="java" |
| data-path="SDK_JAVA_Create" |
| data-show="main_section" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_JAVA_Create%22%2c%22sdk%22%3a%22java%22%2c%22show%22%3a%22main_section%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2> |
| <p>N/A</p></description></item><item><title>Documentation: Create</title><link>/documentation/transforms/python/other/create/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/python/other/create/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="create">Create</h1> |
| <table align="left"> |
| <a target="_blank" class="button" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Create"> |
| <img src="/images/logos/sdks/python.png" width="20px" height="20px" |
| alt="Pydoc" /> |
| Pydoc |
| </a> |
| </table> |
| <br><br> |
| <p>Creates a collection containing a specified set of elements. This is |
| useful for testing, as well as creating an initial input to process |
| in parallel. For example, a single element to execute a one-time |
| <code>ParDo</code> or a list of filenames to be read.</p> |
| <h2 id="examples">Examples</h2> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_Create" |
| data-show="create" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_Create%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22create%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2></description></item><item><title>Documentation: Create Your Pipeline</title><link>/documentation/pipelines/create-your-pipeline/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/pipelines/create-your-pipeline/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="create-your-pipeline">Create Your Pipeline</h1> |
| <nav id="TableOfContents"> |
| <ul> |
| <li><a href="#creating-your-pipeline-object">Creating Your Pipeline Object</a></li> |
| <li><a href="#reading-data-into-your-pipeline">Reading Data Into Your Pipeline</a></li> |
| <li><a href="#applying-transforms-to-process-pipeline-data">Applying Transforms to Process Pipeline Data</a></li> |
| <li><a href="#writing-or-outputting-your-final-pipeline-data">Writing or Outputting Your Final Pipeline Data</a></li> |
| <li><a href="#running-your-pipeline">Running Your Pipeline</a></li> |
| <li><a href="#whats-next">What&rsquo;s next</a></li> |
| </ul> |
| </nav> |
| <p>Your Beam program expresses a data processing pipeline, from start to finish. This section explains the mechanics of using the classes in the Beam SDKs to build a pipeline. To construct a pipeline using the classes in the Beam SDKs, your program will need to perform the following general steps:</p> |
| <ul> |
| <li>Create a <code>Pipeline</code> object.</li> |
| <li>Use a <strong>Read</strong> or <strong>Create</strong> transform to create one or more <code>PCollection</code>s for your pipeline data.</li> |
| <li>Apply <strong>transforms</strong> to each <code>PCollection</code>. Transforms can change, filter, group, analyze, or otherwise process the elements in a <code>PCollection</code>. Each transform creates a new output <code>PCollection</code>, to which you can apply additional transforms until processing is complete.</li> |
| <li><strong>Write</strong> or otherwise output the final, transformed <code>PCollection</code>s.</li> |
| <li><strong>Run</strong> the pipeline.</li> |
| </ul> |
| <h2 id="creating-your-pipeline-object">Creating Your Pipeline Object</h2> |
| <p>A Beam program often starts by creating a <code>Pipeline</code> object.</p> |
| <p>In the Beam SDKs, each pipeline is represented by an explicit object of type <code>Pipeline</code>. Each <code>Pipeline</code> object is an independent entity that encapsulates both the data the pipeline operates over and the transforms that get applied to that data.</p> |
| <p>To create a pipeline, declare a <code>Pipeline</code> object, and pass it some <a href="/documentation/programming-guide#configuring-pipeline-options">configuration options</a>.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Start by defining the options for the pipeline. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PipelineOptions</span> <span class="n">options</span> <span class="o">=</span> <span class="n">PipelineOptionsFactory</span><span class="o">.</span><span class="na">create</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Then create the pipeline. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">Pipeline</span> <span class="n">p</span> <span class="o">=</span> <span class="n">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h2 id="reading-data-into-your-pipeline">Reading Data Into Your Pipeline</h2> |
| <p>To create your pipeline&rsquo;s initial <code>PCollection</code>, you apply a root transform to your pipeline object. A root transform creates a <code>PCollection</code> from either an external data source or some local data you specify.</p> |
| <p>There are two kinds of root transforms in the Beam SDKs: <code>Read</code> and <code>Create</code>. <code>Read</code> transforms read data from an external source, such as a text file or a database table. <code>Create</code> transforms create a <code>PCollection</code> from an in-memory <code>java.util.Collection</code>.</p> |
| <p>The following example code shows how to <code>apply</code> a <code>TextIO.Read</code> root transform to read data from a text file. The transform is applied to a <code>Pipeline</code> object <code>p</code>, and returns a pipeline data set in the form of a <code>PCollection&lt;String&gt;</code>:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">lines</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;ReadLines&#34;</span><span class="o">,</span> <span class="n">TextIO</span><span class="o">.</span><span class="na">read</span><span class="o">().</span><span class="na">from</span><span class="o">(</span><span class="s">&#34;gs://some/inputData.txt&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h2 id="applying-transforms-to-process-pipeline-data">Applying Transforms to Process Pipeline Data</h2> |
| <p>You can manipulate your data using the various <a href="/documentation/programming-guide/#transforms">transforms</a> provided in the Beam SDKs. To do this, you <strong>apply</strong> the transforms to your pipeline&rsquo;s <code>PCollection</code> by calling the <code>apply</code> method on each <code>PCollection</code> that you want to process and passing the desired transform object as an argument.</p> |
| <p>The following code shows how to <code>apply</code> a transform to a <code>PCollection</code> of strings. The transform is a user-defined custom transform that reverses the contents of each string and outputs a new <code>PCollection</code> containing the reversed strings.</p> |
| <p>The input is a <code>PCollection&lt;String&gt;</code> called <code>words</code>; the code passes an instance of a <code>PTransform</code> object called <code>ReverseWords</code> to <code>apply</code>, and saves the return value as the <code>PCollection&lt;String&gt;</code> called <code>reversedWords</code>.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">words</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">reversedWords</span> <span class="o">=</span> <span class="n">words</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="k">new</span> <span class="n">ReverseWords</span><span class="o">());</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h2 id="writing-or-outputting-your-final-pipeline-data">Writing or Outputting Your Final Pipeline Data</h2> |
| <p>Once your pipeline has applied all of its transforms, you&rsquo;ll usually need to output the results. To output your pipeline&rsquo;s final <code>PCollection</code>s, you apply a <code>Write</code> transform to that <code>PCollection</code>. <code>Write</code> transforms can output the elements of a <code>PCollection</code> to an external data sink, such as a database table. You can use <code>Write</code> to output a <code>PCollection</code> at any time in your pipeline, although you&rsquo;ll typically write out data at the end of your pipeline.</p> |
| <p>The following example code shows how to <code>apply</code> a <code>TextIO.Write</code> transform to write a <code>PCollection</code> of <code>String</code> to a text file:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">filteredWords</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">filteredWords</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;WriteMyFile&#34;</span><span class="o">,</span> <span class="n">TextIO</span><span class="o">.</span><span class="na">write</span><span class="o">().</span><span class="na">to</span><span class="o">(</span><span class="s">&#34;gs://some/outputData.txt&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h2 id="running-your-pipeline">Running Your Pipeline</h2> |
| <p>Once you have constructed your pipeline, use the <code>run</code> method to execute the pipeline. Pipelines are executed asynchronously: the program you create sends a specification for your pipeline to a <strong>pipeline runner</strong>, which then constructs and runs the actual series of pipeline operations.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">run</span><span class="o">();</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>The <code>run</code> method is asynchronous. If you&rsquo;d like a blocking execution instead, run your pipeline appending the <code>waitUntilFinish</code> method:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">run</span><span class="o">().</span><span class="na">waitUntilFinish</span><span class="o">();</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h2 id="whats-next">What&rsquo;s next</h2> |
| <ul> |
| <li><a href="/documentation/programming-guide">Programming Guide</a> - Learn the details of creating your pipeline, configuring pipeline options, and applying transforms.</li> |
| <li><a href="/documentation/pipelines/test-your-pipeline">Test your pipeline</a>.</li> |
| </ul></description></item><item><title>Documentation: Cross Language RunInference</title><link>/documentation/ml/multi-language-inference/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/ml/multi-language-inference/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="using-runinference-from-java-sdk">Using RunInference from Java SDK</h1> |
| <p>The pipeline in this example is written in Java and reads the input data from Google Cloud Storage. With the help of a <a href="https://beam.apache.org/documentation/programming-guide/#1312-creating-cross-language-python-transforms">PythonExternalTransform</a>, |
| a composite Python transform is called to do the preprocessing, postprocessing, and inference. |
| Lastly, the data is written back to Google Cloud Storage in the Java pipeline.</p> |
| <p>You can find the code used in this example in the <a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/multi_language_inference">Beam repository</a>.</p> |
| <h2 id="nlp-model-and-dataset">NLP model and dataset</h2> |
| <p>A <code>bert-base-uncased</code> natural language processing (NLP) model is used to make inference. This model is open source and available on <a href="https://huggingface.co/bert-base-uncased">HuggingFace</a>. This BERT-model is |
| used to predict the last word of a sentence based on the context of the sentence.</p> |
| <p>We also use an <a href="https://www.kaggle.com/datasets/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews?select=IMDB+Dataset.csv">IMDB movie reviews</a> dataset, which is an open-source dataset that is available on Kaggle.</p> |
| <p>The following is a sample of the data after preprocessing:</p> |
| <table> |
| <thead> |
| <tr> |
| <th><strong>Text</strong></th> |
| <th style="text-align:left"><strong>Last Word</strong></th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><img width=700/></td> |
| <td style="text-align:left"><img width=100/></td> |
| </tr> |
| <tr> |
| <td>One of the other reviewers has mentioned that after watching just 1 Oz episode you&rsquo;ll be [MASK]</td> |
| <td style="text-align:left">hooked</td> |
| </tr> |
| <tr> |
| <td>A wonderful little [MASK]</td> |
| <td style="text-align:left">production</td> |
| </tr> |
| <tr> |
| <td>So im not a big fan of Boll&rsquo;s work but then again not many [MASK]</td> |
| <td style="text-align:left">are</td> |
| </tr> |
| <tr> |
| <td>This a fantastic movie of three prisoners who become [MASK]</td> |
| <td style="text-align:left">famous</td> |
| </tr> |
| <tr> |
| <td>Some films just simply should not be [MASK]</td> |
| <td style="text-align:left">remade</td> |
| </tr> |
| <tr> |
| <td>The Karen Carpenter Story shows a little more about singer Karen Carpenter&rsquo;s complex [MASK]</td> |
| <td style="text-align:left">life</td> |
| </tr> |
| </tbody> |
| </table> |
| <h2 id="multi-language-inference-pipeline">Multi-language Inference pipeline</h2> |
| <p>When using multi-language pipelines, you have access to a much larger pool of transforms. For more information, see <a href="https://beam.apache.org/documentation/programming-guide/#multi-language-pipelines">Multi-language pipelines</a> in the Apache Beam Programming Guide.</p> |
| <h3 id="custom-python-transform">Custom Python transform</h3> |
| <p>In addition to running inference, we also need to perform preprocessing and postprocessing on the data. Postprocessing the data makes it possible to interpret the output. In order to do these three tasks, one single composite custom PTransform is written, with a unit DoFn or PTransform for each of the tasks, as shown in the following snippet:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">expand</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pcoll</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pcoll</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Preprocess&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">Preprocess</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_tokenizer</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Inference&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">RunInference</span><span class="p">(</span><span class="n">KeyedModelHandler</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_model_handler</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Postprocess&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">Postprocess</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">_tokenizer</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">)</span> |
| </span></span></code></pre></div><p>First, the preprocessing of the data. In this case, the raw textual data is cleaned and tokenized for the BERT-model. All these steps are run in the <code>Preprocess</code> DoFn. The <code>Preprocess</code> DoFn takes a single element as input and returns a list with both the original text and the tokenized text.</p> |
| <p>The preprocessed data is then used to make inference. This is done in the <a href="https://beam.apache.org/documentation/ml/overview/#runinference"><code>RunInference</code></a> PTransform, which is already available in the Apache Beam SDK. The <code>RunInference</code> PTransform requires one parameter, a model handler. In this example the <code>KeyedModelHandler</code> is used, because the <code>Preprocess</code> DoFn also outputs the original sentence. You can change how preprocessing is done based on your requirements. This model handler is defined in the following initialization function of the composite PTransform:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">model</span><span class="p">,</span> <span class="n">model_path</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">_model</span> <span class="o">=</span> <span class="n">model</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;Downloading </span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">_model</span><span class="si">}</span><span class="s2"> model from GCS.&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">_model_config</span> <span class="o">=</span> <span class="n">BertConfig</span><span class="o">.</span><span class="n">from_pretrained</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_model</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">_tokenizer</span> <span class="o">=</span> <span class="n">BertTokenizer</span><span class="o">.</span><span class="n">from_pretrained</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_model</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">_model_handler</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">PytorchModelHandlerKeyedTensorWrapper</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">state_dict_path</span><span class="o">=</span><span class="p">(</span><span class="n">model_path</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">model_class</span><span class="o">=</span><span class="n">BertForMaskedLM</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">model_params</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;config&#39;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">_model_config</span><span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">device</span><span class="o">=</span><span class="s1">&#39;cuda:0&#39;</span><span class="p">)</span> |
| </span></span></code></pre></div><p>The <code>PytorchModelHandlerKeyedTensorWrapper</code>, a wrapper around the <code>PytorchModelHandlerKeyedTensor</code> model handler, is used. The <code>PytorchModelHandlerKeyedTensor</code> model handler makes inference on a PyTorch model. Because the tokenized strings generated from <code>BertTokenizer</code> might have different lengths and stack() requires tensors to be the same size, the <code>PytorchModelHandlerKeyedTensorWrapper</code> limits the batch size to 1. Restricting <code>max_batch_size</code> to 1 means the run_inference() call contains one example per batch. The following code shows the definition of the wrapper:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">PytorchModelHandlerKeyedTensorWrapper</span><span class="p">(</span><span class="n">PytorchModelHandlerKeyedTensor</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">batch_elements_kwargs</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="p">{</span><span class="s1">&#39;max_batch_size&#39;</span><span class="p">:</span> <span class="mi">1</span><span class="p">}</span> |
| </span></span></code></pre></div><p>An alternative aproach is to make all the tensors have the same length. This <a href="https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_pytorch_tensorflow_sklearn.ipynb">example</a> shows how to do that.</p> |
| <p>The <code>ModelConfig</code> and <code>ModelTokenizer</code> are loaded in the initialization function. The <code>ModelConfig</code> is used to define the model architecture, and the <code>ModelTokenizer</code> is used to tokenize the input data. The following two parameters are used for these tasks:</p> |
| <ul> |
| <li><code>model</code>: The name of the model that is used for inference. In this example it is <code>bert-base-uncased</code>.</li> |
| <li><code>model_path</code>: The path to the <code>state_dict</code> of the model that is used for inference. In this example it is a path to a Google Cloud Storage bucket, where the <code>state_dict</code> is stored.</li> |
| </ul> |
| <p>Both of these parameters are specified in the Java <code>PipelineOptions</code>.</p> |
| <p>Finally, we postprocess the model predictions in the <code>Postprocess</code> DoFn. The <code>Postprocess</code> DoFn returns the original text, the last word of the sentence, and the predicted word.</p> |
| <h3 id="compile-python-code-into-package">Compile Python code into package</h3> |
| <p>The custom Python code needs to be written in a local package and be compiled as a tarball. This package can then be used by the Java pipeline. The following example shows how to compile the Python package into a tarball:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"> pip install --upgrade build <span class="o">&amp;&amp;</span> python -m build --sdist |
| </span></span></code></pre></div><p>In order to run this, a <code>setup.py</code> is required. The path to the tarball will be used as an argument in the pipeline options of the Java pipeline.</p> |
| <h3 id="run-the-java-pipeline">Run the Java pipeline</h3> |
| <p>The Java pipeline is defined in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/multi_language_inference/last_word_prediction/src/main/java/org/apache/beam/examples/MultiLangRunInference.java#L32"><code>MultiLangRunInference</code></a> class. In this pipeline, the data is read from Google Cloud Storage, the cross-language Python transform is applied, and the output is written back to Google Cloud Storage.</p> |
| <p>The <code>PythonExternalTransform</code> is used to inject the cross-language Python transform into the Java pipeline. <code>PythonExternalTransform</code> takes a string parameter which is the fully qualified name of the Python transform.</p> |
| <p>The <code>withKwarg</code> method is used to specify the parameters that are needed for the Python transform. In this example the <code>model</code> and <code>model_path</code> parameters are specified. These parameters are used in the initialization function of the composite Python PTransform, as shown in the first section. Finally the <code>withExtraPackages</code> method is used to specify the additional Python dependencies that are needed for the Python transform. In this example the <code>local_packages</code> list is used, which contains Python requirements and the path to the compiled tarball.</p> |
| <p>To run the pipeline, use the following command:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">mvn compile exec:java -Dexec.mainClass<span class="o">=</span>org.apache.beam.examples.MultiLangRunInference <span class="se">\ |
| </span></span></span><span class="line"><span class="cl"><span class="se"></span> -Dexec.args<span class="o">=</span><span class="s2">&#34;--runner=DataflowRunner \ |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> --project=</span><span class="nv">$GCP_PROJECT</span><span class="s2">\ |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> --region=</span><span class="nv">$GCP_REGION</span><span class="s2"> \ |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> --gcpTempLocation=gs://</span><span class="nv">$GCP_BUCKET</span><span class="s2">/temp/ \ |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> --inputFile=gs://</span><span class="nv">$GCP_BUCKET</span><span class="s2">/input/imdb_reviews.csv \ |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> --outputFile=gs://</span><span class="nv">$GCP_BUCKET</span><span class="s2">/output/ouput.txt \ |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> --modelPath=gs://</span><span class="nv">$GCP_BUCKET</span><span class="s2">/input/bert-model/bert-base-uncased.pth \ |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> --modelName=</span><span class="nv">$MODEL_NAME</span><span class="s2"> \ |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> --localPackage=</span><span class="nv">$LOCAL_PACKAGE</span><span class="s2">&#34;</span> <span class="se">\ |
| </span></span></span><span class="line"><span class="cl"><span class="se"></span> -Pdataflow-runner |
| </span></span></code></pre></div><p>The standard Google Cloud and Runner parameters are specified. The <code>inputFile</code> and <code>outputFile</code> parameters are used to specify the input and output files. The <code>modelPath</code> and <code>modelName</code> custom parameters are passed to the <code>PythonExternalTransform</code>. Finally the <code>localPackage</code> parameter is used to specify the path to the compiled Python package, which contains the custom Python transform.</p> |
| <h2 id="final-remarks">Final remarks</h2> |
| <p>Use this example as a base to create other custom multi-language inference pipelines. You can also use other SDKs. For example, Go also has a wrapper that can make cross-language transforms. For more information, see <a href="https://beam.apache.org/documentation/programming-guide/#1323-using-cross-language-transforms-in-a-go-pipeline">Using cross-language transforms in a Go pipeline</a> in the Apache Beam Programming Guide.</p> |
| <p>The full code used in this example can be found on <a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/multi_language_inference">GitHub</a>.</p></description></item><item><title>Documentation: Custom I/O patterns</title><link>/documentation/patterns/custom-io/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/patterns/custom-io/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="custom-io-patterns">Custom I/O patterns</h1> |
| <p>This page describes common patterns in pipelines with <a href="/documentation/io/developing-io-overview/">custom I/O connectors</a>. Custom I/O connectors connect pipelines to databases that aren&rsquo;t supported by Beam&rsquo;s <a href="/documentation/io/connectors/">built-in I/O transforms</a>.</p> |
| <nav class="language-switcher"> |
| <strong>Adapt for:</strong> |
| <ul> |
| <li data-value="java" class="active">Java SDK</li> |
| <li data-value="py">Python SDK</li> |
| </ul> |
| </nav> |
| <h2 id="choosing-between-built-in-and-custom-connectors">Choosing between built-in and custom connectors</h2> |
| <p><a href="/documentation/io/connectors/">Built-in I/O connectors</a> are tested and hardened, so use them whenever possible. Only use custom I/O connectors when:</p> |
| <ul> |
| <li>No built-in options exist</li> |
| <li>Your pipeline pulls in a small subset of source data</li> |
| </ul> |
| <p>For instance, use a custom I/O connector to enrich pipeline elements with a small subset of source data. If you’re processing a sales order and adding information to each purchase, you can use a custom I/O connector to pull the small subset of data into your pipeline (instead of processing the entire source).</p> |
| <p>Beam distributes work across many threads, so custom I/O connectors can increase your data source’s load average. You can reduce the load with the <span class="language-java"><a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.StartBundle.html">start</a></span><span class="language-py"><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html?highlight=bundle#apache_beam.transforms.core.DoFn.start_bundle">start</a></span> and <span class="language-java"><a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/DoFn.FinishBundle.html">finish</a></span><span class="language-py"><a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html?highlight=bundle#apache_beam.transforms.core.DoFn.finish_bundle">finish</a></span> bundle annotations.</p></description></item><item><title>Documentation: Custom window patterns</title><link>/documentation/patterns/custom-windows/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/patterns/custom-windows/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="custom-window-patterns">Custom window patterns</h1> |
| <p>The samples on this page demonstrate common custom window patterns. You can create custom windows with <a href="/documentation/programming-guide/#provided-windowing-functions"><code>WindowFn</code> functions</a>. For more information, see the <a href="/documentation/programming-guide/#windowing">programming guide section on windowing</a>.</p> |
| <p><strong>Note</strong>: Custom merging windows isn&rsquo;t supported in Python (with fnapi).</p> |
| <h2 id="using-data-to-dynamically-set-session-window-gaps">Using data to dynamically set session window gaps</h2> |
| <p>You can modify the <a href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/windowing/SlidingWindows.html"><code>assignWindows</code></a> function to use data-driven gaps, then window incoming data into sessions.</p> |
| <p>Access the <code>assignWindows</code> function through <code>WindowFn.AssignContext.element()</code>. The original, fixed-duration <code>assignWindows</code> function is:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">Collection</span><span class="o">&lt;</span><span class="n">IntervalWindow</span><span class="o">&gt;</span> <span class="nf">assignWindows</span><span class="o">(</span><span class="n">WindowFn</span><span class="o">.</span><span class="na">AssignContext</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Assign each element into a window from its timestamp until gapDuration in the |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// future. Overlapping windows (representing elements within gapDuration of |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// each other) will be merged. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="k">new</span> <span class="n">IntervalWindow</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="na">timestamp</span><span class="o">(),</span> <span class="n">gapDuration</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="creating-data-driven-gaps">Creating data-driven gaps</h3> |
| <p>To create data-driven gaps, add the following snippets to the <code>assignWindows</code> function:</p> |
| <ul> |
| <li>A default value for when the custom gap is not present in the data</li> |
| <li>A way to set the attribute from the main pipeline as a method of the custom windows</li> |
| </ul> |
| <p>For example, the following function assigns each element to a window between the timestamp and <code>gapDuration</code>:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="n">Collection</span><span class="o">&lt;</span><span class="n">IntervalWindow</span><span class="o">&gt;</span> <span class="nf">assignWindows</span><span class="o">(</span><span class="n">AssignContext</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Assign each element into a window from its timestamp until gapDuration in the |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// future. Overlapping windows (representing elements within gapDuration of |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// each other) will be merged. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">Duration</span> <span class="n">dataDrivenGap</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TableRow</span> <span class="n">message</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">try</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">dataDrivenGap</span> <span class="o">=</span> <span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="n">Long</span><span class="o">.</span><span class="na">parseLong</span><span class="o">(</span><span class="n">message</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">&#34;gap&#34;</span><span class="o">).</span><span class="na">toString</span><span class="o">()));</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">dataDrivenGap</span> <span class="o">=</span> <span class="n">gapDuration</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="k">new</span> <span class="n">IntervalWindow</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="na">timestamp</span><span class="o">(),</span> <span class="n">dataDrivenGap</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Then, set the <code>gapDuration</code> field in a windowing function:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">DynamicSessions</span> <span class="kd">extends</span> <span class="n">WindowFn</span><span class="o">&lt;</span><span class="n">TableRow</span><span class="o">,</span> <span class="n">IntervalWindow</span><span class="o">&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="cm">/** Duration of the gaps between sessions. */</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">private</span> <span class="kd">final</span> <span class="n">Duration</span> <span class="n">gapDuration</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="cm">/** Creates a {@code DynamicSessions} {@link WindowFn} with the specified gap duration. */</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">private</span> <span class="nf">DynamicSessions</span><span class="o">(</span><span class="n">Duration</span> <span class="n">gapDuration</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">this</span><span class="o">.</span><span class="na">gapDuration</span> <span class="o">=</span> <span class="n">gapDuration</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="windowing-messages-into-sessions">Windowing messages into sessions</h3> |
| <p>After creating data-driven gaps, you can window incoming data into the new, custom sessions.</p> |
| <p>First, set the session length to the gap duration:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="cm">/** Creates a {@code DynamicSessions} {@link WindowFn} with the specified gap duration. */</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">static</span> <span class="n">DynamicSessions</span> <span class="nf">withDefaultGapDuration</span><span class="o">(</span><span class="n">Duration</span> <span class="n">gapDuration</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="k">new</span> <span class="n">DynamicSessions</span><span class="o">(</span><span class="n">gapDuration</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Lastly, window data into sessions in your pipeline:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;Window into sessions&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Window</span><span class="o">.&lt;</span><span class="n">TableRow</span><span class="o">&gt;</span><span class="n">into</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">DynamicSessions</span><span class="o">.</span><span class="na">withDefaultGapDuration</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="n">10</span><span class="o">))));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="example-data-and-windows">Example data and windows</h3> |
| <p>The following test data tallies two users&rsquo; scores with and without the <code>gap</code> attribute:</p> |
| <pre tabindex="0"><code>.apply(&#34;Create data&#34;, Create.timestamped( |
| TimestampedValue.of(&#34;{\&#34;user\&#34;:\&#34;user-1\&#34;,\&#34;score\&#34;:\&#34;12\&#34;,\&#34;gap\&#34;:\&#34;5\&#34;}&#34;, new Instant()), |
| TimestampedValue.of(&#34;{\&#34;user\&#34;:\&#34;user-2\&#34;,\&#34;score\&#34;:\&#34;4\&#34;}&#34;, new Instant()), |
| TimestampedValue.of(&#34;{\&#34;user\&#34;:\&#34;user-1\&#34;,\&#34;score\&#34;:\&#34;-3\&#34;,\&#34;gap\&#34;:\&#34;5\&#34;}&#34;, new Instant().plus(2000)), |
| TimestampedValue.of(&#34;{\&#34;user\&#34;:\&#34;user-1\&#34;,\&#34;score\&#34;:\&#34;2\&#34;,\&#34;gap\&#34;:\&#34;5\&#34;}&#34;, new Instant().plus(9000)), |
| TimestampedValue.of(&#34;{\&#34;user\&#34;:\&#34;user-1\&#34;,\&#34;score\&#34;:\&#34;7\&#34;,\&#34;gap\&#34;:\&#34;5\&#34;}&#34;, new Instant().plus(12000)), |
| TimestampedValue.of(&#34;{\&#34;user\&#34;:\&#34;user-2\&#34;,\&#34;score\&#34;:\&#34;10\&#34;}&#34;, new Instant().plus(12000))) |
| .withCoder(StringUtf8Coder.of())) |
| </code></pre><p>The diagram below visualizes the test data:</p> |
| <p><img src="/images/standard-vs-dynamic-sessions.png" alt="Two sets of data and the standard and dynamic sessions with which the data is windowed."></p> |
| <h4 id="standard-sessions">Standard sessions</h4> |
| <p>Standard sessions use the following windows and scores:</p> |
| <pre tabindex="0"><code>user=user-2, score=4, window=[2019-05-26T13:28:49.122Z..2019-05-26T13:28:59.122Z) |
| user=user-1, score=18, window=[2019-05-26T13:28:48.582Z..2019-05-26T13:29:12.774Z) |
| user=user-2, score=10, window=[2019-05-26T13:29:03.367Z..2019-05-26T13:29:13.367Z) |
| </code></pre><p>User #1 sees two events separated by 12 seconds. With standard sessions, the gap defaults to 10 seconds; both scores are in different sessions, so the scores aren&rsquo;t added.</p> |
| <p>User #2 sees four events, separated by two, seven, and three seconds, respectively. Since none of the gaps are greater than the default, the four events are in the same standard session and added together (18 points).</p> |
| <h4 id="dynamic-sessions">Dynamic sessions</h4> |
| <p>The dynamic sessions specify a five-second gap, so they use the following windows and scores:</p> |
| <pre tabindex="0"><code>user=user-2, score=4, window=[2019-05-26T14:30:22.969Z..2019-05-26T14:30:32.969Z) |
| user=user-1, score=9, window=[2019-05-26T14:30:22.429Z..2019-05-26T14:30:30.553Z) |
| user=user-1, score=9, window=[2019-05-26T14:30:33.276Z..2019-05-26T14:30:41.849Z) |
| user=user-2, score=10, window=[2019-05-26T14:30:37.357Z..2019-05-26T14:30:47.357Z) |
| </code></pre><p>With dynamic sessions, User #2 gets different scores. The third messages arrives seven seconds after the second message, so it&rsquo;s grouped into a different session. The large, 18-point session is split into two 9-point sessions.</p></description></item><item><title>Documentation: Data exploration</title><link>/documentation/ml/data-processing/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/ml/data-processing/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="data-exploration">Data exploration</h1> |
| <p>Several types of Apache Beam data processing are applicable to AI/ML projects:</p> |
| <ul> |
| <li>Data exploration: Learn about your data (properties, distributions, statistics) when you start to deploy your project or when the data changes.</li> |
| <li>Data preprocessing: Transform your data so that it is ready to be used to train your model.</li> |
| <li>Data postprocessing: After running inference, you might need to transform the output of your model so that it is meaningful.</li> |
| <li>Data validation: Check the quality of your data to detect outliers and calculate standard deviations and class distributions.</li> |
| </ul> |
| <p>Data processing can be grouped into two main topics. This example first examimes data exploration and then data pipelines in ML that use both data preprocessing and validation. Data postprocessing is not covered because it is similar to prepressing. Postprocessing differs only in the order and type of pipeline.</p> |
| <h2 id="initial-data-exploration">Initial data exploration</h2> |
| <p><a href="https://pandas.pydata.org/">Pandas</a> is a popular tool for performing data exploration. Pandas is a data analysis and manipulation tool for Python. It uses DataFrames, which is a data structure that contains two-dimensional tabular data and that provides labeled rows and columns for the data. The Apache Beam Python SDK provides a <a href="/documentation/dsls/dataframes/overview/">DataFrame API</a> for working with Pandas-like DataFrame objects.</p> |
| <p>The Beam DataFrame API is intended to provide access to a familiar programming interface within an Apache Beam pipeline. This API allows you to perform data exploration. You can reuse the code for your data preprocessing pipeline. Using the DataFrame API, you can build complex data processing pipelines by invoking standard Pandas commands.</p> |
| <p>You can use the DataFrame API in combination with the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/README.md">Beam interactive runner</a> in a <a href="https://cloud.google.com/dataflow/docs/guides/interactive-pipeline-development">JupyterLab notebook</a>. Use the notebook to iteratively develop pipelines and display the results of your individual pipeline steps.</p> |
| <p>The following is an example of data exploration in Apache Beam in a notebook:</p> |
| <pre tabindex="0"><code>import apache_beam as beam |
| from apache_beam.runners.interactive.interactive_runner import InteractiveRunner |
| import apache_beam.runners.interactive.interactive_beam as ib |
| p = beam.Pipeline(InteractiveRunner()) |
| beam_df = p | beam.dataframe.io.read_csv(input_path) |
| # Investigate columns and data types |
| beam_df.dtypes |
| # Generate descriptive statistics |
| ib.collect(beam_df.describe()) |
| # Investigate missing values |
| ib.collect(beam_df.isnull()) |
| </code></pre><p>For a full end-to-end example that implements data exploration and data preprocessing with Apache Beam and the DataFrame API for your AI/ML project, see the <a href="https://github.com/apache/beam/tree/master/examples/notebooks/beam-ml/dataframe_api_preprocessing.ipynb">Beam Dataframe API tutorial for AI/ML</a>.</p> |
| <h2 id="data-pipeline-for-ml">Data pipeline for ML</h2> |
| <p>A typical data preprocessing pipeline consists of the following steps:</p> |
| <ol> |
| <li>Read and write data: Read and write the data from your file system, database, or messaging queue. Apache Beam has a rich set of <a href="/documentation/io/built-in/">IO connectors</a> for ingesting and writing data.</li> |
| <li>Data cleaning: Filter and clean your data before using it in your ML model. You might remove duplicate or irrelevant data, correct mistakes in your dataset, filter out unwanted outliers, or handle missing data.</li> |
| <li>Data transformations: Your data needs to fit the expected input your model needs to train. You might need to normalize, one-hot encode, scale, or vectorize your data.</li> |
| <li>Data enrichment: You might want to enrich your data with external data sources to make your data more meaningful or easier for an ML model to interpret. For example, you might want to transform a city name or address into a set of coordinates.</li> |
| <li>Data validation and metrics: Make sure your data adheres to a specific set of requirements that can be validated in your pipeline. Report metrics from your data, such as the class distributions.</li> |
| </ol> |
| <p>You can use an Apache Beam pipeline to implement all of these steps. This example shows a pipeline that demonstrates all of the steps previously mentioned:</p> |
| <pre tabindex="0"><code>import apache_beam as beam |
| from apache_beam.metrics import Metrics |
| with beam.Pipeline() as pipeline: |
| # Create data |
| input_data = ( |
| pipeline |
| | beam.Create([ |
| {&#39;age&#39;: 25, &#39;height&#39;: 176, &#39;weight&#39;: 60, &#39;city&#39;: &#39;London&#39;}, |
| {&#39;age&#39;: 61, &#39;height&#39;: 192, &#39;weight&#39;: 95, &#39;city&#39;: &#39;Brussels&#39;}, |
| {&#39;age&#39;: 48, &#39;height&#39;: 163, &#39;weight&#39;: None, &#39;city&#39;: &#39;Berlin&#39;}])) |
| # Clean data |
| def filter_missing_data(row): |
| return row[&#39;weight&#39;] is not None |
| cleaned_data = input_data | beam.Filter(filter_missing_data) |
| # Transform data |
| def scale_min_max_data(row): |
| row[&#39;age&#39;] = (row[&#39;age&#39;]/100) |
| row[&#39;height&#39;] = (row[&#39;height&#39;]-150)/50 |
| row[&#39;weight&#39;] = (row[&#39;weight&#39;]-50)/50 |
| yield row |
| transformed_data = cleaned_data | beam.FlatMap(scale_min_max_data) |
| # Enrich data |
| side_input = pipeline | beam.io.ReadFromText(&#39;coordinates.csv&#39;) |
| def coordinates_lookup(row, coordinates): |
| row[&#39;coordinates&#39;] = coordinates.get(row[&#39;city&#39;], (0, 0)) |
| del row[&#39;city&#39;] |
| yield row |
| enriched_data = ( |
| transformed_data |
| | beam.FlatMap(coordinates_lookup, coordinates=beam.pvalue.AsDict(side_input))) |
| # Metrics |
| counter = Metrics.counter(&#39;main&#39;, &#39;counter&#39;) |
| def count_data(row): |
| counter.inc() |
| yield row |
| output_data = enriched_data | beam.FlatMap(count_data) |
| # Write data |
| output_data | beam.io.WriteToText(&#39;output.csv&#39;) |
| </code></pre></description></item><item><title>Documentation: Design Your Pipeline</title><link>/documentation/pipelines/design-your-pipeline/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/pipelines/design-your-pipeline/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="design-your-pipeline">Design Your Pipeline</h1> |
| <nav id="TableOfContents"> |
| <ul> |
| <li><a href="#what-to-consider-when-designing-your-pipeline">What to consider when designing your pipeline</a></li> |
| <li><a href="#a-basic-pipeline">A basic pipeline</a></li> |
| <li><a href="#branching-pcollections">Branching PCollections</a> |
| <ul> |
| <li><a href="#multiple-transforms-process-the-same-pcollection">Multiple transforms process the same PCollection</a></li> |
| <li><a href="#a-single-transform-that-produces-multiple-outputs">A single transform that produces multiple outputs</a></li> |
| </ul> |
| </li> |
| <li><a href="#merging-pcollections">Merging PCollections</a></li> |
| <li><a href="#multiple-sources">Multiple sources</a></li> |
| <li><a href="#whats-next">What&rsquo;s next</a></li> |
| </ul> |
| </nav> |
| <p>This page helps you design your Apache Beam pipeline. It includes information about how to determine your pipeline&rsquo;s structure, how to choose which transforms to apply to your data, and how to determine your input and output methods.</p> |
| <p>Before reading this section, it is recommended that you become familiar with the information in the <a href="/documentation/programming-guide">Beam programming guide</a>.</p> |
| <h2 id="what-to-consider-when-designing-your-pipeline">What to consider when designing your pipeline</h2> |
| <p>When designing your Beam pipeline, consider a few basic questions:</p> |
| <ul> |
| <li><strong>Where is your input data stored?</strong> How many sets of input data do you have? This will determine what kinds of <code>Read</code> transforms you&rsquo;ll need to apply at the start of your pipeline.</li> |
| <li><strong>What does your data look like?</strong> It might be plaintext, formatted log files, or rows in a database table. Some Beam transforms work exclusively on <code>PCollection</code>s of key/value pairs; you&rsquo;ll need to determine if and how your data is keyed and how to best represent that in your pipeline&rsquo;s <code>PCollection</code>(s).</li> |
| <li><strong>What do you want to do with your data?</strong> The core transforms in the Beam SDKs are general purpose. Knowing how you need to change or manipulate your data will determine how you build core transforms like <a href="/documentation/programming-guide/#pardo">ParDo</a>, or when you use pre-written transforms included with the Beam SDKs.</li> |
| <li><strong>What does your output data look like, and where should it go?</strong> This will determine what kinds of <code>Write</code> transforms you&rsquo;ll need to apply at the end of your pipeline.</li> |
| </ul> |
| <h2 id="a-basic-pipeline">A basic pipeline</h2> |
| <p>The simplest pipelines represent a linear flow of operations, as shown in figure 1.</p> |
| <p><img src="/images/design-your-pipeline-linear.svg" alt="A linear pipeline starts with one input collection, sequentially appliesthree transforms, and ends with one output collection."></p> |
| <p><em>Figure 1: A linear pipeline.</em></p> |
| <p>However, your pipeline can be significantly more complex. A pipeline represents a <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph">Directed Acyclic Graph</a> of steps. It can have multiple input sources, multiple output sinks, and its operations (<code>PTransform</code>s) can both read and output multiple <code>PCollection</code>s. The following examples show some of the different shapes your pipeline can take.</p> |
| <h2 id="branching-pcollections">Branching PCollections</h2> |
| <p>It&rsquo;s important to understand that transforms do not consume <code>PCollection</code>s; instead, they consider each individual element of a <code>PCollection</code> and create a new <code>PCollection</code> as output. This way, you can do different things to different elements in the same <code>PCollection</code>.</p> |
| <h3 id="multiple-transforms-process-the-same-pcollection">Multiple transforms process the same PCollection</h3> |
| <p>You can use the same <code>PCollection</code> as input for multiple transforms without consuming the input or altering it.</p> |
| <p>The pipeline in figure 2 is a branching pipeline. The pipeline reads its input (first names represented as strings) from a database table and creates a <code>PCollection</code> of table rows. Then, the pipeline applies multiple transforms to the <strong>same</strong> <code>PCollection</code>. Transform A extracts all the names in that <code>PCollection</code> that start with the letter &lsquo;A&rsquo;, and Transform B extracts all the names in that <code>PCollection</code> that start with the letter &lsquo;B&rsquo;. Both transforms A and B have the same input <code>PCollection</code>.</p> |
| <p><img src="/images/design-your-pipeline-multiple-pcollections.svg" alt="The pipeline applies two transforms to a single input collection. Eachtransform produces an output collection."></p> |
| <p><em>Figure 2: A branching pipeline. Two transforms are applied to a single |
| PCollection of database table rows.</em></p> |
| <p>The following example code applies two transforms to a single input collection.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">dbRowCollection</span> <span class="o">=</span> <span class="o">...;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">aCollection</span> <span class="o">=</span> <span class="n">dbRowCollection</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;aTrans&#34;</span><span class="o">,</span> <span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;(){</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">startsWith</span><span class="o">(</span><span class="s">&#34;A&#34;</span><span class="o">)){</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">c</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">bCollection</span> <span class="o">=</span> <span class="n">dbRowCollection</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;bTrans&#34;</span><span class="o">,</span> <span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;(){</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">startsWith</span><span class="o">(</span><span class="s">&#34;B&#34;</span><span class="o">)){</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">c</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="a-single-transform-that-produces-multiple-outputs">A single transform that produces multiple outputs</h3> |
| <p>Another way to branch a pipeline is to have a <strong>single</strong> transform output to multiple <code>PCollection</code>s by using <a href="/documentation/programming-guide/#additional-outputs">tagged outputs</a>. Transforms that produce more than one output process each element of the input once, and output to zero or more <code>PCollection</code>s.</p> |
| <p>Figure 3 illustrates the same example described above, but with one transform that produces multiple outputs. Names that start with &lsquo;A&rsquo; are added to the main output <code>PCollection</code>, and names that start with &lsquo;B&rsquo; are added to an additional output <code>PCollection</code>.</p> |
| <p><img src="/images/design-your-pipeline-additional-outputs.svg" alt="The pipeline applies one transform that produces multiple output collections."></p> |
| <p><em>Figure 3: A pipeline with a transform that outputs multiple PCollections.</em></p> |
| <p>If we compare the pipelines in figure 2 and figure 3, you can see they perform |
| the same operation in different ways. The pipeline in figure 2 contains two |
| transforms that process the elements in the same input <code>PCollection</code>. One |
| transform uses the following logic:</p> |
| <pre>if (starts with 'A') { outputToPCollectionA }</pre> |
| <p>while the other transform uses:</p> |
| <pre>if (starts with 'B') { outputToPCollectionB }</pre> |
| <p>Because each transform reads the entire input <code>PCollection</code>, each element in the input <code>PCollection</code> is processed twice.</p> |
| <p>The pipeline in figure 3 performs the same operation in a different way - with only one transform that uses the following logic:</p> |
| <pre>if (starts with 'A') { outputToPCollectionA } else if (starts with 'B') { outputToPCollectionB }</pre> |
| <p>where each element in the input <code>PCollection</code> is processed once.</p> |
| <p>The following example code applies one transform that processes each element |
| once and outputs two collections.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Define two TupleTags, one for each output. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="kd">final</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">startsWithATag</span> <span class="o">=</span> <span class="k">new</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;(){};</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">final</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">startsWithBTag</span> <span class="o">=</span> <span class="k">new</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;(){};</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollectionTuple</span> <span class="n">mixedCollection</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">dbRowCollection</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">startsWith</span><span class="o">(</span><span class="s">&#34;A&#34;</span><span class="o">))</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Emit to main output, which is the output with tag startsWithATag. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">c</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> <span class="k">else</span> <span class="k">if</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">startsWith</span><span class="o">(</span><span class="s">&#34;B&#34;</span><span class="o">))</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Emit to output with tag startsWithBTag. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">c</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">startsWithBTag</span><span class="o">,</span> <span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">})</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Specify main output. In this example, it is the output |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// with tag startsWithATag. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">withOutputTags</span><span class="o">(</span><span class="n">startsWithATag</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Specify the output with tag startsWithBTag, as a TupleTagList. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">TupleTagList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">startsWithBTag</span><span class="o">)));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Get subset of the output with tag startsWithATag. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">mixedCollection</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">startsWithATag</span><span class="o">).</span><span class="na">apply</span><span class="o">(...);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Get subset of the output with tag startsWithBTag. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">mixedCollection</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">startsWithBTag</span><span class="o">).</span><span class="na">apply</span><span class="o">(...);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>You can use either mechanism to produce multiple output <code>PCollection</code>s. The first option is recommended if it logically does not make sense to combine the processing logic into one <code>ParDo</code>. However, using the second option (a single transform that produces multiple outputs) makes more sense if the transform&rsquo;s computation per element is time-consuming, and is more scalable if you plan to add more output types in the future.</p> |
| <h2 id="merging-pcollections">Merging PCollections</h2> |
| <p>Often, after you&rsquo;ve branched your <code>PCollection</code> into multiple <code>PCollection</code>s via multiple transforms, you&rsquo;ll want to merge some or all of those resulting <code>PCollection</code>s back together. You can do so by using one of the following:</p> |
| <ul> |
| <li><strong>Flatten</strong> - You can use the <code>Flatten</code> transform in the Beam SDKs to merge multiple <code>PCollection</code>s of the <strong>same type</strong>.</li> |
| <li><strong>Join</strong> - You can use the <code>CoGroupByKey</code> transform in the Beam SDK to perform a relational join between two <code>PCollection</code>s. The <code>PCollection</code>s must be keyed (i.e. they must be collections of key/value pairs) and they must use the same key type.</li> |
| </ul> |
| <p>The example in figure 4 is a continuation of the example in figure 2 in <a href="#multiple-transforms-process-the-same-pcollection">the |
| section above</a>. After |
| branching into two <code>PCollection</code>s, one with names that begin with &lsquo;A&rsquo; and one |
| with names that begin with &lsquo;B&rsquo;, the pipeline merges the two together into a |
| single <code>PCollection</code> that now contains all names that begin with either &lsquo;A&rsquo; or |
| &lsquo;B&rsquo;. Here, it makes sense to use <code>Flatten</code> because the <code>PCollection</code>s being |
| merged both contain the same type.</p> |
| <p><img src="/images/design-your-pipeline-flatten.svg" alt="The pipeline merges two collections into one collection with the Flatten transform."></p> |
| <p><em>Figure 4: A pipeline that merges two collections into one collection with the Flatten transform.</em></p> |
| <p>The following example code applies <code>Flatten</code> to merge two collections.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">//merge the two PCollections with Flatten |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollectionList</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">collectionList</span> <span class="o">=</span> <span class="n">PCollectionList</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">aCollection</span><span class="o">).</span><span class="na">and</span><span class="o">(</span><span class="n">bCollection</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">mergedCollectionWithFlatten</span> <span class="o">=</span> <span class="n">collectionList</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Flatten</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">pCollections</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// continue with the new merged PCollection |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">mergedCollectionWithFlatten</span><span class="o">.</span><span class="na">apply</span><span class="o">(...);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h2 id="multiple-sources">Multiple sources</h2> |
| <p>Your pipeline can read its input from one or more sources. If your pipeline reads from multiple sources and the data from those sources is related, it can be useful to join the inputs together. In the example illustrated in figure 5 below, the pipeline reads names and addresses from a database table, and names and order numbers from a Kafka topic. The pipeline then uses <code>CoGroupByKey</code> to join this information, where the key is the name; the resulting <code>PCollection</code> contains all the combinations of names, addresses, and orders.</p> |
| <p><img src="/images/design-your-pipeline-join.svg" alt="The pipeline joins two input collections into one collection with the Join transform."></p> |
| <p><em>Figure 5: A pipeline that does a relational join of two input collections.</em></p> |
| <p>The following example code applies <code>Join</code> to join two input collections.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;&gt;</span> <span class="n">userAddress</span> <span class="o">=</span> <span class="n">pipeline</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">JdbcIO</span><span class="o">.&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;&gt;</span><span class="n">read</span><span class="o">()...);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;&gt;</span> <span class="n">userOrder</span> <span class="o">=</span> <span class="n">pipeline</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">KafkaIO</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span><span class="n">read</span><span class="o">()...);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">final</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">addressTag</span> <span class="o">=</span> <span class="k">new</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;();</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">final</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">orderTag</span> <span class="o">=</span> <span class="k">new</span> <span class="n">TupleTag</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;();</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Merge collection values into a CoGbkResult collection. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">CoGbkResult</span><span class="o">&gt;&gt;</span> <span class="n">joinedCollection</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KeyedPCollectionTuple</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">addressTag</span><span class="o">,</span> <span class="n">userAddress</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">and</span><span class="o">(</span><span class="n">orderTag</span><span class="o">,</span> <span class="n">userOrder</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">CoGroupByKey</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">create</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">joinedCollection</span><span class="o">.</span><span class="na">apply</span><span class="o">(...);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h2 id="whats-next">What&rsquo;s next</h2> |
| <ul> |
| <li><a href="/documentation/pipelines/create-your-pipeline">Create your own pipeline</a>.</li> |
| <li><a href="/documentation/pipelines/test-your-pipeline">Test your pipeline</a>.</li> |
| </ul></description></item><item><title>Documentation: Distinct</title><link>/documentation/transforms/java/aggregation/distinct/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/java/aggregation/distinct/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="distinct">Distinct</h1> |
| <table align="left"> |
| <a target="_blank" class="button" |
| href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Distinct.html"> |
| <img src="/images/logos/sdks/java.png" width="20px" height="20px" |
| alt="Javadoc" /> |
| Javadoc |
| </a> |
| </table> |
| <br><br> |
| <p>Produces a collection containing distinct elements of the input collection.</p> |
| <p>On some data sets, it might be more efficient to compute an approximate |
| answer using <code>ApproximateUnique</code>, which also allows for determining distinct |
| values for each key.</p> |
| <h2 id="examples">Examples</h2> |
| <p><strong>Example 1</strong>: Find the distinct element from a <code>PCollection</code> of <code>String</code>.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">static</span> <span class="kd">final</span> <span class="n">String</span><span class="o">[]</span> <span class="n">WORDS_ARRAY</span> <span class="o">=</span> <span class="k">new</span> <span class="n">String</span><span class="o">[]{</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;hi&#34;</span><span class="o">,</span> <span class="s">&#34;hi&#34;</span><span class="o">,</span> <span class="s">&#34;sue&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;sue&#34;</span><span class="o">,</span> <span class="s">&#34;bob&#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">};</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">static</span> <span class="kd">final</span> <span class="n">List</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">WORDS</span> <span class="o">=</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="n">WORDS_ARRAY</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">input</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pipeline</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Create</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">WORDS</span><span class="o">)).</span><span class="na">withCoder</span><span class="o">(</span><span class="n">StringUtf8Coder</span><span class="o">.</span><span class="na">of</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">distinctWords</span> <span class="o">=</span> <span class="n">input</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Distinct</span><span class="o">.</span><span class="na">create</span><span class="o">());</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p><strong>Example 2</strong>: Find the distinct element from a <code>PCollection</code> of <code>Integer</code>.</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-java playground-snippet" |
| data-sdk="java" |
| data-path="SDK_JAVA_Distinct" |
| data-show="main_section" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_JAVA_Distinct%22%2c%22sdk%22%3a%22java%22%2c%22show%22%3a%22main_section%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2> |
| <ul> |
| <li><a href="/documentation/transforms/java/aggregation/count">Count</a> |
| counts the number of elements within each aggregation.</li> |
| <li><a href="/documentation/transforms/java/aggregation/approximateunique">ApproximateUnique</a> |
| estimates the number of distinct elements in a collection.</li> |
| </ul></description></item><item><title>Documentation: Distinct</title><link>/documentation/transforms/python/aggregation/distinct/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/python/aggregation/distinct/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="distinct">Distinct</h1> |
| <script type="text/javascript"> |
| localStorage.setItem("language", "language-py") |
| </script> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Distinct" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p> |
| <p>Produces a collection containing distinct elements of the input collection.</p> |
| <h2 id="examples">Examples</h2> |
| <p>In the following example, we create a pipeline with two <code>PCollection</code>s of produce.</p> |
| <p>We use <code>Distinct</code> to get rid of duplicate elements, which outputs a <code>PCollection</code> of all the unique elements.</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_Distinct" |
| data-show="distinct" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_PYTHON_Distinct%22%2c%22sdk%22%3a%22python%22%2c%22show%22%3a%22distinct%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2> |
| <ul> |
| <li><a href="/documentation/transforms/python/aggregation/count">Count</a> counts the number of elements within each aggregation.</li> |
| </ul> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html#apache_beam.transforms.util.Distinct" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p></description></item><item><title>Documentation: Enrichment</title><link>/documentation/transforms/python/elementwise/enrichment/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/python/elementwise/enrichment/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="enrichment-transform">Enrichment transform</h1> |
| <script type="text/javascript"> |
| localStorage.setItem("language", "language-py") |
| </script> |
| <table> |
| <tr> |
| <td> |
| <a> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment.html#apache_beam.transforms.enrichment.Enrichment" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p> |
| </a> |
| </td> |
| </tr> |
| </table> |
| <p>The enrichment transform lets you dynamically enrich data in a pipeline by doing a key-value lookup to a remote service. The transform uses <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.io.requestresponseio.html#apache_beam.io.requestresponseio.RequestResponseIO"><code>RequestResponeIO</code></a> internally. This feature uses client-side throttling to ensure that the remote service isn&rsquo;t overloaded with requests. If service-side errors occur, like <code>TooManyRequests</code> and <code>Timeout</code> exceptions, it retries the requests by using exponential backoff.</p> |
| <p>This transform is available in Apache Beam 2.54.0 and later versions.</p> |
| <h2 id="examples">Examples</h2> |
| <p>The following examples demonstrate how to create a pipeline that use the enrichment transform to enrich data from external services.</p> |
| <div class="table-wrapper"><table> |
| <thead> |
| <tr> |
| <th style="text-align:left">Service</th> |
| <th style="text-align:left">Example</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td style="text-align:left">Cloud Bigtable</td> |
| <td style="text-align:left"><a href="/documentation/transforms/python/elementwise/enrichment-bigtable/#example">Enrichment with Bigtable</a></td> |
| </tr> |
| <tr> |
| <td style="text-align:left">Vertex AI Feature Store</td> |
| <td style="text-align:left"><a href="/documentation/transforms/python/elementwise/enrichment-vertexai/#example-1-enrichment-with-vertex-ai-feature-store">Enrichment with Vertex AI Feature Store</a></td> |
| </tr> |
| <tr> |
| <td style="text-align:left">Vertex AI Feature Store (Legacy)</td> |
| <td style="text-align:left"><a href="/documentation/transforms/python/elementwise/enrichment-vertexai/#example-2-enrichment-with-vertex-ai-feature-store-legacy">Enrichment with Legacy Vertex AI Feature Store</a></td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2> |
| <p>Not applicable.</p> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment.html#apache_beam.transforms.enrichment.Enrichment" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p></description></item><item><title>Documentation: Enrichment with Bigtable</title><link>/documentation/transforms/python/elementwise/enrichment-bigtable/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/python/elementwise/enrichment-bigtable/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="use-bigtable-to-enrich-data">Use Bigtable to enrich data</h1> |
| <script type="text/javascript"> |
| localStorage.setItem("language", "language-py") |
| </script> |
| <table> |
| <tr> |
| <td> |
| <a> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#apache_beam.transforms.enrichment_handlers.bigtable.BigTableEnrichmentHandler" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p> |
| </a> |
| </td> |
| </tr> |
| </table> |
| <p>In Apache Beam 2.54.0 and later versions, the enrichment transform includes a built-in enrichment handler for <a href="https://cloud.google.com/bigtable/docs/overview">Bigtable</a>. |
| The following example demonstrates how to create a pipeline that use the enrichment transform with the <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#apache_beam.transforms.enrichment_handlers.bigtable.BigTableEnrichmentHandler"><code>BigTableEnrichmentHandler</code></a> handler.</p> |
| <p>The data stored in the Bigtable cluster uses the following format:</p> |
| <div class="table-wrapper"><table> |
| <thead> |
| <tr> |
| <th style="text-align:center">Row key</th> |
| <th style="text-align:center">product:product_id</th> |
| <th style="text-align:center">product:product_name</th> |
| <th style="text-align:center">product:product_stock</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td style="text-align:center">1</td> |
| <td style="text-align:center">1</td> |
| <td style="text-align:center">pixel 5</td> |
| <td style="text-align:center">2</td> |
| </tr> |
| <tr> |
| <td style="text-align:center">2</td> |
| <td style="text-align:center">2</td> |
| <td style="text-align:center">pixel 6</td> |
| <td style="text-align:center">4</td> |
| </tr> |
| <tr> |
| <td style="text-align:center">3</td> |
| <td style="text-align:center">3</td> |
| <td style="text-align:center">pixel 7</td> |
| <td style="text-align:center">20</td> |
| </tr> |
| <tr> |
| <td style="text-align:center">4</td> |
| <td style="text-align:center">4</td> |
| <td style="text-align:center">pixel 8</td> |
| <td style="text-align:center">10</td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="k">as</span> <span class="nn">beam</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.transforms.enrichment</span> <span class="kn">import</span> <span class="n">Enrichment</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.transforms.enrichment_handlers.bigtable</span> <span class="kn">import</span> <span class="n">BigTableEnrichmentHandler</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">project_id</span> <span class="o">=</span> <span class="s1">&#39;apache-beam-testing&#39;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">instance_id</span> <span class="o">=</span> <span class="s1">&#39;beam-test&#39;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">table_id</span> <span class="o">=</span> <span class="s1">&#39;bigtable-enrichment-test&#39;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">row_key</span> <span class="o">=</span> <span class="s1">&#39;product_id&#39;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">data</span> <span class="o">=</span> <span class="p">[</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">Row</span><span class="p">(</span><span class="n">sale_id</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">customer_id</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">product_id</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">quantity</span><span class="o">=</span><span class="mi">1</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">Row</span><span class="p">(</span><span class="n">sale_id</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">customer_id</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span> <span class="n">product_id</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">quantity</span><span class="o">=</span><span class="mi">3</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">Row</span><span class="p">(</span><span class="n">sale_id</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">customer_id</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">product_id</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">quantity</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">bigtable_handler</span> <span class="o">=</span> <span class="n">BigTableEnrichmentHandler</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">project_id</span><span class="o">=</span><span class="n">project_id</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">instance_id</span><span class="o">=</span><span class="n">instance_id</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">table_id</span><span class="o">=</span><span class="n">table_id</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">row_key</span><span class="o">=</span><span class="n">row_key</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">p</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">_</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">p</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s2">&#34;Create&#34;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s2">&#34;Enrich W/ BigTable&#34;</span> <span class="o">&gt;&gt;</span> <span class="n">Enrichment</span><span class="p">(</span><span class="n">bigtable_handler</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s2">&#34;Print&#34;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="nb">print</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p> |
| <p class="notebook-skip">Output:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>Row(sale_id=1, customer_id=1, product_id=1, quantity=1, product={&#39;product_id&#39;: &#39;1&#39;, &#39;product_name&#39;: &#39;pixel 5&#39;, &#39;product_stock&#39;: &#39;2&#39;}) |
| Row(sale_id=3, customer_id=3, product_id=2, quantity=3, product={&#39;product_id&#39;: &#39;2&#39;, &#39;product_name&#39;: &#39;pixel 6&#39;, &#39;product_stock&#39;: &#39;4&#39;}) |
| Row(sale_id=5, customer_id=5, product_id=4, quantity=2, product={&#39;product_id&#39;: &#39;4&#39;, &#39;product_name&#39;: &#39;pixel 8&#39;, &#39;product_stock&#39;: &#39;10&#39;})</code></pre> |
| </div> |
| </div> |
| </p> |
| <h2 id="related-transforms">Related transforms</h2> |
| <p>Not applicable.</p> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.bigtable.html#apache_beam.transforms.enrichment_handlers.bigtable.BigTableEnrichmentHandler" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p></description></item><item><title>Documentation: Enrichment with Vertex AI Feature Store</title><link>/documentation/transforms/python/elementwise/enrichment-vertexai/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/python/elementwise/enrichment-vertexai/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="enrichment-with-google-cloud-vertex-ai-feature-store">Enrichment with Google Cloud Vertex AI Feature Store</h1> |
| <script type="text/javascript"> |
| localStorage.setItem("language", "language-py") |
| </script> |
| <table> |
| <tr> |
| <td> |
| <a> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.vertex_ai_feature_store.html#apache_beam.transforms.enrichment_handlers.vertex_ai_feature_store.VertexAIFeatureStoreEnrichmentHandler" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p> |
| </a> |
| </td> |
| </tr> |
| </table> |
| <p>In Apache Beam 2.55.0 and later versions, the enrichment transform includes a built-in enrichment handler for <a href="https://cloud.google.com/vertex-ai/docs/featurestore">Vertex AI Feature Store</a>. |
| The following example demonstrates how to create a pipeline that use the enrichment transform with the <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.vertex_ai_feature_store.html#apache_beam.transforms.enrichment_handlers.vertex_ai_feature_store.VertexAIFeatureStoreEnrichmentHandler"><code>VertexAIFeatureStoreEnrichmentHandler</code></a> handler and the <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.vertex_ai_feature_store.html#apache_beam.transforms.enrichment_handlers.vertex_ai_feature_store.VertexAIFeatureStoreLegacyEnrichmentHandler"><code>VertexAIFeatureStoreLegacyEnrichmentHandler</code></a> handler.</p> |
| <h2 id="example-1-enrichment-with-vertex-ai-feature-store">Example 1: Enrichment with Vertex AI Feature Store</h2> |
| <p>The precomputed feature values stored in Vertex AI Feature Store uses the following format:</p> |
| <div class="table-wrapper"><table> |
| <thead> |
| <tr> |
| <th style="text-align:center">user_id</th> |
| <th style="text-align:center">age</th> |
| <th style="text-align:center">gender</th> |
| <th style="text-align:center">state</th> |
| <th style="text-align:center">country</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td style="text-align:center">21422</td> |
| <td style="text-align:center">12</td> |
| <td style="text-align:center">0</td> |
| <td style="text-align:center">0</td> |
| <td style="text-align:center">0</td> |
| </tr> |
| <tr> |
| <td style="text-align:center">2963</td> |
| <td style="text-align:center">12</td> |
| <td style="text-align:center">1</td> |
| <td style="text-align:center">1</td> |
| <td style="text-align:center">1</td> |
| </tr> |
| <tr> |
| <td style="text-align:center">20592</td> |
| <td style="text-align:center">12</td> |
| <td style="text-align:center">1</td> |
| <td style="text-align:center">2</td> |
| <td style="text-align:center">2</td> |
| </tr> |
| <tr> |
| <td style="text-align:center">76538</td> |
| <td style="text-align:center">12</td> |
| <td style="text-align:center">1</td> |
| <td style="text-align:center">3</td> |
| <td style="text-align:center">0</td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="k">as</span> <span class="nn">beam</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.transforms.enrichment</span> <span class="kn">import</span> <span class="n">Enrichment</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.transforms.enrichment_handlers.vertex_ai_feature_store</span> \ |
| </span></span><span class="line"><span class="cl"> <span class="kn">import</span> <span class="nn">VertexAIFeatureStoreEnrichmentHandler</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">project_id</span> <span class="o">=</span> <span class="s1">&#39;apache-beam-testing&#39;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">location</span> <span class="o">=</span> <span class="s1">&#39;us-central1&#39;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">api_endpoint</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">location</span><span class="si">}</span><span class="s2">-aiplatform.googleapis.com&#34;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">data</span> <span class="o">=</span> <span class="p">[</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">Row</span><span class="p">(</span><span class="n">user_id</span><span class="o">=</span><span class="s1">&#39;2963&#39;</span><span class="p">,</span> <span class="n">product_id</span><span class="o">=</span><span class="mi">14235</span><span class="p">,</span> <span class="n">sale_price</span><span class="o">=</span><span class="mf">15.0</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">Row</span><span class="p">(</span><span class="n">user_id</span><span class="o">=</span><span class="s1">&#39;21422&#39;</span><span class="p">,</span> <span class="n">product_id</span><span class="o">=</span><span class="mi">11203</span><span class="p">,</span> <span class="n">sale_price</span><span class="o">=</span><span class="mf">12.0</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">Row</span><span class="p">(</span><span class="n">user_id</span><span class="o">=</span><span class="s1">&#39;20592&#39;</span><span class="p">,</span> <span class="n">product_id</span><span class="o">=</span><span class="mi">8579</span><span class="p">,</span> <span class="n">sale_price</span><span class="o">=</span><span class="mf">9.0</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">vertex_ai_handler</span> <span class="o">=</span> <span class="n">VertexAIFeatureStoreEnrichmentHandler</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">project</span><span class="o">=</span><span class="n">project_id</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">location</span><span class="o">=</span><span class="n">location</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">api_endpoint</span><span class="o">=</span><span class="n">api_endpoint</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">feature_store_name</span><span class="o">=</span><span class="s2">&#34;vertexai_enrichment_example&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">feature_view_name</span><span class="o">=</span><span class="s2">&#34;users&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">row_key</span><span class="o">=</span><span class="s2">&#34;user_id&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">p</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">_</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">p</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s2">&#34;Create&#34;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s2">&#34;Enrich W/ Vertex AI&#34;</span> <span class="o">&gt;&gt;</span> <span class="n">Enrichment</span><span class="p">(</span><span class="n">vertex_ai_handler</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s2">&#34;Print&#34;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="nb">print</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p> |
| <p class="notebook-skip">Output:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>Row(user_id=&#39;2963&#39;, product_id=14235, sale_price=15.0, age=12.0, state=&#39;1&#39;, gender=&#39;1&#39;, country=&#39;1&#39;) |
| Row(user_id=&#39;21422&#39;, product_id=11203, sale_price=12.0, age=12.0, state=&#39;0&#39;, gender=&#39;0&#39;, country=&#39;0&#39;) |
| Row(user_id=&#39;20592&#39;, product_id=8579, sale_price=9.0, age=12.0, state=&#39;2&#39;, gender=&#39;1&#39;, country=&#39;2&#39;)</code></pre> |
| </div> |
| </div> |
| </p> |
| <h2 id="example-2-enrichment-with-vertex-ai-feature-store-legacy">Example 2: Enrichment with Vertex AI Feature Store (legacy)</h2> |
| <p>The precomputed feature values stored in Vertex AI Feature Store (Legacy) use the following format:</p> |
| <div class="table-wrapper"><table> |
| <thead> |
| <tr> |
| <th style="text-align:left">entity_id</th> |
| <th style="text-align:left">title</th> |
| <th style="text-align:left">genres</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td style="text-align:left">movie_01</td> |
| <td style="text-align:left">The Shawshank Redemption</td> |
| <td style="text-align:left">Drama</td> |
| </tr> |
| <tr> |
| <td style="text-align:left">movie_02</td> |
| <td style="text-align:left">The Shining</td> |
| <td style="text-align:left">Horror</td> |
| </tr> |
| <tr> |
| <td style="text-align:left">movie_04</td> |
| <td style="text-align:left">The Dark Knight</td> |
| <td style="text-align:left">Action</td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="k">as</span> <span class="nn">beam</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.transforms.enrichment</span> <span class="kn">import</span> <span class="n">Enrichment</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.transforms.enrichment_handlers.vertex_ai_feature_store</span> \ |
| </span></span><span class="line"><span class="cl"> <span class="kn">import</span> <span class="nn">VertexAIFeatureStoreLegacyEnrichmentHandler</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">project_id</span> <span class="o">=</span> <span class="s1">&#39;apache-beam-testing&#39;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">location</span> <span class="o">=</span> <span class="s1">&#39;us-central1&#39;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">api_endpoint</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">location</span><span class="si">}</span><span class="s2">-aiplatform.googleapis.com&#34;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">data</span> <span class="o">=</span> <span class="p">[</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">Row</span><span class="p">(</span><span class="n">entity_id</span><span class="o">=</span><span class="s2">&#34;movie_01&#34;</span><span class="p">,</span> <span class="n">title</span><span class="o">=</span><span class="s1">&#39;The Shawshank Redemption&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">Row</span><span class="p">(</span><span class="n">entity_id</span><span class="o">=</span><span class="s2">&#34;movie_02&#34;</span><span class="p">,</span> <span class="n">title</span><span class="o">=</span><span class="s2">&#34;The Shining&#34;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">Row</span><span class="p">(</span><span class="n">entity_id</span><span class="o">=</span><span class="s2">&#34;movie_04&#34;</span><span class="p">,</span> <span class="n">title</span><span class="o">=</span><span class="s1">&#39;The Dark Knight&#39;</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">vertex_ai_handler</span> <span class="o">=</span> <span class="n">VertexAIFeatureStoreLegacyEnrichmentHandler</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">project</span><span class="o">=</span><span class="n">project_id</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">location</span><span class="o">=</span><span class="n">location</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">api_endpoint</span><span class="o">=</span><span class="n">api_endpoint</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">entity_type_id</span><span class="o">=</span><span class="s1">&#39;movies&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">feature_store_id</span><span class="o">=</span><span class="s2">&#34;movie_prediction_unique&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">feature_ids</span><span class="o">=</span><span class="p">[</span><span class="s2">&#34;title&#34;</span><span class="p">,</span> <span class="s2">&#34;genres&#34;</span><span class="p">],</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">row_key</span><span class="o">=</span><span class="s2">&#34;entity_id&#34;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">p</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">_</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">p</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s2">&#34;Create&#34;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s2">&#34;Enrich W/ Vertex AI&#34;</span> <span class="o">&gt;&gt;</span> <span class="n">Enrichment</span><span class="p">(</span><span class="n">vertex_ai_handler</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s2">&#34;Print&#34;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="nb">print</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p> |
| <p class="notebook-skip">Output:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>Row(entity_id=&#39;movie_01&#39;, title=&#39;The Shawshank Redemption&#39;, genres=&#39;Drama&#39;) |
| Row(entity_id=&#39;movie_02&#39;, title=&#39;The Shining&#39;, genres=&#39;Horror&#39;) |
| Row(entity_id=&#39;movie_04&#39;, title=&#39;The Dark Knight&#39;, genres=&#39;Action&#39;)</code></pre> |
| </div> |
| </div> |
| </p> |
| <h2 id="related-transforms">Related transforms</h2> |
| <p>Not applicable.</p> |
| <table align="left" style="margin-right:1em"> |
| <td> |
| <a |
| class="button" |
| target="_blank" |
| href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.enrichment_handlers.vertex_ai_feature_store.html#apache_beam.transforms.enrichment_handlers.vertex_ai_feature_store.VertexAIFeatureStoreEnrichmentHandler" |
| ><img |
| src="https://beam.apache.org/images/logos/sdks/python.png" |
| width="32px" |
| height="32px" |
| alt="Pydoc" |
| /> |
| Pydoc</a |
| > |
| </td> |
| </table> |
| <p><br><br><br></p></description></item><item><title>Documentation: Execution model</title><link>/documentation/runtime/model/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/runtime/model/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="execution-model">Execution model</h1> |
| <p>The Beam model allows runners to execute your pipeline in different ways. You |
| may observe various effects as a result of the runner’s choices. This page |
| describes these effects so you can better understand how Beam pipelines execute.</p> |
| <h2 id="processing-of-elements">Processing of elements</h2> |
| <p>The serialization and communication of elements between machines is one of the |
| most expensive operations in a distributed execution of your pipeline. Avoiding |
| this serialization may require re-processing elements after failures or may |
| limit the distribution of output to other machines.</p> |
| <h3 id="serialization-and-communication">Serialization and communication</h3> |
| <p>The runner might serialize elements between machines for communication purposes |
| and for other reasons such as persistence.</p> |
| <p>A runner may decide to transfer elements between transforms in a variety of |
| ways, such as:</p> |
| <ul> |
| <li>Routing elements to a worker for processing as part of a grouping operation. |
| This may involve serializing elements and grouping or sorting them by their |
| key.</li> |
| <li>Redistributing elements between workers to adjust parallelism. This may |
| involve serializing elements and communicating them to other workers.</li> |
| <li>Using the elements in a side input to a <code>ParDo</code>. This may require |
| serializing the elements and broadcasting them to all the workers executing |
| the <code>ParDo</code>.</li> |
| <li>Passing elements between transforms that are running on the same worker. |
| This may allow the runner to avoid serializing elements; instead, the runner |
| can just pass the elements in memory. This is done as part of an |
| optimization that is known as |
| <a href="/documentation/glossary/#fusion">fusion</a>.</li> |
| </ul> |
| <p>Some situations where the runner may serialize and persist elements are:</p> |
| <ol> |
| <li>When used as part of a stateful <code>DoFn</code>, the runner may persist values to some |
| state mechanism.</li> |
| <li>When committing the results of processing, the runner may persist the outputs |
| as a checkpoint.</li> |
| </ol> |
| <h3 id="bundling-and-persistence">Bundling and persistence</h3> |
| <p>Beam pipelines often focus on &ldquo;<a href="https://en.wikipedia.org/wiki/embarrassingly_parallel">embarassingly parallel</a>&rdquo; |
| problems. Because of this, the APIs emphasize processing elements in parallel, |
| which makes it difficult to express actions like &ldquo;assign a sequence number to |
| each element in a PCollection&rdquo;. This is intentional as such algorithms are much |
| more likely to suffer from scalability problems.</p> |
| <p>Processing all elements in parallel also has some drawbacks. Specifically, it |
| makes it impossible to batch any operations, such as writing elements to a sink |
| or checkpointing progress during processing.</p> |
| <p>Instead of processing all elements simultaneously, the elements in a |
| <code>PCollection</code> are processed in <em>bundles</em>. The division of the collection into |
| bundles is arbitrary and selected by the runner. This allows the runner to |
| choose an appropriate middle-ground between persisting results after every |
| element, and having to retry everything if there is a failure. For example, a |
| streaming runner may prefer to process and commit small bundles, and a batch |
| runner may prefer to process larger bundles.</p> |
| <h3 id="data-partitioning-and-inter-stage-execution">Data partitioning and inter-stage execution</h3> |
| <p>Partitioning and parallelization of element processing within a Beam pipeline is |
| dependent on two things:</p> |
| <ul> |
| <li>Data source implementation</li> |
| <li>Inter-stage key parallelism</li> |
| </ul> |
| <p>Beam pipelines read data from a source (e.g. <code>KafkaIO</code>, <code>BigQueryIO</code>, <code>JdbcIO</code>, |
| or your own source implementation). To implement a Source in Beam one must |
| implement it as a Splittable <code>DoFn</code>. A Splittable <code>DoFn</code> provides the runner |
| with interfaces to facilitate the splitting of work.</p> |
| <p>When running key-based operations in Beam (e.g. <code>GroupByKey</code>, <code>Combine</code>, |
| <code>Reshuffle.perKey</code>, and stateful <code>DoFn</code>s), Beam runners perform serialization |
| and transfer of data known as <em>shuffle</em><sup>1</sup>. Shuffle allows data |
| elements of the same key to be processed together.</p> |
| <p>The way in which runners <em>shuffle</em> data may be slightly different for Batch and |
| Streaming execution modes.</p> |
| <p><sup>1</sup>Not to be confused with the <code>shuffle</code> operation in some runners.</p> |
| <h4 id="data-ordering-in-a-pipeline-execution">Data ordering in a pipeline execution</h4> |
| <p>The Beam model does not define strict guidelines regarding the order in which |
| runners process elements or transport them across <code>PTransforms</code>. Runners are |
| free to implement data transfer semantics in different forms.</p> |
| <p>Some use cases exist where user pipelines may need to rely on specific ordering |
| semantics in pipeline execution. The <a href="/documentation/runners/capability-matrix/additional-common-features-not-yet-part-of-the-beam-model/index.html">capability matrix documents</a> |
| runner behavior for <strong>key-ordered delivery</strong>.</p> |
| <p>Consider a single Beam worker processing a series of bundles from the same Beam |
| transform, and consider a <code>PTransform</code> that outputs data from this Stage into a |
| downstream <code>PCollection</code>. Finally, consider two events <em>with the same key</em> |
| emitted in a certain order by this worker (within the same bundle or as part of |
| different bundles).</p> |
| <p>We say that the Beam runner supports <strong>key-ordered delivery</strong> if it guarantees |
| that these two events will be observed in the same order by a PTransform that is |
| immediately downstream independently of the kind of data transmission method.</p> |
| <p>This characteristic will hold true in runners and operations that have |
| key-limited parallelism.</p> |
| <h2 id="parallelism">Failures and parallelism within and between transforms</h2> |
| <p>In this section, we discuss how elements in the input collection are processed |
| in parallel, and how transforms are retried when failures occur.</p> |
| <h3 id="data-parallelism">Data-parallelism within one transform</h3> |
| <p>When executing a single <code>ParDo</code>, a runner might divide an example input |
| collection of nine elements into two bundles as shown in figure 1.</p> |
| <p><img src="/images/execution_model_bundling.svg" alt="Bundle A contains five elements. Bundle B contains four elements."></p> |
| <p><em>Figure 1: A runner divides an input collection into two bundles.</em></p> |
| <p>When the <code>ParDo</code> executes, workers may process the two bundles in parallel as |
| shown in figure 2.</p> |
| <p><img src="/images/execution_model_bundling_gantt.svg" alt="Two workers process the two bundles in parallel. Worker one processes bundle A. Worker two processes bundle B."></p> |
| <p><em>Figure 2: Two workers process the two bundles in parallel.</em></p> |
| <p>Since elements cannot be split, the maximum parallelism for a transform depends |
| on the number of elements in the collection. In figure 3, the input collection |
| has nine elements, so the maximum parallelism is nine.</p> |
| <p><img src="/images/execution_model_bundling_gantt_max.svg" alt="Nine workers process a nine element input collection in parallel."></p> |
| <p><em>Figure 3: Nine workers process a nine element input collection in parallel.</em></p> |
| <p>Note: Splittable ParDo allows splitting the processing of a single input across |
| multiple bundles. This feature is a work in progress.</p> |
| <h3 id="dependent-parallellism">Dependent-parallelism between transforms</h3> |
| <p><code>ParDo</code> transforms that are in sequence may be <em>dependently parallel</em> if the |
| runner chooses to execute the consuming transform on the producing transform&rsquo;s |
| output elements without altering the bundling. In figure 4, <code>ParDo1</code> and |
| <code>ParDo2</code> are <em>dependently parallel</em> if the output of <code>ParDo1</code> for a given |
| element must be processed on the same worker.</p> |
| <p><img src="/images/execution_model_bundling_multi.svg" alt="ParDo1 processes an input collection that contains bundles A and B. ParDo2 then processes the output collection from ParDo1, which contains bundles C and D."></p> |
| <p><em>Figure 4: Two transforms in sequence and their corresponding input collections.</em></p> |
| <p>Figure 5 shows how these dependently parallel transforms might execute. The |
| first worker executes <code>ParDo1</code> on the elements in bundle A (which results in |
| bundle C), and then executes <code>ParDo2</code> on the elements in bundle C. Similarly, |
| the second worker executes <code>ParDo1</code> on the elements in bundle B (which results |
| in bundle D), and then executes <code>ParDo2</code> on the elements in bundle D.</p> |
| <p><img src="/images/execution_model_bundling_multi_gantt.svg" alt="Worker one executes ParDo1 on bundle A and Pardo2 on bundle C. Worker two executes ParDo1 on bundle B and ParDo2 on bundle D."></p> |
| <p><em>Figure 5: Two workers execute dependently parallel ParDo transforms.</em></p> |
| <p>Executing transforms this way allows a runner to avoid redistributing elements |
| between workers, which saves on communication costs. However, the maximum parallelism |
| now depends on the maximum parallelism of the first of the dependently parallel |
| steps.</p> |
| <h3 id="failures-within-one-transform">Failures within one transform</h3> |
| <p>If processing of an element within a bundle fails, the entire bundle fails. The |
| elements in the bundle must be retried (otherwise the entire pipeline fails), |
| although they do not need to be retried with the same bundling.</p> |
| <p>For this example, we will use the <code>ParDo</code> from figure 1 that has an input |
| collection with nine elements and is divided into two bundles.</p> |
| <p>In figure 6, the first worker successfully processes all five elements in bundle |
| A. The second worker processes the four elements in bundle B: the first two |
| elements were successfully processed, the third element’s processing failed, and |
| there is one element still awaiting processing.</p> |
| <p>We see that the runner retries all elements in bundle B and the processing |
| completes successfully the second time. Note that the retry does not necessarily |
| happen on the same worker as the original processing attempt, as shown in the |
| figure.</p> |
| <p><img src="/images/execution_model_failure_retry.svg" alt="Worker two fails to process an element in bundle B. Worker one finishes processing bundle A and then successfully retries to execute bundle B."></p> |
| <p><em>Figure 6: The processing of an element within bundle B fails, and another worker |
| retries the entire bundle.</em></p> |
| <p>Because we encountered a failure while processing an element in the input |
| bundle, we had to reprocess <em>all</em> of the elements in the input bundle. This means |
| the runner must throw away the entire output of the bundle (including any state |
| mutations and set timers) since all of the results it contains will be recomputed.</p> |
| <p>Note that if the failed transform is a <code>ParDo</code>, then the <code>DoFn</code> instance is torn |
| down and abandoned.</p> |
| <h3 id="coupled-failure">Coupled failure: Failures between transforms</h3> |
| <p>If a failure to process an element in <code>ParDo2</code> causes <code>ParDo1</code> to re-execute, |
| these two steps are said to be <em>co-failing</em>.</p> |
| <p>For this example, we will use the two <code>ParDo</code>s from figure 4.</p> |
| <p>In figure 7, worker two successfully executes <code>ParDo1</code> on all elements in bundle |
| B. However, the worker fails to process an element in bundle D, so <code>ParDo2</code> |
| fails (shown as the red X). As a result, the runner must discard and recompute |
| the output of <code>ParDo2</code>. Because the runner was executing <code>ParDo1</code> and <code>ParDo2</code> |
| together, the output bundle from <code>ParDo1</code> must also be thrown away, and all |
| elements in the input bundle must be retried. These two <code>ParDo</code>s are co-failing.</p> |
| <p><img src="/images/execution_model_bundling_coupled_failure.svg" alt="Worker two fails to process en element in bundle D, so all elements in both bundle B and bundle D must be retried."></p> |
| <p><em>Figure 7: Processing of an element within bundle D fails, so all elements in |
| the input bundle are retried.</em></p> |
| <p>Note that the retry does not necessarily have the same processing time as the |
| original attempt, as shown in the diagram.</p> |
| <p>All <code>DoFns</code> that experience coupled failures are terminated and must be torn |
| down since they aren’t following the normal <code>DoFn</code> lifecycle .</p> |
| <p>Executing transforms this way allows a runner to avoid persisting elements |
| between transforms, saving on persistence costs.</p></description></item><item><title>Documentation: File processing patterns</title><link>/documentation/patterns/file-processing/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/patterns/file-processing/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="file-processing-patterns">File processing patterns</h1> |
| <p>This page describes common file processing tasks. For more information on file-based I/O, see <a href="/documentation/programming-guide/#pipeline-io">Pipeline I/O</a> and <a href="/documentation/programming-guide/#file-based-data">File-based input and output data</a>.</p> |
| <nav class="language-switcher"> |
| <strong>Adapt for:</strong> |
| <ul> |
| <li data-value="java" class="active">Java SDK</li> |
| <li data-value="py">Python SDK</li> |
| </ul> |
| </nav> |
| <h2 id="processing-files-as-they-arrive">Processing files as they arrive</h2> |
| <p>This section shows you how to process files as they arrive in your file system or object store (like Google Cloud Storage). You can continuously read files or trigger stream and processing pipelines when a file arrives.</p> |
| <h3 id="continuous-read-mode">Continuous read mode</h3> |
| <p class="language-java">You can use <code>FileIO</code> or <code>TextIO</code> to continuously read the source for new files.</p> |
| <p class="language-java">Use the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileIO.html"><code>FileIO</code></a> class to continuously watch a single file pattern. The following example matches a file pattern repeatedly every 30 seconds, continuously returns new matched files as an unbounded <code>PCollection&lt;Metadata&gt;</code>, and stops if no new files appear for one hour:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// This produces PCollection&lt;MatchResult.Metadata&gt; |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">FileIO</span><span class="o">.</span><span class="na">match</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">filepattern</span><span class="o">(</span><span class="s">&#34;...&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">continuously</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Duration</span><span class="o">.</span><span class="na">standardSeconds</span><span class="o">(</span><span class="n">30</span><span class="o">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Watch</span><span class="o">.</span><span class="na">Growth</span><span class="o">.</span><span class="na">afterTimeSinceNewOutput</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardHours</span><span class="o">(</span><span class="n">1</span><span class="o">))));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">The <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html"><code>TextIO</code></a> class <code>watchForNewFiles</code> property streams new file matches.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// This produces PCollection&lt;String&gt; |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TextIO</span><span class="o">.</span><span class="na">read</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">from</span><span class="o">(</span><span class="s">&#34;&lt;path-to-files&gt;/*&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">watchForNewFiles</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Check for new files every minute. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="n">1</span><span class="o">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Stop watching the file pattern if no new files appear for an hour. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">Watch</span><span class="o">.</span><span class="na">Growth</span><span class="o">.</span><span class="na">afterTimeSinceNewOutput</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardHours</span><span class="o">(</span><span class="n">1</span><span class="o">))));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">Some runners may retain file lists during updates, but file lists don’t persist when you restart a pipeline. You can save file lists by:</p> |
| <p class="language-java"><ul> |
| <li>Storing processed filenames in an external file and deduplicating the lists at the next transform</li> |
| <li>Adding timestamps to filenames, writing a glob pattern to pull in only new files, and matching the pattern when the pipeline restarts</li> |
| </ul> |
| </p> |
| <p class="language-py">The continuous-read option is not available for Python.</p> |
| <h3 id="stream-processing-triggered-from-external-source">Stream processing triggered from external source</h3> |
| <p>A streaming pipeline can process data from an unbounded source. For example, to trigger stream processing with Google Cloud Pub/Sub:</p> |
| <ol> |
| <li>Use an external process to detect when new files arrive.</li> |
| <li>Send a Google Cloud Pub/Sub message with a URI to the file.</li> |
| <li>Access the URI from a <code>DoFn</code> that follows the Google Cloud Pub/Sub source.</li> |
| <li>Process the file.</li> |
| </ol> |
| <h3 id="batch-processing-triggered-from-external-source">Batch processing triggered from external source</h3> |
| <p>To start or schedule a batch pipeline job when a file arrives, write the triggering event in the source file itself. This has the most latency because the pipeline must initialize before processing. It’s best suited for low-frequency, large, file-size updates.</p> |
| <h2 id="accessing-filenames">Accessing filenames</h2> |
| <p class="language-java">Use the <code>FileIO</code> class to read filenames in a pipeline job. <code>FileIO</code> returns a <code>PCollection&lt;ReadableFile&gt;</code> object, and the <code>ReadableFile</code> instance contains the filename.</p> |
| <p class="language-java">To access filenames:</p> |
| <p class="language-java"><ol> |
| <li>Create a <code>ReadableFile</code> instance with <code>FileIO</code>. <code>FileIO</code> returns a <code>PCollection&lt;ReadableFile&gt;</code> object. The <code>ReadableFile</code> class contains the filename.</li> |
| <li>Call the <code>readFullyAsUTF8String()</code> method to read the file into memory and return the filename as a <code>String</code> object. If memory is limited, you can use utility classes like <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileSystems.html"><code>FileSystems</code></a> to work directly with the file.</li> |
| </ol> |
| </p> |
| <p class="language-py">To read filenames in a pipeline job:</p> |
| <p class="language-py"><ol> |
| <li>Collect the list of file URIs. You can use the <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.io.filesystems.html?highlight=filesystems#module-apache_beam.io.filesystems"><code>FileSystems</code></a> module to get a list of files that match a glob pattern.</li> |
| <li>Pass the file URIs to a <code>PCollection</code>.</li> |
| </ol> |
| </p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">FileIO</span><span class="o">.</span><span class="na">match</span><span class="o">().</span><span class="na">filepattern</span><span class="o">(</span><span class="s">&#34;hdfs://path/to/*.gz&#34;</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// The withCompression method is optional. By default, the Beam SDK detects compression from |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// the filename. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">FileIO</span><span class="o">.</span><span class="na">readMatches</span><span class="o">().</span><span class="na">withCompression</span><span class="o">(</span><span class="n">Compression</span><span class="o">.</span><span class="na">GZIP</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">FileIO</span><span class="o">.</span><span class="na">ReadableFile</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">process</span><span class="o">(</span><span class="nd">@Element</span> <span class="n">FileIO</span><span class="o">.</span><span class="na">ReadableFile</span> <span class="n">file</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// We can now access the file and its metadata. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">LOG</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">&#34;File Metadata resourceId is {} &#34;</span><span class="o">,</span> <span class="n">file</span><span class="o">.</span><span class="na">getMetadata</span><span class="o">().</span><span class="na">resourceId</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">readable_files</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">fileio</span><span class="o">.</span><span class="n">MatchFiles</span><span class="p">(</span><span class="s1">&#39;hdfs://path/to/*.txt&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">fileio</span><span class="o">.</span><span class="n">ReadMatches</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Reshuffle</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">files_and_contents</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">readable_files</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="p">(</span><span class="n">x</span><span class="o">.</span><span class="n">metadata</span><span class="o">.</span><span class="n">path</span><span class="p">,</span> <span class="n">x</span><span class="o">.</span><span class="n">read_utf8</span><span class="p">())))</span></span></span></code></pre></div> |
| </div> |
| </div></description></item><item><title>Documentation: Filter</title><link>/documentation/transforms/java/elementwise/filter/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/java/elementwise/filter/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="filter">Filter</h1> |
| <table align="left"> |
| <a target="_blank" class="button" |
| href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/transforms/Filter.html"> |
| <img src="/images/logos/sdks/java.png" width="20px" height="20px" |
| alt="Javadoc" /> |
| Javadoc |
| </a> |
| </table> |
| <br><br> |
| <p>Given a predicate, filter out all elements that don&rsquo;t satisfy that predicate. |
| May also be used to filter based on an inequality with a given value based |
| on the natural ordering of the element.</p> |
| <h2 id="examples">Examples</h2> |
| <p><strong>Example 1</strong>: Filtering with a predicate</p> |
| <p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">allStrings</span> <span class="o">=</span> <span class="n">Create</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;Hello&#34;</span><span class="o">,</span> <span class="s">&#34;world&#34;</span><span class="o">,</span> <span class="s">&#34;hi&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">longStrings</span> <span class="o">=</span> <span class="n">allStrings</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Filter</span><span class="o">.</span><span class="na">by</span><span class="o">(</span><span class="k">new</span> <span class="n">SerializableFunction</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Boolean</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">Boolean</span> <span class="nf">apply</span><span class="o">(</span><span class="n">String</span> <span class="n">input</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">input</span><span class="o">.</span><span class="na">length</span><span class="o">()</span> <span class="o">&gt;</span> <span class="n">3</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| The result is a <code>PCollection</code> containing &ldquo;Hello&rdquo; and &ldquo;world&rdquo;.</p> |
| <p><strong>Example 2</strong>: Filtering with an inequality</p> |
| <p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">&gt;</span> <span class="n">numbers</span> <span class="o">=</span> <span class="n">Create</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">1L</span><span class="o">,</span> <span class="n">2L</span><span class="o">,</span> <span class="n">3L</span><span class="o">,</span> <span class="n">4L</span><span class="o">,</span> <span class="n">5L</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">&gt;</span> <span class="n">bigNumbers</span> <span class="o">=</span> <span class="n">numbers</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Filter</span><span class="o">.</span><span class="na">greaterThan</span><span class="o">(</span><span class="n">3</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">Long</span><span class="o">&gt;</span> <span class="n">smallNumbers</span> <span class="o">=</span> <span class="n">numbers</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Filter</span><span class="o">.</span><span class="na">lessThanEq</span><span class="o">(</span><span class="n">3</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| Other variants include <code>Filter.greaterThanEq</code>, <code>Filter.lessThan</code> and <code>Filter.equal</code>.</p> |
| <p><strong>Example 3</strong>: Filtering with lambda</p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-java playground-snippet" |
| data-sdk="java" |
| data-path="SDK_JAVA_Filter" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_JAVA_Filter%22%2c%22sdk%22%3a%22java%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="related-transforms">Related transforms</h2> |
| <ul> |
| <li><a href="/documentation/transforms/java/elementwise/flatmapelements">FlatMapElements</a> behaves the same as <code>Map</code>, but for |
| each input it might produce zero or more outputs.</li> |
| <li><a href="/documentation/transforms/java/elementwise/pardo">ParDo</a> is the most general element-wise mapping |
| operation, and includes other abilities such as multiple output collections and side-inputs.</li> |
| </ul></description></item></channel></rss> |
