| <rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache Beam – Use Beam</title><link>/get-started/</link><description>Recent content in Use Beam on Apache Beam</description><generator>Hugo -- gohugo.io</generator><language>en</language><atom:link href="/get-started/index.xml" rel="self" type="application/rss+xml"/><item><title>Get-Started: An Interactive Overview of Beam</title><link>/get-started/an-interactive-overview-of-beam/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/an-interactive-overview-of-beam/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="an-interactive-overview-of-beam">An Interactive Overview of Beam</h1> |
| <p>Here you can find a collection of the interactive notebooks available for Apache Beam, which are hosted in |
| <a href="https://colab.research.google.com">Colab</a>. |
| The notebooks allow you to interactively play with the code and see how your changes affect the pipeline. |
| You don&rsquo;t need to install anything or modify your computer in any way to use these notebooks.</p> |
| <p>You can also <a href="/get-started/try-apache-beam">try an Apache Beam pipeline</a> using the Java, Python, and Go SDKs.</p> |
| <h2 id="get-started">Get started</h2> |
| <h3 id="learn-the-basics">Learn the basics</h3> |
| <p>In this notebook we go through the basics of what is Apache Beam and how to get started. |
| We learn what is a data pipeline, a PCollection, a PTransform, as well as some basic transforms like <code>Map</code>, <code>FlatMap</code>, <code>Filter</code>, <code>Combine</code>, and <code>GroupByKey</code>.</p> |
| <table align="left"> |
| <td> |
| <a class="button" target="_blank" href="https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/interactive-overview/getting-started.ipynb"> |
| <img alt="Run in Colab" width="32px" height="32px" |
| src="https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png" /> |
| Run in Colab |
| </a> |
| </td> |
| </table> |
| <p><br><br><br><br></p> |
| <h3 id="reading-and-writing-data">Reading and writing data</h3> |
| <p>In this notebook we go through some examples on how to read and write data to and from different data formats. |
| We introduce the built-in <code>ReadFromText</code> and <code>WriteToText</code> transforms. |
| We also see how we can read from CSV files, read from a SQLite database, write fixed-sized batches of elements, and write windows of elements.</p> |
| <table align="left"> |
| <td> |
| <a class="button" target="_blank" href="https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/interactive-overview/reading-and-writing-data.ipynb"> |
| <img alt="Run in Colab" width="32px" height="32px" |
| src="https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png" /> |
| Run in Colab |
| </a> |
| </td> |
| </table> |
| <p><br><br><br><br></p> |
| <h3 id="windowing">Windowing</h3> |
| <p>In this notebook we go through how to aggregate data based on time intervals, or in streaming pipelines. |
| We introduce the <code>GlobalWindow</code>, <code>FixedWindows</code>, <code>SlidingWindows</code>, and <code>Sessions</code>.</p> |
| <table align="left"> |
| <td> |
| <a class="button" target="_blank" href="https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/interactive-overview/windowing.ipynb"> |
| <img alt="Run in Colab" width="32px" height="32px" |
| src="https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png" /> |
| Run in Colab |
| </a> |
| </td> |
| </table> |
| <p><br><br><br><br></p> |
| <h3 id="dataframes">DataFrames</h3> |
| <p>Beam DataFrames provide a pandas-like <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html">DataFrame</a> |
| API to declare Beam pipelines. |
| To learn more about Beam DataFrames, take a look at the |
| <a href="/documentation/dsls/dataframes/overview">Beam DataFrames overview</a> page.</p> |
| <table align="left"> |
| <td> |
| <a class="button" target="_blank" href="https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/interactive-overview/dataframes.ipynb"> |
| <img alt="Run in Colab" width="32px" height="32px" |
| src="https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png" /> |
| Run in Colab |
| </a> |
| </td> |
| </table> |
| <p><br><br><br><br></p> |
| <h2 id="transforms">Transforms</h2> |
| <p>Check the <a href="/documentation/transforms/python/overview/">Python transform catalog</a> |
| for a complete list of the available transforms.</p> |
| <h3 id="element-wise-transforms">Element-wise transforms</h3> |
| <h4 id="map">Map</h4> |
| <p>Applies a simple one-to-one mapping function over each element in the collection.</p> |
| <table align="left"> |
| <td> |
| <a class="button" target="_blank" href="https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/elementwise/map-py.ipynb"> |
| <img alt="Run in Colab" width="32px" height="32px" |
| src="https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png" /> |
| Run in Colab |
| </a> |
| </td> |
| </table> |
| <p><br><br><br><br></p> |
| <h4 id="flatmap">FlatMap</h4> |
| <p>Applies a simple one-to-many mapping function over each element in the collection. The many elements are flattened into the resulting collection.</p> |
| <table align="left"> |
| <td> |
| <a class="button" target="_blank" href="https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/elementwise/flatmap-py.ipynb"> |
| <img alt="Run in Colab" width="32px" height="32px" |
| src="https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png" /> |
| Run in Colab |
| </a> |
| </td> |
| </table> |
| <p><br><br><br><br></p> |
| <h4 id="filter">Filter</h4> |
| <p>Given a predicate, filter out all elements that don’t satisfy that predicate.</p> |
| <table align="left"> |
| <td> |
| <a class="button" target="_blank" href="https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb"> |
| <img alt="Run in Colab" width="32px" height="32px" |
| src="https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png" /> |
| Run in Colab |
| </a> |
| </td> |
| </table> |
| <p><br><br><br><br></p> |
| <h4 id="partition">Partition</h4> |
| <p>Separates elements in a collection into multiple output collections.</p> |
| <table align="left"> |
| <td> |
| <a class="button" target="_blank" href="https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/elementwise/partition-py.ipynb"> |
| <img alt="Run in Colab" width="32px" height="32px" |
| src="https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png" /> |
| Run in Colab |
| </a> |
| </td> |
| </table> |
| <p><br><br><br><br></p> |
| <h4 id="pardo">ParDo</h4> |
| <p>A transform for generic parallel processing. It&rsquo;s recommended to use <code>Map</code>, <code>FlatMap</code>, <code>Filter</code> or other more specific transforms when possible.</p> |
| <table align="left"> |
| <td> |
| <a class="button" target="_blank" href="https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/documentation/transforms/python/elementwise/pardo-py.ipynb"> |
| <img alt="Run in Colab" width="32px" height="32px" |
| src="https://github.com/googlecolab/open_in_colab/raw/master/images/icon32.png" /> |
| Run in Colab |
| </a> |
| </td> |
| </table> |
| <p><br><br><br><br></p></description></item><item><title>Get-Started: Beam Mobile Gaming Example</title><link>/get-started/mobile-gaming-example/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/mobile-gaming-example/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="apache-beam-mobile-gaming-pipeline-examples">Apache Beam Mobile Gaming Pipeline Examples</h1> |
| <nav id="TableOfContents"> |
| <ul> |
| <li><a href="#userscore-basic-score-processing-in-batch">UserScore: Basic Score Processing in Batch</a> |
| <ul> |
| <li><a href="#what-does-userscore-do">What Does UserScore Do?</a></li> |
| <li><a href="#limitations">Limitations</a></li> |
| </ul> |
| </li> |
| <li><a href="#hourlyteamscore-advanced-processing-in-batch-with-windowing">HourlyTeamScore: Advanced Processing in Batch with Windowing</a> |
| <ul> |
| <li><a href="#what-does-hourlyteamscore-do">What Does HourlyTeamScore Do?</a> |
| <ul> |
| <li><a href="#fixed-time-windowing">Fixed-Time Windowing</a></li> |
| <li><a href="#filtering-based-on-event-time">Filtering Based On Event Time</a></li> |
| <li><a href="#calculating-score-per-team-per-window">Calculating Score Per Team, Per Window</a></li> |
| </ul> |
| </li> |
| <li><a href="#limitations-1">Limitations</a></li> |
| </ul> |
| </li> |
| <li><a href="#leaderboard-streaming-processing-with-real-time-game-data">LeaderBoard: Streaming Processing with Real-Time Game Data</a> |
| <ul> |
| <li><a href="#what-does-leaderboard-do">What Does LeaderBoard Do?</a> |
| <ul> |
| <li><a href="#calculating-user-score-based-on-processing-time">Calculating User Score based on Processing Time</a></li> |
| <li><a href="#calculating-team-score-based-on-event-time">Calculating Team Score based on Event Time</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li><a href="#gamestats-abuse-detection-and-usage-analysis">GameStats: Abuse Detection and Usage Analysis</a> |
| <ul> |
| <li><a href="#what-does-gamestats-do">What Does GameStats Do?</a> |
| <ul> |
| <li><a href="#abuse-detection">Abuse Detection</a></li> |
| <li><a href="#analyzing-usage-patterns">Analyzing Usage Patterns</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li><a href="#next-steps">Next Steps</a></li> |
| </ul> |
| </nav> |
| <nav class="language-switcher"> |
| <strong>Adapt for:</strong> |
| <ul> |
| <li data-value="java" class="active">Java SDK</li> |
| <li data-value="py">Python SDK</li> |
| </ul> |
| </nav> |
| <p>This section provides a walkthrough of a series of example Apache Beam pipelines that demonstrate more complex functionality than the basic <a href="/get-started/wordcount-example">WordCount</a> examples. The pipelines in this section process data from a hypothetical game that users play on their mobile phones. The pipelines demonstrate processing at increasing levels of complexity; the first pipeline, for example, shows how to run a batch analysis job to obtain relatively simple score data, while the later pipelines use Beam&rsquo;s windowing and triggers features to provide low-latency data analysis and more complex intelligence about user&rsquo;s play patterns.</p> |
| <p class="language-java"><blockquote> |
| <p><strong>Note</strong>: These examples assume some familiarity with the Beam programming model. If you haven&rsquo;t already, we recommend familiarizing yourself with the programming model documentation and running a basic example pipeline before continuing. Note also that these examples use the Java 8 lambda syntax, and thus require Java 8. However, you can create pipelines with equivalent functionality using Java 7.</p> |
| </blockquote> |
| </p> |
| <p class="language-py"><blockquote> |
| <p><strong>Note</strong>: These examples assume some familiarity with the Beam programming model. If you haven&rsquo;t already, we recommend familiarizing yourself with the programming model documentation and running a basic example pipeline before continuing.</p> |
| </blockquote> |
| </p> |
| <blockquote> |
| <p><strong>Note</strong>: MobileGaming is not yet available for the Go SDK. There is an open issue for this |
| (<a href="https://github.com/apache/beam/issues/18806">Issue 18806</a>).</p> |
| </blockquote> |
| <p>Every time a user plays an instance of our hypothetical mobile game, they generate a data event. Each data event consists of the following information:</p> |
| <ul> |
| <li>The unique ID of the user playing the game.</li> |
| <li>The team ID for the team to which the user belongs.</li> |
| <li>A score value for that particular instance of play.</li> |
| <li>A timestamp that records when the particular instance of play happened&ndash;this is the event time for each game data event.</li> |
| </ul> |
| <p>When the user completes an instance of the game, their phone sends the data event to a game server, where the data is logged and stored in a file. Generally the data is sent to the game server immediately upon completion. However, sometimes delays can happen in the network at various points. Another possible scenario involves users who play the game &ldquo;offline&rdquo;, when their phones are out of contact with the server (such as on an airplane, or outside network coverage area). When the user&rsquo;s phone comes back into contact with the game server, the phone will send all accumulated game data. In these cases, some data events may arrive delayed and out of order.</p> |
| <p>The following diagram shows the ideal situation (events are processed as they occur) vs. reality (there is often a time delay before processing).</p> |
| <p><img src="/images/gaming-example-basic.png" alt="There is often a time delay before processing events."></p> |
| <p><em>Figure 1: The X-axis represents event time: the actual time a game event |
| occurred. The Y-axis represents processing time: the time at which a game event |
| was processed. Ideally, events should be processed as they occur, depicted by |
| the dotted line in the diagram. However, in reality that is not the case and it |
| looks more like what is depicted by the red squiggly line above the ideal line.</em></p> |
| <p>The data events might be received by the game server significantly later than users generate them. This time difference (called <strong>skew</strong>) can have processing implications for pipelines that make calculations that consider when each score was generated. Such pipelines might track scores generated during each hour of a day, for example, or they calculate the length of time that users are continuously playing the game—both of which depend on each data record&rsquo;s event time.</p> |
| <p>Because some of our example pipelines use data files (like logs from the game server) as input, the event timestamp for each game might be embedded in the data&ndash;that is, it&rsquo;s a field in each data record. Those pipelines need to parse the event timestamp from each data record after reading it from the input file.</p> |
| <p>For pipelines that read unbounded game data from an unbounded source, the data source sets the intrinsic <a href="/documentation/programming-guide/#element-timestamps">timestamp</a> for each PCollection element to the appropriate event time.</p> |
| <p>The Mobile Gaming example pipelines vary in complexity, from simple batch analysis to more complex pipelines that can perform real-time analysis and abuse detection. This section walks you through each example and demonstrates how to use Beam features like windowing and triggers to expand your pipeline&rsquo;s capabilites.</p> |
| <h2 id="userscore-basic-score-processing-in-batch">UserScore: Basic Score Processing in Batch</h2> |
| <p>The <code>UserScore</code> pipeline is the simplest example for processing mobile game data. <code>UserScore</code> determines the total score per user over a finite data set (for example, one day&rsquo;s worth of scores stored on the game server). Pipelines like <code>UserScore</code> are best run periodically after all relevant data has been gathered. For example, <code>UserScore</code> could run as a nightly job over data gathered during that day.</p> |
| <p class="language-java"><blockquote> |
| <p><strong>Note:</strong> See <a href="https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/UserScore.java">UserScore on GitHub</a> for the complete example pipeline program.</p> |
| </blockquote> |
| </p> |
| <p class="language-py"><blockquote> |
| <p><strong>Note:</strong> See <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/user_score.py">UserScore on GitHub</a> for the complete example pipeline program.</p> |
| </blockquote> |
| </p> |
| <h3 id="what-does-userscore-do">What Does UserScore Do?</h3> |
| <p>In a day&rsquo;s worth of scoring data, each user ID may have multiple records (if the user plays more than one instance of the game during the analysis window), each with their own score value and timestamp. If we want to determine the total score over all the instances a user plays during the day, our pipeline will need to group all the records together per individual user.</p> |
| <p>As the pipeline processes each event, the event score gets added to the sum total for that particular user.</p> |
| <p><code>UserScore</code> parses out only the data that it needs from each record, specifically the user ID and the score value. The pipeline doesn&rsquo;t consider the event time for any record; it simply processes all data present in the input files that you specify when you run the pipeline.</p> |
| <blockquote> |
| <p><strong>Note:</strong> To use the <code>UserScore</code> pipeline effectively, you&rsquo;d need to ensure that you supply input data that has already been grouped by the desired event time period — that is, that you specify an input file that only contains data from the day you care about.</p> |
| </blockquote> |
| <p><code>UserScore</code>&rsquo;s basic pipeline flow does the following:</p> |
| <ol> |
| <li>Read the day&rsquo;s score data from a text file.</li> |
| <li>Sum the score values for each unique user by grouping each game event by user ID and combining the score values to get the total score for that particular user.</li> |
| <li>Write the result data to a text file.</li> |
| </ol> |
| <p>The following diagram shows score data for several users over the pipeline analysis period. In the diagram, each data point is an event that results in one user/score pair.</p> |
| <img src="/images/gaming-example.gif" alt="A pipeline processes score data for three users." width="850px"> |
| <p><em>Figure 2: Score data for three users.</em></p> |
| <p>This example uses batch processing, and the diagram&rsquo;s Y axis represents processing time: the pipeline processes events lower on the Y-axis first, and events higher up the axis later. The diagram&rsquo;s X axis represents the event time for each game event, as denoted by that event&rsquo;s timestamp. Note that the individual events in the diagram are not processed by the pipeline in the same order as they occurred (according to their timestamps).</p> |
| <p>After reading the score events from the input file, the pipeline groups all of those user/score pairs together and sums the score values into one total value per unique user. <code>UserScore</code> encapsulates the core logic for that step as the <a href="/documentation/programming-guide/#composite-transforms">user-defined composite transform</a> <code>ExtractAndSumScore</code>:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">ExtractAndSumScore</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">extends</span> <span class="n">PTransform</span><span class="o">&lt;</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">GameActionInfo</span><span class="o">&gt;,</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="kd">private</span> <span class="kd">final</span> <span class="n">String</span> <span class="n">field</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">ExtractAndSumScore</span><span class="o">(</span><span class="n">String</span> <span class="n">field</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">this</span><span class="o">.</span><span class="na">field</span> <span class="o">=</span> <span class="n">field</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="nf">expand</span><span class="o">(</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">GameActionInfo</span><span class="o">&gt;</span> <span class="n">gameInfo</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">gameInfo</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">kvs</span><span class="o">(</span><span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">(),</span> <span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">integers</span><span class="o">()))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">via</span><span class="o">((</span><span class="n">GameActionInfo</span> <span class="n">gInfo</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">gInfo</span><span class="o">.</span><span class="na">getKey</span><span class="o">(</span><span class="n">field</span><span class="o">),</span> <span class="n">gInfo</span><span class="o">.</span><span class="na">getScore</span><span class="o">())))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Sum</span><span class="o">.</span><span class="na">integersPerKey</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">ExtractAndSumScore</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">PTransform</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;&#34;&#34;A transform to extract key/score information and sum the scores. |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> The constructor argument `field` determines whether &#39;team&#39; or &#39;user&#39; info is |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> extracted. |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> &#34;&#34;&#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">field</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># super().__init__()</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">PTransform</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">field</span> <span class="o">=</span> <span class="n">field</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">expand</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pcoll</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pcoll</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">elem</span><span class="p">:</span> <span class="p">(</span><span class="n">elem</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">field</span><span class="p">],</span> <span class="n">elem</span><span class="p">[</span><span class="s1">&#39;score&#39;</span><span class="p">]))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombinePerKey</span><span class="p">(</span><span class="nb">sum</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p><code>ExtractAndSumScore</code> is written to be more general, in that you can pass in the field by which you want to group the data (in the case of our game, by unique user or unique team). This means we can re-use <code>ExtractAndSumScore</code> in other pipelines that group score data by team, for example.</p> |
| <p>Here&rsquo;s the main method of <code>UserScore</code>, showing how we apply all three steps of the pipeline:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Begin constructing a pipeline configured by commandline flags. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">Options</span> <span class="n">options</span> <span class="o">=</span> <span class="n">PipelineOptionsFactory</span><span class="o">.</span><span class="na">fromArgs</span><span class="o">(</span><span class="n">args</span><span class="o">).</span><span class="na">withValidation</span><span class="o">().</span><span class="na">as</span><span class="o">(</span><span class="n">Options</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Pipeline</span> <span class="n">pipeline</span> <span class="o">=</span> <span class="n">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Read events from a text file and parse them. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">TextIO</span><span class="o">.</span><span class="na">read</span><span class="o">().</span><span class="na">from</span><span class="o">(</span><span class="n">options</span><span class="o">.</span><span class="na">getInput</span><span class="o">()))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;ParseGameEvent&#34;</span><span class="o">,</span> <span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">ParseEventFn</span><span class="o">()))</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Extract and sum username/score pairs from the event data. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;ExtractUserScore&#34;</span><span class="o">,</span> <span class="k">new</span> <span class="n">ExtractAndSumScore</span><span class="o">(</span><span class="s">&#34;user&#34;</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;WriteUserScoreSums&#34;</span><span class="o">,</span> <span class="k">new</span> <span class="n">WriteToText</span><span class="o">&lt;&gt;(</span><span class="n">options</span><span class="o">.</span><span class="na">getOutput</span><span class="o">(),</span> <span class="n">configureOutput</span><span class="o">(),</span> <span class="kc">false</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Run the batch pipeline. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">pipeline</span><span class="o">.</span><span class="na">run</span><span class="o">().</span><span class="na">waitUntilFinish</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="n">argv</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">save_main_session</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;&#34;&#34;Main entry point; defines and runs the user_score pipeline.&#34;&#34;&#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">parser</span> <span class="o">=</span> <span class="n">argparse</span><span class="o">.</span><span class="n">ArgumentParser</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># The default maps to two large Google Cloud Storage files (each ~12GB)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># holding two subsequent day&#39;s worth (roughly) of data.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;--input&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nb">type</span><span class="o">=</span><span class="nb">str</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">default</span><span class="o">=</span><span class="s1">&#39;gs://apache-beam-samples/game/small/gaming_data.csv&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;Path to the data file(s) containing game data.&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;--output&#39;</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">str</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;Path to the output file(s).&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">args</span><span class="p">,</span> <span class="n">pipeline_args</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_known_args</span><span class="p">(</span><span class="n">argv</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">options</span> <span class="o">=</span> <span class="n">PipelineOptions</span><span class="p">(</span><span class="n">pipeline_args</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># We use the save_main_session option because one or more DoFn&#39;s in this</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># workflow rely on global context (e.g., a module imported at module level).</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">options</span><span class="o">.</span><span class="n">view_as</span><span class="p">(</span><span class="n">SetupOptions</span><span class="p">)</span><span class="o">.</span><span class="n">save_main_session</span> <span class="o">=</span> <span class="n">save_main_session</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">(</span><span class="n">options</span><span class="o">=</span><span class="n">options</span><span class="p">)</span> <span class="k">as</span> <span class="n">p</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">format_user_score_sums</span><span class="p">(</span><span class="n">user_score</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="n">score</span><span class="p">)</span> <span class="o">=</span> <span class="n">user_score</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="s1">&#39;user: </span><span class="si">%s</span><span class="s1">, total_score: </span><span class="si">%s</span><span class="s1">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="n">score</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span> <span class="c1"># pylint: disable=expression-not-assigned</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">p</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;ReadInputText&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ReadFromText</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">input</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;UserScore&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">UserScore</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;FormatUserScoreSums&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="n">format_user_score_sums</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;WriteUserScoreSums&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">WriteToText</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">output</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="limitations">Limitations</h3> |
| <p>As written in the example, the <code>UserScore</code> pipeline has a few limitations:</p> |
| <ul> |
| <li> |
| <p>Because some score data may be generated by offline players and sent after the daily cutoff, for game data, the result data generated by the <code>UserScore</code> pipeline <strong>may be incomplete</strong>. <code>UserScore</code> only processes the fixed input set present in the input file(s) when the pipeline runs.</p> |
| </li> |
| <li> |
| <p><code>UserScore</code> processes all data events present in the input file at processing time, and <strong>does not examine or otherwise error-check events based on event time</strong>. Therefore, the results may include some values whose event times fall outside the relevant analysis period, such as late records from the previous day.</p> |
| </li> |
| <li> |
| <p>Because <code>UserScore</code> runs only after all the data has been collected, it has <strong>high latency</strong> between when users generate data events (the event time) and when results are computed (the processing time).</p> |
| </li> |
| <li> |
| <p><code>UserScore</code> also only reports the total results for the entire day, and doesn&rsquo;t provide any finer-grained information about how the data accumulated during the day.</p> |
| </li> |
| </ul> |
| <p>Starting with the next pipeline example, we&rsquo;ll discuss how you can use Beam&rsquo;s features to address these limitations.</p> |
| <h2 id="hourlyteamscore-advanced-processing-in-batch-with-windowing">HourlyTeamScore: Advanced Processing in Batch with Windowing</h2> |
| <p>The <code>HourlyTeamScore</code> pipeline expands on the basic batch analysis principles used in the <code>UserScore</code> pipeline and improves upon some of its limitations. <code>HourlyTeamScore</code> performs finer-grained analysis, both by using additional features in the Beam SDKs, and taking into account more aspects of the game data. For example, <code>HourlyTeamScore</code> can filter out data that isn&rsquo;t part of the relevant analysis period.</p> |
| <p>Like <code>UserScore</code>, <code>HourlyTeamScore</code> is best thought of as a job to be run periodically after all the relevant data has been gathered (such as once per day). The pipeline reads a fixed data set from a file, and writes the results <span class="language-java">back to a text file</span><span class="language-py">to a Google Cloud BigQuery table</span>.</p> |
| <p class="language-java"><blockquote> |
| <p><strong>Note:</strong> See <a href="https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java">HourlyTeamScore on GitHub</a> for the complete example pipeline program.</p> |
| </blockquote> |
| </p> |
| <p class="language-py"><blockquote> |
| <p><strong>Note:</strong> See <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py">HourlyTeamScore on GitHub</a> for the complete example pipeline program.</p> |
| </blockquote> |
| </p> |
| <h3 id="what-does-hourlyteamscore-do">What Does HourlyTeamScore Do?</h3> |
| <p><code>HourlyTeamScore</code> calculates the total score per team, per hour, in a fixed data set (such as one day&rsquo;s worth of data).</p> |
| <ul> |
| <li> |
| <p>Rather than operating on the entire data set at once, <code>HourlyTeamScore</code> divides the input data into logical windows and performs calculations on those windows. This allows <code>HourlyUserScore</code> to provide information on scoring data per window, where each window represents the game score progress at fixed intervals in time (like once every hour).</p> |
| </li> |
| <li> |
| <p><code>HourlyTeamScore</code> filters data events based on whether their event time (as indicated by the embedded timestamp) falls within the relevant analysis period. Basically, the pipeline checks each game event&rsquo;s timestamp and ensures that it falls within the range we want to analyze (in this case the day in question). Data events from previous days are discarded and not included in the score totals. This makes <code>HourlyTeamScore</code> more robust and less prone to erroneous result data than <code>UserScore</code>. It also allows the pipeline to account for late-arriving data that has a timestamp within the relevant analysis period.</p> |
| </li> |
| </ul> |
| <p>Below, we&rsquo;ll look at each of these enhancements in <code>HourlyTeamScore</code> in detail:</p> |
| <h4 id="fixed-time-windowing">Fixed-Time Windowing</h4> |
| <p>Using fixed-time windowing lets the pipeline provide better information on how events accumulated in the data set over the course of the analysis period. In our case, it tells us when in the day each team was active and how much the team scored at those times.</p> |
| <p>The following diagram shows how the pipeline processes a day&rsquo;s worth of a single team&rsquo;s scoring data after applying fixed-time windowing:</p> |
| <img src="/images/gaming-example-team-scores-narrow.gif" alt="A pipeline processes score data for two teams." width="800px"> |
| <p><em>Figure 3: Score data for two teams. Each team&rsquo;s scores are divided into |
| logical windows based on when those scores occurred in event time.</em></p> |
| <p>Notice that as processing time advances, the sums are now <em>per window</em>; each window represents an hour of <em>event time</em> during the day in which the scores occurred.</p> |
| <blockquote> |
| <p><strong>Note:</strong> As is shown in the diagram above, using windowing produces an <em>independent total for every interval</em> (in this case, each hour). <code>HourlyTeamScore</code> doesn&rsquo;t provide a running total for the entire data set at each hour&ndash;it provides the total score for all the events that occurred <em>only within that hour</em>.</p> |
| </blockquote> |
| <p>Beam&rsquo;s windowing feature uses the <a href="/documentation/programming-guide/#element-timestamps">intrinsic timestamp information</a> attached to each element of a <code>PCollection</code>. Because we want our pipeline to window based on <em>event time</em>, we <strong>must first extract the timestamp</strong> that&rsquo;s embedded in each data record apply it to the corresponding element in the <code>PCollection</code> of score data. Then, the pipeline can <strong>apply the windowing function</strong> to divide the <code>PCollection</code> into logical windows.</p> |
| <p class="language-java"><code>HourlyTeamScore</code> uses the <a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithTimestamps.java">WithTimestamps</a> and <a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java">Window</a> transforms to perform these operations.</p> |
| <p class="language-py"><code>HourlyTeamScore</code> uses the <code>FixedWindows</code> transform, found in <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/window.py">window.py</a>, to perform these operations.</p> |
| <p>The following code shows this:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Add an element timestamp based on the event log, and apply fixed windowing. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;AddEventTimestamps&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">WithTimestamps</span><span class="o">.</span><span class="na">of</span><span class="o">((</span><span class="n">GameActionInfo</span> <span class="n">i</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="n">i</span><span class="o">.</span><span class="na">getTimestamp</span><span class="o">())))</span> |
| </span></span><span class="line"><span class="cl"><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;FixedWindowsTeam&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Window</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">FixedWindows</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="n">options</span><span class="o">.</span><span class="na">getWindowDuration</span><span class="o">()))))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># Add an element timestamp based on the event log, and apply fixed</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># windowing.</span> |
| </span></span><span class="line"><span class="cl"><span class="o">|</span> <span class="s1">&#39;AddEventTimestamps&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">lambda</span> <span class="n">elem</span><span class="p">:</span> <span class="n">beam</span><span class="o">.</span><span class="n">window</span><span class="o">.</span><span class="n">TimestampedValue</span><span class="p">(</span><span class="n">elem</span><span class="p">,</span> <span class="n">elem</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">]))</span> |
| </span></span><span class="line"><span class="cl"><span class="o">|</span> <span class="s1">&#39;FixedWindowsTeam&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">window</span><span class="o">.</span><span class="n">FixedWindows</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">window_duration_in_seconds</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Notice that the transforms the pipeline uses to specify the windowing are distinct from the actual data processing transforms (such as <code>ExtractAndSumScores</code>). This functionality provides you some flexibility in designing your Beam pipeline, in that you can run existing transforms over datasets with different windowing characteristics.</p> |
| <h4 id="filtering-based-on-event-time">Filtering Based On Event Time</h4> |
| <p><code>HourlyTeamScore</code> uses <strong>filtering</strong> to remove any events from our dataset whose timestamps don&rsquo;t fall within the relevant analysis period (i.e. they weren&rsquo;t generated during the day that we&rsquo;re interested in). This keeps the pipeline from erroneously including any data that was, for example, generated offline during the previous day but sent to the game server during the current day.</p> |
| <p>It also lets the pipeline include relevant <strong>late data</strong>—data events with valid timestamps, but that arrived after our analysis period ended. If our pipeline cutoff time is 12:00 am, for example, we might run the pipeline at 2:00 am, but filter out any events whose timestamps indicate that they occurred after the 12:00 am cutoff. Data events that were delayed and arrived between 12:01 am and 2:00 am, but whose timestamps indicate that they occurred before the 12:00 am cutoff, would be included in the pipeline processing.</p> |
| <p><code>HourlyTeamScore</code> uses the <code>Filter</code> transform to perform this operation. When you apply <code>Filter</code>, you specify a predicate to which each data record is compared. Data records that pass the comparison are included, while events that fail the comparison are excluded. In our case, the predicate is the cut-off time we specify, and we compare just one part of the data—the timestamp field.</p> |
| <p>The following code shows how <code>HourlyTeamScore</code> uses the <code>Filter</code> transform to filter events that occur either before or after the relevant analysis period:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;FilterStartTime&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Filter</span><span class="o">.</span><span class="na">by</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">(</span><span class="n">GameActionInfo</span> <span class="n">gInfo</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">gInfo</span><span class="o">.</span><span class="na">getTimestamp</span><span class="o">()</span> <span class="o">&gt;</span> <span class="n">startMinTimestamp</span><span class="o">.</span><span class="na">getMillis</span><span class="o">()))</span> |
| </span></span><span class="line"><span class="cl"><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;FilterEndTime&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Filter</span><span class="o">.</span><span class="na">by</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">(</span><span class="n">GameActionInfo</span> <span class="n">gInfo</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">gInfo</span><span class="o">.</span><span class="na">getTimestamp</span><span class="o">()</span> <span class="o">&lt;</span> <span class="n">stopMinTimestamp</span><span class="o">.</span><span class="na">getMillis</span><span class="o">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="o">|</span> <span class="s1">&#39;FilterStartTime&#39;</span> <span class="o">&gt;&gt;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">beam</span><span class="o">.</span><span class="n">Filter</span><span class="p">(</span><span class="k">lambda</span> <span class="n">elem</span><span class="p">:</span> <span class="n">elem</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">]</span> <span class="o">&gt;</span> <span class="bp">self</span><span class="o">.</span><span class="n">start_timestamp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="o">|</span> <span class="s1">&#39;FilterEndTime&#39;</span> <span class="o">&gt;&gt;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">beam</span><span class="o">.</span><span class="n">Filter</span><span class="p">(</span><span class="k">lambda</span> <span class="n">elem</span><span class="p">:</span> <span class="n">elem</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">]</span> <span class="o">&lt;</span> <span class="bp">self</span><span class="o">.</span><span class="n">stop_timestamp</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="calculating-score-per-team-per-window">Calculating Score Per Team, Per Window</h4> |
| <p><code>HourlyTeamScore</code> uses the same <code>ExtractAndSumScores</code> transform as the <code>UserScore</code> pipeline, but passes a different key (team, as opposed to user). Also, because the pipeline applies <code>ExtractAndSumScores</code> <em>after</em> applying fixed-time 1-hour windowing to the input data, the data gets grouped by both team <em>and</em> window. You can see the full sequence of transforms in <code>HourlyTeamScore</code>&rsquo;s main method:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">Exception</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Begin constructing a pipeline configured by commandline flags. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">Options</span> <span class="n">options</span> <span class="o">=</span> <span class="n">PipelineOptionsFactory</span><span class="o">.</span><span class="na">fromArgs</span><span class="o">(</span><span class="n">args</span><span class="o">).</span><span class="na">withValidation</span><span class="o">().</span><span class="na">as</span><span class="o">(</span><span class="n">Options</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Pipeline</span> <span class="n">pipeline</span> <span class="o">=</span> <span class="n">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="kd">final</span> <span class="n">Instant</span> <span class="n">stopMinTimestamp</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="n">minFmt</span><span class="o">.</span><span class="na">parseMillis</span><span class="o">(</span><span class="n">options</span><span class="o">.</span><span class="na">getStopMin</span><span class="o">()));</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">final</span> <span class="n">Instant</span> <span class="n">startMinTimestamp</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="n">minFmt</span><span class="o">.</span><span class="na">parseMillis</span><span class="o">(</span><span class="n">options</span><span class="o">.</span><span class="na">getStartMin</span><span class="o">()));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Read &#39;gaming&#39; events from a text file. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">TextIO</span><span class="o">.</span><span class="na">read</span><span class="o">().</span><span class="na">from</span><span class="o">(</span><span class="n">options</span><span class="o">.</span><span class="na">getInput</span><span class="o">()))</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Parse the incoming data. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;ParseGameEvent&#34;</span><span class="o">,</span> <span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">ParseEventFn</span><span class="o">()))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Filter out data before and after the given times so that it is not included |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// in the calculations. As we collect data in batches (say, by day), the batch for the day |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// that we want to analyze could potentially include some late-arriving data from the |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// previous day. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// If so, we want to weed it out. Similarly, if we include data from the following day |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// (to scoop up late-arriving events from the day we&#39;re analyzing), we need to weed out |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// events that fall after the time period we want to analyze. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// [START DocInclude_HTSFilters] |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;FilterStartTime&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Filter</span><span class="o">.</span><span class="na">by</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">(</span><span class="n">GameActionInfo</span> <span class="n">gInfo</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">gInfo</span><span class="o">.</span><span class="na">getTimestamp</span><span class="o">()</span> <span class="o">&gt;</span> <span class="n">startMinTimestamp</span><span class="o">.</span><span class="na">getMillis</span><span class="o">()))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;FilterEndTime&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Filter</span><span class="o">.</span><span class="na">by</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">(</span><span class="n">GameActionInfo</span> <span class="n">gInfo</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">gInfo</span><span class="o">.</span><span class="na">getTimestamp</span><span class="o">()</span> <span class="o">&lt;</span> <span class="n">stopMinTimestamp</span><span class="o">.</span><span class="na">getMillis</span><span class="o">()))</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// [END DocInclude_HTSFilters] |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// [START DocInclude_HTSAddTsAndWindow] |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// Add an element timestamp based on the event log, and apply fixed windowing. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;AddEventTimestamps&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">WithTimestamps</span><span class="o">.</span><span class="na">of</span><span class="o">((</span><span class="n">GameActionInfo</span> <span class="n">i</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="n">i</span><span class="o">.</span><span class="na">getTimestamp</span><span class="o">())))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;FixedWindowsTeam&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Window</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">FixedWindows</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="n">options</span><span class="o">.</span><span class="na">getWindowDuration</span><span class="o">()))))</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// [END DocInclude_HTSAddTsAndWindow] |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Extract and sum teamname/score pairs from the event data. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;ExtractTeamScore&#34;</span><span class="o">,</span> <span class="k">new</span> <span class="n">ExtractAndSumScore</span><span class="o">(</span><span class="s">&#34;team&#34;</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;WriteTeamScoreSums&#34;</span><span class="o">,</span> <span class="k">new</span> <span class="n">WriteToText</span><span class="o">&lt;&gt;(</span><span class="n">options</span><span class="o">.</span><span class="na">getOutput</span><span class="o">(),</span> <span class="n">configureOutput</span><span class="o">(),</span> <span class="kc">true</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">pipeline</span><span class="o">.</span><span class="na">run</span><span class="o">().</span><span class="na">waitUntilFinish</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">HourlyTeamScore</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">PTransform</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">start_min</span><span class="p">,</span> <span class="n">stop_min</span><span class="p">,</span> <span class="n">window_duration</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># super().__init__()</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">PTransform</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">start_timestamp</span> <span class="o">=</span> <span class="n">str2timestamp</span><span class="p">(</span><span class="n">start_min</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">stop_timestamp</span> <span class="o">=</span> <span class="n">str2timestamp</span><span class="p">(</span><span class="n">stop_min</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">window_duration_in_seconds</span> <span class="o">=</span> <span class="n">window_duration</span> <span class="o">*</span> <span class="mi">60</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">expand</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pcoll</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pcoll</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;ParseGameEventFn&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">ParseGameEventFn</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Filter out data before and after the given times so that it is not</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># included in the calculations. As we collect data in batches (say, by</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># day), the batch for the day that we want to analyze could potentially</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># include some late-arriving data from the previous day. If so, we want</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># to weed it out. Similarly, if we include data from the following day</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># (to scoop up late-arriving events from the day we&#39;re analyzing), we</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># need to weed out events that fall after the time period we want to</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># analyze.</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># [START filter_by_time_range]</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;FilterStartTime&#39;</span> <span class="o">&gt;&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">Filter</span><span class="p">(</span><span class="k">lambda</span> <span class="n">elem</span><span class="p">:</span> <span class="n">elem</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">]</span> <span class="o">&gt;</span> <span class="bp">self</span><span class="o">.</span><span class="n">start_timestamp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;FilterEndTime&#39;</span> <span class="o">&gt;&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">Filter</span><span class="p">(</span><span class="k">lambda</span> <span class="n">elem</span><span class="p">:</span> <span class="n">elem</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">]</span> <span class="o">&lt;</span> <span class="bp">self</span><span class="o">.</span><span class="n">stop_timestamp</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># [END filter_by_time_range]</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># [START add_timestamp_and_window]</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Add an element timestamp based on the event log, and apply fixed</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># windowing.</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;AddEventTimestamps&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">lambda</span> <span class="n">elem</span><span class="p">:</span> <span class="n">beam</span><span class="o">.</span><span class="n">window</span><span class="o">.</span><span class="n">TimestampedValue</span><span class="p">(</span><span class="n">elem</span><span class="p">,</span> <span class="n">elem</span><span class="p">[</span><span class="s1">&#39;timestamp&#39;</span><span class="p">]))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;FixedWindowsTeam&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">window</span><span class="o">.</span><span class="n">FixedWindows</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">window_duration_in_seconds</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># [END add_timestamp_and_window]</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Extract and sum teamname/score pairs from the event data.</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;ExtractAndSumScore&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ExtractAndSumScore</span><span class="p">(</span><span class="s1">&#39;team&#39;</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="n">argv</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">save_main_session</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;&#34;&#34;Main entry point; defines and runs the hourly_team_score pipeline.&#34;&#34;&#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">parser</span> <span class="o">=</span> <span class="n">argparse</span><span class="o">.</span><span class="n">ArgumentParser</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># The default maps to two large Google Cloud Storage files (each ~12GB)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># holding two subsequent day&#39;s worth (roughly) of data.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;--input&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nb">type</span><span class="o">=</span><span class="nb">str</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">default</span><span class="o">=</span><span class="s1">&#39;gs://apache-beam-samples/game/gaming_data*.csv&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;Path to the data file(s) containing game data.&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;--dataset&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nb">type</span><span class="o">=</span><span class="nb">str</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">required</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;BigQuery Dataset to write tables to. &#39;</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;Must already exist.&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;--table_name&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">default</span><span class="o">=</span><span class="s1">&#39;leader_board&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;The BigQuery table name. Should not already exist.&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;--window_duration&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">default</span><span class="o">=</span><span class="mi">60</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;Numeric value of fixed window duration, in minutes&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;--start_min&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nb">type</span><span class="o">=</span><span class="nb">str</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">default</span><span class="o">=</span><span class="s1">&#39;1970-01-01-00-00&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;String representation of the first minute after &#39;</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;which to generate results in the format: &#39;</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;yyyy-MM-dd-HH-mm. Any input data timestamped &#39;</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;prior to that minute won</span><span class="se">\&#39;</span><span class="s1">t be included in the &#39;</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;sums.&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;--stop_min&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="nb">type</span><span class="o">=</span><span class="nb">str</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">default</span><span class="o">=</span><span class="s1">&#39;2100-01-01-00-00&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;String representation of the first minute for &#39;</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;which to generate results in the format: &#39;</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;yyyy-MM-dd-HH-mm. Any input data timestamped &#39;</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;after to that minute won</span><span class="se">\&#39;</span><span class="s1">t be included in the &#39;</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;sums.&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">args</span><span class="p">,</span> <span class="n">pipeline_args</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_known_args</span><span class="p">(</span><span class="n">argv</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">options</span> <span class="o">=</span> <span class="n">PipelineOptions</span><span class="p">(</span><span class="n">pipeline_args</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># We also require the --project option to access --dataset</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">options</span><span class="o">.</span><span class="n">view_as</span><span class="p">(</span><span class="n">GoogleCloudOptions</span><span class="p">)</span><span class="o">.</span><span class="n">project</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">print_usage</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="nb">print</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">+</span> <span class="s1">&#39;: error: argument --project is required&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># We use the save_main_session option because one or more DoFn&#39;s in this</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># workflow rely on global context (e.g., a module imported at module level).</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">options</span><span class="o">.</span><span class="n">view_as</span><span class="p">(</span><span class="n">SetupOptions</span><span class="p">)</span><span class="o">.</span><span class="n">save_main_session</span> <span class="o">=</span> <span class="n">save_main_session</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">(</span><span class="n">options</span><span class="o">=</span><span class="n">options</span><span class="p">)</span> <span class="k">as</span> <span class="n">p</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span> <span class="c1"># pylint: disable=expression-not-assigned</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">p</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;ReadInputText&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ReadFromText</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">input</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;HourlyTeamScore&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">HourlyTeamScore</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">args</span><span class="o">.</span><span class="n">start_min</span><span class="p">,</span> <span class="n">args</span><span class="o">.</span><span class="n">stop_min</span><span class="p">,</span> <span class="n">args</span><span class="o">.</span><span class="n">window_duration</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;TeamScoresDict&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">TeamScoresDict</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;WriteTeamScoreSums&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">WriteToBigQuery</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">args</span><span class="o">.</span><span class="n">table_name</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">args</span><span class="o">.</span><span class="n">dataset</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;team&#39;</span><span class="p">:</span> <span class="s1">&#39;STRING&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;total_score&#39;</span><span class="p">:</span> <span class="s1">&#39;INTEGER&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;window_start&#39;</span><span class="p">:</span> <span class="s1">&#39;STRING&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">options</span><span class="o">.</span><span class="n">view_as</span><span class="p">(</span><span class="n">GoogleCloudOptions</span><span class="p">)</span><span class="o">.</span><span class="n">project</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="limitations-1">Limitations</h3> |
| <p>As written, <code>HourlyTeamScore</code> still has a limitation:</p> |
| <ul> |
| <li><code>HourlyTeamScore</code> still has <strong>high latency</strong> between when data events occur (the event time) and when results are generated (the processing time), because, as a batch pipeline, it needs to wait to begin processing until all data events are present.</li> |
| </ul> |
| <h2 id="leaderboard-streaming-processing-with-real-time-game-data">LeaderBoard: Streaming Processing with Real-Time Game Data</h2> |
| <p>One way we can help address the latency issue present in the <code>UserScore</code> and <code>HourlyTeamScore</code> pipelines is by reading the score data from an unbounded source. The <code>LeaderBoard</code> pipeline introduces streaming processing by reading the game score data from an unbounded source that produces an infinite amount of data, rather than from a file on the game server.</p> |
| <p>The <code>LeaderBoard</code> pipeline also demonstrates how to process game score data with respect to both <em>processing time</em> and <em>event time</em>. <code>LeaderBoard</code> outputs data about both individual user scores and about team scores, each with respect to a different time frame.</p> |
| <p>Because the <code>LeaderBoard</code> pipeline reads the game data from an unbounded source as that data is generated, you can think of the pipeline as an ongoing job running concurrently with the game process. <code>LeaderBoard</code> can thus provide low-latency insights into how users are playing the game at any given moment — useful if, for example, we want to provide a live web-based scoreboard so that users can track their progress against other users as they play.</p> |
| <p class="language-java"><blockquote> |
| <p><strong>Note:</strong> See <a href="https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java">LeaderBoard on GitHub</a> for the complete example pipeline program.</p> |
| </blockquote> |
| </p> |
| <p class="language-py"><blockquote> |
| <p><strong>Note:</strong> See <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/leader_board.py">LeaderBoard on GitHub</a> for the complete example pipeline program.</p> |
| </blockquote> |
| </p> |
| <h3 id="what-does-leaderboard-do">What Does LeaderBoard Do?</h3> |
| <p>The <code>LeaderBoard</code> pipeline reads game data published to an unbounded source that produces an infinite amount of data in near real-time, and uses that data to perform two separate processing tasks:</p> |
| <ul> |
| <li> |
| <p><code>LeaderBoard</code> calculates the total score for every unique user and publishes speculative results for every ten minutes of <em>processing time</em>. That is, ten minutes after data is received, the pipeline outputs the total score per user that the pipeline has processed to date. This calculation provides a running &ldquo;leader board&rdquo; in close to real time, regardless of when the actual game events were generated.</p> |
| </li> |
| <li> |
| <p><code>LeaderBoard</code> calculates the team scores for each hour that the pipeline runs. This is useful if we want to, for example, reward the top-scoring team for each hour of play. The team score calculation uses fixed-time windowing to divide the input data into hour-long finite windows based on the <em>event time</em> (indicated by the timestamp) as data arrives in the pipeline.</p> |
| <p>In addition, the team score calculation uses Beam&rsquo;s trigger mechanisms to provide speculative results for each hour (which update every five minutes until the hour is up), and to also capture any late data and add it to the specific hour-long window to which it belongs.</p> |
| </li> |
| </ul> |
| <p>Below, we&rsquo;ll look at both of these tasks in detail.</p> |
| <h4 id="calculating-user-score-based-on-processing-time">Calculating User Score based on Processing Time</h4> |
| <p>We want our pipeline to output a running total score for each user for every ten minutes of processing time. This calculation doesn&rsquo;t consider <em>when</em> the actual score was generated by the user&rsquo;s play instance; it simply outputs the sum of all the scores for that user that have arrived in the pipeline to date. Late data gets included in the calculation whenever it happens to arrive in the pipeline as it&rsquo;s running.</p> |
| <p>Because we want all the data that has arrived in the pipeline every time we update our calculation, we have the pipeline consider all of the user score data in a <strong>single global window</strong>. The single global window is unbounded, but we can specify a kind of temporary cut-off point for each ten-minute calculation by using a processing time <a href="/documentation/programming-guide/#triggers">trigger</a>.</p> |
| <p>When we specify a ten-minute processing time trigger for the single global window, the pipeline effectively takes a &ldquo;snapshot&rdquo; of the contents of the window every time the trigger fires. This snapshot happens after ten minutes have passed since data was received. If no data has arrived, the pipeline takes its next &ldquo;snapshot&rdquo; 10 minutes after an element arrives. Since we&rsquo;re using a single global window, each snapshot contains all the data collected <em>to that point in time</em>. The following diagram shows the effects of using a processing time trigger on the single global window:</p> |
| <img src="/images/gaming-example-proc-time-narrow.gif" alt="A pipeline processes score data for three users." width="850px"> |
| <p><em>Figure 4: Score data for three users. Each user&rsquo;s scores are grouped together |
| in a single global window, with a trigger that generates a snapshot for output |
| ten minutes after data is received.</em></p> |
| <p>As processing time advances and more scores are processed, the trigger outputs the updated sum for each user.</p> |
| <p>The following code example shows how <code>LeaderBoard</code> sets the processing time trigger to output the data for user scores:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="cm">/** |
| </span></span></span><span class="line"><span class="cl"><span class="cm"> * Extract user/score pairs from the event stream using processing time, via global windowing. Get |
| </span></span></span><span class="line"><span class="cl"><span class="cm"> * periodic updates on all users&#39; running scores. |
| </span></span></span><span class="line"><span class="cl"><span class="cm"> */</span> |
| </span></span><span class="line"><span class="cl"><span class="nd">@VisibleForTesting</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">static</span> <span class="kd">class</span> <span class="nc">CalculateUserScores</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">extends</span> <span class="n">PTransform</span><span class="o">&lt;</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">GameActionInfo</span><span class="o">&gt;,</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">private</span> <span class="kd">final</span> <span class="n">Duration</span> <span class="n">allowedLateness</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">CalculateUserScores</span><span class="o">(</span><span class="n">Duration</span> <span class="n">allowedLateness</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">this</span><span class="o">.</span><span class="na">allowedLateness</span> <span class="o">=</span> <span class="n">allowedLateness</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="nf">expand</span><span class="o">(</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">GameActionInfo</span><span class="o">&gt;</span> <span class="n">input</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">input</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;LeaderboardUserGlobalWindow&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Window</span><span class="o">.&lt;</span><span class="n">GameActionInfo</span><span class="o">&gt;</span><span class="n">into</span><span class="o">(</span><span class="k">new</span> <span class="n">GlobalWindows</span><span class="o">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Get periodic results every ten minutes. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">triggering</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Repeatedly</span><span class="o">.</span><span class="na">forever</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">AfterProcessingTime</span><span class="o">.</span><span class="na">pastFirstElementInPane</span><span class="o">().</span><span class="na">plusDelayOf</span><span class="o">(</span><span class="n">TEN_MINUTES</span><span class="o">)))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">accumulatingFiredPanes</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withAllowedLateness</span><span class="o">(</span><span class="n">allowedLateness</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Extract and sum username/score pairs from the event data. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;ExtractUserScore&#34;</span><span class="o">,</span> <span class="k">new</span> <span class="n">ExtractAndSumScore</span><span class="o">(</span><span class="s">&#34;user&#34;</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">CalculateUserScores</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">PTransform</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;&#34;&#34;Extract user/score pairs from the event stream using processing time, via |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> global windowing. Get periodic updates on all users&#39; running scores. |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> &#34;&#34;&#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">allowed_lateness</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># super().__init__()</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">PTransform</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">allowed_lateness_seconds</span> <span class="o">=</span> <span class="n">allowed_lateness</span> <span class="o">*</span> <span class="mi">60</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">expand</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pcoll</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># NOTE: the behavior does not exactly match the Java example</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># TODO: allowed_lateness not implemented yet in FixedWindows</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># TODO: AfterProcessingTime not implemented yet, replace AfterCount</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pcoll</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Get periodic results every ten events.</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;LeaderboardUserGlobalWindows&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">window</span><span class="o">.</span><span class="n">GlobalWindows</span><span class="p">(),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">trigger</span><span class="o">=</span><span class="n">trigger</span><span class="o">.</span><span class="n">Repeatedly</span><span class="p">(</span><span class="n">trigger</span><span class="o">.</span><span class="n">AfterCount</span><span class="p">(</span><span class="mi">10</span><span class="p">)),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">accumulation_mode</span><span class="o">=</span><span class="n">trigger</span><span class="o">.</span><span class="n">AccumulationMode</span><span class="o">.</span><span class="n">ACCUMULATING</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Extract and sum username/score pairs from the event data.</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;ExtractAndSumScore&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ExtractAndSumScore</span><span class="p">(</span><span class="s1">&#39;user&#39;</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p><code>LeaderBoard</code> sets the <a href="/documentation/programming-guide/#window-accumulation-modes">window accumulation mode</a> to accumulate window panes as the trigger fires. This accumulation mode is set by <span class="language-java">invoking <code>.accumulatingFiredPanes</code></span> <span class="language-py">using <code>accumulation_mode=trigger.AccumulationMode.ACCUMULATING</code></span> when setting the trigger, and causes the pipeline to accumulate the previously emitted data together with any new data that&rsquo;s arrived since the last trigger fire. This ensures that <code>LeaderBoard</code> is a running sum for the user scores, rather than a collection of individual sums.</p> |
| <h4 id="calculating-team-score-based-on-event-time">Calculating Team Score based on Event Time</h4> |
| <p>We want our pipeline to also output the total score for each team during each hour of play. Unlike the user score calculation, for team scores, we care about when in <em>event</em> time each score actually occurred, because we want to consider each hour of play individually. We also want to provide speculative updates as each individual hour progresses, and to allow any instances of late data — data that arrives after a given hour&rsquo;s data is considered complete — to be included in our calculation.</p> |
| <p>Because we consider each hour individually, we can apply fixed-time windowing to our input data, just like in <code>HourlyTeamScore</code>. To provide the speculative updates and updates on late data, we&rsquo;ll specify additional trigger parameters. The trigger will cause each window to calculate and emit results at an interval we specify (in this case, every five minutes), and also to keep triggering after the window is considered &ldquo;complete&rdquo; to account for late data. Just like the user score calculation, we&rsquo;ll set the trigger to accumulating mode to ensure that we get a running sum for each hour-long window.</p> |
| <p>The triggers for speculative updates and late data help with the problem of <a href="/documentation/programming-guide/#windowing">time skew</a>. Events in the pipeline aren&rsquo;t necessarily processed in the order in which they actually occurred according to their timestamps; they may arrive in the pipeline out of order, or late (in our case, because they were generated while the user&rsquo;s phone was out of contact with a network). Beam needs a way to determine when it can reasonably assume that it has &ldquo;all&rdquo; of the data in a given window: this is called the <em>watermark</em>.</p> |
| <p>In an ideal world, all data would be processed immediately when it occurs, so the processing time would be equal to (or at least have a linear relationship to) the event time. However, because distributed systems contain some inherent inaccuracy (like our late-reporting phones), Beam often uses a heuristic watermark.</p> |
| <p>The following diagram shows the relationship between ongoing processing time and each score&rsquo;s event time for two teams:</p> |
| <img src="/images/gaming-example-event-time-narrow.gif" alt="A pipeline processes score data by team, windowed by event time." width="800px"> |
| <p><em>Figure 5: Score data by team, windowed by event time. A trigger based on |
| processing time causes the window to emit speculative early results and include |
| late results.</em></p> |
| <p>The dotted line in the diagram is the &ldquo;ideal&rdquo; <strong>watermark</strong>: Beam&rsquo;s notion of when all data in a given window can reasonably be considered to have arrived. The irregular solid line represents the actual watermark, as determined by the data source.</p> |
| <p>Data arriving above the solid watermark line is <em>late data</em> — this is a score event that was delayed (perhaps generated offline) and arrived after the window to which it belongs had closed. Our pipeline&rsquo;s late-firing trigger ensures that this late data is still included in the sum.</p> |
| <p>The following code example shows how <code>LeaderBoard</code> applies fixed-time windowing with the appropriate triggers to have our pipeline perform the calculations we want:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Extract team/score pairs from the event stream, using hour-long windows by default. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="nd">@VisibleForTesting</span> |
| </span></span><span class="line"><span class="cl"><span class="kd">static</span> <span class="kd">class</span> <span class="nc">CalculateTeamScores</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">extends</span> <span class="n">PTransform</span><span class="o">&lt;</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">GameActionInfo</span><span class="o">&gt;,</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">private</span> <span class="kd">final</span> <span class="n">Duration</span> <span class="n">teamWindowDuration</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">private</span> <span class="kd">final</span> <span class="n">Duration</span> <span class="n">allowedLateness</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">CalculateTeamScores</span><span class="o">(</span><span class="n">Duration</span> <span class="n">teamWindowDuration</span><span class="o">,</span> <span class="n">Duration</span> <span class="n">allowedLateness</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">this</span><span class="o">.</span><span class="na">teamWindowDuration</span> <span class="o">=</span> <span class="n">teamWindowDuration</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">this</span><span class="o">.</span><span class="na">allowedLateness</span> <span class="o">=</span> <span class="n">allowedLateness</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="nf">expand</span><span class="o">(</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">GameActionInfo</span><span class="o">&gt;</span> <span class="n">infos</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">infos</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;LeaderboardTeamFixedWindows&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Window</span><span class="o">.&lt;</span><span class="n">GameActionInfo</span><span class="o">&gt;</span><span class="n">into</span><span class="o">(</span><span class="n">FixedWindows</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">teamWindowDuration</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// We will get early (speculative) results as well as cumulative |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// processing of late data. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">triggering</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">AfterWatermark</span><span class="o">.</span><span class="na">pastEndOfWindow</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withEarlyFirings</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">AfterProcessingTime</span><span class="o">.</span><span class="na">pastFirstElementInPane</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">plusDelayOf</span><span class="o">(</span><span class="n">FIVE_MINUTES</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withLateFirings</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">AfterProcessingTime</span><span class="o">.</span><span class="na">pastFirstElementInPane</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">plusDelayOf</span><span class="o">(</span><span class="n">TEN_MINUTES</span><span class="o">)))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withAllowedLateness</span><span class="o">(</span><span class="n">allowedLateness</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">accumulatingFiredPanes</span><span class="o">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Extract and sum teamname/score pairs from the event data. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;ExtractTeamScore&#34;</span><span class="o">,</span> <span class="k">new</span> <span class="n">ExtractAndSumScore</span><span class="o">(</span><span class="s">&#34;team&#34;</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">CalculateTeamScores</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">PTransform</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;&#34;&#34;Calculates scores for each team within the configured window duration. |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> Extract team/score pairs from the event stream, using hour-long windows by |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> default. |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> &#34;&#34;&#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">team_window_duration</span><span class="p">,</span> <span class="n">allowed_lateness</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># TODO(BEAM-6158): Revert the workaround once we can pickle super() on py3.</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># super().__init__()</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">PTransform</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">team_window_duration</span> <span class="o">=</span> <span class="n">team_window_duration</span> <span class="o">*</span> <span class="mi">60</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">allowed_lateness_seconds</span> <span class="o">=</span> <span class="n">allowed_lateness</span> <span class="o">*</span> <span class="mi">60</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">expand</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pcoll</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># NOTE: the behavior does not exactly match the Java example</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># TODO: allowed_lateness not implemented yet in FixedWindows</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># TODO: AfterProcessingTime not implemented yet, replace AfterCount</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pcoll</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># We will get early (speculative) results as well as cumulative</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># processing of late data.</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;LeaderboardTeamFixedWindows&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">window</span><span class="o">.</span><span class="n">FixedWindows</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">team_window_duration</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">trigger</span><span class="o">=</span><span class="n">trigger</span><span class="o">.</span><span class="n">AfterWatermark</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">trigger</span><span class="o">.</span><span class="n">AfterCount</span><span class="p">(</span><span class="mi">10</span><span class="p">),</span> <span class="n">trigger</span><span class="o">.</span><span class="n">AfterCount</span><span class="p">(</span><span class="mi">20</span><span class="p">)),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">accumulation_mode</span><span class="o">=</span><span class="n">trigger</span><span class="o">.</span><span class="n">AccumulationMode</span><span class="o">.</span><span class="n">ACCUMULATING</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Extract and sum teamname/score pairs from the event data.</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;ExtractAndSumScore&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ExtractAndSumScore</span><span class="p">(</span><span class="s1">&#39;team&#39;</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Taken together, these processing strategies let us address the latency and completeness issues present in the <code>UserScore</code> and <code>HourlyTeamScore</code> pipelines, while still using the same basic transforms to process the data—as a matter of fact, both calculations still use the same <code>ExtractAndSumScore</code> transform that we used in both the <code>UserScore</code> and <code>HourlyTeamScore</code> pipelines.</p> |
| <h2 id="gamestats-abuse-detection-and-usage-analysis">GameStats: Abuse Detection and Usage Analysis</h2> |
| <p>While <code>LeaderBoard</code> demonstrates how to use basic windowing and triggers to perform low-latency and flexible data analysis, we can use more advanced windowing techniques to perform more comprehensive analysis. This might include some calculations designed to detect system abuse (like spam) or to gain insight into user behavior. The <code>GameStats</code> pipeline builds on the low-latency functionality in <code>LeaderBoard</code> to demonstrate how you can use Beam to perform this kind of advanced analysis.</p> |
| <p>Like <code>LeaderBoard</code>, <code>GameStats</code> reads data from an unbounded source. It is best thought of as an ongoing job that provides insight into the game as users play.</p> |
| <p class="language-java"><blockquote> |
| <p><strong>Note:</strong> See <a href="https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/complete/game/GameStats.java">GameStats on GitHub</a> for the complete example pipeline program.</p> |
| </blockquote> |
| </p> |
| <p class="language-py"><blockquote> |
| <p><strong>Note:</strong> See <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/game/game_stats.py">GameStats on GitHub</a> for the complete example pipeline program.</p> |
| </blockquote> |
| </p> |
| <h3 id="what-does-gamestats-do">What Does GameStats Do?</h3> |
| <p>Like <code>LeaderBoard</code>, <code>GameStats</code> calculates the total score per team, per hour. However, the pipeline also performs two kinds of more complex analysis:</p> |
| <ul> |
| <li><code>GameStats</code> does <strong>abuse detection</strong> system that performs some simple statistical analysis on the score data to determine which users, if any, might be spammers or bots. It then uses the list of suspected spam/bot users to filter the bots out of the hourly team score calculation.</li> |
| <li><code>GameStats</code> <strong>analyzes usage patterns</strong> by grouping together game data that share similar event times using session windowing. This lets us gain some intelligence on how long users tend to play, and how game length changes over time.</li> |
| </ul> |
| <p>Below, we&rsquo;ll look at these features in more detail.</p> |
| <h4 id="abuse-detection">Abuse Detection</h4> |
| <p>Let&rsquo;s suppose scoring in our game depends on the speed at which a user can &ldquo;click&rdquo; on their phone. <code>GameStats</code>&rsquo;s abuse detection analyzes each user&rsquo;s score data to detect if a user has an abnormally high &ldquo;click rate&rdquo; and thus an abnormally high score. This might indicate that the game is being played by a bot that operates significantly faster than a human could play.</p> |
| <p>To determine whether or not a score is &ldquo;abnormally&rdquo; high, <code>GameStats</code> calculates the average of every score in that fixed-time window, and then checks each individual score against the average score multiplied by an arbitrary weight factor (in our case, 2.5). Thus, any score more than 2.5 times the average is deemed to be the product of spam. The <code>GameStats</code> pipeline tracks a list of &ldquo;spam&rdquo; users and filters those users out of the team score calculations for the team leader board.</p> |
| <p>Since the average depends on the pipeline data, we need to calculate it, and then use that calculated data in a subsequent <code>ParDo</code> transform that filters scores that exceed the weighted value. To do this, we can pass the calculated average to as a <a href="/documentation/programming-guide/#side-inputs">side input</a> to the filtering <code>ParDo</code>.</p> |
| <p>The following code example shows the composite transform that handles abuse detection. The transform uses the <code>Sum.integersPerKey</code> transform to sum all scores per user, and then the <code>Mean.globally</code> transform to determine the average score for all users. Once that&rsquo;s been calculated (as a <code>PCollectionView</code> singleton), we can pass it to the filtering <code>ParDo</code> using <code>.withSideInputs</code>:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">CalculateSpammyUsers</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">extends</span> <span class="n">PTransform</span><span class="o">&lt;</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;,</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="n">Logger</span> <span class="n">LOG</span> <span class="o">=</span> <span class="n">LoggerFactory</span><span class="o">.</span><span class="na">getLogger</span><span class="o">(</span><span class="n">CalculateSpammyUsers</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">double</span> <span class="n">SCORE_WEIGHT</span> <span class="o">=</span> <span class="n">2</span><span class="o">.</span><span class="na">5</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="nf">expand</span><span class="o">(</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">userScores</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Get the sum of scores for each user. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">sumScores</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">userScores</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;UserSum&#34;</span><span class="o">,</span> <span class="n">Sum</span><span class="o">.</span><span class="na">integersPerKey</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Extract the score from each element, and use it to find the global mean. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="kd">final</span> <span class="n">PCollectionView</span><span class="o">&lt;</span><span class="n">Double</span><span class="o">&gt;</span> <span class="n">globalMeanScore</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">sumScores</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Values</span><span class="o">.</span><span class="na">create</span><span class="o">()).</span><span class="na">apply</span><span class="o">(</span><span class="n">Mean</span><span class="o">.&lt;</span><span class="n">Integer</span><span class="o">&gt;</span><span class="n">globally</span><span class="o">().</span><span class="na">asSingletonView</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Filter the user sums using the global mean. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;</span> <span class="n">filtered</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">sumScores</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;ProcessAndFilter&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ParDo</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// use the derived mean total score as a side input |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">of</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;,</span> <span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">private</span> <span class="kd">final</span> <span class="n">Counter</span> <span class="n">numSpammerUsers</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Metrics</span><span class="o">.</span><span class="na">counter</span><span class="o">(</span><span class="s">&#34;main&#34;</span><span class="o">,</span> <span class="s">&#34;SpammerUsers&#34;</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Integer</span> <span class="n">score</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">getValue</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Double</span> <span class="n">gmc</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="na">sideInput</span><span class="o">(</span><span class="n">globalMeanScore</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="o">(</span><span class="n">score</span> <span class="o">&gt;</span> <span class="o">(</span><span class="n">gmc</span> <span class="o">*</span> <span class="n">SCORE_WEIGHT</span><span class="o">))</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">LOG</span><span class="o">.</span><span class="na">info</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;user &#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">+</span> <span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">getKey</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">+</span> <span class="s">&#34; spammer score &#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">+</span> <span class="n">score</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">+</span> <span class="s">&#34; with mean &#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">+</span> <span class="n">gmc</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">numSpammerUsers</span><span class="o">.</span><span class="na">inc</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">c</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">})</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withSideInputs</span><span class="o">(</span><span class="n">globalMeanScore</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">filtered</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">CalculateSpammyUsers</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">PTransform</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;&#34;&#34;Filter out all but those users with a high clickrate, which we will |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> consider as &#39;spammy&#39; uesrs. |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> We do this by finding the mean total score per user, then using that |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> information as a side input to filter out all but those user scores that are |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> larger than (mean * SCORE_WEIGHT). |
| </span></span></span><span class="line"><span class="cl"><span class="s2"> &#34;&#34;&#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">SCORE_WEIGHT</span> <span class="o">=</span> <span class="mf">2.5</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">expand</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">user_scores</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Get the sum of scores for each user.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">sum_scores</span> <span class="o">=</span> <span class="p">(</span><span class="n">user_scores</span> <span class="o">|</span> <span class="s1">&#39;SumUsersScores&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombinePerKey</span><span class="p">(</span><span class="nb">sum</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Extract the score from each element, and use it to find the global mean.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">global_mean_score</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">sum_scores</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Values</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">combiners</span><span class="o">.</span><span class="n">MeanCombineFn</span><span class="p">())</span>\ |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="n">as_singleton_view</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Filter the user sums using the global mean.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">filtered</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">sum_scores</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Use the derived mean total score (global_mean_score) as a side input.</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;ProcessAndFilter&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Filter</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">lambda</span> <span class="n">key_score</span><span class="p">,</span> <span class="n">global_mean</span><span class="p">:</span>\ |
| </span></span><span class="line"><span class="cl"> <span class="n">key_score</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">&gt;</span> <span class="n">global_mean</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">SCORE_WEIGHT</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">global_mean_score</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">filtered</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>The abuse-detection transform generates a view of users suspected to be spambots. Later in the pipeline, we use that view to filter out any such users when we calculate the team score per hour, again by using the side input mechanism. The following code example shows where we insert the spam filter, between windowing the scores into fixed windows and extracting the team scores:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Calculate the total score per team over fixed windows, |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// and emit cumulative updates for late data. Uses the side input derived above-- the set of |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// suspected robots-- to filter out scores from those users from the sum. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// Write the results to BigQuery. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">rawEvents</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;WindowIntoFixedWindows&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Window</span><span class="o">.</span><span class="na">into</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">FixedWindows</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="n">options</span><span class="o">.</span><span class="na">getFixedWindowDuration</span><span class="o">()))))</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Filter out the detected spammer users, using the side input derived above. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;FilterOutSpammers&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">GameActionInfo</span><span class="o">,</span> <span class="n">GameActionInfo</span><span class="o">&gt;()</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// If the user is not in the spammers Map, output the data element. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">if</span> <span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="na">sideInput</span><span class="o">(</span><span class="n">spammersView</span><span class="o">).</span><span class="na">get</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">getUser</span><span class="o">().</span><span class="na">trim</span><span class="o">())</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">c</span><span class="o">.</span><span class="na">output</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">})</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withSideInputs</span><span class="o">(</span><span class="n">spammersView</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Extract and sum teamname/score pairs from the event data. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;ExtractTeamScore&#34;</span><span class="o">,</span> <span class="k">new</span> <span class="n">ExtractAndSumScore</span><span class="o">(</span><span class="s">&#34;team&#34;</span><span class="o">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># Calculate the total score per team over fixed windows, and emit cumulative</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># updates for late data. Uses the side input derived above --the set of</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># suspected robots-- to filter out scores from those users from the sum.</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Write the results to BigQuery.</span> |
| </span></span><span class="line"><span class="cl"><span class="p">(</span> <span class="c1"># pylint: disable=expression-not-assigned</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">raw_events</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;WindowIntoFixedWindows&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">window</span><span class="o">.</span><span class="n">FixedWindows</span><span class="p">(</span><span class="n">fixed_window_duration</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Filter out the detected spammer users, using the side input derived</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># above</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;FilterOutSpammers&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Filter</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">lambda</span> <span class="n">elem</span><span class="p">,</span> <span class="n">spammers</span><span class="p">:</span> <span class="n">elem</span><span class="p">[</span><span class="s1">&#39;user&#39;</span><span class="p">]</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">spammers</span><span class="p">,</span> <span class="n">spammers_view</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Extract and sum teamname/score pairs from the event data.</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;ExtractAndSumScore&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ExtractAndSumScore</span><span class="p">(</span><span class="s1">&#39;team&#39;</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="analyzing-usage-patterns">Analyzing Usage Patterns</h4> |
| <p>We can gain some insight on when users are playing our game, and for how long, by examining the event times for each game score and grouping scores with similar event times into <em>sessions</em>. <code>GameStats</code> uses Beam&rsquo;s built-in <a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Sessions.java">session windowing</a> function to group user scores into sessions based on the time they occurred.</p> |
| <p>When you set session windowing, you specify a <em>minimum gap duration</em> between events. All events whose arrival times are closer together than the minimum gap duration are grouped into the same window. Events where the difference in arrival time is greater than the gap are grouped into separate windows. Depending on how we set our minimum gap duration, we can safely assume that scores in the same session window are part of the same (relatively) uninterrupted stretch of play. Scores in a different window indicate that the user stopped playing the game for at least the minimum gap time before returning to it later.</p> |
| <p>The following diagram shows how data might look when grouped into session windows. Unlike fixed windows, session windows are <em>different for each user</em> and is dependent on each individual user&rsquo;s play pattern:</p> |
| <p><img src="/images/gaming-example-session-windows.png" alt="User sessions with a minimum gap duration."></p> |
| <p><em>Figure 6: User sessions with a minimum gap duration. Each user has different |
| sessions, according to how many instances they play and how long their breaks |
| between instances are.</em></p> |
| <p>We can use the session-windowed data to determine the average length of uninterrupted play time for all of our users, as well as the total score they achieve during each session. We can do this in the code by first applying session windows, summing the score per user and session, and then using a transform to calculate the length of each individual session:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Detect user sessions-- that is, a burst of activity separated by a gap from further |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// activity. Find and record the mean session lengths. |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// This information could help the game designers track the changing user engagement |
| </span></span></span><span class="line"><span class="cl"><span class="c1">// as their set of games changes. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="n">userEvents</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;WindowIntoSessions&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Window</span><span class="o">.&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Integer</span><span class="o">&gt;&gt;</span><span class="n">into</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Sessions</span><span class="o">.</span><span class="na">withGapDuration</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="n">options</span><span class="o">.</span><span class="na">getSessionGap</span><span class="o">())))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withTimestampCombiner</span><span class="o">(</span><span class="n">TimestampCombiner</span><span class="o">.</span><span class="na">END_OF_WINDOW</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// For this use, we care only about the existence of the session, not any particular |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// information aggregated over it, so the following is an efficient way to do that. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Combine</span><span class="o">.</span><span class="na">perKey</span><span class="o">(</span><span class="n">x</span> <span class="o">-&gt;</span> <span class="n">0</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Get the duration per session. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;UserSessionActivity&#34;</span><span class="o">,</span> <span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">UserSessionInfoFn</span><span class="o">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># Detect user sessions-- that is, a burst of activity separated by a gap</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># from further activity. Find and record the mean session lengths.</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># This information could help the game designers track the changing user</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># engagement as their set of game changes.</span> |
| </span></span><span class="line"><span class="cl"><span class="p">(</span> <span class="c1"># pylint: disable=expression-not-assigned</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">user_events</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;WindowIntoSessions&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">window</span><span class="o">.</span><span class="n">Sessions</span><span class="p">(</span><span class="n">session_gap</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">timestamp_combiner</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">window</span><span class="o">.</span><span class="n">TimestampCombiner</span><span class="o">.</span><span class="n">OUTPUT_AT_EOW</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># For this use, we care only about the existence of the session, not any</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># particular information aggregated over it, so we can just group by key</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># and assign a &#34;dummy value&#34; of None.</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombinePerKey</span><span class="p">(</span><span class="k">lambda</span> <span class="n">_</span><span class="p">:</span> <span class="kc">None</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Get the duration of the session</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;UserSessionActivity&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">UserSessionActivity</span><span class="p">())</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>This gives us a set of user sessions, each with an attached duration. We can then calculate the <em>average</em> session length by re-windowing the data into fixed time windows, and then calculating the average for all sessions that end in each hour:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// Re-window to process groups of session sums according to when the sessions complete. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;WindowToExtractSessionMean&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Window</span><span class="o">.</span><span class="na">into</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">FixedWindows</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="n">options</span><span class="o">.</span><span class="na">getUserActivityWindowDuration</span><span class="o">()))))</span> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Find the mean session duration in each window. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Mean</span><span class="o">.&lt;</span><span class="n">Integer</span><span class="o">&gt;</span><span class="n">globally</span><span class="o">().</span><span class="na">withoutDefaults</span><span class="o">())</span> |
| </span></span><span class="line"><span class="cl"><span class="c1">// Write this info to a BigQuery table. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;WriteAvgSessionLength&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">WriteWindowedToBigQuery</span><span class="o">&lt;&gt;(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">options</span><span class="o">.</span><span class="na">as</span><span class="o">(</span><span class="n">GcpOptions</span><span class="o">.</span><span class="na">class</span><span class="o">).</span><span class="na">getProject</span><span class="o">(),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">options</span><span class="o">.</span><span class="na">getDataset</span><span class="o">(),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">options</span><span class="o">.</span><span class="na">getGameStatsTablePrefix</span><span class="o">()</span> <span class="o">+</span> <span class="s">&#34;_sessions&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">configureSessionWindowWrite</span><span class="o">()));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># Re-window to process groups of session sums according to when the</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># sessions complete</span> |
| </span></span><span class="line"><span class="cl"><span class="o">|</span> <span class="s1">&#39;WindowToExtractSessionMean&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">window</span><span class="o">.</span><span class="n">FixedWindows</span><span class="p">(</span><span class="n">user_activity_window_duration</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># Find the mean session duration in each window</span> |
| </span></span><span class="line"><span class="cl"><span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">combiners</span><span class="o">.</span><span class="n">MeanCombineFn</span><span class="p">())</span><span class="o">.</span><span class="n">without_defaults</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"><span class="o">|</span> <span class="s1">&#39;FormatAvgSessionLength&#39;</span> <span class="o">&gt;&gt;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">elem</span><span class="p">:</span> <span class="p">{</span><span class="s1">&#39;mean_duration&#39;</span><span class="p">:</span> <span class="nb">float</span><span class="p">(</span><span class="n">elem</span><span class="p">)})</span> |
| </span></span><span class="line"><span class="cl"><span class="o">|</span> <span class="s1">&#39;WriteAvgSessionLength&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">WriteToBigQuery</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">args</span><span class="o">.</span><span class="n">table_name</span> <span class="o">+</span> <span class="s1">&#39;_sessions&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">args</span><span class="o">.</span><span class="n">dataset</span><span class="p">,</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;mean_duration&#39;</span><span class="p">:</span> <span class="s1">&#39;FLOAT&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">options</span><span class="o">.</span><span class="n">view_as</span><span class="p">(</span><span class="n">GoogleCloudOptions</span><span class="p">)</span><span class="o">.</span><span class="n">project</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>We can use the resulting information to find, for example, what times of day our users are playing the longest, or which stretches of the day are more likely to see shorter play sessions.</p> |
| <h2 id="next-steps">Next Steps</h2> |
| <ul> |
| <li>Take a self-paced tour through our <a href="/documentation/resources/learning-resources">Learning Resources</a>.</li> |
| <li>Dive in to some of our favorite <a href="/get-started/resources/videos-and-podcasts">Videos and Podcasts</a>.</li> |
| <li>Join the Beam <a href="/community/contact-us">users@</a> mailing list.</li> |
| </ul> |
| <p>Please don&rsquo;t hesitate to <a href="/community/contact-us">reach out</a> if you encounter any issues!</p></description></item><item><title>Get-Started: Beam Overview</title><link>/get-started/beam-overview/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/beam-overview/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="apache-beam-overview">Apache Beam Overview</h1> |
| <p>Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the pipeline. The pipeline is then executed by one of Beam&rsquo;s supported <strong>distributed processing back-ends</strong>, which include <a href="https://flink.apache.org">Apache Flink</a>, <a href="https://spark.apache.org">Apache Spark</a>, and <a href="https://cloud.google.com/dataflow">Google Cloud Dataflow</a>.</p> |
| <p>Beam is particularly useful for <a href="https://en.wikipedia.org/wiki/Embarassingly_parallel">embarrassingly parallel</a> data processing tasks, in which the problem can be decomposed into many smaller bundles of data that can be processed independently and in parallel. You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data integration. These tasks are useful for moving data between different storage media and data sources, transforming data into a more desirable format, or loading data onto a new system.</p> |
| <div style="display: flex; justify-content: center"> |
| <img src="/images/learner_graph.png" width="800px" alt="Learner Graph"> |
| </div> |
| <h2 id="apache-beam-sdks">Apache Beam SDKs</h2> |
| <p>The Beam SDKs provide a unified programming model that can represent and transform data sets of any size, whether the input is a finite data set from a batch data source, or an infinite data set from a streaming data source. The Beam SDKs use the same classes to represent both bounded and unbounded data, and the same transforms to operate on that data. You use the Beam SDK of your choice to build a program that defines your data processing pipeline.</p> |
| <p>Beam currently supports the following language-specific SDKs:</p> |
| <ul> |
| <li><a href="/documentation/sdks/java">Apache Beam Java SDK</a> <img src="/images/logos/sdks/java.png" alt="Java logo"></li> |
| <li><a href="/documentation/sdks/python">Apache Beam Python SDK</a> <img src="/images/logos/sdks/python.png" alt="Python logo"></li> |
| <li><a href="/documentation/sdks/go">Apache Beam Go SDK</a> <img src="/images/logos/sdks/go.png" height="45px" alt="Go logo"></li> |
| </ul> |
| <p>A Scala <img src="/images/logos/sdks/scala.png" height="45px" alt="Scala logo"> interface is also available as <a href="https://github.com/spotify/scio">Scio</a>.</p> |
| <h2 id="apache-beam-pipeline-runners">Apache Beam Pipeline Runners</h2> |
| <p>The Beam Pipeline Runners translate the data processing pipeline you define with your Beam program into the API compatible with the distributed processing back-end of your choice. When you run your Beam program, you&rsquo;ll need to specify an <a href="/documentation/runners/capability-matrix">appropriate runner</a> for the back-end where you want to execute your pipeline.</p> |
| <p>Beam currently supports the following runners:</p> |
| <ul> |
| <li><a href="/documentation/runners/direct">Direct Runner</a></li> |
| <li><a href="/documentation/runners/flink">Apache Flink Runner</a> <img src="/images/logos/runners/flink.png" height="50px" alt="Apache Flink logo"></li> |
| <li><a href="/documentation/runners/nemo">Apache Nemo Runner</a></li> |
| <li><a href="/documentation/runners/samza">Apache Samza Runner</a> <img src="/images/logos/runners/samza.png" height="40px" alt="Apache Samza logo"></li> |
| <li><a href="/documentation/runners/spark">Apache Spark Runner</a> <img src="/images/logos/runners/spark.png" height="50px" alt="Apache Spark logo"></li> |
| <li><a href="/documentation/runners/dataflow">Google Cloud Dataflow Runner</a> <img src="/images/logos/runners/dataflow.png" height="50px" alt="Google Cloud Dataflow logo"></li> |
| <li><a href="/documentation/runners/jet">Hazelcast Jet Runner</a> <img src="/images/logos/runners/jet.png" height="40px" alt="Hazelcast Jet logo"></li> |
| <li><a href="/documentation/runners/twister2">Twister2 Runner</a> <img src="/images/logos/runners/twister2.png" height="50px" alt="Twister2 logo"></li> |
| </ul> |
| <p><strong>Note:</strong> You can always execute your pipeline locally for testing and debugging purposes.</p> |
| <h2 id="get-started">Get Started</h2> |
| <p>Get started using Beam for your data processing tasks.</p> |
| <blockquote> |
| <p>If you already know <a href="https://spark.apache.org/">Apache Spark</a>, |
| check our <a href="/get-started/from-spark">Getting started from Apache Spark</a> page.</p> |
| </blockquote> |
| <ol> |
| <li> |
| <p>Take the <a href="https://tour.beam.apache.org/">Tour of Beam</a> as an online interactive learning experience.</p> |
| </li> |
| <li> |
| <p>Follow the Quickstart for the <a href="/get-started/quickstart-java">Java SDK</a>, the <a href="/get-started/quickstart-py">Python SDK</a>, or the <a href="/get-started/quickstart-go">Go SDK</a>.</p> |
| </li> |
| <li> |
| <p>See the <a href="/get-started/wordcount-example">WordCount Examples Walkthrough</a> for examples that introduce various features of the SDKs.</p> |
| </li> |
| <li> |
| <p>Take a self-paced tour through our <a href="/documentation/resources/learning-resources">Learning Resources</a>.</p> |
| </li> |
| <li> |
| <p>Dive into the <a href="/documentation/">Documentation</a> section for in-depth concepts and reference materials for the Beam model, SDKs, and runners.</p> |
| </li> |
| <li> |
| <p>Dive into the <a href="https://github.com/GoogleCloudPlatform/dataflow-cookbook">cookbook examples</a> for learning how to run Beam on Dataflow.</p> |
| </li> |
| </ol> |
| <h2 id="contribute">Contribute</h2> |
| <p>Beam is an <a href="https://www.apache.org" target="_blank">Apache Software Foundation</a> project, available under the Apache v2 license. Beam is an open source community and contributions are greatly appreciated! If you&rsquo;d like to contribute, please see the <a href="/contribute/">Contribute</a> section.</p></description></item><item><title>Get-Started: Beam Quickstart for Go</title><link>/get-started/quickstart-go/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/quickstart-go/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="wordcount-quickstart-for-go">WordCount quickstart for Go</h1> |
| <p>This Quickstart will walk you through executing your first Beam pipeline to run <a href="/get-started/wordcount-example">WordCount</a>, written using Beam&rsquo;s <a href="/documentation/sdks/go">Go SDK</a>, on a <a href="/documentation#runners">runner</a> of your choice.</p> |
| <p>If you&rsquo;re interested in contributing to the Apache Beam Go codebase, see the <a href="/contribute">Contribution Guide</a>.</p> |
| <nav id="TableOfContents"> |
| <ul> |
| <li><a href="#set-up-your-environment">Set up your environment</a></li> |
| <li><a href="#run-wordcount">Run wordcount</a></li> |
| <li><a href="#next-steps">Next Steps</a></li> |
| </ul> |
| </nav> |
| <h2 id="set-up-your-environment">Set up your environment</h2> |
| <p>The Beam SDK for Go requires <code>go</code> version 1.20 or newer. It can be downloaded <a href="https://golang.org/">here</a>. Check what go version you have by running:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>go version</code></pre> |
| </div> |
| </div> |
| <p>If you are unfamiliar with Go, see the <a href="https://go.dev/doc/tutorial/getting-started">Get Started With Go Tutorial</a>.</p> |
| <h2 id="run-wordcount">Run wordcount</h2> |
| <p>The Apache Beam |
| <a href="https://github.com/apache/beam/tree/master/sdks/go/examples">examples</a> |
| directory has many examples. All examples can be run by passing the |
| required arguments described in the examples.</p> |
| <p>For example, to run <code>wordcount</code>, run:</p> |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">go run github.com/apache/beam/sdks/v2/go/examples/wordcount@latest --input &#34;gs://apache-beam-samples/shakespeare/kinglear.txt&#34; --output counts |
| less counts</code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow">go run github.com/apache/beam/sdks/v2/go/examples/wordcount@latest --input gs://dataflow-samples/shakespeare/kinglear.txt \ |
| --output gs://&lt;your-gcs-bucket&gt;/counts \ |
| --runner dataflow \ |
| --project your-gcp-project \ |
| --region your-gcp-region \ |
| --temp_location gs://&lt;your-gcs-bucket&gt;/tmp/ \ |
| --staging_location gs://&lt;your-gcs-bucket&gt;/binaries/</code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark"># Build and run the Spark job server from Beam source. |
| # -PsparkMasterUrl is optional. If it is unset the job will be run inside an embedded Spark cluster. |
| ./gradlew :runners:spark:3:job-server:runShadow -PsparkMasterUrl=spark://localhost:7077 |
| # In a separate terminal, run: |
| go run github.com/apache/beam/sdks/v2/go/examples/wordcount@latest --input &lt;PATH_TO_INPUT_FILE&gt; \ |
| --output counts \ |
| --runner spark \ |
| --endpoint localhost:8099</code></pre> |
| </div> |
| </div> |
| <h2 id="next-steps">Next Steps</h2> |
| <ul> |
| <li>Learn more about the <a href="/documentation/sdks/go/">Beam SDK for Go</a> |
| and look through the <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam">godoc</a>.</li> |
| <li>Walk through these WordCount examples in the <a href="/get-started/wordcount-example">WordCount Example Walkthrough</a>.</li> |
| <li>Clone the <a href="https://github.com/apache/beam-starter-go">Beam Go starter project</a>.</li> |
| <li>Take a self-paced tour through our <a href="/documentation/resources/learning-resources">Learning Resources</a>.</li> |
| <li>Dive in to some of our favorite <a href="/get-started/resources/videos-and-podcasts">Videos and Podcasts</a>.</li> |
| <li>Join the Beam <a href="/community/contact-us">users@</a> mailing list.</li> |
| </ul> |
| <p>Please don&rsquo;t hesitate to <a href="/community/contact-us">reach out</a> if you encounter any issues!</p></description></item><item><title>Get-Started: Beam Quickstart for Go</title><link>/get-started/quickstart/go/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/quickstart/go/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="apache-beam-go-sdk-quickstart">Apache Beam Go SDK quickstart</h1> |
| <p>This quickstart shows you how to run an |
| <a href="https://github.com/apache/beam-starter-go">example pipeline</a> written with the |
| <a href="/documentation/sdks/go">Apache Beam Go SDK</a>, using the |
| <a href="/documentation/runners/direct/">Direct Runner</a>. The Direct Runner executes |
| pipelines locally on your machine.</p> |
| <p>If you&rsquo;re interested in contributing to the Apache Beam Go codebase, see the |
| <a href="/contribute">Contribution Guide</a>.</p> |
| <p>On this page:</p> |
| <nav id="TableOfContents"> |
| <ul> |
| <li><a href="#set-up-your-development-environment">Set up your development environment</a></li> |
| <li><a href="#clone-the-github-repository">Clone the GitHub repository</a></li> |
| <li><a href="#run-the-quickstart">Run the quickstart</a></li> |
| <li><a href="#explore-the-code">Explore the code</a> |
| <ul> |
| <li><a href="#create-a-pipeline">Create a pipeline</a></li> |
| <li><a href="#create-an-initial-pcollection">Create an initial PCollection</a></li> |
| <li><a href="#apply-transforms-to-the-pcollection">Apply transforms to the PCollection</a></li> |
| <li><a href="#run-the-pipeline">Run the pipeline</a></li> |
| </ul> |
| </li> |
| <li><a href="#next-steps">Next Steps</a></li> |
| </ul> |
| </nav> |
| <h2 id="set-up-your-development-environment">Set up your development environment</h2> |
| <p>Make sure you have a <a href="https://go.dev/">Go</a> development environment ready. If |
| not, follow the instructions in the |
| <a href="https://go.dev/doc/install">Download and install</a> page.</p> |
| <h2 id="clone-the-github-repository">Clone the GitHub repository</h2> |
| <p>Clone or download the |
| <a href="https://github.com/apache/beam-starter-go">apache/beam-starter-go</a> GitHub |
| repository and change into the <code>beam-starter-go</code> directory.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>git clone https://github.com/apache/beam-starter-go.git |
| cd beam-starter-go</code></pre> |
| </div> |
| </div> |
| <h2 id="run-the-quickstart">Run the quickstart</h2> |
| <p>Run the following command:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>go run main.go --input-text=&#34;Greetings&#34;</code></pre> |
| </div> |
| </div> |
| <p>The output is similar to the following:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>Hello |
| World! |
| Greetings</code></pre> |
| </div> |
| </div> |
| <p>The lines might appear in a different order.</p> |
| <h2 id="explore-the-code">Explore the code</h2> |
| <p>The main code file for this quickstart is <strong>main.go</strong> |
| (<a href="https://github.com/apache/beam-starter-go/blob/main/main.go">GitHub</a>). |
| The code performs the following steps:</p> |
| <ol> |
| <li>Create a Beam pipeline.</li> |
| <li>Create an initial <code>PCollection</code>.</li> |
| <li>Apply transforms.</li> |
| <li>Run the pipeline, using the Direct Runner.</li> |
| </ol> |
| <h3 id="create-a-pipeline">Create a pipeline</h3> |
| <p>Before creating a pipeline, call the <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam#Init"><code>Init</code></a> function:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">beam</span><span class="p">.</span><span class="nf">Init</span><span class="p">()</span> |
| </span></span></code></pre></div><p>Then create the pipeline:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">pipeline</span><span class="p">,</span> <span class="nx">scope</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">NewPipelineWithRoot</span><span class="p">()</span> |
| </span></span></code></pre></div><p>The <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam#NewPipelineWithRoot"><code>NewPipelineWithRoot</code></a> function returns a new |
| <code>Pipeline</code> object, along with the pipeline&rsquo;s root scope. A <em>scope</em> is a |
| hierarchical grouping for composite transforms.</p> |
| <h3 id="create-an-initial-pcollection">Create an initial PCollection</h3> |
| <p>The <code>PCollection</code> abstraction represents a potentially distributed, |
| multi-element data set. A Beam pipeline needs a source of data to populate an |
| initial <code>PCollection</code>. The source can be bounded (with a known, fixed size) or |
| unbounded (with unlimited size).</p> |
| <p>This example uses the <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam#Create"><code>Create</code></a> function to create a <code>PCollection</code> |
| from an in-memory array of strings. The resulting <code>PCollection</code> contains the |
| strings &ldquo;hello&rdquo;, &ldquo;world!&rdquo;, and a user-provided input string.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">elements</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">Create</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="s">&#34;hello&#34;</span><span class="p">,</span> <span class="s">&#34;world!&#34;</span><span class="p">,</span> <span class="nx">input_text</span><span class="p">)</span> |
| </span></span></code></pre></div><h3 id="apply-transforms-to-the-pcollection">Apply transforms to the PCollection</h3> |
| <p>Transforms can change, filter, group, analyze, or otherwise process the |
| elements in a <code>PCollection</code>.</p> |
| <p>This example adds a <a href="/documentation/programming-guide/#pardo">ParDo</a> transform |
| to convert the input strings to title case:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">elements</span> <span class="p">=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">scope</span><span class="p">,</span> <span class="nx">strings</span><span class="p">.</span><span class="nx">Title</span><span class="p">,</span> <span class="nx">elements</span><span class="p">)</span> |
| </span></span></code></pre></div><p>The <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam#ParDo"><code>ParDo</code></a> function takes the parent scope, a transform function that |
| will be applied to the data, and the input PCollection. It returns the output |
| PCollection.</p> |
| <p>The previous example uses the built-in <a href="https://pkg.go.dev/strings#Title"><code>strings.Title</code></a> function for |
| the transform. You can also provide an application-defined function to a ParDo. |
| For example:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">logAndEmit</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">element</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beamLog</span><span class="p">.</span><span class="nf">Infoln</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">element</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emit</span><span class="p">(</span><span class="nx">element</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span></code></pre></div><p>This function logs the input element and returns the same element unmodified. |
| Create a ParDo for this function as follows:</p> |
| <pre tabindex="0"><code>beam.ParDo(scope, logAndEmit, elements) |
| </code></pre><p>At runtime, the ParDo will call the <code>logAndEmit</code> function on each element in |
| the input collection.</p> |
| <h3 id="run-the-pipeline">Run the pipeline</h3> |
| <p>The code shown in the previous sections defines a pipeline, but does not |
| process any data yet. To process data, you run the pipeline:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">beamx</span><span class="p">.</span><span class="nf">Run</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="nx">pipeline</span><span class="p">)</span> |
| </span></span></code></pre></div><p>A Beam <a href="https://beam.apache.org/documentation/basics/#runner">runner</a> runs a |
| Beam pipeline on a specific platform. This example uses the Direct Runner, |
| which is the default runner if you don&rsquo;t specify one. The Direct Runner runs |
| the pipeline locally on your machine. It is meant for testing and development, |
| rather than being optimized for efficiency. For more information, see |
| <a href="https://beam.apache.org/documentation/runners/direct/">Using the Direct Runner</a>.</p> |
| <p>For production workloads, you typically use a distributed runner that runs the |
| pipeline on a big data processing system such as Apache Flink, Apache Spark, or |
| Google Cloud Dataflow. These systems support massively parallel processing.</p> |
| <h2 id="next-steps">Next Steps</h2> |
| <ul> |
| <li>Learn more about the <a href="/documentation/sdks/go/">Beam SDK for Go</a> |
| and look through the |
| <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam">Go SDK API reference</a>.</li> |
| <li>Take a self-paced tour through our |
| <a href="/documentation/resources/learning-resources">Learning Resources</a>.</li> |
| <li>Dive in to some of our favorite |
| <a href="/get-started/resources/videos-and-podcasts">Videos and Podcasts</a>.</li> |
| <li>Join the Beam <a href="/community/contact-us">users@</a> mailing list.</li> |
| </ul> |
| <p>Please don&rsquo;t hesitate to <a href="/community/contact-us">reach out</a> if you encounter any |
| issues!</p></description></item><item><title>Get-Started: Beam Quickstart for Java</title><link>/get-started/quickstart/java/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/quickstart/java/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="apache-beam-java-sdk-quickstart">Apache Beam Java SDK quickstart</h1> |
| <p>This quickstart shows you how to run an |
| <a href="https://github.com/apache/beam-starter-java">example pipeline</a> written with |
| the <a href="/documentation/sdks/java">Apache Beam Java SDK</a>, using the |
| <a href="/documentation/runners/direct/">Direct Runner</a>. The Direct Runner executes |
| pipelines locally on your machine.</p> |
| <p>If you&rsquo;re interested in contributing to the Apache Beam Java codebase, see the |
| <a href="/contribute">Contribution Guide</a>.</p> |
| <p>On this page:</p> |
| <nav id="TableOfContents"> |
| <ul> |
| <li><a href="#set-up-your-development-environment">Set up your development environment</a></li> |
| <li><a href="#clone-the-github-repository">Clone the GitHub repository</a></li> |
| <li><a href="#run-the-quickstart">Run the quickstart</a></li> |
| <li><a href="#explore-the-code">Explore the code</a> |
| <ul> |
| <li><a href="#create-a-pipeline">Create a pipeline</a></li> |
| <li><a href="#create-an-initial-pcollection">Create an initial PCollection</a></li> |
| <li><a href="#apply-a-transform-to-the-pcollection">Apply a transform to the PCollection</a></li> |
| <li><a href="#run-the-pipeline">Run the pipeline</a></li> |
| </ul> |
| </li> |
| <li><a href="#next-steps">Next Steps</a></li> |
| </ul> |
| </nav> |
| <h2 id="set-up-your-development-environment">Set up your development environment</h2> |
| <p>Use <a href="https://sdkman.io/"><code>sdkman</code></a> to install the Java Development Kit (JDK).</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code># Install sdkman |
| curl -s &#34;https://get.sdkman.io&#34; | bash |
| # Install Java 17 |
| sdk install java 17.0.5-tem</code></pre> |
| </div> |
| </div> |
| <p>You can use either <a href="https://gradle.org/">Gradle</a> or |
| <a href="https://maven.apache.org/">Apache Maven</a> to run this quickstart:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code># Install Gradle |
| sdk install gradle |
| # Install Maven |
| sdk install maven</code></pre> |
| </div> |
| </div> |
| <h2 id="clone-the-github-repository">Clone the GitHub repository</h2> |
| <p>Clone or download the |
| <a href="https://github.com/apache/beam-starter-java">apache/beam-starter-java</a> GitHub |
| repository and change into the <code>beam-starter-java</code> directory.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>git clone https://github.com/apache/beam-starter-java.git |
| cd beam-starter-java</code></pre> |
| </div> |
| </div> |
| <h2 id="run-the-quickstart">Run the quickstart</h2> |
| <p><strong>Gradle</strong>: To run the quickstart with Gradle, run the following command:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>gradle run --args=&#39;--inputText=Greetings&#39;</code></pre> |
| </div> |
| </div> |
| <p><strong>Maven</strong>: To run the quickstart with Maven, run the following command:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>mvn compile exec:java -Dexec.args=--inputText=&#39;Greetings&#39;</code></pre> |
| </div> |
| </div> |
| <p>The output is similar to the following:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>Hello |
| World! |
| Greetings</code></pre> |
| </div> |
| </div> |
| <p>The lines might appear in a different order.</p> |
| <h2 id="explore-the-code">Explore the code</h2> |
| <p>The main code file for this quickstart is <strong>App.java</strong> |
| (<a href="https://github.com/apache/beam-starter-java/blob/main/src/main/java/com/example/App.java">GitHub</a>). |
| The code performs the following steps:</p> |
| <ol> |
| <li>Create a Beam pipeline.</li> |
| <li>Create an initial <code>PCollection</code>.</li> |
| <li>Apply a transform to the <code>PCollection</code>.</li> |
| <li>Run the pipeline, using the Direct Runner.</li> |
| </ol> |
| <h3 id="create-a-pipeline">Create a pipeline</h3> |
| <p>The code first creates a <code>Pipeline</code> object. The <code>Pipeline</code> object builds up the |
| graph of transformations to be executed.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">var</span> <span class="n">options</span> <span class="o">=</span> <span class="n">PipelineOptionsFactory</span><span class="o">.</span><span class="na">fromArgs</span><span class="o">(</span><span class="n">args</span><span class="o">).</span><span class="na">withValidation</span><span class="o">().</span><span class="na">as</span><span class="o">(</span><span class="n">Options</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"><span class="n">var</span> <span class="n">pipeline</span> <span class="o">=</span> <span class="n">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span> |
| </span></span></code></pre></div><p>The <code>PipelineOptions</code> object lets you set various options for the pipeline. The |
| <code>fromArgs</code> method shown in this example parses command-line arguments, which |
| lets you set pipeline options through the command line.</p> |
| <h3 id="create-an-initial-pcollection">Create an initial PCollection</h3> |
| <p>The <code>PCollection</code> abstraction represents a potentially distributed, |
| multi-element data set. A Beam pipeline needs a source of data to populate an |
| initial <code>PCollection</code>. The source can be bounded (with a known, fixed size) or |
| unbounded (with unlimited size).</p> |
| <p>This example uses the |
| <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Create.html"><code>Create.of</code></a> |
| method to create a <code>PCollection</code> from an in-memory array of strings. The |
| resulting <code>PCollection</code> contains the strings &ldquo;Hello&rdquo;, &ldquo;World!&rdquo;, and a |
| user-provided input string.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="k">return</span> <span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;Create elements&#34;</span><span class="o">,</span> <span class="n">Create</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="s">&#34;Hello&#34;</span><span class="o">,</span> <span class="s">&#34;World!&#34;</span><span class="o">,</span> <span class="n">inputText</span><span class="o">)))</span> |
| </span></span></code></pre></div><h3 id="apply-a-transform-to-the-pcollection">Apply a transform to the PCollection</h3> |
| <p>Transforms can change, filter, group, analyze, or otherwise process the |
| elements in a <code>PCollection</code>. This example uses the |
| <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/MapElements.html"><code>MapElements</code></a> |
| transform, which maps the elements of a collection into a new collection:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;Print elements&#34;</span><span class="o">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">MapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">()).</span><span class="na">via</span><span class="o">(</span><span class="n">x</span> <span class="o">-&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">x</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">x</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}));</span> |
| </span></span></code></pre></div><p>where</p> |
| <ul> |
| <li><code>into</code> specifies the data type for the elements in the output collection.</li> |
| <li><code>via</code> defines a mapping function that is called on each element of the input |
| collection to create the output collection.</li> |
| </ul> |
| <p>In this example, the mapping function is a lambda that just returns the |
| original value. It also prints the value to <code>System.out</code> as a side effect.</p> |
| <h3 id="run-the-pipeline">Run the pipeline</h3> |
| <p>The code shown in the previous sections defines a pipeline, but does not |
| process any data yet. To process data, you run the pipeline:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">pipeline</span><span class="o">.</span><span class="na">run</span><span class="o">().</span><span class="na">waitUntilFinish</span><span class="o">();</span> |
| </span></span></code></pre></div><p>A Beam <a href="/documentation/basics/#runner">runner</a> runs a |
| Beam pipeline on a specific platform. This example uses the |
| <a href="https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/runners/direct/DirectRunner.html">Direct Runner</a>, |
| which is the default runner if you don&rsquo;t specify one. The Direct Runner runs |
| the pipeline locally on your machine. It is meant for testing and development, |
| rather than being optimized for efficiency. For more information, see |
| <a href="/documentation/runners/direct/">Using the Direct Runner</a>.</p> |
| <p>For production workloads, you typically use a distributed runner that runs the |
| pipeline on a big data processing system such as Apache Flink, Apache Spark, or |
| Google Cloud Dataflow. These systems support massively parallel processing.</p> |
| <h2 id="next-steps">Next Steps</h2> |
| <ul> |
| <li>Learn more about the <a href="/documentation/sdks/java/">Beam SDK for Java</a> |
| and look through the |
| <a href="https://beam.apache.org/releases/javadoc">Java SDK API reference</a>.</li> |
| <li>Take a self-paced tour through our |
| <a href="/documentation/resources/learning-resources">Learning Resources</a>.</li> |
| <li>Dive in to some of our favorite |
| <a href="/get-started/resources/videos-and-podcasts">Videos and Podcasts</a>.</li> |
| <li>Join the Beam <a href="/community/contact-us">users@</a> mailing list.</li> |
| </ul> |
| <p>Please don&rsquo;t hesitate to <a href="/community/contact-us">reach out</a> if you encounter any |
| issues!</p></description></item><item><title>Get-Started: Beam Quickstart for Python</title><link>/get-started/quickstart/python/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/quickstart/python/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="apache-beam-python-sdk-quickstart">Apache Beam Python SDK quickstart</h1> |
| <p>This quickstart shows you how to run an |
| <a href="https://github.com/apache/beam-starter-python">example pipeline</a> written with |
| the <a href="/documentation/sdks/python">Apache Beam Python SDK</a>, using the |
| <a href="/documentation/runners/direct/">Direct Runner</a>. The Direct Runner executes |
| pipelines locally on your machine.</p> |
| <p>If you&rsquo;re interested in contributing to the Apache Beam Python codebase, see the |
| <a href="/contribute">Contribution Guide</a>.</p> |
| <p>On this page:</p> |
| <nav id="TableOfContents"> |
| <ul> |
| <li><a href="#set-up-your-development-environment">Set up your development environment</a></li> |
| <li><a href="#clone-the-github-repository">Clone the GitHub repository</a></li> |
| <li><a href="#create-and-activate-a-virtual-environment">Create and activate a virtual environment</a></li> |
| <li><a href="#install-the-project-dependences">Install the project dependences</a></li> |
| <li><a href="#run-the-quickstart">Run the quickstart</a></li> |
| <li><a href="#explore-the-code">Explore the code</a> |
| <ul> |
| <li><a href="#create-a-pipeline">Create a pipeline</a></li> |
| <li><a href="#create-an-initial-pcollection">Create an initial PCollection</a></li> |
| <li><a href="#apply-a-transform-to-the-pcollection">Apply a transform to the PCollection</a></li> |
| <li><a href="#run-the-pipeline">Run the pipeline</a></li> |
| </ul> |
| </li> |
| <li><a href="#next-steps">Next Steps</a></li> |
| </ul> |
| </nav> |
| <h2 id="set-up-your-development-environment">Set up your development environment</h2> |
| <p>Apache Beam aims to work on released |
| <a href="https://devguide.python.org/versions/">Python versions</a> that have not yet |
| reached end of life, but it may take a few releases until Apache Beam fully |
| supports the most recently released Python minor version.</p> |
| <p>The minimum required Python version is listed in the <strong>Meta</strong> section of the |
| <a href="https://pypi.org/project/apache-beam/">apache-beam</a> project page under |
| <strong>Requires</strong>. The list of all supported Python versions is listed in the |
| <strong>Classifiers</strong> section at the bottom of the page, under <strong>Programming |
| Language</strong>.</p> |
| <p>Check your Python version by running:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>python3 --version</code></pre> |
| </div> |
| </div> |
| <p>If you don&rsquo;t have a Python interpreter, you can download and install it from |
| the <a href="https://devguide.python.org/versions/">Python downloads</a> page.</p> |
| <p>If you need to install a different version of Python in addition to the version |
| that you already have, you can find some recommendations in our |
| <a href="https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-InstallingPythoninterpreters">Developer Wiki</a>.</p> |
| <h2 id="clone-the-github-repository">Clone the GitHub repository</h2> |
| <p>Clone or download the |
| <a href="https://github.com/apache/beam-starter-python">apache/beam-starter-python</a> |
| GitHub repository and change into the <code>beam-starter-python</code> directory.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>git clone https://github.com/apache/beam-starter-python.git |
| cd beam-starter-python</code></pre> |
| </div> |
| </div> |
| <h2 id="create-and-activate-a-virtual-environment">Create and activate a virtual environment</h2> |
| <p>A virtual environment is a directory tree containing its own Python |
| distribution. We recommend using a virtual environment so that all dependencies |
| of your project are installed in an isolated and self-contained environment. To |
| set up a virtual environment, run the following commands:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code># Create a new Python virtual environment. |
| python3 -m venv env |
| # Activate the virtual environment. |
| source env/bin/activate</code></pre> |
| </div> |
| </div> |
| <p>If these commands do not work on your platform, see the |
| <a href="https://docs.python.org/3/library/venv.html#how-venvs-work"><code>venv</code></a> |
| documentation.</p> |
| <h2 id="install-the-project-dependences">Install the project dependences</h2> |
| <p>Run the following command to install the project&rsquo;s dependencies from the |
| <code>requirements.txt</code> file:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>pip install -e .</code></pre> |
| </div> |
| </div> |
| <h2 id="run-the-quickstart">Run the quickstart</h2> |
| <p>Run the following command:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>python main.py --input-text=&#34;Greetings&#34;</code></pre> |
| </div> |
| </div> |
| <p>The output is similar to the following:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>Hello |
| World! |
| Greetings</code></pre> |
| </div> |
| </div> |
| <p>The lines might appear in a different order.</p> |
| <p>Run the following command to deactivate the virtual environment:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>deactivate</code></pre> |
| </div> |
| </div> |
| <h2 id="explore-the-code">Explore the code</h2> |
| <p>The main code file for this quickstart is <strong>app.py</strong> |
| (<a href="https://github.com/apache/beam-starter-python/blob/main/my_app/app.py">GitHub</a>). |
| The code performs the following steps:</p> |
| <ol> |
| <li>Create a Beam pipeline.</li> |
| <li>Create an initial <code>PCollection</code>.</li> |
| <li>Apply a transform to the <code>PCollection</code>.</li> |
| <li>Run the pipeline, using the Direct Runner.</li> |
| </ol> |
| <h3 id="create-a-pipeline">Create a pipeline</h3> |
| <p>The code first creates a <code>Pipeline</code> object. The <code>Pipeline</code> object builds up the |
| graph of transformations to be executed.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">(</span><span class="n">options</span><span class="o">=</span><span class="n">beam_options</span><span class="p">)</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> |
| </span></span></code></pre></div><p>The <code>beam_option</code> variable shown here is a <code>PipelineOptions</code> object, which |
| is used to set options for the pipeline. For more information, see |
| <a href="/documentation/programming-guide/#configuring-pipeline-options">Configuring pipeline options</a>.</p> |
| <h3 id="create-an-initial-pcollection">Create an initial PCollection</h3> |
| <p>The <code>PCollection</code> abstraction represents a potentially distributed, |
| multi-element data set. A Beam pipeline needs a source of data to populate an |
| initial <code>PCollection</code>. The source can be bounded (with a known, fixed size) or |
| unbounded (with unlimited size).</p> |
| <p>This example uses the |
| <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Create"><code>Create</code></a> |
| method to create a <code>PCollection</code> from an in-memory array of strings. The |
| resulting <code>PCollection</code> contains the strings &ldquo;Hello&rdquo;, &ldquo;World!&rdquo;, and a |
| user-provided input string.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"><span class="o">|</span> <span class="s2">&#34;Create elements&#34;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span><span class="s2">&#34;Hello&#34;</span><span class="p">,</span> <span class="s2">&#34;World!&#34;</span><span class="p">,</span> <span class="n">input_text</span><span class="p">])</span> |
| </span></span></code></pre></div><p>Note: The pipe operator <code>|</code> is used to |
| <a href="/documentation/programming-guide/#applying-transforms">chain</a> transforms.</p> |
| <h3 id="apply-a-transform-to-the-pcollection">Apply a transform to the PCollection</h3> |
| <p>Transforms can change, filter, group, analyze, or otherwise process the |
| elements in a <code>PCollection</code>. This example uses the |
| <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.Map"><code>Map</code></a> |
| transform, which maps the elements of a collection into a new collection:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="o">|</span> <span class="s2">&#34;Print elements&#34;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="nb">print</span><span class="p">)</span> |
| </span></span></code></pre></div><h3 id="run-the-pipeline">Run the pipeline</h3> |
| <p>To run the pipeline, you can call the |
| <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.pipeline.html#apache_beam.pipeline.Pipeline.run"><code>Pipeline.run</code></a> |
| method:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">pipeline</span><span class="o">.</span><span class="n">run</span><span class="o">.</span><span class="n">wait_until_finish</span><span class="p">()</span> |
| </span></span></code></pre></div><p>However, by enclosing the <code>Pipeline</code> object inside a <code>with</code> statement, the |
| <code>run</code> method is automatically invoked.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">(</span><span class="n">options</span><span class="o">=</span><span class="n">beam_options</span><span class="p">)</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># ...</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># run() is called automatically</span> |
| </span></span></code></pre></div><p>A Beam <a href="/documentation/basics/#runner">runner</a> runs a Beam pipeline on a |
| specific platform. If you don&rsquo;t specify a runner, the Direct Runner is the |
| default. The Direct Runner runs the pipeline locally on your machine. It is |
| meant for testing and development, rather than being optimized for efficiency. |
| For more information, see |
| <a href="/documentation/runners/direct/">Using the Direct Runner</a>.</p> |
| <p>For production workloads, you typically use a distributed runner that runs the |
| pipeline on a big data processing system such as Apache Flink, Apache Spark, or |
| Google Cloud Dataflow. These systems support massively parallel processing.</p> |
| <h2 id="next-steps">Next Steps</h2> |
| <ul> |
| <li>Learn more about the <a href="/documentation/sdks/python/">Beam SDK for Python</a> |
| and look through the |
| <a href="https://beam.apache.org/releases/pydoc/current">Python SDK API reference</a>.</li> |
| <li>Take a self-paced tour through our |
| <a href="/documentation/resources/learning-resources">Learning Resources</a>.</li> |
| <li>Dive in to some of our favorite |
| <a href="/documentation/resources/videos-and-podcasts">Videos and Podcasts</a>.</li> |
| <li>Join the Beam <a href="/community/contact-us">users@</a> mailing list.</li> |
| </ul> |
| <p>Please don&rsquo;t hesitate to <a href="/community/contact-us">reach out</a> if you encounter any |
| issues!</p></description></item><item><title>Get-Started: Beam Quickstart for Typescript</title><link>/get-started/quickstart/typescript/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/quickstart/typescript/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="apache-beam-typescript-sdk-quickstart">Apache Beam Typescript SDK quickstart</h1> |
| <p>This quickstart shows you how to run an |
| <a href="https://github.com/apache/beam-starter-typescript">example pipeline</a> written with |
| the <a href="/documentation/sdks/typescript">Apache Beam Typescript SDK</a>, using the |
| <a href="/documentation/runners/direct/">Direct Runner</a>. The Direct Runner executes |
| pipelines locally on your machine.</p> |
| <p>If you&rsquo;re interested in contributing to the Apache Beam Typescript codebase, see the |
| <a href="/contribute">Contribution Guide</a>.</p> |
| <p>On this page:</p> |
| <nav id="TableOfContents"> |
| <ul> |
| <li><a href="#set-up-your-development-environment">Set up your development environment</a></li> |
| <li><a href="#clone-the-github-repository">Clone the GitHub repository</a></li> |
| <li><a href="#install-the-project-dependences">Install the project dependences</a></li> |
| <li><a href="#compile-the-pipeline">Compile the pipeline</a></li> |
| <li><a href="#run-the-quickstart">Run the quickstart</a></li> |
| <li><a href="#explore-the-code">Explore the code</a> |
| <ul> |
| <li><a href="#create-a-pipeline">Create a pipeline</a></li> |
| <li><a href="#create-an-initial-pcollection">Create an initial PCollection</a></li> |
| <li><a href="#apply-a-transform-to-the-pcollection">Apply a transform to the PCollection</a></li> |
| <li><a href="#run-the-pipeline">Run the pipeline</a></li> |
| </ul> |
| </li> |
| <li><a href="#next-steps">Next Steps</a></li> |
| </ul> |
| </nav> |
| <h2 id="set-up-your-development-environment">Set up your development environment</h2> |
| <p>Make sure you have a <a href="https://nodejs.org/">Node.js</a> development environment installed. |
| If you don&rsquo;t, you can download and install it from the |
| <a href="https://nodejs.org/en/download/">downloads page</a>.</p> |
| <p>Due to its extensive use of cross-language transforms, it is recommended that |
| Python 3 and Java be available on the system as well.</p> |
| <h2 id="clone-the-github-repository">Clone the GitHub repository</h2> |
| <p>Clone or download the |
| <a href="https://github.com/apache/beam-starter-typescript">apache/beam-starter-typescript</a> |
| GitHub repository and change into the <code>beam-starter-typescript</code> directory.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>git clone https://github.com/apache/beam-starter-typescript.git |
| cd beam-starter-typescript</code></pre> |
| </div> |
| </div> |
| <h2 id="install-the-project-dependences">Install the project dependences</h2> |
| <p>Run the following command to install the project&rsquo;s dependencies.</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>npm install</code></pre> |
| </div> |
| </div> |
| <h2 id="compile-the-pipeline">Compile the pipeline</h2> |
| <p>The pipeline is then built with</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>npm run build</code></pre> |
| </div> |
| </div> |
| <h2 id="run-the-quickstart">Run the quickstart</h2> |
| <p>Run the following command:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>node dist/src/main.js --input_text=&#34;Greetings&#34;</code></pre> |
| </div> |
| </div> |
| <p>The output is similar to the following:</p> |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>Hello |
| World! |
| Greetings</code></pre> |
| </div> |
| </div> |
| <p>The lines might appear in a different order.</p> |
| <h2 id="explore-the-code">Explore the code</h2> |
| <p>The main code file for this quickstart is <strong>app.ts</strong> |
| (<a href="https://github.com/apache/beam-starter-typescript/blob/main/src/app.ts">GitHub</a>). |
| The code performs the following steps:</p> |
| <ol> |
| <li>Define a Beam pipeline that.</li> |
| </ol> |
| <ul> |
| <li>Creates an initial <code>PCollection</code>.</li> |
| <li>Applies a transform (map) to the <code>PCollection</code>.</li> |
| </ul> |
| <ol start="2"> |
| <li>Run the pipeline, using the Direct Runner.</li> |
| </ol> |
| <h3 id="create-a-pipeline">Create a pipeline</h3> |
| <p>A <code>Pipeline</code> is simply a callable that takes a single <code>root</code> object. |
| The <code>Pipeline</code> function builds up the graph of transformations to be executed.</p> |
| <h3 id="create-an-initial-pcollection">Create an initial PCollection</h3> |
| <p>The <code>PCollection</code> abstraction represents a potentially distributed, |
| multi-element data set. A Beam pipeline needs a source of data to populate an |
| initial <code>PCollection</code>. The source can be bounded (with a known, fixed size) or |
| unbounded (with unlimited size).</p> |
| <p>This example uses the |
| <a href="https://beam.apache.org/releases/typedoc/current/functions/transforms_create.create.html"><code>Create</code></a> |
| method to create a <code>PCollection</code> from an in-memory array of strings. The |
| resulting <code>PCollection</code> contains the strings &ldquo;Hello&rdquo;, &ldquo;World!&rdquo;, and a |
| user-provided input string.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="nx">root</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">create</span><span class="p">([</span><span class="s2">&#34;Hello&#34;</span><span class="p">,</span> <span class="s2">&#34;World!&#34;</span><span class="p">,</span> <span class="nx">input_text</span><span class="p">]))</span> |
| </span></span></code></pre></div><h3 id="apply-a-transform-to-the-pcollection">Apply a transform to the PCollection</h3> |
| <p>Transforms can change, filter, group, analyze, or otherwise process the |
| elements in a <code>PCollection</code>. This example uses the |
| <a href="https://beam.apache.org/releases/typedoc/current/classes/pvalue.PCollection.html#map"><code>Map</code></a> |
| transform, which maps the elements of a collection into a new collection:</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">printAndReturn</span><span class="p">);</span> |
| </span></span></code></pre></div><p>For convenience, <code>PColletion</code> has a <code>map</code> method, but more generally transforms |
| are applied with <code>.apply(someTransform())</code>.</p> |
| <h3 id="run-the-pipeline">Run the pipeline</h3> |
| <p>To run the pipeline, a runner is created (possibly with some options)</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="nx">createRunner</span><span class="p">(</span><span class="nx">options</span><span class="p">)</span> |
| </span></span></code></pre></div><p>and then its <code>run</code> method is invoked on the pipeline callable created above.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="p">.</span><span class="nx">run</span><span class="p">(</span><span class="nx">createPipeline</span><span class="p">(...));</span> |
| </span></span></code></pre></div><p>A Beam <a href="/documentation/basics/#runner">runner</a> runs a Beam pipeline on a |
| specific platform. If you don&rsquo;t specify a runner, the Direct Runner is the |
| default. The Direct Runner runs the pipeline locally on your machine. It is |
| meant for testing and development, rather than being optimized for efficiency. |
| For more information, see |
| <a href="/documentation/runners/direct/">Using the Direct Runner</a>.</p> |
| <p>For production workloads, you typically use a distributed runner that runs the |
| pipeline on a big data processing system such as Apache Flink, Apache Spark, or |
| Google Cloud Dataflow. These systems support massively parallel processing. |
| Different runners can be requested via the runner property on options, e.g. |
| <code>createRunner({runner: &quot;dataflow&quot;})</code> or <code>createRunner({runner: &quot;flink&quot;})</code>. |
| In this example this value can be passed in via the command line as |
| <code>--runner=...</code>, e.g. to run on Dataflow one would write</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl">node dist/src/main.js <span class="se">\ |
| </span></span></span><span class="line"><span class="cl"><span class="se"></span> --runner<span class="o">=</span>dataflow <span class="se">\ |
| </span></span></span><span class="line"><span class="cl"><span class="se"></span> --project<span class="o">=</span><span class="si">${</span><span class="nv">PROJECT_ID</span><span class="si">}</span> <span class="se">\ |
| </span></span></span><span class="line"><span class="cl"><span class="se"></span> --tempLocation<span class="o">=</span>gs://<span class="si">${</span><span class="nv">GCS_BUCKET</span><span class="si">}</span>/wordcount-js/temp --region<span class="o">=</span><span class="si">${</span><span class="nv">REGION</span><span class="si">}</span> |
| </span></span></code></pre></div><h2 id="next-steps">Next Steps</h2> |
| <ul> |
| <li>Learn more about the <a href="/documentation/sdks/typescript/">Beam SDK for Typescript</a> |
| and look through the |
| <a href="https://beam.apache.org/releases/typedoc/current">Typescript SDK API reference</a>.</li> |
| <li>Take a self-paced tour through our |
| <a href="/documentation/resources/learning-resources">Learning Resources</a>.</li> |
| <li>Dive in to some of our favorite |
| <a href="/documentation/resources/videos-and-podcasts">Videos and Podcasts</a>.</li> |
| <li>Join the Beam <a href="/community/contact-us">users@</a> mailing list.</li> |
| </ul> |
| <p>Please don&rsquo;t hesitate to <a href="/community/contact-us">reach out</a> if you encounter any |
| issues!</p></description></item><item><title>Get-Started: Beam Releases</title><link>/get-started/downloads/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/downloads/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="apache-beamsupsup-downloads">Apache Beam<sup>®</sup> Downloads</h1> |
| <blockquote> |
| <p>Beam SDK 2.55.1 is the latest released version.</p> |
| </blockquote> |
| <h2 id="using-a-central-repository">Using a central repository</h2> |
| <p>The easiest way to use Apache Beam is via one of the released versions in a |
| central repository. The Java SDK is available on <a href="https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22">Maven Central Repository</a>, |
| and the Python SDK is available on <a href="https://pypi.python.org/pypi/apache-beam">PyPI</a>.</p> |
| <p>For example, if you are developing using Maven and want to use the SDK for Java |
| with the <code>DirectRunner</code>, add the following dependencies to your <code>pom.xml</code> file:</p> |
| <pre><code>&lt;dependency&gt; |
| &lt;groupId&gt;org.apache.beam&lt;/groupId&gt; |
| &lt;artifactId&gt;beam-sdks-java-core&lt;/artifactId&gt; |
| &lt;version&gt;2.55.1&lt;/version&gt; |
| &lt;/dependency&gt; |
| &lt;dependency&gt; |
| &lt;groupId&gt;org.apache.beam&lt;/groupId&gt; |
| &lt;artifactId&gt;beam-runners-direct-java&lt;/artifactId&gt; |
| &lt;version&gt;2.55.1&lt;/version&gt; |
| &lt;scope&gt;runtime&lt;/scope&gt; |
| &lt;/dependency&gt; |
| </code></pre> |
| <p>Similarly in Python, if you are using PyPI and want to use the SDK for Python |
| with <code>DirectRunner</code>, add the following requirement to your <code>setup.py</code> file:</p> |
| <pre><code>apache-beam==2.55.1 |
| </code></pre> |
| <p>Additionally, you may want to depend on additional SDK modules, such as IO |
| connectors or other extensions, and additional runners to execute your pipeline |
| at scale.</p> |
| <p>The Go SDK is accessible via Go Modules and calling <code>go get</code> from a module subdirectory:</p> |
| <pre><code> go get github.com/apache/beam/sdks/v2/go/pkg/beam |
| </code></pre> |
| <p>Specific versions can be depended on similarly:</p> |
| <pre><code> go get github.com/apache/beam/sdks/v2@v2.55.1/go/pkg/beam |
| </code></pre> |
| <h2 id="downloading-source-code">Downloading source code</h2> |
| <p>You can download the source code package for a release from the links in the |
| <a href="#releases">Releases</a> section.</p> |
| <h3 id="release-integrity">Release integrity</h3> |
| <p>You <em>must</em> <a href="https://www.apache.org/info/verification.html">verify</a> the integrity |
| of downloaded files. We provide OpenPGP signatures for every release file. This |
| signature should be matched against the |
| <a href="https://downloads.apache.org/beam/KEYS">KEYS</a> file which contains the OpenPGP |
| keys of Apache Beam&rsquo;s Release Managers. We also provide SHA-512 checksums for |
| every release file (or SHA-1 and MD5 checksums for older releases). After you |
| download the file, you should calculate a checksum for your download, and make |
| sure it is the same as ours.</p> |
| <h2 id="api-stability">API stability</h2> |
| <p>Apache Beam generally follows the rules of |
| <a href="https://semver.org/">semantic versioning</a> with exceptions. Version numbers use |
| the form <code>major.minor.patch</code> and are incremented as follows:</p> |
| <ul> |
| <li>major version for incompatible API changes</li> |
| <li>minor version for new functionality added in a backward-compatible manner, infrequent incompatible API changes</li> |
| <li>patch version for forward-compatible bug fixes</li> |
| </ul> |
| <p>Please note that APIs marked <a href="https://beam.apache.org/releases/javadoc/2.55.1/org/apache/beam/sdk/annotations/Experimental.html"><code>@Experimental</code></a> |
| may change at any point and are not guaranteed to remain compatible across versions.</p> |
| <p>Additionally, any API may change before the first stable release, i.e., between |
| versions denoted <code>0.x.y</code>.</p> |
| <h2 id="releases">Releases</h2> |
| <h3 id="2551-2024-03-25">2.55.1 (2024-03-25)</h3> |
| <p>Official <a href="https://downloads.apache.org/beam/2.55.1/apache-beam-2.55.1-source-release.zip">source code download</a>. |
| <a href="https://downloads.apache.org/beam/2.55.1/apache-beam-2.55.1-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://downloads.apache.org/beam/2.55.1/apache-beam-2.55.1-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://github.com/apache/beam/releases/tag/v2.55.1">Release notes</a></p> |
| <h3 id="2550-2024-03-25">2.55.0 (2024-03-25)</h3> |
| <p>Official <a href="https://archive.apache.org/beam/2.55.0/apache-beam-2.55.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/beam/2.55.0/apache-beam-2.55.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/beam/2.55.0/apache-beam-2.55.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://github.com/apache/beam/releases/tag/v2.55.0">Release notes</a> |
| <a href="/blog/beam-2.55.0">Blog post</a>.</p> |
| <h3 id="2540-2024-02-14">2.54.0 (2024-02-14)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.54.0/apache-beam-2.54.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.54.0/apache-beam-2.54.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.54.0/apache-beam-2.54.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://github.com/apache/beam/releases/tag/v2.54.0">Release notes</a> |
| <a href="/blog/beam-2.54.0">Blog post</a>.</p> |
| <h3 id="2530-2024-01-04">2.53.0 (2024-01-04)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.53.0/apache-beam-2.53.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.53.0/apache-beam-2.53.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.53.0/apache-beam-2.53.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://github.com/apache/beam/releases/tag/v2.53.0">Release notes</a> |
| <a href="/blog/beam-2.53.0">Blog post</a>.</p> |
| <h3 id="2520-2023-11-17">2.52.0 (2023-11-17)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.52.0/apache-beam-2.52.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.52.0/apache-beam-2.52.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.52.0/apache-beam-2.52.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://github.com/apache/beam/releases/tag/v2.52.0">Release notes</a> |
| <a href="/blog/beam-2.52.0">Blog post</a>.</p> |
| <h3 id="2510-2023-10-11">2.51.0 (2023-10-11)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.51.0/apache-beam-2.51.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.51.0/apache-beam-2.51.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.51.0/apache-beam-2.51.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://github.com/apache/beam/releases/tag/v2.51.0">Release notes</a> |
| <a href="/blog/beam-2.51.0">Blog post</a>.</p> |
| <h3 id="2500-2023-08-30">2.50.0 (2023-08-30)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.50.0/apache-beam-2.50.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.50.0/apache-beam-2.50.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.50.0/apache-beam-2.50.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://github.com/apache/beam/releases/tag/v2.50.0">Release notes</a> |
| <a href="/blog/beam-2.50.0">Blog post</a>.</p> |
| <h3 id="2490-2023-07-17">2.49.0 (2023-07-17)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.49.0/apache-beam-2.49.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.49.0/apache-beam-2.49.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.49.0/apache-beam-2.49.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://github.com/apache/beam/releases/tag/v2.49.0">Release notes</a> |
| <a href="/blog/beam-2.49.0">Blog post</a>.</p> |
| <h3 id="2480-2023-05-31">2.48.0 (2023-05-31)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.48.0/apache-beam-2.48.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.48.0/apache-beam-2.48.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.48.0/apache-beam-2.48.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://github.com/apache/beam/releases/tag/v2.48.0">Release notes</a> |
| <a href="/blog/beam-2.48.0">Blog post</a>.</p> |
| <h3 id="2470-2023-05-10">2.47.0 (2023-05-10)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.47.0/apache-beam-2.47.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.47.0/apache-beam-2.47.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.47.0/apache-beam-2.47.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://github.com/apache/beam/releases/tag/v2.47.0">Release notes</a> |
| <a href="/blog/beam-2.47.0">Blog post</a>.</p> |
| <h3 id="2460-2023-03-10">2.46.0 (2023-03-10)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.46.0/apache-beam-2.46.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.46.0/apache-beam-2.46.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.46.0/apache-beam-2.46.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://github.com/apache/beam/releases/tag/v2.46.0">Release notes</a> |
| <a href="/blog/beam-2.46.0">Blog post</a>.</p> |
| <h3 id="2450-2023-02-15">2.45.0 (2023-02-15)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.45.0/apache-beam-2.45.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.45.0/apache-beam-2.45.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.45.0/apache-beam-2.45.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://github.com/apache/beam/releases/tag/v2.45.0">Release notes</a> |
| <a href="/blog/beam-2.45.0">Blog post</a>.</p> |
| <h3 id="2440-2023-01-12">2.44.0 (2023-01-12)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.44.0/apache-beam-2.44.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.44.0/apache-beam-2.44.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.44.0/apache-beam-2.44.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://github.com/apache/beam/releases/tag/v2.44.0">Release notes</a> |
| <a href="/blog/beam-2.44.0">Blog post</a>.</p> |
| <h3 id="2430-2022-11-17">2.43.0 (2022-11-17)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.43.0/apache-beam-2.43.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.43.0/apache-beam-2.43.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.43.0/apache-beam-2.43.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://github.com/apache/beam/releases/tag/v2.43.0">Release notes</a> |
| <a href="/blog/beam-2.43.0">Blog post</a>.</p> |
| <h3 id="2420-2022-10-17">2.42.0 (2022-10-17)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.42.0/apache-beam-2.42.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.42.0/apache-beam-2.42.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.42.0/apache-beam-2.42.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://github.com/apache/beam/releases/tag/v2.42.0">Release notes</a> |
| <a href="/blog/beam-2.42.0">Blog post</a>.</p> |
| <h3 id="2410-2022-08-23">2.41.0 (2022-08-23)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.41.0/apache-beam-2.41.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.41.0/apache-beam-2.41.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.41.0/apache-beam-2.41.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://github.com/apache/beam/releases/tag/v2.41.0">Release notes</a> |
| <a href="/blog/beam-2.41.0">Blog post</a>.</p> |
| <h3 id="2400-2022-06-25">2.40.0 (2022-06-25)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.40.0/apache-beam-2.40.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.40.0/apache-beam-2.40.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.40.0/apache-beam-2.40.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://github.com/apache/beam/releases/tag/v2.40.0">Release notes</a> |
| <a href="/blog/beam-2.40.0">Blog post</a>.</p> |
| <h3 id="2390-2022-05-25">2.39.0 (2022-05-25)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.39.0/apache-beam-2.39.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.39.0/apache-beam-2.39.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.39.0/apache-beam-2.39.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12351169">Release notes</a> |
| <a href="/blog/beam-2.39.0">Blog post</a>.</p> |
| <h3 id="2380-2022-04-20">2.38.0 (2022-04-20)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.38.0/apache-beam-2.38.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.38.0/apache-beam-2.38.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.38.0/apache-beam-2.38.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12351169">Release notes</a> |
| <a href="/blog/beam-2.38.0">Blog post</a>.</p> |
| <h3 id="2370-2022-03-04">2.37.0 (2022-03-04)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.37.0/apache-beam-2.37.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.37.0/apache-beam-2.37.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.37.0/apache-beam-2.37.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12351168">Release notes</a> |
| <a href="/blog/beam-2.37.0">Blog post</a>.</p> |
| <h3 id="2360-2022-02-07">2.36.0 (2022-02-07)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.36.0/apache-beam-2.36.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.36.0/apache-beam-2.36.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.36.0/apache-beam-2.36.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12350407">Release notes</a> |
| <a href="/blog/beam-2.36.0">Blog post</a>.</p> |
| <h3 id="2350-2021-12-29">2.35.0 (2021-12-29)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.35.0/apache-beam-2.35.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.35.0/apache-beam-2.35.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.35.0/apache-beam-2.35.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12350406">Release notes</a> |
| <a href="/blog/beam-2.35.0">Blog post</a>.</p> |
| <h3 id="2340-2021-11-11">2.34.0 (2021-11-11)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.34.0/apache-beam-2.34.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.34.0/apache-beam-2.34.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.34.0/apache-beam-2.34.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12350405">Release notes</a> |
| <a href="/blog/beam-2.34.0">Blog post</a>.</p> |
| <h3 id="2330-2021-10-07">2.33.0 (2021-10-07)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.33.0/apache-beam-2.33.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.33.0/apache-beam-2.33.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.33.0/apache-beam-2.33.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12350404">Release notes</a> |
| <a href="/blog/beam-2.33.0">Blog post</a>.</p> |
| <h3 id="2320-2021-08-25">2.32.0 (2021-08-25)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.32.0/apache-beam-2.32.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.32.0/apache-beam-2.32.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.32.0/apache-beam-2.32.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12349992">Release notes</a> |
| <a href="/blog/beam-2.32.0">Blog post</a>.</p> |
| <h3 id="2310-2021-07-08">2.31.0 (2021-07-08)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.31.0/apache-beam-2.31.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.31.0/apache-beam-2.31.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.31.0/apache-beam-2.31.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12349991">Release notes</a> |
| <a href="/blog/beam-2.31.0">Blog post</a>.</p> |
| <h3 id="2300-2021-06-09">2.30.0 (2021-06-09)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.30.0/apache-beam-2.30.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.30.0/apache-beam-2.30.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.30.0/apache-beam-2.30.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12349978">Release notes</a> |
| <a href="/blog/beam-2.30.0">Blog post</a>.</p> |
| <h3 id="2290-2021-04-27">2.29.0 (2021-04-27)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.29.0/apache-beam-2.29.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.29.0/apache-beam-2.29.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.29.0/apache-beam-2.29.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12349629">Release notes</a> |
| <a href="/blog/beam-2.29.0">Blog post</a>.</p> |
| <h3 id="2280-2021-02-22">2.28.0 (2021-02-22)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.28.0/apache-beam-2.28.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.28.0/apache-beam-2.28.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.28.0/apache-beam-2.28.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12349499">Release notes</a>. |
| <a href="/blog/beam-2.28.0">Blog post</a>.</p> |
| <h3 id="2270-2020-12-22">2.27.0 (2020-12-22)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.27.0/apache-beam-2.27.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.27.0/apache-beam-2.27.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.27.0/apache-beam-2.27.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12349380">Release notes</a>. |
| <a href="/blog/beam-2.27.0">Blog post</a>.</p> |
| <h3 id="2260-2020-12-11">2.26.0 (2020-12-11)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.26.0/apache-beam-2.26.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.26.0/apache-beam-2.26.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.26.0/apache-beam-2.26.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12348833">Release notes</a>. |
| <a href="/blog/beam-2.26.0">Blog post</a>.</p> |
| <h3 id="2250-2020-10-23">2.25.0 (2020-10-23)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.25.0/apache-beam-2.25.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.25.0/apache-beam-2.25.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.25.0/apache-beam-2.25.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12347147">Release notes</a>. |
| <a href="/blog/beam-2.25.0">Blog post</a>.</p> |
| <h3 id="2240-2020-09-18">2.24.0 (2020-09-18)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.24.0/apache-beam-2.24.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.24.0/apache-beam-2.24.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.24.0/apache-beam-2.24.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12347146">Release notes</a>. |
| <a href="/blog/beam-2.24.0">Blog post</a>.</p> |
| <h3 id="2230-2020-07-29">2.23.0 (2020-07-29)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.23.0/apache-beam-2.23.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.23.0/apache-beam-2.23.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.23.0/apache-beam-2.23.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12347145">Release notes</a>. |
| <a href="/blog/beam-2.23.0">Blog post</a>.</p> |
| <h3 id="2220-2020-06-08">2.22.0 (2020-06-08)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.22.0/apache-beam-2.22.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.22.0/apache-beam-2.22.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.22.0/apache-beam-2.22.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12347144">Release notes</a>. |
| <a href="/blog/beam-2.22.0">Blog post</a>.</p> |
| <h3 id="2210-2020-05-27">2.21.0 (2020-05-27)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.21.0/apache-beam-2.21.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.21.0/apache-beam-2.21.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.21.0/apache-beam-2.21.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12347143">Release notes</a>. |
| <a href="/blog/beam-2.21.0">Blog post</a>.</p> |
| <h3 id="2200-2020-04-15">2.20.0 (2020-04-15)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.20.0/apache-beam-2.20.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.20.0/apache-beam-2.20.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.20.0/apache-beam-2.20.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12346780">Release notes</a>. |
| <a href="/blog/beam-2.20.0">Blog post</a>.</p> |
| <h3 id="2190-2020-02-04">2.19.0 (2020-02-04)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.19.0/apache-beam-2.19.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.19.0/apache-beam-2.19.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.19.0/apache-beam-2.19.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12346582">Release notes</a>. |
| <a href="/blog/beam-2.19.0">Blog post</a>.</p> |
| <h3 id="2180-2020-01-23">2.18.0 (2020-01-23)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.18.0/apache-beam-2.18.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.18.0/apache-beam-2.18.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.18.0/apache-beam-2.18.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12346383&amp;projectId=12319527">Release notes</a>. |
| <a href="/blog/beam-2.18.0">Blog post</a>.</p> |
| <h3 id="2170-2020-01-06">2.17.0 (2020-01-06)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.17.0/apache-beam-2.17.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.17.0/apache-beam-2.17.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.17.0/apache-beam-2.17.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12345970">Release notes</a>. |
| <a href="/blog/beam-2.17.0">Blog post</a>.</p> |
| <h3 id="2160-2019-10-07">2.16.0 (2019-10-07)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.16.0/apache-beam-2.16.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.16.0/apache-beam-2.16.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.16.0/apache-beam-2.16.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12345494">Release notes</a>. |
| <a href="/blog/beam-2.16.0">Blog post</a>.</p> |
| <h3 id="2150-2019-08-22">2.15.0 (2019-08-22)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.15.0/apache-beam-2.15.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.15.0/apache-beam-2.15.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.15.0/apache-beam-2.15.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12345489">Release notes</a>. |
| <a href="/blog/beam-2.15.0">Blog post</a>.</p> |
| <h3 id="2140-2019-08-01">2.14.0 (2019-08-01)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.14.0/apache-beam-2.14.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.14.0/apache-beam-2.14.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.14.0/apache-beam-2.14.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12345431">Release notes</a>. |
| <a href="/blog/beam-2.14.0">Blog post</a>.</p> |
| <h3 id="2130-2019-05-21">2.13.0 (2019-05-21)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.13.0/apache-beam-2.13.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.13.0/apache-beam-2.13.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.13.0/apache-beam-2.13.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://jira.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12345166">Release notes</a>. |
| <a href="/blog/beam-2.13.0">Blog post</a>.</p> |
| <h3 id="2120-2019-04-25">2.12.0 (2019-04-25)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.12.0/apache-beam-2.12.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.12.0/apache-beam-2.12.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.12.0/apache-beam-2.12.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://jira.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12344944">Release notes</a>. |
| <a href="/blog/beam-2.12.0">Blog post</a>.</p> |
| <h3 id="2110-2019-02-26">2.11.0 (2019-02-26)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.11.0/apache-beam-2.11.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.11.0/apache-beam-2.11.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.11.0/apache-beam-2.11.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12344775">Release notes</a>. |
| <a href="/blog/beam-2.11.0">Blog post</a>.</p> |
| <h3 id="2100-2019-02-01">2.10.0 (2019-02-01)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.10.0/apache-beam-2.10.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.10.0/apache-beam-2.10.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.10.0/apache-beam-2.10.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12344540">Release notes</a>. |
| <a href="/blog/beam-2.10.0">Blog post</a>.</p> |
| <h3 id="290-2018-12-13">2.9.0 (2018-12-13)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.9.0/apache-beam-2.9.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.9.0/apache-beam-2.9.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.9.0/apache-beam-2.9.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12344258">Release notes</a>. |
| <a href="/blog/beam-2.9.0">Blog post</a>.</p> |
| <h3 id="280-2018-10-26">2.8.0 (2018-10-26)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.8.0/apache-beam-2.8.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.8.0/apache-beam-2.8.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.8.0/apache-beam-2.8.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12343985">Release notes</a>. |
| <a href="/blog/beam-2.8.0">Blog post</a>.</p> |
| <h3 id="270-lts-2018-10-02">2.7.0 LTS (2018-10-02)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.7.0/apache-beam-2.7.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.7.0/apache-beam-2.7.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.7.0/apache-beam-2.7.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12343654">Release notes</a>. |
| <a href="/blog/beam-2.7.0">Blog post</a>.</p> |
| <p>2.7.0 was <a href="https://lists.apache.org/thread.html/896cbc9fef2e60f19b466d6b1e12ce1aeda49ce5065a0b1156233f01@%3Cdev.beam.apache.org%3E">designated</a> by the Beam community as a long term support (LTS) version. LTS versions are supported for a window of 6 months starting from the day it is marked as an LTS. Beam community will decide on which issues will be backported and when patch releases on the branch will be made on a case by case basis.</p> |
| <p><em>LTS Update (2020-04-06):</em> Due to the lack of interest from users the Beam community decided not to maintain or publish new LTS releases. We encourage users to update early and often to the most recent releases.</p> |
| <h3 id="260-2018-08-08">2.6.0 (2018-08-08)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.6.0/apache-beam-2.6.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.6.0/apache-beam-2.6.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.6.0/apache-beam-2.6.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12343392">Release notes</a>. |
| <a href="/blog/beam-2.6.0">Blog post</a>.</p> |
| <h3 id="250-2018-06-06">2.5.0 (2018-06-06)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.5.0/apache-beam-2.5.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.5.0/apache-beam-2.5.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.5.0/apache-beam-2.5.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12342847">Release notes</a>. |
| <a href="/blog/beam-2.5.0">Blog post</a>.</p> |
| <h3 id="240-2018-03-20">2.4.0 (2018-03-20)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.4.0/apache-beam-2.4.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.4.0/apache-beam-2.4.0-source-release.zip.sha512">SHA-512</a>. |
| <a href="https://archive.apache.org/dist/beam/2.4.0/apache-beam-2.4.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12342682">Release notes</a>. |
| <a href="/blog/beam-2.4.0">Blog post</a>.</p> |
| <h3 id="230-2018-01-30">2.3.0 (2018-01-30)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.3.0/apache-beam-2.3.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.3.0/apache-beam-2.3.0-source-release.zip.sha1">SHA-1</a>. |
| <a href="https://archive.apache.org/dist/beam/2.3.0/apache-beam-2.3.0-source-release.zip.md5">MD5</a>. |
| <a href="https://archive.apache.org/dist/beam/2.3.0/apache-beam-2.3.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12341608">Release notes</a>. |
| <a href="/blog/beam-2.3.0">Blog post</a>.</p> |
| <h3 id="220-2017-12-02">2.2.0 (2017-12-02)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.2.0/apache-beam-2.2.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.2.0/apache-beam-2.2.0-source-release.zip.sha1">SHA-1</a>. |
| <a href="https://archive.apache.org/dist/beam/2.2.0/apache-beam-2.2.0-source-release.zip.md5">MD5</a>. |
| <a href="https://archive.apache.org/dist/beam/2.2.0/apache-beam-2.2.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12341044">Release notes</a>.</p> |
| <h3 id="210-2017-08-23">2.1.0 (2017-08-23)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.1.0/apache-beam-2.1.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.1.0/apache-beam-2.1.0-source-release.zip.sha1">SHA-1</a>. |
| <a href="https://archive.apache.org/dist/beam/2.1.0/apache-beam-2.1.0-source-release.zip.md5">MD5</a>. |
| <a href="https://archive.apache.org/dist/beam/2.1.0/apache-beam-2.1.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12340528">Release notes</a>.</p> |
| <h3 id="200-2017-05-17">2.0.0 (2017-05-17)</h3> |
| <p>Official <a href="https://archive.apache.org/dist/beam/2.0.0/apache-beam-2.0.0-source-release.zip">source code download</a>. |
| <a href="https://archive.apache.org/dist/beam/2.0.0/apache-beam-2.0.0-source-release.zip.sha1">SHA-1</a>. |
| <a href="https://archive.apache.org/dist/beam/2.0.0/apache-beam-2.0.0-source-release.zip.md5">MD5</a>. |
| <a href="https://archive.apache.org/dist/beam/2.0.0/apache-beam-2.0.0-source-release.zip.asc">signature</a>.</p> |
| <p><a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12339746">Release notes</a>.</p></description></item><item><title>Get-Started: Beam WordCount Examples</title><link>/get-started/wordcount-example/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/wordcount-example/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="apache-beam-wordcount-examples">Apache Beam WordCount Examples</h1> |
| <nav id="TableOfContents"> |
| <ul> |
| <li><a href="#minimalwordcount-example">MinimalWordCount example</a> |
| <ul> |
| <li><a href="#creating-the-pipeline">Creating the pipeline</a></li> |
| <li><a href="#applying-pipeline-transforms">Applying pipeline transforms</a></li> |
| <li><a href="#running-the-pipeline">Running the pipeline</a></li> |
| <li><a href="#try-the-full-example-in-playground">Try the full example in Playground</a></li> |
| </ul> |
| </li> |
| <li><a href="#wordcount-example">WordCount example</a> |
| <ul> |
| <li><a href="#specifying-explicit-dofns">Specifying explicit DoFns</a></li> |
| <li><a href="#creating-composite-transforms">Creating composite transforms</a></li> |
| <li><a href="#using-parameterizable-pipelineoptions">Using parameterizable PipelineOptions</a></li> |
| <li><a href="#try-the-full-example-in-playground-1">Try the full example in Playground</a></li> |
| </ul> |
| </li> |
| <li><a href="#debuggingwordcount-example">DebuggingWordCount example</a> |
| <ul> |
| <li><a href="#logging">Logging</a> |
| <ul> |
| <li><a href="#direct-runner">Direct Runner</a></li> |
| <li><a href="#cloud-dataflow-runner">Cloud Dataflow Runner</a></li> |
| <li><a href="#apache-spark-runner">Apache Spark Runner</a></li> |
| <li><a href="#apache-flink-runner">Apache Flink Runner</a></li> |
| <li><a href="#apache-nemo-runner">Apache Nemo Runner</a></li> |
| </ul> |
| </li> |
| <li><a href="#testing-your-pipeline-with-asserts">Testing your pipeline with asserts</a></li> |
| <li><a href="#try-the-full-example-in-playground-2">Try the full example in Playground</a></li> |
| </ul> |
| </li> |
| <li><a href="#windowedwordcount-example">WindowedWordCount example</a> |
| <ul> |
| <li><a href="#unbounded-and-bounded-datasets">Unbounded and bounded datasets</a></li> |
| <li><a href="#adding-timestamps-to-data">Adding timestamps to data</a></li> |
| <li><a href="#windowing">Windowing</a></li> |
| <li><a href="#reusing-ptransforms-over-windowed-pcollections">Reusing PTransforms over windowed PCollections</a></li> |
| </ul> |
| </li> |
| <li><a href="#streamingwordcount-example">StreamingWordCount example</a> |
| <ul> |
| <li><a href="#reading-an-unbounded-dataset">Reading an unbounded dataset</a></li> |
| <li><a href="#writing-unbounded-results">Writing unbounded results</a></li> |
| </ul> |
| </li> |
| <li><a href="#next-steps">Next Steps</a></li> |
| </ul> |
| </nav> |
| <nav class="language-switcher"> |
| <strong>Adapt for:</strong> |
| <ul> |
| <li data-value="java" class="active">Java SDK</li> |
| <li data-value="py">Python SDK</li> |
| <li data-value="go">Go SDK</li> |
| </ul> |
| </nav> |
| <p>The WordCount examples demonstrate how to set up a processing pipeline that can |
| read text, tokenize the text lines into individual words, and perform a |
| frequency count on each of those words. The Beam SDKs contain a series of these |
| four successively more detailed WordCount examples that build on each other. The |
| input text for all the examples is a set of Shakespeare&rsquo;s texts.</p> |
| <p>Each WordCount example introduces different concepts in the Beam programming |
| model. Begin by understanding MinimalWordCount, the simplest of the examples. |
| Once you feel comfortable with the basic principles in building a pipeline, |
| continue on to learn more concepts in the other examples.</p> |
| <ul> |
| <li><strong>MinimalWordCount</strong> demonstrates the basic principles involved in building a |
| pipeline.</li> |
| <li><strong>WordCount</strong> introduces some of the more common best practices in creating |
| re-usable and maintainable pipelines.</li> |
| <li><strong>DebuggingWordCount</strong> introduces logging and debugging practices.</li> |
| <li><strong>WindowedWordCount</strong> demonstrates how you can use Beam&rsquo;s programming model |
| to handle both bounded and unbounded datasets.</li> |
| </ul> |
| <h2 id="minimalwordcount-example">MinimalWordCount example</h2> |
| <p>MinimalWordCount demonstrates a simple pipeline that uses the Direct Runner to |
| read from a text file, apply transforms to tokenize and count the words, and |
| write the data to an output text file.</p> |
| <p class="language-java language-go">This example hard-codes the locations for its input and output files and doesn&rsquo;t |
| perform any error checking; it is intended to only show you the &ldquo;bare bones&rdquo; of |
| creating a Beam pipeline. This lack of parameterization makes this particular |
| pipeline less portable across different runners than standard Beam pipelines. In |
| later examples, we will parameterize the pipeline&rsquo;s input and output sources and |
| show other best practices.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">$</span> <span class="n">mvn</span> <span class="n">compile</span> <span class="n">exec</span><span class="o">:</span><span class="n">java</span> <span class="o">-</span><span class="n">Dexec</span><span class="o">.</span><span class="na">mainClass</span><span class="o">=</span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">beam</span><span class="o">.</span><span class="na">examples</span><span class="o">.</span><span class="na">MinimalWordCount</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">apache_beam</span><span class="o">.</span><span class="n">examples</span><span class="o">.</span><span class="n">wordcount_minimal</span> <span class="o">--</span><span class="nb">input</span> <span class="n">YOUR_INPUT_FILE</span> <span class="o">--</span><span class="n">output</span> <span class="n">counts</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="err">$</span> <span class="k">go</span> <span class="nx">install</span> <span class="nx">github</span><span class="p">.</span><span class="nx">com</span><span class="o">/</span><span class="nx">apache</span><span class="o">/</span><span class="nx">beam</span><span class="o">/</span><span class="nx">sdks</span><span class="o">/</span><span class="nx">v2</span><span class="o">/</span><span class="k">go</span><span class="o">/</span><span class="nx">examples</span><span class="o">/</span><span class="nx">minimal_wordcount</span> |
| </span></span><span class="line"><span class="cl"><span class="err">$</span> <span class="nx">minimal_wordcount</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">To view the full code in Java, see |
| <strong><a href="https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java">MinimalWordCount</a>.</strong></p> |
| <p class="language-py">To view the full code in Python, see |
| <strong><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_minimal.py">wordcount_minimal.py</a>.</strong></p> |
| <p class="language-go">To view the full code in Go, see |
| <strong><a href="https://github.com/apache/beam/blob/master/sdks/go/examples/minimal_wordcount/minimal_wordcount.go">minimal_wordcount.go</a>.</strong></p> |
| <p><strong>Key Concepts:</strong></p> |
| <ul> |
| <li>Creating the Pipeline</li> |
| <li>Applying transforms to the Pipeline</li> |
| <li>Reading input (in this example: reading text files)</li> |
| <li>Applying ParDo transforms</li> |
| <li>Applying SDK-provided transforms (in this example: Count)</li> |
| <li>Writing output (in this example: writing to a text file)</li> |
| <li>Running the Pipeline</li> |
| </ul> |
| <p>The following sections explain these concepts in detail, using the relevant code |
| excerpts from the MinimalWordCount pipeline.</p> |
| <h3 id="creating-the-pipeline">Creating the pipeline</h3> |
| <p class="language-java language-py">In this example, the code first creates a <code>PipelineOptions</code> object. This object |
| lets us set various options for our pipeline, such as the pipeline runner that |
| will execute our pipeline and any runner-specific configuration required by the |
| chosen runner. In this example we set these options programmatically, but more |
| often, command-line arguments are used to set <code>PipelineOptions</code>.</p> |
| <p class="language-java language-py">You can specify a runner for executing your pipeline, such as the |
| <code>DataflowRunner</code> or <code>SparkRunner</code>. If you omit specifying a runner, as in this |
| example, your pipeline executes locally using the <code>DirectRunner</code>. In the next |
| sections, we will specify the pipeline&rsquo;s runner.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="c1">// Create a PipelineOptions object. This object lets us set various execution |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// options for our pipeline, such as the runner you wish to use. This example |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// will run with the DirectRunner by default, based on the class path configured |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="c1">// in its dependencies. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">PipelineOptions</span> <span class="n">options</span> <span class="o">=</span> <span class="n">PipelineOptionsFactory</span><span class="o">.</span><span class="na">create</span><span class="o">();</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.options.pipeline_options</span> <span class="kn">import</span> <span class="n">PipelineOptions</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">input_file</span> <span class="o">=</span> <span class="s1">&#39;gs://dataflow-samples/shakespeare/kinglear.txt&#39;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">output_path</span> <span class="o">=</span> <span class="s1">&#39;gs://my-bucket/counts.txt&#39;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">beam_options</span> <span class="o">=</span> <span class="n">PipelineOptions</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">runner</span><span class="o">=</span><span class="s1">&#39;DataflowRunner&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">project</span><span class="o">=</span><span class="s1">&#39;my-project-id&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">job_name</span><span class="o">=</span><span class="s1">&#39;unique-job-name&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">temp_location</span><span class="o">=</span><span class="s1">&#39;gs://my-bucket/temp&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java language-py">The next step is to create a <code>Pipeline</code> object with the options we&rsquo;ve just |
| constructed. The Pipeline object builds up the graph of transformations to be |
| executed, associated with that particular pipeline.</p> |
| <p class="language-go">The first step is to create a <code>Pipeline</code> object. It builds up the graph of |
| transformations to be executed, associated with that particular pipeline. |
| The scope allows grouping into composite transforms.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">Pipeline</span> <span class="n">p</span> <span class="o">=</span> <span class="n">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">pipeline</span> <span class="o">=</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">(</span><span class="n">options</span><span class="o">=</span><span class="n">beam_options</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">p</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">NewPipeline</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"><span class="nx">s</span> <span class="o">:=</span> <span class="nx">p</span><span class="p">.</span><span class="nf">Root</span><span class="p">()</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="applying-pipeline-transforms">Applying pipeline transforms</h3> |
| <p>The MinimalWordCount pipeline contains several transforms to read data into the |
| pipeline, manipulate or otherwise transform the data, and write out the results. |
| Transforms can consist of an individual operation, or can contain multiple |
| nested transforms (which is a <a href="/documentation/programming-guide#composite-transforms">composite transform</a>).</p> |
| <p>Each transform takes some kind of input data and produces some output data. The |
| input and output data is often represented by the SDK class <code>PCollection</code>. |
| <code>PCollection</code> is a special class, provided by the Beam SDK, that you can use to |
| represent a dataset of virtually any size, including unbounded datasets.</p> |
| <img src="/images/wordcount-pipeline.svg" width="800px" alt="The MinimalWordCount pipeline data flow."> |
| <p><em>Figure 1: The MinimalWordCount pipeline data flow.</em></p> |
| <p>The MinimalWordCount pipeline contains five transforms:</p> |
| <ol> |
| <li>A text file <code>Read</code> transform is applied to the <code>Pipeline</code> object itself, and |
| produces a <code>PCollection</code> as output. Each element in the output <code>PCollection</code> |
| represents one line of text from the input file. This example uses input |
| data stored in a publicly accessible Google Cloud Storage bucket (&ldquo;gs://&rdquo;).</li> |
| </ol> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">TextIO</span><span class="o">.</span><span class="na">read</span><span class="o">().</span><span class="na">from</span><span class="o">(</span><span class="s">&#34;gs://apache-beam-samples/shakespeare/*&#34;</span><span class="o">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"><span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ReadFromText</span><span class="p">(</span><span class="n">input_file</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">lines</span> <span class="o">:=</span> <span class="nx">textio</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="s">&#34;gs://apache-beam-samples/shakespeare/*&#34;</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <ol start="2"> |
| <li>This transform splits the lines in <code>PCollection&lt;String&gt;</code>, where each element |
| is an individual word in Shakespeare&rsquo;s collected texts. |
| As an alternative, it would have been possible to use a |
| <a href="/documentation/programming-guide/#pardo">ParDo</a> |
| transform that invokes a <code>DoFn</code> (defined in-line as an anonymous class) on |
| each element that tokenizes the text lines into individual words. The input |
| for this transform is the <code>PCollection</code> of text lines generated by the |
| previous <code>TextIO.Read</code> transform. The <code>ParDo</code> transform outputs a new |
| <code>PCollection</code>, where each element represents an individual word in the text.</li> |
| </ol> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;ExtractWords&#34;</span><span class="o">,</span> <span class="n">FlatMapElements</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">via</span><span class="o">((</span><span class="n">String</span> <span class="n">line</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="n">line</span><span class="o">.</span><span class="na">split</span><span class="o">(</span><span class="s">&#34;[^\\p{L}]+&#34;</span><span class="o">))))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># The Flatmap transform is a simplified version of ParDo.</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="o">|</span> <span class="s1">&#39;ExtractWords&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="sa">r</span><span class="s1">&#39;[A-Za-z</span><span class="se">\&#39;</span><span class="s1">]+&#39;</span><span class="p">,</span> <span class="n">x</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">words</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="kd">func</span><span class="p">(</span><span class="nx">line</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">word</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">wordRE</span><span class="p">.</span><span class="nf">FindAllString</span><span class="p">(</span><span class="nx">line</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emit</span><span class="p">(</span><span class="nx">word</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">},</span> <span class="nx">lines</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <ol start="3"> |
| <li> |
| <p>The SDK-provided <code>Count</code> transform is a generic transform that takes a |
| <code>PCollection</code> of any type, and returns a <code>PCollection</code> of key/value pairs. |
| Each key represents a unique element from the input collection, and each |
| value represents the number of times that key appeared in the input |
| collection.</p> |
| <p>In this pipeline, the input for <code>Count</code> is the <code>PCollection</code> of individual |
| words generated by the previous <code>ParDo</code>, and the output is a <code>PCollection</code> |
| of key/value pairs where each key represents a unique word in the text and |
| the associated value is the occurrence count for each.</p> |
| </li> |
| </ol> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Count</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">perElement</span><span class="o">())</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">combiners</span><span class="o">.</span><span class="n">Count</span><span class="o">.</span><span class="n">PerElement</span><span class="p">()</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">counted</span> <span class="o">:=</span> <span class="nx">stats</span><span class="p">.</span><span class="nf">Count</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">words</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <ol start="4"> |
| <li> |
| <p>The next transform formats each of the key/value pairs of unique words and |
| occurrence counts into a printable string suitable for writing to an output |
| file.</p> |
| <p>The map transform is a higher-level composite transform that encapsulates a |
| simple <code>ParDo</code>. For each element in the input <code>PCollection</code>, the map |
| transform applies a function that produces exactly one output element.</p> |
| </li> |
| </ol> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;FormatResults&#34;</span><span class="o">,</span> <span class="n">MapElements</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">via</span><span class="o">((</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;</span> <span class="n">wordCount</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">wordCount</span><span class="o">.</span><span class="na">getKey</span><span class="o">()</span> <span class="o">+</span> <span class="s">&#34;: &#34;</span> <span class="o">+</span> <span class="n">wordCount</span><span class="o">.</span><span class="na">getValue</span><span class="o">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">MapTuple</span><span class="p">(</span><span class="k">lambda</span> <span class="n">word</span><span class="p">,</span> <span class="n">count</span><span class="p">:</span> <span class="s1">&#39;</span><span class="si">%s</span><span class="s1">: </span><span class="si">%s</span><span class="s1">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">word</span><span class="p">,</span> <span class="n">count</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">formatted</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="kd">func</span><span class="p">(</span><span class="nx">w</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">c</span> <span class="kt">int</span><span class="p">)</span> <span class="kt">string</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">fmt</span><span class="p">.</span><span class="nf">Sprintf</span><span class="p">(</span><span class="s">&#34;%s: %v&#34;</span><span class="p">,</span> <span class="nx">w</span><span class="p">,</span> <span class="nx">c</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">},</span> <span class="nx">counted</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <ol start="5"> |
| <li>A text file write transform. This transform takes the final <code>PCollection</code> of |
| formatted Strings as input and writes each element to an output text file. |
| Each element in the input <code>PCollection</code> represents one line of text in the |
| resulting output file.</li> |
| </ol> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">TextIO</span><span class="o">.</span><span class="na">write</span><span class="o">().</span><span class="na">to</span><span class="o">(</span><span class="s">&#34;wordcounts&#34;</span><span class="o">));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">WriteToText</span><span class="p">(</span><span class="n">output_path</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">textio</span><span class="p">.</span><span class="nf">Write</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="s">&#34;wordcounts.txt&#34;</span><span class="p">,</span> <span class="nx">formatted</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java language-py">Note that the <code>Write</code> transform produces a trivial result value of type <code>PDone</code>, |
| which in this case is ignored.</p> |
| <p class="language-go">Note that the <code>Write</code> transform returns no PCollections.</p> |
| <h3 id="running-the-pipeline">Running the pipeline</h3> |
| <p class="language-java language-py">Run the pipeline by calling the <code>run</code> method, which sends your pipeline to be |
| executed by the pipeline runner that you specified in your <code>PipelineOptions</code>.</p> |
| <p class="language-go">Run the pipeline by passing it to a runner.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">p</span><span class="o">.</span><span class="na">run</span><span class="o">().</span><span class="na">waitUntilFinish</span><span class="o">();</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">(</span><span class="o">...</span><span class="p">)</span> <span class="k">as</span> <span class="n">p</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">[</span><span class="n">construction</span><span class="p">]</span> |
| </span></span><span class="line"><span class="cl"><span class="c1"># p.run() automatically called</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">direct</span><span class="p">.</span><span class="nf">Execute</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nf">Background</span><span class="p">(),</span> <span class="nx">p</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java language-py">Note that the <code>run</code> method is asynchronous. For a blocking execution, call the |
| <span class="language-java"><code>waitUntilFinish</code></span> |
| <span class="language-py"><code>wait_until_finish</code></span> method on the result object |
| returned by the call to <code>run</code>.</p> |
| <h3 id="try-the-full-example-in-playground">Try the full example in Playground</h3> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-java playground-snippet" |
| data-sdk="java" |
| data-path="SDK_JAVA_MinimalWordCount" |
| ></div> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_WordCountMinimal" |
| ></div> |
| <div |
| class="language-go playground-snippet" |
| data-sdk="go" |
| data-path="SDK_GO_MinimalWordCount" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_JAVA_MinimalWordCount%22%2c%22sdk%22%3a%22java%22%7d%2c%7b%22path%22%3a%22SDK_PYTHON_WordCountMinimal%22%2c%22sdk%22%3a%22python%22%7d%2c%7b%22path%22%3a%22SDK_GO_MinimalWordCount%22%2c%22sdk%22%3a%22go%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="wordcount-example">WordCount example</h2> |
| <p>This WordCount example introduces a few recommended programming practices that |
| can make your pipeline easier to read, write, and maintain. While not explicitly |
| required, they can make your pipeline&rsquo;s execution more flexible, aid in testing |
| your pipeline, and help make your pipeline&rsquo;s code reusable.</p> |
| <p>This section assumes that you have a good understanding of the basic concepts in |
| building a pipeline. If you feel that you aren&rsquo;t at that point yet, read the |
| above section, <a href="#minimalwordcount-example">MinimalWordCount</a>.</p> |
| <p><strong>To run this example in Java:</strong></p> |
| <p>Set up your development environment and generate the Maven archetype as |
| described in the <a href="/get-started/quickstart-java/">Java WordCount quickstart</a>. |
| Then run the pipeline with one of the runners:</p> |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ |
| -Dexec.args=&#34;--inputFile=pom.xml --output=counts&#34; -Pdirect-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-flink snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flink" data-lang="flink">$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ |
| -Dexec.args=&#34;--runner=FlinkRunner --inputFile=pom.xml --output=counts&#34; -Pflink-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-flinkCluster snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flinkCluster" data-lang="flinkCluster">$ mvn package exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ |
| -Dexec.args=&#34;--runner=FlinkRunner --flinkMaster=&lt;flink master&gt; --filesToStage=target/word-count-beam-bundled-0.1.jar \ |
| --inputFile=/path/to/quickstart/pom.xml --output=/tmp/counts&#34; -Pflink-runner |
| You can monitor the running job by visiting the Flink dashboard at http://&lt;flink master&gt;:8081</code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark">$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ |
| -Dexec.args=&#34;--runner=SparkRunner --inputFile=pom.xml --output=counts&#34; -Pspark-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow">$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ |
| -Dexec.args=&#34;--runner=DataflowRunner --gcpTempLocation=gs://YOUR_GCS_BUCKET/tmp \ |
| --project=YOUR_PROJECT --region=GCE_REGION \ |
| --inputFile=gs://apache-beam-samples/shakespeare/* --output=gs://YOUR_GCS_BUCKET/counts&#34; \ |
| -Pdataflow-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-samza snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-samza" data-lang="samza">$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ |
| -Dexec.args=&#34;--inputFile=pom.xml --output=counts --runner=SamzaRunner&#34; -Psamza-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-nemo snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-nemo" data-lang="nemo">$ mvn package -Pnemo-runner &amp;&amp; java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.WordCount \ |
| --runner=NemoRunner --inputFile=`pwd`/pom.xml --output=counts</code></pre> |
| </div> |
| </div> |
| <div class='runner-jet snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-jet" data-lang="jet">$ mvn package -P jet-runner &amp;&amp; java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.WordCount \ |
| --runner=JetRunner --jetLocalMode=3 --inputFile=`pwd`/pom.xml --output=counts</code></pre> |
| </div> |
| </div> |
| <p>To view the full code in Java, see |
| <strong><a href="https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WordCount.java">WordCount</a>.</strong></p> |
| <p><strong>To run this example in Python:</strong></p> |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">python -m apache_beam.examples.wordcount --input YOUR_INPUT_FILE --output counts</code></pre> |
| </div> |
| </div> |
| <div class='runner-flink snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flink" data-lang="flink">python -m apache_beam.examples.wordcount --input /path/to/inputfile \ |
| --output /path/to/write/counts \ |
| --runner FlinkRunner</code></pre> |
| </div> |
| </div> |
| <div class='runner-flinkCluster snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flinkCluster" data-lang="flinkCluster"># Running Beam Python on a distributed Flink cluster requires additional configuration. |
| # See /documentation/runners/flink/ for more information.</code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark">python -m apache_beam.examples.wordcount --input /path/to/inputfile \ |
| --output /path/to/write/counts \ |
| --runner SparkRunner</code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow"># As part of the initial setup, install Google Cloud Platform specific extra components. |
| pip install apache-beam[gcp] |
| python -m apache_beam.examples.wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \ |
| --output gs://YOUR_GCS_BUCKET/counts \ |
| --runner DataflowRunner \ |
| --project YOUR_GCP_PROJECT \ |
| --region YOUR_GCP_REGION \ |
| --temp_location gs://YOUR_GCS_BUCKET/tmp/</code></pre> |
| </div> |
| </div> |
| <div class='runner-samza snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-samza" data-lang="samza">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-nemo snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-nemo" data-lang="nemo">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-jet snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-jet" data-lang="jet">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <p>To view the full code in Python, see |
| <strong><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount.py">wordcount.py</a>.</strong></p> |
| <p><strong>To run this example in Go:</strong></p> |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">$ go install github.com/apache/beam/sdks/v2/go/examples/wordcount |
| $ wordcount --input &lt;PATH_TO_INPUT_FILE&gt; --output counts</code></pre> |
| </div> |
| </div> |
| <div class='runner-flink snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flink" data-lang="flink">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-flinkCluster snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flinkCluster" data-lang="flinkCluster">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow">$ go install github.com/apache/beam/sdks/v2/go/examples/wordcount |
| # As part of the initial setup, for non linux users - install package unix before run |
| $ go get -u golang.org/x/sys/unix |
| $ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \ |
| --output gs://&lt;your-gcs-bucket&gt;/counts \ |
| --runner dataflow \ |
| --project your-gcp-project \ |
| --region your-gcp-region \ |
| --temp_location gs://&lt;your-gcs-bucket&gt;/tmp/ \ |
| --staging_location gs://&lt;your-gcs-bucket&gt;/binaries/ \ |
| --worker_harness_container_image=apache/beam_go_sdk:latest</code></pre> |
| </div> |
| </div> |
| <div class='runner-samza snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-samza" data-lang="samza">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-nemo snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-nemo" data-lang="nemo">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-jet snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-jet" data-lang="jet">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <p>To view the full code in Go, see |
| <strong><a href="https://github.com/apache/beam/blob/master/sdks/go/examples/wordcount/wordcount.go">wordcount.go</a>.</strong></p> |
| <p><strong>New Concepts:</strong></p> |
| <ul> |
| <li>Applying <code>ParDo</code> with an explicit <code>DoFn</code></li> |
| <li>Creating Composite Transforms</li> |
| <li>Using Parameterizable <code>PipelineOptions</code></li> |
| </ul> |
| <p>The following sections explain these key concepts in detail, and break down the |
| pipeline code into smaller sections.</p> |
| <h3 id="specifying-explicit-dofns">Specifying explicit DoFns</h3> |
| <p class="language-java language-py">When using <code>ParDo</code> transforms, you need to specify the processing operation that |
| gets applied to each element in the input <code>PCollection</code>. This processing |
| operation is a subclass of the SDK class <code>DoFn</code>. You can create the <code>DoFn</code> |
| subclasses for each <code>ParDo</code> inline, as an anonymous inner class instance, as is |
| done in the previous example (MinimalWordCount). However, it&rsquo;s often a good |
| idea to define the <code>DoFn</code> at the global level, which makes it easier to unit |
| test and can make the <code>ParDo</code> code more readable.</p> |
| <p class="language-go">When using <code>ParDo</code> transforms, you need to specify the processing operation that |
| gets applied to each element in the input <code>PCollection</code>. This processing |
| operation is either a named function or a struct with specially-named methods. You |
| can use anonymous functions (but not closures). However, it&rsquo;s often a good |
| idea to define the <code>DoFn</code> at the global level, which makes it easier to unit |
| test and can make the <code>ParDo</code> code more readable.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// In this example, ExtractWordsFn is a DoFn that is defined as a static class: |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"><span class="kd">static</span> <span class="kd">class</span> <span class="nc">ExtractWordsFn</span> <span class="kd">extends</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># In this example, the DoFns are defined as classes:</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">FormatAsTextFn</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">word</span><span class="p">,</span> <span class="n">count</span> <span class="o">=</span> <span class="n">element</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="s1">&#39;</span><span class="si">%s</span><span class="s1">: </span><span class="si">%s</span><span class="s1">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">word</span><span class="p">,</span> <span class="n">count</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">formatted</span> <span class="o">=</span> <span class="n">counts</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">ParDo</span><span class="p">(</span><span class="n">FormatAsTextFn</span><span class="p">())</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// In this example, extractFn is a DoFn that is defined as a function: |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">extractFn</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">line</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="creating-composite-transforms">Creating composite transforms</h3> |
| <p class="language-java language-py">If you have a processing operation that consists of multiple transforms or |
| <code>ParDo</code> steps, you can create it as a subclass of <code>PTransform</code>. Creating a |
| <code>PTransform</code> subclass allows you to encapsulate complex transforms, can make |
| your pipeline&rsquo;s structure more clear and modular, and makes unit testing easier.</p> |
| <p class="language-go">If you have a processing operation that consists of multiple transforms or |
| <code>ParDo</code> steps, you can use a normal Go function to encapsulate them. You can |
| furthermore use a named subscope to group them as a composite transform visible |
| for monitoring.</p> |
| <p class="language-java language-py">In this example, two transforms are encapsulated as the <code>PTransform</code> subclass |
| <code>CountWords</code>. <code>CountWords</code> contains the <code>ParDo</code> that runs <code>ExtractWordsFn</code> and |
| the SDK-provided <code>Count</code> transform.</p> |
| <p class="language-go">In this example, two transforms are encapsulated as a <code>CountWords</code> function.</p> |
| <p>When <code>CountWords</code> is defined, we specify its ultimate input and output; the |
| input is the <code>PCollection&lt;String&gt;</code> for the extraction operation, and the output |
| is the <code>PCollection&lt;KV&lt;String, Long&gt;&gt;</code> produced by the count operation.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">CountWords</span> <span class="kd">extends</span> <span class="n">PTransform</span><span class="o">&lt;</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;&gt;&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Override</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;&gt;</span> <span class="nf">expand</span><span class="o">(</span><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">lines</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Convert lines of text into individual words. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">words</span> <span class="o">=</span> <span class="n">lines</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">ExtractWordsFn</span><span class="o">()));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Count the number of times each word occurs. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;&gt;</span> <span class="n">wordCounts</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">words</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Count</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">perElement</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">wordCounts</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Pipeline</span> <span class="n">p</span> <span class="o">=</span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">p</span><span class="o">.</span><span class="na">apply</span><span class="o">(...)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="k">new</span> <span class="n">CountWords</span><span class="o">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="nd">@beam.ptransform_fn</span> |
| </span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">CountWords</span><span class="p">(</span><span class="n">pcoll</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pcoll</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Convert lines of text into individual words.</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;ExtractWords&#39;</span> <span class="o">&gt;&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="sa">r</span><span class="s1">&#39;[A-Za-z</span><span class="se">\&#39;</span><span class="s1">]+&#39;</span><span class="p">,</span> <span class="n">x</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Count the number of times each word occurs.</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">combiners</span><span class="o">.</span><span class="n">Count</span><span class="o">.</span><span class="n">PerElement</span><span class="p">())</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">counts</span> <span class="o">=</span> <span class="n">lines</span> <span class="o">|</span> <span class="n">CountWords</span><span class="p">()</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">CountWords</span><span class="p">(</span><span class="nx">s</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">Scope</span><span class="p">,</span> <span class="nx">lines</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span><span class="p">)</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">PCollection</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">s</span> <span class="p">=</span> <span class="nx">s</span><span class="p">.</span><span class="nf">Scope</span><span class="p">(</span><span class="s">&#34;CountWords&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Convert lines of text into individual words. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">col</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">extractFn</span><span class="p">,</span> <span class="nx">lines</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Count the number of times each word occurs. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="k">return</span> <span class="nx">stats</span><span class="p">.</span><span class="nf">Count</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">col</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="using-parameterizable-pipelineoptions">Using parameterizable PipelineOptions</h3> |
| <p>You can hard-code various execution options when you run your pipeline. However, |
| the more common way is to define your own configuration options via command-line |
| argument parsing. Defining your configuration options via the command-line makes |
| the code more easily portable across different runners.</p> |
| <p class="language-java language-py">Add arguments to be processed by the command-line parser, and specify default |
| values for them. You can then access the options values in your pipeline code.</p> |
| <p class="language-go">You can use the standard <code>flag</code> package for this purpose.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">static</span> <span class="kd">interface</span> <span class="nc">WordCountOptions</span> <span class="kd">extends</span> <span class="n">PipelineOptions</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Description</span><span class="o">(</span><span class="s">&#34;Path of the file to read from&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@Default.String</span><span class="o">(</span><span class="s">&#34;gs://dataflow-samples/shakespeare/kinglear.txt&#34;</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">String</span> <span class="nf">getInputFile</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="kt">void</span> <span class="nf">setInputFile</span><span class="o">(</span><span class="n">String</span> <span class="n">value</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">WordCountOptions</span> <span class="n">options</span> <span class="o">=</span> <span class="n">PipelineOptionsFactory</span><span class="o">.</span><span class="na">fromArgs</span><span class="o">(</span><span class="n">args</span><span class="o">).</span><span class="na">withValidation</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">as</span><span class="o">(</span><span class="n">WordCountOptions</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Pipeline</span> <span class="n">p</span> <span class="o">=</span> <span class="n">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">argparse</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">parser</span> <span class="o">=</span> <span class="n">argparse</span><span class="o">.</span><span class="n">ArgumentParser</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"><span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;--input-file&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">default</span><span class="o">=</span><span class="s1">&#39;gs://dataflow-samples/shakespeare/kinglear.txt&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;The file path for the input text to process.&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s1">&#39;--output-path&#39;</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s1">&#39;The path prefix for output files.&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="n">args</span><span class="p">,</span> <span class="n">beam_args</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_known_args</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">beam_options</span> <span class="o">=</span> <span class="n">PipelineOptions</span><span class="p">(</span><span class="n">beam_args</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">(</span><span class="n">options</span><span class="o">=</span><span class="n">beam_options</span><span class="p">)</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">lines</span> <span class="o">=</span> <span class="n">pipeline</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ReadFromText</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">input_file</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">var</span> <span class="nx">input</span> <span class="p">=</span> <span class="nx">flag</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="s">&#34;input&#34;</span><span class="p">,</span> <span class="s">&#34;gs://apache-beam-samples/shakespeare/kinglear.txt&#34;</span><span class="p">,</span> <span class="s">&#34;File(s) to read.&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">p</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">NewPipeline</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">s</span> <span class="o">:=</span> <span class="nx">p</span><span class="p">.</span><span class="nf">Root</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nx">lines</span> <span class="o">:=</span> <span class="nx">textio</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">*</span><span class="nx">input</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="try-the-full-example-in-playground-1">Try the full example in Playground</h3> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-java playground-snippet" |
| data-sdk="java" |
| data-path="SDK_JAVA_WordCount" |
| ></div> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_WordCount" |
| ></div> |
| <div |
| class="language-go playground-snippet" |
| data-sdk="go" |
| data-path="SDK_GO_WordCount" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_JAVA_WordCount%22%2c%22sdk%22%3a%22java%22%7d%2c%7b%22path%22%3a%22SDK_PYTHON_WordCount%22%2c%22sdk%22%3a%22python%22%7d%2c%7b%22path%22%3a%22SDK_GO_WordCount%22%2c%22sdk%22%3a%22go%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="debuggingwordcount-example">DebuggingWordCount example</h2> |
| <p>The DebuggingWordCount example demonstrates some best practices for |
| instrumenting your pipeline code.</p> |
| <p><strong>To run this example in Java:</strong></p> |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \ |
| -Dexec.args=&#34;--output=counts&#34; -Pdirect-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-flink snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flink" data-lang="flink">$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \ |
| -Dexec.args=&#34;--runner=FlinkRunner --output=counts&#34; -Pflink-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-flinkCluster snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flinkCluster" data-lang="flinkCluster">$ mvn package exec:java -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \ |
| -Dexec.args=&#34;--runner=FlinkRunner --flinkMaster=&lt;flink master&gt; --filesToStage=target/word-count-beam-bundled-0.1.jar \ |
| --output=/tmp/counts&#34; -Pflink-runner |
| You can monitor the running job by visiting the Flink dashboard at http://&lt;flink master&gt;:8081</code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark">$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \ |
| -Dexec.args=&#34;--runner=SparkRunner --output=counts&#34; -Pspark-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow">$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \ |
| -Dexec.args=&#34;--runner=DataflowRunner --gcpTempLocation=gs://&lt;your-gcs-bucket&gt;/tmp \ |
| --project=YOUR_PROJECT --region=GCE_REGION \ |
| --inputFile=gs://apache-beam-samples/shakespeare/* --output=gs://&lt;your-gcs-bucket&gt;/counts&#34; \ |
| -Pdataflow-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-samza snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-samza" data-lang="samza">$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \ |
| -Dexec.args=&#34;--runner=SamzaRunner --output=counts&#34; -Psamza-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-nemo snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-nemo" data-lang="nemo">$ mvn package -Pnemo-runner &amp;&amp; java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.DebuggingWordCount \ |
| --runner=NemoRunner --inputFile=`pwd`/pom.xml --output=counts</code></pre> |
| </div> |
| </div> |
| <div class='runner-jet snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-jet" data-lang="jet">$ mvn package -P jet-runner &amp;&amp; java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.DebuggingWordCount \ |
| --runner=JetRunner --jetLocalMode=3 --output=counts</code></pre> |
| </div> |
| </div> |
| <p>To view the full code in Java, see |
| <a href="https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java">DebuggingWordCount</a>.</p> |
| <p><strong>To run this example in Python:</strong></p> |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">python -m apache_beam.examples.wordcount_debugging --input YOUR_INPUT_FILE --output counts</code></pre> |
| </div> |
| </div> |
| <div class='runner-flink snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flink" data-lang="flink">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-flinkCluster snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flinkCluster" data-lang="flinkCluster">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow"># As part of the initial setup, install Google Cloud Platform specific extra components. |
| pip install apache-beam[gcp] |
| python -m apache_beam.examples.wordcount_debugging --input gs://dataflow-samples/shakespeare/kinglear.txt \ |
| --output gs://YOUR_GCS_BUCKET/counts \ |
| --runner DataflowRunner \ |
| --project YOUR_GCP_PROJECT \ |
| --temp_location gs://YOUR_GCS_BUCKET/tmp/</code></pre> |
| </div> |
| </div> |
| <div class='runner-samza snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-samza" data-lang="samza">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-nemo snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-nemo" data-lang="nemo">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-jet snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-jet" data-lang="jet">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <p>To view the full code in Python, see |
| <strong><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_debugging.py">wordcount_debugging.py</a>.</strong></p> |
| <p><strong>To run this example in Go:</strong></p> |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">$ go install github.com/apache/beam/sdks/v2/go/examples/debugging_wordcount |
| $ debugging_wordcount --input &lt;PATH_TO_INPUT_FILE&gt; --output counts</code></pre> |
| </div> |
| </div> |
| <div class='runner-flink snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flink" data-lang="flink">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-flinkCluster snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flinkCluster" data-lang="flinkCluster">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow">$ go install github.com/apache/beam/sdks/v2/go/examples/debugging_wordcount |
| # As part of the initial setup, for non linux users - install package unix before run |
| $ go get -u golang.org/x/sys/unix |
| $ debugging_wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \ |
| --output gs://&lt;your-gcs-bucket&gt;/counts \ |
| --runner dataflow \ |
| --project your-gcp-project \ |
| --region your-gcp-region \ |
| --temp_location gs://&lt;your-gcs-bucket&gt;/tmp/ \ |
| --staging_location gs://&lt;your-gcs-bucket&gt;/binaries/ \ |
| --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515</code></pre> |
| </div> |
| </div> |
| <div class='runner-samza snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-samza" data-lang="samza">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-nemo snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-nemo" data-lang="nemo">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-jet snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-jet" data-lang="jet">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <p>To view the full code in Go, see |
| <strong><a href="https://github.com/apache/beam/blob/master/sdks/go/examples/debugging_wordcount/debugging_wordcount.go">debugging_wordcount.go</a>.</strong></p> |
| <p><strong>New Concepts:</strong></p> |
| <ul> |
| <li>Logging</li> |
| <li>Testing your Pipeline via <code>PAssert</code></li> |
| </ul> |
| <p>The following sections explain these key concepts in detail, and break down the |
| pipeline code into smaller sections.</p> |
| <h3 id="logging">Logging</h3> |
| <p>Each runner may choose to handle logs in its own way.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="c1">// This example uses .trace and .debug: |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">DebuggingWordCount</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">FilterTextFn</span> <span class="kd">extends</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;,</span> <span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="o">(...)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">LOG</span><span class="o">.</span><span class="na">debug</span><span class="o">(</span><span class="s">&#34;Matched: &#34;</span> <span class="o">+</span> <span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">getKey</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> <span class="k">else</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">LOG</span><span class="o">.</span><span class="na">trace</span><span class="o">(</span><span class="s">&#34;Did not match: &#34;</span> <span class="o">+</span> <span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">().</span><span class="na">getKey</span><span class="o">());</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="c1"># [START example_wordcount_debugging_aggregators]</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">logging</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">FilterTextFn</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="s2">&#34;&#34;&#34;A DoFn that filters for a specific key based on a regular expression.&#34;&#34;&#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pattern</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">pattern</span> <span class="o">=</span> <span class="n">pattern</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># A custom metric can track values in your pipeline as it runs. Create</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># custom metrics matched_word and unmatched_words.</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">matched_words</span> <span class="o">=</span> <span class="n">Metrics</span><span class="o">.</span><span class="n">counter</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="vm">__class__</span><span class="p">,</span> <span class="s1">&#39;matched_words&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">umatched_words</span> <span class="o">=</span> <span class="n">Metrics</span><span class="o">.</span><span class="n">counter</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="vm">__class__</span><span class="p">,</span> <span class="s1">&#39;umatched_words&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">word</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">element</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">re</span><span class="o">.</span><span class="k">match</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">pattern</span><span class="p">,</span> <span class="n">word</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Log at INFO level each element we match. When executing this pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># using the Dataflow service, these log lines will appear in the Cloud</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Logging UI.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">&#39;Matched </span><span class="si">%s</span><span class="s1">&#39;</span><span class="p">,</span> <span class="n">word</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Add 1 to the custom metric counter matched_words</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">matched_words</span><span class="o">.</span><span class="n">inc</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">yield</span> <span class="n">element</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">else</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Log at the &#34;DEBUG&#34; level each element that is not matched. Different</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># log levels can be used to control the verbosity of logging providing</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># an effective mechanism to filter less important information. Note</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># currently only &#34;INFO&#34; and higher level logs are emitted to the Cloud</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Logger. This log message will not be visible in the Cloud Logger.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">&#39;Did not match </span><span class="si">%s</span><span class="s1">&#39;</span><span class="p">,</span> <span class="n">word</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># Add 1 to the custom metric counter umatched_words</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">umatched_words</span><span class="o">.</span><span class="n">inc</span><span class="p">()</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">filterFn</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">f</span> <span class="o">*</span><span class="nx">filterFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">ctx</span> <span class="nx">context</span><span class="p">.</span><span class="nx">Context</span><span class="p">,</span> <span class="nx">word</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">count</span> <span class="kt">int</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">,</span> <span class="kt">int</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="nx">f</span><span class="p">.</span><span class="nx">re</span><span class="p">.</span><span class="nf">MatchString</span><span class="p">(</span><span class="nx">word</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Log at the &#34;INFO&#34; level each element that we match. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">log</span><span class="p">.</span><span class="nf">Infof</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="s">&#34;Matched: %v&#34;</span><span class="p">,</span> <span class="nx">word</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emit</span><span class="p">(</span><span class="nx">word</span><span class="p">,</span> <span class="nx">count</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="c1">// Log at the &#34;DEBUG&#34; level each element that is not matched. |
| </span></span></span><span class="line"><span class="cl"><span class="c1"></span> <span class="nx">log</span><span class="p">.</span><span class="nf">Debugf</span><span class="p">(</span><span class="nx">ctx</span><span class="p">,</span> <span class="s">&#34;Did not match: %v&#34;</span><span class="p">,</span> <span class="nx">word</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="direct-runner">Direct Runner</h4> |
| <p>When executing your pipeline with the <code>DirectRunner</code>, you can print log |
| messages directly to your local console. <span class="language-java">If you use |
| the Beam SDK for Java, you must add <code>Slf4j</code> to your class path.</span></p> |
| <h4 id="cloud-dataflow-runner">Cloud Dataflow Runner</h4> |
| <p>When executing your pipeline with the <code>DataflowRunner</code>, you can use Stackdriver |
| Logging. Stackdriver Logging aggregates the logs from all of your Cloud Dataflow |
| job&rsquo;s workers to a single location in the Google Cloud Platform Console. You can |
| use Stackdriver Logging to search and access the logs from all of the workers |
| that Cloud Dataflow has spun up to complete your job. Logging statements in your |
| pipeline&rsquo;s <code>DoFn</code> instances will appear in Stackdriver Logging as your pipeline |
| runs.</p> |
| <p class="language-java language-py">You can also control the worker log levels. Cloud Dataflow workers that execute |
| user code are configured to log to Stackdriver Logging by default at &ldquo;INFO&rdquo; log |
| level and higher. You can override log levels for specific logging namespaces by |
| specifying: <code>--workerLogLevelOverrides={&quot;Name1&quot;:&quot;Level1&quot;,&quot;Name2&quot;:&quot;Level2&quot;,...}</code>. |
| For example, by specifying <code>--workerLogLevelOverrides={&quot;org.apache.beam.examples&quot;:&quot;DEBUG&quot;}</code> |
| when executing a pipeline using the Cloud Dataflow service, Stackdriver Logging |
| will contain only &ldquo;DEBUG&rdquo; or higher level logs for the package in addition to |
| the default &ldquo;INFO&rdquo; or higher level logs.</p> |
| <p class="language-java language-py">The default Cloud Dataflow worker logging configuration can be overridden by |
| specifying <code>--defaultWorkerLogLevel=&lt;one of TRACE, DEBUG, INFO, WARN, ERROR&gt;</code>. |
| For example, by specifying <code>--defaultWorkerLogLevel=DEBUG</code> when executing a |
| pipeline with the Cloud Dataflow service, Cloud Logging will contain all &ldquo;DEBUG&rdquo; |
| or higher level logs. Note that changing the default worker log level to TRACE |
| or DEBUG significantly increases the amount of logs output.</p> |
| <h4 id="apache-spark-runner">Apache Spark Runner</h4> |
| <blockquote> |
| <p><strong>Note:</strong> This section is yet to be added. There is an open issue for this |
| (<a href="https://github.com/apache/beam/issues/18076">Issue 18076</a>).</p> |
| </blockquote> |
| <h4 id="apache-flink-runner">Apache Flink Runner</h4> |
| <blockquote> |
| <p><strong>Note:</strong> This section is yet to be added. There is an open issue for this |
| (<a href="https://github.com/apache/beam/issues/18075">Issue 18075</a>).</p> |
| </blockquote> |
| <h4 id="apache-nemo-runner">Apache Nemo Runner</h4> |
| <p>When executing your pipeline with the <code>NemoRunner</code>, most log messages are printed |
| directly to your local console. You should add <code>Slf4j</code> to your class path to make |
| full use of the logs. In order to observe the logs on each of the driver and the |
| executor sides, you should observe the folders created by Apache REEF. For example, |
| when running your pipeline through the local runtime, a folder called <code>REEF_LOCAL_RUNTIME</code> |
| will be created on your work directory, and the logs and the metric information can |
| all be found under the directory.</p> |
| <h3 id="testing-your-pipeline-with-asserts">Testing your pipeline with asserts</h3> |
| <p class="language-java language-py"><span class="language-java"><code>PAssert</code></span><span class="language-py"><code>assert_that</code></span> |
| is a set of convenient PTransforms in the style of Hamcrest&rsquo;s collection |
| matchers that can be used when writing pipeline level tests to validate the |
| contents of PCollections. Asserts are best used in unit tests with small datasets.</p> |
| <p class="language-go">The <code>passert</code> package contains convenient PTransforms that can be used when |
| writing pipeline level tests to validate the contents of PCollections. Asserts |
| are best used in unit tests with small datasets.</p> |
| <p class="language-java">The following example verifies that the set of filtered words matches our |
| expected counts. The assert does not produce any output, and the pipeline only |
| succeeds if all of the expectations are met.</p> |
| <p class="language-py language-go">The following example verifies that two collections contain the same values. The |
| assert does not produce any output, and the pipeline only succeeds if all of the |
| expectations are met.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">List</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;&gt;</span> <span class="n">expectedResults</span> <span class="o">=</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;Flourish&#34;</span><span class="o">,</span> <span class="n">3L</span><span class="o">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">KV</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="s">&#34;stomach&#34;</span><span class="o">,</span> <span class="n">1L</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">PAssert</span><span class="o">.</span><span class="na">that</span><span class="o">(</span><span class="n">filteredWords</span><span class="o">).</span><span class="na">containsInAnyOrder</span><span class="o">(</span><span class="n">expectedResults</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.testing.util</span> <span class="kn">import</span> <span class="n">assert_that</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">from</span> <span class="nn">apache_beam.testing.util</span> <span class="kn">import</span> <span class="n">equal_to</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">TestPipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">p</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">assert_that</span><span class="p">(</span><span class="n">p</span> <span class="o">|</span> <span class="n">Create</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]),</span> <span class="n">equal_to</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="nx">passert</span><span class="p">.</span><span class="nf">Equals</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">formatted</span><span class="p">,</span> <span class="s">&#34;Flourish: 3&#34;</span><span class="p">,</span> <span class="s">&#34;stomach: 1&#34;</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java">See <a href="https://github.com/apache/beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/DebuggingWordCountTest.java">DebuggingWordCountTest</a> |
| for an example unit test.</p> |
| <h3 id="try-the-full-example-in-playground-2">Try the full example in Playground</h3> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-java playground-snippet" |
| data-sdk="java" |
| data-path="SDK_JAVA_DebuggingWordCount" |
| ></div> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_WordCountDebugging" |
| ></div> |
| <div |
| class="language-go playground-snippet" |
| data-sdk="go" |
| data-path="SDK_GO_DebuggingWordCount" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_JAVA_DebuggingWordCount%22%2c%22sdk%22%3a%22java%22%7d%2c%7b%22path%22%3a%22SDK_PYTHON_WordCountDebugging%22%2c%22sdk%22%3a%22python%22%7d%2c%7b%22path%22%3a%22SDK_GO_DebuggingWordCount%22%2c%22sdk%22%3a%22go%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="windowedwordcount-example">WindowedWordCount example</h2> |
| <p>The WindowedWordCount example counts words in text just as the previous |
| examples did, but introduces several advanced concepts.</p> |
| <p><strong>New Concepts:</strong></p> |
| <ul> |
| <li>Unbounded and bounded datasets</li> |
| <li>Adding timestamps to data</li> |
| <li>Windowing</li> |
| <li>Reusing PTransforms over windowed PCollections</li> |
| </ul> |
| <p>The following sections explain these key concepts in detail, and break down the |
| pipeline code into smaller sections.</p> |
| <p><strong>To run this example in Java:</strong></p> |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WindowedWordCount \ |
| -Dexec.args=&#34;--inputFile=pom.xml --output=counts&#34; -Pdirect-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-flink snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flink" data-lang="flink">$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WindowedWordCount \ |
| -Dexec.args=&#34;--runner=FlinkRunner --inputFile=pom.xml --output=counts&#34; -Pflink-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-flinkCluster snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flinkCluster" data-lang="flinkCluster">$ mvn package exec:java -Dexec.mainClass=org.apache.beam.examples.WindowedWordCount \ |
| -Dexec.args=&#34;--runner=FlinkRunner --flinkMaster=&lt;flink master&gt; --filesToStage=target/word-count-beam-bundled-0.1.jar \ |
| --inputFile=/path/to/quickstart/pom.xml --output=/tmp/counts&#34; -Pflink-runner |
| You can monitor the running job by visiting the Flink dashboard at http://&lt;flink master&gt;:8081</code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark">$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WindowedWordCount \ |
| -Dexec.args=&#34;--runner=SparkRunner --inputFile=pom.xml --output=counts&#34; -Pspark-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow">$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WindowedWordCount \ |
| -Dexec.args=&#34;--runner=DataflowRunner --gcpTempLocation=gs://YOUR_GCS_BUCKET/tmp \ |
| --project=YOUR_PROJECT --region=GCE_REGION \ |
| --inputFile=gs://apache-beam-samples/shakespeare/* --output=gs://YOUR_GCS_BUCKET/counts&#34; \ |
| -Pdataflow-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-samza snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-samza" data-lang="samza">$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WindowedWordCount \ |
| -Dexec.args=&#34;--runner=SamzaRunner --inputFile=pom.xml --output=counts&#34; -Psamza-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-nemo snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-nemo" data-lang="nemo">$ mvn package -Pnemo-runner &amp;&amp; java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.WindowedWordCount \ |
| --runner=NemoRunner --inputFile=`pwd`/pom.xml --output=counts</code></pre> |
| </div> |
| </div> |
| <div class='runner-jet snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-jet" data-lang="jet">$ mvn package -P jet-runner &amp;&amp; java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.WindowedWordCount \ |
| --runner=JetRunner --jetLocalMode=3 --inputFile=`pwd`/pom.xml --output=counts</code></pre> |
| </div> |
| </div> |
| <p>To view the full code in Java, see |
| <strong><a href="https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java">WindowedWordCount</a>.</strong></p> |
| <p><strong>To run this example in Python:</strong></p> |
| <p>This pipeline writes its results to a BigQuery table <code>--output_table</code> |
| parameter. using the format <code>PROJECT:DATASET.TABLE</code> or |
| <code>DATASET.TABLE</code>.</p> |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">python -m apache_beam.examples.windowed_wordcount --input YOUR_INPUT_FILE --output_table PROJECT:DATASET.TABLE</code></pre> |
| </div> |
| </div> |
| <div class='runner-flink snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flink" data-lang="flink">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-flinkCluster snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flinkCluster" data-lang="flinkCluster">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow"># As part of the initial setup, install Google Cloud Platform specific extra components. |
| pip install apache-beam[gcp] |
| python -m apache_beam.examples.windowed_wordcount --input YOUR_INPUT_FILE \ |
| --output_table PROJECT:DATASET.TABLE \ |
| --runner DataflowRunner \ |
| --project YOUR_GCP_PROJECT \ |
| --temp_location gs://YOUR_GCS_BUCKET/tmp/</code></pre> |
| </div> |
| </div> |
| <div class='runner-samza snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-samza" data-lang="samza">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-nemo snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-nemo" data-lang="nemo">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-jet snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-jet" data-lang="jet">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <p>To view the full code in Python, see |
| <strong><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/windowed_wordcount.py">windowed_wordcount.py</a>.</strong></p> |
| <p><strong>To run this example in Go:</strong></p> |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">$ go install github.com/apache/beam/sdks/v2/go/examples/windowed_wordcount |
| $ windowed_wordcount --input &lt;PATH_TO_INPUT_FILE&gt; --output counts</code></pre> |
| </div> |
| </div> |
| <div class='runner-flink snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flink" data-lang="flink">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-flinkCluster snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flinkCluster" data-lang="flinkCluster">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow">$ go install github.com/apache/beam/sdks/v2/go/examples/windowed_wordcount |
| # As part of the initial setup, for non linux users - install package unix before run |
| $ go get -u golang.org/x/sys/unix |
| $ windowed_wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \ |
| --output gs://&lt;your-gcs-bucket&gt;/counts \ |
| --runner dataflow \ |
| --project your-gcp-project \ |
| --temp_location gs://&lt;your-gcs-bucket&gt;/tmp/ \ |
| --staging_location gs://&lt;your-gcs-bucket&gt;/binaries/ \ |
| --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515</code></pre> |
| </div> |
| </div> |
| <div class='runner-samza snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-samza" data-lang="samza">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-nemo snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-nemo" data-lang="nemo">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-jet snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-jet" data-lang="jet">This runner is not yet available for the Go SDK.</code></pre> |
| </div> |
| </div> |
| <p>To view the full code in Go, see |
| <strong><a href="https://github.com/apache/beam/blob/master/sdks/go/examples/windowed_wordcount/windowed_wordcount.go">windowed_wordcount.go</a>.</strong></p> |
| <h3 id="unbounded-and-bounded-datasets">Unbounded and bounded datasets</h3> |
| <p>Beam allows you to create a single pipeline that can handle both bounded and |
| unbounded datasets. If your dataset has a fixed number of elements, it is a bounded |
| dataset and all of the data can be processed together. For bounded datasets, |
| the question to ask is &ldquo;Do I have all of the data?&rdquo; If data continuously |
| arrives (such as an endless stream of game scores in the |
| <a href="/get-started/mobile-gaming-example/">Mobile gaming example</a>, |
| it is an unbounded dataset. An unbounded dataset is never available for |
| processing at any one time, so the data must be processed using a streaming |
| pipeline that runs continuously. The dataset will only be complete up to a |
| certain point, so the question to ask is &ldquo;Up until what point do I have all of |
| the data?&rdquo; Beam uses <a href="/documentation/programming-guide/#windowing">windowing</a> |
| to divide a continuously updating dataset into logical windows of finite size. |
| If your input is unbounded, you must use a runner that supports streaming.</p> |
| <p>If your pipeline&rsquo;s input is bounded, then all downstream PCollections will also be |
| bounded. Similarly, if the input is unbounded, then all downstream PCollections |
| of the pipeline will be unbounded, though separate branches may be independently |
| bounded.</p> |
| <p>Recall that the input for this example is a set of Shakespeare&rsquo;s texts, which is |
| a finite set of data. Therefore, this example reads bounded data from a text |
| file:</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Options</span> <span class="n">options</span> <span class="o">=</span> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Pipeline</span> <span class="n">pipeline</span> <span class="o">=</span> <span class="n">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">input</span> <span class="o">=</span> <span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">TextIO</span><span class="o">.</span><span class="na">read</span><span class="o">().</span><span class="na">from</span><span class="o">(</span><span class="n">options</span><span class="o">.</span><span class="na">getInputFile</span><span class="o">()))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">arvg</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">parser</span> <span class="o">=</span> <span class="n">argparse</span><span class="o">.</span><span class="n">ArgumentParser</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">parser</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">&#39;--input-file&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">dest</span><span class="o">=</span><span class="s1">&#39;input_file&#39;</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">default</span><span class="o">=</span><span class="s1">&#39;/Users/home/words-example.txt&#39;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">known_args</span><span class="p">,</span> <span class="n">pipeline_args</span> <span class="o">=</span> <span class="n">parser</span><span class="o">.</span><span class="n">parse_known_args</span><span class="p">(</span><span class="n">argv</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pipeline_options</span> <span class="o">=</span> <span class="n">PipelineOptions</span><span class="p">(</span><span class="n">pipeline_args</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">p</span> <span class="o">=</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">(</span><span class="n">options</span><span class="o">=</span><span class="n">pipeline_options</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">lines</span> <span class="o">=</span> <span class="n">p</span> <span class="o">|</span> <span class="s1">&#39;read&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">ReadFromText</span><span class="p">(</span><span class="n">known_args</span><span class="o">.</span><span class="n">input_file</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">p</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">NewPipeline</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">s</span> <span class="o">:=</span> <span class="nx">p</span><span class="p">.</span><span class="nf">Root</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nx">lines</span> <span class="o">:=</span> <span class="nx">textio</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">*</span><span class="nx">input</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">...</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="adding-timestamps-to-data">Adding timestamps to data</h3> |
| <p>Each element in a <code>PCollection</code> has an associated <a href="/documentation/programming-guide#element-timestamps">timestamp</a>. |
| The timestamp for each element is initially assigned by the source that creates |
| the <code>PCollection</code>. Some sources that create unbounded PCollections can assign |
| each new element a timestamp that corresponds to when the element was read or |
| added. You can manually assign or adjust timestamps with a <code>DoFn</code>; however, you |
| can only move timestamps forward in time.</p> |
| <p>In this example the input is bounded. For the purpose of the example, the <code>DoFn</code> |
| method named <code>AddTimestampsFn</code> (invoked by <code>ParDo</code>) will set a timestamp for |
| each element in the <code>PCollection</code>.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">ParDo</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="n">AddTimestampFn</span><span class="o">(</span><span class="n">minTimestamp</span><span class="o">,</span> <span class="n">maxTimestamp</span><span class="o">)));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="n">AddTimestampFn</span><span class="p">(</span><span class="n">min_timestamp</span><span class="p">,</span> <span class="n">max_timestamp</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">timestampedLines</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">addTimestampFn</span><span class="p">{</span><span class="nx">Min</span><span class="p">:</span> <span class="nx">mtime</span><span class="p">.</span><span class="nf">Now</span><span class="p">()},</span> <span class="nx">lines</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>Below is the code for <code>AddTimestampFn</code>, a <code>DoFn</code> invoked by <code>ParDo</code>, that sets |
| the data element of the timestamp given the element itself. For example, if the |
| elements were log lines, this <code>ParDo</code> could parse the time out of the log string |
| and set it as the element&rsquo;s timestamp. There are no timestamps inherent in the |
| works of Shakespeare, so in this case we&rsquo;ve made up random timestamps just to |
| illustrate the concept. Each line of the input text will get a random associated |
| timestamp sometime in a 2-hour period.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">static</span> <span class="kd">class</span> <span class="nc">AddTimestampFn</span> <span class="kd">extends</span> <span class="n">DoFn</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">String</span><span class="o">&gt;</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">private</span> <span class="kd">final</span> <span class="n">Instant</span> <span class="n">minTimestamp</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">private</span> <span class="kd">final</span> <span class="n">Instant</span> <span class="n">maxTimestamp</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">AddTimestampFn</span><span class="o">(</span><span class="n">Instant</span> <span class="n">minTimestamp</span><span class="o">,</span> <span class="n">Instant</span> <span class="n">maxTimestamp</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">this</span><span class="o">.</span><span class="na">minTimestamp</span> <span class="o">=</span> <span class="n">minTimestamp</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">this</span><span class="o">.</span><span class="na">maxTimestamp</span> <span class="o">=</span> <span class="n">maxTimestamp</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nd">@ProcessElement</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">processElement</span><span class="o">(</span><span class="n">ProcessContext</span> <span class="n">c</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Instant</span> <span class="n">randomTimestamp</span> <span class="o">=</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">ThreadLocalRandom</span><span class="o">.</span><span class="na">current</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">nextLong</span><span class="o">(</span><span class="n">minTimestamp</span><span class="o">.</span><span class="na">getMillis</span><span class="o">(),</span> <span class="n">maxTimestamp</span><span class="o">.</span><span class="na">getMillis</span><span class="o">()));</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="cm">/** |
| </span></span></span><span class="line"><span class="cl"><span class="cm"> * Concept #2: Set the data element with that timestamp. |
| </span></span></span><span class="line"><span class="cl"><span class="cm"> */</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">c</span><span class="o">.</span><span class="na">outputWithTimestamp</span><span class="o">(</span><span class="n">c</span><span class="o">.</span><span class="na">element</span><span class="o">(),</span> <span class="k">new</span> <span class="n">Instant</span><span class="o">(</span><span class="n">randomTimestamp</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">AddTimestampFn</span><span class="p">(</span><span class="n">beam</span><span class="o">.</span><span class="n">DoFn</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">min_timestamp</span><span class="p">,</span> <span class="n">max_timestamp</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">min_timestamp</span> <span class="o">=</span> <span class="n">min_timestamp</span> |
| </span></span><span class="line"><span class="cl"> <span class="bp">self</span><span class="o">.</span><span class="n">max_timestamp</span> <span class="o">=</span> <span class="n">max_timestamp</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">element</span><span class="p">):</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="n">window</span><span class="o">.</span><span class="n">TimestampedValue</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">element</span><span class="p">,</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">min_timestamp</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">max_timestamp</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kd">type</span> <span class="nx">addTimestampFn</span> <span class="kd">struct</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">Min</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span> <span class="s">`json:&#34;min&#34;`</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="p">(</span><span class="nx">f</span> <span class="o">*</span><span class="nx">addTimestampFn</span><span class="p">)</span> <span class="nf">ProcessElement</span><span class="p">(</span><span class="nx">x</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">X</span><span class="p">)</span> <span class="p">(</span><span class="nx">beam</span><span class="p">.</span><span class="nx">EventTime</span><span class="p">,</span> <span class="nx">beam</span><span class="p">.</span><span class="nx">X</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">timestamp</span> <span class="o">:=</span> <span class="nx">f</span><span class="p">.</span><span class="nx">Min</span><span class="p">.</span><span class="nf">Add</span><span class="p">(</span><span class="nx">time</span><span class="p">.</span><span class="nf">Duration</span><span class="p">(</span><span class="nx">rand</span><span class="p">.</span><span class="nf">Int63n</span><span class="p">(</span><span class="mi">2</span> <span class="o">*</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Hour</span><span class="p">.</span><span class="nf">Nanoseconds</span><span class="p">())))</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">timestamp</span><span class="p">,</span> <span class="nx">x</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-go">Note that the use of the <code>beam.X</code> &ldquo;type variable&rdquo; allows the transform to be |
| used for any type.</p> |
| <h3 id="windowing">Windowing</h3> |
| <p>Beam uses a concept called <strong>Windowing</strong> to subdivide a <code>PCollection</code> into |
| bounded sets of elements. PTransforms that aggregate multiple elements process |
| each <code>PCollection</code> as a succession of multiple, finite windows, even though the |
| entire collection itself may be of infinite size (unbounded).</p> |
| <p>The WindowedWordCount example applies fixed-time windowing, wherein each |
| window represents a fixed time interval. The fixed window size for this example |
| defaults to 1 minute (you can change this with a command-line option).</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span> <span class="n">windowedWords</span> <span class="o">=</span> <span class="n">input</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">Window</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">into</span><span class="o">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">FixedWindows</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">standardMinutes</span><span class="o">(</span><span class="n">options</span><span class="o">.</span><span class="na">getWindowSize</span><span class="o">()))));</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">windowed_words</span> <span class="o">=</span> <span class="nb">input</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">WindowInto</span><span class="p">(</span><span class="n">window</span><span class="o">.</span><span class="n">FixedWindows</span><span class="p">(</span><span class="mi">60</span> <span class="o">*</span> <span class="n">window_size_minutes</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">windowedLines</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">WindowInto</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">window</span><span class="p">.</span><span class="nf">NewFixedWindows</span><span class="p">(</span><span class="nx">time</span><span class="p">.</span><span class="nx">Minute</span><span class="p">),</span> <span class="nx">timestampedLines</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="reusing-ptransforms-over-windowed-pcollections">Reusing PTransforms over windowed PCollections</h3> |
| <p>You can reuse existing PTransforms that were created for manipulating simple |
| PCollections over windowed PCollections as well.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">PCollection</span><span class="o">&lt;</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;&gt;</span> <span class="n">wordCounts</span> <span class="o">=</span> <span class="n">windowedWords</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="k">new</span> <span class="n">WordCount</span><span class="o">.</span><span class="na">CountWords</span><span class="o">());</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="n">word_counts</span> <span class="o">=</span> <span class="n">windowed_words</span> <span class="o">|</span> <span class="n">CountWords</span><span class="p">()</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">counted</span> <span class="o">:=</span> <span class="nx">wordcount</span><span class="p">.</span><span class="nf">CountWords</span><span class="p">(</span><span class="nx">s</span><span class="p">,</span> <span class="nx">windowedLines</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java language-go"><h3 id="try-the-full-example-in-playground">Try the full example in Playground</h3> |
| </p> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-java playground-snippet" |
| data-sdk="java" |
| data-path="SDK_JAVA_WindowedWordCount" |
| ></div> |
| <div |
| class="language-go playground-snippet" |
| data-sdk="go" |
| data-path="SDK_GO_WindowedWordCount" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_JAVA_WindowedWordCount%22%2c%22sdk%22%3a%22java%22%7d%2c%7b%22path%22%3a%22SDK_GO_WindowedWordCount%22%2c%22sdk%22%3a%22go%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <h2 id="streamingwordcount-example">StreamingWordCount example</h2> |
| <p>The StreamingWordCount example is a streaming pipeline that reads Pub/Sub |
| messages from a Pub/Sub subscription or topic, and performs a frequency count on |
| the words in each message. Similar to WindowedWordCount, this example applies |
| fixed-time windowing, wherein each window represents a fixed time interval. The |
| fixed window size for this example is 15 seconds. The pipeline outputs the |
| frequency count of the words seen in each 15 second window.</p> |
| <p><strong>New Concepts:</strong></p> |
| <ul> |
| <li>Reading an unbounded dataset</li> |
| <li>Writing unbounded results</li> |
| </ul> |
| <p><strong>To run this example in Java:</strong></p> |
| <blockquote> |
| <p><strong>Note:</strong> StreamingWordCount is not yet available for the Java SDK.</p> |
| </blockquote> |
| <p><strong>To run this example in Python:</strong></p> |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">python -m apache_beam.examples.streaming_wordcount \ |
| --input_topic &#34;projects/YOUR_PUBSUB_PROJECT_NAME/topics/YOUR_INPUT_TOPIC&#34; \ |
| --output_topic &#34;projects/YOUR_PUBSUB_PROJECT_NAME/topics/YOUR_OUTPUT_TOPIC&#34; \ |
| --streaming</code></pre> |
| </div> |
| </div> |
| <div class='runner-flink snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flink" data-lang="flink">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-flinkCluster snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flinkCluster" data-lang="flinkCluster">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow"># As part of the initial setup, install Google Cloud Platform specific extra components. |
| pip install apache-beam[gcp] |
| python -m apache_beam.examples.streaming_wordcount \ |
| --runner DataflowRunner \ |
| --project YOUR_GCP_PROJECT \ |
| --region YOUR_GCP_REGION \ |
| --temp_location gs://YOUR_GCS_BUCKET/tmp/ \ |
| --input_topic &#34;projects/YOUR_PUBSUB_PROJECT_NAME/topics/YOUR_INPUT_TOPIC&#34; \ |
| --output_topic &#34;projects/YOUR_PUBSUB_PROJECT_NAME/topics/YOUR_OUTPUT_TOPIC&#34; \ |
| --streaming</code></pre> |
| </div> |
| </div> |
| <div class='runner-samza snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-samza" data-lang="samza">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-nemo snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-nemo" data-lang="nemo">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <div class='runner-jet snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-jet" data-lang="jet">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <p>To view the full code in Python, see |
| <strong><a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/streaming_wordcount.py">streaming_wordcount.py</a>.</strong></p> |
| <p><strong>To run this example in Go:</strong></p> |
| <blockquote> |
| <p><strong>Note:</strong> StreamingWordCount is not yet available for the Go SDK. There is an open issue for this |
| (<a href="https://github.com/apache/beam/issues/18879">Issue 18879</a>).</p> |
| </blockquote> |
| <h3 id="reading-an-unbounded-dataset">Reading an unbounded dataset</h3> |
| <p>This example uses an unbounded dataset as input. The code reads Pub/Sub |
| messages from a Pub/Sub subscription or topic using |
| <a href="https://beam.apache.org/releases/pydoc/2.55.1/apache_beam.io.gcp.pubsub.html#apache_beam.io.gcp.pubsub.ReadFromPubSub"><code>beam.io.ReadFromPubSub</code></a>.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="c1">// This example is not currently available for the Beam SDK for Java. |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># Read from Pub/Sub into a PCollection.</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">if</span> <span class="n">known_args</span><span class="o">.</span><span class="n">input_subscription</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">data</span> <span class="o">=</span> <span class="n">p</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ReadFromPubSub</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">subscription</span><span class="o">=</span><span class="n">known_args</span><span class="o">.</span><span class="n">input_subscription</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">else</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">data</span> <span class="o">=</span> <span class="n">p</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ReadFromPubSub</span><span class="p">(</span><span class="n">topic</span><span class="o">=</span><span class="n">known_args</span><span class="o">.</span><span class="n">input_topic</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">lines</span> <span class="o">=</span> <span class="n">data</span> <span class="o">|</span> <span class="s1">&#39;DecodeString&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">d</span><span class="p">:</span> <span class="n">d</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;utf-8&#39;</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"> <span class="c1">// This example is not currently available for the Beam SDK for Go. |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h3 id="writing-unbounded-results">Writing unbounded results</h3> |
| <p>When the input is unbounded, the same is true of the output <code>PCollection</code>. As |
| such, you must make sure to choose an appropriate I/O for the results. Some I/Os |
| support only bounded output, while others support both bounded and unbounded |
| outputs.</p> |
| <p>This example uses an unbounded <code>PCollection</code> and streams the results to |
| Google Pub/Sub. The code formats the results and writes them to a Pub/Sub topic |
| using <a href="https://beam.apache.org/releases/pydoc/2.55.1/apache_beam.io.gcp.pubsub.html#apache_beam.io.gcp.pubsub.WriteToPubSub"><code>beam.io.WriteToPubSub</code></a>.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="c1">// This example is not currently available for the Beam SDK for Java. |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"> <span class="c1"># Write to Pub/Sub</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">_</span> <span class="o">=</span> <span class="p">(</span><span class="n">output</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;EncodeString&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">s</span><span class="p">:</span> <span class="n">s</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">&#39;utf-8&#39;</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">WriteToPubSub</span><span class="p">(</span><span class="n">known_args</span><span class="o">.</span><span class="n">output_topic</span><span class="p">))</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"> <span class="c1">// This example is not currently available for the Beam SDK for Go. |
| </span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h2 id="next-steps">Next Steps</h2> |
| <ul> |
| <li>Walk through the Mobile Gaming examples in the <a href="/get-started/mobile-gaming-example">Mobile Gaming Example Walkthrough</a>.</li> |
| <li>Take a self-paced tour through our <a href="/documentation/resources/learning-resources">Learning Resources</a>.</li> |
| <li>Dive in to some of our favorite <a href="/get-started/resources/videos-and-podcasts">Videos and Podcasts</a>.</li> |
| <li>Join the Beam <a href="/community/contact-us">users@</a> mailing list.</li> |
| </ul> |
| <p>Please don&rsquo;t hesitate to <a href="/community/contact-us">reach out</a> if you encounter any issues!</p></description></item><item><title>Get-Started: Getting started from Apache Spark</title><link>/get-started/from-spark/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/from-spark/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="getting-started-from-apache-spark">Getting started from Apache Spark</h1> |
| <script type="text/javascript"> |
| localStorage.setItem("language", "language-py") |
| </script> |
| <p>If you already know <a href="https://spark.apache.org/"><em>Apache Spark</em></a>, |
| using Beam should be easy. |
| The basic concepts are the same, and the APIs are similar as well.</p> |
| <p>Spark stores data <em>Spark DataFrames</em> for structured data, |
| and in <em>Resilient Distributed Datasets</em> (RDD) for unstructured data. |
| We are using RDDs for this guide.</p> |
| <p>A Spark RDD represents a collection of elements, |
| while in Beam it&rsquo;s called a <em>Parallel Collection</em> (PCollection). |
| A PCollection in Beam does <em>not</em> have any ordering guarantees.</p> |
| <p>Likewise, a transform in Beam is called a <em>Parallel Transform</em> (PTransform).</p> |
| <p>Here are some examples of common operations and their equivalent between PySpark and Beam.</p> |
| <h2 id="overview">Overview</h2> |
| <p>Here&rsquo;s a simple example of a PySpark pipeline that takes the numbers from one to four, |
| multiplies them by two, adds all the values together, and prints the result.</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">pyspark</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">sc</span> <span class="o">=</span> <span class="n">pyspark</span><span class="o">.</span><span class="n">SparkContext</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"><span class="n">result</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">sc</span><span class="o">.</span><span class="n">parallelize</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span> <span class="o">*</span> <span class="mi">2</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">:</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="n">result</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>In Beam you pipe your data through the pipeline using the |
| <em>pipe operator</em> <code>|</code> like <code>data | beam.Map(...)</code> instead of chaining |
| methods like <code>data.map(...)</code>, but they&rsquo;re doing the same thing.</p> |
| <p>Here&rsquo;s what an equivalent pipeline looks like in Beam.</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="k">as</span> <span class="nn">beam</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">result</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span> <span class="o">*</span> <span class="mi">2</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span><span class="nb">sum</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="nb">print</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <blockquote> |
| <p>ℹ️ Note that we called <code>print</code> inside a <code>Map</code> transform. |
| That&rsquo;s because we can only access the elements of a PCollection |
| from within a PTransform. |
| To inspect the data locally, you can use the <a href="https://cloud.google.com/dataflow/docs/guides/interactive-pipeline-development#creating_your_pipeline">InteractiveRunner</a></p> |
| </blockquote> |
| <p>Another thing to note is that Beam pipelines are constructed lazily. |
| This means that when you pipe <code>|</code> data you&rsquo;re only declaring the |
| transformations and the order you want them to happen, |
| but the actual computation doesn&rsquo;t happen. |
| The pipeline is run after the <code>with beam.Pipeline() as pipeline</code> context has |
| closed.</p> |
| <blockquote> |
| <p>ℹ️ When the <code>with beam.Pipeline() as pipeline</code> context closes, |
| it implicitly calls <code>pipeline.run()</code> which triggers the computation to happen.</p> |
| </blockquote> |
| <p>The pipeline is then sent to your |
| <a href="/documentation/runners/capability-matrix/">runner of choice</a> |
| and it processes the data.</p> |
| <blockquote> |
| <p>ℹ️ The pipeline can run locally with the <em>DirectRunner</em>, |
| or in a distributed runner such as Flink, Spark, or Dataflow. |
| The Spark runner is not related to PySpark.</p> |
| </blockquote> |
| <p>A label can optionally be added to a transform using the |
| <em>right shift operator</em> <code>&gt;&gt;</code> like <code>data | 'My description' &gt;&gt; beam.Map(...)</code>. |
| This serves both as comments and makes your pipeline easier to debug.</p> |
| <p>This is how the pipeline looks after adding labels.</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="k">as</span> <span class="nn">beam</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">result</span> <span class="o">=</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Create numbers&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Multiply by two&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span> <span class="o">*</span> <span class="mi">2</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Sum everything&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span><span class="nb">sum</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Print results&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="nb">print</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h2 id="setup">Setup</h2> |
| <p>Here&rsquo;s a comparison on how to get started both in PySpark and Beam.</p> |
| <div class="table-container-wrapper"> |
| <div class="table-wrapper"><table style="width:100%" class="table-wrapper--equal-p"> |
| <tr> |
| <th style="width:20%"></th> |
| <th style="width:40%">PySpark</th> |
| <th style="width:40%">Beam</th> |
| </tr> |
| <tr> |
| <td><b>Install</b></td> |
| <td><code>$ pip install pyspark</code></td> |
| <td><code>$ pip install apache-beam</code></td> |
| </tr> |
| <tr> |
| <td><b>Imports</b></td> |
| <td><code>import pyspark</code></td> |
| <td><code>import apache_beam as beam</code></td> |
| </tr> |
| <tr> |
| <td><b>Creating a<br>local pipeline</b></td> |
| <td> |
| <code>sc = pyspark.SparkContext() as sc:</code><br> |
| <code># Your pipeline code here.</code> |
| </td> |
| <td> |
| <code>with beam.Pipeline() as pipeline:</code><br> |
| <code>&nbsp;&nbsp;&nbsp;&nbsp;# Your pipeline code here.</code> |
| </td> |
| </tr> |
| <tr> |
| <td><b>Creating values</b></td> |
| <td><code>values = sc.parallelize([1, 2, 3, 4])</code></td> |
| <td><code>values = pipeline | beam.Create([1, 2, 3, 4])</code></td> |
| </tr> |
| <tr> |
| <td><b>Creating<br>key-value pairs</b></td> |
| <td> |
| <code>pairs = sc.parallelize([</code><br> |
| <code>&nbsp;&nbsp;&nbsp;&nbsp;('key1', 'value1'),</code><br> |
| <code>&nbsp;&nbsp;&nbsp;&nbsp;('key2', 'value2'),</code><br> |
| <code>&nbsp;&nbsp;&nbsp;&nbsp;('key3', 'value3'),</code><br> |
| <code>])</code> |
| </td> |
| <td> |
| <code>pairs = pipeline | beam.Create([</code><br> |
| <code>&nbsp;&nbsp;&nbsp;&nbsp;('key1', 'value1'),</code><br> |
| <code>&nbsp;&nbsp;&nbsp;&nbsp;('key2', 'value2'),</code><br> |
| <code>&nbsp;&nbsp;&nbsp;&nbsp;('key3', 'value3'),</code><br> |
| <code>])</code> |
| </td> |
| </tr> |
| <tr> |
| <td><b>Running a<br>local pipeline</b></td> |
| <td><code>$ spark-submit spark_pipeline.py</code></td> |
| <td><code>$ python beam_pipeline.py</code></td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| <h2 id="transforms">Transforms</h2> |
| <p>Here are the equivalents of some common transforms in both PySpark and Beam.</p> |
| <div class="table-container-wrapper"> |
| <div class="table-wrapper"><table style="width:100%" class="table-wrapper--equal-p"> |
| <tr> |
| <th style="width:20%"></th> |
| <th style="width:40%">PySpark</th> |
| <th style="width:40%">Beam</th> |
| </tr> |
| <tr> |
| <td><b><a href="/documentation/transforms/python/elementwise/map/">Map</a></b></td> |
| <td><code>values.map(lambda x: x * 2)</code></td> |
| <td><code>values | beam.Map(lambda x: x * 2)</code></td> |
| </tr> |
| <tr> |
| <td><b><a href="/documentation/transforms/python/elementwise/filter/">Filter</a></b></td> |
| <td><code>values.filter(lambda x: x % 2 == 0)</code></td> |
| <td><code>values | beam.Filter(lambda x: x % 2 == 0)</code></td> |
| </tr> |
| <tr> |
| <td><b><a href="/documentation/transforms/python/elementwise/flatmap/">FlatMap</a></b></td> |
| <td><code>values.flatMap(lambda x: range(x))</code></td> |
| <td><code>values | beam.FlatMap(lambda x: range(x))</code></td> |
| </tr> |
| <tr> |
| <td><b><a href="/documentation/transforms/python/aggregation/groupbykey/">Group by key</a></b></td> |
| <td><code>pairs.groupByKey()</code></td> |
| <td><code>pairs | beam.GroupByKey()</code></td> |
| </tr> |
| <tr> |
| <td><b><a href="/documentation/transforms/python/aggregation/combineglobally/">Reduce</a></b></td> |
| <td><code>values.reduce(lambda x, y: x+y)</code></td> |
| <td><code>values | beam.CombineGlobally(sum)</code></td> |
| </tr> |
| <tr> |
| <td><b><a href="/documentation/transforms/python/aggregation/combineperkey/">Reduce by key</a></b></td> |
| <td><code>pairs.reduceByKey(lambda x, y: x+y)</code></td> |
| <td><code>pairs | beam.CombinePerKey(sum)</code></td> |
| </tr> |
| <tr> |
| <td><b><a href="/documentation/transforms/python/aggregation/distinct/">Distinct</a></b></td> |
| <td><code>values.distinct()</code></td> |
| <td><code>values | beam.Distinct()</code></td> |
| </tr> |
| <tr> |
| <td><b><a href="/documentation/transforms/python/aggregation/count/">Count</a></b></td> |
| <td><code>values.count()</code></td> |
| <td><code>values | beam.combiners.Count.Globally()</code></td> |
| </tr> |
| <tr> |
| <td><b><a href="/documentation/transforms/python/aggregation/count/">Count by key</a></b></td> |
| <td><code>pairs.countByKey()</code></td> |
| <td><code>pairs | beam.combiners.Count.PerKey()</code></td> |
| </tr> |
| <tr> |
| <td><b><a href="/documentation/transforms/python/aggregation/top/">Take smallest</a></b></td> |
| <td><code>values.takeOrdered(3)</code></td> |
| <td><code>values | beam.combiners.Top.Smallest(3)</code></td> |
| </tr> |
| <tr> |
| <td><b><a href="/documentation/transforms/python/aggregation/top/">Take largest</a></b></td> |
| <td><code>values.takeOrdered(3, lambda x: -x)</code></td> |
| <td><code>values | beam.combiners.Top.Largest(3)</code></td> |
| </tr> |
| <tr> |
| <td><b><a href="/documentation/transforms/python/aggregation/sample/">Random sample</a></b></td> |
| <td><code>values.takeSample(False, 3)</code></td> |
| <td><code>values | beam.combiners.Sample.FixedSizeGlobally(3)</code></td> |
| </tr> |
| <tr> |
| <td><b><a href="/documentation/transforms/python/other/flatten/">Union</a></b></td> |
| <td><code>values.union(otherValues)</code></td> |
| <td><code>(values, otherValues) | beam.Flatten()</code></td> |
| </tr> |
| <tr> |
| <td><b><a href="/documentation/transforms/python/aggregation/cogroupbykey/">Co-group</a></b></td> |
| <td><code>pairs.cogroup(otherPairs)</code></td> |
| <td><code>{'Xs': pairs, 'Ys': otherPairs} | beam.CoGroupByKey()</code></td> |
| </tr> |
| </table> |
| </div> |
| </div> |
| <blockquote> |
| <p>ℹ️ To learn more about the transforms available in Beam, check the |
| <a href="/documentation/transforms/python/overview">Python transform gallery</a>.</p> |
| </blockquote> |
| <h2 id="using-calculated-values">Using calculated values</h2> |
| <p>Since we are working in potentially distributed environments, |
| we can&rsquo;t guarantee that the results we&rsquo;ve calculated are available at any given machine.</p> |
| <p>In PySpark, we can get a result from a collection of elements (RDD) by using |
| <code>data.collect()</code>, or other aggregations such as <code>reduce()</code>, <code>count()</code>, and more.</p> |
| <p>Here&rsquo;s an example to scale numbers into a range between zero and one.</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">pyspark</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">sc</span> <span class="o">=</span> <span class="n">pyspark</span><span class="o">.</span><span class="n">SparkContext</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"><span class="n">values</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">parallelize</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"><span class="n">min_value</span> <span class="o">=</span> <span class="n">values</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="nb">min</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="n">max_value</span> <span class="o">=</span> <span class="n">values</span><span class="o">.</span><span class="n">reduce</span><span class="p">(</span><span class="nb">max</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># We can simply use `min_value` and `max_value` since it&#39;s already a Python `int` value from `reduce`.</span> |
| </span></span><span class="line"><span class="cl"><span class="n">scaled_values</span> <span class="o">=</span> <span class="n">values</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="p">(</span><span class="n">x</span> <span class="o">-</span> <span class="n">min_value</span><span class="p">)</span> <span class="o">/</span> <span class="p">(</span><span class="n">max_value</span> <span class="o">-</span> <span class="n">min_value</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="c1"># But to access `scaled_values`, we need to call `collect`.</span> |
| </span></span><span class="line"><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="n">scaled_values</span><span class="o">.</span><span class="n">collect</span><span class="p">())</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>In Beam the results from all transforms result in a PCollection. |
| We use <a href="/documentation/programming-guide/#side-inputs"><em>side inputs</em></a> |
| to feed a PCollection into a transform and access its values.</p> |
| <p>Any transform that accepts a function, like |
| <a href="/documentation/transforms/python/elementwise/map"><code>Map</code></a>, |
| can take side inputs. |
| If we only need a single value, we can use |
| <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.pvalue.html#apache_beam.pvalue.AsSingleton"><code>beam.pvalue.AsSingleton</code></a> and access them as a Python value. |
| If we need multiple values, we can use |
| <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.pvalue.html#apache_beam.pvalue.AsIter"><code>beam.pvalue.AsIter</code></a> |
| and access them as an <a href="https://docs.python.org/3/glossary.html#term-iterable"><code>iterable</code></a>.</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="k">as</span> <span class="nn">beam</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">values</span> <span class="o">=</span> <span class="n">pipeline</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">])</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">min_value</span> <span class="o">=</span> <span class="n">values</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span><span class="nb">min</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">max_value</span> <span class="o">=</span> <span class="n">values</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombineGlobally</span><span class="p">(</span><span class="nb">max</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="c1"># To access `min_value` and `max_value`, we need to pass them as a side input.</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">scaled_values</span> <span class="o">=</span> <span class="n">values</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">minimum</span><span class="p">,</span> <span class="n">maximum</span><span class="p">:</span> <span class="p">(</span><span class="n">x</span> <span class="o">-</span> <span class="n">minimum</span><span class="p">)</span> <span class="o">/</span> <span class="p">(</span><span class="n">maximum</span> <span class="o">-</span> <span class="n">minimum</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">minimum</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">pvalue</span><span class="o">.</span><span class="n">AsSingleton</span><span class="p">(</span><span class="n">min_value</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">maximum</span><span class="o">=</span><span class="n">beam</span><span class="o">.</span><span class="n">pvalue</span><span class="o">.</span><span class="n">AsSingleton</span><span class="p">(</span><span class="n">max_value</span><span class="p">),</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">scaled_values</span> <span class="o">|</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="nb">print</span><span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <blockquote> |
| <p>ℹ️ In Beam we need to pass a side input explicitly, but we get the |
| benefit that a reduction or aggregation does <em>not</em> have to fit into memory. |
| Lazily computing side inputs also allows us to compute <code>values</code> only once, |
| rather than for each distinct reduction (or requiring explicit caching of the RDD).</p> |
| </blockquote> |
| <h2 id="next-steps">Next Steps</h2> |
| <ul> |
| <li>Take a look at all the available transforms in the <a href="/documentation/transforms/python/overview">Python transform gallery</a>.</li> |
| <li>Learn how to read from and write to files in the <a href="/documentation/programming-guide/#pipeline-io"><em>Pipeline I/O</em> section of the <em>Programming guide</em></a></li> |
| <li>Walk through additional WordCount examples in the <a href="/get-started/wordcount-example">WordCount Example Walkthrough</a>.</li> |
| <li>Take a self-paced tour through our <a href="/documentation/resources/learning-resources">Learning Resources</a>.</li> |
| <li>Dive in to some of our favorite <a href="/get-started/resources/videos-and-podcasts">Videos and Podcasts</a>.</li> |
| <li>Join the Beam <a href="/community/contact-us">users@</a> mailing list.</li> |
| <li>If you&rsquo;re interested in contributing to the Apache Beam codebase, see the <a href="/contribute">Contribution Guide</a>.</li> |
| </ul> |
| <p>Please don&rsquo;t hesitate to <a href="/community/contact-us">reach out</a> if you encounter any issues!</p></description></item><item><title>Get-Started: Learning Resources</title><link>/get-started/resources/learning-resources/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/resources/learning-resources/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="learning-resources">Learning Resources</h1> |
| <p>Welcome to our learning resources. This page contains a collection of resources that will help you to get started and use Apache Beam. If you’re just starting, you can view this as a guided tour, otherwise you can jump straight to any section of your interest.</p> |
| <p>If you have additional material that you would like to see here, please let us know at <a href="mailto:user@beam.apache.org">user@beam.apache.org</a>!</p> |
| <nav id="TableOfContents"> |
| <ul> |
| <li><a href="#getting-started">Getting Started</a> |
| <ul> |
| <li><a href="#quickstart">Quickstart</a></li> |
| <li><a href="#learning-the-basics">Learning the Basics</a></li> |
| <li><a href="#fundamentals">Fundamentals</a></li> |
| <li><a href="#common-patterns">Common Patterns</a></li> |
| </ul> |
| </li> |
| <li><a href="#articles">Articles</a> |
| <ul> |
| <li><a href="#data-analysis">Data Analysis</a></li> |
| <li><a href="#data-migration">Data Migration</a></li> |
| <li><a href="#machine-learning">Machine Learning</a></li> |
| <li><a href="#advanced-concepts">Advanced Concepts</a></li> |
| </ul> |
| </li> |
| <li><a href="#videos">Videos</a></li> |
| <li><a href="#courses">Courses</a></li> |
| <li><a href="#books">Books</a> |
| <ul> |
| <li><a href="#building-big-data-pipelines-with-apache-beam">Building Big Data Pipelines with Apache Beam</a></li> |
| <li><a href="#streaming-systems-the-what-where-when-and-how-of-large-scale-data-processing">Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing</a></li> |
| </ul> |
| </li> |
| <li><a href="#certifications">Certifications</a> |
| <ul> |
| <li><a href="#getting-started-with-apache-beam-quest">Getting Started with Apache Beam Quest</a></li> |
| </ul> |
| </li> |
| <li><a href="#interactive-labs">Interactive Labs</a> |
| <ul> |
| <li><a href="#java">Java</a></li> |
| <li><a href="#python">Python</a></li> |
| </ul> |
| </li> |
| <li><a href="#beam-katas">Beam Katas</a> |
| <ul> |
| <li><a href="#java-1">Java</a></li> |
| <li><a href="#python-1">Python</a></li> |
| </ul> |
| </li> |
| <li><a href="#code-examples">Code Examples</a> |
| <ul> |
| <li><a href="#dataflow-cookbook">Dataflow Cookbook</a></li> |
| <li><a href="#java-2">Java</a></li> |
| <li><a href="#python-2">Python</a></li> |
| <li><a href="#beam-playground">Beam Playground</a></li> |
| </ul> |
| </li> |
| <li><a href="#api-reference">API Reference</a></li> |
| <li><a href="#feedback-and-suggestions">Feedback and Suggestions</a></li> |
| <li><a href="#how-to-contribute">How to Contribute</a></li> |
| </ul> |
| </nav> |
| <h2 id="getting-started">Getting Started</h2> |
| <h3 id="quickstart">Quickstart</h3> |
| <ul> |
| <li><strong><a href="/get-started/quickstart-java/">Java Quickstart</a></strong> - How to set up and run a WordCount pipeline on the Java SDK.</li> |
| <li><strong><a href="/get-started/quickstart-py/">Python Quickstart</a></strong> - How to set up and run a WordCount pipeline on the Python SDK.</li> |
| <li><strong><a href="/get-started/quickstart-go/">Go Quickstart</a></strong> - How to set up and run a WordCount pipeline on the Go SDK.</li> |
| <li><strong><a href="https://medium.com/google-cloud/setting-up-a-java-development-environment-for-apache-beam-on-google-cloud-platform-ec0c6c9fbb39">Java Development Environment</a></strong> - Setting up a Java development environment for Apache Beam using IntelliJ and Maven.</li> |
| <li><strong><a href="https://medium.com/google-cloud/python-development-environments-for-apache-beam-on-google-cloud-platform-b6f276b344df">Python Development Environment</a></strong> - Setting up a Python development environment for Apache Beam using PyCharm.</li> |
| </ul> |
| <h3 id="learning-the-basics">Learning the Basics</h3> |
| <ul> |
| <li><strong><a href="/get-started/wordcount-example/">WordCount</a></strong> - Walks you through the code of a simple WordCount pipeline. This is a very basic pipeline intended to show the most basic concepts of data processing. WordCount is the &ldquo;Hello World&rdquo; for data processing.</li> |
| <li><strong><a href="/get-started/mobile-gaming-example/">Mobile Gaming</a></strong> - Introduces how to consider time while processing data, user defined transforms, windowing, filtering data, streaming pipelines, triggers, and session analysis. This is a great place to start once you get the hang of WordCount.</li> |
| </ul> |
| <h3 id="fundamentals">Fundamentals</h3> |
| <ul> |
| <li><strong><a href="/documentation/programming-guide/">Programming Guide</a></strong> - The Programming Guide contains more in-depth information on most topics in the Apache Beam SDK. These include descriptions on how everything works as well as code snippets to see how to use every part. This can be used as a reference guidebook.</li> |
| <li><strong><a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101">The world beyond batch: Streaming 101</a></strong> - Covers some basic background information, terminology, time domains, batch processing, and streaming.</li> |
| <li><strong><a href="https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102">The world beyond batch: Streaming 102</a></strong> - Tour of the unified batch and streaming programming model in Beam, alongside with an example to explain many of the concepts.</li> |
| <li><strong><a href="/documentation/runtime/model">Apache Beam Execution Model</a></strong> - Explanation on how runners execute an Apache Beam pipeline. This includes why serialization is important, and how a runner might distribute the work in parallel to multiple machines.</li> |
| </ul> |
| <h3 id="common-patterns">Common Patterns</h3> |
| <ul> |
| <li><strong><a href="https://cloud.google.com/blog/products/gcp/guide-to-common-cloud-dataflow-use-case-patterns-part-1">Common Use Case Patterns Part 1</a></strong> - Common patterns such as writing data to multiple storage locations, slowly-changing lookup cache, calling external services, dealing with bad data, and starting jobs through a REST endpoint.</li> |
| <li><strong><a href="https://cloud.google.com/blog/products/gcp/guide-to-common-cloud-dataflow-use-case-patterns-part-2">Common Use Case Patterns Part 2</a></strong> - Common patterns such as GroupBy using multiple data properties, joining two PCollections on a common key, streaming large lookup tables, merging two streams with different window lengths, and threshold detection with time-series data.</li> |
| <li><strong><a href="https://nanthrax.blogspot.com/2018/02/apache-beam-easily-implement-backoff_18.html">Retry Policy</a></strong> - Adding a retry policy to a <code>DoFn</code>.</li> |
| </ul> |
| <h2 id="articles">Articles</h2> |
| <h3 id="data-analysis">Data Analysis</h3> |
| <ul> |
| <li><strong><a href="https://medium.com/google-cloud/predicting-social-engagement-for-the-worlds-news-with-tensorflow-and-cloud-dataflow-part-1-b92ba8f14a7">Predicting news social engagement</a></strong> - Using multiple data sources, many common design patterns, and sentiment analysis to get insights into different news articles for TensorFlow and Dataflow.</li> |
| <li><strong><a href="https://cloud.google.com/community/tutorials/cloud-iot-rtdp">Processing IoT Data</a></strong> - IoT sensors are continuously streaming data to the cloud. Learn how to handle the sensor data which can be useful for real-time monitoring, alerts, long-term data storage for analysis, performance improvement, and model training.</li> |
| </ul> |
| <h3 id="data-migration">Data Migration</h3> |
| <ul> |
| <li><strong><a href="https://medium.com/google-cloud/oracle-data-to-google-bigquery-using-google-cloud-dataflow-and-dataprep-20884571a9e5">Oracle Database to Google BigQuery</a></strong> - Migrate data from an <a href="https://www.oracle.com/database/index.html">Oracle Database</a> into <a href="https://cloud.google.com/bigquery">BigQuery</a> using <a href="https://cloud.google.com/dataprep/">Dataprep</a>.</li> |
| <li><strong><a href="https://medium.com/google-cloud/export-bigquery-to-google-datastore-with-apache-beam-google-dataflow-7fff1566f345">Google BigQuery to Google Datastore</a></strong> - Migrate data from a <a href="https://cloud.google.com/bigquery/">BigQuery</a> table into <a href="https://cloud.google.com/datastore/">Datastore</a> without thinking of its schema.</li> |
| <li><strong><a href="https://cloud.google.com/blog/products/gcp/using-apache-beam-and-cloud-dataflow-to-integrate-sap-hana-and-bigquery">SAP HANA to Google BigQuery</a></strong> - Migrate data from a <a href="https://www.sapphiresystems.com/en-us/products/sap-hana">SAP HANA</a> in-memory database into <a href="https://cloud.google.com/bigquery">BigQuery</a>.</li> |
| </ul> |
| <h3 id="machine-learning">Machine Learning</h3> |
| <ul> |
| <li><strong><a href="/documentation/ml/about-ml">Machine Learning using the RunInference API</a></strong> - Use Apache Beam with the RunInference API to use machine learning (ML) models to do local and remote inference with batch and streaming pipelines. Follow the <a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference">RunInference API pipeline examples</a> to do image classification, image segmentation, language modeling, and MNIST digit classification. See examples of <a href="/documentation/transforms/python/elementwise/runinference/">RunInference transforms</a>.</li> |
| <li><strong><a href="https://cloud.google.com/dataflow/examples/molecules-walkthrough">Machine Learning Preprocessing and Prediction</a></strong> - Predict the molecular energy from data stored in the <a href="https://en.wikipedia.org/wiki/Spatial_Data_File">Spatial Data File</a> (SDF) format. Train a <a href="https://www.tensorflow.org/">TensorFlow</a> model with <a href="https://github.com/tensorflow/transform">tf.Transform</a> for preprocessing in Python. This also shows how to create batch and streaming prediction pipelines in Apache Beam.</li> |
| <li><strong><a href="https://cloud.google.com/blog/products/ai-machine-learning/pre-processing-tensorflow-pipelines-tftransform-google-cloud">Machine Learning Preprocessing</a></strong> - Find the optimal parameter settings for simulated physical machines like a bottle filler or cookie machine. The goal of each simulated machine is to have the same input/output of the actual machine, making it a &ldquo;digital twin&rdquo;. This uses <a href="https://github.com/tensorflow/transform">tf.Transform</a> for preprocessing.</li> |
| </ul> |
| <h3 id="advanced-concepts">Advanced Concepts</h3> |
| <ul> |
| <li><strong><a href="https://amygdala.github.io/dataflow/app_engine/2017/10/24/gae_dataflow.html">Running on AppEngine</a></strong> - Use a Dataflow template to launch a pipeline from Google AppEngine, and how to run the pipeline periodically via a cron job.</li> |
| <li><strong><a href="/blog/2017/02/13/stateful-processing.html">Stateful Processing</a></strong> - Learn how to access a persistent mutable state while processing input elements, this allows for <em>side effects</em> in a <code>DoFn</code>. This can be used for arbitrary-but-consistent index assignment, if you want to assign a unique incrementing index to each incoming element where order doesn&rsquo;t matter.</li> |
| <li><strong><a href="/blog/2017/08/28/timely-processing.html">Timely and Stateful Processing</a></strong> - An example on how to do batched RPC calls. The call requests are stored in a mutable state as they are received. Once there are either enough requests or a certain time has passed, the batch of requests is triggered to be sent.</li> |
| <li><strong><a href="https://cloud.google.com/blog/products/gcp/running-external-libraries-with-cloud-dataflow-for-grid-computing-workloads">Running External Libraries</a></strong> - Call an external library written in a language that does not have a native SDK in Apache Beam such as C++.</li> |
| </ul> |
| <h2 id="videos">Videos</h2> |
| <ul> |
| <li><strong><a href="https://www.youtube.com/playlist?list=PLIivdWyY5sqIEiHGunZXg_yoS7unlHNJt">Getting Started with Apache Beam</a></strong> - Five part video series for understanding basic to advanced concepts.</li> |
| <li>See more <a href="/get-started/resources/videos-and-podcasts/">Videos and Podcasts</a></li> |
| </ul> |
| <h2 id="courses">Courses</h2> |
| <ul> |
| <li><strong><a href="https://beamcollege.dev/">Beam College</a></strong> &ndash; Free live and recorded lessons for learning Beam and data processing.</li> |
| <li><strong><a href="https://www.coursera.org/specializations/serverless-data-processing-with-dataflow">Serverless Data Processing</a></strong> - Course specialized for Dataflow runner.</li> |
| </ul> |
| <h2 id="books">Books</h2> |
| <h3 id="building-big-data-pipelines-with-apache-beam">Building Big Data Pipelines with Apache Beam</h3> |
| <p><strong><a href="https://www.packtpub.com/product/building-big-data-pipelines-with-apache-beam/9781800564930">Building Big Data Pipelines with Apache Beam</a></strong> by Jan Lukavský, Packt. (January 2022). A general description of the Apache Beam model including gradually built examples that help create solid understanding of the subject. In the first part the book explains concepts using Java SDK, then SQL DSL and Portability layer with focus on Python SDK. The last part of the book is dedicated to more advanced topics like IO connectors using Splittable DoFn and description of how a typical runner executes Pipeline.</p> |
| <h3 id="streaming-systems-the-what-where-when-and-how-of-large-scale-data-processing">Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing</h3> |
| <p><strong><a href="https://learning.oreilly.com/library/view/streaming-systems/9781491983867/">Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing</a></strong> by Tyler Akidau, Slava Chernyak, Reuven Lax. (August 2018). Expanded from Tyler Akidau’s popular blog posts &ldquo;Streaming 101&rdquo; and &ldquo;Streaming 102&rdquo;, this book takes you from an introductory level to a nuanced understanding of the what, where, when, and how of processing real-time data streams.</p> |
| <h2 id="certifications">Certifications</h2> |
| <h3 id="getting-started-with-apache-beam-quest">Getting Started with Apache Beam Quest</h3> |
| <p><strong><a href="https://www.cloudskillsboost.google/quests/310">Get Started with Apache Beam</a></strong> This quest includes four labs that teach you how to write and test Apache Beam pipelines. Three of the labs use Java and one uses Python. Each lab takes about 1.5 hours to complete. When you complete the quest, you&rsquo;re granted a badge that you can use to show your Beam expertise.</p> |
| <h2 id="interactive-labs">Interactive Labs</h2> |
| <h3 id="java">Java</h3> |
| <ul> |
| <li><strong><a href="https://qwiklabs.com/focuses/608?locale=en&amp;parent=catalog">Big Data Text Processing Pipeline</a></strong> (40m) - Run a word count pipeline on the Dataflow runner.</li> |
| <li><strong><a href="https://qwiklabs.com/focuses/3393?locale=en&amp;parent=catalog">Real Time Machine Learning</a></strong> (45m) - Create a real-time flight delay prediction service using historical data on internal flights in the United States.</li> |
| <li><strong><a href="https://qwiklabs.com/focuses/1160?locale=en&amp;parent=catalog">Visualize Real-Time Geospatial Data</a></strong> (60m) - Process real-time streaming data from a real-time real world historical data set, store the results in BigQuery, and visualize the geospatial data on Data Studio.</li> |
| <li><strong><a href="https://qwiklabs.com/focuses/3392?locale=en&amp;parent=catalog">Processing Time Windowed Data</a></strong> (90m) - Implement time-windowed aggregation to augment the raw data in order to produce a consistent training and test datasets for a machine learning model.</li> |
| </ul> |
| <h3 id="python">Python</h3> |
| <ul> |
| <li><strong><a href="https://www.qwiklabs.com/focuses/1098?parent=catalog">Python Qwik Start</a></strong> (30m) - Run a word count pipeline on the Dataflow runner.</li> |
| <li><strong><a href="https://qwiklabs.com/focuses/1159?locale=en&amp;parent=catalog">Simulate historic flights</a></strong> (60m) - Simulate real-time historic internal flights in the United States and store the resulting simulated data in BigQuery.</li> |
| </ul> |
| <h2 id="beam-katas">Beam Katas</h2> |
| <p>Beam Katas are interactive Beam coding exercises (i.e. <a href="http://codekata.com/">code katas</a>) |
| that can help you to learn Apache Beam concepts and programming model hands-on. |
| Built based on <a href="https://www.jetbrains.com/education/">JetBrains Educational Products</a>, Beam Katas |
| objective is to provide a series of structured hands-on learning experiences for learners |
| to understand about Apache Beam and its SDKs by solving exercises with gradually increasing |
| complexity. Beam Katas are available for both Java and Python SDKs.</p> |
| <h3 id="java-1">Java</h3> |
| <ul> |
| <li>Download <a href="https://www.jetbrains.com/education/download/#section=idea">IntelliJ Edu</a></li> |
| <li>Upon opening the IDE, expand the &ldquo;Learn and Teach&rdquo; menu, then select &ldquo;Browse Courses&rdquo;</li> |
| <li>Search for &ldquo;Beam Katas - Java&rdquo;</li> |
| <li>Expand the &ldquo;Advanced Settings&rdquo; and modify the &ldquo;Location&rdquo; and &ldquo;Jdk&rdquo; appropriately</li> |
| <li>Click &ldquo;Join&rdquo;</li> |
| <li><a href="https://www.jetbrains.com/help/education/learner-start-guide.html?section=Introduction%20to%20Java#explore_course">Learn more</a> about how to use the Education product</li> |
| </ul> |
| <h3 id="python-1">Python</h3> |
| <ul> |
| <li>Download <a href="https://www.jetbrains.com/education/download/#section=pycharm-edu">PyCharm Edu</a></li> |
| <li>Upon opening the IDE, expand the &ldquo;Learn and Teach&rdquo; menu, then select &ldquo;Browse Courses&rdquo;</li> |
| <li>Search for &ldquo;Beam Katas - Python&rdquo;</li> |
| <li>Expand the &ldquo;Advanced Settings&rdquo; and modify the &ldquo;Location&rdquo; and &ldquo;Interpreter&rdquo; appropriately</li> |
| <li>Click &ldquo;Join&rdquo;</li> |
| <li><a href="https://www.jetbrains.com/help/education/learner-start-guide.html?section=Introduction%20to%20Python#explore_course">Learn more</a> about how to use the Education product</li> |
| </ul> |
| <h2 id="code-examples">Code Examples</h2> |
| <h3 id="dataflow-cookbook">Dataflow Cookbook</h3> |
| <p>The <a href="https://github.com/GoogleCloudPlatform/dataflow-cookbook">cookbook</a> includes examples in Java, Python, and Scala (via Scio), provides ready-to-launch and self-contained Beam pipelines.</p> |
| <h3 id="java-2">Java</h3> |
| <ul> |
| <li><strong><a href="https://github.com/apache/beam/tree/master/examples/java/src/main/java/org/apache/beam/examples/cookbook">Snippets 1</a></strong> - Commonly-used data analysis patterns such as how to use <a href="https://cloud.google.com/bigquery">BigQuery</a>, a CombinePerKey transform, remove duplicate lines in files, filtering, joining PCollections, getting the maximum value of a PCollection, etc.</li> |
| <li><strong><a href="https://github.com/apache/beam/tree/master/examples/java/src/main/java/org/apache/beam/examples/common">Snippets 2</a></strong> - Additional examples on common tasks such as configuring <a href="https://cloud.google.com/bigquery">BigQuery</a>, <a href="https://cloud.google.com/pubsub/">PubSub</a>, writing one file per window, etc.</li> |
| <li><strong><a href="https://github.com/apache/beam/tree/master/examples/java/src/main/java/org/apache/beam/examples/complete">Complete Examples</a></strong> - End-to-end example pipelines such as an auto complete, a streaming word extract, calculating the Term Frequency-Inverse Document Frequency (<a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf">TF-IDF</a>), getting the top Wikipedia sessions, traffic max lane flow, traffic routes, etc.</li> |
| <li><strong><a href="https://github.com/GoogleCloudPlatform/cloud-code-samples/tree/v1/java/java-dataflow-samples/read-pubsub-write-bigquery">Pub/Sub to BigQuery</a></strong> - A complete example demonstrates using Apache Beam on Dataflow to convert JSON encoded Pub/Sub subscription message strings into structured data and write that data to a BigQuery table.</li> |
| </ul> |
| <h3 id="python-2">Python</h3> |
| <ul> |
| <li><strong><a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/cookbook">Snippets</a></strong> - Commonly-used data analysis patterns such as how to use <a href="https://cloud.google.com/bigquery">BigQuery</a>, <a href="https://cloud.google.com/datastore/">Datastore</a>, coders, combiners, filters, custom PTransforms, etc.</li> |
| <li><strong><a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/complete">Complete Examples</a></strong> - End-to-end example pipelines such as an auto complete, getting mobile gaming statistics, calculating the <a href="https://en.wikipedia.org/wiki/Julia_set">Julia set</a>, solving distributing optimization tasks, estimating PI, calculating the Term Frequency-Inverse Document Frequency (<a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf">TF-IDF</a>), getting the top Wikipedia sessions, etc.</li> |
| </ul> |
| <h3 id="beam-playground">Beam Playground</h3> |
| <ul> |
| <li><a href="https://play.beam.apache.org">Beam Playground</a> is an interactive environment to try out Beam transforms and examples without having to install Apache Beam in your environment. |
| You can try the available Apache Beam examples at <a href="https://play.beam.apache.org">Beam Playground</a>.</li> |
| <li>Learn more about how to add an Apache Beam example/test/kata into Beam Playground catalog <a href="/get-started/try-beam-playground/#how-to-add-new-examples">here</a>.</li> |
| </ul> |
| <h2 id="api-reference">API Reference</h2> |
| <ul> |
| <li><strong><a href="/documentation/sdks/javadoc/">Java API Reference</a></strong> - Official API Reference for the Java SDK.</li> |
| <li><strong><a href="/documentation/sdks/pydoc/">Python API Reference</a></strong> - Official API Reference for the Python SDK.</li> |
| <li><strong><a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam">Go API Reference</a></strong> - Official API Reference for the Go SDK.</li> |
| </ul> |
| <h2 id="feedback-and-suggestions">Feedback and Suggestions</h2> |
| <p>We are open for feedback and suggestions, you can find different ways to reach out to the community in the <a href="/community/contact-us/">Contact Us</a> page.</p> |
| <p>If you have a bug report or want to suggest a new feature, you can let us know by <a href="https://github.com/apache/beam/issues/new/choose">submitting a new issue</a>.</p> |
| <h2 id="how-to-contribute">How to Contribute</h2> |
| <p>We welcome contributions from everyone! To learn more on how to contribute, check our <a href="/contribute/">Contribution Guide</a>.</p></description></item><item><title>Get-Started: The Tour of Beam</title><link>/get-started/tour-of-beam/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/tour-of-beam/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="the-tour-of-beam">The Tour of Beam</h1> |
| <p>The &ldquo;Tour of Beam&rdquo; is an interactive way of learning to write Beam code with a sandbox, where you can write and run pipelines while walking through various concepts. Please <a href="https://tour.beam.apache.org/">click here</a> to try it out.</p></description></item><item><title>Get-Started: Try Apache Beam</title><link>/get-started/try-apache-beam/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/try-apache-beam/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="try-apache-beam">Try Apache Beam</h1> |
| <p>You can try an Apache Beam pipeline using our interactive notebooks.</p> |
| <nav class="language-switcher"> |
| <strong>Adapt for:</strong> |
| <ul> |
| <li data-value="java" class="active">Java SDK</li> |
| <li data-value="py">Python SDK</li> |
| <li data-value="go">Go SDK</li> |
| </ul> |
| </nav> |
| <h2 id="interactive-wordcount-in-colab">Interactive WordCount in Colab</h2> |
| <p>This interactive notebook shows you what a simple, minimal version of WordCount looks like.</p> |
| <div class='language-java snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kn">package</span> <span class="nn">samples.quickstart</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">org.apache.beam.sdk.Pipeline</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">org.apache.beam.sdk.io.TextIO</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">org.apache.beam.sdk.options.PipelineOptions</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">org.apache.beam.sdk.options.PipelineOptionsFactory</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">org.apache.beam.sdk.transforms.Count</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">org.apache.beam.sdk.transforms.Filter</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">org.apache.beam.sdk.transforms.FlatMapElements</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">org.apache.beam.sdk.transforms.MapElements</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">org.apache.beam.sdk.values.KV</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">org.apache.beam.sdk.values.TypeDescriptors</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">java.util.Arrays</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">WordCount</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">String</span> <span class="n">inputsDir</span> <span class="o">=</span> <span class="s">&#34;data/*&#34;</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">String</span> <span class="n">outputsPrefix</span> <span class="o">=</span> <span class="s">&#34;outputs/part&#34;</span><span class="o">;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="n">PipelineOptions</span> <span class="n">options</span> <span class="o">=</span> <span class="n">PipelineOptionsFactory</span><span class="o">.</span><span class="na">fromArgs</span><span class="o">(</span><span class="n">args</span><span class="o">).</span><span class="na">create</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">Pipeline</span> <span class="n">pipeline</span> <span class="o">=</span> <span class="n">Pipeline</span><span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">options</span><span class="o">);</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;Read lines&#34;</span><span class="o">,</span> <span class="n">TextIO</span><span class="o">.</span><span class="na">read</span><span class="o">().</span><span class="na">from</span><span class="o">(</span><span class="n">inputsDir</span><span class="o">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;Find words&#34;</span><span class="o">,</span> <span class="n">FlatMapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">via</span><span class="o">((</span><span class="n">String</span> <span class="n">line</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="n">line</span><span class="o">.</span><span class="na">split</span><span class="o">(</span><span class="s">&#34;[^\\p{L}]+&#34;</span><span class="o">))))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;Filter empty words&#34;</span><span class="o">,</span> <span class="n">Filter</span><span class="o">.</span><span class="na">by</span><span class="o">((</span><span class="n">String</span> <span class="n">word</span><span class="o">)</span> <span class="o">-&gt;</span> <span class="o">!</span><span class="n">word</span><span class="o">.</span><span class="na">isEmpty</span><span class="o">()))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;Count words&#34;</span><span class="o">,</span> <span class="n">Count</span><span class="o">.</span><span class="na">perElement</span><span class="o">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="s">&#34;Write results&#34;</span><span class="o">,</span> <span class="n">MapElements</span><span class="o">.</span><span class="na">into</span><span class="o">(</span><span class="n">TypeDescriptors</span><span class="o">.</span><span class="na">strings</span><span class="o">())</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">via</span><span class="o">((</span><span class="n">KV</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;</span> <span class="n">wordCount</span><span class="o">)</span> <span class="o">-&gt;</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">wordCount</span><span class="o">.</span><span class="na">getKey</span><span class="o">()</span> <span class="o">+</span> <span class="s">&#34;: &#34;</span> <span class="o">+</span> <span class="n">wordCount</span><span class="o">.</span><span class="na">getValue</span><span class="o">()))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">TextIO</span><span class="o">.</span><span class="na">write</span><span class="o">().</span><span class="na">to</span><span class="o">(</span><span class="n">outputsPrefix</span><span class="o">));</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pipeline</span><span class="o">.</span><span class="na">run</span><span class="o">();</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">}</span> |
| </span></span><span class="line"><span class="cl"><span class="o">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-java"><a class="button button--primary" target="_blank" |
| href="https://colab.sandbox.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-java.ipynb"> |
| Run in Colab |
| </a> |
| <a class="button button--primary" target="_blank" |
| href="https://github.com/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-java.ipynb"> |
| View on GitHub |
| </a></p> |
| <p class="language-java">To learn how to install and run the Apache Beam Java SDK on your own computer, follow the instructions in the <a href="/get-started/quickstart-java">Java Quickstart</a>.</p> |
| <div class='language-py snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-py" data-lang="py"><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">apache_beam</span> <span class="k">as</span> <span class="nn">beam</span> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">re</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="n">inputs_pattern</span> <span class="o">=</span> <span class="s1">&#39;data/*&#39;</span> |
| </span></span><span class="line"><span class="cl"><span class="n">outputs_prefix</span> <span class="o">=</span> <span class="s1">&#39;outputs/part&#39;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">()</span> <span class="k">as</span> <span class="n">pipeline</span><span class="p">:</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">pipeline</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Read lines&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ReadFromText</span><span class="p">(</span><span class="n">inputs_pattern</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Find words&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">FlatMap</span><span class="p">(</span><span class="k">lambda</span> <span class="n">line</span><span class="p">:</span> <span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="sa">r</span><span class="s2">&#34;[a-zA-Z&#39;]+&#34;</span><span class="p">,</span> <span class="n">line</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Pair words with 1&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">word</span><span class="p">:</span> <span class="p">(</span><span class="n">word</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Group and sum&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">CombinePerKey</span><span class="p">(</span><span class="nb">sum</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Format results&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">Map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">word_count</span><span class="p">:</span> <span class="nb">str</span><span class="p">(</span><span class="n">word_count</span><span class="p">))</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">|</span> <span class="s1">&#39;Write results&#39;</span> <span class="o">&gt;&gt;</span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">WriteToText</span><span class="p">(</span><span class="n">outputs_prefix</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">)</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-py"><a class="button button--primary" target="_blank" |
| href="https://colab.sandbox.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-py.ipynb"> |
| Run in Colab |
| </a> |
| <a class="button button--primary" target="_blank" |
| href="https://github.com/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-py.ipynb"> |
| View on GitHub |
| </a></p> |
| <p class="language-py">To learn how to install and run the Apache Beam Python SDK on your own computer, follow the instructions in the <a href="/get-started/quickstart-py">Python Quickstart</a>.</p> |
| <div class='language-go snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="kn">package</span> <span class="nx">main</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;context&#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;flag&#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;fmt&#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;regexp&#34;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;github.com/apache/beam/sdks/v2/go/pkg/beam&#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;github.com/apache/beam/sdks/v2/go/pkg/beam/io/textio&#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;github.com/apache/beam/sdks/v2/go/pkg/beam/runners/direct&#34;</span> |
| </span></span><span class="line"><span class="cl"> <span class="s">&#34;github.com/apache/beam/sdks/v2/go/pkg/beam/transforms/stats&#34;</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nx">_</span> <span class="s">&#34;github.com/apache/beam/sdks/v2/go/pkg/beam/io/filesystem/local&#34;</span> |
| </span></span><span class="line"><span class="cl"><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">var</span> <span class="p">(</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">input</span> <span class="p">=</span> <span class="nx">flag</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="s">&#34;input&#34;</span><span class="p">,</span> <span class="s">&#34;data/*&#34;</span><span class="p">,</span> <span class="s">&#34;File(s) to read.&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">output</span> <span class="p">=</span> <span class="nx">flag</span><span class="p">.</span><span class="nf">String</span><span class="p">(</span><span class="s">&#34;output&#34;</span><span class="p">,</span> <span class="s">&#34;outputs/wordcounts.txt&#34;</span><span class="p">,</span> <span class="s">&#34;Output filename.&#34;</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">var</span> <span class="nx">wordRE</span> <span class="p">=</span> <span class="nx">regexp</span><span class="p">.</span><span class="nf">MustCompile</span><span class="p">(</span><span class="s">`[a-zA-Z]+(&#39;[a-z])?`</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"><span class="kd">func</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">flag</span><span class="p">.</span><span class="nf">Parse</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nx">beam</span><span class="p">.</span><span class="nf">Init</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nx">pipeline</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">NewPipeline</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">root</span> <span class="o">:=</span> <span class="nx">pipeline</span><span class="p">.</span><span class="nf">Root</span><span class="p">()</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nx">lines</span> <span class="o">:=</span> <span class="nx">textio</span><span class="p">.</span><span class="nf">Read</span><span class="p">(</span><span class="nx">root</span><span class="p">,</span> <span class="o">*</span><span class="nx">input</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">words</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">root</span><span class="p">,</span> <span class="kd">func</span><span class="p">(</span><span class="nx">line</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">emit</span> <span class="kd">func</span><span class="p">(</span><span class="kt">string</span><span class="p">))</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">for</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">word</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">wordRE</span><span class="p">.</span><span class="nf">FindAllString</span><span class="p">(</span><span class="nx">line</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="nf">emit</span><span class="p">(</span><span class="nx">word</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">}</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="nx">lines</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">counted</span> <span class="o">:=</span> <span class="nx">stats</span><span class="p">.</span><span class="nf">Count</span><span class="p">(</span><span class="nx">root</span><span class="p">,</span> <span class="nx">words</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">formatted</span> <span class="o">:=</span> <span class="nx">beam</span><span class="p">.</span><span class="nf">ParDo</span><span class="p">(</span><span class="nx">root</span><span class="p">,</span> <span class="kd">func</span><span class="p">(</span><span class="nx">word</span> <span class="kt">string</span><span class="p">,</span> <span class="nx">count</span> <span class="kt">int</span><span class="p">)</span> <span class="kt">string</span> <span class="p">{</span> |
| </span></span><span class="line"><span class="cl"> <span class="k">return</span> <span class="nx">fmt</span><span class="p">.</span><span class="nf">Sprintf</span><span class="p">(</span><span class="s">&#34;%s: %v&#34;</span><span class="p">,</span> <span class="nx">word</span><span class="p">,</span> <span class="nx">count</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="p">},</span> <span class="nx">counted</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="nx">textio</span><span class="p">.</span><span class="nf">Write</span><span class="p">(</span><span class="nx">root</span><span class="p">,</span> <span class="o">*</span><span class="nx">output</span><span class="p">,</span> <span class="nx">formatted</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"> |
| </span></span><span class="line"><span class="cl"> <span class="nx">direct</span><span class="p">.</span><span class="nf">Execute</span><span class="p">(</span><span class="nx">context</span><span class="p">.</span><span class="nf">Background</span><span class="p">(),</span> <span class="nx">pipeline</span><span class="p">)</span> |
| </span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p class="language-go"><a class="button button--primary" target="_blank" |
| href="https://colab.sandbox.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-go.ipynb"> |
| Run in Colab |
| </a> |
| <a class="button button--primary" target="_blank" |
| href="https://github.com/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-go.ipynb"> |
| View on GitHub |
| </a></p> |
| <p class="language-go">To learn how to install and run the Apache Beam Go SDK on your own computer, follow the instructions in the <a href="/get-started/quickstart-go">Go Quickstart</a>.</p> |
| <p>For a more detailed explanation about how WordCount works, see the <a href="/get-started/wordcount-example">WordCount Example Walkthrough</a>.</p> |
| <h2 id="next-steps">Next Steps</h2> |
| <ul> |
| <li>Walk through additional WordCount examples in the <a href="/get-started/wordcount-example">WordCount Example Walkthrough</a>.</li> |
| <li>Take a self-paced tour through our <a href="/documentation/resources/learning-resources">Learning Resources</a>.</li> |
| <li>Dive in to some of our favorite <a href="/get-started/resources/videos-and-podcasts">Videos and Podcasts</a>.</li> |
| <li>Join the Beam <a href="/community/contact-us">users@</a> mailing list.</li> |
| <li>If you&rsquo;re interested in contributing to the Apache Beam codebase, see the <a href="/contribute">Contribution Guide</a>.</li> |
| </ul> |
| <p>Please don&rsquo;t hesitate to <a href="/community/contact-us">reach out</a> if you encounter any issues!</p></description></item><item><title>Get-Started: Try Beam Playground</title><link>/get-started/try-beam-playground/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/try-beam-playground/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="try-beam-playground">Try Beam Playground</h1> |
| <p>Beam Playground is an interactive environment to try out Beam transforms and examples |
| without having to install Apache Beam in your environment.</p> |
| <p>You can try the available Apache Beam examples at |
| <a href="https://play.beam.apache.org/">Beam Playground</a>.</p> |
| <h2 id="beam-playground-wordcount-example">Beam Playground WordCount Example</h2> |
| <div class="playground-wrapper"> |
| <div class="playground-snippets"> |
| <div |
| class="language-java playground-snippet" |
| data-sdk="java" |
| data-path="SDK_JAVA_MinimalWordCount" |
| ></div> |
| <div |
| class="language-py playground-snippet" |
| data-sdk="python" |
| data-path="SDK_PYTHON_WordCountWithMetrics" |
| ></div> |
| <div |
| class="language-go playground-snippet" |
| data-sdk="go" |
| data-path="SDK_GO_MinimalWordCount" |
| ></div> |
| <div |
| class="language-scio playground-snippet" |
| data-sdk="scio" |
| data-path="SDK_SCIO_MinimalWordCount" |
| ></div> |
| </div> |
| <div |
| class="code-snippet code-snippet-playground" |
| data-src="https://play.beam.apache.org/embedded?editable=1&examples=%5b%7b%22path%22%3a%22SDK_JAVA_MinimalWordCount%22%2c%22sdk%22%3a%22java%22%7d%2c%7b%22path%22%3a%22SDK_PYTHON_WordCountWithMetrics%22%2c%22sdk%22%3a%22python%22%7d%2c%7b%22path%22%3a%22SDK_GO_MinimalWordCount%22%2c%22sdk%22%3a%22go%22%7d%2c%7b%22path%22%3a%22SDK_SCIO_MinimalWordCount%22%2c%22sdk%22%3a%22scio%22%7d%5d" |
| data-width="100%" |
| data-height="700px" |
| ></div> |
| </div> |
| <p>See <a href="https://github.com/apache/beam/blob/master/playground/load_your_code.md">here</a> for adding new examples.</p> |
| <h2 id="next-steps">Next Steps</h2> |
| <ul> |
| <li>Try examples in <a href="https://play.beam.apache.org/">Apache Beam Playground</a>.</li> |
| <li>Submit feedback using &ldquo;Enjoying Playground?&rdquo; in |
| <a href="https://play.beam.apache.org/">Apache Beam Playground</a> or via |
| <a href="https://docs.google.com/forms/d/e/1FAIpQLSd5_5XeOwwW2yjEVHUXmiBad8Lxk-4OtNcgG45pbyAZzd4EbA/viewform?usp=pp_url">this form</a>.</li> |
| <li>Join the Beam <a href="/community/contact-us">users@</a> mailing list.</li> |
| <li>If you&rsquo;re interested in contributing to the Apache Beam Playground codebase, see the <a href="/contribute">Contribution Guide</a>.</li> |
| </ul> |
| <p>Please don&rsquo;t hesitate to <a href="/community/contact-us">reach out</a> if you encounter any issues!</p></description></item><item><title>Get-Started: Videos and Podcasts</title><link>/get-started/resources/videos-and-podcasts/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/resources/videos-and-podcasts/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="videos-and-podcasts">Videos and Podcasts</h1> |
| <p>This page provides links to some of our favorite videos and podcasts that will help you get started and learn more about Apache Beam.</p> |
| <ul><li><a href="">Introduction</a></li></ul> |
| <iframe class="video video--medium-size" width="560" height="315" src="https://www.youtube.com/embed/videoseries?list=PLIivdWyY5sqIEiHGunZXg_yoS7unlHNJt" frameborder="0" allowfullscreen></iframe> |
| <br> |
| <nav id="TableOfContents"> |
| <ul> |
| <li><a href="#general">General</a> |
| <ul> |
| <li><a href="#fundamentals-of-stream-processing-with-apache-beam">Fundamentals of Stream Processing with Apache Beam</a></li> |
| <li><a href="#apache-beam-a-unified-model-for-batch-and-streaming-data-processing">Apache Beam: A Unified Model for Batch and Streaming Data Processing</a></li> |
| <li><a href="#fundamentals-of-stream-processing-with-apache-beam-1">Fundamentals of Stream Processing with Apache Beam</a></li> |
| <li><a href="#software-engineering-radio-podcast-episode-272-apache-beam">Software Engineering Radio Podcast Episode 272: Apache Beam</a></li> |
| <li><a href="#how-to-run-ml-inference-with-apache-beam">How to run ML Inference with Apache Beam</a></li> |
| </ul> |
| </li> |
| <li><a href="#beam--friends">Beam &amp; Friends</a> |
| <ul> |
| <li><a href="#distributed-processing-for-machine-learning-production-pipelines">Distributed Processing for Machine Learning Production Pipelines</a></li> |
| <li><a href="#tensorflow-extended-an-end-to-end-machine-learning-platform-for-tensorflow">TensorFlow Extended: An End-to-End Machine Learning Platform for TensorFlow</a></li> |
| <li><a href="#flink-and-beam-current-state--roadmap">Flink and Beam: Current State &amp; Roadmap</a></li> |
| <li><a href="#lessons-learned-from-developing-a-stream-processing-platform-at-scale">Lessons learned from developing a stream processing platform at scale</a></li> |
| </ul> |
| </li> |
| <li><a href="#technical-details">Technical Details</a> |
| <ul> |
| <li><a href="#watermarks-time-and-progress-in-apache-beam-and-beyond">Watermarks: Time and Progress in Apache Beam and Beyond</a></li> |
| <li><a href="#triggers-in-apache-beam">Triggers in Apache Beam</a></li> |
| <li><a href="#nexmark-evaluating-big-data-systems-with-apache-beam">Nexmark Evaluating Big Data systems with Apache Beam</a></li> |
| </ul> |
| </li> |
| <li><a href="#next-steps">Next Steps</a></li> |
| </ul> |
| </nav> |
| <h2 id="general">General</h2> |
| <p>The following resources provide general overviews and fundamentals of Apache Beam.</p> |
| <h3 id="fundamentals-of-stream-processing-with-apache-beam">Fundamentals of Stream Processing with Apache Beam</h3> |
| <p>Data Science Summit, Jerusalem, 2016</p> |
| <p>Presented by Tyler Akidau, <em>Apache Beam PPMC member</em></p> |
| <iframe class="video video--medium-size" width="560" height="315" src="https://www.youtube.com/embed/V35MwYcXEX0" frameborder="0" allowfullscreen></iframe> |
| <br> |
| <h3 id="apache-beam-a-unified-model-for-batch-and-streaming-data-processing">Apache Beam: A Unified Model for Batch and Streaming Data Processing</h3> |
| <p>Hadoop Summit, San Jose, CA, 2016</p> |
| <p>Presented by Davor Bonaci, <em>Apache Beam PPMC member</em></p> |
| <iframe class="video video--medium-size" width="560" height="315" src="https://www.youtube.com/embed/7DZ8ONmeP5A" frameborder="0" allowfullscreen></iframe> |
| <br> |
| <h3 id="fundamentals-of-stream-processing-with-apache-beam-1">Fundamentals of Stream Processing with Apache Beam</h3> |
| <p>@Scale Conference, San Jose, CA, 2016</p> |
| <p>Presented by Dan Halperin, <em>Apache Beam PPMC member</em></p> |
| <p><a href="https://www.facebook.com/plugins/video.php?href=https%3A%2F%2Fwww.facebook.com%2Fatscaleevents%2Fvideos%2F1775945569345206%2F&amp;show_text=0&amp;width=560">Link to Video</a> |
| <br></p> |
| <h3 id="software-engineering-radio-podcast-episode-272-apache-beam">Software Engineering Radio Podcast Episode 272: Apache Beam</h3> |
| <p>Presented by Frances Perry, <em>Apache Beam PPMC member</em></p> |
| <p><a href="https://www.se-radio.net/2016/10/se-radio-episode-272-frances-perry-on-apache-beam/" target="_blank"><img src="/images/resources/se-radio-podcast.png" alt="alt text"></a> |
| <br></p> |
| <h3 id="how-to-run-ml-inference-with-apache-beam">How to run ML Inference with Apache Beam</h3> |
| <p>Video by Cassie Kozyrkov</p> |
| <iframe class="video video--medium-size" width="560" height="315" src="https://www.youtube.com/embed/ga2TNdrFRoU" frameborder="0" allowfullscreen></iframe> |
| <br> |
| <h2 id="beam--friends">Beam &amp; Friends</h2> |
| <p>The following resources present Apache Beam partnerships.</p> |
| <h3 id="distributed-processing-for-machine-learning-production-pipelines">Distributed Processing for Machine Learning Production Pipelines</h3> |
| <p>Flink Forward, 2020</p> |
| <p>Presented by Ahmet Altay, Robert Crowe, Reza Rokni</p> |
| <iframe class="video video--medium-size" width="560" height="315" src="https://www.youtube.com/embed/jV1WFTmm4qg" frameborder="0" allowfullscreen></iframe> |
| <br> |
| <h3 id="tensorflow-extended-an-end-to-end-machine-learning-platform-for-tensorflow">TensorFlow Extended: An End-to-End Machine Learning Platform for TensorFlow</h3> |
| <p>Spark+AI, San Francisco, 2019</p> |
| <p>Presented by Konstantinos Katsiapis, Ahmet Altay</p> |
| <iframe class="video video--medium-size" width="560" height="315" src="https://www.youtube.com/embed/GTibgKo7WaI" frameborder="0" allowfullscreen></iframe> |
| <br> |
| <h3 id="flink-and-beam-current-state--roadmap">Flink and Beam: Current State &amp; Roadmap</h3> |
| <p>Flink Forward, Berlin, 2016</p> |
| <p>Presented by Maximilian Michels, <em>Apache Beam PPMC member</em></p> |
| <iframe class="video video--medium-size" width="560" height="315" src="https://www.youtube.com/embed/msdjh6KRXC8" frameborder="0" allowfullscreen></iframe> |
| <br> |
| <h3 id="lessons-learned-from-developing-a-stream-processing-platform-at-scale">Lessons learned from developing a stream processing platform at scale</h3> |
| <p>Big Things Meetup, Tel Aviv, 2016</p> |
| <p>By Amit Sela, <em>Apache Beam PPMC member</em></p> |
| <iframe class="video video--medium-size" width="560" height="315" src="https://www.youtube.com/embed/fc-YigLn_gs" frameborder="0" allowfullscreen></iframe> |
| <br> |
| <h2 id="technical-details">Technical Details</h2> |
| <p>The following resources provide detailed explanations about technical concepts in Apache Beam.</p> |
| <h3 id="watermarks-time-and-progress-in-apache-beam-and-beyond">Watermarks: Time and Progress in Apache Beam and Beyond</h3> |
| <p>Strata+Hadoop World, New York, 2016</p> |
| <p>Presented by Slava Chernyak, <em>Software Engineer at Google</em></p> |
| <iframe class="video video--medium-size" width="560" height="315" src="https://www.youtube.com/embed/TWxSLmkWPm4" frameborder="0" allowfullscreen></iframe> |
| <br> |
| <h3 id="triggers-in-apache-beam">Triggers in Apache Beam</h3> |
| <p>Strata+Hadoop World, New York, 2016</p> |
| <p>Presented by Kenneth Knowles, <em>Apache Beam PPMC member</em></p> |
| <iframe class="video video--medium-size" width="560" height="315" src="https://www.youtube.com/embed/E1k0B9LN46M" frameborder="0" allowfullscreen></iframe> |
| <h3 id="nexmark-evaluating-big-data-systems-with-apache-beam">Nexmark Evaluating Big Data systems with Apache Beam</h3> |
| <p>ApacheCon, Miami, 2017</p> |
| <p>Presented by Etienne Chauchot and Ismaël Mejia, <em>Apache Beam PMC members</em></p> |
| <p><a href="www.slideshare.net/slideshow/embed_code/key/auWXjEK7GTkiUK">Link to Slides</a> |
| <audio controls></p> |
| <source src="https://feathercastapache.files.wordpress.com/2017/05/0517-04-mejia.mp3" type="audio/mpeg"> |
| Your browser does not support the audio element. |
| </audio> |
| ### Universal metrics with Apache Beam |
| <p>ApacheCon, Montreal, 2018</p> |
| <p>Presented by Etienne Chauchot, <em>Apache Beam PMC member</em></p> |
| <p><a href="www.slideshare.net/slideshow/embed_code/key/kKJRzR8HxkxLsR">Link to Slides</a> |
| <audio controls></p> |
| <source src="//feathercastapache.files.wordpress.com/2018/09/03-universal-metrics-with-beam-etienne-chauchot.mp3" type="audio/mpeg"> |
| Your browser does not support the audio element. |
| </audio> |
| <h2 id="next-steps">Next Steps</h2> |
| <ul> |
| <li>Take a self-paced tour through our <a href="/documentation/resources/learning-resources">Learning Resources</a>.</li> |
| </ul></description></item><item><title>Get-Started: WordCount quickstart for Java</title><link>/get-started/quickstart-java/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/quickstart-java/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="wordcount-quickstart-for-java">WordCount quickstart for Java</h1> |
| <p>This quickstart shows you how to set up a Java development environment and run |
| an <a href="/get-started/wordcount-example">example pipeline</a> written with the |
| <a href="/documentation/sdks/java">Apache Beam Java SDK</a>, using a |
| <a href="/documentation#runners">runner</a> of your choice.</p> |
| <p>If you&rsquo;re interested in contributing to the Apache Beam Java codebase, see the |
| <a href="/contribute">Contribution Guide</a>.</p> |
| <p>On this page:</p> |
| <nav id="TableOfContents"> |
| <ul> |
| <li><a href="#set-up-your-development-environment">Set up your development environment</a></li> |
| <li><a href="#get-the-example-code">Get the example code</a></li> |
| <li><a href="#optional-convert-from-maven-to-gradle">Optional: Convert from Maven to Gradle</a></li> |
| <li><a href="#get-sample-text">Get sample text</a></li> |
| <li><a href="#run-a-pipeline">Run a pipeline</a> |
| <ul> |
| <li><a href="#run-wordcount-using-maven">Run WordCount using Maven</a></li> |
| <li><a href="#run-wordcount-using-gradle">Run WordCount using Gradle</a></li> |
| </ul> |
| </li> |
| <li><a href="#inspect-the-results">Inspect the results</a></li> |
| <li><a href="#next-steps">Next Steps</a></li> |
| </ul> |
| </nav> |
| <h2 id="set-up-your-development-environment">Set up your development environment</h2> |
| <ol> |
| <li>Download and install the |
| <a href="https://www.oracle.com/technetwork/java/javase/downloads/index.html">Java Development Kit (JDK)</a> |
| version 8, 11, or 17. Verify that the |
| <a href="https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars001.html">JAVA_HOME</a> |
| environment variable is set and points to your JDK installation.</li> |
| <li>Download and install <a href="https://maven.apache.org/download.cgi">Apache Maven</a> by |
| following the <a href="https://maven.apache.org/install.html">installation guide</a> |
| for your operating system.</li> |
| <li>Optional: If you want to convert your Maven project to Gradle, install |
| <a href="https://gradle.org/install/">Gradle</a>.</li> |
| </ol> |
| <h2 id="get-the-example-code">Get the example code</h2> |
| <ol> |
| <li> |
| <p>Generate a Maven example project that builds against the latest Beam release: |
| <div class='shell-unix snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-unix" data-lang="unix">mvn archetype:generate \ |
| -DarchetypeGroupId=org.apache.beam \ |
| -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \ |
| -DarchetypeVersion=2.55.1 \ |
| -DgroupId=org.example \ |
| -DartifactId=word-count-beam \ |
| -Dversion=&#34;0.1&#34; \ |
| -Dpackage=org.apache.beam.examples \ |
| -DinteractiveMode=false |
| </code></pre> |
| </div> |
| </div> |
| <div class='shell-powerShell snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-powerShell" data-lang="powerShell"><span class="line"><span class="cl"><span class="n">mvn</span> <span class="n">archetype</span><span class="err">:</span><span class="n">generate</span> <span class="p">`</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">-D</span> <span class="n">archetypeGroupId</span><span class="p">=</span><span class="n">org</span><span class="p">.</span><span class="py">apache</span><span class="p">.</span><span class="py">beam</span> <span class="p">`</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">-D</span> <span class="n">archetypeArtifactId</span><span class="p">=</span><span class="nb">beam-sdks</span><span class="n">-java-maven-archetypes-examples</span> <span class="p">`</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">-D</span> <span class="n">archetypeVersion</span><span class="p">=</span><span class="mf">2.55</span><span class="p">.</span><span class="py">1</span> <span class="p">`</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">-D</span> <span class="n">groupId</span><span class="p">=</span><span class="n">org</span><span class="p">.</span><span class="py">example</span> <span class="p">`</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">-D</span> <span class="n">artifactId</span><span class="p">=</span><span class="nb">word-count</span><span class="n">-beam</span> <span class="p">`</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">-D</span> <span class="n">version</span><span class="p">=</span><span class="s2">&#34;0.1&#34;</span> <span class="p">`</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">-D</span> <span class="n">package</span><span class="p">=</span><span class="n">org</span><span class="p">.</span><span class="py">apache</span><span class="p">.</span><span class="py">beam</span><span class="p">.</span><span class="py">examples</span> <span class="p">`</span> |
| </span></span><span class="line"><span class="cl"> <span class="n">-D</span> <span class="n">interactiveMode</span><span class="p">=</span><span class="n">false</span> |
| </span></span><span class="line"><span class="cl"> </span></span></code></pre></div> |
| </div> |
| </div> |
| </p> |
| <p>Maven creates a new project in the <strong>word-count-beam</strong> directory.</p> |
| </li> |
| <li> |
| <p>Change into <strong>word-count-beam</strong>: |
| <div class='shell-unix snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-unix" data-lang="unix">cd word-count-beam/ |
| </code></pre> |
| </div> |
| </div> |
| <div class='shell-powerShell snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-powerShell" data-lang="powerShell"><span class="line"><span class="cl"><span class="nb">cd </span><span class="p">.\</span><span class="nb">word-count</span><span class="n">-beam</span> |
| </span></span><span class="line"><span class="cl"> </span></span></code></pre></div> |
| </div> |
| </div> |
| The directory contains a <strong>pom.xml</strong> and a <strong>src</strong> directory with example |
| pipelines.</p> |
| </li> |
| <li> |
| <p>List the example pipelines: |
| <div class='shell-unix snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-unix" data-lang="unix">ls src/main/java/org/apache/beam/examples/ |
| </code></pre> |
| </div> |
| </div> |
| <div class='shell-powerShell snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-powerShell" data-lang="powerShell"><span class="line"><span class="cl"><span class="nb">dir </span><span class="p">.\</span><span class="n">src</span><span class="p">\</span><span class="n">main</span><span class="p">\</span><span class="n">java</span><span class="p">\</span><span class="n">org</span><span class="p">\</span><span class="n">apache</span><span class="p">\</span><span class="n">beam</span><span class="p">\</span><span class="n">examples</span> |
| </span></span><span class="line"><span class="cl"> </span></span></code></pre></div> |
| </div> |
| </div> |
| You should see the following examples:</p> |
| <ul> |
| <li><strong>DebuggingWordCount.java</strong> (<a href="https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/DebuggingWordCount.java">GitHub</a>)</li> |
| <li><strong>MinimalWordCount.java</strong> (<a href="https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java">GitHub</a>)</li> |
| <li><strong>WindowedWordCount.java</strong> (<a href="https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WindowedWordCount.java">GitHub</a>)</li> |
| <li><strong>WordCount.java</strong> (<a href="https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/WordCount.java">GitHub</a>)</li> |
| </ul> |
| <p>The example used in this tutorial, <strong>WordCount.java</strong>, defines a |
| Beam pipeline that counts words from an input file (by default, a <strong>.txt</strong> |
| file containing Shakespeare&rsquo;s &ldquo;King Lear&rdquo;). To learn more about the examples, |
| see the <a href="/get-started/wordcount-example">WordCount Example Walkthrough</a>.</p> |
| </li> |
| </ol> |
| <h2 id="optional-convert-from-maven-to-gradle">Optional: Convert from Maven to Gradle</h2> |
| <p>The steps below explain how to convert the build from Maven to Gradle for the |
| following runners:</p> |
| <ul> |
| <li>Direct runner</li> |
| <li>Dataflow runner</li> |
| </ul> |
| <p>The conversion process for other runners is similar. For additional guidance, |
| see |
| <a href="https://docs.gradle.org/current/userguide/migrating_from_maven.html">Migrating Builds From Apache Maven</a>.</p> |
| <ol> |
| <li>In the directory with the <strong>pom.xml</strong> file, run the automated Maven-to-Gradle |
| conversion: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>gradle init |
| </code></pre> |
| </div> |
| </div> |
| You&rsquo;ll be asked if you want to generate a Gradle build. Enter <strong>yes</strong>. You&rsquo;ll |
| also be prompted to choose a DSL (Groovy or Kotlin). For this tutorial, enter |
| <strong>2</strong> for Kotlin.</li> |
| <li>Open the generated <strong>build.gradle.kts</strong> file and make the following changes: |
| <ol> |
| <li>In <code>repositories</code>, replace <code>mavenLocal()</code> with <code>mavenCentral()</code>.</li> |
| <li>In <code>repositories</code>, declare a repository for Confluent Kafka dependencies: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>maven { |
| url = uri(&#34;https://packages.confluent.io/maven/&#34;) |
| } |
| </code></pre> |
| </div> |
| </div> |
| </li> |
| <li>At the end of the build script, add the following conditional dependency: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>if (project.hasProperty(&#34;dataflow-runner&#34;)) { |
| dependencies { |
| runtimeOnly(&#34;org.apache.beam:beam-runners-google-cloud-dataflow-java:2.55.1&#34;) |
| } |
| } |
| </code></pre> |
| </div> |
| </div> |
| </li> |
| <li>At the end of the build script, add the following task: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>task(&#34;execute&#34;, JavaExec::class) { |
| classpath = sourceSets[&#34;main&#34;].runtimeClasspath |
| mainClass.set(System.getProperty(&#34;mainClass&#34;)) |
| } |
| </code></pre> |
| </div> |
| </div> |
| </li> |
| </ol> |
| </li> |
| <li>Build your project: |
| <div class="snippet"> |
| <div class="notebook-skip code-snippet without_switcher"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code>gradle build |
| </code></pre> |
| </div> |
| </div> |
| </li> |
| </ol> |
| <h2 id="get-sample-text">Get sample text</h2> |
| <blockquote> |
| <p>If you&rsquo;re planning to use the DataflowRunner, you can skip this step. The |
| runner will pull text directly from Google Cloud Storage.</p> |
| </blockquote> |
| <ol> |
| <li>In the <strong>word-count-beam</strong> directory, create a file called <strong>sample.txt</strong>.</li> |
| <li>Add some text to the file. For this example, use the text of Shakespeare&rsquo;s |
| <a href="https://storage.cloud.google.com/apache-beam-samples/shakespeare/kinglear.txt">King Lear</a>.</li> |
| </ol> |
| <h2 id="run-a-pipeline">Run a pipeline</h2> |
| <p>A single Beam pipeline can run on multiple Beam |
| <a href="/documentation#runners">runners</a>. The |
| <a href="/documentation/runners/direct">DirectRunner</a> is useful for getting started, |
| because it runs on your machine and requires no specific setup. If you&rsquo;re just |
| trying out Beam and you&rsquo;re not sure what to use, use the |
| <a href="/documentation/runners/direct">DirectRunner</a>.</p> |
| <p>The general process for running a pipeline goes like this:</p> |
| <ol> |
| <li>Complete any runner-specific setup.</li> |
| <li>Build your command line: |
| <ol> |
| <li>Specify a runner with <code>--runner=&lt;runner&gt;</code> (defaults to the |
| <a href="/documentation/runners/direct">DirectRunner</a>).</li> |
| <li>Add any runner-specific required options.</li> |
| <li>Choose input files and an output location that are accessible to the |
| runner. (For example, you can&rsquo;t access a local file if you are running |
| the pipeline on an external cluster.)</li> |
| </ol> |
| </li> |
| <li>Run the command.</li> |
| </ol> |
| <p>To run the WordCount pipeline:</p> |
| <ol> |
| <li> |
| <p>Follow the setup steps for your runner:</p> |
| <ul> |
| <li><a href="/documentation/runners/flink">FlinkRunner</a></li> |
| <li><a href="/documentation/runners/spark">SparkRunner</a></li> |
| <li><a href="/documentation/runners/dataflow">DataflowRunner</a></li> |
| <li><a href="/documentation/runners/samza">SamzaRunner</a></li> |
| <li><a href="/documentation/runners/nemo">NemoRunner</a></li> |
| <li><a href="/documentation/runners/jet">JetRunner</a></li> |
| </ul> |
| <p>The DirectRunner will work without additional setup.</p> |
| </li> |
| <li> |
| <p>Run the corresponding Maven or Gradle command below.</p> |
| </li> |
| </ol> |
| <h3 id="run-wordcount-using-maven">Run WordCount using Maven</h3> |
| <p>For Unix shells:</p> |
| <p> |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ |
| -Dexec.args=&#34;--inputFile=sample.txt --output=counts&#34; -Pdirect-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-flink snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flink" data-lang="flink">mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ |
| -Dexec.args=&#34;--runner=FlinkRunner --inputFile=sample.txt --output=counts&#34; -Pflink-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-flinkCluster snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flinkCluster" data-lang="flinkCluster">mvn package exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ |
| -Dexec.args=&#34;--runner=FlinkRunner --flinkMaster=&lt;flink master&gt; --filesToStage=target/word-count-beam-bundled-0.1.jar \ |
| --inputFile=sample.txt --output=/tmp/counts&#34; -Pflink-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark">mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ |
| -Dexec.args=&#34;--runner=SparkRunner --inputFile=sample.txt --output=counts&#34; -Pspark-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow">mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ |
| -Dexec.args=&#34;--runner=DataflowRunner --project=&lt;your-gcp-project&gt; \ |
| --region=&lt;your-gcp-region&gt; \ |
| --gcpTempLocation=gs://&lt;your-gcs-bucket&gt;/tmp \ |
| --inputFile=gs://apache-beam-samples/shakespeare/* --output=gs://&lt;your-gcs-bucket&gt;/counts&#34; \ |
| -Pdataflow-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-samza snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-samza" data-lang="samza">mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \ |
| -Dexec.args=&#34;--inputFile=sample.txt --output=/tmp/counts --runner=SamzaRunner&#34; -Psamza-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-nemo snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-nemo" data-lang="nemo">mvn package -Pnemo-runner &amp;&amp; java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.WordCount \ |
| --runner=NemoRunner --inputFile=`pwd`/sample.txt --output=counts</code></pre> |
| </div> |
| </div> |
| <div class='runner-jet snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-jet" data-lang="jet">mvn package -Pjet-runner |
| java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.WordCount \ |
| --runner=JetRunner --jetLocalMode=3 --inputFile=`pwd`/sample.txt --output=counts</code></pre> |
| </div> |
| </div> |
| </p> |
| <p>For Windows PowerShell:</p> |
| <p> |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">mvn compile exec:java -D exec.mainClass=org.apache.beam.examples.WordCount ` |
| -D exec.args=&#34;--inputFile=sample.txt --output=counts&#34; -P direct-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-flink snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flink" data-lang="flink">mvn compile exec:java -D exec.mainClass=org.apache.beam.examples.WordCount ` |
| -D exec.args=&#34;--runner=FlinkRunner --inputFile=sample.txt --output=counts&#34; -P flink-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-flinkCluster snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flinkCluster" data-lang="flinkCluster">mvn package exec:java -D exec.mainClass=org.apache.beam.examples.WordCount ` |
| -D exec.args=&#34;--runner=FlinkRunner --flinkMaster=&lt;flink master&gt; --filesToStage=.\target\word-count-beam-bundled-0.1.jar ` |
| --inputFile=C:\path\to\quickstart\sample.txt --output=C:\tmp\counts&#34; -P flink-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark">mvn compile exec:java -D exec.mainClass=org.apache.beam.examples.WordCount ` |
| -D exec.args=&#34;--runner=SparkRunner --inputFile=sample.txt --output=counts&#34; -P spark-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow">mvn compile exec:java -D exec.mainClass=org.apache.beam.examples.WordCount ` |
| -D exec.args=&#34;--runner=DataflowRunner --project=&lt;your-gcp-project&gt; ` |
| --region=&lt;your-gcp-region&gt; \ |
| --gcpTempLocation=gs://&lt;your-gcs-bucket&gt;/tmp ` |
| --inputFile=gs://apache-beam-samples/shakespeare/* --output=gs://&lt;your-gcs-bucket&gt;/counts&#34; ` |
| -P dataflow-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-samza snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-samza" data-lang="samza">mvn compile exec:java -D exec.mainClass=org.apache.beam.examples.WordCount ` |
| -D exec.args=&#34;--inputFile=sample.txt --output=/tmp/counts --runner=SamzaRunner&#34; -P samza-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-nemo snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-nemo" data-lang="nemo">mvn package -P nemo-runner -DskipTests |
| java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.WordCount ` |
| --runner=NemoRunner --inputFile=`pwd`/sample.txt --output=counts</code></pre> |
| </div> |
| </div> |
| <div class='runner-jet snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-jet" data-lang="jet">mvn package -P jet-runner |
| java -cp target/word-count-beam-bundled-0.1.jar org.apache.beam.examples.WordCount ` |
| --runner=JetRunner --jetLocalMode=3 --inputFile=$pwd/sample.txt --output=counts</code></pre> |
| </div> |
| </div> |
| </p> |
| <h3 id="run-wordcount-using-gradle">Run WordCount using Gradle</h3> |
| <p>For Unix shells:</p> |
| <p> |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">gradle clean execute -DmainClass=org.apache.beam.examples.WordCount \ |
| --args=&#34;--inputFile=sample.txt --output=counts&#34;</code></pre> |
| </div> |
| </div> |
| <div class='runner-flink snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flink" data-lang="flink">TODO: document Flink on Gradle: https://github.com/apache/beam/issues/21498</code></pre> |
| </div> |
| </div> |
| <div class='runner-flinkCluster snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flinkCluster" data-lang="flinkCluster">TODO: document FlinkCluster on Gradle: https://github.com/apache/beam/issues/21499</code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark">TODO: document Spark on Gradle: https://github.com/apache/beam/issues/21502</code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow">gradle clean execute -DmainClass=org.apache.beam.examples.WordCount \ |
| --args=&#34;--project=&lt;your-gcp-project&gt; --inputFile=gs://apache-beam-samples/shakespeare/* \ |
| --output=gs://&lt;your-gcs-bucket&gt;/counts&#34; -Pdataflow-runner</code></pre> |
| </div> |
| </div> |
| <div class='runner-samza snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-samza" data-lang="samza">TODO: document Samza on Gradle: https://github.com/apache/beam/issues/21500</code></pre> |
| </div> |
| </div> |
| <div class='runner-nemo snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-nemo" data-lang="nemo">TODO: document Nemo on Gradle: https://github.com/apache/beam/issues/21503</code></pre> |
| </div> |
| </div> |
| <div class='runner-jet snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-jet" data-lang="jet">TODO: document Jet on Gradle: https://github.com/apache/beam/issues/21501</code></pre> |
| </div> |
| </div> |
| </p> |
| <h2 id="inspect-the-results">Inspect the results</h2> |
| <p>After the pipeline has completed, you can view the output. There might be |
| multiple output files prefixed by <code>count</code>. The number of output files is decided |
| by the runner, giving it the flexibility to do efficient, distributed execution.</p> |
| <ol> |
| <li>View the output files in a Unix shell: |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">ls counts* |
| </code></pre> |
| </div> |
| </div> |
| <div class='runner-flink snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flink" data-lang="flink">ls counts* |
| </code></pre> |
| </div> |
| </div> |
| <div class='runner-flinkCluster snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flinkCluster" data-lang="flinkCluster">ls /tmp/counts* |
| </code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark">ls counts* |
| </code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow">gsutil ls gs://&lt;your-gcs-bucket&gt;/counts* |
| </code></pre> |
| </div> |
| </div> |
| <div class='runner-samza snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-samza" data-lang="samza">ls /tmp/counts* |
| </code></pre> |
| </div> |
| </div> |
| <div class='runner-nemo snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-nemo" data-lang="nemo">ls counts* |
| </code></pre> |
| </div> |
| </div> |
| <div class='runner-jet snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-jet" data-lang="jet">ls counts* |
| </code></pre> |
| </div> |
| </div> |
| The output files contain unique words and the number of occurrences of each |
| word.</li> |
| <li>View the output content in a Unix shell: |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">more counts* |
| </code></pre> |
| </div> |
| </div> |
| <div class='runner-flink snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flink" data-lang="flink">more counts* |
| </code></pre> |
| </div> |
| </div> |
| <div class='runner-flinkCluster snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flinkCluster" data-lang="flinkCluster">more /tmp/counts* |
| </code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark">more counts* |
| </code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow">gsutil cat gs://&lt;your-gcs-bucket&gt;/counts* |
| </code></pre> |
| </div> |
| </div> |
| <div class='runner-samza snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-samza" data-lang="samza">more /tmp/counts* |
| </code></pre> |
| </div> |
| </div> |
| <div class='runner-nemo snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-nemo" data-lang="nemo">more counts* |
| </code></pre> |
| </div> |
| </div> |
| <div class='runner-jet snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-jet" data-lang="jet">more counts* |
| </code></pre> |
| </div> |
| </div> |
| The order of elements is not guaranteed, to allow runners to optimize for |
| efficiency. But the output should look something like this: |
| <pre tabindex="0"><code>... |
| Think: 3 |
| slower: 1 |
| Having: 1 |
| revives: 1 |
| these: 33 |
| wipe: 1 |
| arrives: 1 |
| concluded: 1 |
| begins: 3 |
| ... |
| </code></pre></li> |
| </ol> |
| <h2 id="next-steps">Next Steps</h2> |
| <ul> |
| <li>Learn more about the <a href="/documentation/sdks/java/">Beam SDK for Java</a> |
| and look through the |
| <a href="https://beam.apache.org/releases/javadoc">Java SDK API reference</a>.</li> |
| <li>Walk through the WordCount examples in the |
| <a href="/get-started/wordcount-example">WordCount Example Walkthrough</a>.</li> |
| <li>Take a self-paced tour through our |
| <a href="/documentation/resources/learning-resources">Learning Resources</a>.</li> |
| <li>Dive in to some of our favorite |
| <a href="/get-started/resources/videos-and-podcasts">Videos and Podcasts</a>.</li> |
| <li>Join the Beam <a href="/community/contact-us">users@</a> mailing list.</li> |
| </ul> |
| <p>Please don&rsquo;t hesitate to <a href="/community/contact-us">reach out</a> if you encounter any |
| issues!</p></description></item><item><title>Get-Started: WordCount Quickstart for Python</title><link>/get-started/quickstart-py/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/get-started/quickstart-py/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="wordcount-quickstart-for-python">WordCount quickstart for Python</h1> |
| <p>This guide shows you how to set up your Python development environment, get the Apache Beam SDK for Python, and run an example pipeline.</p> |
| <p>If you&rsquo;re interested in contributing to the Apache Beam Python codebase, see the <a href="/contribute">Contribution Guide</a>.</p> |
| <nav id="TableOfContents"> |
| <ul> |
| <li><a href="#set-up-your-environment">Set up your environment</a></li> |
| <li><a href="#get-apache-beam">Get Apache Beam</a> |
| <ul> |
| <li><a href="#create-and-activate-a-virtual-environment">Create and activate a virtual environment</a></li> |
| <li><a href="#download-and-install">Download and install</a> |
| <ul> |
| <li><a href="#extra-requirements">Extra requirements</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li><a href="#execute-a-pipeline">Execute a pipeline</a></li> |
| <li><a href="#next-steps">Next Steps</a></li> |
| </ul> |
| </nav> |
| <p>The Python SDK supports Python 3.8, 3.9, 3.10 and 3.11. Beam 2.48.0 was the last release with support for Python 3.7.</p> |
| <h2 id="set-up-your-environment">Set up your environment</h2> |
| <p>For details, see |
| <a href="/get-started/quickstart/python#set-up-your-development-environment">Set up your development environment</a>.</p> |
| <h2 id="get-apache-beam">Get Apache Beam</h2> |
| <h3 id="create-and-activate-a-virtual-environment">Create and activate a virtual environment</h3> |
| <p>A virtual environment is a directory tree containing its own Python distribution. To create a virtual environment, run:</p> |
| <div class='shell-unix snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-unix" data-lang="unix">python -m venv /path/to/directory</code></pre> |
| </div> |
| </div> |
| <div class='shell-powerShell snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-powerShell" data-lang="powerShell"><span class="line"><span class="cl"><span class="n">PS</span><span class="p">&gt;</span> <span class="n">python</span> <span class="n">-m</span> <span class="n">venv</span> <span class="n">C:</span><span class="p">\</span><span class="n">path</span><span class="p">\</span><span class="n">to</span><span class="p">\</span><span class="n">directory</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>A virtual environment needs to be activated for each shell that is to use it. |
| Activating it sets some environment variables that point to the virtual |
| environment&rsquo;s directories.</p> |
| <p>To activate a virtual environment in Bash, run:</p> |
| <div class='shell-unix snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-unix" data-lang="unix">. /path/to/directory/bin/activate</code></pre> |
| </div> |
| </div> |
| <div class='shell-powerShell snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-powerShell" data-lang="powerShell"><span class="line"><span class="cl"><span class="n">PS</span><span class="p">&gt;</span> <span class="n">C:</span><span class="p">\</span><span class="n">path</span><span class="p">\</span><span class="n">to</span><span class="p">\</span><span class="n">directory</span><span class="p">\</span><span class="n">Scripts</span><span class="p">\</span><span class="n">activate</span><span class="p">.</span><span class="n">ps1</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <p>That is, execute the <code>activate</code> script under the virtual environment directory you created.</p> |
| <p>For instructions using other shells, see the <a href="https://docs.python.org/3/library/venv.html">venv documentation</a>.</p> |
| <h3 id="download-and-install">Download and install</h3> |
| <p>Install the latest Python SDK from PyPI:</p> |
| <div class='shell-unix snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-unix" data-lang="unix">pip install apache-beam</code></pre> |
| </div> |
| </div> |
| <div class='shell-powerShell snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-powerShell" data-lang="powerShell"><span class="line"><span class="cl"><span class="n">PS</span><span class="p">&gt;</span> <span class="n">python</span> <span class="n">-m</span> <span class="n">pip</span> <span class="n">install</span> <span class="nb">apache-beam</span></span></span></code></pre></div> |
| </div> |
| </div> |
| <h4 id="extra-requirements">Extra requirements</h4> |
| <p>The above installation will not install all the extra dependencies for using features like the Google Cloud Dataflow runner. Information on what extra packages are required for different features are highlighted below. It is possible to install multiple extra requirements using something like <code>pip install 'apache-beam[feature1,feature2]'</code>.</p> |
| <ul> |
| <li><strong>Google Cloud Platform</strong> |
| <ul> |
| <li>Installation Command: <code>pip install 'apache-beam[gcp]'</code></li> |
| <li>Required for: |
| <ul> |
| <li>Google Cloud Dataflow Runner</li> |
| <li>GCS IO</li> |
| <li>Datastore IO</li> |
| <li>BigQuery IO</li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li><strong>Amazon Web Services</strong> |
| <ul> |
| <li>Installation Command: <code>pip install 'apache-beam[aws]'</code></li> |
| <li>Required for I/O connectors interfacing with AWS</li> |
| </ul> |
| </li> |
| <li><strong>Microsoft Azure</strong> |
| <ul> |
| <li>Installation Command: <code>pip install 'apache-beam[azure]'</code></li> |
| <li>Required for I/O connectors interfacing with Microsoft Azure</li> |
| </ul> |
| </li> |
| <li><strong>Beam YAML API</strong> |
| <ul> |
| <li>Installation Command: <code>pip install 'apache-beam[yaml]'</code></li> |
| <li>Required for using <a href="/documentation/sdks/yaml/">Beam YAML API</a></li> |
| </ul> |
| </li> |
| <li><strong>Beam YAML Dataframe API</strong> |
| <ul> |
| <li>Installation Command: <code>pip install 'apache-beam[dataframe]'</code></li> |
| <li>Required for using <a href="/documentation/dsls/dataframes/overview/">Beam Dataframe API</a></li> |
| </ul> |
| </li> |
| <li><strong>Tests</strong> |
| <ul> |
| <li>Installation Command: <code>pip install 'apache-beam[test]'</code></li> |
| <li>Required for developing Beam and running unit tests</li> |
| </ul> |
| </li> |
| <li><strong>Docs</strong> |
| <ul> |
| <li>Installation Command: <code>pip install 'apache-beam[docs]'</code></li> |
| <li>Required for generating API documentation using Sphinx</li> |
| </ul> |
| </li> |
| </ul> |
| <h2 id="execute-a-pipeline">Execute a pipeline</h2> |
| <p>The Apache Beam <a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples">examples</a> directory has many examples. All examples can be run locally by passing the required arguments described in the example script.</p> |
| <p>For example, run <code>wordcount.py</code> with the following command:</p> |
| <div class='runner-direct snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-direct" data-lang="direct">python -m apache_beam.examples.wordcount --input /path/to/inputfile --output /path/to/write/counts</code></pre> |
| </div> |
| </div> |
| <div class='runner-flink snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-flink" data-lang="flink">python -m apache_beam.examples.wordcount --input /path/to/inputfile \ |
| --output /path/to/write/counts \ |
| --runner FlinkRunner</code></pre> |
| </div> |
| </div> |
| <div class='runner-spark snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-spark" data-lang="spark">python -m apache_beam.examples.wordcount --input /path/to/inputfile \ |
| --output /path/to/write/counts \ |
| --runner SparkRunner</code></pre> |
| </div> |
| </div> |
| <div class='runner-dataflow snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-dataflow" data-lang="dataflow"># As part of the initial setup, install Google Cloud Platform specific extra components. Make sure you |
| # complete the setup steps at /documentation/runners/dataflow/#setup |
| pip install apache-beam[gcp] |
| python -m apache_beam.examples.wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \ |
| --output gs://&lt;your-gcs-bucket&gt;/counts \ |
| --runner DataflowRunner \ |
| --project your-gcp-project \ |
| --region your-gcp-region \ |
| --temp_location gs://&lt;your-gcs-bucket&gt;/tmp/</code></pre> |
| </div> |
| </div> |
| <div class='runner-nemo snippet'> |
| <div class="notebook-skip code-snippet"> |
| <a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard"> |
| <img src="/images/copy-icon.svg"/> |
| </a> |
| <pre tabindex="0"><code class="language-nemo" data-lang="nemo">This runner is not yet available for the Python SDK.</code></pre> |
| </div> |
| </div> |
| <p>After the pipeline completes, you can view the output files at your specified |
| output path. For example, if you specify <code>/dir1/counts</code> for the <code>--output</code> |
| parameter, the pipeline writes the files to <code>/dir1/</code> and names the files |
| sequentially in the format <code>counts-0000-of-0001</code>.</p> |
| <h2 id="next-steps">Next Steps</h2> |
| <ul> |
| <li>Learn more about the <a href="/documentation/sdks/python/">Beam SDK for Python</a> |
| and look through the <a href="https://beam.apache.org/releases/pydoc">Python SDK API reference</a>.</li> |
| <li>Get <a href="/get-started/an-interactive-overview-of-beam">An Interactive Overview of Beam</a></li> |
| <li>Walk through these WordCount examples in the <a href="/get-started/wordcount-example">WordCount Example Walkthrough</a>.</li> |
| <li>Take a self-paced tour through our <a href="/documentation/resources/learning-resources">Learning Resources</a>.</li> |
| <li>Dive in to some of our favorite <a href="/get-started/resources/videos-and-podcasts">Videos and Podcasts</a>.</li> |
| <li>Join the Beam <a href="/community/contact-us">users@</a> mailing list.</li> |
| </ul> |
| <p>Please don&rsquo;t hesitate to <a href="/community/contact-us">reach out</a> if you encounter any issues!</p></description></item></channel></rss> |
