| <rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title>Apache Beam</title><description>Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service). Beam also brings DSL in different languages, allowing users to easily implement their data integration processes.</description><link>/</link><generator>Hugo -- gohugo.io</generator><item><title>Apache Beam 2.56.0</title><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.56.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2550-2023-03-25">download page</a> for this release.</p> |
| <p>For more information on changes in 2.56.0, check out the <a href="https://github.com/apache/beam/milestone/20">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Added FlinkRunner for Flink 1.17, removed support for Flink 1.12 and 1.13. Previous version of Pipeline running on Flink 1.16 and below can be upgraded to 1.17, if the Pipeline is first updated to Beam 2.56.0 with the same Flink version. After Pipeline runs with Beam 2.56.0, it should be possible to upgrade to FlinkRunner with Flink 1.17. (<a href="https://github.com/apache/beam/issues/29939">#29939</a>)</li> |
| <li>New Managed I/O Java API (<a href="https://github.com/apache/beam/pull/30830">#30830</a>).</li> |
| <li>New Ordered Processing PTransform added for processing order-sensitive stateful data (<a href="https://github.com/apache/beam/pull/30735">#30735</a>).</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Upgraded Avro version to 1.11.3, kafka-avro-serializer and kafka-schema-registry-client versions to 7.6.0 (Java) (<a href="https://github.com/apache/beam/pull/30638">#30638</a>). |
| The newer Avro package is known to have breaking changes. If you are affected, you can keep pinned to older Avro versions which are also tested with Beam.</li> |
| <li>Iceberg read/write support is available through the new Managed I/O Java API (<a href="https://github.com/apache/beam/pull/30830">#30830</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Profiling of Cythonized code has been disabled by default. This might improve performance for some Python pipelines (<a href="https://github.com/apache/beam/pull/30938">#30938</a>).</li> |
| <li>Bigtable enrichment handler now accepts a custom function to build a composite row key. (Python) (<a href="https://github.com/apache/beam/issues/30975">#30974</a>).</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>Default consumer polling timeout for KafkaIO.Read was increased from 1 second to 2 seconds. Use KafkaIO.read().withConsumerPollingTimeout(Duration duration) to configure this timeout value when necessary (<a href="https://github.com/apache/beam/issues/30870">#30870</a>).</li> |
| <li>Python Dataflow users no longer need to manually specify &ndash;streaming for pipelines using unbounded sources such as ReadFromPubSub.</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed locking issue when shutting down inactive bundle processors. Symptoms of this issue include slowness or stuckness in long-running jobs (Python) (<a href="https://github.com/apache/beam/pull/30679">#30679</a>).</li> |
| <li>Fixed logging issue that caused silecing the pip output when installing of dependencies provided in <code>--requirements_file</code> (Python).</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.56.0 release. Thank you to all contributors!</p> |
| <p>Abacn</p> |
| <p>Ahmed Abualsaud</p> |
| <p>Andrei Gurau</p> |
| <p>Andrey Devyatkin</p> |
| <p>Aravind Pedapudi</p> |
| <p>Arun Pandian</p> |
| <p>Arvind Ram</p> |
| <p>Bartosz Zablocki</p> |
| <p>Brachi Packter</p> |
| <p>Byron Ellis</p> |
| <p>Chamikara Jayalath</p> |
| <p>Clement DAL PALU</p> |
| <p>Damon</p> |
| <p>Danny McCormick</p> |
| <p>Daria Bezkorovaina</p> |
| <p>Dip Patel</p> |
| <p>Evan Burrell</p> |
| <p>Hai Joey Tran</p> |
| <p>Jack McCluskey</p> |
| <p>Jan Lukavský</p> |
| <p>JayajP</p> |
| <p>Jeff Kinard</p> |
| <p>Julien Tournay</p> |
| <p>Kenneth Knowles</p> |
| <p>Luís Bianchin</p> |
| <p>Maciej Szwaja</p> |
| <p>Melody Shen</p> |
| <p>Oleh Borysevych</p> |
| <p>Pablo Estrada</p> |
| <p>Rebecca Szper</p> |
| <p>Ritesh Ghorse</p> |
| <p>Robert Bradshaw</p> |
| <p>Sam Whittle</p> |
| <p>Sergei Lilichenko</p> |
| <p>Shahar Epstein</p> |
| <p>Shunping Huang</p> |
| <p>Svetak Sundhar</p> |
| <p>Timothy Itodo</p> |
| <p>Veronica Wasson</p> |
| <p>Vitaly Terentyev</p> |
| <p>Vlado Djerek</p> |
| <p>Yi Hu</p> |
| <p>akashorabek</p> |
| <p>bzablocki</p> |
| <p>clmccart</p> |
| <p>damccorm</p> |
| <p>dependabot[bot]</p> |
| <p>dmitryor</p> |
| <p>github-actions[bot]</p> |
| <p>liferoad</p> |
| <p>martin trieu</p> |
| <p>tvalentyn</p> |
| <p>xianhualiu</p></description><link>/blog/beam-2.56.0/</link><pubDate>Wed, 01 May 2024 10:00:00 -0400</pubDate><guid>/blog/beam-2.56.0/</guid><category>blog</category><category>release</category></item><item><title>Introducing Beam YAML: Apache Beam's First No-code SDK</title><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>Writing a Beam pipeline can be a daunting task. Learning the Beam model, downloading dependencies for the SDK language |
| of choice, debugging the pipeline, and maintaining the pipeline code is a lot of overhead for users who want to write a |
| simple to intermediate data processing pipeline. There have been strides in making the SDK&rsquo;s entry points easier, but |
| for many, it is still a long way from being a painless process.</p> |
| <p>To address some of these issues and simplify the entry point to Beam, we have introduced a new way to specify Beam |
| pipelines by using configuration files rather than code. This new SDK, known as |
| <a href="https://beam.apache.org/documentation/sdks/yaml/">Beam YAML</a>, employs a declarative approach to creating |
| data processing pipelines using <a href="https://yaml.org/">YAML</a>, a widely used data serialization language.</p> |
| <h1 id="benefits-of-using-beam-yaml">Benefits of using Beam YAML</h1> |
| <p>The primary goal of Beam YAML is to make the entry point to Beam as welcoming as possible. However, this should not |
| come at the expense of sacrificing the rich features that Beam offers.</p> |
| <p>Here are some of the benefits of using Beam YAML:</p> |
| <ul> |
| <li><strong>No-code development:</strong> Allows users to develop pipelines without writing any code. This makes it easier to get |
| started with Beam and to develop pipelines quickly and easily.</li> |
| <li><strong>Maintainability</strong>: Configuration-based pipelines are easier to maintain than code-based pipelines. YAML format |
| enables clear separation of concerns, simplifying changes and updates without affecting other code sections.</li> |
| <li><strong>Declarative language:</strong> Provides a declarative language, which means that it is based on the description of the |
| desired outcome rather than expressing the intent through code. This makes it easy to understand the structure and |
| flow of a pipeline. The YAML syntax is also widely used with a rich community of resources for learning and |
| leveraging the YAML syntax.</li> |
| <li><strong>Powerful features:</strong> Supports a wide range of features, including a variety of data sources and sinks, turn-key |
| transforms, and execution parameters. This makes it possible to develop complex data processing pipelines with Beam |
| YAML.</li> |
| <li><strong>Reusability</strong>: Beam YAML promotes code reuse by providing a way to define and share common pipeline patterns. You |
| can create reusable YAML snippets or blocks that can be easily shared and reused in different pipelines. This reduces |
| the need to write repetitive tasks and helps maintain consistency across pipelines.</li> |
| <li><strong>Extensibility</strong>: Beam YAML offers a structure for integrating custom transformations into a pipeline, enabling |
| organizations to contribute or leverage a pre-existing catalog of transformations that can be seamlessly accessed |
| using the Beam YAML syntax across multiple pipelines. It is also possible to build third-party extensions, including |
| custom parsers and other tools, that do not need to depend on Beam directly.</li> |
| <li><strong>Backwards Compatibility</strong>: Beam YAML is still being actively worked on, bringing exciting new features and |
| capabilities, but as these features are added, backwards compatibility will be preserved. This way, once a pipeline |
| is written, it will continue to work despite future released versions of the SDK.</li> |
| </ul> |
| <p>Overall, using Beam YAML provides a number of advantages. It makes pipeline development and management more efficient |
| and effective, enabling users to focus on the business logic and data processing tasks, rather than spending time on |
| low-level coding details.</p> |
| <h1 id="case-study-a-simple-business-analytics-use-case">Case Study: A simple business analytics use-case</h1> |
| <p>Let&rsquo;s take the following sample transaction data for a department store:</p> |
| <table> |
| <thead> |
| <tr> |
| <th style="text-align:left">transaction_id</th> |
| <th style="text-align:left">product_name</th> |
| <th style="text-align:left">category</th> |
| <th style="text-align:left">price</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td style="text-align:left">T0012</td> |
| <td style="text-align:left">Headphones</td> |
| <td style="text-align:left">Electronics</td> |
| <td style="text-align:left">59.99</td> |
| </tr> |
| <tr> |
| <td style="text-align:left">T5034</td> |
| <td style="text-align:left">Leather Jacket</td> |
| <td style="text-align:left">Apparel</td> |
| <td style="text-align:left">109.99</td> |
| </tr> |
| <tr> |
| <td style="text-align:left">T0024</td> |
| <td style="text-align:left">Aluminum Mug</td> |
| <td style="text-align:left">Kitchen</td> |
| <td style="text-align:left">29.99</td> |
| </tr> |
| <tr> |
| <td style="text-align:left">T0104</td> |
| <td style="text-align:left">Headphones</td> |
| <td style="text-align:left">Electronics</td> |
| <td style="text-align:left">59.99</td> |
| </tr> |
| <tr> |
| <td style="text-align:left">T0302</td> |
| <td style="text-align:left">Monitor</td> |
| <td style="text-align:left">Electronics</td> |
| <td style="text-align:left">249.99</td> |
| </tr> |
| </tbody> |
| </table> |
| <p>Now, let&rsquo;s say that the business wants to get a record of transactions for all purchases made in the Electronics |
| department for audit purposes. Assuming the records are stored as a CSV file, a Beam YAML pipeline may look something |
| like this:</p> |
| <p>Source code for this example can be found |
| <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/simple_filter.yaml">here</a>.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">pipeline</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">transforms</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">ReadFromCsv</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">ReadInputFile</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/path/to/input.csv</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">Filter</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">FilterWithCategory</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">input</span><span class="p">:</span><span class="w"> </span><span class="l">ReadInputFile</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">language</span><span class="p">:</span><span class="w"> </span><span class="l">python</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">keep</span><span class="p">:</span><span class="w"> </span><span class="l">category == &#34;Electronics&#34;</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">WriteToCsv</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">WriteOutputFile</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">input</span><span class="p">:</span><span class="w"> </span><span class="l">FilterWithCategory</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/path/to/output</span><span class="w"> |
| </span></span></span></code></pre></div><p>This would leave us with the following data:</p> |
| <table> |
| <thead> |
| <tr> |
| <th style="text-align:left">transaction_id</th> |
| <th style="text-align:left">product_name</th> |
| <th style="text-align:left">category</th> |
| <th style="text-align:left">price</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td style="text-align:left">T0012</td> |
| <td style="text-align:left">Headphones</td> |
| <td style="text-align:left">Electronics</td> |
| <td style="text-align:left">59.99</td> |
| </tr> |
| <tr> |
| <td style="text-align:left">T0104</td> |
| <td style="text-align:left">Headphones</td> |
| <td style="text-align:left">Electronics</td> |
| <td style="text-align:left">59.99</td> |
| </tr> |
| <tr> |
| <td style="text-align:left">T0302</td> |
| <td style="text-align:left">Monitor</td> |
| <td style="text-align:left">Electronics</td> |
| <td style="text-align:left">249.99</td> |
| </tr> |
| </tbody> |
| </table> |
| <p>Now, let&rsquo;s say the business wants to determine how much of each Electronics item is being sold to ensure that the |
| correct number is being ordered from the supplier. Let&rsquo;s also assume that they want to determine the total revenue for |
| each item. This simple aggregation can follow the Filter from the previous example as such:</p> |
| <p>Source code for this example can be found |
| <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/simple_filter_and_combine.yaml">here</a>.</p> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="cl"><span class="nt">pipeline</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">transforms</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">ReadFromCsv</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">ReadInputFile</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/path/to/input.csv</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">Filter</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">FilterWithCategory</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">input</span><span class="p">:</span><span class="w"> </span><span class="l">ReadInputFile</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">language</span><span class="p">:</span><span class="w"> </span><span class="l">python</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">keep</span><span class="p">:</span><span class="w"> </span><span class="l">category == &#34;Electronics&#34;</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">Combine</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">CountNumberSold</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">input</span><span class="p">:</span><span class="w"> </span><span class="l">FilterWithCategory</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">group_by</span><span class="p">:</span><span class="w"> </span><span class="l">product_name</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">combine</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">num_sold</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">value</span><span class="p">:</span><span class="w"> </span><span class="l">product_name</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">fn</span><span class="p">:</span><span class="w"> </span><span class="l">count</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">total_revenue</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">value</span><span class="p">:</span><span class="w"> </span><span class="l">price</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">fn</span><span class="p">:</span><span class="w"> </span><span class="l">sum</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l">WriteToCsv</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">WriteOutputFile</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">input</span><span class="p">:</span><span class="w"> </span><span class="l">CountNumberSold</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">config</span><span class="p">:</span><span class="w"> |
| </span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/path/to/output</span><span class="w"> |
| </span></span></span></code></pre></div><p>This would leave us with the following data:</p> |
| <table> |
| <thead> |
| <tr> |
| <th style="text-align:left">product_name</th> |
| <th style="text-align:left">num_sold</th> |
| <th style="text-align:left">total_revenue</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td style="text-align:left">Headphones</td> |
| <td style="text-align:left">2</td> |
| <td style="text-align:left">119.98</td> |
| </tr> |
| <tr> |
| <td style="text-align:left">Monitor</td> |
| <td style="text-align:left">1</td> |
| <td style="text-align:left">249.99</td> |
| </tr> |
| </tbody> |
| </table> |
| <p>While this was a relatively simple use-case, it shows the power of Beam YAML and how easy it is to go from business |
| use-case to a prototype data pipeline in just a few lines of YAML.</p> |
| <h1 id="getting-started-with-beam-yaml">Getting started with Beam YAML</h1> |
| <p>There are several resources that have been compiled to help users get familiar with Beam YAML.</p> |
| <h2 id="day-zero-notebook">Day Zero Notebook</h2> |
| <a target="_blank" href="https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/get-started/try-apache-beam-yaml.ipynb"> |
| <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> |
| </a> |
| <p>To help get started with Apache Beam, there is a Day Zero Notebook available on |
| <a href="https://colab.sandbox.google.com/">Google Colab</a>, an online Python notebook environment with a free attachable |
| runtime, containing some basic YAML pipeline examples.</p> |
| <h2 id="documentation">Documentation</h2> |
| <p>The Apache Beam website provides a set of <a href="https://beam.apache.org/documentation/sdks/yaml/">docs</a> that demonstrate the |
| current capabilities of the Beam YAML SDK. There is also a catalog of currently-supported turnkey transforms found |
| <a href="https://beam.apache.org/releases/yamldoc/current/">here</a>.</p> |
| <h2 id="examples">Examples</h2> |
| <p>A catalog of examples can be found |
| <a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/yaml/examples">here</a>. These examples showcase |
| all the turnkey transforms that can be utilized in Beam YAML. There are also a number of Dataflow Cookbook examples |
| that can be found <a href="https://github.com/GoogleCloudPlatform/dataflow-cookbook/tree/main/Python/yaml">here</a>.</p> |
| <h2 id="contributing">Contributing</h2> |
| <p>Developers who wish to help build out and add functionalities are welcome to start contributing to the effort in the |
| Beam YAML module found <a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/yaml">here</a>.</p> |
| <p>There is also a list of open <a href="https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Ayaml">bugs</a> found |
| on the GitHub repo - now marked with the &lsquo;yaml&rsquo; tag.</p> |
| <p>While Beam YAML has been marked stable as of Beam 2.52, it is still under heavy development, with new features being |
| added with each release. Those who wish to be part of the design decisions and give insights to how the framework is |
| being used are highly encouraged to join the dev mailing list as those discussions will be directed there. A link to |
| the dev list can be found <a href="https://beam.apache.org/community/contact-us/">here</a>.</p></description><link>/blog/beam-yaml-release/</link><pubDate>Thu, 11 Apr 2024 10:00:00 -0400</pubDate><guid>/blog/beam-yaml-release/</guid><category>blog</category></item><item><title>Apache Beam 2.55.0</title><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.55.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2550-2023-03-25">download page</a> for this release.</p> |
| <p>For more information on changes in 2.55.0, check out the <a href="https://github.com/apache/beam/milestone/19">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>The Python SDK will now include automatically generated wrappers for external Java transforms! (<a href="https://github.com/apache/beam/pull/29834">#29834</a>)</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Added support for handling bad records to BigQueryIO (<a href="https://github.com/apache/beam/pull/30081">#30081</a>). |
| <ul> |
| <li>Full Support for Storage Read and Write APIs</li> |
| <li>Partial Support for File Loads (Failures writing to files supported, failures loading files to BQ unsupported)</li> |
| <li>No Support for Extract or Streaming Inserts</li> |
| </ul> |
| </li> |
| <li>Added support for handling bad records to PubSubIO (<a href="https://github.com/apache/beam/pull/30372">#30372</a>). |
| <ul> |
| <li>Support is not available for handling schema mismatches, and enabling error handling for writing to Pub/Sub topics with schemas is not recommended</li> |
| </ul> |
| </li> |
| <li><code>--enableBundling</code> pipeline option for BigQueryIO DIRECT_READ is replaced by <code>--enableStorageReadApiV2</code>. Both were considered experimental and subject to change (Java) (<a href="https://github.com/apache/beam/issues/26354">#26354</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Allow writing clustered and not time-partitioned BigQuery tables (Java) (<a href="https://github.com/apache/beam/pull/30094">#30094</a>).</li> |
| <li>Redis cache support added to RequestResponseIO and Enrichment transform (Python) (<a href="https://github.com/apache/beam/pull/30307">#30307</a>)</li> |
| <li>Merged <code>sdks/java/fn-execution</code> and <code>runners/core-construction-java</code> into the main SDK. These artifacts were never meant for users, but noting |
| that they no longer exist. These are steps to bring portability into the core SDK alongside all other core functionality.</li> |
| <li>Added Vertex AI Feature Store handler for Enrichment transform (Python) (<a href="https://github.com/apache/beam/pull/30388">#30388</a>)</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>Arrow version was bumped to 15.0.0 from 5.0.0 (<a href="https://github.com/apache/beam/pull/30181">#30181</a>).</li> |
| <li>Go SDK users who build custom worker containers may run into issues with the move to distroless containers as a base (see Security Fixes). |
| <ul> |
| <li>The issue stems from distroless containers lacking additional tools, which current custom container processes may rely on.</li> |
| <li>See <a href="https://beam.apache.org/documentation/runtime/environments/#from-scratch-go">https://beam.apache.org/documentation/runtime/environments/#from-scratch-go</a> for instructions on building and using a custom container.</li> |
| </ul> |
| </li> |
| <li>Python SDK has changed the default value for the <code>--max_cache_memory_usage_mb</code> pipeline option from 100 to 0. This option was first introduced in the 2.52.0 SDK version. This change restores the behavior of the 2.51.0 SDK, which does not use the state cache. If your pipeline uses iterable side inputs views, consider increasing the cache size by setting the option manually. (<a href="https://github.com/apache/beam/issues/30360">#30360</a>).</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>N/A</li> |
| </ul> |
| <h2 id="bug-fixes">Bug fixes</h2> |
| <ul> |
| <li>Fixed <code>SpannerIO.readChangeStream</code> to support propagating credentials from pipeline options |
| to the <code>getDialect</code> calls for authenticating with Spanner (Java) (<a href="https://github.com/apache/beam/pull/30361">#30361</a>).</li> |
| <li>Reduced the number of HTTP requests in GCSIO function calls (Python) (<a href="https://github.com/apache/beam/pull/30205">#30205</a>)</li> |
| </ul> |
| <h2 id="security-fixes">Security Fixes</h2> |
| <ul> |
| <li>Go SDK base container image moved to distroless/base-nossl-debian12, reducing vulnerable container surface to kernel and glibc (<a href="https://github.com/apache/beam/pull/30011">#30011</a>).</li> |
| </ul> |
| <h2 id="known-issues">Known Issues</h2> |
| <ul> |
| <li>In Python pipelines, when shutting down inactive bundle processors, shutdown logic can overaggressively hold the lock, blocking acceptance of new work. Symptoms of this issue include slowness or stuckness in long-running jobs. Fixed in 2.56.0 (<a href="https://github.com/apache/beam/pull/30679">#30679</a>).</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.55.0 release. Thank you to all contributors!</p> |
| <p>Ahmed Abualsaud</p> |
| <p>Anand Inguva</p> |
| <p>Andrew Crites</p> |
| <p>Andrey Devyatkin</p> |
| <p>Arun Pandian</p> |
| <p>Arvind Ram</p> |
| <p>Chamikara Jayalath</p> |
| <p>Chris Gray</p> |
| <p>Claire McGinty</p> |
| <p>Damon Douglas</p> |
| <p>Dan Ellis</p> |
| <p>Danny McCormick</p> |
| <p>Daria Bezkorovaina</p> |
| <p>Dima I</p> |
| <p>Edward Cui</p> |
| <p>Ferran Fernández Garrido</p> |
| <p>GStravinsky</p> |
| <p>Jan Lukavský</p> |
| <p>Jason Mitchell</p> |
| <p>JayajP</p> |
| <p>Jeff Kinard</p> |
| <p>Jeffrey Kinard</p> |
| <p>Kenneth Knowles</p> |
| <p>Mattie Fu</p> |
| <p>Michel Davit</p> |
| <p>Oleh Borysevych</p> |
| <p>Ritesh Ghorse</p> |
| <p>Ritesh Tarway</p> |
| <p>Robert Bradshaw</p> |
| <p>Robert Burke</p> |
| <p>Sam Whittle</p> |
| <p>Scott Strong</p> |
| <p>Shunping Huang</p> |
| <p>Steven van Rossum</p> |
| <p>Svetak Sundhar</p> |
| <p>Talat UYARER</p> |
| <p>Ukjae Jeong (Jay)</p> |
| <p>Vitaly Terentyev</p> |
| <p>Vlado Djerek</p> |
| <p>Yi Hu</p> |
| <p>akashorabek</p> |
| <p>case-k</p> |
| <p>clmccart</p> |
| <p>dengwe1</p> |
| <p>dhruvdua</p> |
| <p>hardshah</p> |
| <p>johnjcasey</p> |
| <p>liferoad</p> |
| <p>martin trieu</p> |
| <p>tvalentyn</p></description><link>/blog/beam-2.55.0/</link><pubDate>Mon, 25 Mar 2024 10:00:00 -0400</pubDate><guid>/blog/beam-2.55.0/</guid><category>blog</category><category>release</category></item><item><title>Apache Beam 2.54.0</title><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.54.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/">download page</a> for this release.</p> |
| <p>For more information on changes in 2.54.0, check out the <a href="https://github.com/apache/beam/milestone/18">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li><a href="https://s.apache.org/enrichment-transform">Enrichment Transform</a> along with GCP BigTable handler added to Python SDK (<a href="https://github.com/apache/beam/pull/30001">#30001</a>).</li> |
| <li>Beam Java Batch pipelines run on Google Cloud Dataflow will default to the Portable Runner (v2) starting with this version. (All other languages are already on Runner V2.) See <a href="https://cloud.google.com/dataflow/docs/runner-v2">Runner V2 documentation</a> for how to enable or disable it intentionally.</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Added support for writing to BigQuery dynamic destinations with Python&rsquo;s Storage Write API (<a href="https://github.com/apache/beam/pull/30045">#30045</a>)</li> |
| <li>Adding support for Tuples DataType in ClickHouse (Java) (<a href="https://github.com/apache/beam/pull/29715">#29715</a>).</li> |
| <li>Added support for handling bad records to FileIO, TextIO, AvroIO (<a href="https://github.com/apache/beam/pull/29670">#29670</a>).</li> |
| <li>Added support for handling bad records to BigtableIO (<a href="https://github.com/apache/beam/pull/29885">#29885</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li><a href="https://s.apache.org/enrichment-transform">Enrichment Transform</a> along with GCP BigTable handler added to Python SDK (<a href="https://github.com/apache/beam/pull/30001">#30001</a>).</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>N/A</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>N/A</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed a memory leak affecting some Go SDK since 2.46.0. (<a href="https://github.com/apache/beam/pull/28142">#28142</a>)</li> |
| </ul> |
| <h2 id="security-fixes">Security Fixes</h2> |
| <ul> |
| <li>N/A</li> |
| </ul> |
| <h2 id="known-issues">Known Issues</h2> |
| <ul> |
| <li>Some Python pipelines that run with 2.52.0-2.54.0 SDKs and use large materialized side inputs might be affected by a performance regression. To restore the prior behavior on these SDK versions, supply the <code>--max_cache_memory_usage_mb=0</code> pipeline option. (<a href="https://github.com/apache/beam/issues/30360">#30360</a>).</li> |
| <li>Python pipelines that run with 2.53.0-2.54.0 SDKs and perform file operations on GCS might be affected by excess HTTP requests. This could lead to a performance regression or a permission issue. (<a href="https://github.com/apache/beam/issues/28398">#28398</a>)</li> |
| <li>In Python pipelines, when shutting down inactive bundle processors, shutdown logic can overaggressively hold the lock, blocking acceptance of new work. Symptoms of this issue include slowness or stuckness in long-running jobs. Fixed in 2.56.0 (<a href="https://github.com/apache/beam/pull/30679">#30679</a>).</li> |
| </ul> |
| <p>For the most up to date list of known issues, see <a href="https://github.com/apache/beam/blob/master/CHANGES.md">https://github.com/apache/beam/blob/master/CHANGES.md</a></p> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.54.0 release. Thank you to all contributors!</p> |
| <p>Ahmed Abualsaud</p> |
| <p>Alexey Romanenko</p> |
| <p>Anand Inguva</p> |
| <p>Andrew Crites</p> |
| <p>Arun Pandian</p> |
| <p>Bruno Volpato</p> |
| <p>caneff</p> |
| <p>Chamikara Jayalath</p> |
| <p>Changyu Li</p> |
| <p>Cheskel Twersky</p> |
| <p>Claire McGinty</p> |
| <p>clmccart</p> |
| <p>Damon</p> |
| <p>Danny McCormick</p> |
| <p>dependabot[bot]</p> |
| <p>Edward Cheng</p> |
| <p>Ferran Fernández Garrido</p> |
| <p>Hai Joey Tran</p> |
| <p>hugo-syn</p> |
| <p>Issac</p> |
| <p>Jack McCluskey</p> |
| <p>Jan Lukavský</p> |
| <p>JayajP</p> |
| <p>Jeffrey Kinard</p> |
| <p>Jerry Wang</p> |
| <p>Jing</p> |
| <p>Joey Tran</p> |
| <p>johnjcasey</p> |
| <p>Kenneth Knowles</p> |
| <p>Knut Olav Løite</p> |
| <p>liferoad</p> |
| <p>Marc</p> |
| <p>Mark Zitnik</p> |
| <p>martin trieu</p> |
| <p>Mattie Fu</p> |
| <p>Naireen Hussain</p> |
| <p>Neeraj Bansal</p> |
| <p>Niel Markwick</p> |
| <p>Oleh Borysevych</p> |
| <p>pablo rodriguez defino</p> |
| <p>Rebecca Szper</p> |
| <p>Ritesh Ghorse</p> |
| <p>Robert Bradshaw</p> |
| <p>Robert Burke</p> |
| <p>Sam Whittle</p> |
| <p>Shunping Huang</p> |
| <p>Svetak Sundhar</p> |
| <p>S. Veyrié</p> |
| <p>Talat UYARER</p> |
| <p>tvalentyn</p> |
| <p>Vlado Djerek</p> |
| <p>Yi Hu</p> |
| <p>Zechen Jian</p></description><link>/blog/beam-2.54.0/</link><pubDate>Wed, 14 Feb 2024 09:00:00 -0400</pubDate><guid>/blog/beam-2.54.0/</guid><category>blog</category><category>release</category></item><item><title>Behind the Scenes: Crafting an Autoscaler for Apache Beam in a High-Volume Streaming Environment</title><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h3 id="introduction-to-the-design-of-our-autoscaler-for-apache-beam-jobs">Introduction to the Design of Our Autoscaler for Apache Beam Jobs</h3> |
| <p>Welcome to the third and final part of our blog series on building a scalable, self-managed streaming infrastructure with Beam and Flink. <a href="https://beam.apache.org/blog/apache-beam-flink-and-kubernetes/">In our previous post</a>, we delved into the scale of our streaming platforms, highlighting our capacity to manage over 40,000 streaming jobs and process upwards of 10 million events per second. This impressive scale sets the stage for the challenge we address today: the intricate task of resource allocation in a dynamic streaming environment.</p> |
| <p>In this blog post <a href="https://www.linkedin.com/in/talatuyarer/">Talat Uyarer (Architect / Senior Principal Engineer)</a> and <a href="https://www.linkedin.com/in/rishabhkedia/">Rishabh Kedia (Principal Engineer)</a> describe more details about our Autoscaler. Imagine a scenario where your streaming system is inundated with fluctuating workloads. Our case presents a unique challenge, as our customers, equipped with firewalls distributed globally, generate logs at various times of the day. This results in workloads that not only vary by time but also escalate over time due to changes in settings or the addition of new cybersecurity solutions from PANW. Furthermore, updates to our codebase necessitate rolling out changes across all streaming jobs, leading to a temporary surge in demand as the system processes unprocessed data.</p> |
| <p><img class="center-block" |
| src="/images/blog/apache-beam-flink-and-kubernetes-part3/resource-allocation.png" |
| alt="Resource Allocation"></p> |
| <p>Traditionally, managing this ebb and flow of demand involves a manual, often inefficient approach. One might over-provision resources to handle peak loads, inevitably leading to resource wastage during off-peak hours. Conversely, a more cost-conscious strategy might involve accepting delays during peak times, with the expectation of catching up later. However, both methods demand constant monitoring and manual adjustment - a far from ideal situation.</p> |
| <p>In this modern era, where automated scaling of web front-ends is a given, we aspire to bring the same level of efficiency and automation to streaming infrastructure. Our goal is to develop a system that can dynamically track and adjust to the workload demands of our streaming operations. In this blog post, we will introduce you to our innovative solution - an autoscaler designed specifically for Apache Beam jobs.</p> |
| <p><img class="center-block" |
| src="/images/blog/apache-beam-flink-and-kubernetes-part3/auto-tuned-worker.png" |
| alt="Auto Tuned Resource Allocation"></p> |
| <p>For clarity, when we refer to &ldquo;resources&rdquo; in this context, we mean the number of Flink Task Managers, or Kubernetes Pods, that process your streaming pipeline. These Task Managers aren&rsquo;t just about CPU; they also involve RAM, Network, Disk IO, and other computational resources.</p> |
| <p>However, our solution is predicated on certain assumptions. Primarily, it&rsquo;s geared towards operations processing substantial data volumes. If your workload only requires a couple of Task Managers, this system might not be the best fit. In Our case we have 10K+ workload and each each of them has different workload. Manual tuning was not an option for us. We also assume that the data is evenly distributed, allowing for increased throughput with the addition of more Task Managers. This assumption is crucial for effective horizontal scaling. While there are real-world complexities that might challenge these assumptions, for the scope of this discussion, we will focus on scenarios where these conditions hold true.</p> |
| <p>Join us as we delve into the design and functionality of our autoscaler, a solution tailored to bring efficiency, adaptability, and a touch of intelligence to the world of streaming infrastructure.</p> |
| <h2 id="identifying-the-right-signals-for-autoscaling">Identifying the Right Signals for Autoscaling</h2> |
| <p>When we&rsquo;re overseeing a system like Apache Beam jobs on Flink, it&rsquo;s crucial to identify key signals that help us understand the relationship between our workload and resources. These signals are our guiding lights, showing us when we&rsquo;re lagging behind or wasting resources. By accurately identifying these signals, we can formulate effective scaling policies and implement changes in real-time. Imagine needing to expand from 100 to 200 TaskManagers — how do we smoothly make that transition? That&rsquo;s where these signals come into play.</p> |
| <p>Remember, we&rsquo;re aiming for a universal solution applicable to any workload and pipeline. While specific problems might benefit from unique signals, our focus here is on creating a one-size-fits-all approach.</p> |
| <p>In Flink, tasks form the basic execution unit and consist of one or more operators, such as map, filter, or reduce. Flink optimizes performance by chaining these operators into single tasks when possible, minimizing overheads like thread context switching and network I/O. Your pipeline, when optimized, turns into a directed acyclic graph of stages, each processing elements based on your code. Don&rsquo;t confuse stages with physical machines — they&rsquo;re separate concepts. In our job we measure backlog information by using Apache Beam&rsquo;s <a href="https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/SourceMetrics.java#L32"><code>backlog_bytes</code> and <code>backlog_elements</code></a> metrics.</p> |
| <p><img class="center-block" |
| src="/images/blog/apache-beam-flink-and-kubernetes-part3/flink-operator-chaining.png" |
| alt="Apache Beam Pipeline Optimization by Apache Flink"></p> |
| <h5 id="upscaling-signals"><strong>Upscaling Signals</strong></h5> |
| <h5 id="backlog-growth"><em>Backlog Growth</em></h5> |
| <p>Let’s take a practical example. Consider a pipeline reading from Kafka, where different operators handle data parsing, formatting, and accumulation. The key metric here is throughput — how much data each operstor processes over time. But throughput alone isn&rsquo;t enough. We need to examine the queue size or backlog at each operator. A growing backlog indicates we&rsquo;re falling behind. We measure this as backlog growth — the first derivative of backlog size over time, highlighting our processing deficit.</p> |
| <p><img class="center-block" |
| src="/images/blog/apache-beam-flink-and-kubernetes-part3/backlog_growth.png" |
| alt="Backlog Growth Calculation"></p> |
| <h5 id="backlog-time"><em>Backlog Time</em></h5> |
| <p>This leads us to backlog time, a derived metric that compares backlog size with throughput. It’s a measure of how long it would take to clear the current backlog, assuming no new data arrives. This helps us identify if a backlog of a certain size is acceptable or problematic, based on our specific processing needs and thresholds.</p> |
| <p><img class="center-block" |
| src="/images/blog/apache-beam-flink-and-kubernetes-part3/backlog_time.png" |
| alt="Backlog Time Calculation"></p> |
| <p><img class="center-block" |
| src="/images/blog/apache-beam-flink-and-kubernetes-part3/operator-backlog.png" |
| alt="Close looks at Operator Backlog"></p> |
| <h4 id="downscaling-when-less-is-more"><strong>Downscaling: When Less is More</strong></h4> |
| <h5 id="cpu-utilization"><em>CPU Utilization</em></h5> |
| <p>A key signal for downscaling is CPU utilization. Low CPU utilization suggests we&rsquo;re using more resources than necessary. By monitoring this, we can scale down efficiently without compromising performance.</p> |
| <h4 id="signals-summary"><strong>Signals Summary</strong></h4> |
| <p>In summary, the signals we&rsquo;ve identified for effective autoscaling are:</p> |
| <ol> |
| <li><strong>Throughput:</strong> The baseline of our performance.</li> |
| <li><strong>Backlog Growth:</strong> Indicates if we’re keeping pace with incoming data.</li> |
| <li><strong>Backlog Time:</strong> Helps understand the severity of backlog.</li> |
| <li><strong>CPU Utilization:</strong> Guides us in resource optimization.</li> |
| </ol> |
| <p>These signals might seem straightforward, but their simplicity is key to a scalable, workload-agnostic autoscaling solution.</p> |
| <h2 id="simplifying-autoscaling-policies-for-apache-beam-jobs-on-flink">Simplifying Autoscaling Policies for Apache Beam Jobs on Flink</h2> |
| <p>In the world of Apache Beam jobs running on Flink, deciding when to scale up or down is a bit like being a chef in a busy kitchen. You need to keep an eye on several ingredients — your workload, virtual machines (VMs), and how they interact. It&rsquo;s about maintaining a perfect balance. Our main goals? Avoid falling behind in processing (no backlog growth), ensure that any existing backlog is manageable (short backlog time), and use our resources (like CPU) efficiently.</p> |
| <h4 id="up-scaling-keeping-up-and-catching-up"><strong>Up-scaling: Keeping Up and Catching Up</strong></h4> |
| <p>Imagine your system is like a team of chefs working together. Here&rsquo;s how we decide when to bring more chefs into the kitchen (a.k.a. upscaling):</p> |
| <ol> |
| <li> |
| <p><strong>Keeping Up:</strong> First, we look at our current team size (number of VMs) and how much they&rsquo;re processing (throughput). We then adjust our team size based on the amount of incoming orders (input rate). It&rsquo;s about ensuring that our team is big enough to handle the current demand.</p> |
| </li> |
| <li> |
| <p><strong>Catching Up:</strong> Sometimes, we might have a backlog of orders. In that case, we decide how many extra chefs we need to clear this backlog within a desired time (like 60 seconds). This part of the policy helps us get back on track swiftly.</p> |
| </li> |
| </ol> |
| <h4 id="scaling-example-a-practical-look"><strong>Scaling Example: A Practical Look</strong></h4> |
| <p>Let&rsquo;s paint a picture with an example. Initially, we have a steady flow of orders (input rate) matching our processing capacity (throughput), so there&rsquo;s no backlog. But suddenly, orders increase, and our team starts falling behind, creating a backlog. We respond by increasing our team size to match the new rate of orders. Though the backlog doesn&rsquo;t grow further, it still exists. Finally, we add a few more chefs to the team, which allows us to clear the backlog quickly and return to a new, balanced state.</p> |
| <h4 id="downscaling-when-to-reduce-resources"><strong>Downscaling: When to Reduce Resources</strong></h4> |
| <p>Downscaling is like knowing when some chefs can take a break after a rush hour. We consider this when:</p> |
| <ul> |
| <li>Our backlog is low — we&rsquo;ve caught up with the orders.</li> |
| <li>The backlog isn&rsquo;t growing — we&rsquo;re keeping up with incoming orders.</li> |
| <li>Our kitchen (CPU) isn&rsquo;t working too hard — we&rsquo;re using our resources efficiently.</li> |
| </ul> |
| <p>Downscaling is all about reducing resources without affecting the quality of service. It&rsquo;s about ensuring that we&rsquo;re not overstaffed when the rush hour is over.</p> |
| <h4 id="summary-a-recipe-for-effective-scaling"><strong>Summary: A Recipe for Effective Scaling</strong></h4> |
| <p>In summary, our scaling policy is for scale up, we first ensure that the time to drain the backlog is beyond the threshold (120s) or the cpu is above the threshold (90%)</p> |
| <p>Increasing Backlog aka Backlog Growth &gt; 0 :</p> |
| <p><img class="center-block" |
| src="/images/blog/apache-beam-flink-and-kubernetes-part3/worker_require.png" |
| alt="Required Worker Calculation"></p> |
| <p>Consistent Backlog aka Backlog Growth = 0:</p> |
| <p><img class="center-block" |
| src="/images/blog/apache-beam-flink-and-kubernetes-part3/worker_extra.png" |
| alt="Extra Worker Calculation"></p> |
| <p>To Sum up:</p> |
| <p><img class="center-block" |
| src="/images/blog/apache-beam-flink-and-kubernetes-part3/worker_scaleup.png" |
| alt="Scale up Worker Calculation"></p> |
| <p>To scale down, we need to ensure the machine utilization is low (&lt; 70%) and there is no backlog growth and current time to drain backlog is less than the limit (10s)</p> |
| <p>So the only driving factor to calculate the required resources after a scale down is CPU</p> |
| <p><img class="center-block" |
| src="/images/blog/apache-beam-flink-and-kubernetes-part3/cpurate_desired.png" |
| alt="Desired Cpu Rate Calculation"></p> |
| <h2 id="executing-autoscaling-decision">Executing Autoscaling Decision</h2> |
| <p>In our setup we use Reactive Mode which uses Adaptive Scheduler and Declarative Resources manager. We wanted to align resources with slots. As advised in most of the Flink documentation we set one per vCPU slot. Most of our jobs use 1 vCPU 4GB Memory combination for TaskManager.</p> |
| <p>Reactive Mode, a unique feature of the Adaptive Scheduler, operates under the principle of one job per cluster, a rule enforced in Application Mode. In this mode, a job is configured to utilize all available resources within the cluster. Adding a TaskManager will increase the job&rsquo;s scale, while removing resources will decrease it. In this setup, Flink autonomously manages the job&rsquo;s parallelism, always maximizing it.</p> |
| <p>During a rescaling event, Reactive Mode restarts the job using the most recent checkpoint. This eliminates the need for creating a savepoint, typically required for manual job rescaling. The volume of data reprocessed after rescaling is influenced by the checkpointing interval(10 seconds for us), and the time it takes to restore depends on the size of the state.</p> |
| <p>The scheduler determines the parallelism of each operator within a job. This setting is not user-configurable and any attempts to set it, whether for individual operators or the entire job, will be overlooked.</p> |
| <p><img class="center-block" |
| src="/images/blog/apache-beam-flink-and-kubernetes-part3/adaptive_scheduler_rescale.png" |
| alt="How Reactive Mode Works"></p> |
| <p>Parallelism can only be influenced by setting a maximum for pipelines, which the scheduler will honor. Our maxParallelism is limited by the total count of partitions that the pipeline will process, as well as by the job itself. We cap the maximum number of TaskManagers with maxWorker count and control the job&rsquo;s key count in shuffle by setting maxParallelism. Additionally, we set maxParallelism per pipeline to manage pipeline parallelism. The job cannot exceed the job&rsquo;s maxParallelism in terms of workers.</p> |
| <p>After autoscaler analysis, we will tag if the job needs to be scaled up, no action or scaled down. To interact with the job, we use a library we have built over Flink Kubernetes Operator. This library allows us to interact with our flink jobs via a simple java method call. Library will convert our method call to a kubernetes command.</p> |
| <p>In the kubernetes world, the call will look like this for a scale up:</p> |
| <p><code>kubectl scale flinkdeployment job-name --replicas=100</code></p> |
| <p>Apache Flink will handle the rest of the work needed to scale up.</p> |
| <h2 id="maintaining-state-for-stateful-streaming-application-with-autoscaling">Maintaining State for Stateful Streaming Application with Autoscaling</h2> |
| <p>Adapting Apache Flink&rsquo;s state recovery mechanisms for autoscaling involves leveraging its robust features like max parallelism, checkpointing, and the Adaptive Scheduler to ensure efficient and resilient stream processing, even as the system dynamically adjusts to varying loads. Here&rsquo;s how these components work together in an autoscaling context:</p> |
| <ol> |
| <li><strong>Max Parallelism</strong> sets an upper limit on how much a job can scale out, ensuring that state can be redistributed across a larger or smaller number of nodes without exceeding predefined boundaries. This is crucial for autoscaling because it allows Flink to manage state effectively, even as the number of task slots changes to accommodate varying workloads.</li> |
| <li><strong>Checkpointing</strong> is at the heart of Flink&rsquo;s fault tolerance mechanism, periodically saving the state of each job to a durable storage (in our case it is GCS bucket). In an autoscaling scenario, checkpointing enables Flink to recover to a consistent state after scaling operations. When the system scales out (adds more resources) or scales in (removes resources), Flink can restore the state from these checkpoints, ensuring data integrity and processing continuity without losing critical information. In scale down or up situations there could be a moment to reprocess data from last checkpoint. To reduce that amount we reduce the checkpointing interval to 10 seconds.</li> |
| <li><strong>Reactive Mode</strong> is a special mode for Adaptive Scheduler, that assumes a single job per-cluster (enforced by the Application Mode). Reactive Mode configures a job so that it always uses all resources available in the cluster. Adding a TaskManager will scale up your job, removing resources will scale it down. Flink will manage the parallelism of the job, always setting it to the highest possible values. When a job undergoes resizing, Reactive Mode triggers a restart using the most recent successful checkpoint.</li> |
| </ol> |
| <h2 id="conclusion">Conclusion</h2> |
| <p>In this blog series, we&rsquo;ve taken a deep dive into the creation of an autoscaler for Apache Beam in a high-volume streaming environment, highlighting the journey from conceptualization to implementation. This endeavor not only tackled the complexities of dynamic resource allocation but also set a new standard for efficiency and adaptability in streaming infrastructure. By marrying intelligent scaling policies with the robust capabilities of Apache Beam and Flink, we&rsquo;ve showcased a scalable solution that optimizes resource use and maintains performance under varying loads. This project stands as a testament to the power of teamwork, innovation, and a forward-thinking approach to streaming data processing. As we wrap up this series, we express our gratitude to all contributors and look forward to the continuous evolution of this technology, inviting the community to join us in further discussions and developments.</p> |
| <h1 id="references">References</h1> |
| <p>[1] Streaming Auto-scaling in Google Cloud Dataflow <a href="https://www.infoq.com/presentations/google-cloud-dataflow/">https://www.infoq.com/presentations/google-cloud-dataflow/</a></p> |
| <p>[2] Pipeline lifecycle <a href="https://cloud.google.com/dataflow/docs/pipeline-lifecycle">https://cloud.google.com/dataflow/docs/pipeline-lifecycle</a></p> |
| <p>[3] Flink Elastic Scaling <a href="https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/">https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/</a></p> |
| <h1 id="acknowledgements">Acknowledgements</h1> |
| <p>This is a large effort to build the new infrastructure and to migrate the large customer based applications from cloud provider managed streaming infrastructure to self-managed Flink based infrastructure at scale. Thanks the Palo Alto Networks CDL streaming team who helped to make this happen: Kishore Pola, Andrew Park, Hemant Kumar, Manan Mangal, Helen Jiang, Mandy Wang, Praveen Kumar Pasupuleti, JM Teo, Rishabh Kedia, Talat Uyarer, Naitik Dani, and David He.</p> |
| <hr> |
| <p><strong>Explore More:</strong></p> |
| <ul> |
| <li><a href="https://beam.apache.org/blog/apache-beam-flink-and-kubernetes/">Part 1: Introduction to Building and Managing Apache Beam Flink Services on Kubernetes</a></li> |
| <li><a href="https://beam.apache.org/blog/apache-beam-flink-and-kubernetes-part2/">Part 2: Build a scalable, self-managed streaming infrastructure with Flink: Tackling Autoscaling Challenges - Part 2</a></li> |
| </ul> |
| <p><em>Join the conversation and share your experiences on our <a href="https://beam.apache.org/community/">Community</a> or contribute to our ongoing projects on <a href="https://github.com/apache/beam">GitHub</a>. Your feedback is invaluable. If you have any comments or questions about this series, please feel free to reach out to us via <a href="https://beam.apache.org/community/contact-us/">User Mailist</a></em></p> |
| <p><em>Stay connected with us for more updates and insights into Apache Beam, Flink, and Kubernetes.</em></p></description><link>/blog/apache-beam-flink-and-kubernetes-part3/</link><pubDate>Mon, 05 Feb 2024 09:00:00 -0400</pubDate><guid>/blog/apache-beam-flink-and-kubernetes-part3/</guid><category>blog</category></item><item><title>Apache Beam 2.53.0</title><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.53.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/">download page</a> for this release.</p> |
| <p>For more information on changes in 2.53.0, check out the <a href="https://github.com/apache/beam/milestone/17">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Python streaming users that use 2.47.0 and newer versions of Beam should update to version 2.53.0, which fixes a known issue: (<a href="https://github.com/apache/beam/issues/27330">#27330</a>).</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>TextIO now supports skipping multiple header lines (Java) (<a href="https://github.com/apache/beam/issues/17990">#17990</a>).</li> |
| <li>Python GCSIO is now implemented with GCP GCS Client instead of apitools (<a href="https://github.com/apache/beam/issues/25676">#25676</a>)</li> |
| <li>Adding support for LowCardinality DataType in ClickHouse (Java) (<a href="https://github.com/apache/beam/pull/29533">#29533</a>).</li> |
| <li>Added support for handling bad records to KafkaIO (Java) (<a href="https://github.com/apache/beam/pull/29546">#29546</a>)</li> |
| <li>Add support for generating text embeddings in MLTransform for Vertex AI and Hugging Face Hub models.(<a href="https://github.com/apache/beam/pull/29564">#29564</a>)</li> |
| <li>NATS IO connector added (Go) (<a href="https://github.com/apache/beam/issues/29000">#29000</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>The Python SDK now type checks <code>collections.abc.Collections</code> types properly. Some type hints that were erroneously allowed by the SDK may now fail. (<a href="https://github.com/apache/beam/pull/29272">#29272</a>)</li> |
| <li>Running multi-language pipelines locally no longer requires Docker. |
| Instead, the same (generally auto-started) subprocess used to perform the |
| expansion can also be used as the cross-language worker.</li> |
| <li>Framework for adding Error Handlers to composite transforms added in Java (<a href="https://github.com/apache/beam/pull/29164">#29164</a>).</li> |
| <li>Python 3.11 images now include google-cloud-profiler (<a href="https://github.com/apache/beam/pull/29651">#29561</a>).</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>Euphoria DSL is deprecated and will be removed in a future release (not before 2.56.0) (<a href="https://github.com/apache/beam/issues/29451">#29451</a>)</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>(Python) Fixed sporadic crashes in streaming pipelines that affected some users of 2.47.0 and newer SDKs (<a href="https://github.com/apache/beam/issues/27330">#27330</a>).</li> |
| <li>(Python) Fixed a bug that caused MLTransform to drop identical elements in the output PCollection (<a href="https://github.com/apache/beam/issues/29600">#29600</a>).</li> |
| </ul> |
| <h2 id="security-fixes">Security Fixes</h2> |
| <ul> |
| <li>Upgraded to go 1.21.5 to build, fixing <a href="https://security-tracker.debian.org/tracker/CVE-2023-45285">CVE-2023-45285</a> and <a href="https://security-tracker.debian.org/tracker/CVE-2023-39326">CVE-2023-39326</a></li> |
| </ul> |
| <h2 id="known-issues">Known Issues</h2> |
| <ul> |
| <li>Potential race condition causing NPE in DataflowExecutionStateSampler in Dataflow Java Streaming pipelines (<a href="https://github.com/apache/beam/issues/29987">#29987</a>).</li> |
| <li>Some Python pipelines that run with 2.52.0-2.54.0 SDKs and use large materialized side inputs might be affected by a performance regression. To restore the prior behavior on these SDK versions, supply the <code>--max_cache_memory_usage_mb=0</code> pipeline option. (<a href="https://github.com/apache/beam/issues/30360">#30360</a>).</li> |
| <li>Python pipelines that run with 2.53.0-2.54.0 SDKs and perform file operations on GCS might be affected by excess HTTP requests. This could lead to a performance regression or a permission issue. (<a href="https://github.com/apache/beam/issues/28398">#28398</a>)</li> |
| <li>In Python pipelines, when shutting down inactive bundle processors, shutdown logic can overaggressively hold the lock, blocking acceptance of new work. Symptoms of this issue include slowness or stuckness in long-running jobs. Fixed in 2.56.0 (<a href="https://github.com/apache/beam/pull/30679">#30679</a>).</li> |
| </ul> |
| <p>For the most up to date list of known issues, see <a href="https://github.com/apache/beam/blob/master/CHANGES.md">https://github.com/apache/beam/blob/master/CHANGES.md</a></p> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.53.0 release. Thank you to all contributors!</p> |
| <p>Ahmed Abualsaud</p> |
| <p>Ahmet Altay</p> |
| <p>Alexey Romanenko</p> |
| <p>Anand Inguva</p> |
| <p>Arun Pandian</p> |
| <p>Balázs Németh</p> |
| <p>Bruno Volpato</p> |
| <p>Byron Ellis</p> |
| <p>Calvin Swenson Jr</p> |
| <p>Chamikara Jayalath</p> |
| <p>Clay Johnson</p> |
| <p>Damon</p> |
| <p>Danny McCormick</p> |
| <p>Ferran Fernández Garrido</p> |
| <p>Georgii Zemlianyi</p> |
| <p>Israel Herraiz</p> |
| <p>Jack McCluskey</p> |
| <p>Jacob Tomlinson</p> |
| <p>Jan Lukavský</p> |
| <p>JayajP</p> |
| <p>Jeffrey Kinard</p> |
| <p>Johanna Öjeling</p> |
| <p>Julian Braha</p> |
| <p>Julien Tournay</p> |
| <p>Kenneth Knowles</p> |
| <p>Lawrence Qiu</p> |
| <p>Mark Zitnik</p> |
| <p>Mattie Fu</p> |
| <p>Michel Davit</p> |
| <p>Mike Williamson</p> |
| <p>Naireen</p> |
| <p>Naireen Hussain</p> |
| <p>Niel Markwick</p> |
| <p>Pablo Estrada</p> |
| <p>Radosław Stankiewicz</p> |
| <p>Rebecca Szper</p> |
| <p>Reuven Lax</p> |
| <p>Ritesh Ghorse</p> |
| <p>Robert Bradshaw</p> |
| <p>Robert Burke</p> |
| <p>Sam Rohde</p> |
| <p>Sam Whittle</p> |
| <p>Shunping Huang</p> |
| <p>Svetak Sundhar</p> |
| <p>Talat UYARER</p> |
| <p>Tom Stepp</p> |
| <p>Tony Tang</p> |
| <p>Vlado Djerek</p> |
| <p>Yi Hu</p> |
| <p>Zechen Jiang</p> |
| <p>clmccart</p> |
| <p>damccorm</p> |
| <p>darshan-sj</p> |
| <p>gabry.wu</p> |
| <p>johnjcasey</p> |
| <p>liferoad</p> |
| <p>lrakla</p> |
| <p>martin trieu</p> |
| <p>tvalentyn</p></description><link>/blog/beam-2.53.0/</link><pubDate>Thu, 04 Jan 2024 09:00:00 -0400</pubDate><guid>/blog/beam-2.53.0/</guid><category>blog</category><category>release</category></item><item><title>Scaling a streaming workload on Apache Beam, 1 million events per second and beyond</title><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/0-intro.png" |
| alt="Streaming Processing"></p> |
| <p>Scaling a streaming workload is critical for ensuring that a pipeline can process large amounts of data while also minimizing latency and executing efficiently. Without proper scaling, a pipeline may experience performance issues or even fail entirely, delaying the time to insights for the business.</p> |
| <p>Given the Apache Beam support for the sources and sinks needed by the workload, developing a streaming pipeline can be easy. You can focus on the processing (transformations, enrichments, or aggregations) and on setting the right configurations for each case.</p> |
| <p>However, you need to identify the key performance bottlenecks and make sure that the pipeline has the resources it needs to handle the load efficiently. This can involve right-sizing the number of workers, understanding the settings needed for the source and sinks of the pipeline, optimizing the processing logic, and even determining the transport formats.</p> |
| <p>This article illustrates how to manage the problem of scaling and optimizing a streaming workload developed in Apache Beam and run on Google Cloud using Dataflow. The goal is to reach one million events per second, while also minimizing latency and resource use during execution. The workload uses Pub/Sub as the streaming source and BigQuery as the sink. We describe the reasoning behind the configuration settings and code changes we used to help the workload achieve the desired scale and beyond.</p> |
| <p>The progression described in this article maps to the evolution of a real-life workload, with simplifications. After the initial business requirements for the pipeline were achieved, the focus shifted to optimizing the performance and reducing the resources needed for the pipeline execution.</p> |
| <h2 id="execution-setup">Execution setup</h2> |
| <p>For this article, we created a test suite that creates the necessary components for the pipelines to execute. You can find the code in <a href="https://github.com/prodriguezdefino/apache-beam-streaming-tests">this Github repository</a>. You can find the subsequent configuration changes that are introduced on every run in this <a href="https://github.com/prodriguezdefino/apache-beam-streaming-tests/tree/main/scaling-streaming-workload-blog">folder</a> as scripts that you can run to achieve similar results.</p> |
| <p>All of the execution scripts can also execute a Terraform-based automation to create a Pub/Sub topic and subscription as well as a BigQuery dataset and table to run the workload. Also, it launches two pipelines: one data generation pipeline that pushes events to the Pub/Sub topic, and an ingestion pipeline that demonstrates the potential improvement points.</p> |
| <p>In all cases, the pipelines start with an empty Pub/Sub topic and subscription and an empty BigQuery table. The plan is to generate one million events per second and, after a few minutes, review how the ingestion pipeline scales with time. The data being autogenerated is based on provided schemas or IDL (or Interface Description Language) given the configuration, and the goal is to have messages ranging between 800 bytes and 2 KB, adding up to approximately 1 GB/s volume throughput. Also, the ingestion pipelines are using the same worker type configuration on all runs (<code>n2d-standard-4</code> GCE machines) and are capping the maximum workers number to avoid very large fleets.</p> |
| <p>All of the executions run on Google Cloud using Dataflow, but you can apply all of the configurations and format changes to the suite while executing on other supported Apache Beam runners. Changes and recommendations are not runner specific.</p> |
| <h3 id="local-environment-requirements">Local environment requirements</h3> |
| <p>Before launching the startup scripts, install the following items in your local environment:</p> |
| <ul> |
| <li><code>gcloud</code>, along with the correct permissions</li> |
| <li>Terraform</li> |
| <li>JDK 17 or later</li> |
| <li>Maven 3.6 or later</li> |
| </ul> |
| <p>For more information, see the <a href="https://github.com/prodriguezdefino/apache-beam-streaming-tests#requisites">requirements</a> section in the GitHub repository.</p> |
| <p>Also, review the service quotas and resources available in your Google Cloud project. Specifically: Pub/Sub regional capacity, BigQuery ingestion quota, and Compute Engine instances available in the selected region for the tests.</p> |
| <h3 id="workload-description">Workload description</h3> |
| <p>Focusing on the ingestion pipeline, our <a href="https://github.com/prodriguezdefino/apache-beam-streaming-tests/blob/main/canonical-streaming-pipelines/src/main/java/com/google/cloud/pso/beam/pipelines/StreamingSourceToBigQuery.java#L55">workload</a> is straightforward. It completes the following steps:</p> |
| <ol> |
| <li>reads data in a specific format from Pub/Sub (Apache Thrift in this case)</li> |
| <li>deals with potential compression and batching settings (not enabled by default)</li> |
| <li>executes a UDF (identity function by default)</li> |
| <li>transforms the input format to one of the formats supported by the <code>BigQueryIO</code> transform</li> |
| <li>writes the data to the configured table</li> |
| </ol> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/0-pipeline.png" |
| alt="Example Workload"></p> |
| <p>The pipeline we used for the tests is highly configurable. For more details about how to tweak the ingestion, see the <a href="https://github.com/prodriguezdefino/apache-beam-streaming-tests/blob/main/canonical-streaming-pipelines/src/main/java/com/google/cloud/pso/beam/pipelines/StreamingSourceToBigQuery.java#L39">options</a> in the file. No code changes are needed on any of our steps. The execution scripts take care of the configurations needed.</p> |
| <p>Although these tests are focused on reading data from Pub/Sub, the ingestion pipeline is capable of reading data from a generic streaming source. The repository contains other <a href="https://github.com/prodriguezdefino/apache-beam-streaming-tests/tree/main/example-suite-scripts">examples</a> that show how to launch this same test suite reading data from Pub/Sub Lite and Kafka. In all cases, the pipeline automation sets up the streaming infrastructure.</p> |
| <p>Finally, you can see in the <a href="https://github.com/prodriguezdefino/apache-beam-ptransforms/blob/a0dd229081625c7b593512543614daf995a9f870/common/src/main/java/com/google/cloud/pso/beam/common/formats/options/TransportFormatOptions.java">configuration options</a> that the pipeline supports many transport format options for the input, such as Thrift, Avro, and JSON. This suite focuses on Thrift, because it is a common open source format, and because it generates a format transformation need. The intent is to put some strain in the workload processing. You can run similar tests for Avro and JSON input data. The streaming data generator pipeline can generate random data for the <a href="https://github.com/prodriguezdefino/apache-beam-streaming-tests/tree/main/streaming-data-generator/src/main/java/com/google/cloud/pso/beam/generator/formats">three supported formats</a> by walking directly on the schema (Avro and JSON) or IDL (Thrift) provided for execution.</p> |
| <h2 id="first-run-default-settings">First run: default settings</h2> |
| <p>The default values for the execution writes the data to BigQuery using <code>STREAMING_INSERTS</code> mode for <code>BigQueryIO</code>. This mode correlates with the <a href="https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/insertAll"><code>tableData insertAll</code> API</a> for BigQuery. This API supports data in JSON format. From the Apache Beam perspective, using the <code>BigQueryIO.writeTableRows</code> method lets us resolve the writes into BigQuery.</p> |
| <p>For our ingestion pipeline, the Thrift format needs to be transformed into <code>TableRow</code>. To do that, we need to translate the Thrift IDL into a BigQuery table schema. That can be achieved by translating the Thrift IDL into an Avro schema, and then using Beam utilities to translate the table schema for BigQuery. We can do this at bootstrap. The schema transformation is cached at the <code>DoFn</code> level.</p> |
| <p>After setting up the data generation and ingestion pipelines, and after letting the pipelines run for some minutes, we see that the pipeline is unable to sustain the desired throughput.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/1-default-ps.png" |
| alt="PubSub metrics"></p> |
| <p>The previous image shows that the number of messages that are not being processed by the ingestion pipeline start to show as unacknowledged messages in Pub/Sub metrics.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/1-default-throughput.png" |
| alt="Throughput"></p> |
| <p>Reviewing the per stage performance metrics, we see that the pipeline shows a saw-like shape, which is often associated with the throttling mechanisms the Dataflow runner uses when some of the stages are acting as bottlenecks for the throughput. Also, we see that the <code>Reshuffle</code> step on the <code>BigQueryIO</code> write transform does not scale as expected.</p> |
| <p>This behavior happens because by default the <a href="https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryOptions.java#L57"><code>BigQueryOptions</code></a> uses 50 different keys to shuffle data to workers before the writes happen on BigQuery. To solve this problem, we can add a configuration to our launch script that enables the write operations to scale to a larger number of workers, which improves performance.</p> |
| <h2 id="second-run-improve-the-write-bottleneck">Second run: improve the write bottleneck</h2> |
| <p>After increasing the number of streaming keys to a higher number, 512 keys in our case, we restarted the test suite. The Pub/Sub metrics started to improve. After an initial ramp on the size of the backlog, the curve started to ease out.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/2-skeys-ps.png" |
| alt="PubSub metrics"></p> |
| <p>This is good, but we should take a look at the throughput per stage numbers to understand if we are achieving the goal we set up for this exercise.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/2-skeys-throughput.png" |
| alt="Throughput"></p> |
| <p>Although the performance has clearly improved, and the Pub/Sub backlog no longer increases monotonically, we are still far from the goal of processing one million events per second (1 GB/s) for our ingestion pipeline. In fact, the throughput metrics jump all over, indicating that bottlenecks are preventing the processing from scaling further.</p> |
| <h2 id="third-run-unleash-autoscale">Third run: unleash autoscale</h2> |
| <p>Luckily for us, when writing into BigQuery, we can autoscale the writes. This step simplifies the configuration so that we don&rsquo;t have to guess the right number of shards. We switched the pipeline’s configuration and enabled this setting for the next <a href="https://github.com/prodriguezdefino/apache-beam-streaming-tests/blob/main/scaling-streaming-workload-blog/3-ps2bq-si-tr-streamingautoshard.sh">launch script</a>.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/3-autoshard-parallelism.png" |
| alt="Key Parallelism"></p> |
| <p>Immediately, we see that the autosharding mechanism tweaks the number of keys very aggressively and in a dynamic way. This change is good, because different moments in time might have different scale needs, such as early backlog recoveries and spikes in the execution.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/3-autoshard-throughput-tr.png" |
| alt="Throughput"></p> |
| <p>Inspecting the throughput performance per stage, we see that as the number of keys increases, the performance of the writes also increases. In fact, it reaches very large numbers!</p> |
| <p>After the initial backlog was consumed and the pipeline stabilized, we saw that the desired performance numbers were reached. The pipeline can sustain processing many more than a million events per second from Pub/Sub and several GB/s of BigQuery ingestion. Yay!</p> |
| <p>Still, we want to see if we can do better. We can introduce several improvements to the pipeline to make the execution more efficient. In most cases, the improvements are configuration changes. We just need to know where to focus next.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/3-autoshard-autoscale.png" |
| alt="Resources"></p> |
| <p>The previous image shows that the number of workers needed to sustain this throughput is still quite high. The workload itself is not CPU intensive. Most of the cost is spent on transforming formats and on I/O interactions, such as shuffles and the actual writes. To understand what to improve, we first investigate the transport formats.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/3-autoshard-tr-input.png" |
| alt="Thrift Input Size"> |
| <img class="center-block" |
| src="/images/blog/scaling-streaming-workload/3-autoshard-tr-output.png" |
| alt="TableRow Output Size"></p> |
| <p>Looking at the input size, right before the identity UDF execution, the data format is binary Thrift, which is a decently compact format even when no compression is used. However, while comparing the <code>PCollection</code> approximated size with the <code>TableRow</code> format needed for BigQuery ingestion, a clear size increase is visible. This is something we can improve by changing the BigQuery write API in use.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/3-autoshard-tr-overhead.png" |
| alt="Translation Overhead"></p> |
| <p>When we inspect the <code>StoreInBigQuery</code> transform, we see that the majority of the wall time is spent on the actual writes. Also, the wall time spent converting data to the destination format (<code>TableRows</code>) compared with how much is spent in the actual writes is quite large: 13 times bigger for the writes. To improve this behavior, we can switch the pipeline write mode.</p> |
| <h2 id="fourth-run-in-with-the-new">Fourth run: in with the new</h2> |
| <p>In this run, we use the <code>StorageWrite</code> API. Enabling the <code>StorageWrite</code> API for this pipeline is straightforward. We set the write mode as <code>STORAGE_WRITE_API</code> and define a write triggering frequency. For this test, we write data at most every ten seconds. The write triggering frequency controls how long the per-stream data accumulate. A higher number defines a larger output to be written after the stream assignment but also imposes a larger end-to-end latency for every element read from Pub/Sub. Similar to the <code>STREAMING_WRITES</code> configuration, <code>BigQueryIO</code> can handle autosharding for the writes, which we already demonstrated to be the best setting for performance.</p> |
| <p>After both pipelines become stable, the performance benefits seen when using the <code>StorageWrite</code> API in <code>BigQueryIO</code> are apparent. After enabling the new implementation, the wall time rate between the format transformation and write operation decreases. The wall time spent on writes is only about 34 percent larger than the format transformation.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/4-format-transformation.png" |
| alt="Translation Overhead"></p> |
| <p>After stabilization, the pipeline throughput is also quite smooth. The runner can quickly and steadily downscale the pipeline resources needed to sustain the desired throughput.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/4-throughput.png" |
| alt="Throughput"></p> |
| <p>Looking at the resource scale needed to process the data, another dramatic improvement is visible. Whereas the streaming inserts-based pipeline needed more than 80 workers to sustain the throughput, the storage writes pipeline only needs 49, a 40 percent improvement.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/4-ingestion-scale.png" |
| alt="Resources"></p> |
| <p>We can use the data generation pipeline as reference. This pipeline only needs to randomly generate data and write the events to Pub/Sub. It runs steadily with an average of 40 workers. The improvements on the ingestion pipeline using the right configuration for the workload makes it closer to those resources needed for the generation.</p> |
| <p>Similar to the streaming inserts-based pipeline, writing the data into BigQuery requires running a format translation, from Thrift to <code>TableRow</code> in the former and from Thrift to Protocol Buffers (protobuf) in the latter. Because we are using the <code>BigQueryIO.writeTableRows</code> method, we add another step in the format translation. Because the <code>TableRow</code> format also increases the size of the <code>PCollection</code> being processed, we want to see if we can improve this step.</p> |
| <h2 id="fifth-run-a-better-write-format">Fifth run: a better write format</h2> |
| <p>When using <code>STORAGE_WRITE_API</code>, the <code>BigQueryIO</code> transform exposes a method that we can use to write the Beam row type directly into BigQuery. This step is useful because of the flexibility that the row type provides for interoperability and schema management. Also, it&rsquo;s both efficient for shuffling and denser than <code>TableRow</code>, so our pipeline will have smaller <code>PCollection</code> sizes.</p> |
| <p>For the next run, because our data volume is not small, we decrease the triggering frequency when writing to BigQuery. Because we use a different format, slightly different code runs. For this change, the test pipeline script is configured with the flag <code>--formatToStore=BEAM_ROW</code>.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/5-input-size.png" |
| alt="Thrift input size"> |
| <img class="center-block" |
| src="/images/blog/scaling-streaming-workload/5-output-size.png" |
| alt="Row output size"></p> |
| <p>The <code>PCollection</code> size written into BigQuery is considerably smaller than on previous executions. In fact, for this particular execution, the Beam row format is a smaller size than the Thrift format. A larger <code>PCollection</code> conformed by bigger per-element sizes can put nontrivial memory pressure in smaller worker configurations, reducing the overall throughput.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/5-format-trasformation.png" |
| alt="Translation overhead"></p> |
| <p>The wall clock rate for the format transformation and the actual BigQuery writes also maintain a very similar rate. Handling the Beam row format does not impose a performance penalty in the format translation and subsequent writes. This is confirmed by the number of workers in use by the pipeline when throughput becomes stable, slightly smaller than the previous run but clearly in the same range.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/5-ingestion-scale.png" |
| alt="Resources"></p> |
| <p>Although we are in a much better position than when we started, given our test pipeline input format, there&rsquo;s still room for improvement.</p> |
| <h2 id="sixth-run-further-reduce-the-format-translation-effort">Sixth run: further reduce the format translation effort</h2> |
| <p>Another supported format for the input <code>PCollection</code> in the <code>BigQueryIO</code> transform might be advantageous for our input format. The method <code>writeGenericRecords</code> enables the transform to transform Avro <code>GenericRecords</code> directly into protobuf before the write operation. Apache Thrift can be transformed into Avro <code>GenericRecords</code> very efficiently. We can make another test run configuring our test ingestion pipeline by setting the option <code>--formatToStore=AVRO_GENERIC_RECORD</code> on our execution script.</p> |
| <p>This time, the difference between format translation and writes increases significantly, improving performance. The translation to Avro <code>GenericRecords</code> is only 20 percent of the write effort spent on writing those records into BigQuery. Given that the test pipelines had similar runtimes and that the wall clock seen in the <code>WriteIntoBigQuery</code> stage is also aligned with other <code>StorageWrite</code> related runs, using this format is appropriate for this workload.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/6-format-transformation.png" |
| alt="Translation overhead"></p> |
| <p>We see further gains when we look at resource utilization. We need less CPU time to execute the format translations for our workload while achieving the desired throughput.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/6-ingestion-scale.png" |
| alt="Resources"></p> |
| <p>This pipeline improves upon the previous run, running steadily on 42 workers when throughput is stable. Given the worker configuration used (<code>nd2-standard-4</code>), and the volume throughput of the workload process (about 1 GB/s), we are achieving about 6 MB/s throughput per CPU core, which is quite impressive for a streaming pipeline with exactly-once semantics.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/6-latencies.png" |
| alt="Latencies"></p> |
| <p>When we add up all of the stages executed in the main path of the pipeline, the latency seen at this scale achieves sub-second end-to-end latencies during sustained periods of time.</p> |
| <p>Given the workload requirements and the implemented pipeline code, this performance is the best that we can extract without further tuning the runner’s specific settings.</p> |
| <h2 id="seventh-run--lets-just-relax-at-least-some-constraints">Seventh run : lets just relax (at least some constraints)</h2> |
| <p>When using the <code>STORAGE\_WRITE\_API</code> setting for <code>BigQueryIO</code>, we enforce exactly-once semantics on the writes. This configuration is great for use cases that need strong consistency on the data that gets processed, but it imposes a performance and cost penalty.</p> |
| <p>From a high-level perspective, writes into BigQuery are made in batches, which are released based on the current sharding and the triggering frequency. If a write fails during the execution of a particular bundle, it is retried. A bundle of data is committed into BigQuery only when all the data in that particular bundle is correctly appended to a stream. This implementation needs to shuffle the full volume of data to create the batches that are written, and also the information of the finished batches for later commit (although this last piece is very small compared with the first).</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/7-previous-data-input.png" |
| alt="Read data size"></p> |
| <p>Looking at the previous pipeline execution, the total data being processed for the pipeline by Streaming Engine is larger than the data being read from Pub/Sub. For example, 7 TB of data is read from Pub/Sub, whereas the processing of data for the whole execution of the pipeline moves 25 TB of data to and from Streaming Engine.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/7-previous-shuffle-total.png" |
| alt="Streamed data size"></p> |
| <p>When data consistency is not a hard requirement for ingestion, you can use at-least-once semantics with <code>BigQueryIO</code> write mode. This implementation avoids shuffling and grouping data for the writes. However, this change might cause a small number of repeated rows to be written into the destination table. This can happen with append errors, infrequent worker restarts, and other even less frequent errors.</p> |
| <p>Therefore, we add the configuration to use <code>STORAGE_API_AT_LEAST_ONCE</code> write mode. To instruct the <code>StorageWrite</code> client to reuse connections while writing data, we also add the configuration flag <code>–useStorageApiConnectionPool</code>. This configuration option only works with <code>STORAGE_API_AT_LEAST_ONCE</code> mode, and it reduces the occurrences of warnings similars to <code>Storage Api write delay more than 8 seconds</code>.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/7-resources.png" |
| alt="Resources"></p> |
| <p>When pipeline throughput stabilizes, we see a similar pattern for resource utilization for the workload. The number of workers in use reaches 40, a small improvement compared with the last run. However, the amount of data being moved from Streaming Engine is much closer to the amount of data read from Pub/Sub.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/7-current-input.png" |
| alt="Read data size"> |
| <img class="center-block" |
| src="/images/blog/scaling-streaming-workload/7-current-shuffle-total.png" |
| alt="Streamed data size"></p> |
| <p>Considering all of these factors, this change further optimizes the workload, achieving a throughput of 6.4 MB/s per CPU core. This improvement is small compared to the same workload when using consistent writes into BigQuery, but it uses less streaming data resources. This configuration represents the most optimal setup for our workload, with the highest throughput per resource and the lowest streaming data across workers.</p> |
| <p><img class="center-block" |
| src="/images/blog/scaling-streaming-workload/7-latency.png" |
| alt="Streamed data size"></p> |
| <p>This configuration also has impressively low latency for the end-to-end processing. Given that the main path of our pipeline has been fused in a single execution stage from reads to writes, we see that even at p99, the latency tends to be below 300 milliseconds at a quite large volume throughput (as previously mentioned around 1 GB/s).</p> |
| <h2 id="recap">Recap</h2> |
| <p>Optimizing Apache Beam streaming workloads for low latency and efficient execution requires careful analysis and decision-making, and the right configurations.</p> |
| <p>Considering the scenario discussed in this article, it is essential to consider factors like overall CPU utilization, throughput and latency per stage, <code>PCollection</code> sizes, wall time per stage, write mode, and transport formats, in addition to writing the right pipeline for the workload.</p> |
| <p>Our experiments revealed that using the <code>StorageWrite</code> API, autosharding for writes, and Avro <code>GenericRecords</code> as the transport format yielded the most efficient results. Relaxing the consistency for writes can further improve performance.</p> |
| <p>The accompanying <a href="https://github.com/prodriguezdefino/apache-beam-streaming-tests">Github repository</a> contains a test suite that you can use to replicate the analysis on your Google Cloud project or with a different runner setup. Feel free to take it for a spin. Comments and PRs are always welcome.</p></description><link>/blog/scaling-streaming-workload/</link><pubDate>Wed, 03 Jan 2024 00:00:01 -0800</pubDate><guid>/blog/scaling-streaming-workload/</guid><category>blog</category></item><item><title>Build a scalable, self-managed streaming infrastructure with Beam and Flink: Tackling Autoscaling Challenges - Part 2</title><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="build-a-scalable-self-managed-streaming-infrastructure-with-flink-tackling-autoscaling-challenges---part-2">Build a scalable, self-managed streaming infrastructure with Flink: Tackling Autoscaling Challenges - Part 2</h1> |
| <p>Welcome to Part 2 of our in-depth series about building and managing a service for Apache Beam Flink on Kubernetes. In this segment, we&rsquo;re taking a closer look at the hurdles we encountered while implementing autoscaling. These challenges weren&rsquo;t just roadblocks. They were opportunities for us to innovate and enhance our system. Let’s break down these issues, understand their context, and explore the solutions we developed.</p> |
| <h2 id="understand-apache-beam-backlog-metrics-in-the-flink-runner-environment">Understand Apache Beam backlog metrics in the Flink runner environment</h2> |
| <p><strong>The Challenge:</strong> In our current setup, we are using Apache Flink for processing data streams. However, we&rsquo;ve encountered a puzzling issue: our Flink job isn&rsquo;t showing the backlog metrics from Apache Beam. These metrics are critical for understanding the state and performance of our data pipelines.</p> |
| <p><strong>What We Found:</strong> Interestingly, we noticed that the metrics are actually being generated in <code>KafkaIO</code>, which is a part of our data pipeline that handles Kafka streams. But when we try to monitor these metrics through the Apache Flink Metric system, we can&rsquo;t find them. We suspected that there might be an issue with the integration (or &lsquo;wiring&rsquo;) between Apache Beam and Apache Flink.</p> |
| <p><strong>Digging Deeper:</strong> On closer inspection, we found that the metrics should be emitted during the &lsquo;Checkpointing&rsquo; phase of the data stream processing. During this crucial step, the system takes a snapshot of the stream&rsquo;s state, and the metrics are typically metrics that are generated for unbounded sources. Unbounded sources are sources that continuously stream data, like Kafka.</p> |
| <p><strong>A Potential Solution:</strong> We believe the root of the problem lies in how the metric context is set during the checkpointing phase. A disconnect appears to prevent the Beam metrics from being properly captured in the Flink Metric system. We proposed a fix for this issue, which you can review and contribute to on our GitHub pull request: <a href="https://github.com/apache/beam/pull/29793">Apache Beam PR #29793</a>.</p> |
| <p><img class="center-block" |
| src="/images/blog/apache-beam-flink-and-kubernetes-part2/flink-backlog-metrics.png" |
| alt="Apache Flink Beam Backlog Metrics"></p> |
| <h2 id="overcoming-challenges-in-checkpoint-size-reduction-for-autoscaling-beam-jobs">Overcoming challenges in checkpoint size reduction for autoscaling Beam jobs</h2> |
| <p>In this section we will discuss strategies for reducing the size of checkpoints in autoscaling Apache Beam jobs, focusing on efficient checkpointing in Apache Flink and optimizing bundle sizes and PipelineOptions to manage frequent checkpoint timeouts and large-scale job requirements.</p> |
| <h3 id="understand-the-basics-of-checkpointing-in-apache-flink">Understand the basics of checkpointing in Apache Flink</h3> |
| <p>In stream processing, maintaining state consistency and fault tolerance is crucial. Apache Flink achieves this through a process called <em>checkpointing</em>. Checkpointing periodically captures the state of a job&rsquo;s operators and stores it in a stable storage location, like Google Cloud Storage or AWS S3. Specifically, Flink checkpoints a job every ten seconds and allows up to one minute for this process to complete. This process is vital for ensuring that, in case of failures, the job can resume from the last checkpoint, providing exactly-once semantics and fault tolerance.</p> |
| <h3 id="the-role-of-bundles-in-apache-beam">The role of bundles in Apache Beam</h3> |
| <p>Apache Beam introduces the concept of a <em>bundle</em>. A bundle is essentially a group of elements that are processed together. This step enhances processing efficiency and throughput by reducing the overhead of handling each element separately. For more information, see <a href="https://beam.apache.org/documentation/runtime/model/#bundling-and-persistence">Bundling and persistence</a>. In the Flink runner <a href="https://beam.apache.org/releases/javadoc/2.52.0/org/apache/beam/runners/flink/FlinkPipelineOptions.html#getMaxBundleSize--">default configuration</a>, a bundle&rsquo;s default size is 1000 elements with a one-second timeout. However, based on our performance tests, we adjusted the bundle size to <em>10,000 elements with a 10-second timeout</em>.</p> |
| <h3 id="challenge-frequent-checkpoint-timeouts">Challenge: frequent checkpoint timeouts</h3> |
| <p>When we configured checkpointing every 10 seconds, we faced frequent checkpoint timeouts, often exceeding 1 minute. This was due to the large size of the checkpoints.</p> |
| <h3 id="solution-manage-checkpoint-size">Solution: Manage checkpoint size</h3> |
| <p>In Apache Beam Flink jobs, the <code>finishBundleBeforeCheckpointing</code> option plays a pivotal role. When enabled, it ensures that all bundles are completely processed before initiating a checkpoint. This results in checkpoints that only contain the state post-bundle completion, significantly reducing checkpoint size. Initially, our checkpoints were around 2 MB per pipeline. With this change, they consistently dropped to 150 KB.</p> |
| <h3 id="address-the-checkpoint-size-in-large-scale-jobs">Address the checkpoint size in large-scale jobs</h3> |
| <p>Despite reducing checkpoint sizes, a 150 KB checkpoint every ten seconds can still be substantial, especially in jobs that run multiple pipelines. For instance, with 100 pipelines in a single job, this size balloons to 15 MB per 10-second interval.</p> |
| <h3 id="further-optimization-reduce-checkpoint-size-with-pipelineoptions">Further optimization: reduce checkpoint size with PipelineOptions</h3> |
| <p>We discovered that due to a specific issue (BEAM-8577), our Flink runner was including our large <code>PipelineOptions</code> objects in every checkpoint. We solved this problem by removing unnecessary application-related options from <code>PipelineOptions</code>, further reducing the checkpoint size to a more manageable 10 KB per pipeline.</p> |
| <h2 id="kafka-reader-wait-time-solving-autoscaling-challenges-in-beam-jobs">Kafka Reader wait time: solving autoscaling challenges in Beam jobs</h2> |
| <h3 id="understand-unaligned-checkpointing">Understand unaligned checkpointing</h3> |
| <p>In our system, we use unaligned checkpointing to speed up the process of checkpointing, which is essential for ensuring data consistency in distributed systems. However, when we activated the <code>finishBundleBeforeCheckpointing</code> feature, we began facing checkpoint timeout issues and delays in checkpointing steps. Apache Beam leverages Apache Flink&rsquo;s legacy source implementation for processing unbounded sources. In Flink, tasks are categorized into two types: source tasks and non-source tasks.</p> |
| <ul> |
| <li><strong>Source tasks</strong>: fetch data from external systems into a Flink job</li> |
| <li><strong>Non-source tasks</strong>: process the incoming data</li> |
| </ul> |
| <p><img class="center-block" |
| src="/images/blog/apache-beam-flink-and-kubernetes-part2/kafkaio-wait-reader.png" |
| alt="Apache Flink Task Types"></p> |
| <p>In the standard configuration, non-source tasks check for an available buffer before pulling data. If source tasks don&rsquo;t perform this check, they might experience checkpointing delays in writing data to the output buffer. This delay affects the efficiency of unaligned checkpoints, which are only recognized by legacy source tasks when an output buffer is available.</p> |
| <h3 id="address-the-challenge-with-unboundedsourcewrapper-in-beam">Address the challenge with UnboundedSourceWrapper in Beam</h3> |
| <p>To solve this problem, Apache Flink introduced a new source implementation that operates in a pull mode. In this mode, a task checks for a free buffer before fetching data, aligning with the approach of non-source tasks.</p> |
| <p>However, the legacy source, still used by Apache Beam&rsquo;s Flink Runner, operates in a push mode. It sends data to downstream tasks immediately. This setup might create bottlenecks when buffers are full, causing delays in detecting unaligned checkpoint barriers.</p> |
| <h3 id="our-solution">Our solution</h3> |
| <p>Despite its deprecation, Apache Beam&rsquo;s Flink Runner still uses the legacy source implementation. To address its issues, we implemented our modifications and the quick workarounds suggested in <a href="https://issues.apache.org/jira/browse/FLINK-26759">FLINK-26759</a>. These enhancements are detailed in our <a href="#">Pull Request</a>. You can also find more information about unaligned checkpoint issues in the <a href="https://blog.51cto.com/u_14286418/7000028">Flink Unaligned Checkpoint</a> blog post.</p> |
| <p><img class="center-block" |
| src="/images/blog/apache-beam-flink-and-kubernetes-part2/checkpoint_monitoring-history-subtasks.png" |
| alt="Apache Flink UI Checkpoint History"></p> |
| <h2 id="address-slow-reads-in-high-traffic-scenarios">Address slow reads in high-traffic scenarios</h2> |
| <p>In our journey with Apache Beam and the Flink Runner, we encountered a significant challenge similar to one documented in the post <a href="https://antonio-si.medium.com/how-intuit-debug-consumer-lags-in-apache-beam-22ca3b39602e">How Intuit Debug Consumer Lags in Apache Beam</a> by <a href="https://antonio-si.medium.com/">Antonio Si</a> in his experience at Intuit. Their real-time data processing pipelines had increasing Kafka consumer lag, particularly with topics experiencing high message traffic. This issue was traced to Apache Beam&rsquo;s handling of Kafka partitions through <code>UnboundedSourceWrapper</code> and <code>KafkaUnboundedReader</code>. Specifically, for topics with lower traffic, the processing thread paused unnecessarily, delaying the processing of high-traffic topics. We faced a parallel situation in our system, where the imbalance in processing speeds between high- and low-traffic topics led to inefficiencies.</p> |
| <p><img class="center-block" |
| src="/images/blog/apache-beam-flink-and-kubernetes-part2/adaptive-timeout-kafka.png" |
| alt="UnboundedSourceWrapper Design"></p> |
| <p>To resolve this issue, we developed an innovative solution: an adaptive timeout strategy in <code>KafkaIO</code>. This strategy dynamically adjusts the timeout duration based on the traffic of each topic. For low-traffic topics, it shortens the timeout, preventing unnecessary delays. For high-traffic topics, it extends the timeout, providing more processing opportunities. This approach is detailed in our recent pull request.</p> |
| <h2 id="unbalanced-partition-distribution-in-beam-job-autoscaling">Unbalanced partition distribution in Beam job autoscaling</h2> |
| <p>At the heart of this system is the adaptive scheduler, a component designed for rapid resource allocation. It intelligently adjusts the number of parallel tasks (parallelism) a job performs based on the availability of computing slots. These slots are like individual workstations, each capable of handling certain parts of the job.</p> |
| <p>However, we encountered a problem. Our jobs consist of multiple independent pipelines, each needing its own set of resources. Initially, the system tended to overburden the first few workers by assigning them more tasks, while others remained underutilized. This issue was due to the way Flink allocated tasks, favoring the first workers for each pipeline.</p> |
| <p><img class="center-block" |
| src="/images/blog/apache-beam-flink-and-kubernetes-part2/flink-partition-assignment.png" |
| alt="Flink split assignment on slots"></p> |
| <p>To address this issue, we developed a custom patch for Flink&rsquo;s <em>SlotSharingSlotAllocator</em>, a component responsible for task distribution. This patch ensures a more balanced workload distribution across all available workers, improving efficiency and preventing bottlenecks. |
| With this improvement, each worker gets a fair share of tasks, leading to better resource utilization and smoother operation of our Beam Jobs.</p> |
| <h2 id="drain-support-in-kubernetes-operator-with-flink">Drain support in Kubernetes Operator with Flink</h2> |
| <h3 id="the-challenge">The challenge</h3> |
| <p>In the world of data processing with Apache Flink, a common task is to manage and update data-processing jobs. These jobs could be either stateful, where they remember past data, or stateless, where they don&rsquo;t.</p> |
| <p>In the past, when we needed to update or delete a Flink job managed by the Kubernetes Operator, the system saved the current state of the job using a savepoint or checkpoint. However, a crucial step was missing: the system didn&rsquo;t stop the job from processing new data (this is what we mean by draining the job). This oversight could lead to two major issues:</p> |
| <ol> |
| <li><strong>For stateful jobs:</strong> potential data inconsistencies, because the job might process new data that wasn&rsquo;t accounted for in the savepoint</li> |
| <li><strong>For stateless jobs:</strong> data duplication, because the job might reprocess data it already processed</li> |
| </ol> |
| <h3 id="the-solution-drain-function">The solution: drain function</h3> |
| <p>This is where the update referenced as <a href="https://issues.apache.org/jira/browse/FLINK-32700">FLINK-32700</a> is needed. This update introduced a drain function. Think of it as telling the job, &ldquo;Finish what you&rsquo;re currently processing, but don&rsquo;t take on anything new.&rdquo; Here&rsquo;s how it works:</p> |
| <ol> |
| <li><strong>Stop new data:</strong> The job stops reading new input.</li> |
| <li><strong>Mark the source:</strong> The job marks the source with an infinite watermark. Think of this watermark as a marker that tells the system that there&rsquo;s no more new data to process.</li> |
| <li><strong>Propagate through the pipeline:</strong> This marker is then passed through the job&rsquo;s processing pipeline, ensuring that every part of the job knows not to expect any new data.</li> |
| </ol> |
| <p>This seemingly small change has a big impact. It ensures that when a job is updated or deleted, the data it processes remains consistent and accurate. This is crucial for any data-processing task, because it maintains the integrity and reliability of the data. Furthermore, in cases where the drainage fails, you can cancel the job without needing a savepoint, which adds a layer of flexibility and safety to the whole process.</p> |
| <h2 id="conclusion">Conclusion</h2> |
| <p>As we conclude Part 2 of our series on building and managing Apache Beam Flink services on Kubernetes, it&rsquo;s evident that the journey of implementing autoscaling has been both challenging and enlightening. The obstacles we faced, from understanding Apache Beam backlog metrics in the Flink Runner environment to addressing slow reads in high-traffic scenarios, pushed us to develop innovative solutions and deepen our understanding of streaming infrastructure.</p> |
| <p>Our exploration into the intricacies of checkpointing, Kafka Reader wait times, and unbalanced partition distribution revealed the complexities of autoscaling Beam jobs. These challenges prompted us to devise strategies like the adaptive timeout in <code>KafkaIO</code> and the balanced workload distribution in Flink&rsquo;s <code>SlotSharingSlotAllocator</code>. Additionally, the introduction of the drain support in Kubernetes Operator with Flink marks a significant advancement in managing stateful and stateless jobs effectively.</p> |
| <p>This journey has not only enhanced the robustness and efficiency of our system but has also contributed valuable insights to the broader community working with Apache Beam and Flink. We hope that our experiences and solutions will aid others facing similar challenges in their projects.</p> |
| <p>Stay tuned for our next blog post, where we&rsquo;ll delve into the specifics of autoscaling in Apache Beam. We&rsquo;ll break down the concepts, strategies, and best practices to effectively scale your Beam jobs. Thank you for following our series, and we look forward to sharing more of our journey and learnings with you.</p> |
| <h2 id="acknowledgements">Acknowledgements</h2> |
| <p>This is a large effort to build the new infrastructure and to migrate the large customer based applications from cloud provider managed streaming infrastructure to self-managed, Flink-based infrastructure at scale. Thanks the Palo Alto Networks CDL streaming team who helped to make this happen: Kishore Pola, Andrew Park, Hemant Kumar, Manan Mangal, Helen Jiang, Mandy Wang, Praveen Kumar Pasupuleti, JM Teo, Rishabh Kedia, Talat Uyarer, Naitk Dani, and David He.</p> |
| <hr> |
| <p><strong>Explore More:</strong></p> |
| <ul> |
| <li><a href="https://beam.apache.org/blog/apache-beam-flink-and-kubernetes/">Part 1: Introduction to Building and Managing Apache Beam Flink Services on Kubernetes</a></li> |
| </ul> |
| <p><em>Join the conversation and share your experiences on our <a href="https://beam.apache.org/community/">Community</a> or contribute to our ongoing projects on <a href="https://github.com/apache/beam">GitHub</a>. Your feedback is invaluable. If you have any comments or questions about this series, please feel free to reach out to us via <a href="https://beam.apache.org/community/contact-us/">User Mailist</a></em></p> |
| <p><em>Stay connected with us for more updates and insights into Apache Beam, Flink, and Kubernetes.</em></p></description><link>/blog/apache-beam-flink-and-kubernetes-part2/</link><pubDate>Mon, 18 Dec 2023 09:00:00 -0400</pubDate><guid>/blog/apache-beam-flink-and-kubernetes-part2/</guid><category>blog</category></item><item><title>Apache Beam 2.52.0</title><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.52.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2520-2023-11-17">download page</a> for this release.</p> |
| <p>For more information on changes in 2.52.0, check out the <a href="https://github.com/apache/beam/milestone/16">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Previously deprecated Avro-dependent code (Beam Release 2.46.0) has been finally removed from Java SDK &ldquo;core&rdquo; package. |
| Please, use <code>beam-sdks-java-extensions-avro</code> instead. This will allow to easily update Avro version in user code without |
| potential breaking changes in Beam &ldquo;core&rdquo; since the Beam Avro extension already supports the latest Avro versions and |
| should handle this. (<a href="https://github.com/apache/beam/issues/25252">#25252</a>).</li> |
| <li>Publishing Java 21 SDK container images now supported as part of Apache Beam release process. (<a href="https://github.com/apache/beam/issues/28120">#28120</a>) |
| <ul> |
| <li>Direct Runner and Dataflow Runner support running pipelines on Java21 (experimental until tests fully setup). For other runners (Flink, Spark, Samza, etc) support status depend on runner projects.</li> |
| </ul> |
| </li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Add <code>UseDataStreamForBatch</code> pipeline option to the Flink runner. When it is set to true, Flink runner will run batch |
| jobs using the DataStream API. By default the option is set to false, so the batch jobs are still executed |
| using the DataSet API.</li> |
| <li><code>upload_graph</code> as one of the Experiments options for DataflowRunner is no longer required when the graph is larger than 10MB for Java SDK (<a href="https://github.com/apache/beam/pull/28621">PR#28621</a>).</li> |
| <li>state amd side input cache has been enabled to a default of 100 MB. Use <code>--max_cache_memory_usage_mb=X</code> to provide cache size for the user state API and side inputs. (Python) (<a href="https://github.com/apache/beam/issues/28770">#28770</a>).</li> |
| <li>Beam YAML stable release. Beam pipelines can now be written using YAML and leverage the Beam YAML framework which includes a preliminary set of IO&rsquo;s and turnkey transforms. More information can be found in the YAML root folder and in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/README.md">README</a>.</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li><code>org.apache.beam.sdk.io.CountingSource.CounterMark</code> uses custom <code>CounterMarkCoder</code> as a default coder since all Avro-dependent |
| classes finally moved to <code>extensions/avro</code>. In case if it&rsquo;s still required to use <code>AvroCoder</code> for <code>CounterMark</code>, then, |
| as a workaround, a copy of &ldquo;old&rdquo; <code>CountingSource</code> class should be placed into a project code and used directly |
| (<a href="https://github.com/apache/beam/issues/25252">#25252</a>).</li> |
| <li>Renamed <code>host</code> to <code>firestoreHost</code> in <code>FirestoreOptions</code> to avoid potential conflict of command line arguments (Java) (<a href="https://github.com/apache/beam/pull/29201">#29201</a>).</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed &ldquo;Desired bundle size 0 bytes must be greater than 0&rdquo; in Java SDK&rsquo;s BigtableIO.BigtableSource when you have more cores than bytes to read (Java) <a href="https://github.com/apache/beam/issues/28793">#28793</a>.</li> |
| <li><code>watch_file_pattern</code> arg of the <a href="https://github.com/apache/beam/blob/104c10b3ee536a9a3ea52b4dbf62d86b669da5d9/sdks/python/apache_beam/ml/inference/base.py#L997">RunInference</a> arg had no effect prior to 2.52.0. To use the behavior of arg <code>watch_file_pattern</code> prior to 2.52.0, follow the documentation at <a href="https://beam.apache.org/documentation/ml/side-input-updates/">https://beam.apache.org/documentation/ml/side-input-updates/</a> and use <code>WatchFilePattern</code> PTransform as a SideInput. (<a href="https://github.com/apache/beam/pulls/28948">#28948</a>)</li> |
| <li><code>MLTransform</code> doesn&rsquo;t output artifacts such as min, max and quantiles. Instead, <code>MLTransform</code> will add a feature to output these artifacts as human readable format - <a href="https://github.com/apache/beam/issues/29017">#29017</a>. For now, to use the artifacts such as min and max that were produced by the eariler <code>MLTransform</code>, use <code>read_artifact_location</code> of <code>MLTransform</code>, which reads artifacts that were produced earlier in a different <code>MLTransform</code> (<a href="https://github.com/apache/beam/pull/29016/">#29016</a>)</li> |
| <li>Fixed a memory leak, which affected some long-running Python pipelines: <a href="https://github.com/apache/beam/issues/28246">#28246</a>.</li> |
| </ul> |
| <h2 id="security-fixes">Security Fixes</h2> |
| <ul> |
| <li>Fixed <a href="https://www.cve.org/CVERecord?id=CVE-2023-39325">CVE-2023-39325</a> (Java/Python/Go) (<a href="https://github.com/apache/beam/issues/29118">#29118</a>).</li> |
| <li>Mitigated <a href="https://nvd.nist.gov/vuln/detail/CVE-2023-47248">CVE-2023-47248</a> (Python) <a href="https://github.com/apache/beam/issues/29392">#29392</a>.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.52.0 release. Thank you to all contributors!</p> |
| <p>Ahmed Abualsaud</p> |
| <p>Ahmet Altay</p> |
| <p>Aleksandr Dudko</p> |
| <p>Alexey Romanenko</p> |
| <p>Anand Inguva</p> |
| <p>Andrei Gurau</p> |
| <p>Andrey Devyatkin</p> |
| <p>BjornPrime</p> |
| <p>Bruno Volpato</p> |
| <p>Bulat</p> |
| <p>Chamikara Jayalath</p> |
| <p>Damon</p> |
| <p>Danny McCormick</p> |
| <p>Devansh Modi</p> |
| <p>Dominik Dębowczyk</p> |
| <p>Ferran Fernández Garrido</p> |
| <p>Hai Joey Tran</p> |
| <p>Israel Herraiz</p> |
| <p>Jack McCluskey</p> |
| <p>Jan Lukavský</p> |
| <p>JayajP</p> |
| <p>Jeff Kinard</p> |
| <p>Jeffrey Kinard</p> |
| <p>Jiangjie Qin</p> |
| <p>Jing</p> |
| <p>Joar Wandborg</p> |
| <p>Johanna Öjeling</p> |
| <p>Julien Tournay</p> |
| <p>Kanishk Karanawat</p> |
| <p>Kenneth Knowles</p> |
| <p>Kerry Donny-Clark</p> |
| <p>Luís Bianchin</p> |
| <p>Minbo Bae</p> |
| <p>Pranav Bhandari</p> |
| <p>Rebecca Szper</p> |
| <p>Reuven Lax</p> |
| <p>Ritesh Ghorse</p> |
| <p>Robert Bradshaw</p> |
| <p>Robert Burke</p> |
| <p>RyuSA</p> |
| <p>Shunping Huang</p> |
| <p>Steven van Rossum</p> |
| <p>Svetak Sundhar</p> |
| <p>Tony Tang</p> |
| <p>Vitaly Terentyev</p> |
| <p>Vivek Sumanth</p> |
| <p>Vlado Djerek</p> |
| <p>Yi Hu</p> |
| <p>aku019</p> |
| <p>brucearctor</p> |
| <p>caneff</p> |
| <p>damccorm</p> |
| <p>ddebowczyk92</p> |
| <p>dependabot[bot]</p> |
| <p>dpcollins-google</p> |
| <p>edman124</p> |
| <p>gabry.wu</p> |
| <p>illoise</p> |
| <p>johnjcasey</p> |
| <p>jonathan-lemos</p> |
| <p>kennknowles</p> |
| <p>liferoad</p> |
| <p>magicgoody</p> |
| <p>martin trieu</p> |
| <p>nancyxu123</p> |
| <p>pablo rodriguez defino</p> |
| <p>tvalentyn</p></description><link>/blog/beam-2.52.0/</link><pubDate>Fri, 17 Nov 2023 09:00:00 -0400</pubDate><guid>/blog/beam-2.52.0/</guid><category>blog</category><category>release</category></item><item><title>Contributor Spotlight: Johanna Öjeling</title><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>Johanna Öjeling is a Senior Software Engineer at <a href="https://normative.io/">Normative</a>. She started using Apache Beam in 2020 at her previous company <a href="http://datatonic.com">Datatonic</a> and began contributing in 2022 at a personal capacity. We interviewed Johanna to learn more about her interests and we hope that this will inspire new, future, diverse set of contributors to participate in OSS projects.</p> |
| <p><strong>What areas of interest are you passionate about in your career?</strong></p> |
| <p>My core interest lies in distributed and data-intensive systems, and I enjoy working on challenges related to performance, scalability and maintainability. I also feel strongly about developer experience, and like to build tools and frameworks that make developers happier and more productive. Aside from that, I take pleasure in mentoring and coaching other software engineers to grow their skills and pursue a fulfilling career.</p> |
| <p><strong>What motivated you to make your first contribution?</strong></p> |
| <p>I was already a user of the Apache Beam Java and Python SDKs and Google Cloud Dataflow in my previous job, and had started to play around with the Go SDK to learn Go. When I noticed that a feature I wanted was missing, it seemed like a great opportunity to implement it. I had been curious about developing open source software for some time, but did not have a good idea until then of what to contribute with.</p> |
| <p><strong>In which way have you contributed to Apache Beam?</strong></p> |
| <p>I have primarily worked on the Go SDK with implementation of new features, bug fixes, tests, documentation and code reviews. Some examples include a MongoDB I/O connector with dynamically scalable reads and writes, a file I/O connector supporting continuous file discovery, and an Amazon S3 file system implementation.</p> |
| <p><strong>How has your open source engagement impacted your personal or professional growth?</strong></p> |
| <p>Contributing to open source is one of the best decisions I have taken professionally. The Beam community has been incredibly welcoming and appreciative, and it has been rewarding to collaborate with talented people around the world to create software that is free for anyone to benefit from. Open source has opened up new opportunities to challenge myself, dive deeper into technologies I like, and learn from highly skilled professionals. To me, it has served as an outlet for creativity, problem solving and purposeful work.</p> |
| <p><strong>How have you noticed contributing to open source is different from contributing to closed source/proprietary software?</strong></p> |
| <p>My observation has been that there are higher requirements for software quality in open source, and it is more important to get things right the first time. My closed source software experience is from startups/scale-ups where speed is prioritized. When not working on public facing APIs or libraries, one can also more easily change things, whereas we need to be mindful about breaking changes in Beam. I care for software quality and value the high standards the Beam committers hold.</p> |
| <p><strong>What do you like to do with your spare time when you&rsquo;re not contributing to Beam?</strong></p> |
| <p>Coding is a passion of mine so I tend to spend a lot of my free time on hobby projects, reading books and articles, listening to talks and attending events. When I was younger I loved learning foreign languages and studied English, French, German and Spanish. Later I discovered an interest in computer science and switched focus to programming languages. I decided to change careers to software engineering and have tried to learn as much as possible ever since. I love that it never ends.</p> |
| <p><strong>What future features/improvements are you most excited about, or would you like to see on Beam?</strong></p> |
| <p>The multi-language pipeline support is an impressive feature of Beam, and I like that new SDKs such as TypeScript and Swift are emerging, which enables developers to write pipelines in their preferred language. Naturally, I am also excited to see where the Go SDK is headed and how we can make use of newer features of the Go language.</p> |
| <p><strong>What types of contributions or support do you think the Beam community needs more of?</strong></p> |
| <p>Many data and machine learning engineers feel more comfortable with Python than Java and wish the Python SDK were as feature rich as the Java SDK. This presents great opportunities for Python developers to start contributing to Beam. As an SDK author, one can take advantage of Beam&rsquo;s multiple SDKs. When I have developed in Go I have often studied the Java and Python implementations to get ideas for how to solve specific problems and make sure the Go SDK follows a similar pattern.</p> |
| <p><strong>What advice would you give to someone who wants to contribute but does not know where to begin?</strong></p> |
| <p>Start with asking yourself what prior knowledge you have and what you would like to learn, then look for opportunities that match that. The contribution guidelines will tell you where to find open issues and what the process looks like. There are tasks labeled as &ldquo;good first issue&rdquo; which can be a good starting point. I was quite nervous about making my first contribution and had my mentor pre-review my PR. There was no need to worry though, as people will be grateful for your effort to improve the project. The pride I felt when a committer approved my PR and welcomed me to Beam is something I still remember.</p> |
| <p><strong>What advice would you give to the Beam community? What could we improve?</strong></p> |
| <p>We can make it easier for new community members to get involved by providing more examples of tasks that we need help with, both in the form of code and non-code contributions. I will take it as an action point myself to label more issues accordingly and tailor the descriptions for newcomers. However, this is contingent on community members visiting the GitHub project. To address this, we could also proactively promote opportunities through social channels and the user mailing list.</p> |
| <p><em>We thank Johanna for the interview and for her contributions! If you would like to learn more about contributing to Beam you can learn more about it here: <a href="https://beam.apache.org/contribute/">https://beam.apache.org/contribute/</a>.</em></p></description><link>/blog/contributor-spotlight-johanna-ojeling/</link><pubDate>Sat, 11 Nov 2023 15:00:00 -0800</pubDate><guid>/blog/contributor-spotlight-johanna-ojeling/</guid><category>blog</category></item></channel></rss> |
