| <rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache Beam – release</title><link>/categories/release/</link><description>Recent content in release on Apache Beam</description><generator>Hugo -- gohugo.io</generator><language>en</language><lastBuildDate>Wed, 26 Jun 2024 13:00:00 -0800</lastBuildDate><atom:link href="/categories/release/index.xml" rel="self" type="application/rss+xml"/><item><title>Blog: Apache Beam 2.57.0</title><link>/blog/beam-2.57.0/</link><pubDate>Wed, 26 Jun 2024 13:00:00 -0800</pubDate><guid>/blog/beam-2.57.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.57.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2570-2024-06-26">download page</a> for this release.</p> |
| <p>For more information on changes in 2.57.0, check out the <a href="https://github.com/apache/beam/milestone/21">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Apache Beam adds Python 3.12 support (<a href="https://github.com/apache/beam/issues/29149">#29149</a>).</li> |
| <li>Added FlinkRunner for Flink 1.18 (<a href="https://github.com/apache/beam/issues/30789">#30789</a>).</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Ensure that BigtableIO closes the reader streams (<a href="https://github.com/apache/beam/issues/31477">#31477</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Added Feast feature store handler for enrichment transform (Python) (<a href="https://github.com/apache/beam/issues/30964">#30957</a>).</li> |
| <li>BigQuery per-worker metrics are reported by default for Streaming Dataflow Jobs (Java) (<a href="https://github.com/apache/beam/pull/31015">#31015</a>)</li> |
| <li>Adds <code>inMemory()</code> variant of Java List and Map side inputs for more efficient lookups when the entire side input fits into memory.</li> |
| <li>Beam YAML now supports the jinja templating syntax. |
| Template variables can be passed with the (json-formatted) <code>--jinja_variables</code> flag.</li> |
| <li>DataFrame API now supports pandas 2.1.x and adds 12 more string functions for Series.(<a href="https://github.com/apache/beam/pull/31185">#31185</a>).</li> |
| <li>Added BigQuery handler for enrichment transform (Python) (<a href="https://github.com/apache/beam/pull/31295">#31295</a>)</li> |
| <li>Disable soft delete policy when creating the default bucket for a project (Java) (<a href="https://github.com/apache/beam/pull/31324">#31324</a>).</li> |
| <li>Added <code>DoFn.SetupContextParam</code> and <code>DoFn.BundleContextParam</code> which can be used |
| as a python <code>DoFn.process</code>, <code>Map</code>, or <code>FlatMap</code> parameter to invoke a context |
| manager per DoFn setup or bundle (analogous to using <code>setup</code>/<code>teardown</code> |
| or <code>start_bundle</code>/<code>finish_bundle</code> respectively.)</li> |
| <li>Go SDK Prism Runner |
| <ul> |
| <li>Pre-built Prism binaries are now part of the release and are available via the Github release page. (<a href="https://github.com/apache/beam/issues/29697">#29697</a>).</li> |
| <li>ProcessingTime is now handled synthetically with TestStream pipelines and Non-TestStream pipelines, for fast test pipeline execution by default. (<a href="https://github.com/apache/beam/issues/30083">#30083</a>). |
| <ul> |
| <li>Prism does NOT yet support &ldquo;real time&rdquo; execution for this release.</li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li>Improve processing for large elements to reduce the chances for exceeding 2GB protobuf limits (Python)([https://github.com/apache/beam/issues/31607]).</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>Java&rsquo;s View.asList() side inputs are now optimized for iterating rather than |
| indexing when in the global window. |
| This new implementation still supports all (immutable) List methods as before, |
| but some of the random access methods like get() and size() will be slower. |
| To use the old implementation one can use View.asList().withRandomAccess().</li> |
| <li>SchemaTransforms implemented with TypedSchemaTransformProvider now produce a |
| configuration Schema with snake_case naming convention |
| (<a href="https://github.com/apache/beam/pull/31374">#31374</a>). This will make the following |
| cases problematic: |
| <ul> |
| <li>Running a pre-2.57.0 remote SDK pipeline containing a 2.57.0+ Java SchemaTransform, |
| and vice versa:</li> |
| <li>Running a 2.57.0+ remote SDK pipeline containing a pre-2.57.0 Java SchemaTransform</li> |
| <li>All direct uses of Python&rsquo;s <a href="https://github.com/apache/beam/blob/a998107a1f5c3050821eef6a5ad5843d8adb8aec/sdks/python/apache_beam/transforms/external.py#L381">SchemaAwareExternalTransform</a> |
| should be updated to use new snake_case parameter names.</li> |
| </ul> |
| </li> |
| <li>Upgraded Jackson Databind to 2.15.4 (Java) (<a href="https://github.com/apache/beam/issues/26743">#26743</a>). |
| jackson-2.15 has known breaking changes. An important one is it imposed a buffer limit for parser. |
| If your custom PTransform/DoFn are affected, refer to <a href="https://github.com/apache/beam/pull/31580">#31580</a> for mitigation.</li> |
| </ul> |
| <p>For the most up to date list of known issues, see <a href="https://github.com/apache/beam/blob/master/CHANGES.md">https://github.com/apache/beam/blob/master/CHANGES.md</a></p> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.57.0 release. Thank you to all contributors!</p> |
| <p>Ahmed Abualsaud</p> |
| <p>Ahmet Altay</p> |
| <p>Alexey Romanenko</p> |
| <p>Andrey Devyatkin</p> |
| <p>Anody Zhang</p> |
| <p>Arvind Ram</p> |
| <p>Ben Konz</p> |
| <p>Bruno Volpato</p> |
| <p>Celeste Zeng</p> |
| <p>Chamikara Jayalath</p> |
| <p>Claire McGinty</p> |
| <p>Colm O hEigeartaigh</p> |
| <p>Damon</p> |
| <p>Danny McCormick</p> |
| <p>Evan Galpin</p> |
| <p>Ferran Fernández Garrido</p> |
| <p>Florent Biville</p> |
| <p>Jack Dingilian</p> |
| <p>Jack McCluskey</p> |
| <p>Jan Lukavský</p> |
| <p>JayajP</p> |
| <p>Jeff Kinard</p> |
| <p>Jeffrey Kinard</p> |
| <p>John Casey</p> |
| <p>Justin Uang</p> |
| <p>Kenneth Knowles</p> |
| <p>Kevin Zhou</p> |
| <p>Liam Miller-Cushon</p> |
| <p>Maarten Vercruysse</p> |
| <p>Maciej Szwaja</p> |
| <p>Maja Kontrec Rönn</p> |
| <p>Marc hurabielle</p> |
| <p>Martin Trieu</p> |
| <p>Mattie Fu</p> |
| <p>Min Zhu</p> |
| <p>Naireen Hussain</p> |
| <p>Nick Anikin</p> |
| <p>Pablo Rodriguez Defino</p> |
| <p>Paul King</p> |
| <p>Priyans Desai</p> |
| <p>Radosław Stankiewicz</p> |
| <p>Rebecca Szper</p> |
| <p>Ritesh Ghorse</p> |
| <p>Robert Bradshaw</p> |
| <p>Robert Burke</p> |
| <p>Rodrigo Bozzolo</p> |
| <p>RyuSA</p> |
| <p>Sam Rohde</p> |
| <p>Sam Whittle</p> |
| <p>Sergei Lilichenko</p> |
| <p>Shahar Epstein</p> |
| <p>Shunping Huang</p> |
| <p>Svetak Sundhar</p> |
| <p>Tomo Suzuki</p> |
| <p>Tony Tang</p> |
| <p>Valentyn Tymofieiev</p> |
| <p>Vincent Stollenwerk</p> |
| <p>Vineet Kumar</p> |
| <p>Vitaly Terentyev</p> |
| <p>Vlado Djerek</p> |
| <p>XQ Hu</p> |
| <p>Yi Hu</p> |
| <p>akashorabek</p> |
| <p>bzablocki</p> |
| <p>kberezin</p></description></item><item><title>Blog: Apache Beam 2.56.0</title><link>/blog/beam-2.56.0/</link><pubDate>Wed, 01 May 2024 10:00:00 -0400</pubDate><guid>/blog/beam-2.56.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.56.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2550-2023-03-25">download page</a> for this release.</p> |
| <p>For more information on changes in 2.56.0, check out the <a href="https://github.com/apache/beam/milestone/20">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Added FlinkRunner for Flink 1.17, removed support for Flink 1.12 and 1.13. Previous version of Pipeline running on Flink 1.16 and below can be upgraded to 1.17, if the Pipeline is first updated to Beam 2.56.0 with the same Flink version. After Pipeline runs with Beam 2.56.0, it should be possible to upgrade to FlinkRunner with Flink 1.17. (<a href="https://github.com/apache/beam/issues/29939">#29939</a>)</li> |
| <li>New Managed I/O Java API (<a href="https://github.com/apache/beam/pull/30830">#30830</a>).</li> |
| <li>New Ordered Processing PTransform added for processing order-sensitive stateful data (<a href="https://github.com/apache/beam/pull/30735">#30735</a>).</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Upgraded Avro version to 1.11.3, kafka-avro-serializer and kafka-schema-registry-client versions to 7.6.0 (Java) (<a href="https://github.com/apache/beam/pull/30638">#30638</a>). |
| The newer Avro package is known to have breaking changes. If you are affected, you can keep pinned to older Avro versions which are also tested with Beam.</li> |
| <li>Iceberg read/write support is available through the new Managed I/O Java API (<a href="https://github.com/apache/beam/pull/30830">#30830</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Profiling of Cythonized code has been disabled by default. This might improve performance for some Python pipelines (<a href="https://github.com/apache/beam/pull/30938">#30938</a>).</li> |
| <li>Bigtable enrichment handler now accepts a custom function to build a composite row key. (Python) (<a href="https://github.com/apache/beam/issues/30975">#30974</a>).</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>Default consumer polling timeout for KafkaIO.Read was increased from 1 second to 2 seconds. Use KafkaIO.read().withConsumerPollingTimeout(Duration duration) to configure this timeout value when necessary (<a href="https://github.com/apache/beam/issues/30870">#30870</a>).</li> |
| <li>Python Dataflow users no longer need to manually specify &ndash;streaming for pipelines using unbounded sources such as ReadFromPubSub.</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed locking issue when shutting down inactive bundle processors. Symptoms of this issue include slowness or stuckness in long-running jobs (Python) (<a href="https://github.com/apache/beam/pull/30679">#30679</a>).</li> |
| <li>Fixed logging issue that caused silecing the pip output when installing of dependencies provided in <code>--requirements_file</code> (Python).</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.56.0 release. Thank you to all contributors!</p> |
| <p>Abacn</p> |
| <p>Ahmed Abualsaud</p> |
| <p>Andrei Gurau</p> |
| <p>Andrey Devyatkin</p> |
| <p>Aravind Pedapudi</p> |
| <p>Arun Pandian</p> |
| <p>Arvind Ram</p> |
| <p>Bartosz Zablocki</p> |
| <p>Brachi Packter</p> |
| <p>Byron Ellis</p> |
| <p>Chamikara Jayalath</p> |
| <p>Clement DAL PALU</p> |
| <p>Damon</p> |
| <p>Danny McCormick</p> |
| <p>Daria Bezkorovaina</p> |
| <p>Dip Patel</p> |
| <p>Evan Burrell</p> |
| <p>Hai Joey Tran</p> |
| <p>Jack McCluskey</p> |
| <p>Jan Lukavský</p> |
| <p>JayajP</p> |
| <p>Jeff Kinard</p> |
| <p>Julien Tournay</p> |
| <p>Kenneth Knowles</p> |
| <p>Luís Bianchin</p> |
| <p>Maciej Szwaja</p> |
| <p>Melody Shen</p> |
| <p>Oleh Borysevych</p> |
| <p>Pablo Estrada</p> |
| <p>Rebecca Szper</p> |
| <p>Ritesh Ghorse</p> |
| <p>Robert Bradshaw</p> |
| <p>Sam Whittle</p> |
| <p>Sergei Lilichenko</p> |
| <p>Shahar Epstein</p> |
| <p>Shunping Huang</p> |
| <p>Svetak Sundhar</p> |
| <p>Timothy Itodo</p> |
| <p>Veronica Wasson</p> |
| <p>Vitaly Terentyev</p> |
| <p>Vlado Djerek</p> |
| <p>Yi Hu</p> |
| <p>akashorabek</p> |
| <p>bzablocki</p> |
| <p>clmccart</p> |
| <p>damccorm</p> |
| <p>dependabot[bot]</p> |
| <p>dmitryor</p> |
| <p>github-actions[bot]</p> |
| <p>liferoad</p> |
| <p>martin trieu</p> |
| <p>tvalentyn</p> |
| <p>xianhualiu</p></description></item><item><title>Blog: Apache Beam 2.55.0</title><link>/blog/beam-2.55.0/</link><pubDate>Mon, 25 Mar 2024 10:00:00 -0400</pubDate><guid>/blog/beam-2.55.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.55.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2550-2023-03-25">download page</a> for this release.</p> |
| <p>For more information on changes in 2.55.0, check out the <a href="https://github.com/apache/beam/milestone/19">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>The Python SDK will now include automatically generated wrappers for external Java transforms! (<a href="https://github.com/apache/beam/pull/29834">#29834</a>)</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Added support for handling bad records to BigQueryIO (<a href="https://github.com/apache/beam/pull/30081">#30081</a>). |
| <ul> |
| <li>Full Support for Storage Read and Write APIs</li> |
| <li>Partial Support for File Loads (Failures writing to files supported, failures loading files to BQ unsupported)</li> |
| <li>No Support for Extract or Streaming Inserts</li> |
| </ul> |
| </li> |
| <li>Added support for handling bad records to PubSubIO (<a href="https://github.com/apache/beam/pull/30372">#30372</a>). |
| <ul> |
| <li>Support is not available for handling schema mismatches, and enabling error handling for writing to Pub/Sub topics with schemas is not recommended</li> |
| </ul> |
| </li> |
| <li><code>--enableBundling</code> pipeline option for BigQueryIO DIRECT_READ is replaced by <code>--enableStorageReadApiV2</code>. Both were considered experimental and subject to change (Java) (<a href="https://github.com/apache/beam/issues/26354">#26354</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Allow writing clustered and not time-partitioned BigQuery tables (Java) (<a href="https://github.com/apache/beam/pull/30094">#30094</a>).</li> |
| <li>Redis cache support added to RequestResponseIO and Enrichment transform (Python) (<a href="https://github.com/apache/beam/pull/30307">#30307</a>)</li> |
| <li>Merged <code>sdks/java/fn-execution</code> and <code>runners/core-construction-java</code> into the main SDK. These artifacts were never meant for users, but noting |
| that they no longer exist. These are steps to bring portability into the core SDK alongside all other core functionality.</li> |
| <li>Added Vertex AI Feature Store handler for Enrichment transform (Python) (<a href="https://github.com/apache/beam/pull/30388">#30388</a>)</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>Arrow version was bumped to 15.0.0 from 5.0.0 (<a href="https://github.com/apache/beam/pull/30181">#30181</a>).</li> |
| <li>Go SDK users who build custom worker containers may run into issues with the move to distroless containers as a base (see Security Fixes). |
| <ul> |
| <li>The issue stems from distroless containers lacking additional tools, which current custom container processes may rely on.</li> |
| <li>See <a href="https://beam.apache.org/documentation/runtime/environments/#from-scratch-go">https://beam.apache.org/documentation/runtime/environments/#from-scratch-go</a> for instructions on building and using a custom container.</li> |
| </ul> |
| </li> |
| <li>Python SDK has changed the default value for the <code>--max_cache_memory_usage_mb</code> pipeline option from 100 to 0. This option was first introduced in the 2.52.0 SDK version. This change restores the behavior of the 2.51.0 SDK, which does not use the state cache. If your pipeline uses iterable side inputs views, consider increasing the cache size by setting the option manually. (<a href="https://github.com/apache/beam/issues/30360">#30360</a>).</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>N/A</li> |
| </ul> |
| <h2 id="bug-fixes">Bug fixes</h2> |
| <ul> |
| <li>Fixed <code>SpannerIO.readChangeStream</code> to support propagating credentials from pipeline options |
| to the <code>getDialect</code> calls for authenticating with Spanner (Java) (<a href="https://github.com/apache/beam/pull/30361">#30361</a>).</li> |
| <li>Reduced the number of HTTP requests in GCSIO function calls (Python) (<a href="https://github.com/apache/beam/pull/30205">#30205</a>)</li> |
| </ul> |
| <h2 id="security-fixes">Security Fixes</h2> |
| <ul> |
| <li>Go SDK base container image moved to distroless/base-nossl-debian12, reducing vulnerable container surface to kernel and glibc (<a href="https://github.com/apache/beam/pull/30011">#30011</a>).</li> |
| </ul> |
| <h2 id="known-issues">Known Issues</h2> |
| <ul> |
| <li>In Python pipelines, when shutting down inactive bundle processors, shutdown logic can overaggressively hold the lock, blocking acceptance of new work. Symptoms of this issue include slowness or stuckness in long-running jobs. Fixed in 2.56.0 (<a href="https://github.com/apache/beam/pull/30679">#30679</a>).</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.55.0 release. Thank you to all contributors!</p> |
| <p>Ahmed Abualsaud</p> |
| <p>Anand Inguva</p> |
| <p>Andrew Crites</p> |
| <p>Andrey Devyatkin</p> |
| <p>Arun Pandian</p> |
| <p>Arvind Ram</p> |
| <p>Chamikara Jayalath</p> |
| <p>Chris Gray</p> |
| <p>Claire McGinty</p> |
| <p>Damon Douglas</p> |
| <p>Dan Ellis</p> |
| <p>Danny McCormick</p> |
| <p>Daria Bezkorovaina</p> |
| <p>Dima I</p> |
| <p>Edward Cui</p> |
| <p>Ferran Fernández Garrido</p> |
| <p>GStravinsky</p> |
| <p>Jan Lukavský</p> |
| <p>Jason Mitchell</p> |
| <p>JayajP</p> |
| <p>Jeff Kinard</p> |
| <p>Jeffrey Kinard</p> |
| <p>Kenneth Knowles</p> |
| <p>Mattie Fu</p> |
| <p>Michel Davit</p> |
| <p>Oleh Borysevych</p> |
| <p>Ritesh Ghorse</p> |
| <p>Ritesh Tarway</p> |
| <p>Robert Bradshaw</p> |
| <p>Robert Burke</p> |
| <p>Sam Whittle</p> |
| <p>Scott Strong</p> |
| <p>Shunping Huang</p> |
| <p>Steven van Rossum</p> |
| <p>Svetak Sundhar</p> |
| <p>Talat UYARER</p> |
| <p>Ukjae Jeong (Jay)</p> |
| <p>Vitaly Terentyev</p> |
| <p>Vlado Djerek</p> |
| <p>Yi Hu</p> |
| <p>akashorabek</p> |
| <p>case-k</p> |
| <p>clmccart</p> |
| <p>dengwe1</p> |
| <p>dhruvdua</p> |
| <p>hardshah</p> |
| <p>johnjcasey</p> |
| <p>liferoad</p> |
| <p>martin trieu</p> |
| <p>tvalentyn</p></description></item><item><title>Blog: Apache Beam 2.54.0</title><link>/blog/beam-2.54.0/</link><pubDate>Wed, 14 Feb 2024 09:00:00 -0400</pubDate><guid>/blog/beam-2.54.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.54.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/">download page</a> for this release.</p> |
| <p>For more information on changes in 2.54.0, check out the <a href="https://github.com/apache/beam/milestone/18">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li><a href="https://s.apache.org/enrichment-transform">Enrichment Transform</a> along with GCP BigTable handler added to Python SDK (<a href="https://github.com/apache/beam/pull/30001">#30001</a>).</li> |
| <li>Beam Java Batch pipelines run on Google Cloud Dataflow will default to the Portable Runner (v2) starting with this version. (All other languages are already on Runner V2.) See <a href="https://cloud.google.com/dataflow/docs/runner-v2">Runner V2 documentation</a> for how to enable or disable it intentionally.</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Added support for writing to BigQuery dynamic destinations with Python&rsquo;s Storage Write API (<a href="https://github.com/apache/beam/pull/30045">#30045</a>)</li> |
| <li>Adding support for Tuples DataType in ClickHouse (Java) (<a href="https://github.com/apache/beam/pull/29715">#29715</a>).</li> |
| <li>Added support for handling bad records to FileIO, TextIO, AvroIO (<a href="https://github.com/apache/beam/pull/29670">#29670</a>).</li> |
| <li>Added support for handling bad records to BigtableIO (<a href="https://github.com/apache/beam/pull/29885">#29885</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li><a href="https://s.apache.org/enrichment-transform">Enrichment Transform</a> along with GCP BigTable handler added to Python SDK (<a href="https://github.com/apache/beam/pull/30001">#30001</a>).</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>N/A</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>N/A</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed a memory leak affecting some Go SDK since 2.46.0. (<a href="https://github.com/apache/beam/pull/28142">#28142</a>)</li> |
| </ul> |
| <h2 id="security-fixes">Security Fixes</h2> |
| <ul> |
| <li>N/A</li> |
| </ul> |
| <h2 id="known-issues">Known Issues</h2> |
| <ul> |
| <li>Some Python pipelines that run with 2.52.0-2.54.0 SDKs and use large materialized side inputs might be affected by a performance regression. To restore the prior behavior on these SDK versions, supply the <code>--max_cache_memory_usage_mb=0</code> pipeline option. (<a href="https://github.com/apache/beam/issues/30360">#30360</a>).</li> |
| <li>Python pipelines that run with 2.53.0-2.54.0 SDKs and perform file operations on GCS might be affected by excess HTTP requests. This could lead to a performance regression or a permission issue. (<a href="https://github.com/apache/beam/issues/28398">#28398</a>)</li> |
| <li>In Python pipelines, when shutting down inactive bundle processors, shutdown logic can overaggressively hold the lock, blocking acceptance of new work. Symptoms of this issue include slowness or stuckness in long-running jobs. Fixed in 2.56.0 (<a href="https://github.com/apache/beam/pull/30679">#30679</a>).</li> |
| </ul> |
| <p>For the most up to date list of known issues, see <a href="https://github.com/apache/beam/blob/master/CHANGES.md">https://github.com/apache/beam/blob/master/CHANGES.md</a></p> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.54.0 release. Thank you to all contributors!</p> |
| <p>Ahmed Abualsaud</p> |
| <p>Alexey Romanenko</p> |
| <p>Anand Inguva</p> |
| <p>Andrew Crites</p> |
| <p>Arun Pandian</p> |
| <p>Bruno Volpato</p> |
| <p>caneff</p> |
| <p>Chamikara Jayalath</p> |
| <p>Changyu Li</p> |
| <p>Cheskel Twersky</p> |
| <p>Claire McGinty</p> |
| <p>clmccart</p> |
| <p>Damon</p> |
| <p>Danny McCormick</p> |
| <p>dependabot[bot]</p> |
| <p>Edward Cheng</p> |
| <p>Ferran Fernández Garrido</p> |
| <p>Hai Joey Tran</p> |
| <p>hugo-syn</p> |
| <p>Issac</p> |
| <p>Jack McCluskey</p> |
| <p>Jan Lukavský</p> |
| <p>JayajP</p> |
| <p>Jeffrey Kinard</p> |
| <p>Jerry Wang</p> |
| <p>Jing</p> |
| <p>Joey Tran</p> |
| <p>johnjcasey</p> |
| <p>Kenneth Knowles</p> |
| <p>Knut Olav Løite</p> |
| <p>liferoad</p> |
| <p>Marc</p> |
| <p>Mark Zitnik</p> |
| <p>martin trieu</p> |
| <p>Mattie Fu</p> |
| <p>Naireen Hussain</p> |
| <p>Neeraj Bansal</p> |
| <p>Niel Markwick</p> |
| <p>Oleh Borysevych</p> |
| <p>pablo rodriguez defino</p> |
| <p>Rebecca Szper</p> |
| <p>Ritesh Ghorse</p> |
| <p>Robert Bradshaw</p> |
| <p>Robert Burke</p> |
| <p>Sam Whittle</p> |
| <p>Shunping Huang</p> |
| <p>Svetak Sundhar</p> |
| <p>S. Veyrié</p> |
| <p>Talat UYARER</p> |
| <p>tvalentyn</p> |
| <p>Vlado Djerek</p> |
| <p>Yi Hu</p> |
| <p>Zechen Jian</p></description></item><item><title>Blog: Apache Beam 2.53.0</title><link>/blog/beam-2.53.0/</link><pubDate>Thu, 04 Jan 2024 09:00:00 -0400</pubDate><guid>/blog/beam-2.53.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.53.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/">download page</a> for this release.</p> |
| <p>For more information on changes in 2.53.0, check out the <a href="https://github.com/apache/beam/milestone/17">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Python streaming users that use 2.47.0 and newer versions of Beam should update to version 2.53.0, which fixes a known issue: (<a href="https://github.com/apache/beam/issues/27330">#27330</a>).</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>TextIO now supports skipping multiple header lines (Java) (<a href="https://github.com/apache/beam/issues/17990">#17990</a>).</li> |
| <li>Python GCSIO is now implemented with GCP GCS Client instead of apitools (<a href="https://github.com/apache/beam/issues/25676">#25676</a>)</li> |
| <li>Adding support for LowCardinality DataType in ClickHouse (Java) (<a href="https://github.com/apache/beam/pull/29533">#29533</a>).</li> |
| <li>Added support for handling bad records to KafkaIO (Java) (<a href="https://github.com/apache/beam/pull/29546">#29546</a>)</li> |
| <li>Add support for generating text embeddings in MLTransform for Vertex AI and Hugging Face Hub models.(<a href="https://github.com/apache/beam/pull/29564">#29564</a>)</li> |
| <li>NATS IO connector added (Go) (<a href="https://github.com/apache/beam/issues/29000">#29000</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>The Python SDK now type checks <code>collections.abc.Collections</code> types properly. Some type hints that were erroneously allowed by the SDK may now fail. (<a href="https://github.com/apache/beam/pull/29272">#29272</a>)</li> |
| <li>Running multi-language pipelines locally no longer requires Docker. |
| Instead, the same (generally auto-started) subprocess used to perform the |
| expansion can also be used as the cross-language worker.</li> |
| <li>Framework for adding Error Handlers to composite transforms added in Java (<a href="https://github.com/apache/beam/pull/29164">#29164</a>).</li> |
| <li>Python 3.11 images now include google-cloud-profiler (<a href="https://github.com/apache/beam/pull/29651">#29561</a>).</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>Euphoria DSL is deprecated and will be removed in a future release (not before 2.56.0) (<a href="https://github.com/apache/beam/issues/29451">#29451</a>)</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>(Python) Fixed sporadic crashes in streaming pipelines that affected some users of 2.47.0 and newer SDKs (<a href="https://github.com/apache/beam/issues/27330">#27330</a>).</li> |
| <li>(Python) Fixed a bug that caused MLTransform to drop identical elements in the output PCollection (<a href="https://github.com/apache/beam/issues/29600">#29600</a>).</li> |
| </ul> |
| <h2 id="security-fixes">Security Fixes</h2> |
| <ul> |
| <li>Upgraded to go 1.21.5 to build, fixing <a href="https://security-tracker.debian.org/tracker/CVE-2023-45285">CVE-2023-45285</a> and <a href="https://security-tracker.debian.org/tracker/CVE-2023-39326">CVE-2023-39326</a></li> |
| </ul> |
| <h2 id="known-issues">Known Issues</h2> |
| <ul> |
| <li>Potential race condition causing NPE in DataflowExecutionStateSampler in Dataflow Java Streaming pipelines (<a href="https://github.com/apache/beam/issues/29987">#29987</a>).</li> |
| <li>Some Python pipelines that run with 2.52.0-2.54.0 SDKs and use large materialized side inputs might be affected by a performance regression. To restore the prior behavior on these SDK versions, supply the <code>--max_cache_memory_usage_mb=0</code> pipeline option. (<a href="https://github.com/apache/beam/issues/30360">#30360</a>).</li> |
| <li>Python pipelines that run with 2.53.0-2.54.0 SDKs and perform file operations on GCS might be affected by excess HTTP requests. This could lead to a performance regression or a permission issue. (<a href="https://github.com/apache/beam/issues/28398">#28398</a>)</li> |
| <li>In Python pipelines, when shutting down inactive bundle processors, shutdown logic can overaggressively hold the lock, blocking acceptance of new work. Symptoms of this issue include slowness or stuckness in long-running jobs. Fixed in 2.56.0 (<a href="https://github.com/apache/beam/pull/30679">#30679</a>).</li> |
| </ul> |
| <p>For the most up to date list of known issues, see <a href="https://github.com/apache/beam/blob/master/CHANGES.md">https://github.com/apache/beam/blob/master/CHANGES.md</a></p> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.53.0 release. Thank you to all contributors!</p> |
| <p>Ahmed Abualsaud</p> |
| <p>Ahmet Altay</p> |
| <p>Alexey Romanenko</p> |
| <p>Anand Inguva</p> |
| <p>Arun Pandian</p> |
| <p>Balázs Németh</p> |
| <p>Bruno Volpato</p> |
| <p>Byron Ellis</p> |
| <p>Calvin Swenson Jr</p> |
| <p>Chamikara Jayalath</p> |
| <p>Clay Johnson</p> |
| <p>Damon</p> |
| <p>Danny McCormick</p> |
| <p>Ferran Fernández Garrido</p> |
| <p>Georgii Zemlianyi</p> |
| <p>Israel Herraiz</p> |
| <p>Jack McCluskey</p> |
| <p>Jacob Tomlinson</p> |
| <p>Jan Lukavský</p> |
| <p>JayajP</p> |
| <p>Jeffrey Kinard</p> |
| <p>Johanna Öjeling</p> |
| <p>Julian Braha</p> |
| <p>Julien Tournay</p> |
| <p>Kenneth Knowles</p> |
| <p>Lawrence Qiu</p> |
| <p>Mark Zitnik</p> |
| <p>Mattie Fu</p> |
| <p>Michel Davit</p> |
| <p>Mike Williamson</p> |
| <p>Naireen</p> |
| <p>Naireen Hussain</p> |
| <p>Niel Markwick</p> |
| <p>Pablo Estrada</p> |
| <p>Radosław Stankiewicz</p> |
| <p>Rebecca Szper</p> |
| <p>Reuven Lax</p> |
| <p>Ritesh Ghorse</p> |
| <p>Robert Bradshaw</p> |
| <p>Robert Burke</p> |
| <p>Sam Rohde</p> |
| <p>Sam Whittle</p> |
| <p>Shunping Huang</p> |
| <p>Svetak Sundhar</p> |
| <p>Talat UYARER</p> |
| <p>Tom Stepp</p> |
| <p>Tony Tang</p> |
| <p>Vlado Djerek</p> |
| <p>Yi Hu</p> |
| <p>Zechen Jiang</p> |
| <p>clmccart</p> |
| <p>damccorm</p> |
| <p>darshan-sj</p> |
| <p>gabry.wu</p> |
| <p>johnjcasey</p> |
| <p>liferoad</p> |
| <p>lrakla</p> |
| <p>martin trieu</p> |
| <p>tvalentyn</p></description></item><item><title>Blog: Apache Beam 2.52.0</title><link>/blog/beam-2.52.0/</link><pubDate>Fri, 17 Nov 2023 09:00:00 -0400</pubDate><guid>/blog/beam-2.52.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.52.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2520-2023-11-17">download page</a> for this release.</p> |
| <p>For more information on changes in 2.52.0, check out the <a href="https://github.com/apache/beam/milestone/16">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Previously deprecated Avro-dependent code (Beam Release 2.46.0) has been finally removed from Java SDK &ldquo;core&rdquo; package. |
| Please, use <code>beam-sdks-java-extensions-avro</code> instead. This will allow to easily update Avro version in user code without |
| potential breaking changes in Beam &ldquo;core&rdquo; since the Beam Avro extension already supports the latest Avro versions and |
| should handle this. (<a href="https://github.com/apache/beam/issues/25252">#25252</a>).</li> |
| <li>Publishing Java 21 SDK container images now supported as part of Apache Beam release process. (<a href="https://github.com/apache/beam/issues/28120">#28120</a>) |
| <ul> |
| <li>Direct Runner and Dataflow Runner support running pipelines on Java21 (experimental until tests fully setup). For other runners (Flink, Spark, Samza, etc) support status depend on runner projects.</li> |
| </ul> |
| </li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Add <code>UseDataStreamForBatch</code> pipeline option to the Flink runner. When it is set to true, Flink runner will run batch |
| jobs using the DataStream API. By default the option is set to false, so the batch jobs are still executed |
| using the DataSet API.</li> |
| <li><code>upload_graph</code> as one of the Experiments options for DataflowRunner is no longer required when the graph is larger than 10MB for Java SDK (<a href="https://github.com/apache/beam/pull/28621">PR#28621</a>).</li> |
| <li>state amd side input cache has been enabled to a default of 100 MB. Use <code>--max_cache_memory_usage_mb=X</code> to provide cache size for the user state API and side inputs. (Python) (<a href="https://github.com/apache/beam/issues/28770">#28770</a>).</li> |
| <li>Beam YAML stable release. Beam pipelines can now be written using YAML and leverage the Beam YAML framework which includes a preliminary set of IO&rsquo;s and turnkey transforms. More information can be found in the YAML root folder and in the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/README.md">README</a>.</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li><code>org.apache.beam.sdk.io.CountingSource.CounterMark</code> uses custom <code>CounterMarkCoder</code> as a default coder since all Avro-dependent |
| classes finally moved to <code>extensions/avro</code>. In case if it&rsquo;s still required to use <code>AvroCoder</code> for <code>CounterMark</code>, then, |
| as a workaround, a copy of &ldquo;old&rdquo; <code>CountingSource</code> class should be placed into a project code and used directly |
| (<a href="https://github.com/apache/beam/issues/25252">#25252</a>).</li> |
| <li>Renamed <code>host</code> to <code>firestoreHost</code> in <code>FirestoreOptions</code> to avoid potential conflict of command line arguments (Java) (<a href="https://github.com/apache/beam/pull/29201">#29201</a>).</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed &ldquo;Desired bundle size 0 bytes must be greater than 0&rdquo; in Java SDK&rsquo;s BigtableIO.BigtableSource when you have more cores than bytes to read (Java) <a href="https://github.com/apache/beam/issues/28793">#28793</a>.</li> |
| <li><code>watch_file_pattern</code> arg of the <a href="https://github.com/apache/beam/blob/104c10b3ee536a9a3ea52b4dbf62d86b669da5d9/sdks/python/apache_beam/ml/inference/base.py#L997">RunInference</a> arg had no effect prior to 2.52.0. To use the behavior of arg <code>watch_file_pattern</code> prior to 2.52.0, follow the documentation at <a href="https://beam.apache.org/documentation/ml/side-input-updates/">https://beam.apache.org/documentation/ml/side-input-updates/</a> and use <code>WatchFilePattern</code> PTransform as a SideInput. (<a href="https://github.com/apache/beam/pulls/28948">#28948</a>)</li> |
| <li><code>MLTransform</code> doesn&rsquo;t output artifacts such as min, max and quantiles. Instead, <code>MLTransform</code> will add a feature to output these artifacts as human readable format - <a href="https://github.com/apache/beam/issues/29017">#29017</a>. For now, to use the artifacts such as min and max that were produced by the eariler <code>MLTransform</code>, use <code>read_artifact_location</code> of <code>MLTransform</code>, which reads artifacts that were produced earlier in a different <code>MLTransform</code> (<a href="https://github.com/apache/beam/pull/29016/">#29016</a>)</li> |
| <li>Fixed a memory leak, which affected some long-running Python pipelines: <a href="https://github.com/apache/beam/issues/28246">#28246</a>.</li> |
| </ul> |
| <h2 id="security-fixes">Security Fixes</h2> |
| <ul> |
| <li>Fixed <a href="https://www.cve.org/CVERecord?id=CVE-2023-39325">CVE-2023-39325</a> (Java/Python/Go) (<a href="https://github.com/apache/beam/issues/29118">#29118</a>).</li> |
| <li>Mitigated <a href="https://nvd.nist.gov/vuln/detail/CVE-2023-47248">CVE-2023-47248</a> (Python) <a href="https://github.com/apache/beam/issues/29392">#29392</a>.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.52.0 release. Thank you to all contributors!</p> |
| <p>Ahmed Abualsaud</p> |
| <p>Ahmet Altay</p> |
| <p>Aleksandr Dudko</p> |
| <p>Alexey Romanenko</p> |
| <p>Anand Inguva</p> |
| <p>Andrei Gurau</p> |
| <p>Andrey Devyatkin</p> |
| <p>BjornPrime</p> |
| <p>Bruno Volpato</p> |
| <p>Bulat</p> |
| <p>Chamikara Jayalath</p> |
| <p>Damon</p> |
| <p>Danny McCormick</p> |
| <p>Devansh Modi</p> |
| <p>Dominik Dębowczyk</p> |
| <p>Ferran Fernández Garrido</p> |
| <p>Hai Joey Tran</p> |
| <p>Israel Herraiz</p> |
| <p>Jack McCluskey</p> |
| <p>Jan Lukavský</p> |
| <p>JayajP</p> |
| <p>Jeff Kinard</p> |
| <p>Jeffrey Kinard</p> |
| <p>Jiangjie Qin</p> |
| <p>Jing</p> |
| <p>Joar Wandborg</p> |
| <p>Johanna Öjeling</p> |
| <p>Julien Tournay</p> |
| <p>Kanishk Karanawat</p> |
| <p>Kenneth Knowles</p> |
| <p>Kerry Donny-Clark</p> |
| <p>Luís Bianchin</p> |
| <p>Minbo Bae</p> |
| <p>Pranav Bhandari</p> |
| <p>Rebecca Szper</p> |
| <p>Reuven Lax</p> |
| <p>Ritesh Ghorse</p> |
| <p>Robert Bradshaw</p> |
| <p>Robert Burke</p> |
| <p>RyuSA</p> |
| <p>Shunping Huang</p> |
| <p>Steven van Rossum</p> |
| <p>Svetak Sundhar</p> |
| <p>Tony Tang</p> |
| <p>Vitaly Terentyev</p> |
| <p>Vivek Sumanth</p> |
| <p>Vlado Djerek</p> |
| <p>Yi Hu</p> |
| <p>aku019</p> |
| <p>brucearctor</p> |
| <p>caneff</p> |
| <p>damccorm</p> |
| <p>ddebowczyk92</p> |
| <p>dependabot[bot]</p> |
| <p>dpcollins-google</p> |
| <p>edman124</p> |
| <p>gabry.wu</p> |
| <p>illoise</p> |
| <p>johnjcasey</p> |
| <p>jonathan-lemos</p> |
| <p>kennknowles</p> |
| <p>liferoad</p> |
| <p>magicgoody</p> |
| <p>martin trieu</p> |
| <p>nancyxu123</p> |
| <p>pablo rodriguez defino</p> |
| <p>tvalentyn</p></description></item><item><title>Blog: Apache Beam 2.51.0</title><link>/blog/beam-2.51.0/</link><pubDate>Wed, 11 Oct 2023 09:00:00 -0400</pubDate><guid>/blog/beam-2.51.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.51.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2510-2023-10-03">download page</a> for this release.</p> |
| <p>For more information on changes in 2.51.0, check out the <a href="https://github.com/apache/beam/milestone/15">detailed release notes</a>.</p> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>In Python, <a href="https://beam.apache.org/documentation/sdks/python-machine-learning/#why-use-the-runinference-api">RunInference</a> now supports loading many models in the same transform using a <a href="https://beam.apache.org/documentation/sdks/python-machine-learning/#use-a-keyed-modelhandler">KeyedModelHandler</a> (<a href="https://github.com/apache/beam/issues/27628">#27628</a>).</li> |
| <li>In Python, the <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.ml.inference.vertex_ai_inference.html#apache_beam.ml.inference.vertex_ai_inference.VertexAIModelHandlerJSON">VertexAIModelHandlerJSON</a> now supports passing in inference_args. These will be passed through to the Vertex endpoint as parameters.</li> |
| <li>Added support to run <code>mypy</code> on user pipelines (<a href="https://github.com/apache/beam/issues/27906">#27906</a>)</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>Removed fastjson library dependency for Beam SQL. Table property is changed to be based on jackson ObjectNode (Java) (<a href="https://github.com/apache/beam/issues/24154">#24154</a>).</li> |
| <li>Removed TensorFlow from Beam Python container images <a href="https://github.com/apache/beam/pull/28424">PR</a>. If you have been negatively affected by this change, please comment on <a href="https://github.com/apache/beam/issues/20605">#20605</a>.</li> |
| <li>Removed the parameter <code>t reflect.Type</code> from <code>parquetio.Write</code>. The element type is derived from the input PCollection (Go) (<a href="https://github.com/apache/beam/issues/28490">#28490</a>)</li> |
| <li>Refactor BeamSqlSeekableTable.setUp adding a parameter joinSubsetType. <a href="https://github.com/apache/beam/issues/28283">#28283</a></li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed exception chaining issue in GCS connector (Python) (<a href="https://github.com/apache/beam/issues/26769#issuecomment-1700422615">#26769</a>).</li> |
| <li>Fixed streaming inserts exception handling, GoogleAPICallErrors are now retried according to retry strategy and routed to failed rows where appropriate rather than causing a pipeline error (Python) (<a href="https://github.com/apache/beam/issues/21080">#21080</a>).</li> |
| <li>Fixed a bug in Python SDK&rsquo;s cross-language Bigtable sink that mishandled records that don&rsquo;t have an explicit timestamp set: <a href="https://github.com/apache/beam/issues/28632">#28632</a>.</li> |
| </ul> |
| <h2 id="security-fixes">Security Fixes</h2> |
| <ul> |
| <li>Python containers updated, fixing <a href="https://nvd.nist.gov/vuln/detail/CVE-2021-30474">CVE-2021-30474</a>, <a href="https://nvd.nist.gov/vuln/detail/CVE-2021-30475">CVE-2021-30475</a>, <a href="https://nvd.nist.gov/vuln/detail/CVE-2021-30473">CVE-2021-30473</a>, <a href="https://nvd.nist.gov/vuln/detail/CVE-2020-36133">CVE-2020-36133</a>, <a href="https://nvd.nist.gov/vuln/detail/CVE-2020-36131">CVE-2020-36131</a>, <a href="https://nvd.nist.gov/vuln/detail/CVE-2020-36130">CVE-2020-36130</a>, and <a href="https://nvd.nist.gov/vuln/detail/CVE-2020-36135">CVE-2020-36135</a></li> |
| <li>Used go 1.21.1 to build, fixing <a href="https://security-tracker.debian.org/tracker/CVE-2023-39320">CVE-2023-39320</a></li> |
| </ul> |
| <h2 id="known-issues">Known Issues</h2> |
| <ul> |
| <li>Python pipelines using BigQuery Storage Read API must pin <code>fastavro</code> dependency to 1.8.3 |
| or earlier: <a href="https://github.com/apache/beam/issues/28811">#28811</a></li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.50.0 release. Thank you to all contributors!</p> |
| <p>Adam Whitmore</p> |
| <p>Ahmed Abualsaud</p> |
| <p>Ahmet Altay</p> |
| <p>Aleksandr Dudko</p> |
| <p>Alexey Romanenko</p> |
| <p>Anand Inguva</p> |
| <p>Andrey Devyatkin</p> |
| <p>Arvind Ram</p> |
| <p>Arwin Tio</p> |
| <p>BjornPrime</p> |
| <p>Bruno Volpato</p> |
| <p>Bulat</p> |
| <p>Celeste Zeng</p> |
| <p>Chamikara Jayalath</p> |
| <p>Clay Johnson</p> |
| <p>Damon</p> |
| <p>Danny McCormick</p> |
| <p>David Cavazos</p> |
| <p>Dip Patel</p> |
| <p>Hai Joey Tran</p> |
| <p>Hao Xu</p> |
| <p>Haruka Abe</p> |
| <p>Jack Dingilian</p> |
| <p>Jack McCluskey</p> |
| <p>Jeff Kinard</p> |
| <p>Jeffrey Kinard</p> |
| <p>Joey Tran</p> |
| <p>Johanna Öjeling</p> |
| <p>Julien Tournay</p> |
| <p>Kenneth Knowles</p> |
| <p>Kerry Donny-Clark</p> |
| <p>Mattie Fu</p> |
| <p>Melissa Pashniak</p> |
| <p>Michel Davit</p> |
| <p>Moritz Mack</p> |
| <p>Pranav Bhandari</p> |
| <p>Rebecca Szper</p> |
| <p>Reeba Qureshi</p> |
| <p>Reuven Lax</p> |
| <p>Ritesh Ghorse</p> |
| <p>Robert Bradshaw</p> |
| <p>Robert Burke</p> |
| <p>Ruwann</p> |
| <p>Ryan Tam</p> |
| <p>Sam Rohde</p> |
| <p>Sereana Seim</p> |
| <p>Svetak Sundhar</p> |
| <p>Tim Grein</p> |
| <p>Udi Meiri</p> |
| <p>Valentyn Tymofieiev</p> |
| <p>Vitaly Terentyev</p> |
| <p>Vlado Djerek</p> |
| <p>Xinyu Liu</p> |
| <p>Yi Hu</p> |
| <p>Zbynek Konecny</p> |
| <p>Zechen Jiang</p> |
| <p>bzablocki</p> |
| <p>caneff</p> |
| <p>dependabot[bot]</p> |
| <p>gDuperran</p> |
| <p>gabry.wu</p> |
| <p>johnjcasey</p> |
| <p>kberezin-nshl</p> |
| <p>kennknowles</p> |
| <p>liferoad</p> |
| <p>lostluck</p> |
| <p>magicgoody</p> |
| <p>martin trieu</p> |
| <p>mosche</p> |
| <p>olalamichelle</p> |
| <p>tvalentyn</p> |
| <p>xqhu</p> |
| <p>Łukasz Spyra</p></description></item><item><title>Blog: Apache Beam 2.50.0</title><link>/blog/beam-2.50.0/</link><pubDate>Wed, 30 Aug 2023 09:00:00 -0400</pubDate><guid>/blog/beam-2.50.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.50.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2500-2023-08-30">download page</a> for this release.</p> |
| <p>For more information on changes in 2.50.0, check out the <a href="https://github.com/apache/beam/milestone/14">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Spark 3.2.2 is used as default version for Spark runner (<a href="https://github.com/apache/beam/issues/23804">#23804</a>).</li> |
| <li>The Go SDK has a new default local runner, called Prism (<a href="https://github.com/apache/beam/issues/24789">#24789</a>).</li> |
| <li>All Beam released container images are now <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/build-multi-arch-for-arm#what_is_a_multi-arch_image">multi-arch images</a> that support both x86 and ARM CPU architectures.</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Java KafkaIO now supports picking up topics via topicPattern (<a href="https://github.com/apache/beam/pull/26948">#26948</a>)</li> |
| <li>Support for read from Cosmos DB Core SQL API (<a href="https://github.com/apache/beam/issues/23604">#23604</a>)</li> |
| <li>Upgraded to HBase 2.5.5 for HBaseIO. (Java) (<a href="https://github.com/apache/beam/issues/19554">#27711</a>)</li> |
| <li>Added support for GoogleAdsIO source (Java) (<a href="https://github.com/apache/beam/pull/27681">#27681</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>The Go SDK now requires Go 1.20 to build. (<a href="https://github.com/apache/beam/issues/27558">#27558</a>)</li> |
| <li>The Go SDK has a new default local runner, Prism. (<a href="https://github.com/apache/beam/issues/24789">#24789</a>). |
| <ul> |
| <li>Prism is a portable runner that executes each transform independantly, ensuring coders.</li> |
| <li>At this point it supercedes the Go direct runner in functionality. The Go direct runner is now deprecated.</li> |
| <li>See <a href="https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/prism/README.md">https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/prism/README.md</a> for the goals and features of Prism.</li> |
| </ul> |
| </li> |
| <li>Hugging Face Model Handler for RunInference added to Python SDK. (<a href="https://github.com/apache/beam/pull/26632">#26632</a>)</li> |
| <li>Hugging Face Pipelines support for RunInference added to Python SDK. (<a href="https://github.com/apache/beam/pull/27399">#27399</a>)</li> |
| <li>Vertex AI Model Handler for RunInference now supports private endpoints (<a href="https://github.com/apache/beam/pull/27696">#27696</a>)</li> |
| <li>MLTransform transform added with support for common ML pre/postprocessing operations (<a href="https://github.com/apache/beam/pull/26795">#26795</a>)</li> |
| <li>Upgraded the Kryo extension for the Java SDK to Kryo 5.5.0. This brings in bug fixes, performance improvements, and serialization of Java 14 records. (<a href="https://github.com/apache/beam/issues/27635">#27635</a>)</li> |
| <li>All Beam released container images are now <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/build-multi-arch-for-arm#what_is_a_multi-arch_image">multi-arch images</a> that support both x86 and ARM CPU architectures. (<a href="https://github.com/apache/beam/issues/27674">#27674</a>). The multi-arch container images include: |
| <ul> |
| <li>All versions of Go, Python, Java and Typescript SDK containers.</li> |
| <li>All versions of Flink job server containers.</li> |
| <li>Java and Python expansion service containers.</li> |
| <li>Transform service controller container.</li> |
| <li>Spark3 job server container.</li> |
| </ul> |
| </li> |
| <li>Added support for batched writes to AWS SQS for improved throughput (Java, AWS 2).(<a href="https://github.com/apache/beam/issues/21429">#21429</a>)</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>Python SDK: Legacy runner support removed from Dataflow, all pipelines must use runner v2.</li> |
| <li>Python SDK: Dataflow Runner will no longer stage Beam SDK from PyPI in the <code>--staging_location</code> at pipeline submission. Custom container images that are not based on Beam&rsquo;s default image must include Apache Beam installation.(<a href="https://github.com/apache/beam/issues/26996">#26996</a>)</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>The Go Direct Runner is now Deprecated. It remains available to reduce migration churn. |
| <ul> |
| <li>Tests can be set back to the direct runner by overriding TestMain: <code>func TestMain(m *testing.M) { ptest.MainWithDefault(m, &quot;direct&quot;) }</code></li> |
| <li>It&rsquo;s recommended to fix issues seen in tests using Prism, as they can also happen on any portable runner.</li> |
| <li>Use the generic register package for your pipeline DoFns to ensure pipelines function on portable runners, like prism.</li> |
| <li>Do not rely on closures or using package globals for DoFn configuration. They don&rsquo;t function on portable runners.</li> |
| </ul> |
| </li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed DirectRunner bug in Python SDK where GroupByKey gets empty PCollection and fails when pipeline option <code>direct_num_workers!=1</code>.(<a href="https://github.com/apache/beam/pull/27373">#27373</a>)</li> |
| <li>Fixed BigQuery I/O bug when estimating size on queries that utilize row-level security (<a href="https://github.com/apache/beam/pull/27474">#27474</a>)</li> |
| <li>Beam Python containers rely on a version of Debian/aom that has several security vulnerabilities: <a href="https://nvd.nist.gov/vuln/detail/CVE-2021-30474">CVE-2021-30474</a>, <a href="https://nvd.nist.gov/vuln/detail/CVE-2021-30475">CVE-2021-30475</a>, <a href="https://nvd.nist.gov/vuln/detail/CVE-2021-30473">CVE-2021-30473</a>, <a href="https://nvd.nist.gov/vuln/detail/CVE-2020-36133">CVE-2020-36133</a>, <a href="https://nvd.nist.gov/vuln/detail/CVE-2020-36131">CVE-2020-36131</a>, <a href="https://nvd.nist.gov/vuln/detail/CVE-2020-36130">CVE-2020-36130</a>, and <a href="https://nvd.nist.gov/vuln/detail/CVE-2020-36135">CVE-2020-36135</a>.</li> |
| </ul> |
| <h2 id="known-issues">Known Issues</h2> |
| <ul> |
| <li>Long-running Python pipelines might experience a memory leak: <a href="https://github.com/apache/beam/issues/28246">#28246</a>.</li> |
| <li>Python Pipelines using BigQuery IO or <code>orjson</code> dependency might experience segmentation faults or get stuck: <a href="https://github.com/apache/beam/issues/28318">#28318</a>.</li> |
| <li>Python SDK&rsquo;s cross-language Bigtable sink mishandles records that don&rsquo;t have an explicit timestamp set: <a href="https://github.com/apache/beam/issues/28632">#28632</a>. To avoid this issue, set explicit timestamps for all records before writing to Bigtable.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.50.0 release. Thank you to all contributors!</p> |
| <p>Abacn</p> |
| <p>acejune</p> |
| <p>AdalbertMemSQL</p> |
| <p>ahmedabu98</p> |
| <p>Ahmed Abualsaud</p> |
| <p>al97</p> |
| <p>Aleksandr Dudko</p> |
| <p>Alexey Romanenko</p> |
| <p>Anand Inguva</p> |
| <p>Andrey Devyatkin</p> |
| <p>Anton Shalkovich</p> |
| <p>ArjunGHUB</p> |
| <p>Bjorn Pedersen</p> |
| <p>BjornPrime</p> |
| <p>Brett Morgan</p> |
| <p>Bruno Volpato</p> |
| <p>Buqian Zheng</p> |
| <p>Burke Davison</p> |
| <p>Byron Ellis</p> |
| <p>bzablocki</p> |
| <p>case-k</p> |
| <p>Celeste Zeng</p> |
| <p>Chamikara Jayalath</p> |
| <p>Clay Johnson</p> |
| <p>Connor Brett</p> |
| <p>Damon</p> |
| <p>Damon Douglas</p> |
| <p>Dan Hansen</p> |
| <p>Danny McCormick</p> |
| <p>Darkhan Nausharipov</p> |
| <p>Dip Patel</p> |
| <p>Dmytro Sadovnychyi</p> |
| <p>Florent Biville</p> |
| <p>Gabriel Lacroix</p> |
| <p>Hai Joey Tran</p> |
| <p>Hong Liang Teoh</p> |
| <p>Jack McCluskey</p> |
| <p>James Fricker</p> |
| <p>Jeff Kinard</p> |
| <p>Jeff Zhang</p> |
| <p>Jing</p> |
| <p>johnjcasey</p> |
| <p>jon esperanza</p> |
| <p>Josef Šimánek</p> |
| <p>Kenneth Knowles</p> |
| <p>Laksh</p> |
| <p>Liam Miller-Cushon</p> |
| <p>liferoad</p> |
| <p>magicgoody</p> |
| <p>Mahmud Ridwan</p> |
| <p>Manav Garg</p> |
| <p>Marco Vela</p> |
| <p>martin trieu</p> |
| <p>Mattie Fu</p> |
| <p>Michel Davit</p> |
| <p>Moritz Mack</p> |
| <p>mosche</p> |
| <p>Peter Sobot</p> |
| <p>Pranav Bhandari</p> |
| <p>Reeba Qureshi</p> |
| <p>Reuven Lax</p> |
| <p>Ritesh Ghorse</p> |
| <p>Robert Bradshaw</p> |
| <p>Robert Burke</p> |
| <p>RyuSA</p> |
| <p>Saba Sathya</p> |
| <p>Sam Whittle</p> |
| <p>Steven Niemitz</p> |
| <p>Steven van Rossum</p> |
| <p>Svetak Sundhar</p> |
| <p>Tony Tang</p> |
| <p>Valentyn Tymofieiev</p> |
| <p>Vitaly Terentyev</p> |
| <p>Vlado Djerek</p> |
| <p>Yichi Zhang</p> |
| <p>Yi Hu</p> |
| <p>Zechen Jiang</p></description></item><item><title>Blog: Apache Beam 2.49.0</title><link>/blog/beam-2.49.0/</link><pubDate>Mon, 17 Jul 2023 09:00:00 -0400</pubDate><guid>/blog/beam-2.49.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.49.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2490-2023-07-17">download page</a> for this release.</p> |
| <p>For more information on changes in 2.49.0, check out the <a href="https://github.com/apache/beam/milestone/13">detailed release notes</a>.</p> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Support for Bigtable Change Streams added in Java <code>BigtableIO.ReadChangeStream</code> (<a href="https://github.com/apache/beam/issues/27183">#27183</a>).</li> |
| <li>Added Bigtable Read and Write cross-language transforms to Python SDK ((<a href="https://github.com/apache/beam/issues/26593">#26593</a>), (<a href="https://github.com/apache/beam/issues/27146">#27146</a>)).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Allow prebuilding large images when using <code>--prebuild_sdk_container_engine=cloud_build</code>, like images depending on <code>tensorflow</code> or <code>torch</code> (<a href="https://github.com/apache/beam/pull/27023">#27023</a>).</li> |
| <li>Disabled <code>pip</code> cache when installing packages on the workers. This reduces the size of prebuilt Python container images (<a href="https://github.com/apache/beam/pull/27035">#27035</a>).</li> |
| <li>Select dedicated avro datum reader and writer (Java) (<a href="https://github.com/apache/beam/issues/18874">#18874</a>).</li> |
| <li>Timer API for the Go SDK (Go) (<a href="https://github.com/apache/beam/issues/22737">#22737</a>).</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>Remove Python 3.7 support. (<a href="https://github.com/apache/beam/issues/26447">#26447</a>)</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed KinesisIO <code>NullPointerException</code> when a progress check is made before the reader is started (IO) (<a href="https://github.com/apache/beam/issues/23868">#23868</a>)</li> |
| </ul> |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li>Long-running Python pipelines might experience a memory leak: <a href="https://github.com/apache/beam/issues/28246">#28246</a>.</li> |
| <li>Python SDK&rsquo;s cross-language Bigtable sink mishandles records that don&rsquo;t have an explicit timestamp set: <a href="https://github.com/apache/beam/issues/28632">#28632</a>. To avoid this issue, set explicit timestamps for all records before writing to Bigtable.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.49.0 release. Thank you to all contributors!</p> |
| <p>Abzal Tuganbay</p> |
| <p>AdalbertMemSQL</p> |
| <p>Ahmed Abualsaud</p> |
| <p>Ahmet Altay</p> |
| <p>Alan Zhang</p> |
| <p>Alexey Romanenko</p> |
| <p>Anand Inguva</p> |
| <p>Andrei Gurau</p> |
| <p>Arwin Tio</p> |
| <p>Bartosz Zablocki</p> |
| <p>Bruno Volpato</p> |
| <p>Burke Davison</p> |
| <p>Byron Ellis</p> |
| <p>Chamikara Jayalath</p> |
| <p>Charles Rothrock</p> |
| <p>Chris Gavin</p> |
| <p>Claire McGinty</p> |
| <p>Clay Johnson</p> |
| <p>Damon</p> |
| <p>Daniel Dopierała</p> |
| <p>Danny McCormick</p> |
| <p>Darkhan Nausharipov</p> |
| <p>David Cavazos</p> |
| <p>Dip Patel</p> |
| <p>Dmitry Repin</p> |
| <p>Gavin McDonald</p> |
| <p>Jack Dingilian</p> |
| <p>Jack McCluskey</p> |
| <p>James Fricker</p> |
| <p>Jan Lukavský</p> |
| <p>Jasper Van den Bossche</p> |
| <p>John Casey</p> |
| <p>John Gill</p> |
| <p>Joseph Crowley</p> |
| <p>Kanishk Karanawat</p> |
| <p>Katie Liu</p> |
| <p>Kenneth Knowles</p> |
| <p>Kyle Galloway</p> |
| <p>Liam Miller-Cushon</p> |
| <p>MakarkinSAkvelon</p> |
| <p>Masato Nakamura</p> |
| <p>Mattie Fu</p> |
| <p>Michel Davit</p> |
| <p>Naireen Hussain</p> |
| <p>Nathaniel Young</p> |
| <p>Nelson Osacky</p> |
| <p>Nick Li</p> |
| <p>Oleh Borysevych</p> |
| <p>Pablo Estrada</p> |
| <p>Reeba Qureshi</p> |
| <p>Reuven Lax</p> |
| <p>Ritesh Ghorse</p> |
| <p>Robert Bradshaw</p> |
| <p>Robert Burke</p> |
| <p>Rouslan</p> |
| <p>Saadat Su</p> |
| <p>Sam Rohde</p> |
| <p>Sam Whittle</p> |
| <p>Sanil Jain</p> |
| <p>Shunping Huang</p> |
| <p>Smeet nagda</p> |
| <p>Svetak Sundhar</p> |
| <p>Timur Sultanov</p> |
| <p>Udi Meiri</p> |
| <p>Valentyn Tymofieiev</p> |
| <p>Vlado Djerek</p> |
| <p>WuA</p> |
| <p>XQ Hu</p> |
| <p>Xianhua Liu</p> |
| <p>Xinyu Liu</p> |
| <p>Yi Hu</p> |
| <p>Zachary Houfek</p> |
| <p>alexeyinkin</p> |
| <p>bigduu</p> |
| <p>bullet03</p> |
| <p>bzablocki</p> |
| <p>jonathan-lemos</p> |
| <p>jubebo</p> |
| <p>magicgoody</p> |
| <p>ruslan-ikhsan</p> |
| <p>sultanalieva-s</p> |
| <p>vitaly.terentyev</p></description></item><item><title>Blog: Apache Beam 2.48.0</title><link>/blog/beam-2.48.0/</link><pubDate>Wed, 31 May 2023 11:30:00 -0400</pubDate><guid>/blog/beam-2.48.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.48.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2480-2023-05-31">download page</a> for this release.</p> |
| <p>For more information on changes in 2.48.0, check out the <a href="https://github.com/apache/beam/milestone/12">detailed release notes</a>.</p> |
| <p><strong>Note: The release tag for Go SDK for this release is sdks/v2.48.2 instead of sdks/v2.48.0 because of incorrect commit attached to the release tag sdks/v2.48.0.</strong></p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>&ldquo;Experimental&rdquo; annotation cleanup: the annotation and concept have been removed from Beam to avoid |
| the misperception of code as &ldquo;not ready&rdquo;. Any proposed breaking changes will be subject to |
| case-by-case pro/con decision making (and generally avoided) rather than using the &ldquo;Experimental&rdquo; |
| to allow them.</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Added rename for GCS and copy for local filesystem (Go) (<a href="https://github.com/apache/beam/issues/26064">#25779</a>).</li> |
| <li>Added support for enhanced fan-out in KinesisIO.Read (Java) (<a href="https://github.com/apache/beam/issues/19967">#19967</a>). |
| <ul> |
| <li>This change is not compatible with Flink savepoints created by Beam 2.46.0 applications which had KinesisIO sources.</li> |
| </ul> |
| </li> |
| <li>Added textio.ReadWithFilename transform (Go) (<a href="https://github.com/apache/beam/issues/25812">#25812</a>).</li> |
| <li>Added fileio.MatchContinuously transform (Go) (<a href="https://github.com/apache/beam/issues/26186">#26186</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Allow passing service name for google-cloud-profiler (Python) (<a href="https://github.com/apache/beam/issues/26280">#26280</a>).</li> |
| <li>Dead letter queue support added to RunInference in Python (<a href="https://github.com/apache/beam/issues/24209">#24209</a>).</li> |
| <li>Support added for defining pre/postprocessing operations on the RunInference transform (<a href="https://github.com/apache/beam/issues/26308">#26308</a>)</li> |
| <li>Adds a Docker Compose based transform service that can be used to discover and use portable Beam transforms (<a href="https://github.com/apache/beam/pull/26023">#26023</a>).</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>Passing a tag into MultiProcessShared is now required in the Python SDK (<a href="https://github.com/apache/beam/issues/26168">#26168</a>).</li> |
| <li>CloudDebuggerOptions is removed (deprecated in Beam v2.47.0) for Dataflow runner as the Google Cloud Debugger service is <a href="https://cloud.google.com/debugger/docs/deprecations">shutting down</a>. (Java) (<a href="https://github.com/apache/beam/issues/25959">#25959</a>).</li> |
| <li>AWS 2 client providers (deprecated in Beam <a href="#2380---2022-04-20">v2.38.0</a>) are finally removed (<a href="https://github.com/apache/beam/issues/26681">#26681</a>).</li> |
| <li>AWS 2 SnsIO.writeAsync (deprecated in Beam v2.37.0 due to risk of data loss) was finally removed (<a href="https://github.com/apache/beam/issues/26710">#26710</a>).</li> |
| <li>AWS 2 coders (deprecated in Beam v2.43.0 when adding Schema support for AWS Sdk Pojos) are finally removed (<a href="https://github.com/apache/beam/issues/23315">#23315</a>).</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed Java bootloader failing with Too Long Args due to long classpaths, with a pathing jar. (Java) (<a href="https://github.com/apache/beam/issues/25582">#25582</a>).</li> |
| </ul> |
| <h2 id="known-issues">Known Issues</h2> |
| <ul> |
| <li>PubsubIO writes will throw <em>SizeLimitExceededException</em> for any message above 100 bytes, when used in batch (bounded) mode. (Java) (<a href="https://github.com/apache/beam/issues/27000">#27000</a>).</li> |
| <li>Long-running Python pipelines might experience a memory leak: <a href="https://github.com/apache/beam/issues/28246">#28246</a>.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.48.0 release. Thank you to all contributors!</p> |
| <p>Abzal Tuganbay</p> |
| <p>Ahmed Abualsaud</p> |
| <p>Alexey Romanenko</p> |
| <p>Anand Inguva</p> |
| <p>Andrei Gurau</p> |
| <p>Andrey Devyatkin</p> |
| <p>Balázs Németh</p> |
| <p>Bazyli Polednia</p> |
| <p>Bruno Volpato</p> |
| <p>Chamikara Jayalath</p> |
| <p>Clay Johnson</p> |
| <p>Damon</p> |
| <p>Daniel Arn</p> |
| <p>Danny McCormick</p> |
| <p>Darkhan Nausharipov</p> |
| <p>Dip Patel</p> |
| <p>Dmitry Repin</p> |
| <p>George Novitskiy</p> |
| <p>Israel Herraiz</p> |
| <p>Jack Dingilian</p> |
| <p>Jack McCluskey</p> |
| <p>Jan Lukavský</p> |
| <p>Jasper Van den Bossche</p> |
| <p>Jeff Zhang</p> |
| <p>Jeremy Edwards</p> |
| <p>Johanna Öjeling</p> |
| <p>John Casey</p> |
| <p>Katie Liu</p> |
| <p>Kenneth Knowles</p> |
| <p>Kerry Donny-Clark</p> |
| <p>Kuba Rauch</p> |
| <p>Liam Miller-Cushon</p> |
| <p>MakarkinSAkvelon</p> |
| <p>Mattie Fu</p> |
| <p>Michel Davit</p> |
| <p>Moritz Mack</p> |
| <p>Nick Li</p> |
| <p>Oleh Borysevych</p> |
| <p>Pablo Estrada</p> |
| <p>Pranav Bhandari</p> |
| <p>Pranjal Joshi</p> |
| <p>Rebecca Szper</p> |
| <p>Reuven Lax</p> |
| <p>Ritesh Ghorse</p> |
| <p>Robert Bradshaw</p> |
| <p>Robert Burke</p> |
| <p>Rouslan</p> |
| <p>RuiLong J</p> |
| <p>RyujiTamaki</p> |
| <p>Sam Whittle</p> |
| <p>Sanil Jain</p> |
| <p>Svetak Sundhar</p> |
| <p>Timur Sultanov</p> |
| <p>Tony Tang</p> |
| <p>Udi Meiri</p> |
| <p>Valentyn Tymofieiev</p> |
| <p>Vishal Bhise</p> |
| <p>Vitaly Terentyev</p> |
| <p>Xinyu Liu</p> |
| <p>Yi Hu</p> |
| <p>bullet03</p> |
| <p>darshan-sj</p> |
| <p>kellen</p> |
| <p>liferoad</p> |
| <p>mokamoka03210120</p> |
| <p>psolomin</p></description></item><item><title>Blog: Apache Beam 2.47.0</title><link>/blog/beam-2.47.0/</link><pubDate>Wed, 10 May 2023 12:00:00 -0500</pubDate><guid>/blog/beam-2.47.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.47.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2470-2023-05-10">download page</a> for this release.</p> |
| <p>For more information on changes in 2.47.0, check out the <a href="https://github.com/apache/beam/milestone/10">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Apache Beam adds Python 3.11 support (<a href="https://github.com/apache/beam/issues/23848">#23848</a>).</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>BigQuery Storage Write API is now available in Python SDK via cross-language (<a href="https://github.com/apache/beam/issues/21961">#21961</a>).</li> |
| <li>Added HbaseIO support for writing RowMutations (ordered by rowkey) to Hbase (Java) (<a href="https://github.com/apache/beam/issues/25830">#25830</a>).</li> |
| <li>Added fileio transforms MatchFiles, MatchAll and ReadMatches (Go) (<a href="https://github.com/apache/beam/issues/25779">#25779</a>).</li> |
| <li>Add integration test for JmsIO + fix issue with multiple connections (Java) (<a href="https://github.com/apache/beam/issues/25887">#25887</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>The Flink runner now supports Flink 1.16.x (<a href="https://github.com/apache/beam/issues/25046">#25046</a>).</li> |
| <li>Schema&rsquo;d PTransforms can now be directly applied to Beam dataframes just like PCollections. |
| (Note that when doing multiple operations, it may be more efficient to explicitly chain the operations |
| like <code>df | (Transform1 | Transform2 | ...)</code> to avoid excessive conversions.)</li> |
| <li>The Go SDK adds new transforms periodic.Impulse and periodic.Sequence that extends support |
| for slowly updating side input patterns. (<a href="https://github.com/apache/beam/issues/23106">#23106</a>)</li> |
| <li>Several Google client libraries in Python SDK dependency chain were updated to latest available major versions. (<a href="https://github.com/apache/beam/pull/24599">#24599</a>)</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>If a main session fails to load, the pipeline will now fail at worker startup. (<a href="https://github.com/apache/beam/issues/25401">#25401</a>).</li> |
| <li>Python pipeline options will now ignore unparsed command line flags prefixed with a single dash. (<a href="https://github.com/apache/beam/issues/25943">#25943</a>).</li> |
| <li>The SmallestPerKey combiner now requires keyword-only arguments for specifying optional parameters, such as <code>key</code> and <code>reverse</code>. (<a href="https://github.com/apache/beam/issues/25888">#25888</a>).</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>Cloud Debugger support and its pipeline options are deprecated and will be removed in the next Beam version, |
| in response to the Google Cloud Debugger service <a href="https://cloud.google.com/debugger/docs/deprecations">turning down</a>. (Java) (<a href="https://github.com/apache/beam/issues/25959">#25959</a>).</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>BigQuery sink in STORAGE_WRITE_API mode in batch pipelines might result in data consistency issues during the handling of other unrelated transient errors for Beam SDKs 2.35.0 - 2.46.0 (inclusive). For more details see: <a href="https://github.com/apache/beam/issues/26521">https://github.com/apache/beam/issues/26521</a></li> |
| </ul> |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li>BigQueryIO Storage API write with autoUpdateSchema may cause data corruption for Beam SDKs 2.45.0 - 2.47.0 (inclusive) (<a href="https://github.com/apache/beam/issues/26789">#26789</a>)</li> |
| <li>Long-running Python pipelines might experience a memory leak: <a href="https://github.com/apache/beam/issues/28246">#28246</a>.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.47.0 release. Thank you to all contributors!</p> |
| <p>Ahmed Abualsaud</p> |
| <p>Ahmet Altay</p> |
| <p>Alexey Romanenko</p> |
| <p>Amir Fayazi</p> |
| <p>Amrane Ait Zeouay</p> |
| <p>Anand Inguva</p> |
| <p>Andrew Pilloud</p> |
| <p>Andrey Kot</p> |
| <p>Bjorn Pedersen</p> |
| <p>Bruno Volpato</p> |
| <p>Buqian Zheng</p> |
| <p>Chamikara Jayalath</p> |
| <p>ChangyuLi28</p> |
| <p>Damon</p> |
| <p>Danny McCormick</p> |
| <p>Dmitry Repin</p> |
| <p>George Ma</p> |
| <p>Jack Dingilian</p> |
| <p>Jack McCluskey</p> |
| <p>Jasper Van den Bossche</p> |
| <p>Jeremy Edwards</p> |
| <p>Jiangjie (Becket) Qin</p> |
| <p>Johanna Öjeling</p> |
| <p>Juta Staes</p> |
| <p>Kenneth Knowles</p> |
| <p>Kyle Weaver</p> |
| <p>Mattie Fu</p> |
| <p>Moritz Mack</p> |
| <p>Nick Li</p> |
| <p>Oleh Borysevych</p> |
| <p>Pablo Estrada</p> |
| <p>Rebecca Szper</p> |
| <p>Reuven Lax</p> |
| <p>Reza Rokni</p> |
| <p>Ritesh Ghorse</p> |
| <p>Robert Bradshaw</p> |
| <p>Robert Burke</p> |
| <p>Saadat Su</p> |
| <p>Saifuddin53</p> |
| <p>Sam Rohde</p> |
| <p>Shubham Krishna</p> |
| <p>Svetak Sundhar</p> |
| <p>Theodore Ni</p> |
| <p>Thomas Gaddy</p> |
| <p>Timur Sultanov</p> |
| <p>Udi Meiri</p> |
| <p>Valentyn Tymofieiev</p> |
| <p>Xinyu Liu</p> |
| <p>Yanan Hao</p> |
| <p>Yi Hu</p> |
| <p>Yuvi Panda</p> |
| <p>andres-vv</p> |
| <p>bochap</p> |
| <p>dannikay</p> |
| <p>darshan-sj</p> |
| <p>dependabot[bot]</p> |
| <p>harrisonlimh</p> |
| <p>hnnsgstfssn</p> |
| <p>jrmccluskey</p> |
| <p>liferoad</p> |
| <p>tvalentyn</p> |
| <p>xianhualiu</p> |
| <p>zhangskz</p></description></item><item><title>Blog: Apache Beam 2.46.0</title><link>/blog/beam-2.46.0/</link><pubDate>Fri, 10 Mar 2023 13:00:00 -0500</pubDate><guid>/blog/beam-2.46.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.46.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2460-2023-03-10">download page</a> for this release.</p> |
| <p>For more information on changes in 2.46.0, check out the <a href="https://github.com/apache/beam/milestone/9?closed=1">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Java SDK containers migrated to <a href="https://hub.docker.com/_/eclipse-temurin">Eclipse Temurin</a> |
| as a base. This change migrates away from the deprecated <a href="https://hub.docker.com/_/openjdk">OpenJDK</a> |
| container. Eclipse Temurin is currently based upon Ubuntu 22.04 while the OpenJDK |
| container was based upon Debian 11.</li> |
| <li>RunInference PTransform will accept model paths as SideInputs in Python SDK. (<a href="https://github.com/apache/beam/issues/24042">#24042</a>)</li> |
| <li>RunInference supports ONNX runtime in Python SDK (<a href="https://github.com/apache/beam/issues/22972">#22972</a>)</li> |
| <li>Tensorflow Model Handler for RunInference in Python SDK (<a href="https://github.com/apache/beam/issues/25366">#25366</a>)</li> |
| <li>Java SDK modules migrated to use <code>:sdks:java:extensions:avro</code> (<a href="https://github.com/apache/beam/issues/24748">#24748</a>)</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Added in JmsIO a retry policy for failed publications (Java) (<a href="https://github.com/apache/beam/issues/24971">#24971</a>).</li> |
| <li>Support for <code>LZMA</code> compression/decompression of text files added to the Python SDK (<a href="https://github.com/apache/beam/issues/25316">#25316</a>)</li> |
| <li>Added ReadFrom/WriteTo Csv/Json as top-level transforms to the Python SDK.</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Add UDF metrics support for Samza portable mode.</li> |
| <li>Option for SparkRunner to avoid the need of SDF output to fit in memory (<a href="https://github.com/apache/beam/issues/23852">#23852</a>). |
| This helps e.g. with ParquetIO reads. Turn the feature on by adding experiment <code>use_bounded_concurrent_output_for_sdf</code>.</li> |
| <li>Add <code>WatchFilePattern</code> transform, which can be used as a side input to the RunInference PTransfrom to watch for model updates using a file pattern. (<a href="https://github.com/apache/beam/issues/24042">#24042</a>)</li> |
| <li>Add support for loading TorchScript models with <code>PytorchModelHandler</code>. The TorchScript model path can be |
| passed to PytorchModelHandler using <code>torch_script_model_path=&lt;path_to_model&gt;</code>. (<a href="https://github.com/apache/beam/pull/25321">#25321</a>)</li> |
| <li>The Go SDK now requires Go 1.19 to build. (<a href="https://github.com/apache/beam/pull/25545">#25545</a>)</li> |
| <li>The Go SDK now has an initial native Go implementation of a portable Beam Runner called Prism. (<a href="https://github.com/apache/beam/pull/24789">#24789</a>) |
| <ul> |
| <li>For more details and current state see <a href="https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/runners/prism">https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/runners/prism</a>.</li> |
| </ul> |
| </li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>The deprecated SparkRunner for Spark 2 (see <a href="#2410---2022-08-23">2.41.0</a>) was removed (<a href="https://github.com/apache/beam/pull/25263">#25263</a>).</li> |
| <li>Python&rsquo;s BatchElements performs more aggressive batching in some cases, |
| capping at 10 second rather than 1 second batches by default and excluding |
| fixed cost in this computation to better handle cases where the fixed cost |
| is larger than a single second. To get the old behavior, one can pass |
| <code>target_batch_duration_secs_including_fixed_cost=1</code> to BatchElements.</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>Avro related classes are deprecated in module <code>beam-sdks-java-core</code> and will be eventually removed. Please, migrate to a new module <code>beam-sdks-java-extensions-avro</code> instead by importing the classes from <code>org.apache.beam.sdk.extensions.avro</code> package. |
| For the sake of migration simplicity, the relative package path and the whole class hierarchy of Avro related classes in new module is preserved the same as it was before. |
| For example, import <code>org.apache.beam.sdk.extensions.avro.coders.AvroCoder</code> class instead of<code>org.apache.beam.sdk.coders.AvroCoder</code>. (<a href="https://github.com/apache/beam/issues/24749">#24749</a>).</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.46.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay</p> |
| <p>Alan Zhang</p> |
| <p>Alexey Romanenko</p> |
| <p>Amrane Ait Zeouay</p> |
| <p>Anand Inguva</p> |
| <p>Andrew Pilloud</p> |
| <p>Brian Hulette</p> |
| <p>Bruno Volpato</p> |
| <p>Byron Ellis</p> |
| <p>Chamikara Jayalath</p> |
| <p>Damon</p> |
| <p>Danny McCormick</p> |
| <p>Darkhan Nausharipov</p> |
| <p>David Katz</p> |
| <p>Dmitry Repin</p> |
| <p>Doug Judd</p> |
| <p>Egbert van der Wal</p> |
| <p>Elizaveta Lomteva</p> |
| <p>Evan Galpin</p> |
| <p>Herman Mak</p> |
| <p>Jack McCluskey</p> |
| <p>Jan Lukavský</p> |
| <p>Johanna Öjeling</p> |
| <p>John Casey</p> |
| <p>Jozef Vilcek</p> |
| <p>Junhao Liu</p> |
| <p>Juta Staes</p> |
| <p>Katie Liu</p> |
| <p>Kiley Sok</p> |
| <p>Liam Miller-Cushon</p> |
| <p>Luke Cwik</p> |
| <p>Moritz Mack</p> |
| <p>Ning Kang</p> |
| <p>Oleh Borysevych</p> |
| <p>Pablo E</p> |
| <p>Pablo Estrada</p> |
| <p>Reuven Lax</p> |
| <p>Ritesh Ghorse</p> |
| <p>Robert Bradshaw</p> |
| <p>Robert Burke</p> |
| <p>Ruslan Altynnikov</p> |
| <p>Ryan Zhang</p> |
| <p>Sam Rohde</p> |
| <p>Sam Whittle</p> |
| <p>Sam sam</p> |
| <p>Sergei Lilichenko</p> |
| <p>Shivam</p> |
| <p>Shubham Krishna</p> |
| <p>Theodore Ni</p> |
| <p>Timur Sultanov</p> |
| <p>Tony Tang</p> |
| <p>Vachan</p> |
| <p>Veronica Wasson</p> |
| <p>Vincent Devillers</p> |
| <p>Vitaly Terentyev</p> |
| <p>William Ross Morrow</p> |
| <p>Xinyu Liu</p> |
| <p>Yi Hu</p> |
| <p>ZhengLin Li</p> |
| <p>Ziqi Ma</p> |
| <p>ahmedabu98</p> |
| <p>alexeyinkin</p> |
| <p>aliftadvantage</p> |
| <p>bullet03</p> |
| <p>dannikay</p> |
| <p>darshan-sj</p> |
| <p>dependabot[bot]</p> |
| <p>johnjcasey</p> |
| <p>kamrankoupayi</p> |
| <p>kileys</p> |
| <p>liferoad</p> |
| <p>nancyxu123</p> |
| <p>nickuncaged1201</p> |
| <p>pablo rodriguez defino</p> |
| <p>tvalentyn</p> |
| <p>xqhu</p></description></item><item><title>Blog: Apache Beam 2.45.0</title><link>/blog/beam-2.45.0/</link><pubDate>Wed, 15 Feb 2023 09:00:00 -0700</pubDate><guid>/blog/beam-2.45.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.45.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2430-2023-01-13">download page</a> for this release.</p> |
| <p>For more information on changes in 2.45.0, check out the <a href="https://github.com/apache/beam/milestone/8?closed=1">detailed release notes</a>.</p> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>MongoDB IO connector added (Go) (<a href="https://github.com/apache/beam/issues/24575">#24575</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>RunInference Wrapper with Sklearn Model Handler support added in Go SDK (<a href="https://github.com/apache/beam/issues/23382">#24497</a>).</li> |
| <li>Adding override of allowed TLS algorithms (Java), now maintaining the disabled/legacy algorithms |
| present in 2.43.0 (up to 1.8.0_342, 11.0.16, 17.0.2 for respective Java versions). This is accompanied |
| by an explicit re-enabling of TLSv1 and TLSv1.1 for Java 8 and Java 11.</li> |
| <li>Add UDF metrics support for Samza portable mode.</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>Portable Java pipelines, Go pipelines, Python streaming pipelines, and portable Python batch |
| pipelines on Dataflow are required to use Runner V2. The <code>disable_runner_v2</code>, |
| <code>disable_runner_v2_until_2023</code>, <code>disable_prime_runner_v2</code> experiments will raise an error during |
| pipeline construction. You can no longer specify the Dataflow worker jar override. Note that |
| non-portable Java jobs and non-portable Python batch jobs are not impacted. (<a href="https://github.com/apache/beam/issues/24515">#24515</a>).</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Avoids Cassandra syntax error when user-defined query has no where clause in it (Java) (<a href="https://github.com/apache/beam/issues/24829">#24829</a>).</li> |
| <li>Fixed JDBC connection failures (Java) during handshake due to deprecated TLSv1(.1) protocol for the JDK. (<a href="https://github.com/apache/beam/issues/24623">#24623</a>)</li> |
| <li>Fixed Python BigQuery Batch Load write may truncate valid data when deposition sets to WRITE_TRUNCATE and incoming data is large (Python) (<a href="https://github.com/apache/beam/issues/24535">#24623</a>).</li> |
| <li>Fixed Kafka watermark issue with sparse data on many partitions (<a href="https://github.com/apache/beam/pull/24205">#24205</a>)</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.45.0 release. Thank you to all contributors!</p> |
| <p>AdalbertMemSQL</p> |
| <p>Ahmed Abualsaud</p> |
| <p>Ahmet Altay</p> |
| <p>Alexey Romanenko</p> |
| <p>Anand Inguva</p> |
| <p>Andrea Nardelli</p> |
| <p>Andrei Gurau</p> |
| <p>Andrew Pilloud</p> |
| <p>Benjamin Gonzalez</p> |
| <p>BjornPrime</p> |
| <p>Brian Hulette</p> |
| <p>Bulat</p> |
| <p>Byron Ellis</p> |
| <p>Chamikara Jayalath</p> |
| <p>Charles Rothrock</p> |
| <p>Damon</p> |
| <p>Daniela Martín</p> |
| <p>Danny McCormick</p> |
| <p>Darkhan Nausharipov</p> |
| <p>Dejan Spasic</p> |
| <p>Diego Gomez</p> |
| <p>Dmitry Repin</p> |
| <p>Doug Judd</p> |
| <p>Elias Segundo Antonio</p> |
| <p>Evan Galpin</p> |
| <p>Evgeny Antyshev</p> |
| <p>Fernando Morales</p> |
| <p>Jack McCluskey</p> |
| <p>Johanna Öjeling</p> |
| <p>John Casey</p> |
| <p>Junhao Liu</p> |
| <p>Kanishk Karanawat</p> |
| <p>Kenneth Knowles</p> |
| <p>Kiley Sok</p> |
| <p>Liam Miller-Cushon</p> |
| <p>Lucas Marques</p> |
| <p>Luke Cwik</p> |
| <p>MakarkinSAkvelon</p> |
| <p>Marco Robles</p> |
| <p>Mark Zitnik</p> |
| <p>Melanie</p> |
| <p>Moritz Mack</p> |
| <p>Ning Kang</p> |
| <p>Oleh Borysevych</p> |
| <p>Pablo Estrada</p> |
| <p>Philippe Moussalli</p> |
| <p>Piyush Sagar</p> |
| <p>Rebecca Szper</p> |
| <p>Reuven Lax</p> |
| <p>Rick Viscomi</p> |
| <p>Ritesh Ghorse</p> |
| <p>Robert Bradshaw</p> |
| <p>Robert Burke</p> |
| <p>Sam Whittle</p> |
| <p>Sergei Lilichenko</p> |
| <p>Seung Jin An</p> |
| <p>Shane Hansen</p> |
| <p>Sho Nakatani</p> |
| <p>Shunya Ueta</p> |
| <p>Siddharth Agrawal</p> |
| <p>Timur Sultanov</p> |
| <p>Veronica Wasson</p> |
| <p>Vitaly Terentyev</p> |
| <p>Xinbin Huang</p> |
| <p>Xinyu Liu</p> |
| <p>Xinyue Zhang</p> |
| <p>Yi Hu</p> |
| <p>ZhengLin Li</p> |
| <p>alexeyinkin</p> |
| <p>andoni-guzman</p> |
| <p>andthezhang</p> |
| <p>bullet03</p> |
| <p>camphillips22</p> |
| <p>gabihodoroaga</p> |
| <p>harrisonlimh</p> |
| <p>pablo rodriguez defino</p> |
| <p>ruslan-ikhsan</p> |
| <p>tvalentyn</p> |
| <p>yyy1000</p> |
| <p>zhengbuqian</p></description></item><item><title>Blog: Apache Beam 2.44.0</title><link>/blog/beam-2.44.0/</link><pubDate>Tue, 17 Jan 2023 09:00:00 -0700</pubDate><guid>/blog/beam-2.44.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.44.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2430-2023-01-13">download page</a> for this release.</p> |
| <p>For more information on changes in 2.44.0, check out the <a href="https://github.com/apache/beam/milestone/7?closed=1">detailed release notes</a>.</p> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Support for Bigtable sink (Write and WriteBatch) added (Go) (<a href="https://github.com/apache/beam/issues/23324">#23324</a>).</li> |
| <li>S3 implementation of the Beam filesystem (Go) (<a href="https://github.com/apache/beam/issues/23991">#23991</a>).</li> |
| <li>Support for SingleStoreDB source and sink added (Java) (<a href="https://github.com/apache/beam/issues/22617">#22617</a>).</li> |
| <li>Added support for DefaultAzureCredential authentication in Azure Filesystem (Python) (<a href="https://github.com/apache/beam/issues/24210">#24210</a>).</li> |
| <li>Added new CdapIO for CDAP Batch and Streaming Source/Sinks (Java) (<a href="https://github.com/apache/beam/issues/24961">#24961</a>).</li> |
| <li>Added new SparkReceiverIO for Spark Receivers 2.4.* (Java) (<a href="https://github.com/apache/beam/issues/24960">#24960</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Beam now provides a portable &ldquo;runner&rdquo; that can render pipeline graphs with |
| graphviz. See <code>python -m apache_beam.runners.render --help</code> for more details.</li> |
| <li>Local packages can now be used as dependencies in the requirements.txt file, rather |
| than requiring them to be passed separately via the <code>--extra_package</code> option |
| (Python) (<a href="https://github.com/apache/beam/pull/23684">#23684</a>).</li> |
| <li>Pipeline Resource Hints now supported via <code>--resource_hints</code> flag (Go) (<a href="https://github.com/apache/beam/pull/23990">#23990</a>).</li> |
| <li>Make Python SDK containers reusable on portable runners by installing dependencies to temporary venvs (<a href="https://issues.apache.org/jira/browse/BEAM-12792">BEAM-12792</a>).</li> |
| <li>RunInference model handlers now support the specification of a custom inference function in Python (<a href="https://github.com/apache/beam/issues/22572">#22572</a>)</li> |
| <li>Support for <code>map_windows</code> urn added to Go SDK (<a href="https://github.apache/beam/pull/24307">#24307</a>).</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li><code>ParquetIO.withSplit</code> was removed since splittable reading has been the default behavior since 2.35.0. The effect of |
| this change is to drop support for non-splittable reading (Java)(<a href="https://github.com/apache/beam/issues/23832">#23832</a>).</li> |
| <li><code>beam-sdks-java-extensions-google-cloud-platform-core</code> is no longer a |
| dependency of the Java SDK Harness. Some users of a portable runner (such as Dataflow Runner v2) |
| may have an undeclared dependency on this package (for example using GCS with |
| TextIO) and will now need to declare the dependency.</li> |
| <li><code>beam-sdks-java-core</code> is no longer a dependency of the Java SDK Harness. Users of a portable |
| runner (such as Dataflow Runner v2) will need to provide this package and its dependencies.</li> |
| <li>Slices now use the Beam Iterable Coder. This enables cross language use, but breaks pipeline updates |
| if a Slice type is used as a PCollection element or State API element. (Go)<a href="https://github.com/apache/beam/issues/24339">#24339</a></li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed JmsIO acknowledgment issue (Java) (<a href="https://github.com/apache/beam/issues/20814">#20814</a>)</li> |
| <li>Fixed Beam SQL CalciteUtils (Java) and Cross-language JdbcIO (Python) did not support JDBC CHAR/VARCHAR, BINARY/VARBINARY logical types (<a href="https://github.com/apache/beam/issues/23747">#23747</a>, <a href="https://github.com/apache/beam/issues/23526">#23526</a>).</li> |
| <li>Ensure iterated and emitted types are used with the generic register package are registered with the type and schema registries.(Go) (<a href="https://github.com/apache/beam/pull/23889">#23889</a>)</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.44.0 release. Thank you to all contributors!</p> |
| <p>Ahmed Abualsaud</p> |
| <p>Ahmet Altay</p> |
| <p>Alex Merose</p> |
| <p>Alexey Inkin</p> |
| <p>Alexey Romanenko</p> |
| <p>Anand Inguva</p> |
| <p>Andrei Gurau</p> |
| <p>Andrej Galad</p> |
| <p>Andrew Pilloud</p> |
| <p>Ayush Sharma</p> |
| <p>Benjamin Gonzalez</p> |
| <p>Bjorn Pedersen</p> |
| <p>Brian Hulette</p> |
| <p>Bruno Volpato</p> |
| <p>Bulat Safiullin</p> |
| <p>Chamikara Jayalath</p> |
| <p>Chris Gavin</p> |
| <p>Damon Douglas</p> |
| <p>Danielle Syse</p> |
| <p>Danny McCormick</p> |
| <p>Darkhan Nausharipov</p> |
| <p>David Cavazos</p> |
| <p>Dmitry Repin</p> |
| <p>Doug Judd</p> |
| <p>Elias Segundo Antonio</p> |
| <p>Evan Galpin</p> |
| <p>Evgeny Antyshev</p> |
| <p>Heejong Lee</p> |
| <p>Henrik Heggelund-Berg</p> |
| <p>Israel Herraiz</p> |
| <p>Jack McCluskey</p> |
| <p>Jan Lukavský</p> |
| <p>Janek Bevendorff</p> |
| <p>Johanna Öjeling</p> |
| <p>John J. Casey</p> |
| <p>Jozef Vilcek</p> |
| <p>Kanishk Karanawat</p> |
| <p>Kenneth Knowles</p> |
| <p>Kiley Sok</p> |
| <p>Laksh</p> |
| <p>Liam Miller-Cushon</p> |
| <p>Luke Cwik</p> |
| <p>MakarkinSAkvelon</p> |
| <p>Minbo Bae</p> |
| <p>Moritz Mack</p> |
| <p>Nancy Xu</p> |
| <p>Ning Kang</p> |
| <p>Nivaldo Tokuda</p> |
| <p>Oleh Borysevych</p> |
| <p>Pablo Estrada</p> |
| <p>Philippe Moussalli</p> |
| <p>Pranav Bhandari</p> |
| <p>Rebecca Szper</p> |
| <p>Reuven Lax</p> |
| <p>Rick Smit</p> |
| <p>Ritesh Ghorse</p> |
| <p>Robert Bradshaw</p> |
| <p>Robert Burke</p> |
| <p>Ryan Thompson</p> |
| <p>Sam Whittle</p> |
| <p>Sanil Jain</p> |
| <p>Scott Strong</p> |
| <p>Shubham Krishna</p> |
| <p>Steven van Rossum</p> |
| <p>Svetak Sundhar</p> |
| <p>Thiago Nunes</p> |
| <p>Tianyang Hu</p> |
| <p>Trevor Gevers</p> |
| <p>Valentyn Tymofieiev</p> |
| <p>Vitaly Terentyev</p> |
| <p>Vladislav Chunikhin</p> |
| <p>Xinyu Liu</p> |
| <p>Yi Hu</p> |
| <p>Yichi Zhang</p> |
| <p>AdalbertMemSQL</p> |
| <p>agvdndor</p> |
| <p>andremissaglia</p> |
| <p>arne-alex</p> |
| <p>bullet03</p> |
| <p>camphillips22</p> |
| <p>capthiron</p> |
| <p>creste</p> |
| <p>fab-jul</p> |
| <p>illoise</p> |
| <p>kn1kn1</p> |
| <p>nancyxu123</p> |
| <p>peridotml</p> |
| <p>shinannegans</p> |
| <p>smeet07</p></description></item><item><title>Blog: Apache Beam 2.43.0</title><link>/blog/beam-2.43.0/</link><pubDate>Thu, 17 Nov 2022 09:00:00 -0700</pubDate><guid>/blog/beam-2.43.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.43.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2430-2022-11-17">download page</a> for this release.</p> |
| <p>For more information on changes in 2.43.0, check out the <a href="https://github.com/apache/beam/milestone/5?closed=1">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Python 3.10 support in Apache Beam (<a href="https://github.com/apache/beam/issues/21458">#21458</a>).</li> |
| <li>An initial implementation of a runner that allows us to run Beam pipelines on Dask. Try it out and give us feedback! (Python) (<a href="https://github.com/apache/beam/issues/18962">#18962</a>).</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Decreased TextSource CPU utilization by 2.3x (Java) (<a href="https://github.com/apache/beam/issues/23193">#23193</a>).</li> |
| <li>Fixed bug when using SpannerIO with RuntimeValueProvider options (Java) (<a href="https://github.com/apache/beam/issues/22146">#22146</a>).</li> |
| <li>Fixed issue for unicode rendering on WriteToBigQuery (<a href="https://github.com/apache/beam/issues/22312">#22312</a>)</li> |
| <li>Remove obsolete variants of BigQuery Read and Write, always using Beam-native variant |
| (<a href="https://github.com/apache/beam/issues/23564">#23564</a> and <a href="https://github.com/apache/beam/issues/23559">#23559</a>).</li> |
| <li>Bumped google-cloud-spanner dependency version to 3.x for Python SDK (<a href="https://github.com/apache/beam/issues/21198">#21198</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Dataframe wrapper added in Go SDK via Cross-Language (with automatic expansion service). (Go) (<a href="https://github.com/apache/beam/issues/23384">#23384</a>).</li> |
| <li>Name all Java threads to aid in debugging (<a href="https://github.com/apache/beam/issues/23049">#23049</a>).</li> |
| <li>An initial implementation of a runner that allows us to run Beam pipelines on Dask. (Python) (<a href="https://github.com/apache/beam/issues/18962">#18962</a>).</li> |
| <li>Allow configuring GCP OAuth scopes via pipeline options. This unblocks usages of Beam IOs that require additional scopes. |
| For example, this feature makes it possible to access Google Drive backed tables in BigQuery (<a href="https://github.com/apache/beam/issues/23290">#23290</a>).</li> |
| <li>An example for using Python RunInference from Java (<a href="https://github.com/apache/beam/pull/23619">#23290</a>).</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>CoGroupByKey transform in Python SDK has changed the output typehint. The typehint component representing grouped values changed from List to Iterable, |
| which more accurately reflects the nature of the arbitrarily large output collection. <a href="https://github.com/apache/beam/issues/21556">#21556</a> Beam users may see an error on transforms downstream from CoGroupByKey. Users must change methods expecting a List to expect an Iterable going forward. See <a href="https://docs.google.com/document/d/1RIzm8-g-0CyVsPb6yasjwokJQFoKHG4NjRUcKHKINu0">document</a> for information and fixes.</li> |
| <li>The PortableRunner for Spark assumes Spark 3 as default Spark major version unless configured otherwise using <code>--spark_version</code>. |
| Spark 2 support is deprecated and will be removed soon (<a href="https://github.com/apache/beam/issues/23728">#23728</a>).</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed Python cross-language JDBC IO Connector cannot read or write rows containing Numeric/Decimal type values (<a href="https://github.com/apache/beam/issues/19817">#19817</a>).</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.43.0 release. Thank you to all contributors!</p> |
| <p>Ahmed Abualsaud |
| AlexZMLyu |
| Alexey Romanenko |
| Anand Inguva |
| Andrew Pilloud |
| Andy Ye |
| Arnout Engelen |
| Benjamin Gonzalez |
| Bharath Kumarasubramanian |
| BjornPrime |
| Brian Hulette |
| Bruno Volpato |
| Chamikara Jayalath |
| Colin Versteeg |
| Damon |
| Daniel Smilkov |
| Daniela Martín |
| Danny McCormick |
| Darkhan Nausharipov |
| David Huntsperger |
| Denis Pyshev |
| Dmitry Repin |
| Evan Galpin |
| Evgeny Antyshev |
| Fernando Morales |
| Geddy05 |
| Harshit Mehrotra |
| Iñigo San Jose Visiers |
| Ismaël Mejía |
| Israel Herraiz |
| Jan Lukavský |
| Juta Staes |
| Kanishk Karanawat |
| Kenneth Knowles |
| KevinGG |
| Kiley Sok |
| Liam Miller-Cushon |
| Luke Cwik |
| Mc |
| Melissa Pashniak |
| Moritz Mack |
| Ning Kang |
| Pablo Estrada |
| Philippe Moussalli |
| Pranav Bhandari |
| Rebecca Szper |
| Reuven Lax |
| Ritesh Ghorse |
| Robert Bradshaw |
| Robert Burke |
| Ryan Thompson |
| Ryohei Nagao |
| Sam Rohde |
| Sam Whittle |
| Sanil Jain |
| Seunghwan Hong |
| Shane Hansen |
| Shubham Krishna |
| Shunsuke Otani |
| Steve Niemitz |
| Steven van Rossum |
| Svetak Sundhar |
| Thiago Nunes |
| Toran Sahu |
| Veronica Wasson |
| Vitaly Terentyev |
| Vladislav Chunikhin |
| Xinyu Liu |
| Yi Hu |
| Yixiao Shen |
| alexeyinkin |
| arne-alex |
| azhurkevich |
| bulat safiullin |
| bullet03 |
| coldWater |
| dpcollins-google |
| egalpin |
| johnjcasey |
| liferoad |
| rvballada |
| shaojwu |
| tvalentyn</p></description></item><item><title>Blog: Apache Beam 2.42.0</title><link>/blog/beam-2.42.0/</link><pubDate>Mon, 17 Oct 2022 09:00:00 -0700</pubDate><guid>/blog/beam-2.42.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.42.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2420-2022-10-17">download page</a> for this release.</p> |
| <p>For more information on changes in 2.42.0, check out the <a href="https://github.com/apache/beam/milestone/4?closed=1">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Added support for stateful DoFns to the Go SDK.</li> |
| <li>Added support for <a href="/documentation/programming-guide/#batched-dofns">Batched |
| DoFns</a> |
| to the Python SDK.</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Added support for Zstd compression to the Python SDK.</li> |
| <li>Added support for Google Cloud Profiler to the Go SDK.</li> |
| <li>Added support for stateful DoFns to the Go SDK.</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>The Go SDK&rsquo;s Row Coder now uses a different single-precision float encoding for float32 types to match Java&rsquo;s behavior (<a href="https://github.com/apache/beam/issues/22629">#22629</a>).</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed Python cross-language JDBC IO Connector cannot read or write rows containing Timestamp type values <a href="https://github.com/apache/beam/issues/19817">19817</a>.</li> |
| </ul> |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li>Go SDK doesn&rsquo;t yet support Slowly Changing Side Input pattern (<a href="https://github.com/apache/beam/issues/23106">#23106</a>)</li> |
| <li>See a full list of open <a href="https://github.com/apache/beam/milestone/4">issues that affect</a> this version.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.42.0 release. Thank you to all contributors!</p> |
| <p>Abirdcfly |
| Ahmed Abualsaud |
| Alexander Zhuravlev |
| Alexey Inkin |
| Alexey Romanenko |
| Anand Inguva |
| Andrej Galad |
| Andrew Pilloud |
| Andy Ye |
| Balázs Németh |
| Brian Hulette |
| Bruno Volpato |
| bulat safiullin |
| bullet03 |
| Chamikara Jayalath |
| ChangyuLi28 |
| Clément Guillaume |
| Damon |
| Danny McCormick |
| Darkhan Nausharipov |
| David Huntsperger |
| dpcollins-google |
| Evgeny Antyshev |
| grufino |
| Heejong Lee |
| Ismaël Mejía |
| Jack McCluskey |
| johnjcasey |
| Jonathan Shen |
| Kenneth Knowles |
| Ke Wu |
| Kiley Sok |
| Liam Miller-Cushon |
| liferoad |
| Lucas Nogueira |
| Luke Cwik |
| MakarkinSAkvelon |
| Manit Gupta |
| masahitojp |
| Michael Hu |
| Michel Davit |
| Moritz Mack |
| Naireen Hussain |
| nancyxu123 |
| Nikhil Nadig |
| oborysevych |
| Pablo Estrada |
| Pranav Bhandari |
| Rajat Bhatta |
| Rebecca Szper |
| Reuven Lax |
| Ritesh Ghorse |
| Robert Bradshaw |
| Robert Burke |
| Ryan Thompson |
| Sam Whittle |
| Sergey Pronin |
| Shivam |
| Shunsuke Otani |
| Shunya Ueta |
| Steven Niemitz |
| Stuart |
| Svetak Sundhar |
| Valentyn Tymofieiev |
| Vitaly Terentyev |
| Vlad |
| Vladislav Chunikhin |
| Yichi Zhang |
| Yi Hu |
| Yixiao Shen</p></description></item><item><title>Blog: Apache Beam 2.41.0</title><link>/blog/beam-2.41.0/</link><pubDate>Tue, 23 Aug 2022 09:00:00 -0700</pubDate><guid>/blog/beam-2.41.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.41.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2410-2022-08-23">download page</a> for this release.</p> |
| <p>For more information on changes in 2.41.0, check out the <a href="https://github.com/apache/beam/milestone/3?closed=1">detailed release notes</a>.</p> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Projection Pushdown optimizer is now on by default for streaming, matching the behavior of batch pipelines since 2.38.0. If you encounter a bug with the optimizer, please file an issue and disable the optimizer using pipeline option <code>--experiments=disable_projection_pushdown</code>.</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Previously available in Java sdk, Python sdk now also supports logging level overrides per module. (<a href="https://github.com/apache/beam/issues/18222">#18222</a>).</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>Projection Pushdown optimizer may break Dataflow upgrade compatibility for optimized pipelines when it removes unused fields. If you need to upgrade and encounter a compatibility issue, disable the optimizer using pipeline option <code>--experiments=disable_projection_pushdown</code>.</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>Support for Spark 2.4.x is deprecated and will be dropped with the release of Beam 2.44.0 or soon after (Spark runner) (<a href="https://github.com/apache/beam/issues/22094">#22094</a>).</li> |
| <li>The modules <a href="https://github.com/apache/beam/tree/master/sdks/java/io/amazon-web-services">amazon-web-services</a> and |
| <a href="https://github.com/apache/beam/tree/master/sdks/java/io/kinesis">kinesis</a> for AWS Java SDK v1 are deprecated |
| in favor of <a href="https://github.com/apache/beam/tree/master/sdks/java/io/amazon-web-services2">amazon-web-services2</a> |
| and will be eventually removed after a few Beam releases (Java) (<a href="https://github.com/apache/beam/issues/21249">#21249</a>).</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed a condition where retrying queries would yield an incorrect cursor in the Java SDK Firestore Connector (<a href="https://github.com/apache/beam/issues/22089">#22089</a>).</li> |
| <li>Fixed plumbing allowed lateness in Go SDK. It was ignoring the user set value earlier and always used to set to 0. (<a href="https://github.com/apache/beam/issues/22474">#22474</a>).</li> |
| </ul> |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li>See a full list of open <a href="https://github.com/apache/beam/milestone/3">issues that affect</a> this version.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.41.0 release. Thank you to all contributors!</p> |
| <p>Ahmed Abualsaud |
| Ahmet Altay |
| akashorabek |
| Alexey Inkin |
| Alexey Romanenko |
| Anand Inguva |
| andoni-guzman |
| Andrew Pilloud |
| Andrey |
| Andy Ye |
| Balázs Németh |
| Benjamin Gonzalez |
| BjornPrime |
| Brian Hulette |
| bulat safiullin |
| bullet03 |
| Byron Ellis |
| Chamikara Jayalath |
| Damon Douglas |
| Daniel Oliveira |
| Daniel Thevessen |
| Danny McCormick |
| David Huntsperger |
| Dheeraj Gharde |
| Etienne Chauchot |
| Evan Galpin |
| Fernando Morales |
| Heejong Lee |
| Jack McCluskey |
| johnjcasey |
| Kenneth Knowles |
| Ke Wu |
| Kiley Sok |
| Liam Miller-Cushon |
| Lucas Nogueira |
| Luke Cwik |
| MakarkinSAkvelon |
| Manu Zhang |
| Minbo Bae |
| Moritz Mack |
| Naireen Hussain |
| Ning Kang |
| Oleh Borysevych |
| Pablo Estrada |
| pablo rodriguez defino |
| Pranav Bhandari |
| Rebecca Szper |
| Red Daly |
| Reuven Lax |
| Ritesh Ghorse |
| Robert Bradshaw |
| Robert Burke |
| Ryan Thompson |
| Sam Whittle |
| Steven Niemitz |
| Valentyn Tymofieiev |
| Vincent Marquez |
| Vitaly Terentyev |
| Vlad |
| Vladislav Chunikhin |
| Yichi Zhang |
| Yi Hu |
| yirutang |
| Yixiao Shen |
| Yu Feng</p></description></item><item><title>Blog: Apache Beam 2.40.0</title><link>/blog/beam-2.40.0/</link><pubDate>Sat, 25 Jun 2022 09:00:00 -0700</pubDate><guid>/blog/beam-2.40.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.40.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2400-2022-06-25">download page</a> for this |
| release.</p> |
| <p>For more information on changes in 2.40.0 check out the <a href="https://github.com/apache/beam/releases/tag/v2.40.0">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Added <a href="https://s.apache.org/inference-sklearn-pytorch">RunInference</a> API, a framework agnostic transform for inference. With this release, PyTorch and Scikit-learn are supported by the transform. |
| See also example at apache_beam/examples/inference/pytorch_image_classification.py</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Upgraded to Hive 3.1.3 for HCatalogIO. Users can still provide their own version of Hive. (Java) (<a href="https://github.com/apache/beam/issues/19554">Issue-19554</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Go SDK users can now use generic registration functions to optimize their DoFn execution. (<a href="https://issues.apache.org/jira/browse/BEAM-14347">BEAM-14347</a>)</li> |
| <li>Go SDK users may now write self-checkpointing Splittable DoFns to read from streaming sources. (<a href="https://issues.apache.org/jira/browse/BEAM-11104">BEAM-11104</a>)</li> |
| <li>Go SDK textio Reads have been moved to Splittable DoFns exclusively. (<a href="https://issues.apache.org/jira/browse/BEAM-14489">BEAM-14489</a>)</li> |
| <li>Pipeline drain support added for Go SDK has now been tested. (<a href="https://issues.apache.org/jira/browse/BEAM-11106">BEAM-11106</a>)</li> |
| <li>Go SDK users can now see heap usage, sideinput cache stats, and active process bundle stats in Worker Status. (<a href="https://issues.apache.org/jira/browse/BEAM-13829">BEAM-13829</a>)</li> |
| <li>The serialization (pickling) library for Python is dill==0.3.1.1 (<a href="https://issues.apache.org/jira/browse/BEAM-11167">BEAM-11167</a>)</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>The Go Sdk now requires a minimum version of 1.18 in order to support generics (<a href="https://issues.apache.org/jira/browse/BEAM-14347">BEAM-14347</a>).</li> |
| <li>synthetic.SourceConfig field types have changed to int64 from int for better compatibility with Flink&rsquo;s use of Logical types in Schemas (Go) (<a href="https://issues.apache.org/jira/browse/BEAM-14173">BEAM-14173</a>)</li> |
| <li>Default coder updated to compress sources used with <code>BoundedSourceAsSDFWrapperFn</code> and <code>UnboundedSourceAsSDFWrapper</code>.</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed Java expansion service to allow specific files to stage (<a href="https://issues.apache.org/jira/browse/BEAM-14160">BEAM-14160</a>).</li> |
| <li>Fixed Elasticsearch connection when using both ssl and username/password (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-14000">BEAM-14000</a>)</li> |
| </ul> |
| <h2 id="known-issues">Known Issues</h2> |
| <ul> |
| <li>Python&rsquo;s <code>beam.FlatMap</code> will raise <code>AttributeError: 'builtin_function_or_method' object has no attribute '__func__'</code> when |
| constructed with some |
| <a href="https://docs.python.org/3/library/functions.html">built-ins</a>, like <code>sum</code> |
| and <code>len</code> (<a href="https://github.com/apache/beam/issues/22091">#22091</a>).</li> |
| <li>Java&rsquo;s <code>BigQueryIO.Write</code> can have an exception where it attempts to output a timestamp beyond the max timestamp range |
| <code>Cannot output with timestamp 294247-01-10T04:00:54.776Z. Output timestamps must be no earlier than the timestamp of the current input or timer (294247-01-10T04:00:54.776Z) minus the allowed skew (0 milliseconds) and no later than 294247-01-10T04:00:54.775Z. See the DoFn#getAllowedTimestampSkew() Javadoc for details on changing the allowed skew.</code> |
| This happens when a sink is idle, causing the idle timeout to trigger, or when a specific table is idle long enough when using dynamic destinations. |
| When this happens, the job is no longer able to be drained. This has been fixed for the 2.41 release.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.40.0 release. Thank you to all contributors!</p> |
| <p>Ahmed Abualsaud |
| Ahmet Altay |
| Aizhamal Nurmamat kyzy |
| Alejandro Rodriguez-Morantes |
| Alexander Zhuravlev |
| Alexey Romanenko |
| Anand Inguva |
| andoni-guzman |
| Andy Ye |
| Balázs Németh |
| Benjamin Gonzalez |
| Brian Hulette |
| bulat safiullin |
| bullet03 |
| Chamikara Jayalath |
| Damon Douglas |
| Daniel Oliveira |
| Danny McCormick |
| Darkhan Nausharipov |
| David Huntsperger |
| Diego Gomez |
| dpcollins-google |
| Ekaterina Tatanova |
| Elias Segundo |
| Etienne Chauchot |
| Evan Galpin |
| fbeevikm |
| Fernando Morales |
| Heejong Lee |
| Igor Krasavin |
| Ilion Beyst |
| Israel Herraiz |
| Jack McCluskey |
| Jan Kuehle |
| Jan Lukavský |
| johnjcasey |
| Jonathan Lui |
| jrmccluskey |
| Julien Tournay |
| Kenneth Knowles |
| Kerry Donny-Clark |
| Kevin Puthusseri |
| Kiley Sok |
| Kyle Weaver |
| kynx |
| Lucas Nogueira |
| Luke Cwik |
| LuNing Wang |
| Marco Robles |
| masahitojp |
| Minbo Bae |
| Moritz Mack |
| Naireen Hussain |
| Nancy Xu |
| Niel Markwick |
| Ning Kang |
| nishant jain |
| nishantjain91 |
| Oskar Firlej |
| Pablo Estrada |
| pablo rodriguez defino |
| Rebecca Szper |
| Red Daly |
| Reuven Lax |
| Ritesh Ghorse |
| Robert Bradshaw |
| Robert Burke |
| Ryan Thompson |
| Sam Whittle |
| Thiago Nunes |
| Tom Stepp |
| vachan-shetty |
| Valentyn Tymofieiev |
| vikash2310 |
| Vitaly Terentyev |
| Vladislav Chunikhin |
| Yichi Zhang |
| Yi Hu |
| Yiru Tang |
| yixiaoshen |
| zwestrick</p></description></item><item><title>Blog: Apache Beam 2.39.0</title><link>/blog/beam-2.39.0/</link><pubDate>Wed, 25 May 2022 09:00:00 -0700</pubDate><guid>/blog/beam-2.39.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.39.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2390-2022-05-25">download page</a> for this |
| release.</p> |
| <p>For more information on changes in 2.39.0 check out the <a href="https://issues.apache.org/jira/secure/ConfigureReleaseNote.jspa?projectId=12319527&amp;version=12351170">detailed release notes</a>.</p> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>JmsIO gains the ability to map any kind of input to any subclass of <code>javax.jms.Message</code> (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-16308">BEAM-16308</a>).</li> |
| <li>JmsIO introduces the ability to write to dynamic topics (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-16308">BEAM-16308</a>). |
| <ul> |
| <li>A <code>topicNameMapper</code> must be set to extract the topic name from the input value.</li> |
| <li>A <code>valueMapper</code> must be set to convert the input value to JMS message.</li> |
| </ul> |
| </li> |
| <li>Reduce number of threads spawned by BigqueryIO StreamingInserts ( |
| <a href="https://issues.apache.org/jira/browse/BEAM-14283">BEAM-14283</a>).</li> |
| <li>Implemented Apache PulsarIO (<a href="https://issues.apache.org/jira/browse/BEAM-8218">BEAM-8218</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Support for flink scala 2.12, because most of the libraries support version 2.12 onwards. (<a href="https://issues.apache.org/jira/browse/BEAM-14386">beam-14386</a>)</li> |
| <li>&lsquo;Manage Clusters&rsquo; JupyterLab extension added for users to configure usage of Dataproc clusters managed by Interactive Beam (Python) (<a href="https://issues.apache.org/jira/browse/BEAM-14130">BEAM-14130</a>).</li> |
| <li>Pipeline drain support added for Go SDK (<a href="https://issues.apache.org/jira/browse/BEAM-11106">BEAM-11106</a>). <strong>Note: this feature is not yet fully validated and should be treated as experimental in this release.</strong></li> |
| <li><code>DataFrame.unstack()</code>, <code>DataFrame.pivot() </code> and <code>Series.unstack()</code> |
| implemented for DataFrame API (<a href="https://issues.apache.org/jira/browse/BEAM-13948">BEAM-13948</a>, <a href="https://issues.apache.org/jira/browse/BEAM-13966">BEAM-13966</a>).</li> |
| <li>Support for impersonation credentials added to dataflow runner in the Java and Python SDK (<a href="https://issues.apache.org/jira/browse/BEAM-14014">BEAM-14014</a>).</li> |
| <li>Implemented Jupyterlab extension for managing Dataproc clusters (<a href="https://issues.apache.org/jira/browse/BEAM-14130">BEAM-14130</a>).</li> |
| <li>ExternalPythonTransform API added for easily invoking Python transforms from |
| Java (<a href="https://issues.apache.org/jira/browse/BEAM-14143">BEAM-14143</a>).</li> |
| <li>Added Add support for Elasticsearch 8.x (<a href="https://issues.apache.org/jira/browse/BEAM-14003">BEAM-14003</a>).</li> |
| <li>Shard aware Kinesis record aggregation (AWS Sdk v2), (<a href="https://issues.apache.org/jira/browse/BEAM-14104">BEAM-14104</a>).</li> |
| <li>Upgrade to ZetaSQL 2022.04.1 (<a href="https://issues.apache.org/jira/browse/BEAM-14348">BEAM-14348</a>).</li> |
| <li>Fixed ReadFromBigQuery cannot be used with the interactive runner (<a href="https://issues.apache.org/jira/browse/BEAM-14112">BEAM-14112</a>).</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>Unused functions <code>ShallowCloneParDoPayload()</code>, <code>ShallowCloneSideInput()</code>, and <code>ShallowCloneFunctionSpec()</code> have been removed from the Go SDK&rsquo;s pipelinex package (<a href="https://issues.apache.org/jira/browse/BEAM-13739">BEAM-13739</a>).</li> |
| <li>JmsIO requires an explicit <code>valueMapper</code> to be set (<a href="https://issues.apache.org/jira/browse/BEAM-16308">BEAM-16308</a>). You can use the <code>TextMessageMapper</code> to convert <code>String</code> inputs to JMS <code>TestMessage</code>s:</li> |
| </ul> |
| <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"> <span class="n">JmsIO</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="n">write</span><span class="o">()</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withConnectionFactory</span><span class="o">(</span><span class="n">jmsConnectionFactory</span><span class="o">)</span> |
| </span></span><span class="line"><span class="cl"> <span class="o">.</span><span class="na">withValueMapper</span><span class="o">(</span><span class="k">new</span> <span class="n">TextMessageMapper</span><span class="o">());</span> |
| </span></span></code></pre></div><ul> |
| <li>Coders in Python are expected to inherit from Coder. (<a href="https://issues.apache.org/jira/browse/BEAM-14351">BEAM-14351</a>).</li> |
| <li>New abstract method <code>metadata()</code> added to io.filesystem.FileSystem in the |
| Python SDK. (<a href="https://issues.apache.org/jira/browse/BEAM-14314">BEAM-14314</a>)</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>Flink 1.11 is no longer supported (<a href="https://issues.apache.org/jira/browse/BEAM-14139">BEAM-14139</a>).</li> |
| <li>Python 3.6 is no longer supported (<a href="https://issues.apache.org/jira/browse/BEAM-13657">BEAM-13657</a>).</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed Java Spanner IO NPE when ProjectID not specified in template executions (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-14405">BEAM-14405</a>).</li> |
| <li>Fixed potential NPE in BigQueryServicesImpl.getErrorInfo (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-14133">BEAM-14133</a>).</li> |
| </ul> |
| <h2 id="known-issues">Known Issues</h2> |
| <ul> |
| <li>See a full list of open <a href="https://issues.apache.org/jira/browse/BEAM-14412?jql=project%20%3D%20BEAM%20AND%20affectedVersion%20%3D%202.39.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC">issues that affect</a> this version.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.39.0 release. Thank you to all contributors!</p> |
| <p>Ahmed Abualsaud, |
| Ahmet Altay, |
| Aizhamal Nurmamat kyzy, |
| Alexander Zhuravlev, |
| Alexey Romanenko, |
| Anand Inguva, |
| Andrei Gurau, |
| Andrew Pilloud, |
| Andy Ye, |
| Arun Pandian, |
| Arwin Tio, |
| Aydar Farrakhov, |
| Aydar Zainutdinov, |
| AydarZaynutdinov, |
| Balázs Németh, |
| Benjamin Gonzalez, |
| Brian Hulette, |
| Buqian Zheng, |
| Chamikara Jayalath, |
| Chun Yang, |
| Daniel Oliveira, |
| Daniela Martín, |
| Danny McCormick, |
| David Huntsperger, |
| Deepak Nagaraj, |
| Denise Case, |
| Esun Kim, |
| Etienne Chauchot, |
| Evan Galpin, |
| Hector Miuler Malpica Gallegos, |
| Heejong Lee, |
| Hengfeng Li, |
| Ilango Rajagopal, |
| Ilion Beyst, |
| Israel Herraiz, |
| Jack McCluskey, |
| Kamil Bregula, |
| Kamil Breguła, |
| Ke Wu, |
| Kenneth Knowles, |
| KevinGG, |
| Kiley, |
| Kiley Sok, |
| Kyle Weaver, |
| Liam Miller-Cushon, |
| Luke Cwik, |
| Marco Robles, |
| Matt Casters, |
| Michael Li, |
| MiguelAnzoWizeline, |
| Milan Patel, |
| Minbo Bae, |
| Moritz Mack, |
| Nick Caballero, |
| Niel Markwick, |
| Ning Kang, |
| Oskar Firlej, |
| Pablo Estrada, |
| Pavel Avilov, |
| Reuven Lax, |
| Reza Rokni, |
| Ritesh Ghorse, |
| Robert Bradshaw, |
| Robert Burke, |
| Ryan Thompson, |
| Sam Whittle, |
| Steven Niemitz, |
| Thiago Nunes, |
| Tomo Suzuki, |
| Valentyn Tymofieiev, |
| Victor, |
| Yi Hu, |
| Yichi Zhang, |
| Yiru Tang, |
| ahmedabu98, |
| andoni-guzman, |
| brachipa, |
| bulat safiullin, |
| bullet03, |
| dannymartinm, |
| daria.malkova, |
| dpcollins-google, |
| egalpin, |
| emily, |
| fbeevikm, |
| johnjcasey, |
| kileys, |
| <a href="mailto:msbukal@google.com">msbukal@google.com</a>, |
| nguyennk92, |
| pablo rodriguez defino, |
| rszper, |
| rvballada, |
| sachinag, |
| tvalentyn, |
| vachan-shetty, |
| yirutang</p></description></item><item><title>Blog: Apache Beam 2.38.0</title><link>/blog/beam-2.38.0/</link><pubDate>Wed, 20 Apr 2022 09:00:00 -0700</pubDate><guid>/blog/beam-2.38.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.38.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2380-2022-04-20">download page</a> for this release.</p> |
| <p>For more information on changes in 2.38.0 check out the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12351169">detailed release notes</a>.</p> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Introduce projection pushdown optimizer to the Java SDK (<a href="https://issues.apache.org/jira/browse/BEAM-12976">BEAM-12976</a>). The optimizer currently only works on the <a href="/documentation/io/built-in/google-bigquery/#storage-api">BigQuery Storage API</a>, but more I/Os will be added in future releases. If you encounter a bug with the optimizer, please file a JIRA and disable the optimizer using pipeline option <code>--experiments=disable_projection_pushdown</code>.</li> |
| <li>A new IO for Neo4j graph databases was added. (<a href="https://issues.apache.org/jira/browse/BEAM-1857">BEAM-1857</a>) It has the ability to update nodes and relationships using UNWIND statements and to read data using cypher statements with parameters.</li> |
| <li><code>amazon-web-services2</code> has reached feature parity and is finally recommended over the earlier <code>amazon-web-services</code> and <code>kinesis</code> modules (Java). These will be deprecated in one of the next releases (<a href="https://issues.apache.org/jira/browse/BEAM-13174">BEAM-13174</a>). |
| <ul> |
| <li>Long outstanding write support for <code>Kinesis</code> was added (<a href="https://issues.apache.org/jira/browse/BEAM-13175">BEAM-13175</a>).</li> |
| <li>Configuration was simplified and made consistent across all IOs, including the usage of <code>AwsOptions</code> (<a href="https://issues.apache.org/jira/browse/BEAM-13563">BEAM-13563</a>, <a href="https://issues.apache.org/jira/browse/BEAM-13663">BEAM-13663</a>, <a href="https://issues.apache.org/jira/browse/BEAM-13587">BEAM-13587</a>).</li> |
| <li>Additionally, there&rsquo;s a long list of recent improvements and fixes to |
| <code>S3</code> Filesystem (<a href="https://issues.apache.org/jira/browse/BEAM-13245">BEAM-13245</a>, <a href="https://issues.apache.org/jira/browse/BEAM-13246">BEAM-13246</a>, <a href="https://issues.apache.org/jira/browse/BEAM-13441">BEAM-13441</a>, <a href="https://issues.apache.org/jira/browse/BEAM-13445">BEAM-13445</a>, <a href="https://issues.apache.org/jira/browse/BEAM-14011">BEAM-14011</a>), |
| <code>DynamoDB</code> IO (<a href="https://issues.apache.org/jira/browse/BEAM-13009">BEAM-13209</a>, <a href="https://issues.apache.org/jira/browse/BEAM-13209">BEAM-13209</a>), |
| <code>SQS</code> IO (<a href="https://issues.apache.org/jira/browse/BEAM-13631">BEAM-13631</a>, <a href="https://issues.apache.org/jira/browse/BEAM-13510">BEAM-13510</a>) and others.</li> |
| </ul> |
| </li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Pipeline dependencies supplied through <code>--requirements_file</code> will now be staged to the runner using binary distributions (wheels) of the PyPI packages for linux_x86_64 platform (<a href="https://issues.apache.org/jira/browse/BEAM-4032">BEAM-4032</a>). To restore the behavior to use source distributions, set pipeline option <code>--requirements_cache_only_sources</code>. To skip staging the packages at submission time, set pipeline option <code>--requirements_cache=skip</code> (Python).</li> |
| <li>The Flink runner now supports Flink 1.14.x (<a href="https://issues.apache.org/jira/browse/BEAM-13106">BEAM-13106</a>).</li> |
| <li>Interactive Beam now supports remotely executing Flink pipelines on Dataproc (Python) (<a href="https://issues.apache.org/jira/browse/BEAM-14071">BEAM-14071</a>).</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>(Python) Previously <code>DoFn.infer_output_types</code> was expected to return <code>Iterable[element_type]</code> where <code>element_type</code> is the PCollection elemnt type. It is now expected to return <code>element_type</code>. Take care if you have overriden <code>infer_output_type</code> in a <code>DoFn</code> (this is not common). See <a href="https://issues.apache.org/jira/browse/BEAM-13860">BEAM-13860</a>.</li> |
| <li>(<code>amazon-web-services2</code>) The types of <code>awsRegion</code> / <code>endpoint</code> in <code>AwsOptions</code> changed from String to <code>Region</code> / <code>URI</code> (<a href="https://issues.apache.org/jira/browse/BEAM-13563">BEAM-13563</a>).</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>Beam 2.38.0 will be the last minor release to support Flink 1.11.</li> |
| <li>(<code>amazon-web-services2</code>) Client providers (<code>withXYZClientProvider()</code>) as well as IO specific <code>RetryConfiguration</code>s are deprecated, instead use <code>withClientConfiguration()</code> or <code>AwsOptions</code> to configure AWS IOs / clients. |
| Custom implementations of client providers shall be replaced with a respective <code>ClientBuilderFactory</code> and configured through <code>AwsOptions</code> (<a href="https://issues.apache.org/jira/browse/BEAM-13563">BEAM-13563</a>).</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fix S3 copy for large objects (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-14011">BEAM-14011</a>)</li> |
| <li>Fix quadratic behavior of pipeline canonicalization (Go) (<a href="https://issues.apache.org/jira/browse/BEAM-14128">BEAM-14128</a>) |
| <ul> |
| <li>This caused unnecessarily long pre-processing times before job submission for large complex pipelines.</li> |
| </ul> |
| </li> |
| <li>Fix <code>pyarrow</code> version parsing (Python)(<a href="https://issues.apache.org/jira/browse/BEAM-14235">BEAM-14235</a>)</li> |
| </ul> |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li>See a full list of open <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20affectedVersion%20%3D%202.38.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC">issues that affect</a> this version.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.38.0 release. Thank you to all contributors!</p> |
| <p>abhijeet-lele |
| Ahmet Altay |
| akustov |
| Alexander |
| Alexander Zhuravlev |
| Alexey Romanenko |
| AlikRodriguez |
| Anand Inguva |
| andoni-guzman |
| andreukus |
| Andy Ye |
| Ankur Goenka |
| ansh0l |
| Artur Khanin |
| Aydar Farrakhov |
| Aydar Zainutdinov |
| Benjamin Gonzalez |
| Brian Hulette |
| brucearctor |
| bulat safiullin |
| bullet03 |
| Carl Mastrangelo |
| Chamikara Jayalath |
| Chun Yang |
| Daniela Martín |
| Daniel Oliveira |
| Danny McCormick |
| daria.malkova |
| David Cavazos |
| David Huntsperger |
| dmitryor |
| Dmytro Sadovnychyi |
| dpcollins-google |
| egalpin |
| Elias Segundo Antonio |
| emily |
| Etienne Chauchot |
| Hengfeng Li |
| Ismaël Mejía |
| Israel Herraiz |
| Jack McCluskey |
| Jakub Kukul |
| Janek Bevendorff |
| Jeff Klukas |
| Johan Sternby |
| Kamil Breguła |
| Kenneth Knowles |
| Ke Wu |
| Kiley |
| Kyle Weaver |
| laraschmidt |
| Lara Schmidt |
| LE QUELLEC Olivier |
| Luka Kalinovcic |
| Luke Cwik |
| Marcin Kuthan |
| masahitojp |
| Masato Nakamura |
| Matt Casters |
| Melissa Pashniak |
| Michael Li |
| Miguel Hernandez |
| Moritz Mack |
| mosche |
| nancyxu123 |
| Nathan J Mehl |
| Niel Markwick |
| Ning Kang |
| Pablo Estrada |
| paul-tlh |
| Pavel Avilov |
| Rahul Iyer |
| Reuven Lax |
| Ritesh Ghorse |
| Robert Bradshaw |
| Robert Burke |
| Ryan Skraba |
| Ryan Thompson |
| Sam Whittle |
| Seth Vargo |
| sp029619 |
| Steven Niemitz |
| Thiago Nunes |
| Udi Meiri |
| Valentyn Tymofieiev |
| Victor |
| vitaly.terentyev |
| Yichi Zhang |
| Yi Hu |
| yirutang |
| Zachary Houfek |
| Zoe</p></description></item><item><title>Blog: Apache Beam 2.37.0</title><link>/blog/beam-2.37.0/</link><pubDate>Fri, 04 Mar 2022 08:30:00 -0800</pubDate><guid>/blog/beam-2.37.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.37.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2370-2022-03-04">download page</a> for this release.</p> |
| <p>For more information on changes in 2.37.0 check out the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12351168">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Java 17 support for Dataflow (<a href="https://issues.apache.org/jira/browse/BEAM-12240">BEAM-12240</a>). |
| <ul> |
| <li>Users using Dataflow Runner V2 may see issues with state cache due to inaccurate object sizes (<a href="https://issues.apache.org/jira/browse/BEAM-13695">BEAM-13695</a>).</li> |
| <li>ZetaSql is currently unsupported (<a href="https://github.com/google/zetasql/issues/89">issue</a>).</li> |
| </ul> |
| </li> |
| <li>Python 3.9 support in Apache Beam (<a href="https://issues.apache.org/jira/browse/BEAM-12000">BEAM-12000</a>). |
| <ul> |
| <li>Dataflow support for Python 3.9 is expected to be available with 2.37.0, |
| but may not be fully available yet when the release is announced (<a href="https://issues.apache.org/jira/browse/BEAM-13864">BEAM-13864</a>).</li> |
| <li>Users of Dataflow Runner V2 can run Python 3.9 pipelines with 2.37.0 release right away.</li> |
| </ul> |
| </li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Go SDK now has wrappers for the following Cross Language Transforms from Java, along with automatic expansion service startup for each. |
| <ul> |
| <li>JDBCIO (<a href="https://issues.apache.org/jira/browse/BEAM-13293">BEAM-13293</a>).</li> |
| <li>Debezium (<a href="https://issues.apache.org/jira/browse/BEAM-13761">BEAM-13761</a>).</li> |
| <li>BeamSQL (<a href="https://issues.apache.org/jira/browse/BEAM-13683">BEAM-13683</a>).</li> |
| <li>BiqQuery (<a href="https://issues.apache.org/jira/browse/BEAM-13732">BEAM-13732</a>).</li> |
| <li>KafkaIO now also has automatic expansion service startup. (<a href="https://issues.apache.org/jira/browse/BEAM-13821">BEAM-13821</a>).</li> |
| </ul> |
| </li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>DataFrame API now supports pandas 1.4.x (<a href="https://issues.apache.org/jira/browse/BEAM-13605">BEAM-13605</a>).</li> |
| <li>Go SDK DoFns can now observe trigger panes directly (<a href="https://issues.apache.org/jira/browse/BEAM-13757">BEAM-13757</a>).</li> |
| </ul> |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li>See a full list of open <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20affectedVersion%20%3D%202.37.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC">issues that affect</a> this version.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.37.0 release. Thank you to all contributors!</p> |
| <p>Aizhamal Nurmamat kyzy |
| Alexander |
| Alexander Chermenin |
| Alexandr Zhuravlev |
| Alexey Romanenko |
| Anand Inguva |
| andoni-guzman |
| andreukus |
| Andy Ye |
| Artur Khanin |
| Aydar Farrakhov |
| Aydar Zainutdinov |
| AydarZaynutdinov |
| Benjamin Gonzalez |
| Brian Hulette |
| Chamikara Jayalath |
| Daniel Oliveira |
| Danny McCormick |
| daria-malkova |
| daria.malkova |
| darshan-sj |
| David Huntsperger |
| dprieto91 |
| emily |
| Etienne Chauchot |
| Fernando Morales |
| Heejong Lee |
| Ismaël Mejía |
| Jack McCluskey |
| Jan Lukavský |
| johnjcasey |
| Kamil Breguła |
| kellen |
| Kenneth Knowles |
| kileys |
| Kyle Weaver |
| Luke Cwik |
| Marcin Kuthan |
| Marco Robles |
| Matt Rudary |
| Miguel Hernandez |
| Milena Bukal |
| Moritz Mack |
| Mostafa Aghajani |
| Ning Kang |
| Pablo Estrada |
| Pavel Avilov |
| Reuven Lax |
| Ritesh Ghorse |
| Robert Bradshaw |
| Robert Burke |
| Sam Whittle |
| Sandy Chapman |
| Sergey Kalinin |
| Thiago Nunes |
| thorbjorn444 |
| Tim Robertson |
| Tomo Suzuki |
| Valentyn Tymofieiev |
| Victor |
| Victor Chen |
| Vitaly Ivanov |
| Yichi Zhang</p></description></item><item><title>Blog: Apache Beam 2.36.0</title><link>/blog/beam-2.36.0/</link><pubDate>Mon, 07 Feb 2022 10:11:00 -0800</pubDate><guid>/blog/beam-2.36.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.36.0 release of Apache Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2360-2022-02-07">download page</a> for this release.</p> |
| <p>For more information on changes in 2.36.0, check out the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12350407">detailed release |
| notes</a>.</p> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Support for stopReadTime on KafkaIO SDF (Java).(<a href="https://issues.apache.org/jira/browse/BEAM-13171">BEAM-13171</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>💻 Support for ARM64 / Mac M1 out of the box. (<a href="https://issues.apache.org/jira/browse/BEAM-11703">BEAM-11703</a>).</li> |
| <li>Added support for cloudpickle as a pickling library for Python SDK (<a href="https://issues.apache.org/jira/browse/BEAM-8123">BEAM-8123</a>). To use cloudpickle, set pipeline option: &ndash;pickle_library=cloudpickle</li> |
| <li>Added option to specify triggering frequency when streaming to BigQuery (Python) (<a href="https://issues.apache.org/jira/browse/BEAM-12865">BEAM-12865</a>).</li> |
| <li>Added option to enable caching uploaded artifacts across job runs for Python Dataflow jobs (<a href="https://issues.apache.org/jira/browse/BEAM-13459">BEAM-13459</a>). To enable, set pipeline option: &ndash;enable_artifact_caching, this will be enabled by default in a future release.</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>Updated the jedis from 3.x to 4.x to Java RedisIO. If you are using RedisIO and using jedis directly, please refer to <a href="https://github.com/redis/jedis/blob/v4.0.0/docs/3to4.md">this page</a> to update it. (<a href="https://issues.apache.org/jira/browse/BEAM-12092">BEAM-12092</a>).</li> |
| <li>Datatype of timestamp fields in <code>SqsMessage</code> for AWS IOs for SDK v2 was changed from <code>String</code> to <code>long</code>, visibility of all fields was fixed from <code>package private</code> to <code>public</code> <a href="https://issues.apache.org/jira/browse/BEAM-13638">BEAM-13638</a>.</li> |
| <li>Properly check output timestamps on elements output from DoFns, timers, and onWindowExpiration in Java <a href="https://issues.apache.org/jira/browse/BEAM-12931">BEAM-12931</a>.</li> |
| <li>Fixed a bug with DeferredDataFrame.xs when used with a non-tuple key |
| (<a href="https://issues.apache.org/jira/browse/BEAM-13421%5D">BEAM-13421</a>).</li> |
| <li>Beam Python now requires <code>google-cloud-pubsub&gt;=2.1.0</code>. The API surface for <code>apache_beam.io.gcp.pubsub</code> has not changed, but code that uses the PubSub client directly may need to be updated.</li> |
| </ul> |
| <h2 id="known-issues">Known Issues</h2> |
| <ul> |
| <li>Users may encounter an unexpected java.lang.ArithmeticException when outputting a timestamp |
| for an element further than allowedSkew from an allowed DoFN skew set to a value more than |
| Integer.MAX_VALUE.</li> |
| <li>S3 object metadata retrieval broken in Python SDK (<a href="https://issues.apache.org/jira/browse/BEAM-13980">BEAM-13980</a>)</li> |
| <li>See a full list of open <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20affectedVersion%20%3D%202.36.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC">issues that affect</a> this version.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.36.0 release. Thank you to all contributors!</p> |
| <p>Ada Wong |
| Ahmet Altay |
| Alexander |
| Alexander Dahl |
| Alexandr Zhuravlev |
| Alexey Romanenko |
| AlikRodriguez |
| Anand Inguva |
| Andrew Pilloud |
| Andy Ye |
| Arkadiusz Gasiński |
| Artur Khanin |
| Arun Pandian |
| Aydar Farrakhov |
| Aydar Zainutdinov |
| AydarZaynutdinov |
| Benjamin Gonzalez |
| Brian Hulette |
| Chamikara Jayalath |
| Daniel Collins |
| Daniel Oliveira |
| Daniel Thevessen |
| Daniela Martín |
| David Hinkes |
| David Huntsperger |
| Emily Ye |
| Etienne Chauchot |
| Evan Galpin |
| Heejong Lee |
| Ilya |
| Ilya Kozyrev |
| In-Ho Yi |
| Jack McCluskey |
| Janek Bevendorff |
| Jarek Potiuk |
| Ke Wu |
| KevinGG |
| Kyle Hersey |
| Kyle Weaver |
| Luís Bianchin |
| Luke Cwik |
| Masato Nakamura |
| Matthias Baetens |
| Mehdi Drissi |
| Melissa Pashniak |
| Michel Davit |
| Miguel Hernandez |
| MiguelAnzoWizeline |
| Milena Bukal |
| Moritz Mack |
| Mostafa Aghajani |
| Nathan J Mehl |
| Niel Markwick |
| Ning Kang |
| Pablo Estrada |
| Pavel Avilov |
| Quentin Sommer |
| Reuben van Ammers |
| Reuven Lax |
| Ritesh Ghorse |
| Robert Bradshaw |
| Robert Burke |
| Ryan Thompson |
| Sam Whittle |
| Sayat |
| Sergei Lebedev |
| Sergey Kalinin |
| Steve Niemitz |
| Talat Uyarer |
| Thiago Nunes |
| Tianyang Hu |
| Tim Robertson |
| Valentyn Tymofieiev |
| Vitaly Ivanov |
| Yichi Zhang |
| Yiru Tang |
| Yu Feng |
| Yu ISHIKAWA |
| Zachary Houfek |
| blais |
| daria-malkova |
| daria.malkova |
| darshan-sj |
| dpcollins-google |
| emily |
| ewianda |
| johnjcasey |
| kileys |
| lam206 |
| laraschmidt |
| mosche |
| <a href="mailto:msbukal@google.com">msbukal@google.com</a> |
| tvalentyn</p></description></item><item><title>Blog: Apache Beam 2.35.0</title><link>/blog/beam-2.35.0/</link><pubDate>Wed, 29 Dec 2021 10:11:00 -0800</pubDate><guid>/blog/beam-2.35.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.35.0 release of Apache Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2350-2021-12-29">download page</a> for this release.</p> |
| <p>For more information on changes in 2.35.0, check out the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12350406">detailed release |
| notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>MultiMap side inputs are now supported by the Go SDK (<a href="https://issues.apache.org/jira/browse/BEAM-3293">BEAM-3293</a>).</li> |
| <li>Side inputs are supported within Splittable DoFns for Dataflow Runner V1 and Dataflow Runner V2. (<a href="https://issues.apache.org/jira/browse/BEAM-12522">BEAM-12522</a>).</li> |
| <li>Upgrades Log4j version used in test suites (Apache Beam testing environment only, not for end user consumption) to 2.17.0(<a href="https://issues.apache.org/jira/browse/BEAM-13434">BEAM-13434</a>). |
| Note that Apache Beam versions do not depend on the Log4j 2 dependency (log4j-core) impacted by <a href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228">CVE-2021-44228</a>. |
| However we urge users to update direct and indirect dependencies (if any) on Log4j 2 to the latest version by updating their build configuration and redeploying impacted pipelines.</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>We changed the data type for ranges in <code>JdbcIO.readWithPartitions</code> from <code>int</code> to <code>long</code> (<a href="https://issues.apache.org/jira/browse/BEAM-13149">BEAM-13149</a>). |
| This is a relatively minor breaking change, which we&rsquo;re implementing to improve the usability of the transform without increasing cruft. |
| This transform is relatively new, so we may implement other breaking changes in the future to improve its usability.</li> |
| <li>Side inputs are supported within Splittable DoFns for Dataflow Runner V1 and Dataflow Runner V2. (<a href="https://issues.apache.org/jira/browse/BEAM-12522">BEAM-12522</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Added custom delimiters to Python TextIO reads (<a href="https://issues.apache.org/jira/browse/BEAM-12730">BEAM-12730</a>).</li> |
| <li>Added escapechar parameter to Python TextIO reads (<a href="https://issues.apache.org/jira/browse/BEAM-13189">BEAM-13189</a>).</li> |
| <li>Splittable reading is enabled by default while reading data with ParquetIO (<a href="https://issues.apache.org/jira/browse/BEAM-12070">BEAM-12070</a>).</li> |
| <li>DoFn Execution Time metrics added to Go (<a href="https://issues.apache.org/jira/browse/BEAM-13001">BEAM-13001</a>).</li> |
| <li>Cross-bundle side input caching is now available in the Go SDK for runners that support the feature by setting the EnableSideInputCache hook (<a href="https://issues.apache.org/jira/browse/BEAM-11097">BEAM-11097</a>).</li> |
| <li>Upgraded the GCP Libraries BOM version to 24.0.0 and associated dependencies (<a href="https://issues.apache.org/jira/browse/BEAM-11205">BEAM-11205</a>). For Google Cloud client library versions set by this BOM, |
| see <a href="https://storage.googleapis.com/cloud-opensource-java-dashboard/com.google.cloud/libraries-bom/24.0.0/artifact_details.html">this table</a>.</li> |
| <li>Removed avro-python3 dependency in AvroIO. Fastavro has already been our Avro library of choice on Python 3. Boolean use_fastavro is left for api compatibility, but will have no effect.(<a href="https://github.com/apache/beam/pull/15900">BEAM-13016</a>).</li> |
| <li>MultiMap side inputs are now supported by the Go SDK (<a href="https://issues.apache.org/jira/browse/BEAM-3293">BEAM-3293</a>).</li> |
| <li>Remote packages can now be downloaded from locations supported by apache_beam.io.filesystems. The files will be downloaded on Stager and uploaded to staging location. For more information, see <a href="https://issues.apache.org/jira/browse/BEAM-11275">BEAM-11275</a></li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>A new URN convention was adopted for cross-language transforms and existing URNs were updated. This may break advanced use-cases, for example, if a custom expansion service is used to connect diffrent Beam Java and Python versions. (<a href="https://issues.apache.org/jira/browse/BEAM-12047">BEAM-12047</a>).</li> |
| <li>The upgrade to Calcite 1.28.0 introduces a breaking change in the SUBSTRING function in SqlTransform, when used with the Calcite dialect (<a href="https://issues.apache.org/jira/browse/BEAM-13099">BEAM-13099</a>, <a href="https://issues.apache.org/jira/browse/CALCITE-4427">CALCITE-4427</a>).</li> |
| <li>ListShards (with DescribeStreamSummary) is used instead of DescribeStream to list shards in Kinesis streams (AWS SDK v2). Due to this change, as mentioned in <a href="https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ListShards.html">AWS documentation</a>, for fine-grained IAM policies it is required to update them to allow calls to ListShards and DescribeStreamSummary APIs. For more information, see <a href="https://docs.aws.amazon.com/streams/latest/dev/controlling-access.html">Controlling Access to Amazon Kinesis Data Streams</a> (<a href="https://issues.apache.org/jira/browse/BEAM-13233">BEAM-13233</a>).</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>Non-splittable reading is deprecated while reading data with ParquetIO (<a href="https://issues.apache.org/jira/browse/BEAM-12070">BEAM-12070</a>).</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Properly map main input windows to side input windows by default (Go) |
| (<a href="https://issues.apache.org/jira/browse/BEAM-11087">BEAM-11087</a>).</li> |
| <li>Fixed data loss when writing to DynamoDB without setting deduplication key names (Java) |
| (<a href="https://issues.apache.org/jira/browse/BEAM-13009">BEAM-13009</a>).</li> |
| <li>Go SDK Examples now have types and functions registered. (Go) (<a href="https://issues.apache.org/jira/browse/BEAM-5378">BEAM-5378</a>)</li> |
| <li>Fixed data loss when using Python WriteToFiles in streaming pipeline (<a href="https://issues.apache.org/jira/browse/BEAM-12950">BEAM-12950</a>).</li> |
| </ul> |
| <h2 id="known-issues">Known Issues</h2> |
| <ul> |
| <li>Users of beam-sdks-java-io-hcatalog (and beam-sdks-java-extensions-sql-hcatalog) must take care to override the transitive log4j dependency when they add a hive dependency (<a href="https://issues.apache.org/jira/browse/BEAM-13499">BEAM-13499</a>).</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.35.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay |
| Alexandr Zhuravlev |
| Alexey Romanenko |
| AlikRodriguez |
| Anand Inguva |
| Andrew Pilloud |
| Ankur Goenka |
| Anthony Sottile |
| Artur Khanin |
| Aydar Farrakhov |
| Aydar Zainutdinov |
| Benjamin Gonzalez |
| brachipa |
| Brian Hulette |
| Calvin Leung |
| Chamikara Jayalath |
| Chris Gray |
| Damon Douglas |
| Daniel Collins |
| Daniel Oliveira |
| daria.malkova |
| darshan-sj |
| David Huntsperger |
| David Prieto Rivera |
| Dmitrii Kuzin |
| dpcollins-google |
| dprieto |
| egalpin |
| Etienne Chauchot |
| Eugene Nikolaiev |
| Fernando Morales |
| Hector Lagos |
| Heejong Lee |
| Ilya Kozyrev |
| Iñigo San Jose Visiers |
| Jack McCluskey |
| Jiayang Wu |
| jrhy |
| Kenneth Knowles |
| KevinGG |
| kileys |
| klmilam |
| Kyle Weaver |
| Luís Bianchin |
| Luke Cwik |
| Melissa Pashniak |
| Michael Luckey |
| Miguel Hernandez |
| Milena Bukal |
| Minbo Bae |
| minherz |
| Moritz Mack |
| mosche |
| Natalie |
| Ning Kang |
| Pablo Estrada |
| Pavel Avilov |
| Reuven Lax |
| Ritesh Ghorse |
| Robert Bradshaw |
| Robert Burke |
| Rogan Morrow |
| Ruslan Altynnikov |
| Sam Whittle |
| Sergey Kalinin |
| Slava Chernyak |
| Svetak Sundhar |
| Tianyang Hu |
| Tim Robertson |
| Tomo Suzuki |
| tuorhador |
| Udi Meiri |
| vachan-shetty |
| Valentyn Tymofieiev |
| Yichi Zhang |
| zhoufek</p></description></item><item><title>Blog: Apache Beam 2.34.0</title><link>/blog/beam-2.34.0/</link><pubDate>Thu, 11 Nov 2021 00:11:00 -0800</pubDate><guid>/blog/beam-2.34.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.34.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2340-2021-11-11">download page</a> for this release.</p> |
| <p>For more information on changes in 2.34.0, check out the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12350405">detailed release |
| notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>The Beam Java API for Calcite SqlTransform is no longer experimental (<a href="https://issues.apache.org/jira/browse/BEAM-12680">BEAM-12680</a>).</li> |
| <li>Python&rsquo;s ParDo (Map, FlatMap, etc.) transforms now suport a <code>with_exception_handling</code> option for easily ignoring bad records and implementing the dead letter pattern.</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li><code>ReadFromBigQuery</code> and <code>ReadAllFromBigQuery</code> now run queries with BATCH priority by default. The <code>query_priority</code> parameter is introduced to the same transforms to allow configuring the query priority (Python) (<a href="https://issues.apache.org/jira/browse/BEAM-12913">BEAM-12913</a>).</li> |
| <li>[EXPERIMENTAL] Support for <a href="https://cloud.google.com/bigquery/docs/reference/storage">BigQuery Storage Read API</a> added to <code>ReadFromBigQuery</code>. The newly introduced <code>method</code> parameter can be set as <code>DIRECT_READ</code> to use the Storage Read API. The default is <code>EXPORT</code> which invokes a BigQuery export request. (Python) (<a href="https://issues.apache.org/jira/browse/BEAM-10917">BEAM-10917</a>).</li> |
| <li>[EXPERIMENTAL] Added <code>use_native_datetime</code> parameter to <code>ReadFromBigQuery</code> to configure the return type of <a href="https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#datetime_type">DATETIME</a> fields when using <code>ReadFromBigQuery</code>. This parameter can <em>only</em> be used when <code>method = DIRECT_READ</code>(Python) (<a href="https://issues.apache.org/jira/browse/BEAM-10917">BEAM-10917</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Upgrade to Calcite 1.26.0 (<a href="https://issues.apache.org/jira/browse/BEAM-9379">BEAM-9379</a>).</li> |
| <li>Added a new <code>dataframe</code> extra to the Python SDK that tracks <code>pandas</code> versions |
| we&rsquo;ve verified compatibility with. We now recommend installing Beam with <code>pip install apache-beam[dataframe]</code> when you intend to use the DataFrame API |
| (<a href="https://issues.apache.org/jira/browse/BEAM-12906">BEAM-12906</a>).</li> |
| <li>Add an <a href="https://github.com/cometta/python-apache-beam-spark">example</a> of deploying Python Apache Beam job with Spark Cluster</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>SQL Rows are no longer flattened (<a href="https://issues.apache.org/jira/browse/BEAM-5505">BEAM-5505</a>).</li> |
| <li>[Go SDK] beam.TryCrossLanguage&rsquo;s signature now matches beam.CrossLanguage. Like other Try functions it returns an error instead of panicking. (<a href="https://issues.apache.org/jira/browse/BEAM-9918">BEAM-9918</a>).</li> |
| <li><a href="https://jira.apache.org/jira/browse/BEAM-12925">BEAM-12925</a> was fixed. It used to silently pass incorrect null data read from JdbcIO. Pipelines affected by this will now start throwing failures instead of silently passing incorrect data.</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed error while writing multiple DeferredFrames to csv (Python) (<a href="https://issues.apache.org/jira/browse/BEAM-12701">BEAM-12701</a>).</li> |
| <li>Fixed error when importing the DataFrame API with pandas 1.0.x installed (<a href="https://issues.apache.org/jira/browse/BEAM-12945">BEAM-12945</a>).</li> |
| <li>Fixed top.SmallestPerKey implementation in the Go SDK (<a href="https://issues.apache.org/jira/browse/BEAM-12946">BEAM-12946</a>).</li> |
| </ul> |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li>Large Java BigQueryIO writes with the FILE_LOADS method will fail in batch mode (specifically, when copy jobs are used). |
| This results in the error message: <code>IllegalArgumentException: Attempting to access unknown side input</code>. |
| Please upgrade to a newer version (&gt; 2.34.0) or use another write method (e.g. <code>STORAGE_WRITE_API</code>).</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.34.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, |
| Aizhamal Nurmamat kyzy, |
| Alex Amato, |
| Alexander Chermenin, |
| Alexey Romanenko, |
| AlikRodriguez, |
| Andrew Pilloud, |
| Andy Xu, |
| Ankur Goenka, |
| Aydar Farrakhov, |
| Aydar Zainutdinov, |
| Aydar Zaynutdinov, |
| AydarZaynutdinov, |
| Benjamin Gonzalez, |
| BenWhitehead, |
| Brachi Packter, |
| Brian Hulette, |
| Bu Sun Kim, |
| Chamikara Jayalath, |
| Chris Gray, |
| Chuck Yang, |
| Chun Yang, |
| Claire McGinty, |
| comet, |
| Daniel Collins, |
| Daniel Oliveira, |
| Daniel Thevessen, |
| daria.malkova, |
| David Cavazos, |
| David Huntsperger, |
| Dmytro Kozhevin, |
| dpcollins-google, |
| Eduardo Sánchez López, |
| Elias Djurfeldt, |
| emily, |
| Emily Ye, |
| Enis Sert, |
| Etienne Chauchot, |
| Fernando Morales, |
| Heejong Lee, |
| Ihor Indyk, |
| Ismaël Mejía, |
| Israel Herraiz, |
| Jack McCluskey, |
| Jonathan Hourany, |
| Judah Rand, |
| Kenneth Knowles, |
| KevinGG, |
| Ke Wu, |
| kileys, |
| Kyle Weaver, |
| Luke Cwik, |
| masahitojp, |
| MiguelAnzoWizeline, |
| Minbo Bae, |
| Niels Basjes, |
| Ning Kang, |
| Pablo Estrada, |
| pareshsarafmdb, |
| Paul Féraud, |
| Piotr Szczepanik, |
| Reuven Lax, |
| Ritesh Ghorse, |
| R. Miles McCain, |
| Robert Bradshaw, |
| Robert Burke, |
| Rogan Morrow, |
| Ruwan Lambrichts, |
| rvballada, |
| Ryan Thompson, |
| Sam Rohde, |
| Sam Whittle, |
| Ștefan Istrate, |
| Steve Niemitz, |
| Thomas Li Fredriksen, |
| Tomo Suzuki, |
| tvalentyn, |
| Udi Meiri, |
| Vachan, |
| Valentyn Tymofieiev, |
| Vincent Marquez, |
| WinsonT, |
| Yichi Zhang, |
| Yifan Mai, |
| Yilei &ldquo;Dolee&rdquo; Yang, |
| zhoufek</p></description></item><item><title>Blog: Apache Beam 2.33.0</title><link>/blog/beam-2.33.0/</link><pubDate>Thu, 07 Oct 2021 00:00:01 -0800</pubDate><guid>/blog/beam-2.33.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.33.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2330-2021-10-07">download page</a> for this release.</p> |
| <p>For more information on changes in 2.33.0, check out the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12350404">detailed release |
| notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Go SDK is no longer experimental, and is officially part of the Beam release process. |
| <ul> |
| <li>Matching Go SDK containers are published on release.</li> |
| <li>Batch usage is well supported, and tested on Flink, Spark, and the Python Portable Runner. |
| <ul> |
| <li>SDK Tests are also run against Google Cloud Dataflow, but this doesn&rsquo;t indicate reciprocal support.</li> |
| </ul> |
| </li> |
| <li>The SDK supports Splittable DoFns, Cross Language transforms, and most Beam Model basics.</li> |
| <li>Go Modules are now used for dependency management. |
| <ul> |
| <li>This is a breaking change, see Breaking Changes for resolution.</li> |
| <li>Easier path to contribute to the Go SDK, no need to set up a GO_PATH.</li> |
| <li>Minimum Go version is now Go v1.16</li> |
| </ul> |
| </li> |
| <li>See the announcement blogpost for full information once published.</li> |
| </ul> |
| </li> |
| </ul> |
| <!-- |
| {$TOPICS e.g.:} |
| ### I/Os |
| * Support for X source added (Java) ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)). |
| {$TOPICS} |
| --> |
| <h3 id="new-features--improvements">New Features / Improvements</h3> |
| <ul> |
| <li>Projection pushdown in SchemaIO (<a href="https://issues.apache.org/jira/browse/BEAM-12609">BEAM-12609</a>).</li> |
| <li>Upgrade Flink runner to Flink versions 1.13.2, 1.12.5 and 1.11.4 (<a href="https://issues.apache.org/jira/browse/BEAM-10955">BEAM-10955</a>).</li> |
| </ul> |
| <h3 id="breaking-changes">Breaking Changes</h3> |
| <ul> |
| <li>Since release 2.30.0, &ldquo;The AvroCoder changes for BEAM-2303 [changed] the reader/writer from the Avro ReflectDatum* classes to the SpecificDatum* classes&rdquo; (Java). This default behavior change has been reverted in this release. Use the <code>useReflectApi</code> setting to control it (<a href="https://issues.apache.org/jira/browse/BEAM-12628">BEAM-12628</a>).</li> |
| </ul> |
| <h3 id="deprecations">Deprecations</h3> |
| <ul> |
| <li>Python GBK will stop supporting unbounded PCollections that have global windowing and a default trigger in Beam 2.34. This can be overriden with <code>--allow_unsafe_triggers</code>. (<a href="https://issues.apache.org/jira/browse/BEAM-9487">BEAM-9487</a>).</li> |
| <li>Python GBK will start requiring safe triggers or the <code>--allow_unsafe_triggers</code> flag starting with Beam 2.34. (<a href="https://issues.apache.org/jira/browse/BEAM-9487">BEAM-9487</a>).</li> |
| </ul> |
| <h3 id="bugfixes">Bugfixes</h3> |
| <ul> |
| <li>UnsupportedOperationException when reading from BigQuery tables and converting |
| TableRows to Beam Rows (Java) |
| (<a href="https://issues.apache.org/jira/browse/BEAM-12479">BEAM-12479</a>).</li> |
| <li>SDFBoundedSourceReader behaves much slower compared with the original behavior |
| of BoundedSource (Python) |
| (<a href="https://issues.apache.org/jira/browse/BEAM-12781">BEAM-12781</a>).</li> |
| <li>ORDER BY column not in SELECT crashes (ZetaSQL) |
| (<a href="https://issues.apache.org/jira/browse/BEAM-12759">BEAM-12759</a>).</li> |
| </ul> |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li>Spark 2.x users will need to update Spark&rsquo;s Jackson runtime dependencies (<code>spark.jackson.version</code>) to at least version 2.9.2, due to Beam updating its dependencies.</li> |
| <li>See a full list of open <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20affectedVersion%20%3D%202.33.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC">issues that affect</a> this version.</li> |
| <li>Go SDK jobs may produce &ldquo;Failed to deduce Step from MonitoringInfo&rdquo; messages following successful job execution. The messages are benign and don&rsquo;t indicate job failure. These are due to not yet handling PCollection metrics.</li> |
| <li>Large Java BigQueryIO writes with the FILE_LOADS method will fail in batch mode (specifically, when copy jobs are used). |
| This results in the error message: <code>IllegalArgumentException: Attempting to access unknown side input</code>. |
| Please upgrade to a newer version (&gt; 2.34.0) or use another write method (e.g. <code>STORAGE_WRITE_API</code>).</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.33.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, |
| Alex Amato, |
| Alexey Romanenko, |
| Andreas Bergmeier, |
| Andres Rodriguez, |
| Andrew Pilloud, |
| Andy Xu, |
| Ankur Goenka, |
| anthonyqzhu, |
| Benjamin Gonzalez, |
| Bhupinder Sindhwani, |
| Chamikara Jayalath, |
| Claire McGinty, |
| Daniel Mateus Pires, |
| Daniel Oliveira, |
| David Huntsperger, |
| Dylan Hercher, |
| emily, |
| Emily Ye, |
| Etienne Chauchot, |
| Eugene Nikolaiev, |
| Heejong Lee, |
| iindyk, |
| Iñigo San Jose Visiers, |
| Ismaël Mejía, |
| Jack McCluskey, |
| Jan Lukavský, |
| Jeff Ruane, |
| Jeremy Lewi, |
| KevinGG, |
| Ke Wu, |
| Kyle Weaver, |
| lostluck, |
| Luke Cwik, |
| Marwan Tammam, |
| masahitojp, |
| Mehdi Drissi, |
| Minbo Bae, |
| Ning Kang, |
| Pablo Estrada, |
| Pascal Gillet, |
| Pawas Chhokra, |
| Reuven Lax, |
| Ritesh Ghorse, |
| Robert Bradshaw, |
| Robert Burke, |
| Rodrigo Benenson, |
| Ryan Thompson, |
| Saksham Gupta, |
| Sam Rohde, |
| Sam Whittle, |
| Sayat, |
| Sayat Satybaldiyev, |
| Siyuan Chen, |
| Slava Chernyak, |
| Steve Niemitz, |
| Steven Niemitz, |
| tvalentyn, |
| Tyson Hamilton, |
| Udi Meiri, |
| vachan-shetty, |
| Venkatramani Rajgopal, |
| Yichi Zhang, |
| zhoufek</p></description></item><item><title>Blog: Apache Beam 2.32.0</title><link>/blog/beam-2.32.0/</link><pubDate>Wed, 25 Aug 2021 00:00:01 -0800</pubDate><guid>/blog/beam-2.32.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.32.0 release of Apache Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2320-2021-08-11">download page</a> for this release.</p> |
| <!-- more --> |
| <p>For more information on changes in 2.32.0, check out the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12349992">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>The <a href="/documentation/dsls/dataframes/overview/">Beam DataFrame |
| API</a> is no |
| longer experimental! We&rsquo;ve spent the time since the <a href="/blog/dataframe-api-preview-available/">2.26.0 preview |
| announcement</a> |
| implementing the most frequently used pandas operations |
| (<a href="https://issues.apache.org/jira/browse/BEAM-9547">BEAM-9547</a>), improving |
| <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.dataframe.html">documentation</a> |
| and <a href="https://issues.apache.org/jira/browse/BEAM-12028">error messages</a>, |
| adding |
| <a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/dataframe">examples</a>, |
| integrating DataFrames with <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.runners.interactive.interactive_beam.html">interactive |
| Beam</a>, |
| and of course finding and fixing |
| <a href="https://issues.apache.org/jira/issues/?jql=project%3DBEAM%20AND%20issuetype%3DBug%20AND%20status%3DResolved%20AND%20component%3Ddsl-dataframe">bugs</a>. |
| Leaving experimental just means that we now have high confidence in the API |
| and recommend its use for production workloads. We will continue to improve |
| the API, guided by your |
| <a href="/community/contact-us/">feedback</a>.</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Added ability to use JdbcIO.Write.withResults without statement and preparedStatementSetter. (<a href="https://issues.apache.org/jira/browse/BEAM-12511">BEAM-12511</a>)</li> |
| </ul> |
| <ul> |
| <li>Added ability to register URI schemes to use the S3 protocol via FileIO. (<a href="https://issues.apache.org/jira/browse/BEAM-12435">BEAM-12435</a>).</li> |
| </ul> |
| <ul> |
| <li>Respect number of shards set in SnowflakeWrite batch mode. (<a href="https://issues.apache.org/jira/browse/BEAM-12715">BEAM-12715</a>)</li> |
| <li>Java SDK: Update Google Cloud Healthcare IO connectors from using v1beta1 to using the GA version.</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Add support to convert Beam Schema to Avro Schema for JDBC LogicalTypes: |
| <code>VARCHAR</code>, <code>NVARCHAR</code>, <code>LONGVARCHAR</code>, <code>LONGNVARCHAR</code>, <code>DATE</code>, <code>TIME</code> |
| (Java)(<a href="https://issues.apache.org/jira/browse/BEAM-12385">BEAM-12385</a>).</li> |
| <li>Reading from JDBC source by partitions (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-12456">BEAM-12456</a>).</li> |
| <li>PubsubIO can now write to a dead-letter topic after a parsing error (Java)(<a href="https://issues.apache.org/jira/browse/BEAM-12474">BEAM-12474</a>).</li> |
| <li>New append-only option for Elasticsearch sink (Java) <a href="https://issues.apache.org/jira/browse/BEAM-12601">BEAM-12601</a></li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>ListShards (with DescribeStreamSummary) is used instead of DescribeStream to list shards in Kinesis streams. Due to this change, as mentioned in <a href="https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ListShards.html">AWS documentation</a>, for fine-grained IAM policies it is required to update them to allow calls to ListShards and DescribeStreamSummary APIs. For more information, see <a href="https://docs.aws.amazon.com/streams/latest/dev/controlling-access.html">Controlling Access to Amazon Kinesis Data Streams</a> (<a href="https://issues.apache.org/jira/browse/BEAM-12225">BEAM-12225</a>).</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>Python GBK will stop supporting unbounded PCollections that have global windowing and a default trigger in Beam 2.33. This can be overriden with <code>--allow_unsafe_triggers</code>. (<a href="https://issues.apache.org/jira/browse/BEAM-9487">BEAM-9487</a>).</li> |
| <li>Python GBK will start requiring safe triggers or the <code>--allow_unsafe_triggers</code> flag starting with Beam 2.33. (<a href="https://issues.apache.org/jira/browse/BEAM-9487">BEAM-9487</a>).</li> |
| </ul> |
| <h2 id="bugfixes">Bugfixes</h2> |
| <ul> |
| <li>Fixed race condition in RabbitMqIO causing duplicate acks (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-6516">BEAM-6516</a>))</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.32.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, Ajo Thomas, Alex Amato, Alexey Romanenko, Alex Koay, allenpradeep, Anant Damle, Andrew Pilloud, Ankur Goenka, Ashwin Ramaswami, Benjamin Gonzalez, BenWhitehead, Blake Williams, Boyuan Zhang, Brian Hulette, Chamikara Jayalath, Daniel Oliveira, Daniel Thevessen, daria-malkova, David Cavazos, David Huntsperger, dennisylyung, Dennis Yung, dmkozh, egalpin, emily, Esun Kim, Gabriel Melo de Paula, Harch Vardhan, Heejong Lee, heidimhurst, hoshimura, Iñigo San Jose Visiers, Ismaël Mejía, Jack McCluskey, Jan Lukavský, Justin King, Kenneth Knowles, KevinGG, Ke Wu, kileys, Kyle Weaver, Luke Cwik, Maksym Skorupskyi, masahitojp, Matthew Ouyang, Matthias Baetens, Matt Rudary, MiguelAnzoWizeline, Miguel Hernandez, Nikita Petunin, Ning Ding, Ning Kang, odidev, Pablo Estrada, Pascal Gillet, rafal.ochyra, raphael.sanamyan, Reuven Lax, Robert Bradshaw, Robert Burke, roger-mike, Ryan McDowell, Sam Rohde, Sam Whittle, Siyuan Chen, Teng Qiu, Tianzi Cai, Tobias Hermann, Tomo Suzuki, tvalentyn, Tyson Hamilton, Udi Meiri, Valentyn Tymofieiev, Vitaly Terentyev, Yichi Zhang, Yifan Mai, yoshiki.obata, Yu Feng, YuqiHuai, yzhang559, Zachary Houfek, zhoufek</p></description></item><item><title>Blog: Apache Beam 2.31.0</title><link>/blog/beam-2.31.0/</link><pubDate>Thu, 08 Jul 2021 09:00:00 -0700</pubDate><guid>/blog/beam-2.31.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.31.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2310-2021-07-08">download page</a> for this release.</p> |
| <p>For more information on changes in 2.31.0, check out the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12349991">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <h3 id="ios">I/Os</h3> |
| <ul> |
| <li>Fixed bug in ReadFromBigQuery when a RuntimeValueProvider is used as value of table argument (Python) (<a href="https://issues.apache.org/jira/browse/BEAM-12514">BEAM-12514</a>).</li> |
| </ul> |
| <h3 id="new-features--improvements">New Features / Improvements</h3> |
| <ul> |
| <li><code>CREATE FUNCTION</code> DDL statement added to Calcite SQL syntax. <code>JAR</code> and <code>AGGREGATE</code> are now reserved keywords. (<a href="https://issues.apache.org/jira/browse/BEAM-12339">BEAM-12339</a>).</li> |
| <li>Flink 1.13 is now supported by the Flink runner (<a href="https://issues.apache.org/jira/browse/BEAM-12277">BEAM-12277</a>).</li> |
| <li>DatastoreIO: Write and delete operations now follow automatic gradual ramp-up, |
| in line with best practices (Java/Python) (<a href="https://issues.apache.org/jira/browse/BEAM-12260">BEAM-12260</a>, <a href="https://issues.apache.org/jira/browse/BEAM-12272">BEAM-12272</a>).</li> |
| <li>Python <code>TriggerFn</code> has a new <code>may_lose_data</code> method to signal potential data loss. Default behavior assumes safe (necessary for backwards compatibility). See Deprecations for potential impact of overriding this. (<a href="https://issues.apache.org/jira/browse/BEAM-9487">BEAM-9487</a>).</li> |
| </ul> |
| <h3 id="breaking-changes">Breaking Changes</h3> |
| <ul> |
| <li>Python Row objects are now sensitive to field order. So <code>Row(x=3, y=4)</code> is no |
| longer considered equal to <code>Row(y=4, x=3)</code> (BEAM-11929).</li> |
| <li>Kafka Beam SQL tables now ascribe meaning to the LOCATION field; previously |
| it was ignored if provided.</li> |
| <li><code>TopCombineFn</code> disallow <code>compare</code> as its argument (Python) (<a href="https://issues.apache.org/jira/browse/BEAM-7372">BEAM-7372</a>).</li> |
| <li>Drop support for Flink 1.10 (<a href="https://issues.apache.org/jira/browse/BEAM-12281">BEAM-12281</a>).</li> |
| </ul> |
| <h3 id="deprecations">Deprecations</h3> |
| <ul> |
| <li>Python GBK will stop supporting unbounded PCollections that have global windowing and a default trigger in Beam 2.33. This can be overriden with <code>--allow_unsafe_triggers</code>. (<a href="https://issues.apache.org/jira/browse/BEAM-9487">BEAM-9487</a>).</li> |
| <li>Python GBK will start requiring safe triggers or the <code>--allow_unsafe_triggers</code> flag starting with Beam 2.33. (<a href="https://issues.apache.org/jira/browse/BEAM-9487">BEAM-9487</a>).</li> |
| </ul> |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li>See a full list of <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20affectedVersion%20%3D%202.31.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC">issues that affect</a> this version.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to <code>git shortlog</code>, the following people contributed to the 2.31.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, ajo thomas, Alan Myrvold, Alex Amato, Alexey Romanenko, |
| AlikRodriguez, Anant Damle, Andrew Pilloud, Benjamin Gonzalez, Boyuan Zhang, |
| Brian Hulette, Chamikara Jayalath, Daniel Oliveira, David Cavazos, |
| David Huntsperger, David Moravek, Dmytro Kozhevin, dpcollins-google, Emily Ye, |
| Ernesto Valentino, Evan Galpin, Fernando Morales, Heejong Lee, Ismaël Mejía, |
| Jan Lukavský, Josias Rico, jrynd, Kenneth Knowles, Ke Wu, kileys, Kyle Weaver, |
| masahitojp, Matthias Baetens, Maximilian Michels, Milena Bukal, |
| Nathan J. Mehl, Pablo Estrada, Peter Sobot, Reuven Lax, Robert Bradshaw, |
| Robert Burke, roger-mike, Sam Rohde, Sam Whittle, Stephan Hoyer, Tom Underhill, |
| tvalentyn, Uday Singh, Udi Meiri, Vitaly Terentyev, Xinyu Liu, Yichi Zhang, |
| Yifan Mai, yoshiki.obata, zhoufek</p></description></item><item><title>Blog: Apache Beam 2.30.0</title><link>/blog/beam-2.30.0/</link><pubDate>Wed, 09 Jun 2021 09:00:00 -0700</pubDate><guid>/blog/beam-2.30.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.30.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2300-2021-06-09">download page</a> for this release.</p> |
| <p>For more information on changes in 2.30.0, check out the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12349978">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Legacy Read transform (non-SDF based Read) is used by default for non-FnAPI opensource runners. Use <code>use_sdf_read</code> experimental flag to re-enable SDF based Read transforms (<a href="https://issues.apache.org/jira/browse/BEAM-10670">BEAM-10670</a>)</li> |
| <li>Upgraded vendored gRPC dependency to 1.36.0 (<a href="https://issues.apache.org/jira/browse/BEAM-11227">BEAM-11227</a>)</li> |
| </ul> |
| <h3 id="ios">I/Os</h3> |
| <ul> |
| <li>Fixed the issue that WriteToBigQuery with batch file loads does not respect schema update options when there are multiple load jobs (<a href="https://issues.apache.org/jira/browse/BEAM-11277">BEAM-11277</a>)</li> |
| <li>Fixed the issue that the job didn&rsquo;t properly retry since BigQuery sink swallows HttpErrors when performing streaming inserts (<a href="https://issues.apache.org/jira/browse/BEAM-12362">BEAM-12362</a>)</li> |
| </ul> |
| <h3 id="new-features--improvements">New Features / Improvements</h3> |
| <ul> |
| <li>Added capability to declare resource hints in Java and Python SDKs (<a href="https://issues.apache.org/jira/browse/BEAM-2085">BEAM-2085</a>)</li> |
| <li>Added Spanner IO Performance tests for read and write in Python SDK (<a href="https://issues.apache.org/jira/browse/BEAM-10029">BEAM-10029</a>)</li> |
| <li>Added support for accessing GCP PubSub Message ordering keys, message IDs and message publish timestamp in Python SDK (<a href="https://issues.apache.org/jira/browse/BEAM-7819">BEAM-7819</a>)</li> |
| <li>DataFrame API: Added support for collecting DataFrame objects in interactive Beam (<a href="https://issues.apache.org/jira/browse/BEAM-11855">BEAM-11855</a>)</li> |
| <li>DataFrame API: Added <a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/dataframe">apache_beam.examples.dataframe</a> module (<a href="https://issues.apache.org/jira/browse/BEAM-12024">BEAM-12024</a>)</li> |
| <li>Upgraded the GCP Libraries BOM version to 20.0.0 (<a href="https://issues.apache.org/jira/browse/BEAM-11205">BEAM-11205</a>). For Google Cloud client library versions set by this BOM, see <a href="https://storage.googleapis.com/cloud-opensource-java-dashboard/com.google.cloud/libraries-bom/20.0.0/artifact_details.html">this table</a></li> |
| <li>Added <code>sdkContainerImage</code> flag to (eventually) replace <code>workerHarnessContainerImage</code> (<a href="https://issues.apache.org/jira/browse/BEAM-12212">BEAM-12212</a>)</li> |
| <li>Added support for Dataflow update when schemas are used (<a href="https://issues.apache.org/jira/browse/BEAM-12198">BEAM-12198</a>)</li> |
| <li>Fixed the issue that <code>ZipFiles.zipDirectory</code> leaks native JVM memory (<a href="https://issues.apache.org/jira/browse/BEAM-12220">BEAM-12220</a>)</li> |
| <li>Fixed the issue that <code>Reshuffle.withNumBuckets</code> creates <code>(N*2)-1</code> buckets (<a href="https://issues.apache.org/jira/browse/BEAM-12361">BEAM-12361</a>)</li> |
| </ul> |
| <h3 id="breaking-changes">Breaking Changes</h3> |
| <ul> |
| <li>Drop support for Flink 1.8 and 1.9 (<a href="https://issues.apache.org/jira/browse/BEAM-11948">BEAM-11948</a>)</li> |
| <li>MongoDbIO: Read.withFilter() and Read.withProjection() are removed since they are deprecated since Beam 2.12.0 (<a href="https://issues.apache.org/jira/browse/BEAM-12217">BEAM-12217</a>)</li> |
| <li>RedisIO.readAll() was removed since it was deprecated since Beam 2.13.0. Please use RedisIO.readKeyPatterns() for the equivalent functionality (<a href="https://issues.apache.org/jira/browse/BEAM-12214">BEAM-12214</a>)</li> |
| <li>MqttIO.create() with clientId constructor removed because it was deprecated since Beam 2.13.0 (<a href="https://issues.apache.org/jira/browse/BEAM-12216">BEAM-12216</a>)</li> |
| </ul> |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li>See a full list of open <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20affectedVersion%20%3D%202.30.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC">issues that affect</a> this version.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to <code>git shortlog</code>, the following people contributed to the 2.30.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, Alex Amato, Alexey Romanenko, Anant Damle, Andreas Bergmeier, Andrew Pilloud, Ankur Goenka, |
| Anup D, Artur Khanin, Benjamin Gonzalez, Bipin Upadhyaya, Boyuan Zhang, Brian Hulette, Bulat Shakirzyanov, |
| Chamikara Jayalath, Chun Yang, Daniel Kulp, Daniel Oliveira, David Cavazos, Elliotte Rusty Harold, Emily Ye, |
| Eric Roshan-Eisner, Evan Galpin, Fabien Caylus, Fernando Morales, Heejong Lee, Iñigo San Jose Visiers, |
| Isidro Martínez, Ismaël Mejía, Ke Wu, Kenneth Knowles, KevinGG, Kyle Weaver, Ludovic Post, MATTHEW Ouyang (LCL), |
| Mackenzie Clark, Masato Nakamura, Matthias Baetens, Max, Nicholas Azar, Ning Kang, Pablo Estrada, Patrick McCaffrey, |
| Quentin Sommer, Reuven Lax, Robert Bradshaw, Robert Burke, Rui Wang, Sam Rohde, Sam Whittle, Shoaib Zafar, |
| Siyuan Chen, Sruthi Sree Kumar, Steve Niemitz, Sylvain Veyrié, Tomo Suzuki, Udi Meiri, Valentyn Tymofieiev, |
| Vitaly Terentyev, Wenbing, Xinyu Liu, Yichi Zhang, Yifan Mai, Yueyang Qiu, Yunqing Zhou, ajo thomas, brucearctor, |
| dmkozh, dpcollins-google, emily, jordan-moore, kileys, lostluck, masahitojp, roger-mike, sychen, tvalentyn, |
| vachan-shetty, yoshiki.obata</p></description></item><item><title>Blog: Apache Beam 2.29.0</title><link>/blog/beam-2.29.0/</link><pubDate>Thu, 29 Apr 2021 09:00:00 -0700</pubDate><guid>/blog/beam-2.29.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.29.0 release of Beam. |
| This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2290-2021-04-15">download page</a> for this release.</p> |
| <p>For more information on changes in 2.29.0, check out the <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12349629">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Spark Classic and Portable runners officially support Spark 3 (<a href="https://issues.apache.org/jira/browse/BEAM-7093">BEAM-7093</a>).</li> |
| <li>Official Java 11 support for most runners (Dataflow, Flink, Spark) (<a href="https://issues.apache.org/jira/browse/BEAM-2530">BEAM-2530</a>).</li> |
| <li>DataFrame API now supports GroupBy.apply (<a href="https://issues.apache.org/jira/browse/BEAM-11628">BEAM-11628</a>).</li> |
| </ul> |
| <h3 id="ios">I/Os</h3> |
| <ul> |
| <li>Added support for S3 filesystem on AWS SDK V2 (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-7637">BEAM-7637</a>)</li> |
| <li>GCP BigQuery sink (file loads) uses runner determined sharding for unbounded data (<a href="https://issues.apache.org/jira/browse/BEAM-11772">BEAM-11772</a>)</li> |
| <li>KafkaIO now recognizes the <code>partition</code> property in writing records (<a href="https://issues.apache.org/jira/browse/BEAM-11806">BEAM-11806</a>)</li> |
| <li>Support for Hadoop configuration on ParquetIO (<a href="https://issues.apache.org/jira/browse/BEAM-11913">BEAM-11913</a>)</li> |
| </ul> |
| <h3 id="new-features--improvements">New Features / Improvements</h3> |
| <ul> |
| <li>DataFrame API now supports pandas 1.2.x (<a href="https://issues.apache.org/jira/browse/BEAM-11531">BEAM-11531</a>).</li> |
| <li>Multiple DataFrame API bugfixes (<a href="https://issues.apache.org/jira/browse/BEAM-12071">BEAM-12071</a>, <a href="https://issues.apache.org/jira/browse/BEAM-11929">BEAM-11929</a>)</li> |
| <li>DDL supported in SQL transforms (<a href="https://issues.apache.org/jira/browse/BEAM-11850">BEAM-11850</a>)</li> |
| <li>Upgrade Flink runner to Flink version 1.12.2 (<a href="https://issues.apache.org/jira/browse/BEAM-11941">BEAM-11941</a>)</li> |
| </ul> |
| <h3 id="breaking-changes">Breaking Changes</h3> |
| <ul> |
| <li>Deterministic coding enforced for GroupByKey and Stateful DoFns. Previously non-deterministic coding was allowed, resulting in keys not properly being grouped in some cases. (<a href="https://issues.apache.org/jira/browse/BEAM-11719">BEAM-11719</a>) |
| To restore the old behavior, one can register <code>FakeDeterministicFastPrimitivesCoder</code> with |
| <code>beam.coders.registry.register_fallback_coder(beam.coders.coders.FakeDeterministicFastPrimitivesCoder())</code> |
| or use the <code>allow_non_deterministic_key_coders</code> pipeline option.</li> |
| </ul> |
| <h3 id="deprecations">Deprecations</h3> |
| <ul> |
| <li>Support for Flink 1.8 and 1.9 will be removed in the next release (2.30.0) (<a href="https://issues.apache.org/jira/browse/BEAM-11948">BEAM-11948</a>).</li> |
| </ul> |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li>See a full list of open <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20affectedVersion%20%3D%202.29.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC">issues that affect</a> this version.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to <code>git shortlog</code>, the following people contributed to the 2.29.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, Alan Myrvold, Alex Amato, Alexander Chermenin, Alexey Romanenko, |
| Allen Pradeep Xavier, Amy Wu, Anant Damle, Andreas Bergmeier, Andrei Balici, |
| Andrew Pilloud, Andy Xu, Ankur Goenka, Bashir Sadjad, Benjamin Gonzalez, Boyuan |
| Zhang, Brian Hulette, Chamikara Jayalath, Chinmoy Mandayam, Chuck Yang, |
| dandy10, Daniel Collins, Daniel Oliveira, David Cavazos, David Huntsperger, |
| David Moravek, Dmytro Kozhevin, Emily Ye, Esun Kim, Evgeniy Belousov, Filip |
| Popić, Fokko Driesprong, Gris Cuevas, Heejong Lee, Ihor Indyk, Ismaël Mejía, |
| Jakub-Sadowski, Jan Lukavský, John Edmonds, Juan Sandoval, 谷口恵輔, Kenneth |
| Jung, Kenneth Knowles, KevinGG, Kiley Sok, Kyle Weaver, MabelYC, Mackenzie |
| Clark, Masato Nakamura, Milena Bukal, Miltos, Minbo Bae, Miraç Vuslat Başaran, |
| mynameborat, Nahian-Al Hasan, Nam Bui, Niel Markwick, Niels Basjes, Ning Kang, |
| Nir Gazit, Pablo Estrada, Ramazan Yapparov, Raphael Sanamyan, Reuven Lax, Rion |
| Williams, Robert Bradshaw, Robert Burke, Rui Wang, Sam Rohde, Sam Whittle, |
| Shehzaad Nakhoda, Shehzaad Nakhoda, Siyuan Chen, Sonam Ramchand, Steve Niemitz, |
| sychen, Sylvain Veyrié, Tim Robertson, Tobias Kaymak, Tomasz Szerszeń, Tomasz |
| Szerszeń, Tomo Suzuki, Tyson Hamilton, Udi Meiri, Valentyn Tymofieiev, Yichi |
| Zhang, Yifan Mai, Yixing Zhang, Yoshiki Obata</p></description></item><item><title>Blog: Apache Beam 2.28.0</title><link>/blog/beam-2.28.0/</link><pubDate>Mon, 22 Feb 2021 12:00:00 -0800</pubDate><guid>/blog/beam-2.28.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.28.0 release of Apache Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2280-2021-02-22">download page</a> for this release. |
| For more information on changes in 2.28.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12349499">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Many improvements related to Parquet support (<a href="https://issues.apache.org/jira/browse/BEAM-11460">BEAM-11460</a>, <a href="https://issues.apache.org/jira/browse/BEAM-8202">BEAM-8202</a>, and <a href="https://issues.apache.org/jira/browse/BEAM-11526">BEAM-11526</a>)</li> |
| <li>Hash Functions in BeamSQL (<a href="https://issues.apache.org/jira/browse/BEAM-10074">BEAM-10074</a>)</li> |
| <li>Hash functions in ZetaSQL (<a href="https://issues.apache.org/jira/browse/BEAM-11624">BEAM-11624</a>)</li> |
| <li>Create ApproximateDistinct using HLL Impl (<a href="https://issues.apache.org/jira/browse/BEAM-10324">BEAM-10324</a>)</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>SpannerIO supports using BigDecimal for Numeric fields (<a href="https://issues.apache.org/jira/browse/BEAM-11643">BEAM-11643</a>)</li> |
| <li>Add Beam schema support to ParquetIO (<a href="https://issues.apache.org/jira/browse/BEAM-11526">BEAM-11526</a>)</li> |
| <li>Support ParquetTable Writer (<a href="https://issues.apache.org/jira/browse/BEAM-8202">BEAM-8202</a>)</li> |
| <li>GCP BigQuery sink (streaming inserts) uses runner determined sharding (<a href="https://issues.apache.org/jira/browse/BEAM-11408">BEAM-11408</a>)</li> |
| <li>PubSub support types: TIMESTAMP, DATE, TIME, DATETIME (<a href="https://issues.apache.org/jira/browse/BEAM-11533">BEAM-11533</a>)</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>ParquetIO add methods <em>readGenericRecords</em> and <em>readFilesGenericRecords</em> can read files with an unknown schema. See <a href="https://github.com/apache/beam/pull/13554">PR-13554</a> and (<a href="https://issues.apache.org/jira/browse/BEAM-11460">BEAM-11460</a>)</li> |
| <li>Added support for thrift in KafkaTableProvider (<a href="https://issues.apache.org/jira/browse/BEAM-11482">BEAM-11482</a>)</li> |
| <li>Added support for HadoopFormatIO to skip key/value clone (<a href="https://issues.apache.org/jira/browse/BEAM-11457">BEAM-11457</a>)</li> |
| <li>Support Conversion to GenericRecords in Convert.to transform (<a href="https://issues.apache.org/jira/browse/BEAM-11571">BEAM-11571</a>).</li> |
| <li>Support writes for Parquet Tables in Beam SQL (<a href="https://issues.apache.org/jira/browse/BEAM-8202">BEAM-8202</a>).</li> |
| <li>Support reading Parquet files with unknown schema (<a href="https://issues.apache.org/jira/browse/BEAM-11460">BEAM-11460</a>)</li> |
| <li>Support user configurable Hadoop Configuration flags for ParquetIO (<a href="https://issues.apache.org/jira/browse/BEAM-11527">BEAM-11527</a>)</li> |
| <li>Expose commit_offset_in_finalize and timestamp_policy to ReadFromKafka (<a href="https://issues.apache.org/jira/browse/BEAM-11677">BEAM-11677</a>)</li> |
| <li>S3 options does not provided to boto3 client while using FlinkRunner and Beam worker pool container (<a href="https://issues.apache.org/jira/browse/BEAM-11799">BEAM-11799</a>)</li> |
| <li>HDFS not deduplicating identical configuration paths (<a href="https://issues.apache.org/jira/browse/BEAM-11329">BEAM-11329</a>)</li> |
| <li>Hash Functions in BeamSQL (<a href="https://issues.apache.org/jira/browse/BEAM-10074">BEAM-10074</a>)</li> |
| <li>Create ApproximateDistinct using HLL Impl (<a href="https://issues.apache.org/jira/browse/BEAM-10324">BEAM-10324</a>)</li> |
| <li>Add Beam schema support to ParquetIO (<a href="https://issues.apache.org/jira/browse/BEAM-11526">BEAM-11526</a>)</li> |
| <li>Add a Deque Encoder (<a href="https://issues.apache.org/jira/browse/BEAM-11538">BEAM-11538</a>)</li> |
| <li>Hash functions in ZetaSQL (<a href="https://issues.apache.org/jira/browse/BEAM-11624">BEAM-11624</a>)</li> |
| <li>Refactor ParquetTableProvider (<a href="https://issues.apache.org/jira/browse/"></a>)</li> |
| <li>Add JVM properties to JavaJobServer (<a href="https://issues.apache.org/jira/browse/BEAM-8344">BEAM-8344</a>)</li> |
| <li>Single source of truth for supported Flink versions (<a href="https://issues.apache.org/jira/browse/"></a>)</li> |
| <li>Use metric for Python BigQuery streaming insert API latency logging (<a href="https://issues.apache.org/jira/browse/BEAM-11018">BEAM-11018</a>)</li> |
| <li>Use metric for Java BigQuery streaming insert API latency logging (<a href="https://issues.apache.org/jira/browse/BEAM-11032">BEAM-11032</a>)</li> |
| <li>Upgrade Flink runner to Flink versions 1.12.1 and 1.11.3 (<a href="https://issues.apache.org/jira/browse/BEAM-11697">BEAM-11697</a>)</li> |
| <li>Upgrade Beam base image to use Tensorflow 2.4.1 (<a href="https://issues.apache.org/jira/browse/BEAM-11762">BEAM-11762</a>)</li> |
| <li>Create Beam GCP BOM (<a href="https://issues.apache.org/jira/browse/BEAM-11665">BEAM-11665</a>)</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>The Java artifacts &ldquo;beam-sdks-java-io-kinesis&rdquo;, &ldquo;beam-sdks-java-io-google-cloud-platform&rdquo;, and |
| &ldquo;beam-sdks-java-extensions-sql-zetasql&rdquo; declare Guava 30.1-jre dependency (It was 25.1-jre in Beam 2.27.0). |
| This new Guava version may introduce dependency conflicts if your project or dependencies rely |
| on removed APIs. If affected, ensure to use an appropriate Guava version via <code>dependencyManagement</code> in Maven and |
| <code>force</code> in Gradle.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.28.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, Alex Amato, Alexey Romanenko, Allen Pradeep Xavier, Anant Damle, Artur Khanin, |
| Boyuan Zhang, Brian Hulette, Chamikara Jayalath, Chris Roth, Costi Ciudatu, Damon Douglas, |
| Daniel Collins, Daniel Oliveira, David Cavazos, David Huntsperger, Elliotte Rusty Harold, |
| Emily Ye, Etienne Chauchot, Etta Rapp, Evan Palmer, Eyal, Filip Krakowski, Fokko Driesprong, |
| Heejong Lee, Ismaël Mejía, janeliulwq, Jan Lukavský, John Edmonds, Jozef Vilcek, Kenneth Knowles |
| Ke Wu, kileys, Kyle Weaver, MabelYC, masahitojp, Masato Nakamura, Milena Bukal, Miraç Vuslat Başaran, |
| Nelson Osacky, Niel Markwick, Ning Kang, omarismail94, Pablo Estrada, Piotr Szuberski, |
| ramazan-yapparov, Reuven Lax, Reza Rokni, rHermes, Robert Bradshaw, Robert Burke, Robert Gruener, |
| Romster, Rui Wang, Sam Whittle, shehzaadn-vd, Siyuan Chen, Sonam Ramchand, Tobiasz Kędzierski, |
| Tomo Suzuki, tszerszen, tvalentyn, Tyson Hamilton, Udi Meiri, Xinbin Huang, Yichi Zhang, |
| Yifan Mai, yoshiki.obata, Yueyang Qiu, Yusaku Matsuki</p></description></item><item><title>Blog: Apache Beam 2.27.0</title><link>/blog/beam-2.27.0/</link><pubDate>Thu, 07 Jan 2021 12:00:00 -0800</pubDate><guid>/blog/beam-2.27.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.27.0 release of Apache Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2270-2020-12-22">download page</a> for this release. |
| For more information on changes in 2.27.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12349380">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Java 11 Containers are now published with all Beam releases.</li> |
| <li>There is a new transform <code>ReadAllFromBigQuery</code> that can receive multiple requests to read data from BigQuery at pipeline runtime. See <a href="https://github.com/apache/beam/pull/13170">PR 13170</a>, and <a href="https://issues.apache.org/jira/browse/BEAM-9650">BEAM-9650</a>.</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>ReadFromMongoDB can now be used with MongoDB Atlas (Python) (<a href="https://issues.apache.org/jira/browse/BEAM-11266">BEAM-11266</a>.)</li> |
| <li>ReadFromMongoDB/WriteToMongoDB will mask password in display_data (Python) (<a href="https://issues.apache.org/jira/browse/BEAM-11444">BEAM-11444</a>.)</li> |
| <li>There is a new transform <code>ReadAllFromBigQuery</code> that can receive multiple requests to read data from BigQuery at pipeline runtime. See <a href="https://github.com/apache/beam/pull/13170">PR 13170</a>, and <a href="https://issues.apache.org/jira/browse/BEAM-9650">BEAM-9650</a>.</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Beam modules that depend on Hadoop are now tested for compatibility with Hadoop 3 (<a href="https://issues.apache.org/jira/browse/BEAM-8569">BEAM-8569</a>). (Hive/HCatalog pending)</li> |
| <li>Publishing Java 11 SDK container images now supported as part of Apache Beam release process. (<a href="https://issues.apache.org/jira/browse/BEAM-8106">BEAM-8106</a>)</li> |
| <li>Added Cloud Bigtable Provider extension to Beam SQL (<a href="https://issues.apache.org/jira/browse/BEAM-11173">BEAM-11173</a>, <a href="https://issues.apache.org/jira/browse/BEAM-11373">BEAM-11373</a>)</li> |
| <li>Added a schema provider for thrift data (<a href="https://issues.apache.org/jira/browse/BEAM-11338">BEAM-11338</a>)</li> |
| <li>Added combiner packing pipeline optimization to Dataflow runner. (<a href="https://issues.apache.org/jira/browse/BEAM-10641">BEAM-10641</a>)</li> |
| <li>Added an example to ingest data from Apache Kafka to Google Pub/Sub. (<a href="https://issues.apache.org/jira/browse/BEAM-11065">BEAM-11065</a>)</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>HBaseIO hbase-shaded-client dependency should be now provided by the users (<a href="https://issues.apache.org/jira/browse/BEAM-9278">BEAM-9278</a>).</li> |
| <li><code>--region</code> flag in amazon-web-services2 was replaced by <code>--awsRegion</code> (<a href="https://issues.apache.org/jira/projects/BEAM/issues/BEAM-11331">BEAM-11331</a>).</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.27.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, Alan Myrvold, Alex Amato, Alexey Romanenko, Aliraza Nagamia, Allen Pradeep Xavier, |
| Andrew Pilloud, andreyKaparulin, Ashwin Ramaswami, Boyuan Zhang, Brent Worden, Brian Hulette, |
| Carlos Marin, Chamikara Jayalath, Costi Ciudatu, Damon Douglas, Daniel Collins, |
| Daniel Oliveira, David Huntsperger, David Lu, David Moravek, David Wrede, |
| dennis, Dennis Yung, dpcollins-google, Emily Ye, emkornfield, |
| Esun Kim, Etienne Chauchot, Eugene Nikolaiev, Frank Zhao, Haizhou Zhao, |
| Hector Acosta, Heejong Lee, Ilya, Iñigo San Jose Visiers, InigoSJ, |
| Ismaël Mejía, janeliulwq, Jan Lukavský, Kamil Wasilewski, Kenneth Jung, |
| Kenneth Knowles, Ke Wu, kileys, Kyle Weaver, lostluck, |
| Matt Casters, Maximilian Michels, Michal Walenia, Mike Dewar, nehsyc, |
| Nelson Osacky, Niels Basjes, Ning Kang, Pablo Estrada, palmere-google, |
| Pawel Pasterz, Piotr Szuberski, purbanow, Reuven Lax, rHermes, |
| Robert Bradshaw, Robert Burke, Rui Wang, Sam Rohde, Sam Whittle, |
| Siyuan Chen, Tim Robertson, Tobiasz Kędzierski, tszerszen, |
| Valentyn Tymofieiev, Tyson Hamilton, Udi Meiri, vachan-shetty, Xinyu Liu, |
| Yichi Zhang, Yifan Mai, yoshiki.obata, Yueyang Qiu</p></description></item><item><title>Blog: Apache Beam 2.26.0</title><link>/blog/beam-2.26.0/</link><pubDate>Fri, 11 Dec 2020 12:00:00 -0800</pubDate><guid>/blog/beam-2.26.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.26.0 release of Apache Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2260-2020-12-11">download page</a> for this release.</p> |
| <p>For more information on changes in 2.26.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12348833">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Splittable DoFn is now the default for executing the Read transform for Java based runners (Spark with bounded pipelines) in addition to existing runners from the 2.25.0 release (Direct, Flink, Jet, Samza, Twister2). The expected output of the Read transform is unchanged. Users can opt-out using <code>--experiments=use_deprecated_read</code>. The Apache Beam community is looking for feedback for this change as the community is planning to make this change permanent with no opt-out. If you run into an issue requiring the opt-out, please send an e-mail to <a href="mailto:user@beam.apache.org">user@beam.apache.org</a> specifically referencing BEAM-10670 in the subject line and why you needed to opt-out. (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-10670">BEAM-10670</a>)</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Java BigQuery streaming inserts now have timeouts enabled by default. Pass <code>--HTTPWriteTimeout=0</code> to revert to the old behavior. (<a href="https://issues.apache.org/jira/browse/BEAM-6103">BEAM-6103</a>)</li> |
| <li>Added support for Contextual Text IO (Java), a version of text IO that provides metadata about the records (<a href="https://issues.apache.org/jira/browse/BEAM-10124">BEAM-10124</a>). Support for this IO is currently experimental. Specifically, <strong>there are no update-compatibility guarantees</strong> for streaming jobs with this IO between current future verisons of Apache Beam SDK.</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Added support for avro payload format in Beam SQL Kafka Table (<a href="https://issues.apache.org/jira/browse/BEAM-10885">BEAM-10885</a>)</li> |
| <li>Added support for json payload format in Beam SQL Kafka Table (<a href="https://issues.apache.org/jira/browse/BEAM-10893">BEAM-10893</a>)</li> |
| <li>Added support for protobuf payload format in Beam SQL Kafka Table (<a href="https://issues.apache.org/jira/browse/BEAM-10892">BEAM-10892</a>)</li> |
| <li>Added support for avro payload format in Beam SQL Pubsub Table (<a href="https://issues.apache.org/jira/browse/BEAM-5504">BEAM-5504</a>)</li> |
| <li>Added option to disable unnecessary copying between operators in Flink Runner (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-11146">BEAM-11146</a>)</li> |
| <li>Added CombineFn.setup and CombineFn.teardown to Python SDK. These methods let you initialize the CombineFn&rsquo;s state before any of the other methods of the CombineFn is executed and clean that state up later on. If you are using Dataflow, you need to enable Dataflow Runner V2 by passing <code>--experiments=use_runner_v2</code> before using this feature. (<a href="https://issues.apache.org/jira/browse/BEAM-3736">BEAM-3736</a>)</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>BigQuery&rsquo;s DATETIME type now maps to Beam logical type org.apache.beam.sdk.schemas.logicaltypes.SqlTypes.DATETIME</li> |
| <li>Pandas 1.x is now required for dataframe operations.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.26.0 release. Thank you to all contributors!</p> |
| <p>Abhishek Yadav, AbhiY98, Ahmet Altay, Alan Myrvold, Alex Amato, Alexey Romanenko, |
| Andrew Pilloud, Ankur Goenka, Boyuan Zhang, Brian Hulette, Chad Dombrova, |
| Chamikara Jayalath, Curtis &ldquo;Fjord&rdquo; Hawthorne, Damon Douglas, dandy10, Daniel Oliveira, |
| David Cavazos, dennis, Derrick Qin, dpcollins-google, Dylan Hercher, emily, Esun Kim, |
| Gleb Kanterov, Heejong Lee, Ismaël Mejía, Jan Lukavský, Jean-Baptiste Onofré, Jing, |
| Jozef Vilcek, Justin White, Kamil Wasilewski, Kenneth Knowles, kileys, Kyle Weaver, |
| lostluck, Luke Cwik, Mark, Maximilian Michels, Milan Cermak, Mohammad Hossein Sekhavat, |
| Nelson Osacky, Neville Li, Ning Kang, pabloem, Pablo Estrada, pawelpasterz, |
| Pawel Pasterz, Piotr Szuberski, PoojaChandak, purbanow, rarokni, Ravi Magham, |
| Reuben van Ammers, Reuven Lax, Reza Rokni, Robert Bradshaw, Robert Burke, |
| Romain Manni-Bucau, Rui Wang, rworley-monster, Sam Rohde, Sam Whittle, shollyman, |
| Simone Primarosa, Siyuan Chen, Steve Niemitz, Steven van Rossum, sychen, Teodor Spæren, |
| Tim Clemons, Tim Robertson, Tobiasz Kędzierski, tszerszen, Tudor Marian, tvalentyn, |
| Tyson Hamilton, Udi Meiri, Vasu Gupta, xasm83, Yichi Zhang, yichuan66, Yifan Mai, |
| yoshiki.obata, Yueyang Qiu, yukihira1992</p></description></item><item><title>Blog: Apache Beam 2.25.0</title><link>/blog/beam-2.25.0/</link><pubDate>Fri, 23 Oct 2020 14:00:00 -0800</pubDate><guid>/blog/beam-2.25.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.25.0 release of Apache Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2250-2020-10-23">download page</a> for this release.</p> |
| <p>For more information on changes in 2.25.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12347147">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Splittable DoFn is now the default for executing the Read transform for Java based runners (Direct, Flink, Jet, Samza, Twister2). The expected output of the Read transform is unchanged. Users can opt-out using <code>--experiments=use_deprecated_read</code>. The Apache Beam community is looking for feedback for this change as the community is planning to make this change permanent with no opt-out. If you run into an issue requiring the opt-out, please send an e-mail to <a href="mailto:user@beam.apache.org">user@beam.apache.org</a> specifically referencing BEAM-10670 in the subject line and why you needed to opt-out. (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-10670">BEAM-10670</a>)</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Added cross-language support to Java&rsquo;s KinesisIO, now available in the Python module <code>apache_beam.io.kinesis</code> (<a href="https://issues.apache.org/jira/browse/BEAM-10138">BEAM-10138</a>, <a href="https://issues.apache.org/jira/browse/BEAM-10137">BEAM-10137</a>).</li> |
| <li>Update Snowflake JDBC dependency for SnowflakeIO (<a href="https://issues.apache.org/jira/browse/BEAM-10864">BEAM-10864</a>)</li> |
| <li>Added cross-language support to Java&rsquo;s SnowflakeIO.Write, now available in the Python module <code>apache_beam.io.snowflake</code> (<a href="https://issues.apache.org/jira/browse/BEAM-9898">BEAM-9898</a>).</li> |
| <li>Added delete function to Java&rsquo;s <code>ElasticsearchIO#Write</code>. Now, Java&rsquo;s ElasticsearchIO can be used to selectively delete documents using <code>withIsDeleteFn</code> function (<a href="https://issues.apache.org/jira/browse/BEAM-5757">BEAM-5757</a>).</li> |
| <li>Java SDK: Added new IO connector for InfluxDB - InfluxDbIO (<a href="https://issues.apache.org/jira/browse/BEAM-2546">BEAM-2546</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Support for repeatable fields in JSON decoder for <code>ReadFromBigQuery</code> added. (Python) (<a href="https://issues.apache.org/jira/browse/BEAM-10524">BEAM-10524</a>)</li> |
| <li>Added an opt-in, performance-driven runtime type checking system for the Python SDK (<a href="https://issues.apache.org/jira/browse/BEAM-10549">BEAM-10549</a>). |
| More details will be in an upcoming <a href="/blog/python-performance-runtime-type-checking/index.html">blog post</a>.</li> |
| <li>Added support for Python 3 type annotations on PTransforms using typed PCollections (<a href="https://issues.apache.org/jira/browse/BEAM-10258">BEAM-10258</a>). |
| More details will be in an upcoming <a href="/blog/python-improved-annotations/index.html">blog post</a>.</li> |
| <li>Improved the Interactive Beam API where recording streaming jobs now start a long running background recording job. Running ib.show() or ib.collect() samples from the recording (<a href="https://issues.apache.org/jira/browse/BEAM-10603">BEAM-10603</a>).</li> |
| <li>In Interactive Beam, ib.show() and ib.collect() now have &ldquo;n&rdquo; and &ldquo;duration&rdquo; as parameters. These mean read only up to &ldquo;n&rdquo; elements and up to &ldquo;duration&rdquo; seconds of data read from the recording (<a href="https://issues.apache.org/jira/browse/BEAM-10603">BEAM-10603</a>).</li> |
| <li>Initial preview of <a href="https://s.apache.org/simpler-python-pipelines-2020#slide=id.g905ac9257b_1_21">Dataframes</a> support. |
| See also example at apache_beam/examples/wordcount_dataframe.py</li> |
| <li>Fixed support for type hints on <code>@ptransform_fn</code> decorators in the Python SDK. |
| (<a href="https://issues.apache.org/jira/browse/BEAM-4091">BEAM-4091</a>) |
| This has not enabled by default to preserve backwards compatibility; use the |
| <code>--type_check_additional=ptransform_fn</code> flag to enable. It may be enabled by |
| default in future versions of Beam.</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>Python 2 and Python 3.5 support dropped (<a href="https://issues.apache.org/jira/browse/BEAM-10644">BEAM-10644</a>, <a href="https://issues.apache.org/jira/browse/BEAM-9372">BEAM-9372</a>).</li> |
| <li>Pandas 1.x allowed. Older version of Pandas may still be used, but may not be as well tested.</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>Python transform ReadFromSnowflake has been moved from <code>apache_beam.io.external.snowflake</code> to <code>apache_beam.io.snowflake</code>. The previous path will be removed in the future versions.</li> |
| </ul> |
| <h2 id="known-issues">Known Issues</h2> |
| <ul> |
| <li>Dataflow streaming timers once against not strictly time ordered when set earlier mid-bundle, as the fix for <a href="https://issues.apache.org/jira/browse/BEAM-8543">BEAM-8543</a> introduced more severe bugs and has been rolled back.</li> |
| <li>Default compressor change breaks dataflow python streaming job update compatibility. Please use python SDK version &lt;= 2.23.0 or &gt; 2.25.0 if job update is critical.(<a href="https://issues.apache.org/jira/browse/BEAM-11113">BEAM-11113</a>)</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.25.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, Alan Myrvold, Aldair Coronel Ruiz, Alexey Romanenko, Andrew Pilloud, Ankur Goenka, |
| Ayoub ENNASSIRI, Bipin Upadhyaya, Boyuan Zhang, Brian Hulette, Brian Michalski, Chad Dombrova, |
| Chamikara Jayalath, Damon Douglas, Daniel Oliveira, David Cavazos, David Janicek, Doug Roeper, Eric |
| Roshan-Eisner, Etta Rapp, Eugene Kirpichov, Filipe Regadas, Heejong Lee, Ihor Indyk, Irvi Firqotul |
| Aini, Ismaël Mejía, Jan Lukavský, Jayendra, Jiadai Xia, Jithin Sukumar, Jozsef Bartok, Kamil |
| Gałuszka, Kamil Wasilewski, Kasia Kucharczyk, Kenneth Jung, Kenneth Knowles, Kevin Puthusseri, Kevin |
| Sijo Puthusseri, KevinGG, Kyle Weaver, Leiyi Zhang, Lourens Naudé, Luke Cwik, Matthew Ouyang, |
| Maximilian Michels, Michal Walenia, Milan Cermak, Monica Song, Nelson Osacky, Neville Li, Ning Kang, |
| Pablo Estrada, Piotr Szuberski, Qihang, Rehman, Reuven Lax, Robert Bradshaw, Robert Burke, Rui Wang, |
| Saavan Nanavati, Sam Bourne, Sam Rohde, Sam Whittle, Sergiy Kolesnikov, Sindy Li, Siyuan Chen, Steve |
| Niemitz, Terry Xian, Thomas Weise, Tobiasz Kędzierski, Truc Le, Tyson Hamilton, Udi Meiri, Valentyn |
| Tymofieiev, Yichi Zhang, Yifan Mai, Yueyang Qiu, annaqin418, danielxjd, dennis, dp, fuyuwei, |
| lostluck, nehsyc, odeshpande, odidev, pulasthi, purbanow, rworley-monster, sclukas77, terryxian78, |
| tvalentyn, yoshiki.obata</p></description></item><item><title>Blog: Apache Beam 2.24.0</title><link>/blog/beam-2.24.0/</link><pubDate>Fri, 18 Sep 2020 00:00:01 -0800</pubDate><guid>/blog/beam-2.24.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.24.0 release of Apache Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2240-2020-09-18">download page</a> for this release.</p> |
| <p>For more information on changes in 2.24.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12347146">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Apache Beam 2.24.0 is the last release with Python 2 and Python 3.5 |
| support.</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>New overloads for BigtableIO.Read.withKeyRange() and BigtableIO.Read.withRowFilter() |
| methods that take ValueProvider as a parameter (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-10283">BEAM-10283</a>).</li> |
| <li>The WriteToBigQuery transform (Python) in Dataflow Batch no longer relies on BigQuerySink by default. It relies on |
| a new, fully-featured transform based on file loads into BigQuery. To revert the behavior to the old implementation, |
| you may use <code>--experiments=use_legacy_bq_sink</code>.</li> |
| <li>Add cross-language support to Java&rsquo;s JdbcIO, now available in the Python module <code>apache_beam.io.jdbc</code> (<a href="https://issues.apache.org/jira/browse/BEAM-10135">BEAM-10135</a>, <a href="https://issues.apache.org/jira/browse/BEAM-10136">BEAM-10136</a>).</li> |
| <li>Add support of AWS SDK v2 for KinesisIO.Read (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-9702">BEAM-9702</a>).</li> |
| <li>Add streaming support to SnowflakeIO in Java SDK (<a href="https://issues.apache.org/jira/browse/BEAM-9896">BEAM-9896</a>)</li> |
| <li>Support reading and writing to Google Healthcare DICOM APIs in Python SDK (<a href="https://issues.apache.org/jira/browse/BEAM-10601">BEAM-10601</a>)</li> |
| <li>Add dispositions for SnowflakeIO.write (<a href="https://issues.apache.org/jira/browse/BEAM-10343">BEAM-10343</a>)</li> |
| <li>Add cross-language support to SnowflakeIO.Read now available in the Python module <code>apache_beam.io.external.snowflake</code> (<a href="https://issues.apache.org/jira/browse/BEAM-9897">BEAM-9897</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Shared library for simplifying management of large shared objects added to Python SDK. Example use case is sharing a large TF model object across threads (<a href="https://issues.apache.org/jira/browse/BEAM-10417">BEAM-10417</a>).</li> |
| <li>Dataflow streaming timers are not strictly time ordered when set earlier mid-bundle (<a href="https://issues.apache.org/jira/browse/BEAM-8543">BEAM-8543</a>).</li> |
| <li>OnTimerContext should not create a new one when processing each element/timer in FnApiDoFnRunner (<a href="https://issues.apache.org/jira/browse/BEAM-9839">BEAM-9839</a>)</li> |
| <li>Key should be available in @OnTimer methods (Spark Runner) (<a href="https://issues.apache.org/jira/browse/BEAM-9850">BEAM-9850</a>)</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>WriteToBigQuery transforms now require a GCS location to be provided through either |
| custom_gcs_temp_location in the constructor of WriteToBigQuery or the fallback option |
| &ndash;temp_location, or pass method=&ldquo;STREAMING_INSERTS&rdquo; to WriteToBigQuery (<a href="https://issues.apache.org/jira/browse/BEAM-6928">BEAM-6928</a>).</li> |
| <li>Python SDK now understands <code>typing.FrozenSet</code> type hints, which are not interchangeable with <code>typing.Set</code>. You may need to update your pipelines if type checking fails. (<a href="https://issues.apache.org/jira/browse/BEAM-10197">BEAM-10197</a>)</li> |
| </ul> |
| <h2 id="known-issues">Known Issues</h2> |
| <ul> |
| <li>Default compressor change breaks dataflow python streaming job update compatibility. Please use python SDK version &lt;= 2.23.0 or &gt; 2.25.0 if job update is critical.(<a href="https://issues.apache.org/jira/browse/BEAM-11113">BEAM-11113</a>)</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.24.0 release. Thank you to all contributors!</p> |
| <p>adesormi, Ahmet Altay, Alex Amato, Alexey Romanenko, Andrew Pilloud, Ashwin Ramaswami, Borzoo, |
| Boyuan Zhang, Brian Hulette, Brian M, Bu Sun Kim, Chamikara Jayalath, Colm O hEigeartaigh, |
| Corvin Deboeser, Damian Gadomski, Damon Douglas, Daniel Oliveira, Dariusz Aniszewski, |
| davidak09, David Cavazos, David Moravek, David Yan, dhodun, Doug Roeper, Emil Hessman, Emily Ye, |
| Etienne Chauchot, Etta Rapp, Eugene Kirpichov, fuyuwei, Gleb Kanterov, |
| Harrison Green, Heejong Lee, Henry Suryawirawan, InigoSJ, Ismaël Mejía, Israel Herraiz, |
| Jacob Ferriero, Jan Lukavský, Jayendra, jfarr, jhnmora000, Jiadai Xia, JIahao wu, Jie Fan, |
| Jiyong Jung, Julius Almeida, Kamil Gałuszka, Kamil Wasilewski, Kasia Kucharczyk, Kenneth Knowles, |
| Kevin Puthusseri, Kyle Weaver, Łukasz Gajowy, Luke Cwik, Mark-Zeng, Maximilian Michels, |
| Michal Walenia, Niel Markwick, Ning Kang, Pablo Estrada, pawel.urbanowicz, Piotr Szuberski, |
| Rafi Kamal, rarokni, Rehman Murad Ali, Reuben van Ammers, Reuven Lax, Ricardo Bordon, |
| Robert Bradshaw, Robert Burke, Robin Qiu, Rui Wang, Saavan Nanavati, sabhyankar, Sam Rohde, |
| Scott Lukas, Siddhartha Thota, Simone Primarosa, Sławomir Andrian, |
| Steve Niemitz, Tobiasz Kędzierski, Tomo Suzuki, Tyson Hamilton, Udi Meiri, |
| Valentyn Tymofieiev, viktorjonsson, Xinyu Liu, Yichi Zhang, Yixing Zhang, yoshiki.obata, |
| Yueyang Qiu, zijiesong</p></description></item><item><title>Blog: Apache Beam 2.23.0</title><link>/blog/beam-2.23.0/</link><pubDate>Wed, 29 Jul 2020 00:00:01 -0800</pubDate><guid>/blog/beam-2.23.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.23.0 release of Apache Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2230-2020-07-29">download page</a> for this release.</p> |
| <p>For more information on changes in 2.23.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12347145">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Twister2 Runner (<a href="https://issues.apache.org/jira/browse/BEAM-7304">BEAM-7304</a>).</li> |
| <li>Python 3.8 support (<a href="https://issues.apache.org/jira/browse/BEAM-8494">BEAM-8494</a>).</li> |
| </ul> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Support for reading from Snowflake added (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-9722">BEAM-9722</a>).</li> |
| <li>Support for writing to Splunk added (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-8596">BEAM-8596</a>).</li> |
| <li>Support for assume role added (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-10335">BEAM-10335</a>).</li> |
| <li>A new transform to read from BigQuery has been added: <code>apache_beam.io.gcp.bigquery.ReadFromBigQuery</code>. This transform |
| is experimental. It reads data from BigQuery by exporting data to Avro files, and reading those files. It also supports |
| reading data by exporting to JSON files. This has small differences in behavior for Time and Date-related fields. See |
| Pydoc for more information.</li> |
| <li>Add dispositions for SnowflakeIO.write (<a href="https://issues.apache.org/jira/browse/BEAM-10343">BEAM-10343</a>)</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li>Update Snowflake JDBC dependency and add application=beam to connection URL (<a href="https://issues.apache.org/jira/browse/BEAM-10383">BEAM-10383</a>).</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li><code>RowJson.RowJsonDeserializer</code>, <code>JsonToRow</code>, and <code>PubsubJsonTableProvider</code> now accept &ldquo;implicit |
| nulls&rdquo; by default when deserializing JSON (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-10220">BEAM-10220</a>). |
| Previously nulls could only be represented with explicit null values, as in |
| <code>{&quot;foo&quot;: &quot;bar&quot;, &quot;baz&quot;: null}</code>, whereas an implicit null like <code>{&quot;foo&quot;: &quot;bar&quot;}</code> would raise an |
| exception. Now both JSON strings will yield the same result by default. This behavior can be |
| overridden with <code>RowJson.RowJsonDeserializer#withNullBehavior</code>.</li> |
| <li>Fixed a bug in <code>GroupIntoBatches</code> experimental transform in Python to actually group batches by key. |
| This changes the output type for this transform (<a href="https://issues.apache.org/jira/browse/BEAM-6696">BEAM-6696</a>).</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>Remove Gearpump runner. (<a href="https://issues.apache.org/jira/browse/BEAM-9999">BEAM-9999</a>)</li> |
| <li>Remove Apex runner. (<a href="https://issues.apache.org/jira/browse/BEAM-9999">BEAM-9999</a>)</li> |
| <li>RedisIO.readAll() is deprecated and will be removed in 2 versions, users must use RedisIO.readKeyPatterns() as a replacement (<a href="https://issues.apache.org/jira/browse/BEAM-9747">BEAM-9747</a>).</li> |
| </ul> |
| <h2 id="known-issues">Known Issues</h2> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.23.0 release. Thank you to all contributors!</p> |
| <p>Aaron, Abhishek Yadav, Ahmet Altay, aiyangar, Aizhamal Nurmamat kyzy, Ajo Thomas, Akshay-Iyangar, Alan Pryor, Alex Amato, Alexey Romanenko, Allen Pradeep Xavier, Andrew Crites, Andrew Pilloud, Ankur Goenka, Anna Qin, Ashwin Ramaswami, bntnam, Borzoo Esmailloo, Boyuan Zhang, Brian Hulette, Brian Michalski, brucearctor, Chamikara Jayalath, chi-chi weng, Chuck Yang, Chun Yang, Colm O hEigeartaigh, Corvin Deboeser, Craig Chambers, Damian Gadomski, Damon Douglas, Daniel Oliveira, Dariusz Aniszewski, darshanj, darshan jani, David Cavazos, David Moravek, David Yan, Esun Kim, Etienne Chauchot, Filipe Regadas, fuyuwei, Graeme Morgan, Hannah-Jiang, Harch Vardhan, Heejong Lee, Henry Suryawirawan, InigoSJ, Ismaël Mejía, Israel Herraiz, Jacob Ferriero, Jan Lukavský, Jie Fan, John Mora, Jozef Vilcek, Julien Phalip, Justine Koa, Kamil Gabryjelski, Kamil Wasilewski, Kasia Kucharczyk, Kenneth Jung, Kenneth Knowles, kevingg, Kevin Sijo Puthusseri, kshivvy, Kyle Weaver, Kyoungha Min, Kyungwon Jo, Luke Cwik, Mark Liu, Mark-Zeng, Matthias Baetens, Maximilian Michels, Michal Walenia, Mikhail Gryzykhin, Nam Bui, Nathan Fisher, Niel Markwick, Ning Kang, Omar Ismail, Pablo Estrada, paul fisher, Pawel Pasterz, perkss, Piotr Szuberski, pulasthi, purbanow, Rahul Patwari, Rajat Mittal, Rehman, Rehman Murad Ali, Reuben van Ammers, Reuven Lax, Reza Rokni, Rion Williams, Robert Bradshaw, Robert Burke, Rui Wang, Ruoyun Huang, sabhyankar, Sam Rohde, Sam Whittle, sclukas77, Sebastian Graca, Shoaib Zafar, Sruthi Sree Kumar, Stephen O&rsquo;Kennedy, Steve Koonce, Steve Niemitz, Steven van Rossum, Ted Romer, Tesio, Thinh Ha, Thomas Weise, Tobias Kaymak, tobiaslieber-cognitedata, Tobiasz Kędzierski, Tomo Suzuki, Tudor Marian, tvs, Tyson Hamilton, Udi Meiri, Valentyn Tymofieiev, Vasu Nori, xuelianhan, Yichi Zhang, Yifan Zou, Yixing Zhang, yoshiki.obata, Yueyang Qiu, Yu Feng, Yuwei Fu, Zhuo Peng, ZijieSong946.</p></description></item><item><title>Blog: Apache Beam 2.22.0</title><link>/blog/beam-2.22.0/</link><pubDate>Mon, 08 Jun 2020 00:00:01 -0800</pubDate><guid>/blog/beam-2.22.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.22.0 release of Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2220-2020-06-08">download page</a> for this release.</p> |
| <p>For more information on changes in 2.22.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12347144">detailed release notes</a>.</p> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Basic Kafka read/write support for DataflowRunner (Python) (<a href="https://issues.apache.org/jira/browse/BEAM-8019">BEAM-8019</a>).</li> |
| <li>Sources and sinks for Google Healthcare APIs (Java)(<a href="https://issues.apache.org/jira/browse/BEAM-9468">BEAM-9468</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li><code>--workerCacheMB</code> flag is supported in Dataflow streaming pipeline (<a href="https://issues.apache.org/jira/browse/BEAM-9964">BEAM-9964</a>)</li> |
| <li><code>--direct_num_workers=0</code> is supported for FnApi runner. It will set the number of threads/subprocesses to number of cores of the machine executing the pipeline (<a href="https://issues.apache.org/jira/browse/BEAM-9443">BEAM-9443</a>).</li> |
| <li>Python SDK now has experimental support for SqlTransform (<a href="https://issues.apache.org/jira/browse/BEAM-8603">BEAM-8603</a>).</li> |
| <li>Add OnWindowExpiration method to Stateful DoFn (<a href="https://issues.apache.org/jira/browse/BEAM-1589">BEAM-1589</a>).</li> |
| <li>Added PTransforms for Google Cloud DLP (Data Loss Prevention) services integration (<a href="https://issues.apache.org/jira/browse/BEAM-9723">BEAM-9723</a>): |
| <ul> |
| <li>Inspection of data,</li> |
| <li>Deidentification of data,</li> |
| <li>Reidentification of data.</li> |
| </ul> |
| </li> |
| <li>Add a more complete I/O support matrix in the documentation site (<a href="https://issues.apache.org/jira/browse/BEAM-9916">BEAM-9916</a>).</li> |
| <li>Upgrade Sphinx to 3.0.3 for building PyDoc.</li> |
| <li>Added a PTransform for image annotation using Google Cloud AI image processing service |
| (<a href="https://issues.apache.org/jira/browse/BEAM-9646">BEAM-9646</a>)</li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>The Python SDK now requires <code>--job_endpoint</code> to be set when using <code>--runner=PortableRunner</code> (<a href="https://issues.apache.org/jira/browse/BEAM-9860">BEAM-9860</a>). Users seeking the old default behavior should set <code>--runner=FlinkRunner</code> instead.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.22.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, aiyangar, Ajo Thomas, Akshay-Iyangar, Alan Pryor, Alexey Romanenko, Allen Pradeep Xavier, amaliujia, Andrew Pilloud, Ankur Goenka, Ashwin Ramaswami, bntnam, Borzoo Esmailloo, Boyuan Zhang, Brian Hulette, Chamikara Jayalath, Colm O hEigeartaigh, Craig Chambers, Damon Douglas, Daniel Oliveira, David Cavazos, David Moravek, Esun Kim, Etienne Chauchot, Filipe Regadas, Graeme Morgan, Hannah Jiang, Hannah-Jiang, Harch Vardhan, Heejong Lee, Henry Suryawirawan, Ismaël Mejía, Israel Herraiz, Jacob Ferriero, Jan Lukavský, John Mora, Kamil Wasilewski, Kenneth Jung, Kenneth Knowles, kevingg, Kyle Weaver, Kyoungha Min, Kyungwon Jo, Luke Cwik, Mark Liu, Matthias Baetens, Maximilian Michels, Michal Walenia, Mikhail Gryzykhin, Nam Bui, Niel Markwick, Ning Kang, Omar Ismail, omarismail94, Pablo Estrada, paul fisher, pawelpasterz, Pawel Pasterz, Piotr Szuberski, Rahul Patwari, rarokni, Rehman, Rehman Murad Ali, Reuven Lax, Robert Bradshaw, Robert Burke, Rui Wang, Ruoyun Huang, Sam Rohde, Sam Whittle, Sebastian Graca, Shoaib Zafar, Sruthi Sree Kumar, Stephen O&rsquo;Kennedy, Steve Koonce, Steve Niemitz, Steven van Rossum, Tesio, Thomas Weise, tobiaslieber-cognitedata, Tomo Suzuki, Tudor Marian, tvalentyn, Tyson Hamilton, Udi Meiri, Valentyn Tymofieiev, Vasu Nori, xuelianhan, Yichi Zhang, Yifan Zou, yoshiki.obata, Yueyang Qiu, Zhuo Peng</p></description></item><item><title>Blog: Apache Beam 2.21.0</title><link>/blog/beam-2.21.0/</link><pubDate>Wed, 27 May 2020 00:00:01 -0800</pubDate><guid>/blog/beam-2.21.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.21.0 release of Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2210-2020-05-27">download page</a> for this release.</p> |
| <p>For more information on changes in 2.21.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12347143">detailed release notes</a>.</p> |
| <h2 id="ios">I/Os</h2> |
| <ul> |
| <li>Python: Deprecated module <code>apache_beam.io.gcp.datastore.v1</code> has been removed |
| as the client it uses is out of date and does not support Python 3 |
| (<a href="https://issues.apache.org/jira/browse/BEAM-9529">BEAM-9529</a>). |
| Please migrate your code to use |
| <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.datastore.v1new.datastoreio.html">apache_beam.io.gcp.datastore.<strong>v1new</strong></a>. |
| See the updated |
| <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py">datastore_wordcount</a> |
| for example usage.</li> |
| <li>Python SDK: Added integration tests and updated batch write functionality for Google Cloud Spanner transform (<a href="https://issues.apache.org/jira/browse/BEAM-8949">BEAM-8949</a>).</li> |
| </ul> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <ul> |
| <li> |
| <p>Python SDK will now use Python 3 type annotations as pipeline type hints. |
| (<a href="https://github.com/apache/beam/pull/10717">#10717</a>)</p> |
| <p>If you suspect that this feature is causing your pipeline to fail, calling |
| <code>apache_beam.typehints.disable_type_annotations()</code> before pipeline creation |
| will disable is completely, and decorating specific functions (such as |
| <code>process()</code>) with <code>@apache_beam.typehints.no_annotations</code> will disable it |
| for that function.</p> |
| <p>More details can be found in |
| <a href="/documentation/sdks/python-type-safety/">Ensuring Python Type Safety</a> |
| and the Python SDK Typing Changes |
| <a href="/blog/python-typing/">blog post</a>.</p> |
| </li> |
| <li> |
| <p>Java SDK: Introducing the concept of options in Beam Schema’s. These options add extra |
| context to fields and schemas. This replaces the current Beam metadata that is present |
| in a FieldType only, options are available in fields and row schemas. Schema options are |
| fully typed and can contain complex rows. <em>Remark: Schema aware is still experimental.</em> |
| (<a href="https://issues.apache.org/jira/browse/BEAM-9035">BEAM-9035</a>)</p> |
| </li> |
| <li> |
| <p>Java SDK: The protobuf extension is fully schema aware and also includes protobuf option |
| conversion to beam schema options. <em>Remark: Schema aware is still experimental.</em> |
| (<a href="https://issues.apache.org/jira/browse/BEAM-9044">BEAM-9044</a>)</p> |
| </li> |
| <li> |
| <p>Added ability to write to BigQuery via Avro file loads (Python) (<a href="https://issues.apache.org/jira/browse/BEAM-8841">BEAM-8841</a>)</p> |
| <p>By default, file loads will be done using JSON, but it is possible to |
| specify the temp_file_format parameter to perform file exports with AVRO. |
| AVRO-based file loads work by exporting Python types into Avro types, so |
| to switch to Avro-based loads, you will need to change your data types |
| from Json-compatible types (string-type dates and timestamp, long numeric |
| values as strings) into Python native types that are written to Avro |
| (Python&rsquo;s date, datetime types, decimal, etc). For more information |
| see <a href="https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro#avro_conversions">https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro#avro_conversions</a>.</p> |
| </li> |
| <li> |
| <p>Added integration of Java SDK with Google Cloud AI VideoIntelligence service |
| (<a href="https://issues.apache.org/jira/browse/BEAM-9147">BEAM-9147</a>)</p> |
| </li> |
| <li> |
| <p>Added integration of Java SDK with Google Cloud AI natural language processing API |
| (<a href="https://issues.apache.org/jira/browse/BEAM-9634">BEAM-9634</a>)</p> |
| </li> |
| <li> |
| <p><code>docker-pull-licenses</code> tag was introduced. Licenses/notices of third party dependencies will be added to the docker images when <code>docker-pull-licenses</code> was set. |
| The files are added to <code>/opt/apache/beam/third_party_licenses/</code>. |
| By default, no licenses/notices are added to the docker images. (<a href="https://issues.apache.org/jira/browse/BEAM-9136">BEAM-9136</a>)</p> |
| </li> |
| </ul> |
| <h2 id="breaking-changes">Breaking Changes</h2> |
| <ul> |
| <li>Dataflow runner now requires the <code>--region</code> option to be set, unless a default value is set in the environment (<a href="https://issues.apache.org/jira/browse/BEAM-9199">BEAM-9199</a>). See <a href="https://cloud.google.com/dataflow/docs/concepts/regional-endpoints">here</a> for more details.</li> |
| <li>HBaseIO.ReadAll now requires a PCollection of HBaseIO.Read objects instead of HBaseQuery objects (<a href="https://issues.apache.org/jira/browse/BEAM-9279">BEAM-9279</a>).</li> |
| <li>ProcessContext.updateWatermark has been removed in favor of using a WatermarkEstimator (<a href="https://issues.apache.org/jira/browse/BEAM-9430">BEAM-9430</a>).</li> |
| <li>Coder inference for PCollection of Row objects has been disabled (<a href="https://issues.apache.org/jira/browse/BEAM-9569">BEAM-9569</a>).</li> |
| <li>Go SDK docker images are no longer released until further notice.</li> |
| </ul> |
| <h2 id="deprecations">Deprecations</h2> |
| <ul> |
| <li>Java SDK: Beam Schema FieldType.getMetadata is now deprecated and is replaced by the Beam |
| Schema Options, it will be removed in version <code>2.23.0</code>. (<a href="https://issues.apache.org/jira/browse/BEAM-9704">BEAM-9704</a>)</li> |
| <li>The <code>--zone</code> option in the Dataflow runner is now deprecated. Please use <code>--worker_zone</code> instead. (<a href="https://issues.apache.org/jira/browse/BEAM-9716">BEAM-9716</a>)</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.21.0 release. Thank you to all contributors!</p> |
| <p>Aaron Meihm, Adrian Eka, Ahmet Altay, AldairCoronel, Alex Van Boxel, Alexey Romanenko, Andrew Crites, Andrew Pilloud, Ankur Goenka, Badrul (Taki) Chowdhury, Bartok Jozsef, Boyuan Zhang, Brian Hulette, brucearctor, bumblebee-coming, Chad Dombrova, Chamikara Jayalath, Chie Hayashida, Chris Gorgolewski, Chuck Yang, Colm O hEigeartaigh, Curtis &ldquo;Fjord&rdquo; Hawthorne, Daniel Mills, Daniel Oliveira, David Yan, Elias Djurfeldt, Emiliano Capoccia, Etienne Chauchot, Fernando Diaz, Filipe Regadas, Gleb Kanterov, Hai Lu, Hannah Jiang, Harch Vardhan, Heejong Lee, Henry Suryawirawan, Hk-tang, Ismaël Mejía, Jacoby, Jan Lukavský, Jeroen Van Goey, jfarr, Jozef Vilcek, Kai Jiang, Kamil Wasilewski, Kenneth Knowles, KevinGG, Kyle Weaver, Kyoungha Min, Luke Cwik, Maximilian Michels, Michal Walenia, Ning Kang, Pablo Estrada, paul fisher, Piotr Szuberski, Reuven Lax, Robert Bradshaw, Robert Burke, Rose Nguyen, Rui Wang, Sam Rohde, Sam Whittle, Spoorti Kundargi, Steve Koonce, sunjincheng121, Ted Yun, Tesio, Thomas Weise, Tomo Suzuki, Udi Meiri, Valentyn Tymofieiev, Vasu Nori, Yichi Zhang, yoshiki.obata, Yueyang Qiu</p></description></item><item><title>Blog: Apache Beam 2.20.0</title><link>/blog/beam-2.20.0/</link><pubDate>Wed, 15 Apr 2020 00:00:01 -0800</pubDate><guid>/blog/beam-2.20.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.20.0 release of Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2190-2020-02-04">download page</a> for this release.</p> |
| <p>For more information on changes in 2.20.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12346780">detailed release notes</a>.</p> |
| <h3 id="ios">I/Os</h3> |
| <p>Python SDK: . (#10223).</p> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8561">BEAM-8561</a> Adds support for Thrift encoded data via ThriftIO</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-7310">BEAM-7310</a> KafkaIO supports schema resolution using Confluent Schema Registry</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-7246">BEAM-7246</a> Support for Google Cloud Spanner. This is an experimental module for reading and writing data from Google Cloud Spanner</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8399">BEAM-8399</a> Adds support for standard HDFS URLs (with server name)</li> |
| </ul> |
| <h3 id="new-features--improvements">New Features / Improvements</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9146">BEAM-9146</a> New AnnotateVideo &amp; AnnotateVideoWithContext PTransform&rsquo;s that integrates GCP Video Intelligence functionality</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9247">BEAM-9247</a> New AnnotateImage &amp; AnnotateImageWithContext PTransform&rsquo;s for element-wise &amp; batch image annotation using Google Cloud Vision API</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9258">BEAM-9258</a> Added a PTransform for inspection and deidentification of text using Google Cloud DLP</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9248">BEAM-9248</a> New AnnotateText PTransform that integrates Google Cloud Natural Language functionality</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9305">BEAM-9305</a> ReadFromBigQuery now supports value providers for the query string</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8841">BEAM-8841</a> Added ability to write to BigQuery via Avro file loads</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9228">BEAM-9228</a> Direct runner for FnApi supports further parallelism</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8550">BEAM-8550</a> Support for @RequiresTimeSortedInput in Flink and Spark</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-6857">BEAM-6857</a> Added support for dynamic timers</li> |
| </ul> |
| <h3 id="breaking-changes">Breaking Changes</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-3453">BEAM-3453</a> Backwards incompatible change in ReadFromPubSub(topic=) in Python</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9310">BEAM-9310</a> SpannerAccessor in Java is now package-private to reduce API surface</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8616">BEAM-8616</a> ParquetIO hadoop dependency should be now provided by the users</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9063">BEAM-9063</a> Docker images will be deployed to apache/beam repositories from 2.20</li> |
| </ul> |
| <h3 id="bugfixes">Bugfixes</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9579">BEAM-9579</a> Fixed numpy operators in ApproximateQuantiles</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9277">BEAM-9277</a> Fixed exception when running in IPython notebook</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-1833">BEAM-1833</a> Restructure Python pipeline construction to better follow the Runner API</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9225">BEAM-9225</a> Fixed Flink uberjar job termination bug</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9503">BEAM-9503</a> Fixed SyntaxError in process worker startup</li> |
| <li>Various bug fixes and performance improvements.</li> |
| </ul> |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9322">BEAM-9322</a> Python SDK ignores manually set PCollection tags</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9445">BEAM-9445</a> Python SDK pre_optimize=all experiment may cause error</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9725">BEAM-9725</a> Python SDK performance regression for reshuffle transform</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.20.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, Alex Amato, Alexey Romanenko, Andrew Pilloud, Ankur Goenka, Anton Kedin, Boyuan Zhang, Brian Hulette, Brian Martin, Chamikara Jayalath |
| , Charles Chen, Craig Chambers, Daniel Oliveira, David Moravek, David Rieber, Dustin Rhodes, Etienne Chauchot, Gleb Kanterov, Hai Lu, Heejong Lee |
| , Ismaël Mejía, J Ross Thomson, Jan Lukavský, Jason Kuster, Jean-Baptiste Onofré, Jeff Klukas, João Cabrita, Juan Rael, Juta, Kasia Kucharczyk |
| , Kengo Seki, Kenneth Jung, Kenneth Knowles, Kyle Weaver, Kyle Winkelman, Lukas Drbal, Marek Simunek, Mark Liu, Maximilian Michels, Melissa Pashniak |
| , Michael Luckey, Michal Walenia, Mike Pedersen, Mikhail Gryzykhin, Niel Markwick, Pablo Estrada, Pascal Gula, Rehman Murad Ali, Reuven Lax, Rob, Robbe Sneyders |
| , Robert Bradshaw, Robert Burke, Rui Wang, Ruoyun Huang, Ryan Williams, Sam Rohde, Sam Whittle, Scott Wegner, Shoaib Zafar, Thomas Weise, Tianyang Hu, Tyler Akidau |
| , Udi Meiri, Valentyn Tymofieiev, Xinyu Liu, XuMingmin, ttanay, tvalentyn, Łukasz Gajowy</p></description></item><item><title>Blog: Apache Beam 2.19.0</title><link>/blog/beam-2.19.0/</link><pubDate>Tue, 04 Feb 2020 00:00:01 -0800</pubDate><guid>/blog/beam-2.19.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.19.0 release of Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2190-2020-02-04">download page</a> for this release.</p> |
| <p>For more information on changes in 2.19.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12346582">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Multiple improvements made into Python SDK harness: |
| <a href="https://issues.apache.org/jira/browse/BEAM-8624">BEAM-8624</a>, |
| <a href="https://issues.apache.org/jira/browse/BEAM-8623">BEAM-8623</a>, |
| <a href="https://issues.apache.org/jira/browse/BEAM-7949">BEAM-7949</a>, |
| <a href="https://issues.apache.org/jira/browse/BEAM-8935">BEAM-8935</a>, |
| <a href="https://issues.apache.org/jira/browse/BEAM-8816">BEAM-8816</a></li> |
| </ul> |
| <h3 id="ios">I/Os</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-1440">BEAM-1440</a> Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-2572">BEAM-2572</a> Implement an S3 filesystem for Python SDK</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-5192">BEAM-5192</a> Support Elasticsearch 7.x</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8745">BEAM-8745</a> More fine-grained controls for the size of a BigQuery Load job</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8801">BEAM-8801</a> PubsubMessageToRow should not check useFlatSchema() in processElement</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8953">BEAM-8953</a> Extend ParquetIO.Read/ReadFiles.Builder to support Avro GenericData model</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8946">BEAM-8946</a> Report collection size from MongoDBIOIT</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8978">BEAM-8978</a> Report saved data size from HadoopFormatIOIT</li> |
| </ul> |
| <h3 id="new-features--improvements">New Features / Improvements</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-6008">BEAM-6008</a> Improve error reporting in Java/Python PortableRunner</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8296">BEAM-8296</a> Containerize the Spark job server</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8746">BEAM-8746</a> Allow the local job service to work from inside docker</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8837">BEAM-8837</a> PCollectionVisualizationTest: possible bug</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8139">BEAM-8139</a> Execute portable Spark application jar</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9019">BEAM-9019</a> Improve Spark Encoders (wrappers of beam coders)</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9053">BEAM-9053</a> Improve error message when unable to get the correct filesystem for specified path in Python SDK) Improve error message when unable to get the correct filesystem for specified path in Python SDK</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9055">BEAM-9055</a> Unify the config names of Fn Data API across languages</li> |
| </ul> |
| <h3 id="sql">SQL</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-5690">BEAM-5690</a> Issue with GroupByKey in BeamSql using SparkRunner</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8993">BEAM-8993</a> [SQL] MongoDb should use predicate push-down</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8844">BEAM-8844</a> [SQL] Create performance tests for BigQueryTable</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9023">BEAM-9023</a> Upgrade to ZetaSQL 2019.12.1</li> |
| </ul> |
| <h3 id="breaking-changes">Breaking Changes</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8989">BEAM-8989</a> Backwards incompatible change in ParDo.getSideInputs (caught by failure when running Apache Nemo quickstart)</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8402">BEAM-8402</a> Backwards incompatible change related to how Environments are represented in Python <code>DirectRunner</code>.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9218">BEAM-9218</a> Template staging broken on Beam 2.18.0</li> |
| </ul> |
| <h3 id="dependency-changes">Dependency Changes</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8696">BEAM-8696</a> Beam Dependency Update Request: com.google.protobuf:protobuf-java</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8701">BEAM-8701</a> Beam Dependency Update Request: commons-io:commons-io</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8716">BEAM-8716</a> Beam Dependency Update Request: org.apache.commons:commons-csv</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8717">BEAM-8717</a> Beam Dependency Update Request: org.apache.commons:commons-lang3</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8749">BEAM-8749</a> Beam Dependency Update Request: com.datastax.cassandra:cassandra-driver-mapping</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-5546">BEAM-5546</a> Beam Dependency Update Request: commons-codec:commons-codec</li> |
| </ul> |
| <h3 id="bugfixes">Bugfixes</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9123">BEAM-9123</a> HadoopResourceId returns wrong directory name</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8962">BEAM-8962</a> FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-5495">BEAM-5495</a> PipelineResources algorithm is not working in most environments</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8025">BEAM-8025</a> Cassandra IO classMethod test is flaky</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8577">BEAM-8577</a> FileSystems may have not be initialized during ResourceId deserialization</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8582">BEAM-8582</a> Python SDK emits duplicate records for Default and AfterWatermark triggers</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8943">BEAM-8943</a> SDK harness servers don&rsquo;t shut down properly when SDK harness environment cleanup fails</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8995">BEAM-8995</a> apache_beam.io.gcp.bigquery_read_it_test failing on Py3.5 PC with: TypeError: the JSON object must be str, not &lsquo;bytes&rsquo;</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8999">BEAM-8999</a> PGBKCVOperation does not respect timestamp combiners</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9050">BEAM-9050</a> Beam pickler doesn&rsquo;t pickle classes that have <strong>module</strong> set to None.</li> |
| <li></li> |
| <li>Various bug fixes and performance improvements.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.19.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, Alex Amato, Alexey Romanenko, Andrew Pilloud, Ankur Goenka, Anton Kedin, Boyuan Zhang, Brian Hulette, Brian Martin, Chamikara Jayalath, Charles Chen, Craig Chambers, Daniel Oliveira, David Moravek, David Rieber, Dustin Rhodes, Etienne Chauchot, Gleb Kanterov, Hai Lu, Heejong Lee, Ismaël Mejía, Jan Lukavský, Jason Kuster, Jean-Baptiste Onofré, Jeff Klukas, João Cabrita, J Ross Thomson, Juan Rael, Juta, Kasia Kucharczyk, Kengo Seki, Kenneth Jung, Kenneth Knowles, Kyle Weaver, Kyle Winkelman, Lukas Drbal, Łukasz Gajowy, Marek Simunek, Mark Liu, Maximilian Michels, Melissa Pashniak, Michael Luckey, Michal Walenia, Mike Pedersen, Mikhail Gryzykhin, Niel Markwick, Pablo Estrada, Pascal Gula, Reuven Lax, Rob, Robbe Sneyders, Robert Bradshaw, Robert Burke, Rui Wang, Ruoyun Huang, Ryan Williams, Sam Rohde, Sam Whittle, Scott Wegner, Thomas Weise, Tianyang Hu, ttanay, tvalentyn, Tyler Akidau, Udi Meiri, Valentyn Tymofieiev, Xinyu Liu, XuMingmin</p></description></item><item><title>Blog: Apache Beam 2.18.0</title><link>/blog/beam-2.18.0/</link><pubDate>Thu, 23 Jan 2020 00:00:01 -0800</pubDate><guid>/blog/beam-2.18.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.18.0 release of Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2180-2020-01-23">download page</a> for this release.</p> |
| <p>For more information on changes in 2.18.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12346383&amp;projectId=12319527">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8470">BEAM-8470</a> - Create a new Spark runner based on Spark Structured streaming framework</li> |
| </ul> |
| <h3 id="ios">I/Os</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-7636">BEAM-7636</a> - Added SqsIO v2 support.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8513">BEAM-8513</a> - RabbitMqIO: Allow reads from exchange-bound queue without declaring the exchange.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8540">BEAM-8540</a> - Fix CSVSink example in FileIO docs</li> |
| </ul> |
| <h3 id="new-features--improvements">New Features / Improvements</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-5878">BEAM-5878</a> - Added support DoFns with Keyword-only arguments in Python 3.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-6756">BEAM-6756</a> - Improved support for lazy iterables in schemas (Java).</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-4776">BEAM-4776</a> AND <a href="https://issues.apache.org/jira/browse/BEAM-4777">BEAM-4777</a> - Added metrics supports to portable runners.</li> |
| <li>Various improvements to Interactive Beam: <a href="https://issues.apache.org/jira/browse/BEAM-7760">BEAM-7760</a>, <a href="https://issues.apache.org/jira/browse/BEAM-8379">BEAM-8379</a>, <a href="https://issues.apache.org/jira/browse/BEAM-8016">BEAM-8016</a>, <a href="https://issues.apache.org/jira/browse/BEAM-8016">BEAM-8016</a>.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8658">BEAM-8658</a> - Optionally set artifact staging port in FlinkUberJarJobServer.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8660">BEAM-8660</a> - Override returned artifact staging endpoint</li> |
| </ul> |
| <h3 id="sql">SQL</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8343">BEAM-8343</a> - [SQL] Add means for IO APIs to support predicate and/or project push-down when running SQL pipelines. And <a href="https://issues.apache.org/jira/browse/BEAM-8468">BEAM-8468</a>, <a href="https://issues.apache.org/jira/browse/BEAM-8365">BEAM-8365</a>, <a href="https://issues.apache.org/jira/browse/BEAM-8508">BEAM-8508</a>.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8427">BEAM-8427</a> - [SQL] Add support for MongoDB source.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8456">BEAM-8456</a> - Add pipeline option to control truncate of BigQuery data processed by Beam SQL.</li> |
| </ul> |
| <h3 id="breaking-changes">Breaking Changes</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8814">BEAM-8814</a> - &ndash;no_auth flag changed to boolean type.</li> |
| </ul> |
| <h3 id="deprecations">Deprecations</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8252">BEAM-8252</a> AND <a href="https://issues.apache.org/jira/browse/BEAM-8254">BEAM-8254</a> Add worker_region and worker_zone options. Deprecated &ndash;zone flag and &ndash;worker_region experiment argument.</li> |
| </ul> |
| <h3 id="dependency-changes">Dependency Changes</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-7078">BEAM-7078</a> - com.amazonaws:amazon-kinesis-client updated to 1.13.0.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8822">BEAM-8822</a> - Upgrade Hadoop dependencies to version 2.8.</li> |
| </ul> |
| <h3 id="bugfixes">Bugfixes</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-7917">BEAM-7917</a> - Python datastore v1new fails on retry.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-7981">BEAM-7981</a> - ParDo function wrapper doesn&rsquo;t support Iterable output types.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8146">BEAM-8146</a> - SchemaCoder/RowCoder have no equals() function.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8347">BEAM-8347</a> - UnboundedRabbitMqReader can fail to advance watermark if no new data comes in.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8352">BEAM-8352</a> - Reading records in background may lead to OOM errors</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8480">BEAM-8480</a> - Explicitly set restriction coder for bounded reader wrapper SDF.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8515">BEAM-8515</a> - Ensure that ValueProvider types have equals/hashCode implemented for comparison reasons.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8579">BEAM-8579</a> - Strip UTF-8 BOM bytes (if present) in TextSource.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8657">BEAM-8657</a> - Not doing Combiner lifting for data-driven triggers.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8663">BEAM-8663</a> - BundleBasedRunner Stacked Bundles don&rsquo;t respect PaneInfo.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8667">BEAM-8667</a> - Data channel should to avoid unlimited buffering in Python SDK.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8802">BEAM-8802</a> - Timestamp combiner not respected across bundles in streaming mode.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8803">BEAM-8803</a> - Default behaviour for Python BQ Streaming inserts sink should be to retry always.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8825">BEAM-8825</a> - OOM when writing large numbers of &rsquo;narrow&rsquo; rows.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8835">BEAM-8835</a> - Artifact retrieval fails with FlinkUberJarJobServer</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8836">BEAM-8836</a> - ExternalTransform is not providing a unique name</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8884">BEAM-8884</a> - Python MongoDBIO TypeError when splitting.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9041">BEAM-9041</a> - SchemaCoder equals should not rely on from/toRowFunction equality.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9042">BEAM-9042</a> - AvroUtils.schemaCoder(schema) produces a not serializable SchemaCoder.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-9065">BEAM-9065</a> - Spark runner accumulates metrics (incorrectly) between runs.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-6303">BEAM-6303</a> - Add .parquet extension to files in ParquetIO.</li> |
| <li>Various bug fixes and performance improvements.</li> |
| </ul> |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li> |
| <p><a href="https://issues.apache.org/jira/browse/BEAM-8882">BEAM-8882</a> - Python: <code>beam.Create</code> no longer preserves order unless <code>reshuffle=False</code> is passed in as an argument.</p> |
| <p>You may encounter this issue when using DirectRunner.</p> |
| </li> |
| <li> |
| <p><a href="https://issues.apache.org/jira/browse/BEAM-9065">BEAM-9065</a> - Spark runner accumulates metrics (incorrectly) between runs</p> |
| </li> |
| <li> |
| <p><a href="https://issues.apache.org/jira/browse/BEAM-9123">BEAM-9123</a> - HadoopResourceId returns wrong directory name</p> |
| </li> |
| <li> |
| <p>See a full list of open <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20affectedVersion%20%3D%202.18.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC">issues that affect</a> this version.</p> |
| </li> |
| <li> |
| <p><a href="https://issues.apache.org/jira/browse/BEAM-9144">BEAM-9144</a> - If you are using Avro 1.9.x with Beam you should not upgrade to this version. There is an issue with timestamp conversions. A fix will be available in the next release.</p> |
| </li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.18.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, Aizhamal Nurmamat kyzy, Alan Myrvold, Alexey Romanenko, Alex Van Boxel, Andre Araujo, Andrew Crites, Andrew Pilloud, Aryan Naraghi, Boyuan Zhang, Brian Hulette, bumblebee-coming, Cerny Ondrej, Chad Dombrova, Chamikara Jayalath, Changming Ma, Chun Yang, cmachgodaddy, Colm O hEigeartaigh, Craig Chambers, Daniel Oliveira, Daniel Robert, David Cavazos, David Moravek, David Song, dependabot[bot], Derek, Dmytro Sadovnychyi, Elliotte Rusty Harold, Etienne Chauchot, Hai Lu, Henry Suryawirawan, Ismaël Mejía, Jack Whelpton, Jan Lukavský, Jean-Baptiste Onofré, Jeff Klukas, Jincheng Sun, Jing, Jing Chen, Joe Tsai, Jonathan Alvarez-Gutierrez, Kamil Wasilewski, KangZhiDong, Kasia Kucharczyk, Kenneth Knowles, kirillkozlov, Kirill Kozlov, Kyle Weaver, liumomo315, lostluck, Łukasz Gajowy, Luke Cwik, Mark Liu, Maximilian Michels, Michal Walenia, Mikhail Gryzykhin, Niel Markwick, Ning Kang, nlofeudo, pabloem, Pablo Estrada, Pankaj Gudlani, Piotr Szczepanik, Primevenn, Reuven Lax, Robert Bradshaw, Robert Burke, Rui Wang, Ruoyun Huang, RusOr10n, Ryan Skraba, Saikat Maitra, sambvfx, Sam Rohde, Samuel Husso, Stefano, Steve Koonce, Steve Niemitz, sunjincheng121, Thomas Weise, Tianyang Hu, Tim Robertson, Tomo Suzuki, tvalentyn, Udi Meiri, Valentyn Tymofieiev, Viola Lyu, Wenjia Liu, Yichi Zhang, Yifan Zou, yoshiki.obata, Yueyang Qiu, ziel, 康智冬</p></description></item><item><title>Blog: Apache Beam 2.17.0</title><link>/blog/beam-2.17.0/</link><pubDate>Mon, 06 Jan 2020 00:00:01 -0800</pubDate><guid>/blog/beam-2.17.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.17.0 release of Beam. This release includes both improvements and new functionality. |
| Users of the MongoDbIO connector are encouraged to upgrade to this release to address a <a href="/security/CVE-2020-1929/">security vulnerability</a>.</p> |
| <p>See the <a href="/get-started/downloads/#2170-2020-01-06">download page</a> for this release.</p> |
| <p>For more information on changes in 2.17.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12345970&amp;projectId=12319527">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-7962">BEAM-7962</a> - Drop support for Flink 1.5 and 1.6</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-7635">BEAM-7635</a> - Migrate SnsIO to AWS SDK for Java 2</li> |
| <li>Improved usability for portable Flink Runner |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8183">BEAM-8183</a> - Optionally bundle multiple pipelines into a single Flink jar.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8372">BEAM-8372</a> - Allow submission of Flink UberJar directly to flink cluster.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8471">BEAM-8471</a> - Flink native job submission for portable pipelines.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8312">BEAM-8312</a> - Flink portable pipeline jars do not need to stage artifacts remotely.</li> |
| </ul> |
| </li> |
| </ul> |
| <h3 id="new-features--improvements">New Features / Improvements</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-7730">BEAM-7730</a> - Add Flink 1.9 build target and Make FlinkRunner compatible with Flink 1.9.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-7990">BEAM-7990</a> - Add ability to read parquet files into PCollection of pyarrow.Table.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8355">BEAM-8355</a> - Make BooleanCoder a standard coder.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8394">BEAM-8394</a> - Add withDataSourceConfiguration() method in JdbcIO.ReadRows class.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-5428">BEAM-5428</a> - Implement cross-bundle state caching.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-5967">BEAM-5967</a> - Add handling of DynamicMessage in ProtoCoder.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-7473">BEAM-7473</a> - Update RestrictionTracker within Python to not be required to be thread safe.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-7920">BEAM-7920</a> - Added AvroTableProvider to Beam SQL.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8098">BEAM-8098</a> - Improve documentation on BigQueryIO.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8100">BEAM-8100</a> - Add exception handling to Json transforms in Java SDK.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8306">BEAM-8306</a> - Improve estimation of data byte size reading from source in ElasticsearchIO.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8351">BEAM-8351</a> - Support passing in arbitrary KV pairs to sdk worker via external environment config.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8396">BEAM-8396</a> - Default to LOOPBACK mode for local flink (spark, &hellip;) runner.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8410">BEAM-8410</a> - JdbcIO should support setConnectionInitSqls in its DataSource.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8609">BEAM-8609</a> - Add HllCount to Java transform catalog.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8861">BEAM-8861</a> - Disallow self-signed certificates by default in ElasticsearchIO.</li> |
| </ul> |
| <h3 id="dependency-changes">Dependency Changes</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8285">BEAM-8285</a> - Upgrade ZetaSQL to 2019.09.1.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8392">BEAM-8392</a> - Upgrade pyarrow version bounds: 0.15.1&lt;= to &lt;0.16.0.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-5895">BEAM-5895</a> - Upgrade com.rabbitmq:amqp-client to 5.7.3.</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-6896">BEAM-6896</a> - Upgrade PyYAML version bounds: 3.12&lt;= to &lt;6.0.0.</li> |
| </ul> |
| <h3 id="bugfixes">Bugfixes</h3> |
| <ul> |
| <li>[BEAM-8819] - AvroCoder for SpecificRecords is not serialized correctly since 2.13.0</li> |
| <li>Various bug fixes and performance improvements.</li> |
| </ul> |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8989">BEAM-8989</a> Apache Nemo |
| runner broken due to backwards incompatible change since 2.16.0.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.17.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, Alan Myrvold, Alexey Romanenko, Andre-Philippe Paquet, Andrew |
| Pilloud, angulartist, Ankit Jhalaria, Ankur Goenka, Anton Kedin, Aryan Naraghi, |
| Aurélien Geron, B M VISHWAS, Bartok Jozsef, Boyuan Zhang, Brian Hulette, Cerny |
| Ondrej, Chad Dombrova, Chamikara Jayalath, ChethanU, cmach, Colm O hEigeartaigh, |
| Cyrus Maden, Daniel Oliveira, Daniel Robert, Dante, David Cavazos, David |
| Moravek, David Yan, Enrico Canzonieri, Etienne Chauchot, gxercavins, Hai Lu, |
| Hannah Jiang, Ian Lance Taylor, Ismaël Mejía, Israel Herraiz, James Wen, Jan |
| Lukavský, Jean-Baptiste Onofré, Jeff Klukas, jesusrv1103, Jofre, Kai Jiang, |
| Kamil Wasilewski, Kasia Kucharczyk, Kenneth Knowles, Kirill Kozlov, |
| kirillkozlov, Kohki YAMAGIWA, Kyle Weaver, Leonardo Alves Miguel, lloigor, |
| lostluck, Luis Enrique Ortíz Ramirez, Luke Cwik, Mark Liu, Maximilian Michels, |
| Michal Walenia, Mikhail Gryzykhin, mrociorg, Nicolas Delsaux, Ning Kang, NING |
| KANG, Pablo Estrada, pabloem, Piotr Szczepanik, rahul8383, Rakesh Kumar, Renat |
| Nasyrov, Reuven Lax, Robert Bradshaw, Robert Burke, Rui Wang, Ruslan Altynnikov, |
| Ryan Skraba, Salman Raza, Saul Chavez, Sebastian Jambor, sunjincheng121, Tatu |
| Saloranta, tchiarato, Thomas Weise, Tomo Suzuki, Tudor Marian, tvalentyn, Udi |
| Meiri, Valentyn Tymofieiev, Viola Lyu, Vishwas, Yichi Zhang, Yifan Zou, Yueyang |
| Qiu, Łukasz Gajowy</p></description></item><item><title>Blog: Apache Beam 2.16.0</title><link>/blog/beam-2.16.0/</link><pubDate>Mon, 07 Oct 2019 00:00:01 -0800</pubDate><guid>/blog/beam-2.16.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.16.0 release of Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2160-2019-10-07">download page</a> for this release.</p> |
| <p>For more information on changes in 2.16.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12345494">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Customizable Docker container images released and supported by Beam portable runners on Python 2.7, 3.5, 3.6, 3.7. (<a href="https://issues.apache.org/jira/browse/BEAM-7907">BEAM-7907</a>)</li> |
| <li>Integration improvements for Python Streaming on Dataflow including service features like autoscaling, drain, update, streaming engine and counter updates.</li> |
| </ul> |
| <h3 id="new-features--improvements">New Features / Improvements</h3> |
| <ul> |
| <li>A new count distinct transform based on BigQuery compatible HyperLogLog++ implementation. (<a href="https://issues.apache.org/jira/browse/BEAM-7013">BEAM-7013</a>)</li> |
| <li>Element counters in the Web UI graph representations for transforms for Python streaming jobs in Google Cloud Dataflow. (<a href="https://issues.apache.org/jira/browse/BEAM-7045">BEAM-7045</a>)</li> |
| <li>Add SetState in Python sdk. (<a href="https://issues.apache.org/jira/browse/BEAM-7741">BEAM-7741</a>)</li> |
| <li>Add hot key detection to Dataflow Runner. (<a href="https://issues.apache.org/jira/browse/BEAM-7820">BEAM-7820</a>)</li> |
| <li>Add ability to get the list of submitted jobs from gRPC JobService. (<a href="https://issues.apache.org/jira/browse/BEAM-7927">BEAM-7927</a>)</li> |
| <li>Portable Flink pipelines can now be bundled into executable jars. (<a href="https://issues.apache.org/jira/browse/BEAM-7966">BEAM-7966</a>, <a href="https://issues.apache.org/jira/browse/BEAM-7967">BEAM-7967</a>)</li> |
| <li>SQL join selection should be done in planner, not in expansion to PTransform. (<a href="https://issues.apache.org/jira/browse/BEAM-6114">BEAM-6114</a>)</li> |
| <li>A Python Sink for BigQuery with File Loads in Streaming. (<a href="https://issues.apache.org/jira/browse/BEAM-6611">BEAM-6611</a>)</li> |
| <li>Python BigQuery sink should be able to handle 15TB load job quota. (<a href="https://issues.apache.org/jira/browse/BEAM-7588">BEAM-7588</a>)</li> |
| <li>Spark portable runner: reuse SDK harness. (<a href="https://issues.apache.org/jira/browse/BEAM-7600">BEAM-7600</a>)</li> |
| <li>BigQuery File Loads to work well with load job size limits. (<a href="https://issues.apache.org/jira/browse/BEAM-7742">BEAM-7742</a>)</li> |
| <li>External environment with containerized worker pool. (<a href="https://issues.apache.org/jira/browse/BEAM-7980">BEAM-7980</a>)</li> |
| <li>Use OffsetRange as restriction for OffsetRestrictionTracker. (<a href="https://issues.apache.org/jira/browse/BEAM-8014">BEAM-8014</a>)</li> |
| <li>Get logs for SDK worker Docker containers. (<a href="https://issues.apache.org/jira/browse/BEAM-8015">BEAM-8015</a>)</li> |
| <li>PCollection boundedness is tracked and propagated in python sdk. (<a href="https://issues.apache.org/jira/browse/BEAM-8088">BEAM-8088</a>)</li> |
| </ul> |
| <h3 id="dependency-changes">Dependency Changes</h3> |
| <ul> |
| <li>Upgrade &ldquo;com.amazonaws:amazon-kinesis-producer&rdquo; to version 0.13.1. (<a href="https://issues.apache.org/jira/browse/BEAM-7894">BEAM-7894</a>)</li> |
| <li>Upgrade to joda time 2.10.3 to get updated TZDB. (<a href="https://issues.apache.org/jira/browse/BEAM-8161">BEAM-8161</a>)</li> |
| <li>Upgrade Jackson to version 2.9.10. (<a href="https://issues.apache.org/jira/browse/BEAM-8299">BEAM-8299</a>)</li> |
| <li>Upgrade grpcio minimum required version to 1.12.1. (<a href="https://issues.apache.org/jira/browse/BEAM-7986">BEAM-7986</a>)</li> |
| <li>Upgrade funcsigs minimum required version to 1.0.2 in Python2. (<a href="https://issues.apache.org/jira/browse/BEAM-7060">BEAM-7060</a>)</li> |
| <li>Upgrade google-cloud-pubsub maximum required version to 1.0.0. (<a href="https://issues.apache.org/jira/browse/BEAM-5539">BEAM-5539</a>)</li> |
| <li>Upgrade google-cloud-bigtable maximum required version to 1.0.0. (<a href="https://issues.apache.org/jira/browse/BEAM-5539">BEAM-5539</a>)</li> |
| <li>Upgrade dill version to 0.3.0. (<a href="https://issues.apache.org/jira/browse/BEAM-8324">BEAM-8324</a>)</li> |
| </ul> |
| <h3 id="bugfixes">Bugfixes</h3> |
| <ul> |
| <li>Various bug fixes and performance improvements.</li> |
| </ul> |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li>Given that Python 2 will reach EOL on Jan 1 2020, Python 2 users of Beam will now receive a warning that new releases of Apache Beam will soon support Python 3 only.</li> |
| <li>Filesystems not properly registered using FileIO.write in FlinkRunner. (<a href="https://issues.apache.org/jira/browse/BEAM-8303">BEAM-8303</a>)</li> |
| <li>Performance regression in Java DirectRunner in streaming mode. (<a href="https://issues.apache.org/jira/browse/BEAM-8363">BEAM-8363</a>)</li> |
| <li>Can&rsquo;t install the Python SDK on macOS 10.15. (<a href="https://issues.apache.org/jira/browse/BEAM-8368">BEAM-8368</a>)</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.16.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, Alex Van Boxel, Alexey Romanenko, Alexey Strokach, Alireza Samadian, |
| Andre-Philippe Paquet, Andrew Pilloud, Ankur Goenka, Anton Kedin, Aryan Naraghi, |
| B M VISHWAS, Bartok Jozsef, Bill Neubauer, Boyuan Zhang, Brian Hulette, Bruno Volpato, |
| Chad Dombrova, Chamikara Jayalath, Charith Ellawala, Charles Chen, Claire McGinty, |
| Cyrus Maden, Daniel Oliveira, Dante, David Cavazos, David Moravek, David Yan, |
| Dominic Mitchell, Elias Djurfeldt, Enrico Canzonieri, Etienne Chauchot, Gleb Kanterov, |
| Hai Lu, Hannah Jiang, Heejong Lee, Ian Lance Taylor, Ismaël Mejía, Jack Whelpton, |
| James Wen, Jan Lukavský, Jean-Baptiste Onofré, Jofre, Kai Jiang, Kamil Wasilewski, |
| Kasia Kucharczyk, Kenneth Jung, Kenneth Knowles, Kirill Kozlov, Kohki YAMAGIWA, |
| Kyle Weaver, Kyle Winkelman, Ludovic Post, Luis Enrique Ortíz Ramirez, Luke Cwik, |
| Mark Liu, Maximilian Michels, Michal Walenia, Mike Kaplinskiy, Mikhail Gryzykhin, |
| NING KANG, Oliver Henlich, Pablo Estrada, Rakesh Kumar, Renat Nasyrov, Reuven Lax, |
| Robert Bradshaw, Robert Burke, Rui Wang, Ruoyun Huang, Ryan Skraba, Sahith Nallapareddy, |
| Salman Raza, Sam Rohde, Saul Chavez, Shoaib, Shoaib Zafar, Slava Chernyak, Tanay Tummalapalli, |
| Thinh Ha, Thomas Weise, Tianzi Cai, Tim van der Lippe, Tomer Zeltzer, Tudor Marian, |
| Udi Meiri, Valentyn Tymofieiev, Yichi Zhang, Yifan Zou, Yueyang Qiu, gxercavins, |
| jesusrv1103, lostluck, matt-darwin, mrociorg, ostrokach, parahul, rahul8383, rosetn, |
| sunjincheng121, the1plummie, ttanay, tvalentyn, venn001, yoshiki.obata, Łukasz Gajowy</p></description></item><item><title>Blog: Apache Beam 2.15.0</title><link>/blog/beam-2.15.0/</link><pubDate>Thu, 22 Aug 2019 00:00:01 -0800</pubDate><guid>/blog/beam-2.15.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.15.0 release of Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2150-2019-08-22">download page</a> for this release.</p> |
| <p>For more information on changes in 2.15.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12345489">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Vendored Guava was upgraded to version 26.0.</li> |
| <li>Support multi-process execution on the FnApiRunner for Python. (<a href="https://issues.apache.org/jira/browse/BEAM-3645">BEAM-3645</a>)</li> |
| </ul> |
| <h3 id="ios">I/Os</h3> |
| <ul> |
| <li>Add AvroIO.sink for IndexedRecord (FileIO compatible). (<a href="https://issues.apache.org/jira/browse/BEAM-6480">BEAM-6480</a>)</li> |
| <li>Add support for writing to BigQuery clustered tables. (<a href="https://issues.apache.org/jira/browse/BEAM-5191">BEAM-5191</a>)</li> |
| </ul> |
| <h3 id="new-features--improvements">New Features / Improvements</h3> |
| <ul> |
| <li>Support ParquetTable in SQL. (<a href="https://issues.apache.org/jira/browse/BEAM-7728">BEAM-7728</a>)</li> |
| <li>Add hot key detection to Dataflow Runner. (<a href="https://issues.apache.org/jira/browse/BEAM-7820">BEAM-7820</a>)</li> |
| <li>Support schemas in the JDBC sink. (<a href="https://issues.apache.org/jira/browse/BEAM-6675">BEAM-6675</a>)</li> |
| <li>Report GCS throttling time to Dataflow autoscaler for better autoscaling. (<a href="https://issues.apache.org/jira/browse/BEAM-7667">BEAM-7667</a>)</li> |
| <li>Support transform_name_mapping option in Python SDK for <code>--update</code> use. (<a href="https://issues.apache.org/jira/browse/BEAM-7761">BEAM-7761</a>)</li> |
| <li>Dependency: Upgrade Jackson databind to version 2.9.9.3 (<a href="https://issues.apache.org/jira/browse/BEAM-7880">BEAM-7880</a>)</li> |
| </ul> |
| <h3 id="bugfixes">Bugfixes</h3> |
| <ul> |
| <li>Various bug fixes and performance improvements.</li> |
| </ul> |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-7616">BEAM-7616</a> urlopen calls may get stuck. (Regression from 2.14.0)</li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-8111">BEAM-8111</a> SchemaCoder fails on Dataflow, preventing the use of SqlTransform and schema-aware transforms. (Regression from 2.14.0)</li> |
| <li>(<a href="https://issues.apache.org/jira/browse/BEAM-8368">BEAM-8368</a>) Can&rsquo;t install the Python SDK on macOS 10.15.</li> |
| </ul> |
| <h3 id="breaking-changes">Breaking Changes</h3> |
| <ul> |
| <li><code>--region</code> flag will be a required flag in the future for Dataflow. A warning is added to warn for this future change. (<a href="https://issues.apache.org/jira/browse/BEAM-7833">BEAM-7833</a>)</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.15.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, Alexey Romanenko, Alex Goos, Alireza Samadian, Andrew Pilloud, Ankur Goenka, |
| Anton Kedin, Aryan Naraghi, Bartok Jozsef, bmv126, B M VISHWAS, Boyuan Zhang, |
| Brian Hulette, brucearctor, Cade Markegard, Cam Mach, Chad Dombrova, |
| Chaim Turkel, Chamikara Jayalath, Charith Ellawala, Claire McGinty, Craig Chambers, |
| Daniel Oliveira, David Cavazos, David Moravek, Dominic Mitchell, Dustin Rhodes, |
| Etienne Chauchot, Filipe Regadas, Gleb Kanterov, Gunnar Schulze, Hannah Jiang, |
| Heejong Lee, Henry Suryawirawan, Ismaël Mejía, Ivo Galic, Jan Lukavský, |
| Jawad, Juta, Juta Staes, Kai Jiang, Kamil Wasilewski, Kasia Kucharczyk, |
| Kenneth Jung, Kenneth Knowles, Kyle Weaver, Lily Li, Logan HAUSPIE, lostluck, |
| Łukasz Gajowy, Luke Cwik, Mark Liu, Matt Helm, Maximilian Michels, |
| Michael Luckey, Mikhail Gryzykhin, Neville Li, Nicholas Rucci, pabloem, |
| Pablo Estrada, Paul King, Paul Suganthan, Raheel Khan, Rakesh Kumar, |
| Reza Rokni, Robert Bradshaw, Robert Burke, rosetn, Rui Wang, Ryan Skraba, RyanSkraba, |
| Sahith Nallapareddy, Sam Rohde, Sam Whittle, Steve Niemitz, Tanay Tummalapalli, Thomas Weise, |
| Tianyang Hu, ttanay, tvalentyn, Udi Meiri, Valentyn Tymofieiev, Wout Scheepers, |
| yanzhi, Yekut, Yichi Zhang, Yifan Zou, yoshiki.obata, Yueyang Qiu, Yunqing Zhou</p></description></item><item><title>Blog: Apache Beam 2.14.0</title><link>/blog/beam-2.14.0/</link><pubDate>Wed, 31 Jul 2019 00:00:01 -0800</pubDate><guid>/blog/beam-2.14.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.14.0 release of Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2140-2019-08-01">download page</a> for this release.</p> |
| <p>For more information on changes in 2.14.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12345431">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <ul> |
| <li>Python 3 support is extended to Python 3.6 and 3.7; in addition to various other Python 3 <a href="https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504&amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16890504">improvements</a>.</li> |
| <li>Spark portable runner (batch) now <a href="https://lists.apache.org/thread.html/c43678fc24c9a1dc9f48c51c51950aedcb9bc0fd3b633df16c3d595a@%3Cuser.beam.apache.org%3E">available</a> for Java, Python, Go.</li> |
| <li>Added new runner: Hazelcast Jet Runner. (<a href="https://issues.apache.org/jira/browse/BEAM-7305">BEAM-7305</a>)</li> |
| </ul> |
| <h3 id="ios">I/Os</h3> |
| <ul> |
| <li>Schema support added to BigQuery reads. (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-6673">BEAM-6673</a>)</li> |
| <li>Schema support added to JDBC source. (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-6674">BEAM-6674</a>)</li> |
| <li>BigQuery support for <code>bytes</code> is fixed. (Python 3) (<a href="https://issues.apache.org/jira/browse/BEAM-6769">BEAM-6769</a>)</li> |
| <li>Added DynamoDB IO. (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-7043">BEAM-7043</a>)</li> |
| <li>Added support unbounded reads with HCatalogIO (Java) (<a href="https://issues.apache.org/jira/browse/BEAM-7450">BEAM-7450</a>)</li> |
| <li>Added BoundedSource wrapper for SDF. (Python) (<a href="https://issues.apache.org/jira/browse/BEAM-7443">BEAM-7443</a>)</li> |
| <li>Added support for INCRBY/DECRBY operations in RedisIO. (<a href="https://issues.apache.org/jira/browse/BEAM-7286">BEAM-7286</a>)</li> |
| <li>Added Support for ValueProvider defined GCS Location for WriteToBigQuery with File Loads. (Java) ((<a href="https://issues.apache.org/jira/browse/BEAM-7603">BEAM-7603</a>))</li> |
| </ul> |
| <h3 id="new-features--improvements">New Features / Improvements</h3> |
| <ul> |
| <li>Python SDK add support for DoFn <code>setup</code> and <code>teardown</code> methods. (<a href="https://issues.apache.org/jira/browse/BEAM-562">BEAM-562</a>)</li> |
| <li>Python SDK adds new transforms: <a href="https://issues.apache.org/jira/browse/BEAM-6693">ApproximateUnique</a>, <a href="https://issues.apache.org/jira/browse/BEAM-6695">Latest</a>, <a href="https://issues.apache.org/jira/browse/BEAM-7019">Reify</a>, <a href="https://issues.apache.org/jira/browse/BEAM-7021">ToString</a>, <a href="https://issues.apache.org/jira/browse/BEAM-7023">WithKeys</a>.</li> |
| <li>Added hook for user-defined JVM initialization in workers. (<a href="https://issues.apache.org/jira/browse/BEAM-6872">BEAM-6872</a>)</li> |
| <li>Added support for SQL Row Estimation for BigQueryTable. (<a href="https://issues.apache.org/jira/browse/BEAM-7513">BEAM-7513</a>)</li> |
| <li>Auto sharding of streaming sinks in FlinkRunner. (<a href="https://issues.apache.org/jira/browse/BEAM-5865">BEAM-5865</a>)</li> |
| <li>Removed the Hadoop dependency from the external sorter. (<a href="https://issues.apache.org/jira/browse/BEAM-7268">BEAM-7268</a>)</li> |
| <li>Added option to expire portable SDK worker environments. (<a href="https://issues.apache.org/jira/browse/BEAM-7348">BEAM-7348</a>)</li> |
| <li>Beam does not relocate Guava anymore and depends only on its own vendored version of Guava. (<a href="https://issues.apache.org/jira/browse/BEAM-6620">BEAM-6620</a>)</li> |
| </ul> |
| <h3 id="breaking-changes">Breaking Changes</h3> |
| <ul> |
| <li>Deprecated set/getClientConfiguration in Jdbc IO. (<a href="https://issues.apache.org/jira/browse/BEAM-7263">BEAM-7263</a>)</li> |
| </ul> |
| <h3 id="bugfixes">Bugfixes</h3> |
| <ul> |
| <li>Fixed reading of concatenated compressed files. (Python) (<a href="https://issues.apache.org/jira/browse/BEAM-6952">BEAM-6952</a>)</li> |
| <li>Fixed re-scaling issues on Flink &gt;= 1.6 versions. (<a href="https://issues.apache.org/jira/browse/BEAM-7144">BEAM-7144</a>)</li> |
| <li>Fixed SQL EXCEPT DISTINCT behavior. (<a href="https://issues.apache.org/jira/browse/BEAM-7194">BEAM-7194</a>)</li> |
| <li>Fixed OOM issues with bounded Reads for Flink Runner. (<a href="https://issues.apache.org/jira/browse/BEAM-7442">BEAM-7442</a>)</li> |
| <li>Fixed HdfsFileSystem to correctly match directories. (<a href="https://issues.apache.org/jira/browse/BEAM-7561">BEAM-7561</a>)</li> |
| <li>Upgraded Spark runner to use spark version 2.4.3. (<a href="https://issues.apache.org/jira/browse/BEAM-7265">BEAM-7265</a>)</li> |
| <li>Upgraded Jackson to version 2.9.9. (<a href="https://issues.apache.org/jira/browse/BEAM-7465">BEAM-7465</a>)</li> |
| <li>Various other bug fixes and performance improvements.</li> |
| </ul> |
| <h3 id="known-issues">Known Issues</h3> |
| <ul> |
| <li>Do <strong>NOT</strong> use Python MongoDB source in this release. Python MongoDB source <a href="https://issues.apache.org/jira/browse/BEAM-5148">added</a> in this release has a known issue that can result in data loss. See (<a href="https://issues.apache.org/jira/browse/BEAM-7866">BEAM-7866</a>) for details.</li> |
| <li>Can&rsquo;t install the Python SDK on macOS 10.15. See (<a href="https://issues.apache.org/jira/browse/BEAM-8368">BEAM-8368</a>) for details.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.14.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, Aizhamal Nurmamat kyzy, Ajo Thomas, Alex Amato, Alexey Romanenko, |
| Alexey Strokach, Alex Van Boxel, Alireza Samadian, Andrew Pilloud, |
| Ankit Jhalaria, Ankur Goenka, Anton Kedin, Aryan Naraghi, Bartok Jozsef, |
| Bora Kaplan, Boyuan Zhang, Brian Hulette, Cam Mach, Chamikara Jayalath, |
| Charith Ellawala, Charles Chen, Colm O hEigeartaigh, Cyrus Maden, |
| Daniel Mills, Daniel Oliveira, David Cavazos, David Moravek, David Yan, |
| Daniel Lescohier, Elwin Arens, Etienne Chauchot, Fábio Franco Uechi, |
| Finch Keung, Frederik Bode, Gregory Kovelman, Graham Polley, Hai Lu, Hannah Jiang, |
| Harshit Dwivedi, Harsh Vardhan, Heejong Lee, Henry Suryawirawan, |
| Ismaël Mejía, Jan Lukavský, Jean-Baptiste Onofré, Jozef Vilcek, Juta, Kai Jiang, |
| Kamil Wu, Kasia Kucharczyk, Kenneth Knowles, Kyle Weaver, Lara Schmidt, |
| Łukasz Gajowy, Luke Cwik, Manu Zhang, Mark Liu, Matthias Baetens, |
| Maximilian Michels, Melissa Pashniak, Michael Luckey, Michal Walenia, |
| Mikhail Gryzykhin, Ming Liang, Neville Li, Pablo Estrada, Paul Suganthan, |
| Peter Backx, Rakesh Kumar, Rasmi Elasmar, Reuven Lax, Reza Rokni, Robbe Sneyders, |
| Robert Bradshaw, Robert Burke, Rose Nguyen, Rui Wang, Ruoyun Huang, |
| Shoaib Zafar, Slava Chernyak, Steve Niemitz, Tanay Tummalapalli, Thomas Weise, |
| Tim Robertson, Tim van der Lippe, Udi Meiri, Valentyn Tymofieiev, Varun Dhussa, |
| Viktor Gerdin, Yichi Zhang, Yifan Mai, Yifan Zou, Yueyang Qiu.</p></description></item><item><title>Blog: Apache Beam 2.13.0</title><link>/blog/beam-2.13.0/</link><pubDate>Fri, 07 Jun 2019 00:00:01 -0800</pubDate><guid>/blog/beam-2.13.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.13.0 release of Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2130-2019-05-21">download page</a> for this release.</p> |
| <p>For more information on changes in 2.13.0, check out the |
| <a href="https://jira.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12345166">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <h3 id="ios">I/Os</h3> |
| <ul> |
| <li>Support reading query results with the BigQuery storage API.</li> |
| <li>Support KafkaIO to be configured externally for use with other SDKs.</li> |
| <li>BigQuery IO now supports BYTES datatype on Python 3.</li> |
| <li>Avro IO support enabled on Python 3.</li> |
| <li>For Python 3 pipelines, the default Avro library used by Beam AvroIO and Dataflow workers was switched from avro-python3 to fastavro.</li> |
| </ul> |
| <h3 id="new-features--improvements">New Features / Improvements</h3> |
| <ul> |
| <li>Flink 1.8 support added.</li> |
| <li>Support to run word count on Portable Spark runner.</li> |
| <li>ElementCount metrics in FnApi Dataflow Runner.</li> |
| <li>Support to create BinaryCombineFn from lambdas.</li> |
| </ul> |
| <h3 id="breaking-changes">Breaking Changes</h3> |
| <ul> |
| <li>When writing BYTES Datatype into Bigquery with Beam Bigquery IO on Python DirectRunner, users need to base64-encode bytes values before passing them to Bigquery IO. Accordingly, when reading bytes data from BigQuery, the IO will also return base64-encoded bytes. This change only affects Bigquery IO on Python DirectRunner. New DirectRunner behavior is consistent with treatment of Bytes by Beam Java Bigquery IO, and Python Dataflow Runner.</li> |
| </ul> |
| <h3 id="bugfixes">Bugfixes</h3> |
| <ul> |
| <li>Various bug fixes and performance improvements.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed to the 2.13.0 release. Thank you to all contributors!</p> |
| <p>Aaron Li, Ahmet Altay, Aizhamal Nurmamat kyzy, Alex Amato, Alexey Romanenko, |
| Andrew Pilloud, Ankur Goenka, Anton Kedin, apstndb, Boyuan Zhang, Brian Hulette, |
| Brian Quinlan, Chamikara Jayalath, Cyrus Maden, Daniel Chen, Daniel Oliveira, |
| David Cavazos, David Moravek, David Yan, EdgarLGB, Etienne Chauchot, frederik2, |
| Gleb Kanterov, Harshit Dwivedi, Harsh Vardhan, Heejong Lee, Hennadiy Leontyev, |
| Henri-Mayeul de Benque, Ismaël Mejía, Jae-woo Kim, Jamie Kirkpatrick, Jan Lukavský, |
| Jason Kuster, Jean-Baptiste Onofré, JohnZZGithub, Jozef Vilcek, Juta, Kenneth Jung, |
| Kenneth Knowles, Kyle Weaver, Łukasz Gajowy, Luke Cwik, Mark Liu, Mathieu Blanchard, |
| Maximilian Michels, Melissa Pashniak, Michael Luckey, Michal Walenia, Mike Kaplinskiy, |
| Mike Pedersen, Mikhail Gryzykhin, Mikhail-Ivanov, Niklas Hansson, pabloem, |
| Pablo Estrada, Pranay Nanda, Reuven Lax, Richard Moorhead, Robbe Sneyders, |
| Robert Bradshaw, Robert Burke, Roman van der Krogt, rosetn, Rui Wang, Ryan Yuan, |
| Sam Whittle, sudhan499, Sylwester Kardziejonek, Ted, Thomas Weise, Tim Robertson, |
| ttanay, tvalentyn, Udi Meiri, Valentyn Tymofieiev, Xinyu Liu, Yifan Zou, |
| yoshiki.obata, Yueyang Qiu</p></description></item><item><title>Blog: Apache Beam 2.12.0</title><link>/blog/beam-2.12.0/</link><pubDate>Thu, 25 Apr 2019 00:00:01 -0800</pubDate><guid>/blog/beam-2.12.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.12.0 release of Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2120-2019-04-25">download page</a> for this release.</p> |
| <p>For more information on changes in 2.12.0, check out the |
| <a href="https://jira.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12344944">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <h3 id="ios">I/Os</h3> |
| <ul> |
| <li>Add support for a BigQuery custom sink for Python SDK.</li> |
| <li>Add support to specify a query in CassandraIO for Java SDK.</li> |
| <li>Add experimental support for cross-language transforms, |
| please see <a href="https://issues.apache.org/jira/browse/BEAM-6730">BEAM-6730</a></li> |
| <li>Add support in the Flink Runner for exactly-once Writes with KafkaIO</li> |
| </ul> |
| <h3 id="new-features--improvements">New Features / Improvements</h3> |
| <ul> |
| <li>Enable Bundle Finalization in Python SDK for portable runners.</li> |
| <li>Add support to the Java SDK harness to merge windows.</li> |
| <li>Add Kafka Sink EOS support on Flink runner.</li> |
| <li>Added a dead letter queue to Python streaming BigQuery sink.</li> |
| <li>Add Experimental Python 3.6 and 3.7 workloads enabled. |
| Beam 2.12 supports starting Dataflow pipelines under Python 3.6, 3.7, however 3.5 remains the only recommended minor version for Dataflow runner. In addition to announced 2.11 limitations, Beam typehint annotations are currently not supported on Python &gt;= 3.6.</li> |
| </ul> |
| <h3 id="bugfixes">Bugfixes</h3> |
| <ul> |
| <li>Various bug fixes and performance improvements.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed |
| to the 2.12.0 release. Thank you to all contributors!</p> |
| <p>Ahmed El.Hussaini, Ahmet Altay, Alan Myrvold, Alex Amato, Alexander Savchenko, |
| Alexey Romanenko, Andrew Brampton, Andrew Pilloud, Ankit Jhalaria, |
| Ankur Goenka, Anton Kedin, Boyuan Zhang, Brian Hulette, Chamikara Jayalath, |
| Charles Chen, Colm O hEigeartaigh, Craig Chambers, Dan Duong, Daniel Mescheder, |
| Daniel Oliveira, David Moravek, David Rieber, David Yan, Eric Roshan-Eisner, |
| Etienne Chauchot, Gleb Kanterov, Heejong Lee, Ho Tien Vu, Ismaël Mejía, |
| Jan Lukavský, Jean-Baptiste Onofré, Jeff Klukas, Juta, Kasia Kucharczyk, |
| Kengo Seki, Kenneth Jung, Kenneth Knowles, kevin, Kyle Weaver, Kyle Winkelman, |
| Łukasz Gajowy, Mark Liu, Mathieu Blanchard, Max Charas, Maximilian Michels, |
| Melissa Pashniak, Michael Luckey, Michal Walenia, Mike Kaplinskiy, |
| Mikhail Gryzykhin, Niel Markwick, Pablo Estrada, Radoslaw Stankiewicz, |
| Reuven Lax, Robbe Sneyders, Robert Bradshaw, Robert Burke, Rui Wang, |
| Ruoyun Huang, Ryan Williams, Slava Chernyak, Shahar Frank, Sunil Pedapudi, |
| Thomas Weise, Tim Robertson, Tanay Tummalapalli, Udi Meiri, |
| Valentyn Tymofieiev, Xinyu Liu, Yifan Zou, Yueyang Qiu</p></description></item><item><title>Blog: Apache Beam 2.11.0</title><link>/blog/beam-2.11.0/</link><pubDate>Tue, 05 Mar 2019 00:00:01 -0800</pubDate><guid>/blog/beam-2.11.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.11.0 release of Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2110-2019-02-26">download page</a> for this release.</p> |
| <p>For more information on changes in 2.11.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12344775">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <h3 id="dependency-upgradeschanges">Dependency Upgrades/Changes</h3> |
| <ul> |
| <li>Java: antlr: 4.7</li> |
| <li>Java: antlr_runtime: 4.7</li> |
| <li>Java: bigdataoss_gcsio: 1.9.16</li> |
| <li>Java: bigdataoss_util: 1.9.16</li> |
| <li>Java: bigtable_client_core: 1.8.0</li> |
| <li>Java: cassandra-driver-core: 3.6.0</li> |
| <li>Java: cassandra-driver-mapping: 3.6.0</li> |
| <li>Java: commons-compress: 1.18</li> |
| <li>Java: gax_grpc: 1.38.0</li> |
| <li>Java: google_api_common: 1.7.0</li> |
| <li>Java: google_api_services_dataflow: v1b3-rev20190126-1.27.0</li> |
| <li>Java: google_cloud_bigquery_storage: 0.79.0-alpha</li> |
| <li>Java: google_cloud_bigquery_storage_proto: 0.44.0</li> |
| <li>Java: google_auth_library_credentials: 0.12.0</li> |
| <li>Java: google_auth_library_oauth2_http: 0.12.0</li> |
| <li>Java: google_cloud_core: 1.61.0</li> |
| <li>Java: google_cloud_core_grpc: 1.61.0</li> |
| <li>Java: google_cloud_spanner: 1.6.0</li> |
| <li>Java: grpc_all: 1.17.1</li> |
| <li>Java: grpc_auth: 1.17.1</li> |
| <li>Java: grpc_core: 1.17.1</li> |
| <li>Java: grpc_google_cloud_pubsub_v1: 1.17.1</li> |
| <li>Java: grpc_protobuf: 1.17.1</li> |
| <li>Java: grpc_protobuf_lite: 1.17.1</li> |
| <li>Java: grpc_netty: 1.17.1</li> |
| <li>Java: grpc_stub: 1.17.1</li> |
| <li>Java: netty_handler: 4.1.30.Final</li> |
| <li>Java: netty_tcnative_boringssl_static: 2.0.17.Final</li> |
| <li>Java: netty_transport_native_epoll: 4.1.30.Final</li> |
| <li>Java: proto_google_cloud_spanner_admin_database_v1: 1.6.0</li> |
| <li>Java: zstd_jni: 1.3.8-3</li> |
| <li>Python: futures&gt;=3.2.0,&lt;4.0.0; python_version &lt; &ldquo;3.0&rdquo;</li> |
| <li>Python: pyvcf&gt;=0.6.8,&lt;0.7.0; python_version &lt; &ldquo;3.0&rdquo;</li> |
| <li>Python: google-apitools&gt;=0.5.26,&lt;0.5.27</li> |
| <li>Python: google-cloud-core==0.28.1</li> |
| <li>Python: google-cloud-bigtable==0.31.1</li> |
| </ul> |
| <h3 id="ios">I/Os</h3> |
| <ul> |
| <li>Portable Flink runner support for running cross-language transforms.</li> |
| <li>Add Cloud KMS support to GCS copies.</li> |
| <li>Add parameters for offsetConsumer in KafkaIO.read().</li> |
| <li>Allow setting compression codec in ParquetIO write.</li> |
| <li>Add kms_key to BigQuery transforms, pass to Dataflow.</li> |
| </ul> |
| <h3 id="new-features--improvements">New Features / Improvements</h3> |
| <ul> |
| <li>Python 3 (experimental) suppport for DirectRunner and DataflowRunner.</li> |
| <li>Add ZStandard compression support for Java SDK.</li> |
| <li>Python: Add CombineFn.compact, similar to Java.</li> |
| <li>SparkRunner: GroupByKey optimized for non-merging windows.</li> |
| <li>SparkRunner: Add bundleSize parameter to control splitting of Spark sources.</li> |
| <li>FlinkRunner: Portable runner savepoint / upgrade support.</li> |
| </ul> |
| <h3 id="bugfixes">Bugfixes</h3> |
| <ul> |
| <li>Various bug fixes and performance improvements.</li> |
| </ul> |
| <h3 id="deprecations">Deprecations</h3> |
| <ul> |
| <li>Deprecate MongoDb <code>withKeepAlive</code> because it is deprecated in the Mongo driver.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed |
| to the 2.11.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, Alex Amato. Alexey Romanenko, Andrew Pilloud, Ankur Goenka, Anton Kedin, |
| Boyuan Zhang, Brian Hulette, Brian Martin, Chamikara Jayalath, Charles Chen, Craig Chambers, |
| Daniel Oliveira, David Moravek, David Rieber, Dustin Rhodes, Etienne Chauchot, Gleb Kanterov, |
| Hai Lu, Heejong Lee, Ismaël Mejía, J Ross Thomson, Jan Lukavsky, Jason Kuster, Jean-Baptiste Onofré, |
| Jeff Klukas, João Cabrita, Juan Rael, Juta Staes, Kasia Kucharczyk, Kengo Seki, Kenneth Jung, |
| Kenneth Knowles, Kyle Weaver, Kyle Winkelman, Lukas Drbal, Marek Simunek, Mark Liu, |
| Maximilian Michels, Melissa Pashniak, Michael Luckey, Michal Walenia, Mike Pedersen, |
| Mikhail Gryzykhin, Niel Markwick, Pablo Estrada, Pascal Gula, Reuven Lax, Robbe Sneyders, |
| Robert Bradshaw, Robert Burke, Rui Wang, Ruoyun Huang, Ryan Williams, Sam Rohde, Sam Whittle, |
| Scott Wegner, Tanay Tummalapalli, Thomas Weise, Tianyang Hu, Tyler Akidau, Udi Meiri, |
| Valentyn Tymofieiev, Xinyu Liu, Xu Mingmin, Łukasz Gajowy.</p></description></item><item><title>Blog: Apache Beam 2.10.0</title><link>/blog/beam-2.10.0/</link><pubDate>Fri, 15 Feb 2019 00:00:01 -0800</pubDate><guid>/blog/beam-2.10.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.10.0 release of Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#2100-2019-02-01">download page</a> for this release.</p> |
| <p>For more information on changes in 2.10.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12344540">detailed release notes</a>.</p> |
| <h2 id="highlights">Highlights</h2> |
| <h3 id="dependency-upgradeschanges">Dependency Upgrades/Changes</h3> |
| <ul> |
| <li>FlinkRunner: Flink 1.5.x/1.6.x/1.7.x</li> |
| <li>Java: AutoValue 1.6.3</li> |
| <li>Java: Jackson 2.9.8</li> |
| <li>Java: google_cloud_bigdataoss 1.9.13</li> |
| <li>Java: Apache Commons Codec: 1.10</li> |
| <li>Python: avro&gt;=1.8.1,&lt;2.0.0; python_version &lt; &ldquo;3.0&rdquo;</li> |
| <li>Python: avro-python3&gt;=1.8.1,&lt;2.0.0; python_version &gt;= &ldquo;3.0&rdquo;</li> |
| <li>Python: bigdataoss_gcsio 1.9.12</li> |
| <li>Python: dill&gt;=0.2.9,&lt;0.2.10</li> |
| <li>Python: gcsio 1.9.13</li> |
| <li>Python: google-cloud-pubsub 0.39.0</li> |
| <li>Python: pytz&gt;=2018.3</li> |
| <li>Python: pyyaml&gt;=3.12,&lt;4.0.0</li> |
| <li>MongoDbIO: mongo client 3.9.1</li> |
| </ul> |
| <h3 id="ios">I/Os</h3> |
| <ul> |
| <li>ParquetIO for Python SDK</li> |
| <li>HadoopOutputFormatIO: Add batching support</li> |
| <li>HadoopOutputFormatIO: Add streaming support</li> |
| <li>MongoDbIO: Add projections</li> |
| <li>MongoDbIO: Add support for server with self signed SSL</li> |
| <li>MongoDbIO add ordered option (inserts documents even if errors)</li> |
| <li>KafkaIO: Add support to write to multiple topics</li> |
| <li>KafkaIO: add writing support with ProducerRecord</li> |
| <li>CassandraIO: Add ability to delete data</li> |
| <li>JdbcIO: Add ValueProvider support for Statement in JdbcIO.write(), so it can be templatized</li> |
| </ul> |
| <h3 id="new-features--improvements">New Features / Improvements</h3> |
| <ul> |
| <li>FlinkRunner: support Flink config directory</li> |
| <li>FlinkRunner: master url now supports IPv6 addresses</li> |
| <li>FlinkRunner: portable runner savepoint / upgrade support</li> |
| <li>FlinkRunner: can be built against different Flink versions</li> |
| <li>FlinkRunner: Send metrics to Flink in portable runner</li> |
| <li>Java: Migrate to vendored gRPC (no conflicts with user gRPC, smaller jars)</li> |
| <li>Java: Migrate to vendored Guava (no conflicts with user Guava, smaller jars)</li> |
| <li>SQL: support joining unbounded to bounded sources via side input (and is no longer sensitive to left vs right join)</li> |
| <li>SQL: support table macro</li> |
| <li>Schemas: support for Avro, with automatic schema registration</li> |
| <li>Schemas: Automatic schema registration for AutoValue classes</li> |
| </ul> |
| <h3 id="bugfixes">Bugfixes</h3> |
| <ul> |
| <li>Watch PTransform fixed (affects FileIO)</li> |
| <li>FlinkRunner: no longer fails if GroupByKey contains null values (streaming mode only)</li> |
| <li>FlinkRunner: no longer prepares to-be-staged file too late</li> |
| <li>FlinkRunner: sets number of shards for writes with runner determined sharding</li> |
| <li>FlinkRunner: prevents CheckpointMarks from not getting acknowledged</li> |
| <li>Schemas: Generated row object for POJOs, Avros, and JavaBeans should work if the wrapped class is package private</li> |
| <li>Schemas: Nested collection types in schemas no longer cause NullPointerException when converting to a POJO</li> |
| <li>BigQueryIO: now handles quotaExceeded errors properly</li> |
| <li>BigQueryIO: now handles triggering correctly in certain very large load jobs</li> |
| <li>FileIO and other file-based IOs: Beam LocalFilesystem now matches glob patterns in windows</li> |
| <li>SQL: joins no longer moves timestamps to the end of the window</li> |
| <li>SQL: was missing some transitive dependencies</li> |
| <li>SQL: JDBC driver no longer breaks interactions with other JDBC sources</li> |
| <li>pyarrow supported on Windows Python 2</li> |
| </ul> |
| <h3 id="deprecations">Deprecations</h3> |
| <ul> |
| <li>Deprecate HadoopInputFormatIO</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed |
| to the 2.10.0 release. Thank you to all contributors!</p> |
| <p>Ahmet Altay, Alan Myrvold, Alex Amato, Alexey Romanenko, Anton Kedin, Rui Wang, |
| Andrew Brampton Andrew Pilloud, Ankur Goenka, Antonio D&rsquo;souza, Bingfeng Shu, |
| Boyuan Zhang, brucearctor, Cade Markegard, Chaim Turkel, Chamikara Jayalath, |
| Charles Chen, Colm O hEigeartaigh, Cory, Craig Chambers, Cristian, Daniel |
| Mills, Daniel Oliveira, David Cavazos, David Hrbacek, David Moravek, Dawid |
| Wysakowicz, djhworld, Dustin Rhodes, Etienne Chauchot, Fabien Rousseau, Garrett |
| Jones, Gleb Kanterov, Heejong Lee, Ismaël Mejía, Jason Kuster, Jean-Baptiste |
| Onofré, Jeff Klukas, Joar Wandborg, Jozef Vilcek, Kadir Cetinkaya, Kasia |
| Kucharczyk, Kengo Seki, Kenneth Knowles, lcaggio, Lukasz Cwik, Łukasz Gajowy, |
| Manu Zhang, marek.simunek, Mark Daoust, Mark Liu, Maximilian Michels, Melissa |
| Pashniak, Michael Luckey, Mikhail Gryzykhin, mlotstein, morokosi, Niel |
| Markwick, Pablo Estrada, Prem Kumar Karunakaran, Reuven Lax, robbe, Robbe |
| Sneyders, Robert Bradshaw, Robert Burke, Ruoyun Huang, Ryan Williams, Sam |
| Whittle, Scott Wegner, Slava Chernyak, Theodore Siu, Thomas Weise, Udi Meiri, |
| <a href="mailto:vaclav.plajt@gmail.com">vaclav.plajt@gmail.com</a>, Valentyn Tymofieiev, Won Wook SONG, Wout Scheepers, |
| Xinyu Liu, Yueyang Qiu, Zhuo Peng</p></description></item><item><title>Blog: Apache Beam 2.9.0</title><link>/blog/beam-2.9.0/</link><pubDate>Thu, 13 Dec 2018 00:00:01 -0800</pubDate><guid>/blog/beam-2.9.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.9.0 release of Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#290-2018-12-13">download page</a> for this release.</p> |
| <p>For more information on changes in 2.9.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12344258">detailed release notes</a>.</p> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <h3 id="dependency-upgrades">Dependency Upgrades</h3> |
| <ul> |
| <li>Update google-api-client libraries to 1.27.0.</li> |
| <li>Update byte-buddy to 1.9.3</li> |
| <li>Update Flink Runner to 1.5.5</li> |
| <li>Upgrade google-apitools to 0.5.24</li> |
| </ul> |
| <h3 id="portability">Portability</h3> |
| <ul> |
| <li>Added support for user state and timers to Flink runner.</li> |
| </ul> |
| <h3 id="ios">I/Os</h3> |
| <ul> |
| <li>I/O connector for RabbitMQ.</li> |
| <li>Update SpannerIO to support unbounded writes.</li> |
| <li>Add PFADD method to RedisIO.</li> |
| </ul> |
| <h3 id="miscellaneous-fixes">Miscellaneous Fixes</h3> |
| <ul> |
| <li>Dataflow runner was updated to <strong>not</strong> use <a href="https://github.com/google/conscrypt">Conscrypt</a> as the default security provider.</li> |
| <li>Support set/delete of timers by ID in Flink runner.</li> |
| <li>Improvements to stabilize integration tests.</li> |
| <li>Updates Spark runner to show Beam metrics in web UI</li> |
| <li>Vendor gRPC and Protobuf separately from beam-model-* Java packages</li> |
| <li>Avoid reshuffle for zero and one element creates</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed |
| to the 2.9.0 release. Thank you to all contributors!</p> |
| <p>Adam Horky, Ahmet Altay, Alan Myrvold, Alex Amato, Alexey Romanenko, Andrea Foegler, Andrew Fulton, Andrew Pilloud, Ankur Goenka, Anton Kedin, Babu, Ben Song, Bingfeng Shu, Boyuan Zhang, Brian Martin, Brian Quinlan, Chamikara Jayalath, Charles Chen, Christian Schneider, Colm O hEigeartaigh, Cory Brzycki, CraigChambersG, Daniel Oliveira, David Moravek, Dusan Rychnovsky, Etienne Chauchot, Eugene Kirpichov, Fabien Rousseau, Gleb Kanterov, Heejong Lee, Henning Rohde, Ismaël Mejía, Jan Lukavský, Jaromir Vanek, Jason Kuster, Jean-Baptiste Onofré, Jeff Klukas, Jeroen Steggink, Julien Tournay, Jára Vaněk, Katarzyna Kucharczyk, Keisuke Kondo, Kenneth Knowles, Liam Miller-Cushon, Luke Cwik, Manu Zhang, Mark Liu, Maximilian Michels, Melissa Pashniak, Micah Wylde, Michael Luckey, Mike Pedersen, Mikhail Gryzykhin, Novotnik, Petr, Ondrej Kvasnicka, Pablo Estrada, Pavel Slechta, Raghu Angadi, Reuven Lax, Robbe Sneyders, Robert Bradshaw, Robert Burke, Ruoyu Liu, Ruoyun Huang, Sam Rohde, Sam sam, Scott Wegner, Simon Plovyt, Thomas Weise, Tim Robertson, Tomas Novak, Udi Meiri, Vaclav Plajt, Valentyn Tymofieiev, Varun Dhussa, Vojtech Janota, Wout Scheepers, Xinyu Liu, XuMingmin, Yifan Zou, Yueyang Qiu, akedin, amaliujia, connelloG, flyisland, huygaa11, jasonkuster, jglezt, kkpoon, mareksimunek, matthiasa4, melissa, mingmxu, nielm, reuvenlax, robbe, ruoyu90, splovyt, svXaverius, <a href="mailto:vaclav.plajt@gmail.com">vaclav.plajt@gmail.com</a>, xinyuiscool, xitep, Łukasz Gajowy</p></description></item><item><title>Blog: Apache Beam 2.8.0</title><link>/blog/beam-2.8.0/</link><pubDate>Mon, 29 Oct 2018 00:00:01 -0800</pubDate><guid>/blog/beam-2.8.0/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>We are happy to present the new 2.8.0 release of Beam. This release includes both improvements and new functionality. |
| See the <a href="/get-started/downloads/#280-2018-10-26">download page</a> for this release.</p> |
| <p>For more information on changes in 2.8.0, check out the |
| <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&amp;version=12343985">detailed release notes</a>.</p> |
| <h2 id="new-features--improvements">New Features / Improvements</h2> |
| <h2 id="known-issues">Known Issues</h2> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-4783">BEAM-4783</a> Performance degradations in certain situations when Spark runner is used.</li> |
| </ul> |
| <h3 id="dependency-upgrades">Dependency Upgrades</h3> |
| <ul> |
| <li>Elastic Search dependency upgraded to 6.3.2</li> |
| <li>google-cloud-pubsub dependency upgraded to 0.35.4</li> |
| <li>google-api-client dependency upgraded to 1.24.1</li> |
| <li>Updated Flink Runner to 1.5.3</li> |
| <li>Updated Spark runner to Spark version 2.3.2</li> |
| </ul> |
| <h3 id="sdks">SDKs</h3> |
| <ul> |
| <li>Python SDK added support for user state and timers.</li> |
| <li>Go SDK added support for side inputs.</li> |
| </ul> |
| <h3 id="portability">Portability</h3> |
| <ul> |
| <li><a href="/roadmap/portability/#python-on-flink">Python on Flink MVP</a> completed.</li> |
| </ul> |
| <h3 id="ios">I/Os</h3> |
| <ul> |
| <li>Fixes to RedisIO non-prefix read operations.</li> |
| </ul> |
| <h2 id="miscellaneous-fixes">Miscellaneous Fixes</h2> |
| <ul> |
| <li>Several bug fixes and performance improvements.</li> |
| </ul> |
| <h2 id="list-of-contributors">List of Contributors</h2> |
| <p>According to git shortlog, the following people contributed |
| to the 2.8.0 release. Thank you to all contributors!</p> |
| <p>Adam Horky, Ahmet Altay, Alan Myrvold, Aleksandr Kokhaniukov, |
| Alex Amato, Alexey Romanenko, Aljoscha Krettek, Andrew Fulton, |
| Andrew Pilloud, Ankur Goenka, Anton Kedin, Babu, Batkhuyag Batsaikhan, Ben Song, |
| Bingfeng Shu, Boyuan Zhang, Chamikara Jayalath, Charles Chen, |
| Christian Schneider, Cody Schroeder, Colm O hEigeartaigh, Daniel Mills, |
| Daniel Oliveira, Dat Tran, David Moravek, Dusan Rychnovsky, Etienne Chauchot, |
| Eugene Kirpichov, Gleb Kanterov, Heejong Lee, Henning Rohde, Ismaël Mejía, |
| Jan Lukavský, Jaromir Vanek, Jean-Baptiste Onofré, Jeff Klukas, Joar Wandborg, |
| Jozef Vilcek, Julien Phalip, Julien Tournay, Juta, Jára Vaněk, |
| Katarzyna Kucharczyk, Kengo Seki, Kenneth Knowles, Kevin Si, Kirill Kozlov, |
| Kyle Winkelman, Logan HAUSPIE, Lukasz Cwik, Manu Zhang, Mark Liu, |
| Matthias Baetens, Matthias Feys, Maximilian Michels, Melissa Pashniak, |
| Micah Wylde, Michael Luckey, Mike Pedersen, Mikhail Gryzykhin, Novotnik, |
| Petr, Ondrej Kvasnicka, Pablo Estrada, PaulVelthuis93, Pavel Slechta, |
| Rafael Fernández, Raghu Angadi, Renat, Reuven Lax, Robbe Sneyders, |
| Robert Bradshaw, Robert Burke, Rodrigo Benenson, Rong Ou, Ruoyun Huang, |
| Ryan Williams, Sam Rohde, Scott Wegner, Shinsuke Sugaya, Shnitz, Simon P, |
| Sindy Li, Stephen Lumenta, Stijn Decubber, Thomas Weise, Tomas Novak, |
| Tomas Roos, Udi Meiri, Vaclav Plajt, Valentyn Tymofieiev, Vitalii Tverdokhlib, |
| Xinyu Liu, XuMingmin, Yifan Zou, Yuan, Yueyang Qiu, aalbatross, amaliujia, |
| cclauss, connelloG, daidokoro, deepyaman, djhworld, flyisland, huygaa11, |
| jasonkuster, jglezt, kkpoon, mareksimunek, nielm, svXaverius, timrobertson100, |
| <a href="mailto:vaclav.plajt@gmail.com">vaclav.plajt@gmail.com</a>, vitaliytv, vvarma, xiliu, xinyuiscool, xitep, |
| Łukasz Gajowy.</p></description></item></channel></rss> |
