| <rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Apache Beam – Apache Beam Roadmap Highlights</title><link>/roadmap/</link><description>Recent content in Apache Beam Roadmap Highlights on Apache Beam</description><generator>Hugo -- gohugo.io</generator><language>en</language><atom:link href="/roadmap/index.xml" rel="self" type="application/rss+xml"/><item><title>Roadmap: Beam SQL Roadmap</title><link>/roadmap/sql/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/roadmap/sql/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="sql">SQL</h1> |
| <p>This roadmap is in progress. In the meantime, here are available resources:</p> |
| <ul> |
| <li><a href="/documentation/dsls/sql/overview">Beam SQL documentation</a></li> |
| <li>Issues: <a href="https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Adsl-sql">dsl-sql</a></li> |
| </ul></description></item><item><title>Roadmap: Connectors - Go SDK</title><link>/roadmap/connectors-go-sdk/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/roadmap/connectors-go-sdk/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>Roadmap for connectors developed using Go SDK.</p> |
| <ul> |
| <li>Go SDK plans to utilize currently available Java and Python connectors |
| through cross-language transforms feature. |
| <ul> |
| <li>KafkaIO via Java - DONE</li> |
| <li>BigQuery via Java - In Progress</li> |
| <li>Beam SQL via Java</li> |
| </ul> |
| </li> |
| <li>The Go SDK supports SplittableDoFns for bounded pipelines, so scalable bounded pipelines are possible. |
| <ul> |
| <li>The textio package supports <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/textio#ReadSdf">ReadSdf</a> and <a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/io/textio#ReadAllSdf">ReadAllSdf</a> for efficient batch text reads.</li> |
| <li>A general FileIO will be produced to simplify adding new file based connectors.</li> |
| </ul> |
| </li> |
| </ul></description></item><item><title>Roadmap: Connectors - Java SDK</title><link>/roadmap/connectors-java-sdk/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/roadmap/connectors-java-sdk/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>Roadmap for connectors developed using Java SDK.</p> |
| <h1 id="couchbase">Couchbase</h1> |
| <p>Couchbase is a NoSQL document-oriented database. See |
| <a href="https://github.com/apache/beam/issues/18381">Issue 18381</a> for more details on the |
| planned Beam connector for Couchbase.</p> |
| <h1 id="influxdb">InfluxDB</h1> |
| <p>InfluxDB is a database for fast and highly available storage and retrieval |
| of time series data. See <a href="https://issues.apache.org/jira/browse/BEAM-2546">BEAM-2546</a> for |
| more details on the planned Beam connector for InfluxDB.</p> |
| <h1 id="memcached">Memcached</h1> |
| <p>Memcached is a distributed memory caching system. See |
| <a href="https://issues.apache.org/jira/browse/BEAM-1678">BEAM-1678</a> for more details on the |
| planned Beam connector for Memcached.</p></description></item><item><title>Roadmap: Connectors - Python SDK</title><link>/roadmap/connectors-python-sdk/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/roadmap/connectors-python-sdk/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>Roadmap for connectors developed using Python SDK.</p> |
| <h1 id="kafka">Kafka</h1> |
| <p>An Apache Kafka connectors for Python SDK that is fully developed using |
| Splittable DoFn API is planned. This is partially blocked till |
| Splittable DoFn work related to portability framework is finalized. |
| See <a href="https://issues.apache.org/jira/browse/BEAM-3788">BEAM-3788</a> for more details.</p> |
| <h1 id="parquet">Parquet</h1> |
| <p>A Python connector for Parquet file format is currently in development. |
| See <a href="https://issues.apache.org/jira/browse/BEAM-4444">BEAM-4444</a> for more details.</p></description></item><item><title>Roadmap: Euphoria API Roadmap</title><link>/roadmap/euphoria/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/roadmap/euphoria/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="euphoria-api">Euphoria API</h1> |
| <p>Easy to use Java 8 DSL for the Beam Java SDK. Provides a high-level abstraction of Beam transformations, which is both easy to read and write. Can be used as a complement to existing Beam pipelines (convertible back and forth). You can have a glimpse of the API at <a href="/documentation/sdks/java/euphoria/#wordcount-example">WordCount example</a>.</p> |
| <ul> |
| <li>Issues: <a href="https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Adsl-euphoria">dsl-euphoria</a> / <a href="https://issues.apache.org/jira/browse/BEAM-3900">BEAM-3900</a></li> |
| <li>Contact: <a href="mailto:dmvk@apache.org">David Moravek</a></li> |
| </ul> |
| <h2 id="salted-join-implementation">&ldquo;Salted&rdquo; join implementation</h2> |
| <p>Implementation of a join, that can handle large scale join of highly skewed data sets. This implementation breaks |
| the large keys into multiple splits, using key distribution approximated by count min sketch data structure.</p> |
| <h2 id="pipeline-sampling">Pipeline sampling</h2> |
| <p>In order to pick the right translation for the operator without user interference, we can leverage knowledge from |
| previous pipeline runs. We want to provide a convenient and portable way to gather this knowledge.</p> |
| <h2 id="fluent-api">Fluent API</h2> |
| <p>Implementation of an easy to use Fluent API on top of Euphoria DSL.</p> |
| <h2 id="side-outputs">Side Outputs</h2> |
| <p>An convenient API for multiple outputs.</p> |
| <h2 id="table-stream-joins">Table-stream joins</h2> |
| <p>Introduce API for converting streams to tables (KStream &lt;-&gt; KTable approach) and various types of (windowed and unwindowed) joins on them.</p></description></item><item><title>Roadmap: Flink Runner Roadmap</title><link>/roadmap/dataflow-runner/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/roadmap/dataflow-runner/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="google-cloud-dataflow-runner-roadmap">Google Cloud Dataflow Runner Roadmap</h1> |
| <p>This roadmap is in progress. In the meantime, here are available resources:</p> |
| <ul> |
| <li><a href="/documentation/runners/dataflow">Runner documentation</a></li> |
| <li>Issues: <a href="https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Arunner-dataflow">runner-dataflow</a></li> |
| </ul></description></item><item><title>Roadmap: Flink Runner Roadmap</title><link>/roadmap/flink-runner/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/roadmap/flink-runner/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="apache-flink-runner-roadmap">Apache Flink Runner Roadmap</h1> |
| <p>This roadmap is in progress. In the meantime, here are available resources:</p> |
| <ul> |
| <li><a href="/documentation/runners/flink">Runner documentation</a></li> |
| <li>Issues: <a href="https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Arunner-flink">runner-flink</a></li> |
| </ul></description></item><item><title>Roadmap: Go SDK Roadmap</title><link>/roadmap/go-sdk/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/roadmap/go-sdk/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="go-sdk-roadmap">Go SDK Roadmap</h1> |
| <p>The Go SDK is <a href="/blog/go-sdk-release/">fully released as of v2.33.0</a>.</p> |
| <p>The Go SDK the first SDK purely on the <a href="/roadmap/portability/">Beam Portability Framework</a> |
| and can execute pipelines on portable runners, like Flink, Spark, Samza, and Google Cloud Dataflow.</p> |
| <p>Current roadmap:</p> |
| <ul> |
| <li>continue building up unbounded pipeline facing features, as described on the <a href="https://cwiki.apache.org/confluence/display/BEAM/Supporting+Streaming+in+the+Go+SDK">Beam Dev Wiki</a>.</li> |
| <li>improve IO support via cross language transforms, and add scalable native transforms. <a href="/roadmap/connectors-go-sdk/">Go SDK Connector Roadmap</a></li> |
| </ul> |
| <p>Otherwise, improving examples and documentation for devs and users alike is ongoing. |
| Contributions are welcome. Please contact the <a href="mailto:dev@beam.apache.org?subject=%5BGo%20SDK%5D%20How%20can%20I%20help%3F">dev list</a> |
| for assistance in finding a place to help out.</p> |
| <ul> |
| <li>Issues: <a href="https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Asdk-go">sdk-go</a></li> |
| <li>Contact: Robert Burke (@lostluck) <a href="mailto:lostluck@apache.org?subject=%5BGo%20SDK%20Roadmap%5D">Email</a> - Please also cc the <a href="mailto:dev@beam.apache.org">dev@beam.apache.org</a> list. I strongly prefer public discussion of Go SDK matters.</li> |
| </ul></description></item><item><title>Roadmap: Java SDK Roadmap</title><link>/roadmap/java-sdk/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/roadmap/java-sdk/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="java-sdk-roadmap">Java SDK Roadmap</h1> |
| <h2 id="next-java-lts-version-support-java-21">Next Java LTS version support (Java 21)</h2> |
| <p>Work to support the next LTS release of Java is in progress. For more details |
| about the scope and info on the various tasks please see the GitHub Issue.</p> |
| <ul> |
| <li>GitHub: <a href="https://github.com/apache/beam/issues/28120">#28120</a></li> |
| </ul></description></item><item><title>Roadmap: Multi-SDK Connector Efforts</title><link>/roadmap/connectors-multi-sdk/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/roadmap/connectors-multi-sdk/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <p>Connector-related efforts that will benefit multiple SDKs.</p> |
| <h1 id="splittable-dofn">Splittable DoFn</h1> |
| <p>Splittable DoFn is the next generation sources framework for Beam that will |
| replace current frameworks for developing bounded and unbounded sources. |
| Splittable DoFn is being developed along side current Beam portability |
| efforts. See <a href="/roadmap/portability/">Beam portability framework roadmap</a> for more details.</p> |
| <h1 id="cross-language-transforms">Cross-language transforms</h1> |
| <p><em>Last updated on May 2020.</em></p> |
| <p>As an added benefit of Beam portability effort, we are able to utilize Beam transforms across SDKs. This has many benefits.</p> |
| <ul> |
| <li>Connector sharing across SDKs. For example, |
| <ul> |
| <li>Beam pipelines written using Python and Go SDKs will be able to utilize the vast selection of connectors that are currently implemented for Java SDK.</li> |
| <li>Java SDK will be able to utilize connectors for systems that only offer a Python API.</li> |
| <li>Go SDK, will be able to utilize connectors currently available for Java and Python SDKs.</li> |
| </ul> |
| </li> |
| <li>Ease of developing and maintaining Beam transforms - in general, with cross-language transforms, Beam transform authors will be able to implement new Beam transforms using a |
| language of choice and utilize these transforms from other languages reducing the maintenance and support overheads.</li> |
| <li><a href="/documentation/dsls/sql/overview/">Beam SQL</a>, that is currently only available to Java SDK, will become available to Python and Go SDKs.</li> |
| <li><a href="https://www.tensorflow.org/tfx/transform/get_started">Beam TFX transforms</a>, that are currently only available to Beam Python SDK pipelines will become available to Java and Go SDKs.</li> |
| </ul> |
| <h2 id="completed-and-ongoing-efforts">Completed and Ongoing Efforts</h2> |
| <p>Many efforts related to cross-language transforms are currently in flux. Some of the completed and ongoing efforts are given below.</p> |
| <h3 id="cross-language-transforms-api-and-expansion-service">Cross-language transforms API and expansion service</h3> |
| <p>Work related to developing/updating the cross-language transforms API for Java/Python/Go SDKs and work related to cross-language transform expansion services.</p> |
| <ul> |
| <li>Basic API for Java SDK - completed</li> |
| <li>Basic API for Python SDK - completed</li> |
| <li>Basic API for Go SDK - In progress</li> |
| <li>Basic cross-language transform expansion service for Java and Python SDKs - completed</li> |
| <li>Artifact staging - mostly completed - <a href="https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E">email thread</a>, <a href="https://docs.google.com/document/d/1XaiNekAY2sptuQRIXpjGAyaYdSc-wlJ-VKjl04c8N48/edit#heading=h.900gc947qrw8">doc</a></li> |
| </ul> |
| <h3 id="support-for-flink-runner">Support for Flink runner</h3> |
| <p>Work related to making cross-language transforms available for Flink runner.</p> |
| <ul> |
| <li>Basic support for executing cross-language transforms on portable Flink runner - completed</li> |
| </ul> |
| <h3 id="support-for-dataflow-runner">Support for Dataflow runner</h3> |
| <p>Work related to making cross-language transforms available for Dataflow runner.</p> |
| <ul> |
| <li>Basic support for executing cross-language transforms on Dataflow runner |
| <ul> |
| <li>This work requires updates to Dataflow service&rsquo;s job submission and job execution logic. This is currently being developed at Google.</li> |
| </ul> |
| </li> |
| </ul> |
| <h3 id="support-for-direct-runner">Support for Direct runner</h3> |
| <p>Work related to making cross-language transforms available on Direct runner</p> |
| <ul> |
| <li>Basic support for executing cross-language transforms on Pyton Direct runner - completed</li> |
| <li>Basic support for executing cross-language transforms on Java Direct runner - Not started</li> |
| </ul> |
| <h3 id="connectortransform-support">Connector/transform support</h3> |
| <p>Ongoing and planned work related to making existing connectors/transforms available to other SDKs through the cross-language transforms framework.</p> |
| <ul> |
| <li>Java JdbcIO - completed - <a href="https://issues.apache.org/jira/browse/BEAM-10135">BEAM-10135</a>, <a href="https://issues.apache.org/jira/browse/BEAM-10136">BEAM-10136</a></li> |
| <li>Java KafkaIO - completed - <a href="https://issues.apache.org/jira/browse/BEAM-7029">BEAM-7029</a></li> |
| <li>Java KinesisIO - completed - <a href="https://issues.apache.org/jira/browse/BEAM-10137">BEAM-10137</a>, <a href="https://issues.apache.org/jira/browse/BEAM-10138">BEAM-10138</a></li> |
| <li>Java PubSubIO - In progress - <a href="https://issues.apache.org/jira/browse/BEAM-7738">BEAM-7738</a></li> |
| <li>Java SnowflakeIO - completed - <a href="https://issues.apache.org/jira/browse/BEAM-9897">BEAM-9897</a>, <a href="https://issues.apache.org/jira/browse/BEAM-9898">BEAM-9898</a></li> |
| <li>Java SpannerIO - In progress - <a href="https://issues.apache.org/jira/browse/BEAM-10139">BEAM-10139</a>, <a href="https://issues.apache.org/jira/browse/BEAM-10140">BEAM-10140</a></li> |
| <li>Java SQL - completed - <a href="https://issues.apache.org/jira/browse/BEAM-8603">BEAM-8603</a></li> |
| </ul> |
| <h3 id="portable-beam-schema">Portable Beam schema</h3> |
| <p>Portable Beam schema support will provide a generalized mechanism for serializing and transferring data across language boundaries which will be extremely useful for pipelines that employ cross-language transforms.</p> |
| <ul> |
| <li>Make row coder a standard coder and implement in python - completed - <a href="https://issues.apache.org/jira/browse/BEAM-7886">BEAM-7886</a></li> |
| </ul> |
| <h3 id="integrationperformance-testing">Integration/Performance testing</h3> |
| <ul> |
| <li>Add an integration test suite for cross-language transforms on Flink runner - In progress - <a href="https://issues.apache.org/jira/browse/BEAM-6683">BEAM-6683</a></li> |
| </ul> |
| <h3 id="documentation">Documentation</h3> |
| <p>Work related to adding documenting on cross-language transforms to Beam Website.</p> |
| <ul> |
| <li>Document cross-language transforms API for Java/Python - Not started</li> |
| <li>Document API for making existing transforms available as cross-language transforms for Java/Python - Not started</li> |
| </ul></description></item><item><title>Roadmap: Nemo Runner Roadmap</title><link>/roadmap/nemo-runner/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/roadmap/nemo-runner/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="apache-nemo-runner-roadmap">Apache Nemo Runner Roadmap</h1> |
| <p>This roadmap is in progress. In the meantime, here are available resources:</p> |
| <ul> |
| <li><a href="/documentation/runners/nemo">Runner documentation</a></li> |
| <li>JIRA: <a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20runner-nemo">runner-nemo</a> / <a href="https://issues.apache.org/jira/projects/NEMO/issues/filter=allopenissues">nemo-jira</a></li> |
| </ul></description></item><item><title>Roadmap: Portability Framework Roadmap</title><link>/roadmap/portability/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/roadmap/portability/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="portability-framework-roadmap">Portability Framework Roadmap</h1> |
| <h2 id="overview">Overview</h2> |
| <p>Interoperability between SDKs and runners is a key aspect of Apache |
| Beam. Previously, the reality was that most runners supported the |
| Java SDK only, because each SDK-runner combination required non-trivial |
| work on both sides. Most runners are also currently written in Java, |
| which makes support of non-Java SDKs far more expensive. The |
| <em>portability framework</em> rectified this situation and provided |
| full interoperability across the Beam ecosystem.</p> |
| <p>The portability framework introduces well-defined, language-neutral |
| data structures and protocols between the SDK and runner. This interop |
| layer &ndash; called the <em>portability API</em> &ndash; ensures that SDKs and runners |
| can work with each other uniformly, reducing the interoperability |
| burden for both SDKs and runners to a constant effort. It notably |
| ensures that <em>new</em> SDKs automatically work with existing runners and |
| vice versa. The framework introduces a new runner, the <em>Universal |
| Local Runner (ULR)</em>, as a practical reference implementation that |
| complements the direct runners. Finally, it enables cross-language |
| pipelines (sharing I/O or transformations across SDKs) and |
| user-customized <a href="/documentation/runtime/environments/">execution environments</a> |
| (&ldquo;custom containers&rdquo;).</p> |
| <p>The portability API consists of a set of smaller contracts that |
| isolate SDKs and runners for job submission, management and |
| execution. These contracts use protobufs and <a href="https://grpc.io">gRPC</a> for broad language |
| support.</p> |
| <ul> |
| <li> |
| <p><strong>Job submission and management</strong>: The <em>Runner API</em> defines a |
| language-neutral pipeline representation with transformations |
| specifying the execution environment as a docker container |
| image. The latter both allows the execution side to set up the |
| right environment as well as opens the door for custom containers |
| and cross-environment pipelines. The <em>Job API</em> allows pipeline |
| execution and configuration to be managed uniformly.</p> |
| </li> |
| <li> |
| <p><strong>Job execution</strong>: The <em>SDK harness</em> is a SDK-provided |
| program responsible for executing user code and is run separately |
| from the runner. The <em>Fn API</em> defines an execution-time binary |
| contract between the SDK harness and the runner that describes how |
| execution tasks are managed and how data is transferred. In |
| addition, the runner needs to handle progress and monitoring in an |
| efficient and language-neutral way. SDK harness initialization |
| relies on the <em>Provision</em> and <em>Artifact APIs</em> for obtaining staged |
| files, pipeline options and environment information. Docker |
| provides isolation between the runner and SDK/user environments to |
| the benefit of both as defined by the <em>container contract</em>. The |
| containerization of the SDK gives it (and the user, unless the SDK |
| is closed) full control over its own environment without risk of |
| dependency conflicts. The runner has significant freedom regarding |
| how it manages the SDK harness containers.</p> |
| </li> |
| </ul> |
| <p>The goal is that all (non-direct) runners and SDKs eventually support |
| the portability API, perhaps exclusively.</p> |
| <p>If you are interested in digging in to the designs, you can find |
| them on the <a href="https://cwiki.apache.org/confluence/display/BEAM/Design+Documents">Beam developers&rsquo; wiki</a>. |
| Another overview can be found <a href="https://docs.google.com/presentation/d/1Yg8Xm4fb-oRjiLQjwLt5153hpwwTLclZrVOKP2hQifo/edit#slide=id.g42e4c9aad6_1_3070">here</a>.</p> |
| <h2 id="status">Status</h2> |
| <p>All SDKs currently support the portability framework. |
| There is also a Python Universal Local Runner and shared java runners library. |
| Performance is good and multi-language pipelines are supported. |
| Currently, the Flink and Spark runners support portable pipeline execution |
| (which is used by default for SDKs other than Java), |
| as does Dataflow when using the <a href="https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#dataflow-runner-v2">Dataflow Runner v2</a>. |
| See the |
| <a href="https://s.apache.org/apache-beam-portability-support-table">Portability support table</a> |
| for details.</p> |
| <h2 id="issues">Issues</h2> |
| <p>The portability effort touches every component, so the &ldquo;portability&rdquo; |
| label is used to identify all portability-related issues. Pure |
| design or proto definitions should use the &ldquo;beam-model&rdquo; component. A |
| common pattern for new portability features is that the overall |
| feature is in &ldquo;beam-model&rdquo; with subtasks for each SDK and runner in |
| their respective components.</p> |
| <p><strong>Issues:</strong> <a href="https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Aportability">query</a></p> |
| <p>Prerequisites: <a href="https://docs.docker.com/compose/install/">Docker</a>, <a href="https://docs.python-guide.org/starting/install3/linux/">Python</a>, <a href="https://openjdk.java.net/install/">Java 8</a></p> |
| <h3 id="python-on-flink">Running Python wordcount on Flink</h3> |
| <p>The Beam Flink runner can run Python pipelines in batch and streaming modes. |
| Please see the <a href="/documentation/runners/flink/">Flink Runner page</a> for more information on |
| how to run portable pipelines on top of Flink.</p> |
| <h3 id="python-on-spark">Running Python wordcount on Spark</h3> |
| <p>The Beam Spark runner can run Python pipelines in batch mode. |
| Please see the <a href="/documentation/runners/spark/">Spark Runner page</a> for more information on |
| how to run portable pipelines on top of Spark.</p> |
| <p>Python streaming mode is not yet supported on Spark.</p> |
| <h2 id="sdk-harness-config">SDK Harness Configuration</h2> |
| <p>See <a href="/documentation/runtime/sdk-harness-config/">here</a> for more information on SDK harness deployment options |
| and <a href="https://docs.google.com/presentation/d/1Cso0XP9dmj77OD9Bd53C1M3W1sPJF0ZnA20gzb2BPhE/edit?usp=sharing">here</a> |
| for what goes into writing a portable SDK.</p></description></item><item><title>Roadmap: Python SDK Roadmap</title><link>/roadmap/python-sdk/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/roadmap/python-sdk/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="python-sdk-roadmap">Python SDK Roadmap</h1> |
| <h2 id="sunsetting-python-2-support">Sunsetting Python 2 Support</h2> |
| <p>The Apache Beam community voted to join <a href="https://python3statement.org/">the pledge</a> to sunset Python 2 support in 2020. The Beam community will discontinue Python 2 support in 2020 and cannot guarantee long-term functional support or maintenance of Beam on Python 2. To ensure minimal disruption to your service, we strongly recommend that you upgrade to Python 3 as soon as possible.</p> |
| <h2 id="python-3-support">Python 3 Support</h2> |
| <p>Apache Beam 2.14.0 and higher support Python 3.5, 3.6, and 3.7. We&rsquo;re continuing to <a href="https://issues.apache.org/jira/browse/BEAM-1251?focusedCommentId=16890504&amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1689050">improve</a> the experience for Python 3 users and add support for Python 3.x minor versions (<a href="https://issues.apache.org/jira/browse/BEAM-8494">BEAM-8494</a>):</p> |
| <ul> |
| <li><a href="https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=245&amp;view=detail">Kanban Board</a></li> |
| <li><a href="https://docs.google.com/document/d/1s1BJVCY65LB_SYK1SU1u7NbZiFANoq-nEYaEvzRbYlA">Python 3 Conversion Quick Start Guide</a></li> |
| <li><a href="https://issues.apache.org/jira/browse/BEAM-1251">Tracking Issue</a></li> |
| <li><a href="https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE">Original Proposal</a></li> |
| </ul> |
| <p>Contributions and feedback are welcome!</p> |
| <p>If you are interested in helping, you can select an unassigned issue on the Kanban board and assign it to yourself. If you cannot assign the issue to yourself, comment on the issue. When submitting a new PR, please tag <a href="https://github.com/aaltay">@aaltay</a>, and <a href="https://github.com/tvalentyn">@tvalentyn</a>.</p> |
| <p>To report a Python 3 related issue, create a subtask in <a href="https://issues.apache.org/jira/browse/BEAM-1251">BEAM-1251</a> and cc: [~altay] and [~tvalentyn] in a JIRA comment. The best way to help us identify and investigate the issue is with a minimal pipeline that reproduces the issue.</p> |
| <p>You can also discuss encountered issues on user@ or dev@ mailing lists as appropriate.</p></description></item><item><title>Roadmap: Samza Runner Roadmap</title><link>/roadmap/samza-runner/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/roadmap/samza-runner/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="samza-runner-roadmap">Samza Runner Roadmap</h1> |
| <p>This roadmap is in progress. In the meantime, here are available resources:</p> |
| <ul> |
| <li><a href="/documentation/runners/samza">Runner documentation</a></li> |
| <li>Issues: <a href="https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Arunner-samza">runner-samza</a></li> |
| </ul></description></item><item><title>Roadmap: Spark Runner Roadmap</title><link>/roadmap/spark-runner/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/roadmap/spark-runner/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="apache-spark-runner-roadmap">Apache Spark Runner Roadmap</h1> |
| <h2 id="spark-3">Spark 3</h2> |
| <p>Support for Spark 3 in Beam&rsquo;s Spark runner is ongoing. For info on the various |
| tasks please refer to the JIRA ticket.</p> |
| <ul> |
| <li>JIRA: <a href="https://issues.apache.org/jira/browse/BEAM-7093">BEAM-7093</a></li> |
| <li>Contact: <a href="mailto:iemejia@apache.org">Ismaël Mejía</a></li> |
| </ul></description></item><item><title>Roadmap: Twister2 Runner Roadmap</title><link>/roadmap/twister2-runner/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/roadmap/twister2-runner/</guid><description> |
| <!-- |
| Licensed under the Apache License, Version 2.0 (the "License"); |
| you may not use this file except in compliance with the License. |
| You may obtain a copy of the License at |
| http://www.apache.org/licenses/LICENSE-2.0 |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <h1 id="twister2-runner-roadmap">Twister2 Runner Roadmap</h1> |
| <p>This roadmap is in progress. In the meantime, here are available resources:</p> |
| <ul> |
| <li><a href="/documentation/runners/twister2">Runner documentation</a></li> |
| <li>Issues: <a href="https://github.com/apache/beam/issues?q=is%3Aopen+is%3Aissue+label%3Arunner-twister2">runner-twister2</a></li> |
| </ul></description></item></channel></rss> |
