| |
| |
| <!DOCTYPE html> |
| <html class="writer-html5" lang="en" data-content_root="./"> |
| <head> |
| <meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" /> |
| |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| <title>apache_beam.pipeline module — Apache Beam 2.67.0 documentation</title> |
| <link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" /> |
| <link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" /> |
| |
| |
| <script src="_static/jquery.js?v=5d32c60e"></script> |
| <script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script> |
| <script src="_static/documentation_options.js?v=959b4fbe"></script> |
| <script src="_static/doctools.js?v=9a2dae69"></script> |
| <script src="_static/sphinx_highlight.js?v=dc90522c"></script> |
| <script src="_static/js/theme.js"></script> |
| <link rel="index" title="Index" href="genindex.html" /> |
| <link rel="search" title="Search" href="search.html" /> |
| <link rel="next" title="apache_beam.pvalue module" href="apache_beam.pvalue.html" /> |
| <link rel="prev" title="apache_beam.error module" href="apache_beam.error.html" /> |
| </head> |
| |
| <body class="wy-body-for-nav"> |
| <div class="wy-grid-for-nav"> |
| <nav data-toggle="wy-nav-shift" class="wy-nav-side"> |
| <div class="wy-side-scroll"> |
| <div class="wy-side-nav-search" > |
| |
| |
| |
| <a href="index.html" class="icon icon-home"> |
| Apache Beam |
| </a> |
| <div role="search"> |
| <form id="rtd-search-form" class="wy-form" action="search.html" method="get"> |
| <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" /> |
| <input type="hidden" name="check_keywords" value="yes" /> |
| <input type="hidden" name="area" value="default" /> |
| </form> |
| </div> |
| </div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu"> |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.coders.html">apache_beam.coders package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.dataframe.html">apache_beam.dataframe package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.io.html">apache_beam.io package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.metrics.html">apache_beam.metrics package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.ml.html">apache_beam.ml package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.options.html">apache_beam.options package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.portability.html">apache_beam.portability package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.runners.html">apache_beam.runners package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.testing.html">apache_beam.testing package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.transforms.html">apache_beam.transforms package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.typehints.html">apache_beam.typehints package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.utils.html">apache_beam.utils package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.yaml.html">apache_beam.yaml package</a></li> |
| </ul> |
| <ul class="current"> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.error.html">apache_beam.error module</a></li> |
| <li class="toctree-l1 current"><a class="current reference internal" href="#">apache_beam.pipeline module</a><ul> |
| <li class="toctree-l2"><a class="reference internal" href="#apache_beam.pipeline.Pipeline"><code class="docutils literal notranslate"><span class="pre">Pipeline</span></code></a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="#apache_beam.pipeline.Pipeline.runner_implemented_transforms"><code class="docutils literal notranslate"><span class="pre">Pipeline.runner_implemented_transforms()</span></code></a></li> |
| <li class="toctree-l3"><a class="reference internal" href="#apache_beam.pipeline.Pipeline.display_data"><code class="docutils literal notranslate"><span class="pre">Pipeline.display_data()</span></code></a></li> |
| <li class="toctree-l3"><a class="reference internal" href="#apache_beam.pipeline.Pipeline.options"><code class="docutils literal notranslate"><span class="pre">Pipeline.options</span></code></a></li> |
| <li class="toctree-l3"><a class="reference internal" href="#apache_beam.pipeline.Pipeline.allow_unsafe_triggers"><code class="docutils literal notranslate"><span class="pre">Pipeline.allow_unsafe_triggers</span></code></a></li> |
| <li class="toctree-l3"><a class="reference internal" href="#apache_beam.pipeline.Pipeline.transform_annotations"><code class="docutils literal notranslate"><span class="pre">Pipeline.transform_annotations()</span></code></a></li> |
| <li class="toctree-l3"><a class="reference internal" href="#apache_beam.pipeline.Pipeline.replace_all"><code class="docutils literal notranslate"><span class="pre">Pipeline.replace_all()</span></code></a></li> |
| <li class="toctree-l3"><a class="reference internal" href="#apache_beam.pipeline.Pipeline.run"><code class="docutils literal notranslate"><span class="pre">Pipeline.run()</span></code></a></li> |
| <li class="toctree-l3"><a class="reference internal" href="#apache_beam.pipeline.Pipeline.visit"><code class="docutils literal notranslate"><span class="pre">Pipeline.visit()</span></code></a></li> |
| <li class="toctree-l3"><a class="reference internal" href="#apache_beam.pipeline.Pipeline.apply"><code class="docutils literal notranslate"><span class="pre">Pipeline.apply()</span></code></a></li> |
| <li class="toctree-l3"><a class="reference internal" href="#apache_beam.pipeline.Pipeline.to_runner_api"><code class="docutils literal notranslate"><span class="pre">Pipeline.to_runner_api()</span></code></a></li> |
| <li class="toctree-l3"><a class="reference internal" href="#apache_beam.pipeline.Pipeline.merge_compatible_environments"><code class="docutils literal notranslate"><span class="pre">Pipeline.merge_compatible_environments()</span></code></a></li> |
| <li class="toctree-l3"><a class="reference internal" href="#apache_beam.pipeline.Pipeline.from_runner_api"><code class="docutils literal notranslate"><span class="pre">Pipeline.from_runner_api()</span></code></a></li> |
| </ul> |
| </li> |
| <li class="toctree-l2"><a class="reference internal" href="#apache_beam.pipeline.transform_annotations"><code class="docutils literal notranslate"><span class="pre">transform_annotations()</span></code></a></li> |
| </ul> |
| </li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.pvalue.html">apache_beam.pvalue module</a></li> |
| </ul> |
| |
| </div> |
| </div> |
| </nav> |
| |
| <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" > |
| <i data-toggle="wy-nav-top" class="fa fa-bars"></i> |
| <a href="index.html">Apache Beam</a> |
| </nav> |
| |
| <div class="wy-nav-content"> |
| <div class="rst-content"> |
| <div role="navigation" aria-label="Page navigation"> |
| <ul class="wy-breadcrumbs"> |
| <li><a href="index.html" class="icon icon-home" aria-label="Home"></a></li> |
| <li class="breadcrumb-item active">apache_beam.pipeline module</li> |
| <li class="wy-breadcrumbs-aside"> |
| <a href="_sources/apache_beam.pipeline.rst.txt" rel="nofollow"> View page source</a> |
| </li> |
| </ul> |
| <hr/> |
| </div> |
| <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> |
| <div itemprop="articleBody"> |
| |
| <section id="module-apache_beam.pipeline"> |
| <span id="apache-beam-pipeline-module"></span><h1>apache_beam.pipeline module<a class="headerlink" href="#module-apache_beam.pipeline" title="Link to this heading"></a></h1> |
| <p>Pipeline, the top-level Beam object.</p> |
| <p>A pipeline holds a DAG of data transforms. Conceptually the nodes of the DAG |
| are transforms (<a class="reference internal" href="apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal notranslate"><span class="pre">PTransform</span></code></a> objects) |
| and the edges are values (mostly <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> |
| objects). The transforms take as inputs one or more PValues and output one or |
| more <code class="xref py py-class docutils literal notranslate"><span class="pre">PValue</span></code> s.</p> |
| <p>The pipeline offers functionality to traverse the graph. The actual operation |
| to be executed for each node visited is specified through a runner object.</p> |
| <p>Typical usage:</p> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># Create a pipeline object using a local runner for execution.</span> |
| <span class="k">with</span> <span class="n">beam</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">(</span><span class="s1">'DirectRunner'</span><span class="p">)</span> <span class="k">as</span> <span class="n">p</span><span class="p">:</span> |
| |
| <span class="c1"># Add to the pipeline a "Create" transform. When executed this</span> |
| <span class="c1"># transform will produce a PCollection object with the specified values.</span> |
| <span class="n">pcoll</span> <span class="o">=</span> <span class="n">p</span> <span class="o">|</span> <span class="s1">'Create'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">Create</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">])</span> |
| |
| <span class="c1"># Another transform could be applied to pcoll, e.g., writing to a text file.</span> |
| <span class="c1"># For other transforms, refer to transforms/ directory.</span> |
| <span class="n">pcoll</span> <span class="o">|</span> <span class="s1">'Write'</span> <span class="o">>></span> <span class="n">beam</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">WriteToText</span><span class="p">(</span><span class="s1">'./output'</span><span class="p">)</span> |
| |
| <span class="c1"># run() will execute the DAG stored in the pipeline. The execution of the</span> |
| <span class="c1"># nodes visited is done using the specified local runner.</span> |
| </pre></div> |
| </div> |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.pipeline.Pipeline"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.pipeline.</span></span><span class="sig-name descname"><span class="pre">Pipeline</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">runner</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><span class="pre">str</span></a><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><a class="reference internal" href="apache_beam.runners.runner.html#apache_beam.runners.runner.PipelineRunner" title="apache_beam.runners.runner.PipelineRunner"><span class="pre">PipelineRunner</span></a><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><span class="pre">None</span></a></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">options</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.PipelineOptions" title="apache_beam.options.pipeline_options.PipelineOptions"><span class="pre">PipelineOptions</span></a><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><span class="pre">None</span></a></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">argv</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.List" title="(in Python v3.13)"><span class="pre">List</span></a><span class="p"><span class="pre">[</span></span><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><span class="pre">str</span></a><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><span class="pre">None</span></a></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">display_data</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Dict" title="(in Python v3.13)"><span class="pre">Dict</span></a><span class="p"><span class="pre">[</span></span><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><span class="pre">str</span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Any" title="(in Python v3.13)"><span class="pre">Any</span></a><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><span class="pre">None</span></a></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/pipeline.html#Pipeline"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.pipeline.Pipeline" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference internal" href="apache_beam.transforms.display.html#apache_beam.transforms.display.HasDisplayData" title="apache_beam.transforms.display.HasDisplayData"><code class="xref py py-class docutils literal notranslate"><span class="pre">HasDisplayData</span></code></a></p> |
| <p>A pipeline object that manages a DAG of |
| <a class="reference internal" href="apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal notranslate"><span class="pre">PTransform</span></code></a> s |
| and their <code class="xref py py-class docutils literal notranslate"><span class="pre">PValue</span></code> s.</p> |
| <p>Conceptually the <a class="reference internal" href="apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal notranslate"><span class="pre">PTransform</span></code></a> s are |
| the DAG’s nodes and the <code class="xref py py-class docutils literal notranslate"><span class="pre">PValue</span></code> s are the edges.</p> |
| <p>All the transforms applied to the pipeline must have distinct full labels. |
| If same transform instance needs to be applied then the right shift operator |
| should be used to designate new names |
| (e.g. <code class="docutils literal notranslate"><span class="pre">input</span> <span class="pre">|</span> <span class="pre">"label"</span> <span class="pre">>></span> <span class="pre">my_transform</span></code>).</p> |
| <p>Initialize a pipeline object.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>runner</strong> (<a class="reference internal" href="apache_beam.runners.runner.html#apache_beam.runners.runner.PipelineRunner" title="apache_beam.runners.runner.PipelineRunner"><em>PipelineRunner</em></a>) – An object of |
| type <a class="reference internal" href="apache_beam.runners.runner.html#apache_beam.runners.runner.PipelineRunner" title="apache_beam.runners.runner.PipelineRunner"><code class="xref py py-class docutils literal notranslate"><span class="pre">PipelineRunner</span></code></a> that will be |
| used to execute the pipeline. For registered runners, the runner name |
| can be specified, otherwise a runner object must be supplied.</p></li> |
| <li><p><strong>options</strong> (<a class="reference internal" href="apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.PipelineOptions" title="apache_beam.options.pipeline_options.PipelineOptions"><em>PipelineOptions</em></a>) – A configured |
| <a class="reference internal" href="apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.PipelineOptions" title="apache_beam.options.pipeline_options.PipelineOptions"><code class="xref py py-class docutils literal notranslate"><span class="pre">PipelineOptions</span></code></a> object |
| containing arguments that should be used for running the Beam job.</p></li> |
| <li><p><strong>argv</strong> (<em>List</em><em>[</em><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><em>str</em></a><em>]</em>) – a list of arguments (such as <a class="reference external" href="https://docs.python.org/3/library/sys.html#sys.argv" title="(in Python v3.13)"><code class="xref py py-data docutils literal notranslate"><span class="pre">sys.argv</span></code></a>) |
| to be used for building a |
| <a class="reference internal" href="apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.PipelineOptions" title="apache_beam.options.pipeline_options.PipelineOptions"><code class="xref py py-class docutils literal notranslate"><span class="pre">PipelineOptions</span></code></a> object. |
| This will only be used if argument <strong>options</strong> is <a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><code class="xref py py-data docutils literal notranslate"><span class="pre">None</span></code></a>.</p></li> |
| <li><p><strong>display_data</strong> (<em>Dict</em><em>[</em><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><em>str</em></a><em>, </em><em>Any</em><em>]</em>) – a dictionary of static data associated |
| with this pipeline that can be displayed when it runs.</p></li> |
| </ul> |
| </dd> |
| <dt class="field-even">Raises<span class="colon">:</span></dt> |
| <dd class="field-even"><p><a class="reference external" href="https://docs.python.org/3/library/exceptions.html#ValueError" title="(in Python v3.13)"><strong>ValueError</strong></a> – if either the runner or options argument is not |
| of the expected type.</p> |
| </dd> |
| </dl> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.pipeline.Pipeline.runner_implemented_transforms"> |
| <em class="property"><span class="pre">classmethod</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">runner_implemented_transforms</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.FrozenSet" title="(in Python v3.13)"><span class="pre">FrozenSet</span></a><span class="p"><span class="pre">[</span></span><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><span class="pre">str</span></a><span class="p"><span class="pre">]</span></span></span></span><a class="reference internal" href="_modules/apache_beam/pipeline.html#Pipeline.runner_implemented_transforms"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.pipeline.Pipeline.runner_implemented_transforms" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.pipeline.Pipeline.display_data"> |
| <span class="sig-name descname"><span class="pre">display_data</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Dict" title="(in Python v3.13)"><span class="pre">Dict</span></a><span class="p"><span class="pre">[</span></span><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><span class="pre">str</span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Any" title="(in Python v3.13)"><span class="pre">Any</span></a><span class="p"><span class="pre">]</span></span></span></span><a class="reference internal" href="_modules/apache_beam/pipeline.html#Pipeline.display_data"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.pipeline.Pipeline.display_data" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py property"> |
| <dt class="sig sig-object py" id="apache_beam.pipeline.Pipeline.options"> |
| <em class="property"><span class="pre">property</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">options</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><a class="reference internal" href="apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.PipelineOptions" title="apache_beam.options.pipeline_options.PipelineOptions"><span class="pre">PipelineOptions</span></a></em><a class="headerlink" href="#apache_beam.pipeline.Pipeline.options" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py property"> |
| <dt class="sig sig-object py" id="apache_beam.pipeline.Pipeline.allow_unsafe_triggers"> |
| <em class="property"><span class="pre">property</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">allow_unsafe_triggers</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/functions.html#bool" title="(in Python v3.13)"><span class="pre">bool</span></a></em><a class="headerlink" href="#apache_beam.pipeline.Pipeline.allow_unsafe_triggers" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.pipeline.Pipeline.transform_annotations"> |
| <span class="sig-name descname"><span class="pre">transform_annotations</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">annotations</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/pipeline.html#Pipeline.transform_annotations"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.pipeline.Pipeline.transform_annotations" title="Link to this definition"></a></dt> |
| <dd><p>A context manager for attaching annotations to a set of transforms.</p> |
| <p>All transforms applied while this context is active will have these |
| annotations attached. This includes sub-transforms applied within |
| composite transforms.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.pipeline.Pipeline.replace_all"> |
| <span class="sig-name descname"><span class="pre">replace_all</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">replacements</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Iterable" title="(in Python v3.13)"><span class="pre">Iterable</span></a><span class="p"><span class="pre">[</span></span><span class="pre">PTransformOverride</span><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><span class="pre">None</span></a></span></span><a class="reference internal" href="_modules/apache_beam/pipeline.html#Pipeline.replace_all"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.pipeline.Pipeline.replace_all" title="Link to this definition"></a></dt> |
| <dd><p>Dynamically replaces PTransforms in the currently populated hierarchy.</p> |
| <p>Currently this only works for replacements where input and output types |
| are exactly the same.</p> |
| <p>TODO: Update this to also work for transform overrides where input and |
| output types are different.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><p><strong>replacements</strong> (<em>List</em><em>[</em><em>PTransformOverride</em><em>]</em>) – a list of |
| <code class="xref py py-class docutils literal notranslate"><span class="pre">PTransformOverride</span></code> objects.</p> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.pipeline.Pipeline.run"> |
| <span class="sig-name descname"><span class="pre">run</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">test_runner_api</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/functions.html#bool" title="(in Python v3.13)"><span class="pre">bool</span></a><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><span class="pre">str</span></a></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">'AUTO'</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="apache_beam.runners.interactive.interactive_runner.html#apache_beam.runners.interactive.interactive_runner.PipelineResult" title="apache_beam.runners.interactive.interactive_runner.PipelineResult"><span class="pre">PipelineResult</span></a></span></span><a class="reference internal" href="_modules/apache_beam/pipeline.html#Pipeline.run"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.pipeline.Pipeline.run" title="Link to this definition"></a></dt> |
| <dd><p>Runs the pipeline. Returns whatever our runner returns after running.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.pipeline.Pipeline.visit"> |
| <span class="sig-name descname"><span class="pre">visit</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">visitor</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">PipelineVisitor</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><span class="pre">None</span></a></span></span><a class="reference internal" href="_modules/apache_beam/pipeline.html#Pipeline.visit"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.pipeline.Pipeline.visit" title="Link to this definition"></a></dt> |
| <dd><p>Visits depth-first every node of a pipeline’s DAG.</p> |
| <p>Runner-internal implementation detail; no backwards-compatibility guarantees</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><p><strong>visitor</strong> (<em>PipelineVisitor</em>) – <code class="xref py py-class docutils literal notranslate"><span class="pre">PipelineVisitor</span></code> object whose callbacks |
| will be called for each node visited. See |
| <code class="xref py py-class docutils literal notranslate"><span class="pre">PipelineVisitor</span></code> comments.</p> |
| </dd> |
| <dt class="field-even">Raises<span class="colon">:</span></dt> |
| <dd class="field-even"><ul class="simple"> |
| <li><p><a class="reference external" href="https://docs.python.org/3/library/exceptions.html#TypeError" title="(in Python v3.13)"><strong>TypeError</strong></a> – if node is specified and is not a |
| <code class="xref py py-class docutils literal notranslate"><span class="pre">PValue</span></code>.</p></li> |
| <li><p><a class="reference internal" href="apache_beam.error.html#apache_beam.error.PipelineError" title="apache_beam.error.PipelineError"><strong>PipelineError</strong></a> – if node is specified and does not |
| belong to this pipeline instance.</p></li> |
| </ul> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.pipeline.Pipeline.apply"> |
| <span class="sig-name descname"><span class="pre">apply</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">transform</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><span class="pre">PTransform</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">pvalueish</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">PValue</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><span class="pre">None</span></a></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">label</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><span class="pre">str</span></a><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><span class="pre">None</span></a></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">PValue</span></span></span><a class="reference internal" href="_modules/apache_beam/pipeline.html#Pipeline.apply"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.pipeline.Pipeline.apply" title="Link to this definition"></a></dt> |
| <dd><p>Applies a custom transform using the pvalueish specified.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>transform</strong> (<a class="reference internal" href="apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><em>PTransform</em></a>) – the |
| <a class="reference internal" href="apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal notranslate"><span class="pre">PTransform</span></code></a> to apply.</p></li> |
| <li><p><strong>pvalueish</strong> (<a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><em>PCollection</em></a>) – the input for the |
| <a class="reference internal" href="apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal notranslate"><span class="pre">PTransform</span></code></a> (typically a |
| <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a>).</p></li> |
| <li><p><strong>label</strong> (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><em>str</em></a>) – label of the |
| <a class="reference internal" href="apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal notranslate"><span class="pre">PTransform</span></code></a>.</p></li> |
| </ul> |
| </dd> |
| <dt class="field-even">Raises<span class="colon">:</span></dt> |
| <dd class="field-even"><ul class="simple"> |
| <li><p><a class="reference external" href="https://docs.python.org/3/library/exceptions.html#TypeError" title="(in Python v3.13)"><strong>TypeError</strong></a> – if the transform object extracted from the |
| argument list is not a |
| <a class="reference internal" href="apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal notranslate"><span class="pre">PTransform</span></code></a>.</p></li> |
| <li><p><a class="reference external" href="https://docs.python.org/3/library/exceptions.html#RuntimeError" title="(in Python v3.13)"><strong>RuntimeError</strong></a> – if the transform object was already applied to |
| this pipeline and needs to be cloned in order to apply again.</p></li> |
| </ul> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.pipeline.Pipeline.to_runner_api"> |
| <span class="sig-name descname"><span class="pre">to_runner_api</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">return_context</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/functions.html#bool" title="(in Python v3.13)"><span class="pre">bool</span></a></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">context</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="apache_beam.io.iobase.html#apache_beam.io.iobase.Read.PipelineContext" title="apache_beam.io.iobase.Read.PipelineContext"><span class="pre">PipelineContext</span></a><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><span class="pre">None</span></a></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">use_fake_coders</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/functions.html#bool" title="(in Python v3.13)"><span class="pre">bool</span></a></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">default_environment</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="apache_beam.transforms.environments.html#apache_beam.transforms.environments.Environment" title="apache_beam.transforms.environments.Environment"><span class="pre">environments.Environment</span></a><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><span class="pre">None</span></a></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">beam_runner_api_pb2.Pipeline</span></span></span><a class="reference internal" href="_modules/apache_beam/pipeline.html#Pipeline.to_runner_api"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.pipeline.Pipeline.to_runner_api" title="Link to this definition"></a></dt> |
| <dd><p>For internal use only; no backwards-compatibility guarantees.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.pipeline.Pipeline.merge_compatible_environments"> |
| <em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">merge_compatible_environments</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">proto</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/pipeline.html#Pipeline.merge_compatible_environments"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.pipeline.Pipeline.merge_compatible_environments" title="Link to this definition"></a></dt> |
| <dd><p>Tries to minimize the number of distinct environments by merging |
| those that are compatible (currently defined as identical).</p> |
| <p>Mutates proto as contexts may have references to proto.components.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.pipeline.Pipeline.from_runner_api"> |
| <em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_runner_api</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">proto</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Pipeline</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">runner</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="apache_beam.runners.runner.html#apache_beam.runners.runner.PipelineRunner" title="apache_beam.runners.runner.PipelineRunner"><span class="pre">PipelineRunner</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">options</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="apache_beam.options.pipeline_options.html#apache_beam.options.pipeline_options.PipelineOptions" title="apache_beam.options.pipeline_options.PipelineOptions"><span class="pre">PipelineOptions</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">return_context</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/functions.html#bool" title="(in Python v3.13)"><span class="pre">bool</span></a></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="#apache_beam.pipeline.Pipeline" title="apache_beam.pipeline.Pipeline"><span class="pre">Pipeline</span></a></span></span><a class="reference internal" href="_modules/apache_beam/pipeline.html#Pipeline.from_runner_api"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.pipeline.Pipeline.from_runner_api" title="Link to this definition"></a></dt> |
| <dd><p>For internal use only; no backwards-compatibility guarantees.</p> |
| </dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py function"> |
| <dt class="sig sig-object py" id="apache_beam.pipeline.transform_annotations"> |
| <span class="sig-prename descclassname"><span class="pre">apache_beam.pipeline.</span></span><span class="sig-name descname"><span class="pre">transform_annotations</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">annotations</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/pipeline.html#transform_annotations"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.pipeline.transform_annotations" title="Link to this definition"></a></dt> |
| <dd><p>A context manager for attaching annotations to a set of transforms.</p> |
| <p>All transforms applied while this context is active will have these |
| annotations attached. This includes sub-transforms applied within |
| composite transforms.</p> |
| </dd></dl> |
| |
| </section> |
| |
| |
| </div> |
| </div> |
| <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer"> |
| <a href="apache_beam.error.html" class="btn btn-neutral float-left" title="apache_beam.error module" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a> |
| <a href="apache_beam.pvalue.html" class="btn btn-neutral float-right" title="apache_beam.pvalue module" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a> |
| </div> |
| |
| <hr/> |
| |
| <div role="contentinfo"> |
| <p>© Copyright %Y, Apache Beam.</p> |
| </div> |
| |
| Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a |
| <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a> |
| provided by <a href="https://readthedocs.org">Read the Docs</a>. |
| |
| |
| </footer> |
| </div> |
| </div> |
| </section> |
| </div> |
| <script> |
| jQuery(function () { |
| SphinxRtdTheme.Navigation.enable(true); |
| }); |
| </script> |
| |
| </body> |
| </html> |
