| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" |
| "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| |
| |
| <html xmlns="http://www.w3.org/1999/xhtml"> |
| <head> |
| <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> |
| |
| <title>apache_beam.transforms package — Apache Beam documentation</title> |
| |
| <link rel="stylesheet" href="_static/sphinxdoc.css" type="text/css" /> |
| <link rel="stylesheet" href="_static/pygments.css" type="text/css" /> |
| |
| <script type="text/javascript"> |
| var DOCUMENTATION_OPTIONS = { |
| URL_ROOT: './', |
| VERSION: '', |
| COLLAPSE_INDEX: false, |
| FILE_SUFFIX: '.html', |
| HAS_SOURCE: true, |
| SOURCELINK_SUFFIX: '.txt' |
| }; |
| </script> |
| <script type="text/javascript" src="_static/jquery.js"></script> |
| <script type="text/javascript" src="_static/underscore.js"></script> |
| <script type="text/javascript" src="_static/doctools.js"></script> |
| <link rel="index" title="Index" href="genindex.html" /> |
| <link rel="search" title="Search" href="search.html" /> |
| <link rel="next" title="apache_beam.typehints package" href="apache_beam.typehints.html" /> |
| <link rel="prev" title="apache_beam.testing package" href="apache_beam.testing.html" /> |
| </head> |
| <body role="document"> |
| <div class="related" role="navigation" aria-label="related navigation"> |
| <h3>Navigation</h3> |
| <ul> |
| <li class="right" style="margin-right: 10px"> |
| <a href="genindex.html" title="General Index" |
| accesskey="I">index</a></li> |
| <li class="right" > |
| <a href="py-modindex.html" title="Python Module Index" |
| >modules</a> |</li> |
| <li class="right" > |
| <a href="apache_beam.typehints.html" title="apache_beam.typehints package" |
| accesskey="N">next</a> |</li> |
| <li class="right" > |
| <a href="apache_beam.testing.html" title="apache_beam.testing package" |
| accesskey="P">previous</a> |</li> |
| <li class="nav-item nav-item-0"><a href="index.html">Apache Beam documentation</a> »</li> |
| <li class="nav-item nav-item-1"><a href="apache_beam.html" accesskey="U">apache_beam package</a> »</li> |
| </ul> |
| </div> |
| <div class="sphinxsidebar" role="navigation" aria-label="main navigation"> |
| <div class="sphinxsidebarwrapper"> |
| <h3><a href="index.html">Table Of Contents</a></h3> |
| <ul> |
| <li><a class="reference internal" href="#">apache_beam.transforms package</a><ul> |
| <li><a class="reference internal" href="#submodules">Submodules</a></li> |
| <li><a class="reference internal" href="#module-apache_beam.transforms.combiners">apache_beam.transforms.combiners module</a></li> |
| <li><a class="reference internal" href="#module-apache_beam.transforms.core">apache_beam.transforms.core module</a></li> |
| <li><a class="reference internal" href="#module-apache_beam.transforms.cy_combiners">apache_beam.transforms.cy_combiners module</a></li> |
| <li><a class="reference internal" href="#module-apache_beam.transforms.display">apache_beam.transforms.display module</a></li> |
| <li><a class="reference internal" href="#module-apache_beam.transforms.ptransform">apache_beam.transforms.ptransform module</a></li> |
| <li><a class="reference internal" href="#module-apache_beam.transforms.sideinputs">apache_beam.transforms.sideinputs module</a></li> |
| <li><a class="reference internal" href="#module-apache_beam.transforms.timeutil">apache_beam.transforms.timeutil module</a></li> |
| <li><a class="reference internal" href="#module-apache_beam.transforms.trigger">apache_beam.transforms.trigger module</a></li> |
| <li><a class="reference internal" href="#module-apache_beam.transforms.util">apache_beam.transforms.util module</a></li> |
| <li><a class="reference internal" href="#module-apache_beam.transforms.window">apache_beam.transforms.window module</a></li> |
| <li><a class="reference internal" href="#module-apache_beam.transforms">Module contents</a></li> |
| </ul> |
| </li> |
| </ul> |
| |
| <h4>Previous topic</h4> |
| <p class="topless"><a href="apache_beam.testing.html" |
| title="previous chapter">apache_beam.testing package</a></p> |
| <h4>Next topic</h4> |
| <p class="topless"><a href="apache_beam.typehints.html" |
| title="next chapter">apache_beam.typehints package</a></p> |
| <div role="note" aria-label="source link"> |
| <h3>This Page</h3> |
| <ul class="this-page-menu"> |
| <li><a href="_sources/apache_beam.transforms.rst.txt" |
| rel="nofollow">Show Source</a></li> |
| </ul> |
| </div> |
| <div id="searchbox" style="display: none" role="search"> |
| <h3>Quick search</h3> |
| <form class="search" action="search.html" method="get"> |
| <div><input type="text" name="q" /></div> |
| <div><input type="submit" value="Go" /></div> |
| <input type="hidden" name="check_keywords" value="yes" /> |
| <input type="hidden" name="area" value="default" /> |
| </form> |
| </div> |
| <script type="text/javascript">$('#searchbox').show(0);</script> |
| </div> |
| </div> |
| |
| <div class="document"> |
| <div class="documentwrapper"> |
| <div class="bodywrapper"> |
| <div class="body" role="main"> |
| |
| <div class="section" id="apache-beam-transforms-package"> |
| <h1>apache_beam.transforms package<a class="headerlink" href="#apache-beam-transforms-package" title="Permalink to this headline">¶</a></h1> |
| <div class="section" id="submodules"> |
| <h2>Submodules<a class="headerlink" href="#submodules" title="Permalink to this headline">¶</a></h2> |
| </div> |
| <div class="section" id="module-apache_beam.transforms.combiners"> |
| <span id="apache-beam-transforms-combiners-module"></span><h2>apache_beam.transforms.combiners module<a class="headerlink" href="#module-apache_beam.transforms.combiners" title="Permalink to this headline">¶</a></h2> |
| <p>A library of basic combiner PTransform subclasses.</p> |
| <dl class="class"> |
| <dt id="apache_beam.transforms.combiners.Count"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.combiners.</code><code class="descname">Count</code><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#Count"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.Count" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <p>Combiners for counting elements.</p> |
| <dl class="class"> |
| <dt id="apache_beam.transforms.combiners.Count.Globally"> |
| <em class="property">class </em><code class="descname">Globally</code><span class="sig-paren">(</span><em>label=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#Count.Globally"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.Count.Globally" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.ptransform.PTransform</span></code></a></p> |
| <p>combiners.Count.Globally counts the total number of elements.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.combiners.Count.Globally.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>pcoll</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#Count.Globally.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.Count.Globally.expand" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.combiners.Count.PerElement"> |
| <em class="property">class </em><code class="descname">PerElement</code><span class="sig-paren">(</span><em>label=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#Count.PerElement"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.Count.PerElement" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.ptransform.PTransform</span></code></a></p> |
| <p>combiners.Count.PerElement counts how many times each element occurs.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.combiners.Count.PerElement.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>pcoll</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#Count.PerElement.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.Count.PerElement.expand" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.combiners.Count.PerKey"> |
| <em class="property">class </em><code class="descname">PerKey</code><span class="sig-paren">(</span><em>label=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#Count.PerKey"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.Count.PerKey" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.ptransform.PTransform</span></code></a></p> |
| <p>combiners.Count.PerKey counts how many elements each unique key has.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.combiners.Count.PerKey.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>pcoll</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#Count.PerKey.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.Count.PerKey.expand" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.combiners.Mean"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.combiners.</code><code class="descname">Mean</code><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#Mean"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.Mean" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <p>Combiners for computing arithmetic means of elements.</p> |
| <dl class="class"> |
| <dt id="apache_beam.transforms.combiners.Mean.Globally"> |
| <em class="property">class </em><code class="descname">Globally</code><span class="sig-paren">(</span><em>label=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#Mean.Globally"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.Mean.Globally" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.ptransform.PTransform</span></code></a></p> |
| <p>combiners.Mean.Globally computes the arithmetic mean of the elements.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.combiners.Mean.Globally.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>pcoll</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#Mean.Globally.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.Mean.Globally.expand" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.combiners.Mean.PerKey"> |
| <em class="property">class </em><code class="descname">PerKey</code><span class="sig-paren">(</span><em>label=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#Mean.PerKey"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.Mean.PerKey" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.ptransform.PTransform</span></code></a></p> |
| <p>combiners.Mean.PerKey finds the means of the values for each key.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.combiners.Mean.PerKey.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>pcoll</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#Mean.PerKey.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.Mean.PerKey.expand" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.combiners.Sample"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.combiners.</code><code class="descname">Sample</code><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#Sample"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.Sample" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <p>Combiners for sampling n elements without replacement.</p> |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.combiners.Sample.FixedSizeGlobally"> |
| <code class="descname">FixedSizeGlobally</code><em class="property"> = <CallablePTransform(PTransform) label=[FixedSizeGlobally]></em><a class="headerlink" href="#apache_beam.transforms.combiners.Sample.FixedSizeGlobally" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.combiners.Sample.FixedSizePerKey"> |
| <code class="descname">FixedSizePerKey</code><em class="property"> = <CallablePTransform(PTransform) label=[FixedSizePerKey]></em><a class="headerlink" href="#apache_beam.transforms.combiners.Sample.FixedSizePerKey" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.combiners.Top"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.combiners.</code><code class="descname">Top</code><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#Top"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.Top" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <p>Combiners for obtaining extremal elements.</p> |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.combiners.Top.Largest"> |
| <code class="descname">Largest</code><em class="property"> = <CallablePTransform(PTransform) label=[Largest]></em><a class="headerlink" href="#apache_beam.transforms.combiners.Top.Largest" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.combiners.Top.LargestPerKey"> |
| <code class="descname">LargestPerKey</code><em class="property"> = <CallablePTransform(PTransform) label=[LargestPerKey]></em><a class="headerlink" href="#apache_beam.transforms.combiners.Top.LargestPerKey" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.combiners.Top.Of"> |
| <code class="descname">Of</code><em class="property"> = <CallablePTransform(PTransform) label=[Of]></em><a class="headerlink" href="#apache_beam.transforms.combiners.Top.Of" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.combiners.Top.PerKey"> |
| <code class="descname">PerKey</code><em class="property"> = <CallablePTransform(PTransform) label=[PerKey]></em><a class="headerlink" href="#apache_beam.transforms.combiners.Top.PerKey" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.combiners.Top.Smallest"> |
| <code class="descname">Smallest</code><em class="property"> = <CallablePTransform(PTransform) label=[Smallest]></em><a class="headerlink" href="#apache_beam.transforms.combiners.Top.Smallest" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.combiners.Top.SmallestPerKey"> |
| <code class="descname">SmallestPerKey</code><em class="property"> = <CallablePTransform(PTransform) label=[SmallestPerKey]></em><a class="headerlink" href="#apache_beam.transforms.combiners.Top.SmallestPerKey" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.combiners.ToDict"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.combiners.</code><code class="descname">ToDict</code><span class="sig-paren">(</span><em>label='ToDict'</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#ToDict"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.ToDict" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.ptransform.PTransform</span></code></a></p> |
| <p>A global CombineFn that condenses a PCollection into a single dict.</p> |
| <p>PCollections should consist of 2-tuples, notionally (key, value) pairs. |
| If multiple values are associated with the same key, only one of the values |
| will be present in the resulting dict.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.combiners.ToDict.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>pcoll</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#ToDict.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.ToDict.expand" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.combiners.ToList"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.combiners.</code><code class="descname">ToList</code><span class="sig-paren">(</span><em>label='ToList'</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#ToList"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.ToList" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.ptransform.PTransform</span></code></a></p> |
| <p>A global CombineFn that condenses a PCollection into a single list.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.combiners.ToList.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>pcoll</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/combiners.html#ToList.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.combiners.ToList.expand" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| </div> |
| <div class="section" id="module-apache_beam.transforms.core"> |
| <span id="apache-beam-transforms-core-module"></span><h2>apache_beam.transforms.core module<a class="headerlink" href="#module-apache_beam.transforms.core" title="Permalink to this headline">¶</a></h2> |
| <p>Core PTransform subclasses, such as FlatMap, GroupByKey, and Map.</p> |
| <dl class="class"> |
| <dt id="apache_beam.transforms.core.DoFn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.core.</code><code class="descname">DoFn</code><span class="sig-paren">(</span><em>*unused_args</em>, <em>**unused_kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="apache_beam.typehints.html#apache_beam.typehints.decorators.WithTypeHints" title="apache_beam.typehints.decorators.WithTypeHints"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.typehints.decorators.WithTypeHints</span></code></a>, <a class="reference internal" href="#apache_beam.transforms.display.HasDisplayData" title="apache_beam.transforms.display.HasDisplayData"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.display.HasDisplayData</span></code></a></p> |
| <p>A function object used by a transform with custom processing.</p> |
| <p>The ParDo transform is such a transform. The ParDo.apply |
| method will take an object of type DoFn and apply it to all elements of a |
| PCollection object.</p> |
| <p>In order to have concrete DoFn objects one has to subclass from DoFn and |
| define the desired behavior (start_bundle/finish_bundle and process) or wrap a |
| callable object using the CallableWrapperDoFn class.</p> |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.core.DoFn.DoFnParams"> |
| <code class="descname">DoFnParams</code><em class="property"> = ['ElementParam', 'SideInputParam', 'TimestampParam', 'WindowParam']</em><a class="headerlink" href="#apache_beam.transforms.core.DoFn.DoFnParams" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.core.DoFn.ElementParam"> |
| <code class="descname">ElementParam</code><em class="property"> = 'ElementParam'</em><a class="headerlink" href="#apache_beam.transforms.core.DoFn.ElementParam" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.core.DoFn.SideInputParam"> |
| <code class="descname">SideInputParam</code><em class="property"> = 'SideInputParam'</em><a class="headerlink" href="#apache_beam.transforms.core.DoFn.SideInputParam" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.core.DoFn.TimestampParam"> |
| <code class="descname">TimestampParam</code><em class="property"> = 'TimestampParam'</em><a class="headerlink" href="#apache_beam.transforms.core.DoFn.TimestampParam" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.core.DoFn.WindowParam"> |
| <code class="descname">WindowParam</code><em class="property"> = 'WindowParam'</em><a class="headerlink" href="#apache_beam.transforms.core.DoFn.WindowParam" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.DoFn.default_label"> |
| <code class="descname">default_label</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.default_label"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.default_label" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.DoFn.finish_bundle"> |
| <code class="descname">finish_bundle</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.finish_bundle"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.finish_bundle" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Called after a bundle of elements is processed on a worker.</p> |
| </dd></dl> |
| |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.core.DoFn.from_callable"> |
| <em class="property">static </em><code class="descname">from_callable</code><span class="sig-paren">(</span><em>fn</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.from_callable"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.from_callable" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.DoFn.get_function_arguments"> |
| <code class="descname">get_function_arguments</code><span class="sig-paren">(</span><em>func</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.get_function_arguments"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.get_function_arguments" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Return the function arguments based on the name provided. If they have |
| a _inspect_function attached to the class then use that otherwise default |
| to the python inspect library.</p> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.DoFn.infer_output_type"> |
| <code class="descname">infer_output_type</code><span class="sig-paren">(</span><em>input_type</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.infer_output_type"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.infer_output_type" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.DoFn.is_process_bounded"> |
| <code class="descname">is_process_bounded</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.is_process_bounded"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.is_process_bounded" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Checks if an object is a bound method on an instance.</p> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.DoFn.process"> |
| <code class="descname">process</code><span class="sig-paren">(</span><em>element</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.process"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.process" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Called for each element of a pipeline. The default arguments are needed |
| for the DoFnRunner to be able to pass the parameters correctly.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>element</strong> – The element to be processed</li> |
| <li><strong>*args</strong> – side inputs</li> |
| <li><strong>**kwargs</strong> – keyword side inputs</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.DoFn.start_bundle"> |
| <code class="descname">start_bundle</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.start_bundle"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.start_bundle" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Called before a bundle of elements is processed on a worker.</p> |
| <p>Elements to be processed are split into bundles and distributed |
| to workers. Before a worker calls process() on the first element |
| of its bundle, it calls this method.</p> |
| </dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.core.CombineFn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.core.</code><code class="descname">CombineFn</code><span class="sig-paren">(</span><em>*unused_args</em>, <em>**unused_kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="apache_beam.typehints.html#apache_beam.typehints.decorators.WithTypeHints" title="apache_beam.typehints.decorators.WithTypeHints"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.typehints.decorators.WithTypeHints</span></code></a>, <a class="reference internal" href="#apache_beam.transforms.display.HasDisplayData" title="apache_beam.transforms.display.HasDisplayData"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.display.HasDisplayData</span></code></a></p> |
| <p>A function object used by a Combine transform with custom processing.</p> |
| <p>A CombineFn specifies how multiple values in all or part of a PCollection can |
| be merged into a single value—essentially providing the same kind of |
| information as the arguments to the Python “reduce” builtin (except for the |
| input argument, which is an instance of CombineFnProcessContext). The |
| combining process proceeds as follows:</p> |
| <ol class="arabic simple"> |
| <li>Input values are partitioned into one or more batches.</li> |
| <li>For each batch, the create_accumulator method is invoked to create a fresh |
| initial “accumulator” value representing the combination of zero values.</li> |
| <li>For each input value in the batch, the add_input method is invoked to |
| combine more values with the accumulator for that batch.</li> |
| <li>The merge_accumulators method is invoked to combine accumulators from |
| separate batches into a single combined output accumulator value, once all |
| of the accumulators have had all the input value in their batches added to |
| them. This operation is invoked repeatedly, until there is only one |
| accumulator value left.</li> |
| <li>The extract_output operation is invoked on the final accumulator to get |
| the output value.</li> |
| </ol> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombineFn.add_input"> |
| <code class="descname">add_input</code><span class="sig-paren">(</span><em>accumulator</em>, <em>element</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.add_input"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.add_input" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Return result of folding element into accumulator.</p> |
| <p>CombineFn implementors must override add_input.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>accumulator</strong> – the current accumulator</li> |
| <li><strong>element</strong> – the element to add</li> |
| <li><strong>*args</strong> – Additional arguments and side inputs.</li> |
| <li><strong>**kwargs</strong> – Additional arguments and side inputs.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombineFn.add_inputs"> |
| <code class="descname">add_inputs</code><span class="sig-paren">(</span><em>accumulator</em>, <em>elements</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.add_inputs"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.add_inputs" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Returns the result of folding each element in elements into accumulator.</p> |
| <p>This is provided in case the implementation affords more efficient |
| bulk addition of elements. The default implementation simply loops |
| over the inputs invoking add_input for each one.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>accumulator</strong> – the current accumulator</li> |
| <li><strong>elements</strong> – the elements to add</li> |
| <li><strong>*args</strong> – Additional arguments and side inputs.</li> |
| <li><strong>**kwargs</strong> – Additional arguments and side inputs.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombineFn.apply"> |
| <code class="descname">apply</code><span class="sig-paren">(</span><em>elements</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.apply"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.apply" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Returns result of applying this CombineFn to the input values.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>elements</strong> – the set of values to combine.</li> |
| <li><strong>*args</strong> – Additional arguments and side inputs.</li> |
| <li><strong>**kwargs</strong> – Additional arguments and side inputs.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombineFn.create_accumulator"> |
| <code class="descname">create_accumulator</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.create_accumulator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.create_accumulator" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Return a fresh, empty accumulator for the combine operation.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>*args</strong> – Additional arguments and side inputs.</li> |
| <li><strong>**kwargs</strong> – Additional arguments and side inputs.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombineFn.default_label"> |
| <code class="descname">default_label</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.default_label"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.default_label" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombineFn.extract_output"> |
| <code class="descname">extract_output</code><span class="sig-paren">(</span><em>accumulator</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.extract_output"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.extract_output" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Return result of converting accumulator into the output value.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>accumulator</strong> – the final accumulator value computed by this CombineFn |
| for the entire input key or PCollection.</li> |
| <li><strong>*args</strong> – Additional arguments and side inputs.</li> |
| <li><strong>**kwargs</strong> – Additional arguments and side inputs.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombineFn.for_input_type"> |
| <code class="descname">for_input_type</code><span class="sig-paren">(</span><em>input_type</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.for_input_type"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.for_input_type" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Returns a specialized implementation of self, if it exists.</p> |
| <p>Otherwise, returns self.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>input_type</strong> – the type of input elements.</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.core.CombineFn.from_callable"> |
| <em class="property">static </em><code class="descname">from_callable</code><span class="sig-paren">(</span><em>fn</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.from_callable"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.from_callable" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.core.CombineFn.maybe_from_callable"> |
| <em class="property">static </em><code class="descname">maybe_from_callable</code><span class="sig-paren">(</span><em>fn</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.maybe_from_callable"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.maybe_from_callable" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombineFn.merge_accumulators"> |
| <code class="descname">merge_accumulators</code><span class="sig-paren">(</span><em>accumulators</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.merge_accumulators"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.merge_accumulators" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Returns the result of merging several accumulators |
| to a single accumulator value.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>accumulators</strong> – the accumulators to merge</li> |
| <li><strong>*args</strong> – Additional arguments and side inputs.</li> |
| <li><strong>**kwargs</strong> – Additional arguments and side inputs.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.core.PartitionFn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.core.</code><code class="descname">PartitionFn</code><span class="sig-paren">(</span><em>*unused_args</em>, <em>**unused_kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#PartitionFn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.PartitionFn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="apache_beam.typehints.html#apache_beam.typehints.decorators.WithTypeHints" title="apache_beam.typehints.decorators.WithTypeHints"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.typehints.decorators.WithTypeHints</span></code></a></p> |
| <p>A function object used by a Partition transform.</p> |
| <p>A PartitionFn specifies how individual values in a PCollection will be placed |
| into separate partitions, indexed by an integer.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.PartitionFn.default_label"> |
| <code class="descname">default_label</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#PartitionFn.default_label"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.PartitionFn.default_label" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.PartitionFn.partition_for"> |
| <code class="descname">partition_for</code><span class="sig-paren">(</span><em>element</em>, <em>num_partitions</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#PartitionFn.partition_for"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.PartitionFn.partition_for" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Specify which partition will receive this element.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>element</strong> – An element of the input PCollection.</li> |
| <li><strong>num_partitions</strong> – Number of partitions, i.e., output PCollections.</li> |
| <li><strong>*args</strong> – optional parameters and side inputs.</li> |
| <li><strong>**kwargs</strong> – optional parameters and side inputs.</li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">An integer in [0, num_partitions).</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.core.ParDo"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.core.</code><code class="descname">ParDo</code><span class="sig-paren">(</span><em>fn</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.ptransform.PTransformWithSideInputs</span></code></p> |
| <p>A ParDo transform.</p> |
| <p>Processes an input PCollection by applying a DoFn to each element and |
| returning the accumulated results into an output PCollection. The type of the |
| elements is not fixed as long as the DoFn can deal with it. In reality |
| the type is restrained to some extent because the elements sometimes must be |
| persisted to external storage. See the expand() method comments for a detailed |
| description of all possible arguments.</p> |
| <p>Note that the DoFn must return an iterable for each element of the input |
| PCollection. An easy way to do this is to use the yield keyword in the |
| process method.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>pcoll</strong> – a PCollection to be processed.</li> |
| <li><strong>fn</strong> – a DoFn object to be applied to each element of pcoll argument.</li> |
| <li><strong>*args</strong> – positional arguments passed to the dofn object.</li> |
| <li><strong>**kwargs</strong> – keyword arguments passed to the dofn object.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p>Note that the positional and keyword arguments will be processed in order |
| to detect PCollections that will be computed as side inputs to the |
| transform. During pipeline execution whenever the DoFn object gets executed |
| (its apply() method gets called) the PCollection arguments will be replaced |
| by values from the PCollection in the exact positions where they appear in |
| the argument lists.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.ParDo.default_type_hints"> |
| <code class="descname">default_type_hints</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.default_type_hints"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.default_type_hints" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.ParDo.display_data"> |
| <code class="descname">display_data</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.display_data"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.display_data" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.ParDo.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>pcoll</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.expand" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.ParDo.infer_output_type"> |
| <code class="descname">infer_output_type</code><span class="sig-paren">(</span><em>input_type</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.infer_output_type"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.infer_output_type" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.ParDo.make_fn"> |
| <code class="descname">make_fn</code><span class="sig-paren">(</span><em>fn</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.make_fn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.make_fn" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.ParDo.with_outputs"> |
| <code class="descname">with_outputs</code><span class="sig-paren">(</span><em>*tags</em>, <em>**main_kw</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.with_outputs"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.with_outputs" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Returns a tagged tuple allowing access to the outputs of a ParDo.</p> |
| <p>The resulting object supports access to the |
| PCollection associated with a tag (e.g., o.tag, o[tag]) and iterating over |
| the available tags (e.g., for tag in o: ...).</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>*tags</strong> – if non-empty, list of valid tags. If a list of valid tags is given, |
| it will be an error to use an undeclared tag later in the pipeline.</li> |
| <li><strong>**main_kw</strong> – dictionary empty or with one key ‘main’ defining the tag to be |
| used for the main output (which will not have a tag associated with it).</li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first">An object of type DoOutputsTuple that bundles together all the outputs |
| of a ParDo transform and allows accessing the individual |
| PCollections for each output using an object.tag syntax.</p> |
| </td> |
| </tr> |
| <tr class="field-odd field"><th class="field-name">Raises:</th><td class="field-body"><ul class="first last simple"> |
| <li><code class="xref py py-exc docutils literal"><span class="pre">TypeError</span></code> – if the self object is not a PCollection that is the result of |
| a ParDo transform.</li> |
| <li><code class="xref py py-exc docutils literal"><span class="pre">ValueError</span></code> – if main_kw contains any key other than ‘main’.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="function"> |
| <dt id="apache_beam.transforms.core.FlatMap"> |
| <code class="descclassname">apache_beam.transforms.core.</code><code class="descname">FlatMap</code><span class="sig-paren">(</span><em>fn</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#FlatMap"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.FlatMap" title="Permalink to this definition">¶</a></dt> |
| <dd><p>FlatMap is like ParDo except it takes a callable to specify the |
| transformation.</p> |
| <p>The callable must return an iterable for each element of the input |
| PCollection. The elements of these iterables will be flattened into |
| the output PCollection.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>fn</strong> – a callable object.</li> |
| <li><strong>*args</strong> – positional arguments passed to the transform callable.</li> |
| <li><strong>**kwargs</strong> – keyword arguments passed to the transform callable.</li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first">A PCollection containing the Map outputs.</p> |
| </td> |
| </tr> |
| <tr class="field-odd field"><th class="field-name">Raises:</th><td class="field-body"><p class="first last"><code class="xref py py-exc docutils literal"><span class="pre">TypeError</span></code> – If the fn passed as argument is not a callable. Typical error |
| is to pass a DoFn instance which is supported only for ParDo.</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="function"> |
| <dt id="apache_beam.transforms.core.Map"> |
| <code class="descclassname">apache_beam.transforms.core.</code><code class="descname">Map</code><span class="sig-paren">(</span><em>fn</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Map"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Map" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Map is like FlatMap except its callable returns only a single element.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>fn</strong> – a callable object.</li> |
| <li><strong>*args</strong> – positional arguments passed to the transform callable.</li> |
| <li><strong>**kwargs</strong> – keyword arguments passed to the transform callable.</li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first">A PCollection containing the Map outputs.</p> |
| </td> |
| </tr> |
| <tr class="field-odd field"><th class="field-name">Raises:</th><td class="field-body"><p class="first last"><code class="xref py py-exc docutils literal"><span class="pre">TypeError</span></code> – If the fn passed as argument is not a callable. Typical error |
| is to pass a DoFn instance which is supported only for ParDo.</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="function"> |
| <dt id="apache_beam.transforms.core.Filter"> |
| <code class="descclassname">apache_beam.transforms.core.</code><code class="descname">Filter</code><span class="sig-paren">(</span><em>fn</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Filter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Filter" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Filter is a FlatMap with its callable filtering out elements.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>fn</strong> – a callable object.</li> |
| <li><strong>*args</strong> – positional arguments passed to the transform callable.</li> |
| <li><strong>**kwargs</strong> – keyword arguments passed to the transform callable.</li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first">A PCollection containing the Filter outputs.</p> |
| </td> |
| </tr> |
| <tr class="field-odd field"><th class="field-name">Raises:</th><td class="field-body"><p class="first last"><code class="xref py py-exc docutils literal"><span class="pre">TypeError</span></code> – If the fn passed as argument is not a callable. Typical error |
| is to pass a DoFn instance which is supported only for FlatMap.</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.core.CombineGlobally"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.core.</code><code class="descname">CombineGlobally</code><span class="sig-paren">(</span><em>fn</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineGlobally"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.ptransform.PTransform</span></code></a></p> |
| <p>A CombineGlobally transform.</p> |
| <p>Reduces a PCollection to a single value by progressively applying a CombineFn |
| to portions of the PCollection (and to intermediate values created thereby). |
| See documentation in CombineFn for details on the specifics on how CombineFns |
| are applied.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>pcoll</strong> – a PCollection to be reduced into a single value.</li> |
| <li><strong>fn</strong> – a CombineFn object that will be called to progressively reduce the |
| PCollection into single values, or a callable suitable for wrapping |
| by CallableWrapperCombineFn.</li> |
| <li><strong>*args</strong> – positional arguments passed to the CombineFn object.</li> |
| <li><strong>**kwargs</strong> – keyword arguments passed to the CombineFn object.</li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Raises:</th><td class="field-body"><p class="first"><code class="xref py py-exc docutils literal"><span class="pre">TypeError</span></code> – If the output type of the input PCollection is not compatible |
| with Iterable[A].</p> |
| </td> |
| </tr> |
| <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">A single-element PCollection containing the main output of the Combine |
| transform.</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p>Note that the positional and keyword arguments will be processed in order |
| to detect PObjects that will be computed as side inputs to the transform. |
| During pipeline execution whenever the CombineFn object gets executed (i.e., |
| any of the CombineFn methods get called), the PObject arguments will be |
| replaced by their actual value in the exact position where they appear in |
| the argument lists.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombineGlobally.as_singleton_view"> |
| <code class="descname">as_singleton_view</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineGlobally.as_singleton_view"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.as_singleton_view" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.core.CombineGlobally.as_view"> |
| <code class="descname">as_view</code><em class="property"> = False</em><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.as_view" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombineGlobally.default_label"> |
| <code class="descname">default_label</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineGlobally.default_label"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.default_label" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombineGlobally.display_data"> |
| <code class="descname">display_data</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineGlobally.display_data"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.display_data" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombineGlobally.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>pcoll</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineGlobally.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.expand" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.core.CombineGlobally.has_defaults"> |
| <code class="descname">has_defaults</code><em class="property"> = True</em><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.has_defaults" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombineGlobally.with_defaults"> |
| <code class="descname">with_defaults</code><span class="sig-paren">(</span><em>has_defaults=True</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineGlobally.with_defaults"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.with_defaults" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombineGlobally.without_defaults"> |
| <code class="descname">without_defaults</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineGlobally.without_defaults"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.without_defaults" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.core.CombinePerKey"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.core.</code><code class="descname">CombinePerKey</code><span class="sig-paren">(</span><em>fn</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombinePerKey"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombinePerKey" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.ptransform.PTransformWithSideInputs</span></code></p> |
| <p>A per-key Combine transform.</p> |
| <p>Identifies sets of values associated with the same key in the input |
| PCollection, then applies a CombineFn to condense those sets to single |
| values. See documentation in CombineFn for details on the specifics on how |
| CombineFns are applied.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>pcoll</strong> – input pcollection.</li> |
| <li><strong>fn</strong> – instance of CombineFn to apply to all values under the same key in |
| pcoll, or a callable whose signature is <code class="docutils literal"><span class="pre">f(iterable,</span> <span class="pre">*args,</span> <span class="pre">**kwargs)</span></code> |
| (e.g., sum, max).</li> |
| <li><strong>*args</strong> – arguments and side inputs, passed directly to the CombineFn.</li> |
| <li><strong>**kwargs</strong> – arguments and side inputs, passed directly to the CombineFn.</li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">A PObject holding the result of the combine operation.</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombinePerKey.default_label"> |
| <code class="descname">default_label</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombinePerKey.default_label"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombinePerKey.default_label" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombinePerKey.display_data"> |
| <code class="descname">display_data</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombinePerKey.display_data"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombinePerKey.display_data" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombinePerKey.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>pcoll</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombinePerKey.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombinePerKey.expand" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombinePerKey.make_fn"> |
| <code class="descname">make_fn</code><span class="sig-paren">(</span><em>fn</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombinePerKey.make_fn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombinePerKey.make_fn" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.core.CombineValues"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.core.</code><code class="descname">CombineValues</code><span class="sig-paren">(</span><em>fn</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineValues"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineValues" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.ptransform.PTransformWithSideInputs</span></code></p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombineValues.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>pcoll</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineValues.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineValues.expand" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.CombineValues.make_fn"> |
| <code class="descname">make_fn</code><span class="sig-paren">(</span><em>fn</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineValues.make_fn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineValues.make_fn" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.core.GroupByKey"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.core.</code><code class="descname">GroupByKey</code><span class="sig-paren">(</span><em>label=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupByKey"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupByKey" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.ptransform.PTransform</span></code></a></p> |
| <p>A group by key transform.</p> |
| <p>Processes an input PCollection consisting of key/value pairs represented as a |
| tuple pair. The result is a PCollection where values having a common key are |
| grouped together. For example (a, 1), (b, 2), (a, 3) will result into |
| (a, [1, 3]), (b, [2]).</p> |
| <p>The implementation here is used only when run on the local direct runner.</p> |
| <dl class="class"> |
| <dt id="apache_beam.transforms.core.GroupByKey.ReifyWindows"> |
| <em class="property">class </em><code class="descname">ReifyWindows</code><span class="sig-paren">(</span><em>*unused_args</em>, <em>**unused_kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupByKey.ReifyWindows"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupByKey.ReifyWindows" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.core.DoFn</span></code></a></p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.GroupByKey.ReifyWindows.infer_output_type"> |
| <code class="descname">infer_output_type</code><span class="sig-paren">(</span><em>input_type</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupByKey.ReifyWindows.infer_output_type"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupByKey.ReifyWindows.infer_output_type" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.GroupByKey.ReifyWindows.process"> |
| <code class="descname">process</code><span class="sig-paren">(</span><em>element</em>, <em>window='WindowParam'</em>, <em>timestamp='TimestampParam'</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupByKey.ReifyWindows.process"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupByKey.ReifyWindows.process" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.GroupByKey.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>pcoll</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupByKey.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupByKey.expand" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.core.Partition"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.core.</code><code class="descname">Partition</code><span class="sig-paren">(</span><em>fn</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Partition"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Partition" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.ptransform.PTransformWithSideInputs</span></code></p> |
| <p>Split a PCollection into several partitions.</p> |
| <p>Uses the specified PartitionFn to separate an input PCollection into the |
| specified number of sub-PCollections.</p> |
| <p>When apply()d, a Partition() PTransform requires the following:</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>partitionfn</strong> – a PartitionFn, or a callable with the signature described in |
| CallableWrapperPartitionFn.</li> |
| <li><strong>n</strong> – number of output partitions.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p>The result of this PTransform is a simple list of the output PCollections |
| representing each of n partitions, in order.</p> |
| <dl class="class"> |
| <dt id="apache_beam.transforms.core.Partition.ApplyPartitionFnFn"> |
| <em class="property">class </em><code class="descname">ApplyPartitionFnFn</code><span class="sig-paren">(</span><em>*unused_args</em>, <em>**unused_kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Partition.ApplyPartitionFnFn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Partition.ApplyPartitionFnFn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.core.DoFn</span></code></a></p> |
| <p>A DoFn that applies a PartitionFn.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.Partition.ApplyPartitionFnFn.process"> |
| <code class="descname">process</code><span class="sig-paren">(</span><em>element</em>, <em>partitionfn</em>, <em>n</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Partition.ApplyPartitionFnFn.process"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Partition.ApplyPartitionFnFn.process" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.Partition.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>pcoll</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Partition.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Partition.expand" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.Partition.make_fn"> |
| <code class="descname">make_fn</code><span class="sig-paren">(</span><em>fn</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Partition.make_fn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Partition.make_fn" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.core.Windowing"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.core.</code><code class="descname">Windowing</code><span class="sig-paren">(</span><em>windowfn</em>, <em>triggerfn=None</em>, <em>accumulation_mode=None</em>, <em>timestamp_combiner=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Windowing"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Windowing" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.core.Windowing.from_runner_api"> |
| <em class="property">static </em><code class="descname">from_runner_api</code><span class="sig-paren">(</span><em>proto</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Windowing.from_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Windowing.from_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.Windowing.is_default"> |
| <code class="descname">is_default</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Windowing.is_default"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Windowing.is_default" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.Windowing.to_runner_api"> |
| <code class="descname">to_runner_api</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Windowing.to_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Windowing.to_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.core.WindowInto"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.core.</code><code class="descname">WindowInto</code><span class="sig-paren">(</span><em>windowfn</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WindowInto"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.WindowInto" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.core.ParDo" title="apache_beam.transforms.core.ParDo"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.core.ParDo</span></code></a></p> |
| <p>A window transform assigning windows to each element of a PCollection.</p> |
| <p>Transforms an input PCollection by applying a windowing function to each |
| element. Each transformed element in the result will be a WindowedValue |
| element with the same input value and timestamp, with its new set of windows |
| determined by the windowing function.</p> |
| <dl class="class"> |
| <dt id="apache_beam.transforms.core.WindowInto.WindowIntoFn"> |
| <em class="property">class </em><code class="descname">WindowIntoFn</code><span class="sig-paren">(</span><em>windowing</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WindowInto.WindowIntoFn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.WindowInto.WindowIntoFn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.core.DoFn</span></code></a></p> |
| <p>A DoFn that applies a WindowInto operation.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.WindowInto.WindowIntoFn.process"> |
| <code class="descname">process</code><span class="sig-paren">(</span><em>element</em>, <em>timestamp='TimestampParam'</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WindowInto.WindowIntoFn.process"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.WindowInto.WindowIntoFn.process" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.WindowInto.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>pcoll</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WindowInto.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.WindowInto.expand" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.core.WindowInto.from_runner_api_parameter"> |
| <em class="property">static </em><code class="descname">from_runner_api_parameter</code><span class="sig-paren">(</span><em>proto</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WindowInto.from_runner_api_parameter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.WindowInto.from_runner_api_parameter" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.WindowInto.get_windowing"> |
| <code class="descname">get_windowing</code><span class="sig-paren">(</span><em>unused_inputs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WindowInto.get_windowing"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.WindowInto.get_windowing" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.WindowInto.infer_output_type"> |
| <code class="descname">infer_output_type</code><span class="sig-paren">(</span><em>input_type</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WindowInto.infer_output_type"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.WindowInto.infer_output_type" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.WindowInto.to_runner_api_parameter"> |
| <code class="descname">to_runner_api_parameter</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WindowInto.to_runner_api_parameter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.WindowInto.to_runner_api_parameter" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.core.Flatten"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.core.</code><code class="descname">Flatten</code><span class="sig-paren">(</span><em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Flatten"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Flatten" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.ptransform.PTransform</span></code></a></p> |
| <p>Merges several PCollections into a single PCollection.</p> |
| <p>Copies all elements in 0 or more PCollections into a single output |
| PCollection. If there are no input PCollections, the resulting PCollection |
| will be empty (but see also kwargs below).</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>**kwargs</strong> – Accepts a single named argument “pipeline”, which specifies the |
| pipeline that “owns” this PTransform. Ordinarily Flatten can obtain this |
| information from one of the input PCollections, but if there are none (or |
| if there’s a chance there may be none), this argument is the only way to |
| provide pipeline information and should be considered mandatory.</td> |
| </tr> |
| </tbody> |
| </table> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.Flatten.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>pcolls</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Flatten.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Flatten.expand" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.core.Flatten.from_runner_api_parameter"> |
| <em class="property">static </em><code class="descname">from_runner_api_parameter</code><span class="sig-paren">(</span><em>unused_parameter</em>, <em>unused_context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Flatten.from_runner_api_parameter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Flatten.from_runner_api_parameter" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.Flatten.get_windowing"> |
| <code class="descname">get_windowing</code><span class="sig-paren">(</span><em>inputs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Flatten.get_windowing"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Flatten.get_windowing" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.Flatten.to_runner_api_parameter"> |
| <code class="descname">to_runner_api_parameter</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Flatten.to_runner_api_parameter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Flatten.to_runner_api_parameter" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.core.Create"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.core.</code><code class="descname">Create</code><span class="sig-paren">(</span><em>value</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Create"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Create" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.ptransform.PTransform</span></code></a></p> |
| <p>A transform that creates a PCollection from an iterable.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.Create.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>pbegin</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Create.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Create.expand" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.Create.get_output_type"> |
| <code class="descname">get_output_type</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Create.get_output_type"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Create.get_output_type" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.Create.get_windowing"> |
| <code class="descname">get_windowing</code><span class="sig-paren">(</span><em>unused_inputs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Create.get_windowing"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Create.get_windowing" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.core.Create.infer_output_type"> |
| <code class="descname">infer_output_type</code><span class="sig-paren">(</span><em>unused_input_type</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Create.infer_output_type"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.core.Create.infer_output_type" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| </div> |
| <div class="section" id="module-apache_beam.transforms.cy_combiners"> |
| <span id="apache-beam-transforms-cy-combiners-module"></span><h2>apache_beam.transforms.cy_combiners module<a class="headerlink" href="#module-apache_beam.transforms.cy_combiners" title="Permalink to this headline">¶</a></h2> |
| <p>A library of basic cythonized CombineFn subclasses.</p> |
| <p>For internal use only; no backwards-compatibility guarantees.</p> |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.AccumulatorCombineFn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">AccumulatorCombineFn</code><span class="sig-paren">(</span><em>*unused_args</em>, <em>**unused_kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#AccumulatorCombineFn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.AccumulatorCombineFn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.core.CombineFn" title="apache_beam.transforms.core.CombineFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.core.CombineFn</span></code></a></p> |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.cy_combiners.AccumulatorCombineFn.add_input"> |
| <em class="property">static </em><code class="descname">add_input</code><span class="sig-paren">(</span><em>accumulator</em>, <em>element</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#AccumulatorCombineFn.add_input"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.AccumulatorCombineFn.add_input" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.AccumulatorCombineFn.create_accumulator"> |
| <code class="descname">create_accumulator</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#AccumulatorCombineFn.create_accumulator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.AccumulatorCombineFn.create_accumulator" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.cy_combiners.AccumulatorCombineFn.extract_output"> |
| <em class="property">static </em><code class="descname">extract_output</code><span class="sig-paren">(</span><em>accumulator</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#AccumulatorCombineFn.extract_output"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.AccumulatorCombineFn.extract_output" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.AccumulatorCombineFn.merge_accumulators"> |
| <code class="descname">merge_accumulators</code><span class="sig-paren">(</span><em>accumulators</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#AccumulatorCombineFn.merge_accumulators"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.AccumulatorCombineFn.merge_accumulators" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.AllAccumulator"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">AllAccumulator</code><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#AllAccumulator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.AllAccumulator" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.AllAccumulator.add_input"> |
| <code class="descname">add_input</code><span class="sig-paren">(</span><em>element</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#AllAccumulator.add_input"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.AllAccumulator.add_input" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.AllAccumulator.extract_output"> |
| <code class="descname">extract_output</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#AllAccumulator.extract_output"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.AllAccumulator.extract_output" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.AllAccumulator.merge"> |
| <code class="descname">merge</code><span class="sig-paren">(</span><em>accumulators</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#AllAccumulator.merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.AllAccumulator.merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.AllCombineFn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">AllCombineFn</code><span class="sig-paren">(</span><em>*unused_args</em>, <em>**unused_kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#AllCombineFn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.AllCombineFn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.cy_combiners.AccumulatorCombineFn" title="apache_beam.transforms.cy_combiners.AccumulatorCombineFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.cy_combiners.AccumulatorCombineFn</span></code></a></p> |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.AnyAccumulator"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">AnyAccumulator</code><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#AnyAccumulator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.AnyAccumulator" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.AnyAccumulator.add_input"> |
| <code class="descname">add_input</code><span class="sig-paren">(</span><em>element</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#AnyAccumulator.add_input"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.AnyAccumulator.add_input" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.AnyAccumulator.extract_output"> |
| <code class="descname">extract_output</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#AnyAccumulator.extract_output"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.AnyAccumulator.extract_output" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.AnyAccumulator.merge"> |
| <code class="descname">merge</code><span class="sig-paren">(</span><em>accumulators</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#AnyAccumulator.merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.AnyAccumulator.merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.AnyCombineFn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">AnyCombineFn</code><span class="sig-paren">(</span><em>*unused_args</em>, <em>**unused_kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#AnyCombineFn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.AnyCombineFn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.cy_combiners.AccumulatorCombineFn" title="apache_beam.transforms.cy_combiners.AccumulatorCombineFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.cy_combiners.AccumulatorCombineFn</span></code></a></p> |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.CountAccumulator"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">CountAccumulator</code><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#CountAccumulator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.CountAccumulator" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.CountAccumulator.add_input"> |
| <code class="descname">add_input</code><span class="sig-paren">(</span><em>unused_element</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#CountAccumulator.add_input"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.CountAccumulator.add_input" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.CountAccumulator.extract_output"> |
| <code class="descname">extract_output</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#CountAccumulator.extract_output"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.CountAccumulator.extract_output" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.CountAccumulator.merge"> |
| <code class="descname">merge</code><span class="sig-paren">(</span><em>accumulators</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#CountAccumulator.merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.CountAccumulator.merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.CountCombineFn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">CountCombineFn</code><span class="sig-paren">(</span><em>*unused_args</em>, <em>**unused_kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#CountCombineFn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.CountCombineFn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.cy_combiners.AccumulatorCombineFn" title="apache_beam.transforms.cy_combiners.AccumulatorCombineFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.cy_combiners.AccumulatorCombineFn</span></code></a></p> |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.MaxDoubleAccumulator"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">MaxDoubleAccumulator</code><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MaxDoubleAccumulator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MaxDoubleAccumulator" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MaxDoubleAccumulator.add_input"> |
| <code class="descname">add_input</code><span class="sig-paren">(</span><em>element</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MaxDoubleAccumulator.add_input"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MaxDoubleAccumulator.add_input" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MaxDoubleAccumulator.extract_output"> |
| <code class="descname">extract_output</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MaxDoubleAccumulator.extract_output"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MaxDoubleAccumulator.extract_output" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MaxDoubleAccumulator.merge"> |
| <code class="descname">merge</code><span class="sig-paren">(</span><em>accumulators</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MaxDoubleAccumulator.merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MaxDoubleAccumulator.merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.MaxFloatFn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">MaxFloatFn</code><span class="sig-paren">(</span><em>*unused_args</em>, <em>**unused_kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MaxFloatFn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MaxFloatFn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.cy_combiners.AccumulatorCombineFn" title="apache_beam.transforms.cy_combiners.AccumulatorCombineFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.cy_combiners.AccumulatorCombineFn</span></code></a></p> |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.MaxInt64Accumulator"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">MaxInt64Accumulator</code><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MaxInt64Accumulator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MaxInt64Accumulator" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MaxInt64Accumulator.add_input"> |
| <code class="descname">add_input</code><span class="sig-paren">(</span><em>element</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MaxInt64Accumulator.add_input"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MaxInt64Accumulator.add_input" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MaxInt64Accumulator.extract_output"> |
| <code class="descname">extract_output</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MaxInt64Accumulator.extract_output"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MaxInt64Accumulator.extract_output" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MaxInt64Accumulator.merge"> |
| <code class="descname">merge</code><span class="sig-paren">(</span><em>accumulators</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MaxInt64Accumulator.merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MaxInt64Accumulator.merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.MaxInt64Fn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">MaxInt64Fn</code><span class="sig-paren">(</span><em>*unused_args</em>, <em>**unused_kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MaxInt64Fn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MaxInt64Fn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.cy_combiners.AccumulatorCombineFn" title="apache_beam.transforms.cy_combiners.AccumulatorCombineFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.cy_combiners.AccumulatorCombineFn</span></code></a></p> |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.MeanDoubleAccumulator"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">MeanDoubleAccumulator</code><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MeanDoubleAccumulator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MeanDoubleAccumulator" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MeanDoubleAccumulator.add_input"> |
| <code class="descname">add_input</code><span class="sig-paren">(</span><em>element</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MeanDoubleAccumulator.add_input"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MeanDoubleAccumulator.add_input" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MeanDoubleAccumulator.extract_output"> |
| <code class="descname">extract_output</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MeanDoubleAccumulator.extract_output"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MeanDoubleAccumulator.extract_output" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MeanDoubleAccumulator.merge"> |
| <code class="descname">merge</code><span class="sig-paren">(</span><em>accumulators</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MeanDoubleAccumulator.merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MeanDoubleAccumulator.merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.MeanFloatFn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">MeanFloatFn</code><span class="sig-paren">(</span><em>*unused_args</em>, <em>**unused_kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MeanFloatFn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MeanFloatFn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.cy_combiners.AccumulatorCombineFn" title="apache_beam.transforms.cy_combiners.AccumulatorCombineFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.cy_combiners.AccumulatorCombineFn</span></code></a></p> |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.MeanInt64Accumulator"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">MeanInt64Accumulator</code><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MeanInt64Accumulator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MeanInt64Accumulator" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MeanInt64Accumulator.add_input"> |
| <code class="descname">add_input</code><span class="sig-paren">(</span><em>element</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MeanInt64Accumulator.add_input"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MeanInt64Accumulator.add_input" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MeanInt64Accumulator.extract_output"> |
| <code class="descname">extract_output</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MeanInt64Accumulator.extract_output"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MeanInt64Accumulator.extract_output" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MeanInt64Accumulator.merge"> |
| <code class="descname">merge</code><span class="sig-paren">(</span><em>accumulators</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MeanInt64Accumulator.merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MeanInt64Accumulator.merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.MeanInt64Fn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">MeanInt64Fn</code><span class="sig-paren">(</span><em>*unused_args</em>, <em>**unused_kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MeanInt64Fn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MeanInt64Fn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.cy_combiners.AccumulatorCombineFn" title="apache_beam.transforms.cy_combiners.AccumulatorCombineFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.cy_combiners.AccumulatorCombineFn</span></code></a></p> |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.MinDoubleAccumulator"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">MinDoubleAccumulator</code><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MinDoubleAccumulator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MinDoubleAccumulator" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MinDoubleAccumulator.add_input"> |
| <code class="descname">add_input</code><span class="sig-paren">(</span><em>element</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MinDoubleAccumulator.add_input"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MinDoubleAccumulator.add_input" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MinDoubleAccumulator.extract_output"> |
| <code class="descname">extract_output</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MinDoubleAccumulator.extract_output"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MinDoubleAccumulator.extract_output" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MinDoubleAccumulator.merge"> |
| <code class="descname">merge</code><span class="sig-paren">(</span><em>accumulators</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MinDoubleAccumulator.merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MinDoubleAccumulator.merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.MinFloatFn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">MinFloatFn</code><span class="sig-paren">(</span><em>*unused_args</em>, <em>**unused_kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MinFloatFn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MinFloatFn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.cy_combiners.AccumulatorCombineFn" title="apache_beam.transforms.cy_combiners.AccumulatorCombineFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.cy_combiners.AccumulatorCombineFn</span></code></a></p> |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.MinInt64Accumulator"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">MinInt64Accumulator</code><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MinInt64Accumulator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MinInt64Accumulator" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MinInt64Accumulator.add_input"> |
| <code class="descname">add_input</code><span class="sig-paren">(</span><em>element</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MinInt64Accumulator.add_input"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MinInt64Accumulator.add_input" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MinInt64Accumulator.extract_output"> |
| <code class="descname">extract_output</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MinInt64Accumulator.extract_output"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MinInt64Accumulator.extract_output" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.MinInt64Accumulator.merge"> |
| <code class="descname">merge</code><span class="sig-paren">(</span><em>accumulators</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MinInt64Accumulator.merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MinInt64Accumulator.merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.MinInt64Fn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">MinInt64Fn</code><span class="sig-paren">(</span><em>*unused_args</em>, <em>**unused_kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#MinInt64Fn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.MinInt64Fn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.cy_combiners.AccumulatorCombineFn" title="apache_beam.transforms.cy_combiners.AccumulatorCombineFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.cy_combiners.AccumulatorCombineFn</span></code></a></p> |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.SumDoubleAccumulator"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">SumDoubleAccumulator</code><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#SumDoubleAccumulator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.SumDoubleAccumulator" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.SumDoubleAccumulator.add_input"> |
| <code class="descname">add_input</code><span class="sig-paren">(</span><em>element</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#SumDoubleAccumulator.add_input"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.SumDoubleAccumulator.add_input" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.SumDoubleAccumulator.extract_output"> |
| <code class="descname">extract_output</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#SumDoubleAccumulator.extract_output"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.SumDoubleAccumulator.extract_output" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.SumDoubleAccumulator.merge"> |
| <code class="descname">merge</code><span class="sig-paren">(</span><em>accumulators</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#SumDoubleAccumulator.merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.SumDoubleAccumulator.merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.SumFloatFn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">SumFloatFn</code><span class="sig-paren">(</span><em>*unused_args</em>, <em>**unused_kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#SumFloatFn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.SumFloatFn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.cy_combiners.AccumulatorCombineFn" title="apache_beam.transforms.cy_combiners.AccumulatorCombineFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.cy_combiners.AccumulatorCombineFn</span></code></a></p> |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.SumInt64Accumulator"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">SumInt64Accumulator</code><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#SumInt64Accumulator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.SumInt64Accumulator" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.SumInt64Accumulator.add_input"> |
| <code class="descname">add_input</code><span class="sig-paren">(</span><em>element</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#SumInt64Accumulator.add_input"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.SumInt64Accumulator.add_input" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.SumInt64Accumulator.extract_output"> |
| <code class="descname">extract_output</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#SumInt64Accumulator.extract_output"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.SumInt64Accumulator.extract_output" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.cy_combiners.SumInt64Accumulator.merge"> |
| <code class="descname">merge</code><span class="sig-paren">(</span><em>accumulators</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#SumInt64Accumulator.merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.SumInt64Accumulator.merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.cy_combiners.SumInt64Fn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.cy_combiners.</code><code class="descname">SumInt64Fn</code><span class="sig-paren">(</span><em>*unused_args</em>, <em>**unused_kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/cy_combiners.html#SumInt64Fn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.cy_combiners.SumInt64Fn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.cy_combiners.AccumulatorCombineFn" title="apache_beam.transforms.cy_combiners.AccumulatorCombineFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.cy_combiners.AccumulatorCombineFn</span></code></a></p> |
| </dd></dl> |
| |
| </div> |
| <div class="section" id="module-apache_beam.transforms.display"> |
| <span id="apache-beam-transforms-display-module"></span><h2>apache_beam.transforms.display module<a class="headerlink" href="#module-apache_beam.transforms.display" title="Permalink to this headline">¶</a></h2> |
| <p>DisplayData, its classes, interfaces and methods.</p> |
| <p>The classes in this module allow users and transform developers to define |
| static display data to be displayed when a pipeline runs. PTransforms, DoFns |
| and other pipeline components are subclasses of the HasDisplayData mixin. To |
| add static display data to a component, you can override the display_data |
| method of the HasDisplayData class.</p> |
| <p>Available classes:</p> |
| <ul class="simple"> |
| <li><dl class="first docutils"> |
| <dt>HasDisplayData - Components that inherit from this class can have static</dt> |
| <dd>display data shown in the UI.</dd> |
| </dl> |
| </li> |
| <li>DisplayDataItem - This class represents static display data elements.</li> |
| <li><dl class="first docutils"> |
| <dt>DisplayData - Internal class that is used to create display data and</dt> |
| <dd>communicate it to the API.</dd> |
| </dl> |
| </li> |
| </ul> |
| <dl class="class"> |
| <dt id="apache_beam.transforms.display.HasDisplayData"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.display.</code><code class="descname">HasDisplayData</code><a class="reference internal" href="_modules/apache_beam/transforms/display.html#HasDisplayData"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.display.HasDisplayData" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <p>Basic mixin for elements that contain display data.</p> |
| <p>It implements only the display_data method and a _namespace method.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.display.HasDisplayData.display_data"> |
| <code class="descname">display_data</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/display.html#HasDisplayData.display_data"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.display.HasDisplayData.display_data" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Returns the display data associated to a pipeline component.</p> |
| <p>It should be reimplemented in pipeline components that wish to have |
| static display data.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">value pairs. The value might be an |
| integer, float or string value; a DisplayDataItem for values that |
| have more data (e.g. short value, label, url); or a HasDisplayData |
| instance that has more display data that should be picked up. For |
| example:<dl class="docutils"> |
| <dt>{ ‘key1’: ‘string_value’,</dt> |
| <dd>‘key2’: 1234, |
| ‘key3’: 3.14159265, |
| ‘key4’: DisplayDataItem(‘apache.org’, url=’<a class="reference external" href="http://apache.org">http://apache.org</a>‘), |
| ‘key5’: subComponent }</dd> |
| </dl> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body">A dictionary containing key</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.display.DisplayDataItem"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.display.</code><code class="descname">DisplayDataItem</code><span class="sig-paren">(</span><em>value</em>, <em>url=None</em>, <em>label=None</em>, <em>namespace=None</em>, <em>key=None</em>, <em>shortValue=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/display.html#DisplayDataItem"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.display.DisplayDataItem" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <p>A DisplayDataItem represents a unit of static display data.</p> |
| <p>Each item is identified by a key and the namespace of the component the |
| display item belongs to.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.display.DisplayDataItem.drop_if_default"> |
| <code class="descname">drop_if_default</code><span class="sig-paren">(</span><em>default</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/display.html#DisplayDataItem.drop_if_default"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.display.DisplayDataItem.drop_if_default" title="Permalink to this definition">¶</a></dt> |
| <dd><p>The item should be dropped if its value is equal to its default.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">Returns self.</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.display.DisplayDataItem.drop_if_none"> |
| <code class="descname">drop_if_none</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/display.html#DisplayDataItem.drop_if_none"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.display.DisplayDataItem.drop_if_none" title="Permalink to this definition">¶</a></dt> |
| <dd><p>The item should be dropped if its value is None.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">Returns self.</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.display.DisplayDataItem.get_dict"> |
| <code class="descname">get_dict</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/display.html#DisplayDataItem.get_dict"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.display.DisplayDataItem.get_dict" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Returns the internal-API dictionary representing the DisplayDataItem.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">A dictionary. The internal-API dictionary representing the |
| DisplayDataItem</td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Raises:</th><td class="field-body"><code class="xref py py-exc docutils literal"><span class="pre">ValueError</span></code> – if the item is not valid.</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.display.DisplayDataItem.is_valid"> |
| <code class="descname">is_valid</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/display.html#DisplayDataItem.is_valid"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.display.DisplayDataItem.is_valid" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Checks that all the necessary fields of the DisplayDataItem are |
| filled in. It checks that neither key, namespace, value or type are None.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Raises:</th><td class="field-body"><code class="xref py py-exc docutils literal"><span class="pre">ValueError</span></code> – If the item does not have a key, namespace, value or type.</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.display.DisplayDataItem.should_drop"> |
| <code class="descname">should_drop</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/display.html#DisplayDataItem.should_drop"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.display.DisplayDataItem.should_drop" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Return True if the item should be dropped, or False if it should not |
| be dropped. This depends on the drop_if_none, and drop_if_default calls.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">True or False; depending on whether the item should be dropped or kept.</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.display.DisplayDataItem.typeDict"> |
| <code class="descname">typeDict</code><em class="property"> = {<type 'datetime.datetime'>: 'TIMESTAMP', <type 'bool'>: 'BOOLEAN', <type 'str'>: 'STRING', <type 'datetime.timedelta'>: 'DURATION', <type 'unicode'>: 'STRING', <type 'int'>: 'INTEGER', <type 'float'>: 'FLOAT'}</em><a class="headerlink" href="#apache_beam.transforms.display.DisplayDataItem.typeDict" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.display.DisplayData"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.display.</code><code class="descname">DisplayData</code><span class="sig-paren">(</span><em>namespace</em>, <em>display_data_dict</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/display.html#DisplayData"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.display.DisplayData" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <p>Static display data associated with a pipeline component.</p> |
| <dl class="classmethod"> |
| <dt id="apache_beam.transforms.display.DisplayData.create_from"> |
| <em class="property">classmethod </em><code class="descname">create_from</code><span class="sig-paren">(</span><em>has_display_data</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/display.html#DisplayData.create_from"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.display.DisplayData.create_from" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Creates DisplayData from a HasDisplayData instance.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">A DisplayData instance with populated items.</td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Raises:</th><td class="field-body"><code class="xref py py-exc docutils literal"><span class="pre">ValueError</span></code> – If the has_display_data argument is not an instance of |
| HasDisplayData.</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="classmethod"> |
| <dt id="apache_beam.transforms.display.DisplayData.create_from_options"> |
| <em class="property">classmethod </em><code class="descname">create_from_options</code><span class="sig-paren">(</span><em>pipeline_options</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/display.html#DisplayData.create_from_options"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.display.DisplayData.create_from_options" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Creates DisplayData from a PipelineOptions instance.</p> |
| <p>When creating DisplayData, this method will convert the value of any |
| item of a non-supported type to its string representation. |
| The normal DisplayData.create_from method rejects those items.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">A DisplayData instance with populated items.</td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Raises:</th><td class="field-body"><code class="xref py py-exc docutils literal"><span class="pre">ValueError</span></code> – If the has_display_data argument is not an instance of |
| HasDisplayData.</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| </dd></dl> |
| |
| </div> |
| <div class="section" id="module-apache_beam.transforms.ptransform"> |
| <span id="apache-beam-transforms-ptransform-module"></span><h2>apache_beam.transforms.ptransform module<a class="headerlink" href="#module-apache_beam.transforms.ptransform" title="Permalink to this headline">¶</a></h2> |
| <p>PTransform and descendants.</p> |
| <p>A PTransform is an object describing (not executing) a computation. The actual |
| execution semantics for a transform is captured by a runner object. A transform |
| object always belongs to a pipeline object.</p> |
| <p>A PTransform derived class needs to define the expand() method that describes |
| how one or more PValues are created by the transform.</p> |
| <p>The module defines a few standard transforms: FlatMap (parallel do), |
| GroupByKey (group by key), etc. Note that the expand() methods for these |
| classes contain code that will add nodes to the processing graph associated |
| with a pipeline.</p> |
| <p>As support for the FlatMap transform, the module also defines a DoFn |
| class and wrapper class that allows lambda functions to be used as |
| FlatMap processing functions.</p> |
| <dl class="class"> |
| <dt id="apache_beam.transforms.ptransform.PTransform"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.ptransform.</code><code class="descname">PTransform</code><span class="sig-paren">(</span><em>label=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/ptransform.html#PTransform"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="apache_beam.typehints.html#apache_beam.typehints.decorators.WithTypeHints" title="apache_beam.typehints.decorators.WithTypeHints"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.typehints.decorators.WithTypeHints</span></code></a>, <a class="reference internal" href="#apache_beam.transforms.display.HasDisplayData" title="apache_beam.transforms.display.HasDisplayData"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.display.HasDisplayData</span></code></a></p> |
| <p>A transform object used to modify one or more PCollections.</p> |
| <p>Subclasses must define an expand() method that will be used when the transform |
| is applied to some arguments. Typical usage pattern will be:</p> |
| <blockquote> |
| <div>input | CustomTransform(...)</div></blockquote> |
| <p>The expand() method of the CustomTransform object passed in will be called |
| with input as an argument.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.ptransform.PTransform.default_label"> |
| <code class="descname">default_label</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/ptransform.html#PTransform.default_label"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform.default_label" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.ptransform.PTransform.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>input_or_inputs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/ptransform.html#PTransform.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform.expand" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="classmethod"> |
| <dt id="apache_beam.transforms.ptransform.PTransform.from_runner_api"> |
| <em class="property">classmethod </em><code class="descname">from_runner_api</code><span class="sig-paren">(</span><em>proto</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/ptransform.html#PTransform.from_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform.from_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.ptransform.PTransform.from_runner_api_parameter"> |
| <em class="property">static </em><code class="descname">from_runner_api_parameter</code><span class="sig-paren">(</span><em>spec_parameter</em>, <em>unused_context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/ptransform.html#PTransform.from_runner_api_parameter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform.from_runner_api_parameter" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.ptransform.PTransform.get_windowing"> |
| <code class="descname">get_windowing</code><span class="sig-paren">(</span><em>inputs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/ptransform.html#PTransform.get_windowing"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform.get_windowing" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Returns the window function to be associated with transform’s output.</p> |
| <p>By default most transforms just return the windowing function associated |
| with the input PCollection (or the first input if several).</p> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.ptransform.PTransform.infer_output_type"> |
| <code class="descname">infer_output_type</code><span class="sig-paren">(</span><em>unused_input_type</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/ptransform.html#PTransform.infer_output_type"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform.infer_output_type" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.ptransform.PTransform.label"> |
| <code class="descname">label</code><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform.label" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.ptransform.PTransform.pipeline"> |
| <code class="descname">pipeline</code><em class="property"> = None</em><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform.pipeline" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="classmethod"> |
| <dt id="apache_beam.transforms.ptransform.PTransform.register_urn"> |
| <em class="property">classmethod </em><code class="descname">register_urn</code><span class="sig-paren">(</span><em>urn</em>, <em>parameter_type</em>, <em>constructor</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/ptransform.html#PTransform.register_urn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform.register_urn" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.ptransform.PTransform.side_inputs"> |
| <code class="descname">side_inputs</code><em class="property"> = ()</em><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform.side_inputs" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.ptransform.PTransform.to_runner_api"> |
| <code class="descname">to_runner_api</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/ptransform.html#PTransform.to_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform.to_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.ptransform.PTransform.to_runner_api_parameter"> |
| <code class="descname">to_runner_api_parameter</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/ptransform.html#PTransform.to_runner_api_parameter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform.to_runner_api_parameter" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.ptransform.PTransform.type_check_inputs"> |
| <code class="descname">type_check_inputs</code><span class="sig-paren">(</span><em>pvalueish</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/ptransform.html#PTransform.type_check_inputs"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform.type_check_inputs" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.ptransform.PTransform.type_check_inputs_or_outputs"> |
| <code class="descname">type_check_inputs_or_outputs</code><span class="sig-paren">(</span><em>pvalueish</em>, <em>input_or_output</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/ptransform.html#PTransform.type_check_inputs_or_outputs"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform.type_check_inputs_or_outputs" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.ptransform.PTransform.type_check_outputs"> |
| <code class="descname">type_check_outputs</code><span class="sig-paren">(</span><em>pvalueish</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/ptransform.html#PTransform.type_check_outputs"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform.type_check_outputs" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.ptransform.PTransform.with_input_types"> |
| <code class="descname">with_input_types</code><span class="sig-paren">(</span><em>input_type_hint</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/ptransform.html#PTransform.with_input_types"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform.with_input_types" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Annotates the input type of a PTransform with a type-hint.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>input_type_hint</strong> – An instance of an allowed built-in type, a custom class, |
| or an instance of a typehints.TypeConstraint.</td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Raises:</th><td class="field-body"><code class="xref py py-exc docutils literal"><span class="pre">TypeError</span></code> – If ‘type_hint’ is not a valid type-hint. See |
| typehints.validate_composite_type_param for further details.</td> |
| </tr> |
| <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">A reference to the instance of this particular PTransform object. This |
| allows chaining type-hinting related methods.</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.ptransform.PTransform.with_output_types"> |
| <code class="descname">with_output_types</code><span class="sig-paren">(</span><em>type_hint</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/ptransform.html#PTransform.with_output_types"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.ptransform.PTransform.with_output_types" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Annotates the output type of a PTransform with a type-hint.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>type_hint</strong> – An instance of an allowed built-in type, a custom class, or a |
| typehints.TypeConstraint.</td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Raises:</th><td class="field-body"><code class="xref py py-exc docutils literal"><span class="pre">TypeError</span></code> – If ‘type_hint’ is not a valid type-hint. See |
| typehints.validate_composite_type_param for further details.</td> |
| </tr> |
| <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">A reference to the instance of this particular PTransform object. This |
| allows chaining type-hinting related methods.</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="function"> |
| <dt id="apache_beam.transforms.ptransform.ptransform_fn"> |
| <code class="descclassname">apache_beam.transforms.ptransform.</code><code class="descname">ptransform_fn</code><span class="sig-paren">(</span><em>fn</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/ptransform.html#ptransform_fn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.ptransform.ptransform_fn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>A decorator for a function-based PTransform.</p> |
| <p>Experimental; no backwards-compatibility guarantees.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>fn</strong> – A function implementing a custom PTransform.</td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body">A CallablePTransform instance wrapping the function-based PTransform.</td> |
| </tr> |
| </tbody> |
| </table> |
| <p>This wrapper provides an alternative, simpler way to define a PTransform. |
| The standard method is to subclass from PTransform and override the expand() |
| method. An equivalent effect can be obtained by defining a function that |
| an input PCollection and additional optional arguments and returns a |
| resulting PCollection. For example:</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="nd">@ptransform_fn</span> |
| <span class="k">def</span> <span class="nf">CustomMapper</span><span class="p">(</span><span class="n">pcoll</span><span class="p">,</span> <span class="n">mapfn</span><span class="p">):</span> |
| <span class="k">return</span> <span class="n">pcoll</span> <span class="o">|</span> <span class="n">ParDo</span><span class="p">(</span><span class="n">mapfn</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <p>The equivalent approach using PTransform subclassing:</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">CustomMapper</span><span class="p">(</span><span class="n">PTransform</span><span class="p">):</span> |
| |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">mapfn</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">CustomMapper</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">mapfn</span> <span class="o">=</span> <span class="n">mapfn</span> |
| |
| <span class="k">def</span> <span class="nf">expand</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pcoll</span><span class="p">):</span> |
| <span class="k">return</span> <span class="n">pcoll</span> <span class="o">|</span> <span class="n">ParDo</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">mapfn</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <p>With either method the custom PTransform can be used in pipelines as if |
| it were one of the “native” PTransforms:</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">result_pcoll</span> <span class="o">=</span> <span class="n">input_pcoll</span> <span class="o">|</span> <span class="s1">'Label'</span> <span class="o">>></span> <span class="n">CustomMapper</span><span class="p">(</span><span class="n">somefn</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <p>Note that for both solutions the underlying implementation of the pipe |
| operator (i.e., <cite>|</cite>) will inject the pcoll argument in its proper place |
| (first argument if no label was specified and second argument otherwise).</p> |
| </dd></dl> |
| |
| <dl class="function"> |
| <dt id="apache_beam.transforms.ptransform.label_from_callable"> |
| <code class="descclassname">apache_beam.transforms.ptransform.</code><code class="descname">label_from_callable</code><span class="sig-paren">(</span><em>fn</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/ptransform.html#label_from_callable"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.ptransform.label_from_callable" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </div> |
| <div class="section" id="module-apache_beam.transforms.sideinputs"> |
| <span id="apache-beam-transforms-sideinputs-module"></span><h2>apache_beam.transforms.sideinputs module<a class="headerlink" href="#module-apache_beam.transforms.sideinputs" title="Permalink to this headline">¶</a></h2> |
| <p>Internal side input transforms and implementations.</p> |
| <p>For internal use only; no backwards-compatibility guarantees.</p> |
| <p>Important: this module is an implementation detail and should not be used |
| directly by pipeline writers. Instead, users should use the helper methods |
| AsSingleton, AsIter, AsList and AsDict in apache_beam.pvalue.</p> |
| <dl class="class"> |
| <dt id="apache_beam.transforms.sideinputs.SideInputMap"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.sideinputs.</code><code class="descname">SideInputMap</code><span class="sig-paren">(</span><em>view_class</em>, <em>view_options</em>, <em>iterable</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/sideinputs.html#SideInputMap"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.sideinputs.SideInputMap" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <p>Represents a mapping of windows to side input values.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.sideinputs.SideInputMap.is_globally_windowed"> |
| <code class="descname">is_globally_windowed</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/sideinputs.html#SideInputMap.is_globally_windowed"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.sideinputs.SideInputMap.is_globally_windowed" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="function"> |
| <dt id="apache_beam.transforms.sideinputs.default_window_mapping_fn"> |
| <code class="descclassname">apache_beam.transforms.sideinputs.</code><code class="descname">default_window_mapping_fn</code><span class="sig-paren">(</span><em>target_window_fn</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/sideinputs.html#default_window_mapping_fn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.sideinputs.default_window_mapping_fn" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </div> |
| <div class="section" id="module-apache_beam.transforms.timeutil"> |
| <span id="apache-beam-transforms-timeutil-module"></span><h2>apache_beam.transforms.timeutil module<a class="headerlink" href="#module-apache_beam.transforms.timeutil" title="Permalink to this headline">¶</a></h2> |
| <p>Timestamp utilities.</p> |
| <dl class="class"> |
| <dt id="apache_beam.transforms.timeutil.TimeDomain"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.timeutil.</code><code class="descname">TimeDomain</code><a class="reference internal" href="_modules/apache_beam/transforms/timeutil.html#TimeDomain"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.timeutil.TimeDomain" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <p>Time domain for streaming timers.</p> |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.timeutil.TimeDomain.DEPENDENT_REAL_TIME"> |
| <code class="descname">DEPENDENT_REAL_TIME</code><em class="property"> = 'DEPENDENT_REAL_TIME'</em><a class="headerlink" href="#apache_beam.transforms.timeutil.TimeDomain.DEPENDENT_REAL_TIME" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.timeutil.TimeDomain.REAL_TIME"> |
| <code class="descname">REAL_TIME</code><em class="property"> = 'REAL_TIME'</em><a class="headerlink" href="#apache_beam.transforms.timeutil.TimeDomain.REAL_TIME" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.timeutil.TimeDomain.WATERMARK"> |
| <code class="descname">WATERMARK</code><em class="property"> = 'WATERMARK'</em><a class="headerlink" href="#apache_beam.transforms.timeutil.TimeDomain.WATERMARK" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.timeutil.TimeDomain.from_string"> |
| <em class="property">static </em><code class="descname">from_string</code><span class="sig-paren">(</span><em>domain</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/timeutil.html#TimeDomain.from_string"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.timeutil.TimeDomain.from_string" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| </div> |
| <div class="section" id="module-apache_beam.transforms.trigger"> |
| <span id="apache-beam-transforms-trigger-module"></span><h2>apache_beam.transforms.trigger module<a class="headerlink" href="#module-apache_beam.transforms.trigger" title="Permalink to this headline">¶</a></h2> |
| <p>Support for Dataflow triggers.</p> |
| <p>Triggers control when in processing time windows get emitted.</p> |
| <dl class="class"> |
| <dt id="apache_beam.transforms.trigger.AccumulationMode"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.trigger.</code><code class="descname">AccumulationMode</code><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AccumulationMode"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AccumulationMode" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <p>Controls what to do with data when a trigger fires multiple times.</p> |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.trigger.AccumulationMode.ACCUMULATING"> |
| <code class="descname">ACCUMULATING</code><em class="property"> = 1</em><a class="headerlink" href="#apache_beam.transforms.trigger.AccumulationMode.ACCUMULATING" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.trigger.AccumulationMode.DISCARDING"> |
| <code class="descname">DISCARDING</code><em class="property"> = 0</em><a class="headerlink" href="#apache_beam.transforms.trigger.AccumulationMode.DISCARDING" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.trigger.TriggerFn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.trigger.</code><code class="descname">TriggerFn</code><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#TriggerFn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.TriggerFn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <p>A TriggerFn determines when window (panes) are emitted.</p> |
| <p>See <a class="reference external" href="https://beam.apache.org/documentation/programming-guide/#triggers">https://beam.apache.org/documentation/programming-guide/#triggers</a></p> |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.trigger.TriggerFn.from_runner_api"> |
| <em class="property">static </em><code class="descname">from_runner_api</code><span class="sig-paren">(</span><em>proto</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#TriggerFn.from_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.TriggerFn.from_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.TriggerFn.on_element"> |
| <code class="descname">on_element</code><span class="sig-paren">(</span><em>element</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#TriggerFn.on_element"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.TriggerFn.on_element" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Called when a new element arrives in a window.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>element</strong> – the element being added</li> |
| <li><strong>window</strong> – the window to which the element is being added</li> |
| <li><strong>context</strong> – a context (e.g. a TriggerContext instance) for managing state |
| and setting timers</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.TriggerFn.on_fire"> |
| <code class="descname">on_fire</code><span class="sig-paren">(</span><em>watermark</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#TriggerFn.on_fire"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.TriggerFn.on_fire" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Called when a trigger actually fires.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>watermark</strong> – (a lower bound on) the watermark of the system</li> |
| <li><strong>window</strong> – the window whose trigger is being fired</li> |
| <li><strong>context</strong> – a context (e.g. a TriggerContext instance) for managing state |
| and setting timers</li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">whether this trigger is finished</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.TriggerFn.on_merge"> |
| <code class="descname">on_merge</code><span class="sig-paren">(</span><em>to_be_merged</em>, <em>merge_result</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#TriggerFn.on_merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.TriggerFn.on_merge" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Called when multiple windows are merged.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>to_be_merged</strong> – the set of windows to be merged</li> |
| <li><strong>merge_result</strong> – the window into which the windows are being merged</li> |
| <li><strong>context</strong> – a context (e.g. a TriggerContext instance) for managing state |
| and setting timers</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.TriggerFn.reset"> |
| <code class="descname">reset</code><span class="sig-paren">(</span><em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#TriggerFn.reset"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.TriggerFn.reset" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Clear any state and timers used by this TriggerFn.</p> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.TriggerFn.should_fire"> |
| <code class="descname">should_fire</code><span class="sig-paren">(</span><em>watermark</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#TriggerFn.should_fire"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.TriggerFn.should_fire" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Whether this trigger should cause the window to fire.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>watermark</strong> – (a lower bound on) the watermark of the system</li> |
| <li><strong>window</strong> – the window whose trigger is being considered</li> |
| <li><strong>context</strong> – a context (e.g. a TriggerContext instance) for managing state |
| and setting timers</li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">whether this trigger should cause a firing</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.TriggerFn.to_runner_api"> |
| <code class="descname">to_runner_api</code><span class="sig-paren">(</span><em>unused_context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#TriggerFn.to_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.TriggerFn.to_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.trigger.DefaultTrigger"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.trigger.</code><code class="descname">DefaultTrigger</code><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#DefaultTrigger"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.DefaultTrigger" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.trigger.TriggerFn" title="apache_beam.transforms.trigger.TriggerFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.trigger.TriggerFn</span></code></a></p> |
| <p>Semantically Repeatedly(AfterWatermark()), but more optimized.</p> |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.trigger.DefaultTrigger.from_runner_api"> |
| <em class="property">static </em><code class="descname">from_runner_api</code><span class="sig-paren">(</span><em>proto</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#DefaultTrigger.from_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.DefaultTrigger.from_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.DefaultTrigger.on_element"> |
| <code class="descname">on_element</code><span class="sig-paren">(</span><em>element</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#DefaultTrigger.on_element"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.DefaultTrigger.on_element" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.DefaultTrigger.on_fire"> |
| <code class="descname">on_fire</code><span class="sig-paren">(</span><em>watermark</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#DefaultTrigger.on_fire"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.DefaultTrigger.on_fire" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.DefaultTrigger.on_merge"> |
| <code class="descname">on_merge</code><span class="sig-paren">(</span><em>to_be_merged</em>, <em>merge_result</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#DefaultTrigger.on_merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.DefaultTrigger.on_merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.DefaultTrigger.reset"> |
| <code class="descname">reset</code><span class="sig-paren">(</span><em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#DefaultTrigger.reset"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.DefaultTrigger.reset" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.DefaultTrigger.should_fire"> |
| <code class="descname">should_fire</code><span class="sig-paren">(</span><em>watermark</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#DefaultTrigger.should_fire"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.DefaultTrigger.should_fire" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.DefaultTrigger.to_runner_api"> |
| <code class="descname">to_runner_api</code><span class="sig-paren">(</span><em>unused_context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#DefaultTrigger.to_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.DefaultTrigger.to_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.trigger.AfterWatermark"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.trigger.</code><code class="descname">AfterWatermark</code><span class="sig-paren">(</span><em>early=None</em>, <em>late=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterWatermark"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterWatermark" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.trigger.TriggerFn" title="apache_beam.transforms.trigger.TriggerFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.trigger.TriggerFn</span></code></a></p> |
| <p>Fire exactly once when the watermark passes the end of the window.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>early</strong> – if not None, a speculative trigger to repeatedly evaluate before |
| the watermark passes the end of the window</li> |
| <li><strong>late</strong> – if not None, a speculative trigger to repeatedly evaluate after |
| the watermark passes the end of the window</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.trigger.AfterWatermark.LATE_TAG"> |
| <code class="descname">LATE_TAG</code><em class="property"> = CombiningValueStateTag(is_late, CallableWrapperCombineFn(<built-in function any>))</em><a class="headerlink" href="#apache_beam.transforms.trigger.AfterWatermark.LATE_TAG" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.trigger.AfterWatermark.from_runner_api"> |
| <em class="property">static </em><code class="descname">from_runner_api</code><span class="sig-paren">(</span><em>proto</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterWatermark.from_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterWatermark.from_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterWatermark.is_late"> |
| <code class="descname">is_late</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterWatermark.is_late"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterWatermark.is_late" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterWatermark.on_element"> |
| <code class="descname">on_element</code><span class="sig-paren">(</span><em>element</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterWatermark.on_element"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterWatermark.on_element" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterWatermark.on_fire"> |
| <code class="descname">on_fire</code><span class="sig-paren">(</span><em>watermark</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterWatermark.on_fire"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterWatermark.on_fire" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterWatermark.on_merge"> |
| <code class="descname">on_merge</code><span class="sig-paren">(</span><em>to_be_merged</em>, <em>merge_result</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterWatermark.on_merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterWatermark.on_merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterWatermark.reset"> |
| <code class="descname">reset</code><span class="sig-paren">(</span><em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterWatermark.reset"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterWatermark.reset" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterWatermark.should_fire"> |
| <code class="descname">should_fire</code><span class="sig-paren">(</span><em>watermark</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterWatermark.should_fire"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterWatermark.should_fire" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterWatermark.to_runner_api"> |
| <code class="descname">to_runner_api</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterWatermark.to_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterWatermark.to_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.trigger.AfterCount"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.trigger.</code><code class="descname">AfterCount</code><span class="sig-paren">(</span><em>count</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterCount"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterCount" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.trigger.TriggerFn" title="apache_beam.transforms.trigger.TriggerFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.trigger.TriggerFn</span></code></a></p> |
| <p>Fire when there are at least count elements in this window pane.</p> |
| <p>AfterCount is experimental. No backwards compatibility guarantees.</p> |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.trigger.AfterCount.COUNT_TAG"> |
| <code class="descname">COUNT_TAG</code><em class="property"> = CombiningValueStateTag(count, <apache_beam.transforms.combiners.CountCombineFn object>)</em><a class="headerlink" href="#apache_beam.transforms.trigger.AfterCount.COUNT_TAG" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.trigger.AfterCount.from_runner_api"> |
| <em class="property">static </em><code class="descname">from_runner_api</code><span class="sig-paren">(</span><em>proto</em>, <em>unused_context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterCount.from_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterCount.from_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterCount.on_element"> |
| <code class="descname">on_element</code><span class="sig-paren">(</span><em>element</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterCount.on_element"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterCount.on_element" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterCount.on_fire"> |
| <code class="descname">on_fire</code><span class="sig-paren">(</span><em>watermark</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterCount.on_fire"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterCount.on_fire" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterCount.on_merge"> |
| <code class="descname">on_merge</code><span class="sig-paren">(</span><em>to_be_merged</em>, <em>merge_result</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterCount.on_merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterCount.on_merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterCount.reset"> |
| <code class="descname">reset</code><span class="sig-paren">(</span><em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterCount.reset"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterCount.reset" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterCount.should_fire"> |
| <code class="descname">should_fire</code><span class="sig-paren">(</span><em>watermark</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterCount.should_fire"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterCount.should_fire" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterCount.to_runner_api"> |
| <code class="descname">to_runner_api</code><span class="sig-paren">(</span><em>unused_context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterCount.to_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterCount.to_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.trigger.Repeatedly"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.trigger.</code><code class="descname">Repeatedly</code><span class="sig-paren">(</span><em>underlying</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#Repeatedly"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.Repeatedly" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.trigger.TriggerFn" title="apache_beam.transforms.trigger.TriggerFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.trigger.TriggerFn</span></code></a></p> |
| <p>Repeatedly invoke the given trigger, never finishing.</p> |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.trigger.Repeatedly.from_runner_api"> |
| <em class="property">static </em><code class="descname">from_runner_api</code><span class="sig-paren">(</span><em>proto</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#Repeatedly.from_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.Repeatedly.from_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.Repeatedly.on_element"> |
| <code class="descname">on_element</code><span class="sig-paren">(</span><em>element</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#Repeatedly.on_element"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.Repeatedly.on_element" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.Repeatedly.on_fire"> |
| <code class="descname">on_fire</code><span class="sig-paren">(</span><em>watermark</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#Repeatedly.on_fire"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.Repeatedly.on_fire" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.Repeatedly.on_merge"> |
| <code class="descname">on_merge</code><span class="sig-paren">(</span><em>to_be_merged</em>, <em>merge_result</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#Repeatedly.on_merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.Repeatedly.on_merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.Repeatedly.reset"> |
| <code class="descname">reset</code><span class="sig-paren">(</span><em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#Repeatedly.reset"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.Repeatedly.reset" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.Repeatedly.should_fire"> |
| <code class="descname">should_fire</code><span class="sig-paren">(</span><em>watermark</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#Repeatedly.should_fire"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.Repeatedly.should_fire" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.Repeatedly.to_runner_api"> |
| <code class="descname">to_runner_api</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#Repeatedly.to_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.Repeatedly.to_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.trigger.AfterAny"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.trigger.</code><code class="descname">AfterAny</code><span class="sig-paren">(</span><em>*triggers</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterAny"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterAny" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.trigger._ParallelTriggerFn</span></code></p> |
| <p>Fires when any subtrigger fires.</p> |
| <p>Also finishes when any subtrigger finishes.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterAny.combine_op"> |
| <code class="descname">combine_op</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#apache_beam.transforms.trigger.AfterAny.combine_op" title="Permalink to this definition">¶</a></dt> |
| <dd><p>any(iterable) -> bool</p> |
| <p>Return True if bool(x) is True for any x in the iterable. |
| If the iterable is empty, return False.</p> |
| </dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.trigger.AfterAll"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.trigger.</code><code class="descname">AfterAll</code><span class="sig-paren">(</span><em>*triggers</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterAll"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterAll" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.trigger._ParallelTriggerFn</span></code></p> |
| <p>Fires when all subtriggers have fired.</p> |
| <p>Also finishes when all subtriggers have finished.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterAll.combine_op"> |
| <code class="descname">combine_op</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#apache_beam.transforms.trigger.AfterAll.combine_op" title="Permalink to this definition">¶</a></dt> |
| <dd><p>all(iterable) -> bool</p> |
| <p>Return True if bool(x) is True for all values x in the iterable. |
| If the iterable is empty, return True.</p> |
| </dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.trigger.AfterEach"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.trigger.</code><code class="descname">AfterEach</code><span class="sig-paren">(</span><em>*triggers</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterEach"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterEach" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.trigger.TriggerFn" title="apache_beam.transforms.trigger.TriggerFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.trigger.TriggerFn</span></code></a></p> |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.trigger.AfterEach.INDEX_TAG"> |
| <code class="descname">INDEX_TAG</code><em class="property"> = CombiningValueStateTag(index, CallableWrapperCombineFn(<function <lambda>>))</em><a class="headerlink" href="#apache_beam.transforms.trigger.AfterEach.INDEX_TAG" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.trigger.AfterEach.from_runner_api"> |
| <em class="property">static </em><code class="descname">from_runner_api</code><span class="sig-paren">(</span><em>proto</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterEach.from_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterEach.from_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterEach.on_element"> |
| <code class="descname">on_element</code><span class="sig-paren">(</span><em>element</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterEach.on_element"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterEach.on_element" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterEach.on_fire"> |
| <code class="descname">on_fire</code><span class="sig-paren">(</span><em>watermark</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterEach.on_fire"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterEach.on_fire" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterEach.on_merge"> |
| <code class="descname">on_merge</code><span class="sig-paren">(</span><em>to_be_merged</em>, <em>merge_result</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterEach.on_merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterEach.on_merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterEach.reset"> |
| <code class="descname">reset</code><span class="sig-paren">(</span><em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterEach.reset"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterEach.reset" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterEach.should_fire"> |
| <code class="descname">should_fire</code><span class="sig-paren">(</span><em>watermark</em>, <em>window</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterEach.should_fire"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterEach.should_fire" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.AfterEach.to_runner_api"> |
| <code class="descname">to_runner_api</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#AfterEach.to_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.AfterEach.to_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.trigger.OrFinally"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.trigger.</code><code class="descname">OrFinally</code><span class="sig-paren">(</span><em>*triggers</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#OrFinally"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.OrFinally" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.trigger.AfterAny" title="apache_beam.transforms.trigger.AfterAny"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.trigger.AfterAny</span></code></a></p> |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.trigger.OrFinally.from_runner_api"> |
| <em class="property">static </em><code class="descname">from_runner_api</code><span class="sig-paren">(</span><em>proto</em>, <em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#OrFinally.from_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.OrFinally.from_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.trigger.OrFinally.to_runner_api"> |
| <code class="descname">to_runner_api</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/trigger.html#OrFinally.to_runner_api"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.trigger.OrFinally.to_runner_api" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| </div> |
| <div class="section" id="module-apache_beam.transforms.util"> |
| <span id="apache-beam-transforms-util-module"></span><h2>apache_beam.transforms.util module<a class="headerlink" href="#module-apache_beam.transforms.util" title="Permalink to this headline">¶</a></h2> |
| <p>Simple utility PTransforms.</p> |
| <dl class="class"> |
| <dt id="apache_beam.transforms.util.CoGroupByKey"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.util.</code><code class="descname">CoGroupByKey</code><span class="sig-paren">(</span><em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/util.html#CoGroupByKey"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.util.CoGroupByKey" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.ptransform.PTransform</span></code></a></p> |
| <p>Groups results across several PCollections by key.</p> |
| <p>Given an input dict mapping serializable keys (called “tags”) to 0 or more |
| PCollections of (key, value) tuples, e.g.:</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="p">{</span><span class="s1">'pc1'</span><span class="p">:</span> <span class="n">pcoll1</span><span class="p">,</span> <span class="s1">'pc2'</span><span class="p">:</span> <span class="n">pcoll2</span><span class="p">,</span> <span class="mi">33333</span><span class="p">:</span> <span class="n">pcoll3</span><span class="p">}</span> |
| </pre></div> |
| </div> |
| <p>creates a single output PCollection of (key, value) tuples whose keys are the |
| unique input keys from all inputs, and whose values are dicts mapping each |
| tag to an iterable of whatever values were under the key in the corresponding |
| PCollection:</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="p">(</span><span class="s1">'some key'</span><span class="p">,</span> <span class="p">{</span><span class="s1">'pc1'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'value 1 under "some key" in pcoll1'</span><span class="p">,</span> |
| <span class="s1">'value 2 under "some key" in pcoll1'</span><span class="p">],</span> |
| <span class="s1">'pc2'</span><span class="p">:</span> <span class="p">[],</span> |
| <span class="mi">33333</span><span class="p">:</span> <span class="p">[</span><span class="s1">'only value under "some key" in pcoll3'</span><span class="p">]})</span> |
| </pre></div> |
| </div> |
| <p>Note that pcoll2 had no values associated with “some key”.</p> |
| <p>CoGroupByKey also works for tuples, lists, or other flat iterables of |
| PCollections, in which case the values of the resulting PCollections |
| will be tuples whose nth value is the list of values from the nth |
| PCollection—conceptually, the “tags” are the indices into the input. |
| Thus, for this input:</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="p">(</span><span class="n">pcoll1</span><span class="p">,</span> <span class="n">pcoll2</span><span class="p">,</span> <span class="n">pcoll3</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <p>the output PCollection’s value for “some key” is:</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="p">(</span><span class="s1">'some key'</span><span class="p">,</span> <span class="p">([</span><span class="s1">'value 1 under "some key" in pcoll1'</span><span class="p">,</span> |
| <span class="s1">'value 2 under "some key" in pcoll1'</span><span class="p">],</span> |
| <span class="p">[],</span> |
| <span class="p">[</span><span class="s1">'only value under "some key" in pcoll3'</span><span class="p">]))</span> |
| </pre></div> |
| </div> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>label</strong> – name of this transform instance. Useful while monitoring and |
| debugging a pipeline execution.</li> |
| <li><strong>**kwargs</strong> – Accepts a single named argument “pipeline”, which specifies the |
| pipeline that “owns” this PTransform. Ordinarily CoGroupByKey can obtain |
| this information from one of the input PCollections, but if there are none |
| (or if there’s a chance there may be none), this argument is the only way |
| to provide pipeline information, and should be considered mandatory.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.util.CoGroupByKey.expand"> |
| <code class="descname">expand</code><span class="sig-paren">(</span><em>pcolls</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/util.html#CoGroupByKey.expand"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.util.CoGroupByKey.expand" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Performs CoGroupByKey on argument pcolls; see class docstring.</p> |
| </dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="function"> |
| <dt id="apache_beam.transforms.util.Keys"> |
| <code class="descclassname">apache_beam.transforms.util.</code><code class="descname">Keys</code><span class="sig-paren">(</span><em>label='Keys'</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/util.html#Keys"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.util.Keys" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Produces a PCollection of first elements of 2-tuples in a PCollection.</p> |
| </dd></dl> |
| |
| <dl class="function"> |
| <dt id="apache_beam.transforms.util.KvSwap"> |
| <code class="descclassname">apache_beam.transforms.util.</code><code class="descname">KvSwap</code><span class="sig-paren">(</span><em>label='KvSwap'</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/util.html#KvSwap"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.util.KvSwap" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Produces a PCollection reversing 2-tuples in a PCollection.</p> |
| </dd></dl> |
| |
| <dl class="function"> |
| <dt id="apache_beam.transforms.util.Values"> |
| <code class="descclassname">apache_beam.transforms.util.</code><code class="descname">Values</code><span class="sig-paren">(</span><em>label='Values'</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/util.html#Values"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.util.Values" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Produces a PCollection of second elements of 2-tuples in a PCollection.</p> |
| </dd></dl> |
| |
| </div> |
| <div class="section" id="module-apache_beam.transforms.window"> |
| <span id="apache-beam-transforms-window-module"></span><h2>apache_beam.transforms.window module<a class="headerlink" href="#module-apache_beam.transforms.window" title="Permalink to this headline">¶</a></h2> |
| <p>Windowing concepts.</p> |
| <p>A WindowInto transform logically divides up or groups the elements of a |
| PCollection into finite windows according to a windowing function (derived from |
| WindowFn).</p> |
| <p>The output of WindowInto contains the same elements as input, but they have been |
| logically assigned to windows. The next GroupByKey(s) transforms, including one |
| within a composite transform, will group by the combination of keys and windows.</p> |
| <p>Windowing a PCollection allows chunks of it to be processed individually, before |
| the entire PCollection is available. This is especially important for |
| PCollection(s) with unbounded size, since the full PCollection is never |
| available at once, since more data is continually arriving. For PCollection(s) |
| with a bounded size (aka. conventional batch mode), by default, all data is |
| implicitly in a single window (see GlobalWindows), unless WindowInto is |
| applied.</p> |
| <p>For example, a simple form of windowing divides up the data into fixed-width |
| time intervals, using FixedWindows.</p> |
| <p>Seconds are used as the time unit for the built-in windowing primitives here. |
| Integer or floating point seconds can be passed to these primitives.</p> |
| <p>Internally, seconds, with microsecond granularity, are stored as |
| timeutil.Timestamp and timeutil.Duration objects. This is done to avoid |
| precision errors that would occur with floating point representations.</p> |
| <p>Custom windowing function classes can be created, by subclassing from |
| WindowFn.</p> |
| <dl class="class"> |
| <dt id="apache_beam.transforms.window.TimestampCombiner"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.window.</code><code class="descname">TimestampCombiner</code><a class="reference internal" href="_modules/apache_beam/transforms/window.html#TimestampCombiner"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.TimestampCombiner" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <p>Determines how output timestamps of grouping operations are assigned.</p> |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.window.TimestampCombiner.OUTPUT_AT_EARLIEST"> |
| <code class="descname">OUTPUT_AT_EARLIEST</code><em class="property"> = 2</em><a class="headerlink" href="#apache_beam.transforms.window.TimestampCombiner.OUTPUT_AT_EARLIEST" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.window.TimestampCombiner.OUTPUT_AT_EARLIEST_TRANSFORMED"> |
| <code class="descname">OUTPUT_AT_EARLIEST_TRANSFORMED</code><em class="property"> = 'OUTPUT_AT_EARLIEST_TRANSFORMED'</em><a class="headerlink" href="#apache_beam.transforms.window.TimestampCombiner.OUTPUT_AT_EARLIEST_TRANSFORMED" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.window.TimestampCombiner.OUTPUT_AT_EOW"> |
| <code class="descname">OUTPUT_AT_EOW</code><em class="property"> = 0</em><a class="headerlink" href="#apache_beam.transforms.window.TimestampCombiner.OUTPUT_AT_EOW" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.window.TimestampCombiner.OUTPUT_AT_LATEST"> |
| <code class="descname">OUTPUT_AT_LATEST</code><em class="property"> = 1</em><a class="headerlink" href="#apache_beam.transforms.window.TimestampCombiner.OUTPUT_AT_LATEST" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.window.TimestampCombiner.get_impl"> |
| <em class="property">static </em><code class="descname">get_impl</code><span class="sig-paren">(</span><em>timestamp_combiner</em>, <em>window_fn</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#TimestampCombiner.get_impl"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.TimestampCombiner.get_impl" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.window.WindowFn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.window.</code><code class="descname">WindowFn</code><a class="reference internal" href="_modules/apache_beam/transforms/window.html#WindowFn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.WindowFn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="apache_beam.utils.html#apache_beam.utils.urns.RunnerApiFn" title="apache_beam.utils.urns.RunnerApiFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.utils.urns.RunnerApiFn</span></code></a></p> |
| <p>An abstract windowing function defining a basic assign and merge.</p> |
| <dl class="class"> |
| <dt id="apache_beam.transforms.window.WindowFn.AssignContext"> |
| <em class="property">class </em><code class="descname">AssignContext</code><span class="sig-paren">(</span><em>timestamp</em>, <em>element=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#WindowFn.AssignContext"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.WindowFn.AssignContext" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <p>Context passed to WindowFn.assign().</p> |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.window.WindowFn.MergeContext"> |
| <em class="property">class </em><code class="descname">MergeContext</code><span class="sig-paren">(</span><em>windows</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#WindowFn.MergeContext"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.WindowFn.MergeContext" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <p>Context passed to WindowFn.merge() to perform merging, if any.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.WindowFn.MergeContext.merge"> |
| <code class="descname">merge</code><span class="sig-paren">(</span><em>to_be_merged</em>, <em>merge_result</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#WindowFn.MergeContext.merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.WindowFn.MergeContext.merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.WindowFn.assign"> |
| <code class="descname">assign</code><span class="sig-paren">(</span><em>assign_context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#WindowFn.assign"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.WindowFn.assign" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Associates a timestamp to an element.</p> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.WindowFn.get_transformed_output_time"> |
| <code class="descname">get_transformed_output_time</code><span class="sig-paren">(</span><em>window</em>, <em>input_timestamp</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#WindowFn.get_transformed_output_time"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.WindowFn.get_transformed_output_time" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Given input time and output window, returns output time for window.</p> |
| <p>If TimestampCombiner.OUTPUT_AT_EARLIEST_TRANSFORMED is used in the |
| Windowing, the output timestamp for the given window will be the earliest |
| of the timestamps returned by get_transformed_output_time() for elements |
| of the window.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name" /> |
| <col class="field-body" /> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>window</strong> – Output window of element.</li> |
| <li><strong>input_timestamp</strong> – Input timestamp of element as a timeutil.Timestamp |
| object.</li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">Transformed timestamp.</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.WindowFn.get_window_coder"> |
| <code class="descname">get_window_coder</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#WindowFn.get_window_coder"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.WindowFn.get_window_coder" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.WindowFn.is_merging"> |
| <code class="descname">is_merging</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#WindowFn.is_merging"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.WindowFn.is_merging" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Returns whether this WindowFn merges windows.</p> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.WindowFn.merge"> |
| <code class="descname">merge</code><span class="sig-paren">(</span><em>merge_context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#WindowFn.merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.WindowFn.merge" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Returns a window that is the result of merging a set of windows.</p> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.WindowFn.to_runner_api_parameter"> |
| <code class="descname">to_runner_api_parameter</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="headerlink" href="#apache_beam.transforms.window.WindowFn.to_runner_api_parameter" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.window.BoundedWindow"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.window.</code><code class="descname">BoundedWindow</code><span class="sig-paren">(</span><em>end</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#BoundedWindow"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.BoundedWindow" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <p>A window for timestamps in range (-infinity, end).</p> |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.window.BoundedWindow.end"> |
| <code class="descname">end</code><a class="headerlink" href="#apache_beam.transforms.window.BoundedWindow.end" title="Permalink to this definition">¶</a></dt> |
| <dd><p>End of window.</p> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.BoundedWindow.max_timestamp"> |
| <code class="descname">max_timestamp</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#BoundedWindow.max_timestamp"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.BoundedWindow.max_timestamp" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.window.IntervalWindow"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.window.</code><code class="descname">IntervalWindow</code><span class="sig-paren">(</span><em>start</em>, <em>end</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#IntervalWindow"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.IntervalWindow" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.window.BoundedWindow" title="apache_beam.transforms.window.BoundedWindow"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.window.BoundedWindow</span></code></a></p> |
| <p>A window for timestamps in range [start, end).</p> |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.window.IntervalWindow.start"> |
| <code class="descname">start</code><a class="headerlink" href="#apache_beam.transforms.window.IntervalWindow.start" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Start of window as seconds since Unix epoch.</p> |
| </dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.window.IntervalWindow.end"> |
| <code class="descname">end</code><a class="headerlink" href="#apache_beam.transforms.window.IntervalWindow.end" title="Permalink to this definition">¶</a></dt> |
| <dd><p>End of window as seconds since Unix epoch.</p> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.IntervalWindow.intersects"> |
| <code class="descname">intersects</code><span class="sig-paren">(</span><em>other</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#IntervalWindow.intersects"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.IntervalWindow.intersects" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.IntervalWindow.union"> |
| <code class="descname">union</code><span class="sig-paren">(</span><em>other</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#IntervalWindow.union"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.IntervalWindow.union" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.window.TimestampedValue"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.window.</code><code class="descname">TimestampedValue</code><span class="sig-paren">(</span><em>value</em>, <em>timestamp</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#TimestampedValue"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.TimestampedValue" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">object</span></code></p> |
| <p>A timestamped value having a value and a timestamp.</p> |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.window.TimestampedValue.value"> |
| <code class="descname">value</code><a class="headerlink" href="#apache_beam.transforms.window.TimestampedValue.value" title="Permalink to this definition">¶</a></dt> |
| <dd><p>The underlying value.</p> |
| </dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.window.TimestampedValue.timestamp"> |
| <code class="descname">timestamp</code><a class="headerlink" href="#apache_beam.transforms.window.TimestampedValue.timestamp" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Timestamp associated with the value as seconds since Unix epoch.</p> |
| </dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.window.GlobalWindow"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.window.</code><code class="descname">GlobalWindow</code><a class="reference internal" href="_modules/apache_beam/transforms/window.html#GlobalWindow"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.GlobalWindow" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.window.BoundedWindow" title="apache_beam.transforms.window.BoundedWindow"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.window.BoundedWindow</span></code></a></p> |
| <p>The default window into which all data is placed (via GlobalWindows).</p> |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.window.NonMergingWindowFn"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.window.</code><code class="descname">NonMergingWindowFn</code><a class="reference internal" href="_modules/apache_beam/transforms/window.html#NonMergingWindowFn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.NonMergingWindowFn" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.window.WindowFn" title="apache_beam.transforms.window.WindowFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.window.WindowFn</span></code></a></p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.NonMergingWindowFn.is_merging"> |
| <code class="descname">is_merging</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#NonMergingWindowFn.is_merging"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.NonMergingWindowFn.is_merging" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.NonMergingWindowFn.merge"> |
| <code class="descname">merge</code><span class="sig-paren">(</span><em>merge_context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#NonMergingWindowFn.merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.NonMergingWindowFn.merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.window.GlobalWindows"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.window.</code><code class="descname">GlobalWindows</code><a class="reference internal" href="_modules/apache_beam/transforms/window.html#GlobalWindows"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.GlobalWindows" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.window.NonMergingWindowFn" title="apache_beam.transforms.window.NonMergingWindowFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.window.NonMergingWindowFn</span></code></a></p> |
| <p>A windowing function that assigns everything to one global window.</p> |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.GlobalWindows.assign"> |
| <code class="descname">assign</code><span class="sig-paren">(</span><em>assign_context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#GlobalWindows.assign"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.GlobalWindows.assign" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.window.GlobalWindows.from_runner_api_parameter"> |
| <em class="property">static </em><code class="descname">from_runner_api_parameter</code><span class="sig-paren">(</span><em>unused_fn_parameter</em>, <em>unused_context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#GlobalWindows.from_runner_api_parameter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.GlobalWindows.from_runner_api_parameter" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.GlobalWindows.get_window_coder"> |
| <code class="descname">get_window_coder</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#GlobalWindows.get_window_coder"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.GlobalWindows.get_window_coder" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.GlobalWindows.to_runner_api_parameter"> |
| <code class="descname">to_runner_api_parameter</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#GlobalWindows.to_runner_api_parameter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.GlobalWindows.to_runner_api_parameter" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="classmethod"> |
| <dt id="apache_beam.transforms.window.GlobalWindows.windowed_value"> |
| <em class="property">classmethod </em><code class="descname">windowed_value</code><span class="sig-paren">(</span><em>value</em>, <em>timestamp=Timestamp(-9223372036854.775808)</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#GlobalWindows.windowed_value"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.GlobalWindows.windowed_value" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.window.FixedWindows"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.window.</code><code class="descname">FixedWindows</code><span class="sig-paren">(</span><em>size</em>, <em>offset=0</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#FixedWindows"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.FixedWindows" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.window.NonMergingWindowFn" title="apache_beam.transforms.window.NonMergingWindowFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.window.NonMergingWindowFn</span></code></a></p> |
| <p>A windowing function that assigns each element to one time interval.</p> |
| <p>The attributes size and offset determine in what time interval a timestamp |
| will be slotted. The time intervals have the following formula: |
| [N * size + offset, (N + 1) * size + offset)</p> |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.window.FixedWindows.size"> |
| <code class="descname">size</code><a class="headerlink" href="#apache_beam.transforms.window.FixedWindows.size" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Size of the window as seconds.</p> |
| </dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.window.FixedWindows.offset"> |
| <code class="descname">offset</code><a class="headerlink" href="#apache_beam.transforms.window.FixedWindows.offset" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Offset of this window as seconds since Unix epoch. Windows start at |
| t=N * size + offset where t=0 is the epoch. The offset must be a value |
| in range [0, size). If it is not it will be normalized to this range.</p> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.FixedWindows.assign"> |
| <code class="descname">assign</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#FixedWindows.assign"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.FixedWindows.assign" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.window.FixedWindows.from_runner_api_parameter"> |
| <em class="property">static </em><code class="descname">from_runner_api_parameter</code><span class="sig-paren">(</span><em>fn_parameter</em>, <em>unused_context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#FixedWindows.from_runner_api_parameter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.FixedWindows.from_runner_api_parameter" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.FixedWindows.get_window_coder"> |
| <code class="descname">get_window_coder</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#FixedWindows.get_window_coder"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.FixedWindows.get_window_coder" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.FixedWindows.to_runner_api_parameter"> |
| <code class="descname">to_runner_api_parameter</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#FixedWindows.to_runner_api_parameter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.FixedWindows.to_runner_api_parameter" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.window.SlidingWindows"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.window.</code><code class="descname">SlidingWindows</code><span class="sig-paren">(</span><em>size</em>, <em>period</em>, <em>offset=0</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#SlidingWindows"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.SlidingWindows" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.window.NonMergingWindowFn" title="apache_beam.transforms.window.NonMergingWindowFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.window.NonMergingWindowFn</span></code></a></p> |
| <p>A windowing function that assigns each element to a set of sliding windows.</p> |
| <p>The attributes size and offset determine in what time interval a timestamp |
| will be slotted. The time intervals have the following formula: |
| [N * period + offset, N * period + offset + size)</p> |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.window.SlidingWindows.size"> |
| <code class="descname">size</code><a class="headerlink" href="#apache_beam.transforms.window.SlidingWindows.size" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Size of the window as seconds.</p> |
| </dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.window.SlidingWindows.period"> |
| <code class="descname">period</code><a class="headerlink" href="#apache_beam.transforms.window.SlidingWindows.period" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Period of the windows as seconds.</p> |
| </dd></dl> |
| |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.window.SlidingWindows.offset"> |
| <code class="descname">offset</code><a class="headerlink" href="#apache_beam.transforms.window.SlidingWindows.offset" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Offset of this window as seconds since Unix epoch. Windows start at |
| t=N * period + offset where t=0 is the epoch. The offset must be a value |
| in range [0, period). If it is not it will be normalized to this range.</p> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.SlidingWindows.assign"> |
| <code class="descname">assign</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#SlidingWindows.assign"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.SlidingWindows.assign" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.window.SlidingWindows.from_runner_api_parameter"> |
| <em class="property">static </em><code class="descname">from_runner_api_parameter</code><span class="sig-paren">(</span><em>fn_parameter</em>, <em>unused_context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#SlidingWindows.from_runner_api_parameter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.SlidingWindows.from_runner_api_parameter" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.SlidingWindows.get_window_coder"> |
| <code class="descname">get_window_coder</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#SlidingWindows.get_window_coder"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.SlidingWindows.get_window_coder" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.SlidingWindows.to_runner_api_parameter"> |
| <code class="descname">to_runner_api_parameter</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#SlidingWindows.to_runner_api_parameter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.SlidingWindows.to_runner_api_parameter" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="class"> |
| <dt id="apache_beam.transforms.window.Sessions"> |
| <em class="property">class </em><code class="descclassname">apache_beam.transforms.window.</code><code class="descname">Sessions</code><span class="sig-paren">(</span><em>gap_size</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#Sessions"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.Sessions" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.window.WindowFn" title="apache_beam.transforms.window.WindowFn"><code class="xref py py-class docutils literal"><span class="pre">apache_beam.transforms.window.WindowFn</span></code></a></p> |
| <p>A windowing function that groups elements into sessions.</p> |
| <p>A session is defined as a series of consecutive events |
| separated by a specified gap size.</p> |
| <dl class="attribute"> |
| <dt id="apache_beam.transforms.window.Sessions.gap_size"> |
| <code class="descname">gap_size</code><a class="headerlink" href="#apache_beam.transforms.window.Sessions.gap_size" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Size of the gap between windows as floating-point seconds.</p> |
| </dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.Sessions.assign"> |
| <code class="descname">assign</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#Sessions.assign"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.Sessions.assign" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="staticmethod"> |
| <dt id="apache_beam.transforms.window.Sessions.from_runner_api_parameter"> |
| <em class="property">static </em><code class="descname">from_runner_api_parameter</code><span class="sig-paren">(</span><em>fn_parameter</em>, <em>unused_context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#Sessions.from_runner_api_parameter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.Sessions.from_runner_api_parameter" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.Sessions.get_window_coder"> |
| <code class="descname">get_window_coder</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#Sessions.get_window_coder"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.Sessions.get_window_coder" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.Sessions.merge"> |
| <code class="descname">merge</code><span class="sig-paren">(</span><em>merge_context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#Sessions.merge"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.Sessions.merge" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| <dl class="method"> |
| <dt id="apache_beam.transforms.window.Sessions.to_runner_api_parameter"> |
| <code class="descname">to_runner_api_parameter</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/window.html#Sessions.to_runner_api_parameter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.transforms.window.Sessions.to_runner_api_parameter" title="Permalink to this definition">¶</a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| </div> |
| <div class="section" id="module-apache_beam.transforms"> |
| <span id="module-contents"></span><h2>Module contents<a class="headerlink" href="#module-apache_beam.transforms" title="Permalink to this headline">¶</a></h2> |
| <p>PTransform and descendants.</p> |
| </div> |
| </div> |
| |
| |
| </div> |
| </div> |
| </div> |
| <div class="clearer"></div> |
| </div> |
| <div class="related" role="navigation" aria-label="related navigation"> |
| <h3>Navigation</h3> |
| <ul> |
| <li class="right" style="margin-right: 10px"> |
| <a href="genindex.html" title="General Index" |
| >index</a></li> |
| <li class="right" > |
| <a href="py-modindex.html" title="Python Module Index" |
| >modules</a> |</li> |
| <li class="right" > |
| <a href="apache_beam.typehints.html" title="apache_beam.typehints package" |
| >next</a> |</li> |
| <li class="right" > |
| <a href="apache_beam.testing.html" title="apache_beam.testing package" |
| >previous</a> |</li> |
| <li class="nav-item nav-item-0"><a href="index.html">Apache Beam documentation</a> »</li> |
| <li class="nav-item nav-item-1"><a href="apache_beam.html" >apache_beam package</a> »</li> |
| </ul> |
| </div> |
| <div class="footer" role="contentinfo"> |
| © Copyright . |
| Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.5.5. |
| </div> |
| </body> |
| </html> |