| |
| |
| <!DOCTYPE html> |
| <html class="writer-html5" lang="en" data-content_root="./"> |
| <head> |
| <meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" /> |
| |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> |
| <title>apache_beam.transforms.core module — Apache Beam 2.68.0 documentation</title> |
| <link rel="stylesheet" type="text/css" href="_static/pygments.css?v=b86133f3" /> |
| <link rel="stylesheet" type="text/css" href="_static/css/theme.css?v=e59714d7" /> |
| |
| |
| <script src="_static/jquery.js?v=5d32c60e"></script> |
| <script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script> |
| <script src="_static/documentation_options.js?v=2388e03a"></script> |
| <script src="_static/doctools.js?v=9a2dae69"></script> |
| <script src="_static/sphinx_highlight.js?v=dc90522c"></script> |
| <script src="_static/js/theme.js"></script> |
| <link rel="index" title="Index" href="genindex.html" /> |
| <link rel="search" title="Search" href="search.html" /> |
| <link rel="next" title="apache_beam.transforms.create_source module" href="apache_beam.transforms.create_source.html" /> |
| <link rel="prev" title="apache_beam.transforms.combiners module" href="apache_beam.transforms.combiners.html" /> |
| </head> |
| |
| <body class="wy-body-for-nav"> |
| <div class="wy-grid-for-nav"> |
| <nav data-toggle="wy-nav-shift" class="wy-nav-side"> |
| <div class="wy-side-scroll"> |
| <div class="wy-side-nav-search" > |
| |
| |
| |
| <a href="index.html" class="icon icon-home"> |
| Apache Beam |
| </a> |
| <div role="search"> |
| <form id="rtd-search-form" class="wy-form" action="search.html" method="get"> |
| <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" /> |
| <input type="hidden" name="check_keywords" value="yes" /> |
| <input type="hidden" name="area" value="default" /> |
| </form> |
| </div> |
| </div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu"> |
| <ul class="current"> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.coders.html">apache_beam.coders package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.dataframe.html">apache_beam.dataframe package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.io.html">apache_beam.io package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.metrics.html">apache_beam.metrics package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.ml.html">apache_beam.ml package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.options.html">apache_beam.options package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.portability.html">apache_beam.portability package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.runners.html">apache_beam.runners package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.testing.html">apache_beam.testing package</a></li> |
| <li class="toctree-l1 current"><a class="reference internal" href="apache_beam.transforms.html">apache_beam.transforms package</a><ul class="current"> |
| <li class="toctree-l2"><a class="reference internal" href="apache_beam.transforms.html#subpackages">Subpackages</a></li> |
| <li class="toctree-l2 current"><a class="reference internal" href="apache_beam.transforms.html#submodules">Submodules</a><ul class="current"> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.async_dofn.html">apache_beam.transforms.async_dofn module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.combinefn_lifecycle_pipeline.html">apache_beam.transforms.combinefn_lifecycle_pipeline module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.combiners.html">apache_beam.transforms.combiners module</a></li> |
| <li class="toctree-l3 current"><a class="current reference internal" href="#">apache_beam.transforms.core module</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.DoFn"><code class="docutils literal notranslate"><span class="pre">DoFn</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.CombineFn"><code class="docutils literal notranslate"><span class="pre">CombineFn</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.PartitionFn"><code class="docutils literal notranslate"><span class="pre">PartitionFn</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.ParDo"><code class="docutils literal notranslate"><span class="pre">ParDo</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.FlatMap"><code class="docutils literal notranslate"><span class="pre">FlatMap()</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.FlatMapTuple"><code class="docutils literal notranslate"><span class="pre">FlatMapTuple()</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.Map"><code class="docutils literal notranslate"><span class="pre">Map()</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.MapTuple"><code class="docutils literal notranslate"><span class="pre">MapTuple()</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.Filter"><code class="docutils literal notranslate"><span class="pre">Filter()</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.CombineGlobally"><code class="docutils literal notranslate"><span class="pre">CombineGlobally</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.CombinePerKey"><code class="docutils literal notranslate"><span class="pre">CombinePerKey</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.CombineValues"><code class="docutils literal notranslate"><span class="pre">CombineValues</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.GroupBy"><code class="docutils literal notranslate"><span class="pre">GroupBy</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.GroupByKey"><code class="docutils literal notranslate"><span class="pre">GroupByKey</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.Select"><code class="docutils literal notranslate"><span class="pre">Select</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.Partition"><code class="docutils literal notranslate"><span class="pre">Partition</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.Windowing"><code class="docutils literal notranslate"><span class="pre">Windowing</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.WindowInto"><code class="docutils literal notranslate"><span class="pre">WindowInto</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.Flatten"><code class="docutils literal notranslate"><span class="pre">Flatten</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.FlattenWith"><code class="docutils literal notranslate"><span class="pre">FlattenWith</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.Create"><code class="docutils literal notranslate"><span class="pre">Create</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.Impulse"><code class="docutils literal notranslate"><span class="pre">Impulse</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.RestrictionProvider"><code class="docutils literal notranslate"><span class="pre">RestrictionProvider</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#apache_beam.transforms.core.WatermarkEstimatorProvider"><code class="docutils literal notranslate"><span class="pre">WatermarkEstimatorProvider</span></code></a></li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.create_source.html">apache_beam.transforms.create_source module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.deduplicate.html">apache_beam.transforms.deduplicate module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.display.html">apache_beam.transforms.display module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.enrichment.html">apache_beam.transforms.enrichment module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.environments.html">apache_beam.transforms.environments module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.error_handling.html">apache_beam.transforms.error_handling module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.external.html">apache_beam.transforms.external module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.external_java.html">apache_beam.transforms.external_java module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.external_transform_provider.html">apache_beam.transforms.external_transform_provider module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.fully_qualified_named_transform.html">apache_beam.transforms.fully_qualified_named_transform module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.managed.html">apache_beam.transforms.managed module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.periodicsequence.html">apache_beam.transforms.periodicsequence module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.ptransform.html">apache_beam.transforms.ptransform module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.resources.html">apache_beam.transforms.resources module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.sideinputs.html">apache_beam.transforms.sideinputs module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.sql.html">apache_beam.transforms.sql module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.stats.html">apache_beam.transforms.stats module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.timeutil.html">apache_beam.transforms.timeutil module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.trigger.html">apache_beam.transforms.trigger module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.userstate.html">apache_beam.transforms.userstate module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.util.html">apache_beam.transforms.util module</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="apache_beam.transforms.window.html">apache_beam.transforms.window module</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.typehints.html">apache_beam.typehints package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.utils.html">apache_beam.utils package</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.yaml.html">apache_beam.yaml package</a></li> |
| </ul> |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.error.html">apache_beam.error module</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.pipeline.html">apache_beam.pipeline module</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="apache_beam.pvalue.html">apache_beam.pvalue module</a></li> |
| </ul> |
| |
| </div> |
| </div> |
| </nav> |
| |
| <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" > |
| <i data-toggle="wy-nav-top" class="fa fa-bars"></i> |
| <a href="index.html">Apache Beam</a> |
| </nav> |
| |
| <div class="wy-nav-content"> |
| <div class="rst-content"> |
| <div role="navigation" aria-label="Page navigation"> |
| <ul class="wy-breadcrumbs"> |
| <li><a href="index.html" class="icon icon-home" aria-label="Home"></a></li> |
| <li class="breadcrumb-item"><a href="apache_beam.transforms.html">apache_beam.transforms package</a></li> |
| <li class="breadcrumb-item active">apache_beam.transforms.core module</li> |
| <li class="wy-breadcrumbs-aside"> |
| <a href="_sources/apache_beam.transforms.core.rst.txt" rel="nofollow"> View page source</a> |
| </li> |
| </ul> |
| <hr/> |
| </div> |
| <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> |
| <div itemprop="articleBody"> |
| |
| <section id="module-apache_beam.transforms.core"> |
| <span id="apache-beam-transforms-core-module"></span><h1>apache_beam.transforms.core module<a class="headerlink" href="#module-apache_beam.transforms.core" title="Link to this heading"></a></h1> |
| <p>Core PTransform subclasses, such as FlatMap, GroupByKey, and Map.</p> |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">DoFn</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">unused_args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">unused_kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference internal" href="apache_beam.typehints.decorators.html#apache_beam.typehints.decorators.WithTypeHints" title="apache_beam.typehints.decorators.WithTypeHints"><code class="xref py py-class docutils literal notranslate"><span class="pre">WithTypeHints</span></code></a>, <a class="reference internal" href="apache_beam.transforms.display.html#apache_beam.transforms.display.HasDisplayData" title="apache_beam.transforms.display.HasDisplayData"><code class="xref py py-class docutils literal notranslate"><span class="pre">HasDisplayData</span></code></a>, <a class="reference internal" href="apache_beam.utils.urns.html#apache_beam.utils.urns.RunnerApiFn" title="apache_beam.utils.urns.RunnerApiFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">RunnerApiFn</span></code></a></p> |
| <p>A function object used by a transform with custom processing.</p> |
| <p>The ParDo transform is such a transform. The ParDo.apply |
| method will take an object of type DoFn and apply it to all elements of a |
| PCollection object.</p> |
| <p>In order to have concrete DoFn objects one has to subclass from DoFn and |
| define the desired behavior (start_bundle/finish_bundle and process) or wrap a |
| callable object using the CallableWrapperDoFn class.</p> |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.ElementParam"> |
| <span class="sig-name descname"><span class="pre">ElementParam</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">ElementParam</span></em><a class="headerlink" href="#apache_beam.transforms.core.DoFn.ElementParam" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.SideInputParam"> |
| <span class="sig-name descname"><span class="pre">SideInputParam</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">SideInputParam</span></em><a class="headerlink" href="#apache_beam.transforms.core.DoFn.SideInputParam" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.TimestampParam"> |
| <span class="sig-name descname"><span class="pre">TimestampParam</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">TimestampParam</span></em><a class="headerlink" href="#apache_beam.transforms.core.DoFn.TimestampParam" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.WindowParam"> |
| <span class="sig-name descname"><span class="pre">WindowParam</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">WindowParam</span></em><a class="headerlink" href="#apache_beam.transforms.core.DoFn.WindowParam" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.WindowedValueParam"> |
| <span class="sig-name descname"><span class="pre">WindowedValueParam</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">WindowedValueParam</span></em><a class="headerlink" href="#apache_beam.transforms.core.DoFn.WindowedValueParam" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.PaneInfoParam"> |
| <span class="sig-name descname"><span class="pre">PaneInfoParam</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">PaneInfoParam</span></em><a class="headerlink" href="#apache_beam.transforms.core.DoFn.PaneInfoParam" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.WatermarkEstimatorParam"> |
| <span class="sig-name descname"><span class="pre">WatermarkEstimatorParam</span></span><a class="headerlink" href="#apache_beam.transforms.core.DoFn.WatermarkEstimatorParam" title="Link to this definition"></a></dt> |
| <dd><p>alias of <code class="xref py py-class docutils literal notranslate"><span class="pre">_WatermarkEstimatorParam</span></code></p> |
| </dd></dl> |
| |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.BundleFinalizerParam"> |
| <span class="sig-name descname"><span class="pre">BundleFinalizerParam</span></span><a class="headerlink" href="#apache_beam.transforms.core.DoFn.BundleFinalizerParam" title="Link to this definition"></a></dt> |
| <dd><p>alias of <code class="xref py py-class docutils literal notranslate"><span class="pre">_BundleFinalizerParam</span></code></p> |
| </dd></dl> |
| |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.KeyParam"> |
| <span class="sig-name descname"><span class="pre">KeyParam</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">KeyParam</span></em><a class="headerlink" href="#apache_beam.transforms.core.DoFn.KeyParam" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.BundleContextParam"> |
| <span class="sig-name descname"><span class="pre">BundleContextParam</span></span><a class="headerlink" href="#apache_beam.transforms.core.DoFn.BundleContextParam" title="Link to this definition"></a></dt> |
| <dd><p>alias of <code class="xref py py-class docutils literal notranslate"><span class="pre">_BundleContextParam</span></code></p> |
| </dd></dl> |
| |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.SetupContextParam"> |
| <span class="sig-name descname"><span class="pre">SetupContextParam</span></span><a class="headerlink" href="#apache_beam.transforms.core.DoFn.SetupContextParam" title="Link to this definition"></a></dt> |
| <dd><p>alias of <code class="xref py py-class docutils literal notranslate"><span class="pre">_SetupContextParam</span></code></p> |
| </dd></dl> |
| |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.StateParam"> |
| <span class="sig-name descname"><span class="pre">StateParam</span></span><a class="headerlink" href="#apache_beam.transforms.core.DoFn.StateParam" title="Link to this definition"></a></dt> |
| <dd><p>alias of <code class="xref py py-class docutils literal notranslate"><span class="pre">_StateDoFnParam</span></code></p> |
| </dd></dl> |
| |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.TimerParam"> |
| <span class="sig-name descname"><span class="pre">TimerParam</span></span><a class="headerlink" href="#apache_beam.transforms.core.DoFn.TimerParam" title="Link to this definition"></a></dt> |
| <dd><p>alias of <code class="xref py py-class docutils literal notranslate"><span class="pre">_TimerDoFnParam</span></code></p> |
| </dd></dl> |
| |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.DynamicTimerTagParam"> |
| <span class="sig-name descname"><span class="pre">DynamicTimerTagParam</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">DynamicTimerTagParam</span></em><a class="headerlink" href="#apache_beam.transforms.core.DoFn.DynamicTimerTagParam" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.DoFnProcessParams"> |
| <span class="sig-name descname"><span class="pre">DoFnProcessParams</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">[ElementParam,</span> <span class="pre">SideInputParam,</span> <span class="pre">TimestampParam,</span> <span class="pre">WindowParam,</span> <span class="pre">WindowedValueParam,</span> <span class="pre"><class</span> <span class="pre">'apache_beam.transforms.core._WatermarkEstimatorParam'>,</span> <span class="pre">PaneInfoParam,</span> <span class="pre"><class</span> <span class="pre">'apache_beam.transforms.core._BundleFinalizerParam'>,</span> <span class="pre">KeyParam,</span> <span class="pre"><class</span> <span class="pre">'apache_beam.transforms.core._StateDoFnParam'>,</span> <span class="pre"><class</span> <span class="pre">'apache_beam.transforms.core._TimerDoFnParam'>,</span> <span class="pre"><class</span> <span class="pre">'apache_beam.transforms.core._BundleContextParam'>,</span> <span class="pre"><class</span> <span class="pre">'apache_beam.transforms.core._SetupContextParam'>]</span></em><a class="headerlink" href="#apache_beam.transforms.core.DoFn.DoFnProcessParams" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.RestrictionParam"> |
| <span class="sig-name descname"><span class="pre">RestrictionParam</span></span><a class="headerlink" href="#apache_beam.transforms.core.DoFn.RestrictionParam" title="Link to this definition"></a></dt> |
| <dd><p>alias of <code class="xref py py-class docutils literal notranslate"><span class="pre">_RestrictionDoFnParam</span></code></p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.from_callable"> |
| <em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_callable</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.from_callable"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.from_callable" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.unbounded_per_element"> |
| <em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">unbounded_per_element</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.unbounded_per_element"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.unbounded_per_element" title="Link to this definition"></a></dt> |
| <dd><p>A decorator on process fn specifying that the fn performs an unbounded |
| amount of work per input element.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.yields_elements"> |
| <em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">yields_elements</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.yields_elements"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.yields_elements" title="Link to this definition"></a></dt> |
| <dd><p>A decorator to apply to <code class="docutils literal notranslate"><span class="pre">process_batch</span></code> indicating it yields elements.</p> |
| <p>By default <code class="docutils literal notranslate"><span class="pre">process_batch</span></code> is assumed to both consume and produce |
| “batches”, which are collections of multiple logical Beam elements. This |
| decorator indicates that <code class="docutils literal notranslate"><span class="pre">process_batch</span></code> <strong>produces</strong> individual elements |
| at a time. <code class="docutils literal notranslate"><span class="pre">process_batch</span></code> is always expected to consume batches.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.yields_batches"> |
| <em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">yields_batches</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.yields_batches"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.yields_batches" title="Link to this definition"></a></dt> |
| <dd><p>A decorator to apply to <code class="docutils literal notranslate"><span class="pre">process</span></code> indicating it yields batches.</p> |
| <p>By default <code class="docutils literal notranslate"><span class="pre">process</span></code> is assumed to both consume and produce |
| individual elements at a time. This decorator indicates that <code class="docutils literal notranslate"><span class="pre">process</span></code> |
| <strong>produces</strong> “batches”, which are collections of multiple logical Beam |
| elements.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.default_label"> |
| <span class="sig-name descname"><span class="pre">default_label</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.default_label"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.default_label" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.process"> |
| <span class="sig-name descname"><span class="pre">process</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">element</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.process"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.process" title="Link to this definition"></a></dt> |
| <dd><p>Method to use for processing elements.</p> |
| <p>This is invoked by <code class="docutils literal notranslate"><span class="pre">DoFnRunner</span></code> for each element of a input |
| <code class="docutils literal notranslate"><span class="pre">PCollection</span></code>.</p> |
| <p>The following parameters can be used as default values on <code class="docutils literal notranslate"><span class="pre">process</span></code> |
| arguments to indicate that a DoFn accepts the corresponding parameters. For |
| example, a DoFn might accept the element and its timestamp with the |
| following signature:</p> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">process</span><span class="p">(</span><span class="n">element</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">ElementParam</span><span class="p">,</span> <span class="n">timestamp</span><span class="o">=</span><span class="n">DoFn</span><span class="o">.</span><span class="n">TimestampParam</span><span class="p">):</span> |
| <span class="o">...</span> |
| </pre></div> |
| </div> |
| <p>The full set of parameters is:</p> |
| <ul class="simple"> |
| <li><p><code class="docutils literal notranslate"><span class="pre">DoFn.ElementParam</span></code>: element to be processed, should not be mutated.</p></li> |
| <li><p><code class="docutils literal notranslate"><span class="pre">DoFn.SideInputParam</span></code>: a side input that may be used when processing.</p></li> |
| <li><p><code class="docutils literal notranslate"><span class="pre">DoFn.TimestampParam</span></code>: timestamp of the input element.</p></li> |
| <li><p><code class="docutils literal notranslate"><span class="pre">DoFn.WindowParam</span></code>: <code class="docutils literal notranslate"><span class="pre">Window</span></code> the input element belongs to.</p></li> |
| <li><p><code class="docutils literal notranslate"><span class="pre">DoFn.TimerParam</span></code>: a <code class="docutils literal notranslate"><span class="pre">userstate.RuntimeTimer</span></code> object defined by the |
| spec of the parameter.</p></li> |
| <li><p><code class="docutils literal notranslate"><span class="pre">DoFn.StateParam</span></code>: a <code class="docutils literal notranslate"><span class="pre">userstate.RuntimeState</span></code> object defined by the |
| spec of the parameter.</p></li> |
| <li><p><code class="docutils literal notranslate"><span class="pre">DoFn.KeyParam</span></code>: key associated with the element.</p></li> |
| <li><p><code class="docutils literal notranslate"><span class="pre">DoFn.RestrictionParam</span></code>: an <code class="docutils literal notranslate"><span class="pre">iobase.RestrictionTracker</span></code> will be |
| provided here to allow treatment as a Splittable <code class="docutils literal notranslate"><span class="pre">DoFn</span></code>. The restriction |
| tracker will be derived from the restriction provider in the parameter.</p></li> |
| <li><p><code class="docutils literal notranslate"><span class="pre">DoFn.WatermarkEstimatorParam</span></code>: a function that can be used to track |
| output watermark of Splittable <code class="docutils literal notranslate"><span class="pre">DoFn</span></code> implementations.</p></li> |
| <li><p><code class="docutils literal notranslate"><span class="pre">DoFn.BundleContextParam</span></code>: allows a shared context manager to be used |
| per bundle</p></li> |
| <li><p><code class="docutils literal notranslate"><span class="pre">DoFn.SetupContextParam</span></code>: allows a shared context manager to be used |
| per DoFn</p></li> |
| </ul> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>element</strong> – The element to be processed</p></li> |
| <li><p><strong>*args</strong> – side inputs</p></li> |
| <li><p><strong>**kwargs</strong> – other keyword arguments.</p></li> |
| </ul> |
| </dd> |
| <dt class="field-even">Returns<span class="colon">:</span></dt> |
| <dd class="field-even"><p>An Iterable of output elements or None.</p> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.process_batch"> |
| <span class="sig-name descname"><span class="pre">process_batch</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">batch</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.process_batch"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.process_batch" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.setup"> |
| <span class="sig-name descname"><span class="pre">setup</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.setup"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.setup" title="Link to this definition"></a></dt> |
| <dd><p>Called to prepare an instance for processing bundles of elements.</p> |
| <p>This is a good place to initialize transient in-memory resources, such as |
| network connections. The resources can then be disposed in |
| <code class="docutils literal notranslate"><span class="pre">DoFn.teardown</span></code>.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.start_bundle"> |
| <span class="sig-name descname"><span class="pre">start_bundle</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.start_bundle"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.start_bundle" title="Link to this definition"></a></dt> |
| <dd><p>Called before a bundle of elements is processed on a worker.</p> |
| <p>Elements to be processed are split into bundles and distributed |
| to workers. Before a worker calls process() on the first element |
| of its bundle, it calls this method.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.finish_bundle"> |
| <span class="sig-name descname"><span class="pre">finish_bundle</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.finish_bundle"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.finish_bundle" title="Link to this definition"></a></dt> |
| <dd><p>Called after a bundle of elements is processed on a worker.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.teardown"> |
| <span class="sig-name descname"><span class="pre">teardown</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.teardown"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.teardown" title="Link to this definition"></a></dt> |
| <dd><p>Called to use to clean up this instance before it is discarded.</p> |
| <p>A runner will do its best to call this method on any given instance to |
| prevent leaks of transient resources, however, there may be situations where |
| this is impossible (e.g. process crash, hardware failure, etc.) or |
| unnecessary (e.g. the pipeline is shutting down and the process is about to |
| be killed anyway, so all transient resources will be released automatically |
| by the OS). In these cases, the call may not happen. It will also not be |
| retried, because in such situations the DoFn instance no longer exists, so |
| there’s no instance to retry it on.</p> |
| <p>Thus, all work that depends on input elements, and all externally important |
| side effects, must be performed in <code class="docutils literal notranslate"><span class="pre">DoFn.process</span></code> or |
| <code class="docutils literal notranslate"><span class="pre">DoFn.finish_bundle</span></code>.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.get_function_arguments"> |
| <span class="sig-name descname"><span class="pre">get_function_arguments</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.get_function_arguments"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.get_function_arguments" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.default_type_hints"> |
| <span class="sig-name descname"><span class="pre">default_type_hints</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.default_type_hints"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.default_type_hints" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.infer_output_type"> |
| <span class="sig-name descname"><span class="pre">infer_output_type</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_type</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.infer_output_type"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.infer_output_type" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.get_input_batch_type"> |
| <span class="sig-name descname"><span class="pre">get_input_batch_type</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_element_type</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">TypeConstraint</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/functions.html#type" title="(in Python v3.13)"><span class="pre">type</span></a><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><span class="pre">None</span></a></span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.get_input_batch_type"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.get_input_batch_type" title="Link to this definition"></a></dt> |
| <dd><p>Determine the batch type expected as input to process_batch.</p> |
| <p>The default implementation of <code class="docutils literal notranslate"><span class="pre">get_input_batch_type</span></code> simply observes the |
| input typehint for the first parameter of <code class="docutils literal notranslate"><span class="pre">process_batch</span></code>. A Batched DoFn |
| may override this method if a dynamic approach is required.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><p><strong>input_element_type</strong> – The <strong>element type</strong> of the input PCollection this |
| DoFn is being applied to.</p> |
| </dd> |
| <dt class="field-even">Returns<span class="colon">:</span></dt> |
| <dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">None</span></code> if this DoFn cannot accept batches, else a Beam typehint or |
| a native Python typehint.</p> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.get_output_batch_type"> |
| <span class="sig-name descname"><span class="pre">get_output_batch_type</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_element_type</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">TypeConstraint</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/functions.html#type" title="(in Python v3.13)"><span class="pre">type</span></a><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><span class="pre">None</span></a></span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#DoFn.get_output_batch_type"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.DoFn.get_output_batch_type" title="Link to this definition"></a></dt> |
| <dd><p>Determine the batch type produced by this DoFn’s <code class="docutils literal notranslate"><span class="pre">process_batch</span></code> |
| implementation and/or its <code class="docutils literal notranslate"><span class="pre">process</span></code> implementation with |
| <code class="docutils literal notranslate"><span class="pre">@yields_batch</span></code>.</p> |
| <p>The default implementation of this method observes the return type |
| annotations on <code class="docutils literal notranslate"><span class="pre">process_batch</span></code> and/or <code class="docutils literal notranslate"><span class="pre">process</span></code>. A Batched DoFn may |
| override this method if a dynamic approach is required.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><p><strong>input_element_type</strong> – The <strong>element type</strong> of the input PCollection this |
| DoFn is being applied to.</p> |
| </dd> |
| <dt class="field-even">Returns<span class="colon">:</span></dt> |
| <dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">None</span></code> if this DoFn will never yield batches, else a Beam typehint or |
| a native Python typehint.</p> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.DoFn.to_runner_api_parameter"> |
| <span class="sig-name descname"><span class="pre">to_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">context</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#apache_beam.transforms.core.DoFn.to_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineFn"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">CombineFn</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">unused_args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">unused_kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference internal" href="apache_beam.typehints.decorators.html#apache_beam.typehints.decorators.WithTypeHints" title="apache_beam.typehints.decorators.WithTypeHints"><code class="xref py py-class docutils literal notranslate"><span class="pre">WithTypeHints</span></code></a>, <a class="reference internal" href="apache_beam.transforms.display.html#apache_beam.transforms.display.HasDisplayData" title="apache_beam.transforms.display.HasDisplayData"><code class="xref py py-class docutils literal notranslate"><span class="pre">HasDisplayData</span></code></a>, <a class="reference internal" href="apache_beam.utils.urns.html#apache_beam.utils.urns.RunnerApiFn" title="apache_beam.utils.urns.RunnerApiFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">RunnerApiFn</span></code></a></p> |
| <p>A function object used by a Combine transform with custom processing.</p> |
| <p>A CombineFn specifies how multiple values in all or part of a PCollection can |
| be merged into a single value—essentially providing the same kind of |
| information as the arguments to the Python “reduce” builtin (except for the |
| input argument, which is an instance of CombineFnProcessContext). The |
| combining process proceeds as follows:</p> |
| <ol class="arabic simple"> |
| <li><p>Input values are partitioned into one or more batches.</p></li> |
| <li><p>For each batch, the setup method is invoked.</p></li> |
| <li><p>For each batch, the create_accumulator method is invoked to create a fresh |
| initial “accumulator” value representing the combination of zero values.</p></li> |
| <li><p>For each input value in the batch, the add_input method is invoked to |
| combine more values with the accumulator for that batch.</p></li> |
| <li><p>The merge_accumulators method is invoked to combine accumulators from |
| separate batches into a single combined output accumulator value, once all |
| of the accumulators have had all the input value in their batches added to |
| them. This operation is invoked repeatedly, until there is only one |
| accumulator value left.</p></li> |
| <li><p>The extract_output operation is invoked on the final accumulator to get |
| the output value.</p></li> |
| <li><p>The teardown method is invoked.</p></li> |
| </ol> |
| <p>Note: If this <strong>CombineFn</strong> is used with a transform that has defaults, |
| <strong>apply</strong> will be called with an empty list at expansion time to get the |
| default value.</p> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineFn.default_label"> |
| <span class="sig-name descname"><span class="pre">default_label</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.default_label"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.default_label" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineFn.setup"> |
| <span class="sig-name descname"><span class="pre">setup</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.setup"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.setup" title="Link to this definition"></a></dt> |
| <dd><p>Called to prepare an instance for combining.</p> |
| <p>This method can be useful if there is some state that needs to be loaded |
| before executing any of the other methods. The resources can then be |
| disposed of in <code class="docutils literal notranslate"><span class="pre">CombineFn.teardown</span></code>.</p> |
| <p>If you are using Dataflow, you need to enable Dataflow Runner V2 |
| before using this feature.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>*args</strong> – Additional arguments and side inputs.</p></li> |
| <li><p><strong>**kwargs</strong> – Additional arguments and side inputs.</p></li> |
| </ul> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineFn.create_accumulator"> |
| <span class="sig-name descname"><span class="pre">create_accumulator</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.create_accumulator"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.create_accumulator" title="Link to this definition"></a></dt> |
| <dd><p>Return a fresh, empty accumulator for the combine operation.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>*args</strong> – Additional arguments and side inputs.</p></li> |
| <li><p><strong>**kwargs</strong> – Additional arguments and side inputs.</p></li> |
| </ul> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineFn.add_input"> |
| <span class="sig-name descname"><span class="pre">add_input</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">mutable_accumulator</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">element</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.add_input"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.add_input" title="Link to this definition"></a></dt> |
| <dd><p>Return result of folding element into accumulator.</p> |
| <p>CombineFn implementors must override add_input.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>mutable_accumulator</strong> – the current accumulator, |
| may be modified and returned for efficiency</p></li> |
| <li><p><strong>element</strong> – the element to add, should not be mutated</p></li> |
| <li><p><strong>*args</strong> – Additional arguments and side inputs.</p></li> |
| <li><p><strong>**kwargs</strong> – Additional arguments and side inputs.</p></li> |
| </ul> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineFn.add_inputs"> |
| <span class="sig-name descname"><span class="pre">add_inputs</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">mutable_accumulator</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">elements</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.add_inputs"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.add_inputs" title="Link to this definition"></a></dt> |
| <dd><p>Returns the result of folding each element in elements into accumulator.</p> |
| <p>This is provided in case the implementation affords more efficient |
| bulk addition of elements. The default implementation simply loops |
| over the inputs invoking add_input for each one.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>mutable_accumulator</strong> – the current accumulator, |
| may be modified and returned for efficiency</p></li> |
| <li><p><strong>elements</strong> – the elements to add, should not be mutated</p></li> |
| <li><p><strong>*args</strong> – Additional arguments and side inputs.</p></li> |
| <li><p><strong>**kwargs</strong> – Additional arguments and side inputs.</p></li> |
| </ul> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineFn.merge_accumulators"> |
| <span class="sig-name descname"><span class="pre">merge_accumulators</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">accumulators</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.merge_accumulators"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.merge_accumulators" title="Link to this definition"></a></dt> |
| <dd><p>Returns the result of merging several accumulators |
| to a single accumulator value.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>accumulators</strong> – the accumulators to merge. |
| Only the first accumulator may be modified and returned for efficiency; |
| the other accumulators should not be mutated, because they may be |
| shared with other code and mutating them could lead to incorrect |
| results or data corruption.</p></li> |
| <li><p><strong>*args</strong> – Additional arguments and side inputs.</p></li> |
| <li><p><strong>**kwargs</strong> – Additional arguments and side inputs.</p></li> |
| </ul> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineFn.compact"> |
| <span class="sig-name descname"><span class="pre">compact</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">accumulator</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.compact"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.compact" title="Link to this definition"></a></dt> |
| <dd><p>Optionally returns a more compact representation of the accumulator.</p> |
| <p>This is called before an accumulator is sent across the wire, and can |
| be useful in cases where values are buffered or otherwise lazily |
| kept unprocessed when added to the accumulator. Should return an |
| equivalent, though possibly modified, accumulator.</p> |
| <p>By default returns the accumulator unmodified.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>accumulator</strong> – the current accumulator</p></li> |
| <li><p><strong>*args</strong> – Additional arguments and side inputs.</p></li> |
| <li><p><strong>**kwargs</strong> – Additional arguments and side inputs.</p></li> |
| </ul> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineFn.extract_output"> |
| <span class="sig-name descname"><span class="pre">extract_output</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">accumulator</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.extract_output"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.extract_output" title="Link to this definition"></a></dt> |
| <dd><p>Return result of converting accumulator into the output value.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>accumulator</strong> – the final accumulator value computed by this CombineFn |
| for the entire input key or PCollection. Can be modified for |
| efficiency.</p></li> |
| <li><p><strong>*args</strong> – Additional arguments and side inputs.</p></li> |
| <li><p><strong>**kwargs</strong> – Additional arguments and side inputs.</p></li> |
| </ul> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineFn.teardown"> |
| <span class="sig-name descname"><span class="pre">teardown</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.teardown"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.teardown" title="Link to this definition"></a></dt> |
| <dd><p>Called to clean up an instance before it is discarded.</p> |
| <p>If you are using Dataflow, you need to enable Dataflow Runner V2 |
| before using this feature.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>*args</strong> – Additional arguments and side inputs.</p></li> |
| <li><p><strong>**kwargs</strong> – Additional arguments and side inputs.</p></li> |
| </ul> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineFn.apply"> |
| <span class="sig-name descname"><span class="pre">apply</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">elements</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.apply"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.apply" title="Link to this definition"></a></dt> |
| <dd><p>Returns result of applying this CombineFn to the input values.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>elements</strong> – the set of values to combine.</p></li> |
| <li><p><strong>*args</strong> – Additional arguments and side inputs.</p></li> |
| <li><p><strong>**kwargs</strong> – Additional arguments and side inputs.</p></li> |
| </ul> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineFn.for_input_type"> |
| <span class="sig-name descname"><span class="pre">for_input_type</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_type</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.for_input_type"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.for_input_type" title="Link to this definition"></a></dt> |
| <dd><p>Returns a specialized implementation of self, if it exists.</p> |
| <p>Otherwise, returns self.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><p><strong>input_type</strong> – the type of input elements.</p> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineFn.from_callable"> |
| <em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_callable</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.from_callable"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.from_callable" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineFn.maybe_from_callable"> |
| <em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">maybe_from_callable</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#apache_beam.transforms.core.CombineFn" title="apache_beam.transforms.core.CombineFn"><span class="pre">CombineFn</span></a><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Callable" title="(in Python v3.13)"><span class="pre">Callable</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">has_side_inputs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/functions.html#bool" title="(in Python v3.13)"><span class="pre">bool</span></a></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">True</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="#apache_beam.transforms.core.CombineFn" title="apache_beam.transforms.core.CombineFn"><span class="pre">CombineFn</span></a></span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.maybe_from_callable"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.maybe_from_callable" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineFn.get_accumulator_coder"> |
| <span class="sig-name descname"><span class="pre">get_accumulator_coder</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineFn.get_accumulator_coder"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.get_accumulator_coder" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineFn.to_runner_api_parameter"> |
| <span class="sig-name descname"><span class="pre">to_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">context</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#apache_beam.transforms.core.CombineFn.to_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.PartitionFn"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">PartitionFn</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">unused_args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">unused_kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#PartitionFn"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.PartitionFn" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference internal" href="apache_beam.typehints.decorators.html#apache_beam.typehints.decorators.WithTypeHints" title="apache_beam.typehints.decorators.WithTypeHints"><code class="xref py py-class docutils literal notranslate"><span class="pre">WithTypeHints</span></code></a></p> |
| <p>A function object used by a Partition transform.</p> |
| <p>A PartitionFn specifies how individual values in a PCollection will be placed |
| into separate partitions, indexed by an integer.</p> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.PartitionFn.default_label"> |
| <span class="sig-name descname"><span class="pre">default_label</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#PartitionFn.default_label"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.PartitionFn.default_label" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.PartitionFn.partition_for"> |
| <span class="sig-name descname"><span class="pre">partition_for</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">element</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">T</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">num_partitions</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><span class="pre">int</span></a></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Any" title="(in Python v3.13)"><span class="pre">Any</span></a></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Any" title="(in Python v3.13)"><span class="pre">Any</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><span class="pre">int</span></a></span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#PartitionFn.partition_for"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.PartitionFn.partition_for" title="Link to this definition"></a></dt> |
| <dd><p>Specify which partition will receive this element.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>element</strong> – An element of the input PCollection.</p></li> |
| <li><p><strong>num_partitions</strong> – Number of partitions, i.e., output PCollections.</p></li> |
| <li><p><strong>*args</strong> – optional parameters and side inputs.</p></li> |
| <li><p><strong>**kwargs</strong> – optional parameters and side inputs.</p></li> |
| </ul> |
| </dd> |
| <dt class="field-even">Returns<span class="colon">:</span></dt> |
| <dd class="field-even"><p>An integer in [0, num_partitions).</p> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.ParDo"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">ParDo</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">PTransformWithSideInputs</span></code></p> |
| <p>A <a class="reference internal" href="#apache_beam.transforms.core.ParDo" title="apache_beam.transforms.core.ParDo"><code class="xref py py-class docutils literal notranslate"><span class="pre">ParDo</span></code></a> transform.</p> |
| <p>Processes an input <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> by applying a |
| <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">DoFn</span></code></a> to each element and returning the accumulated results into an |
| output <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a>. The type of the elements is |
| not fixed as long as the <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">DoFn</span></code></a> can deal with it. In reality the type |
| is restrained to some extent because the elements sometimes must be persisted |
| to external storage. See the <a class="reference internal" href="#apache_beam.transforms.core.ParDo.expand" title="apache_beam.transforms.core.ParDo.expand"><code class="xref py py-meth docutils literal notranslate"><span class="pre">expand()</span></code></a> method comments for a |
| detailed description of all possible arguments.</p> |
| <p>Note that the <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">DoFn</span></code></a> must return an iterable for each element of the |
| input <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a>. An easy way to do this is to |
| use the <code class="docutils literal notranslate"><span class="pre">yield</span></code> keyword in the process method.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>pcoll</strong> (<a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><em>PCollection</em></a>) – a <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> to be processed.</p></li> |
| <li><p><strong>fn</strong> (<cite>typing.Union[DoFn, typing.Callable]</cite>) – a <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">DoFn</span></code></a> object to be |
| applied to each element of <strong>pcoll</strong> argument, or a Callable.</p></li> |
| <li><p><strong>*args</strong> – positional arguments passed to the <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">DoFn</span></code></a> object.</p></li> |
| <li><p><strong>**kwargs</strong> – keyword arguments passed to the <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">DoFn</span></code></a> object.</p></li> |
| </ul> |
| </dd> |
| </dl> |
| <p>Note that the positional and keyword arguments will be processed in order |
| to detect <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> s that will be computed as |
| side inputs to the transform. During pipeline execution whenever the |
| <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">DoFn</span></code></a> object gets executed (its <a class="reference internal" href="#apache_beam.transforms.core.DoFn.process" title="apache_beam.transforms.core.DoFn.process"><code class="xref py py-meth docutils literal notranslate"><span class="pre">DoFn.process()</span></code></a> method gets |
| called) the <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> arguments will be |
| replaced by values from the <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> in the |
| exact positions where they appear in the argument lists.</p> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.ParDo.with_exception_handling"> |
| <span class="sig-name descname"><span class="pre">with_exception_handling</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="pre">main_tag='good',</span> <span class="pre">dead_letter_tag='bad',</span> <span class="pre">*,</span> <span class="pre">exc_class=<class</span> <span class="pre">'Exception'>,</span> <span class="pre">partial=False,</span> <span class="pre">use_subprocess=False,</span> <span class="pre">threshold=1,</span> <span class="pre">threshold_windowing=None,</span> <span class="pre">timeout=None,</span> <span class="pre">error_handler=None,</span> <span class="pre">on_failure_callback:</span> <span class="pre">~typing.Callable[[Exception,</span> <span class="pre">~typing.Any],</span> <span class="pre">None]</span> <span class="pre">|</span> <span class="pre">None</span> <span class="pre">=</span> <span class="pre">None,</span> <span class="pre">allow_unsafe_userstate_in_process=False</span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.with_exception_handling"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.with_exception_handling" title="Link to this definition"></a></dt> |
| <dd><p>Automatically provides a dead letter output for saving bad inputs. |
| This can allow a pipeline to continue successfully rather than fail or |
| continuously throw errors on retry when bad elements are encountered.</p> |
| <p>This returns a tagged output with two PCollections, the first being the |
| results of successfully processing the input PCollection, and the second |
| being the set of bad records (those which threw exceptions during |
| processing) along with information about the errors raised.</p> |
| <p>For example, one would write:</p> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">good</span><span class="p">,</span> <span class="n">bad</span> <span class="o">=</span> <span class="n">inputs</span> <span class="o">|</span> <span class="n">Map</span><span class="p">(</span><span class="n">maybe_erroring_fn</span><span class="p">)</span><span class="o">.</span><span class="n">with_exception_handling</span><span class="p">()</span> |
| </pre></div> |
| </div> |
| <p>and <cite>good</cite> will be a PCollection of mapped records and <cite>bad</cite> will contain |
| tuples of the form <cite>(input, error_string</cite>) for each input that raised an |
| exception.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>main_tag</strong> – tag to be used for the main (good) output of the DoFn, |
| useful to avoid possible conflicts if this DoFn already produces |
| multiple outputs. Optional, defaults to ‘good’.</p></li> |
| <li><p><strong>dead_letter_tag</strong> – tag to be used for the bad inputs, useful to avoid |
| possible conflicts if this DoFn already produces multiple outputs. |
| Optional, defaults to ‘bad’.</p></li> |
| <li><p><strong>exc_class</strong> – An exception class, or tuple of exception classes, to catch. |
| Optional, defaults to ‘Exception’.</p></li> |
| <li><p><strong>partial</strong> – Whether to emit outputs for an element as they’re produced |
| (which could result in partial outputs for a ParDo or FlatMap that |
| throws an error part way through execution) or buffer all outputs |
| until successful processing of the entire element. Optional, |
| defaults to False.</p></li> |
| <li><p><strong>use_subprocess</strong> – Whether to execute the DoFn logic in a subprocess. This |
| allows one to recover from errors that can crash the calling process |
| (e.g. from an underlying C/C++ library causing a segfault), but is |
| slower as elements and results must cross a process boundary. Note |
| that this starts up a long-running process that is used to handle |
| all the elements (until hard failure, which should be rare) rather |
| than a new process per element, so the overhead should be minimal |
| (and can be amortized if there’s any per-process or per-bundle |
| initialization that needs to be done). Optional, defaults to False.</p></li> |
| <li><p><strong>threshold</strong> – An upper bound on the ratio of inputs that can be bad before |
| aborting the entire pipeline. Optional, defaults to 1.0 (meaning |
| up to 100% of inputs can be bad and the pipeline will still succeed).</p></li> |
| <li><p><strong>threshold_windowing</strong> – Event-time windowing to use for threshold. Optional, |
| defaults to the windowing of the input.</p></li> |
| <li><p><strong>timeout</strong> – If the element has not finished processing in timeout seconds, |
| raise a TimeoutError. Defaults to None, meaning no time limit.</p></li> |
| <li><p><strong>error_handler</strong> – An ErrorHandler that should be used to consume the bad |
| records, rather than returning the good and bad records as a tuple.</p></li> |
| <li><p><strong>on_failure_callback</strong> – If an element fails or times out, |
| on_failure_callback will be invoked. It will receive the exception |
| and the element being processed in as args. In case of a timeout, |
| the exception will be of type <cite>TimeoutError</cite>. Be careful with this |
| callback - if you set a timeout, it will not apply to the callback, |
| and if the callback fails it will not be retried.</p></li> |
| <li><p><strong>allow_unsafe_userstate_in_process</strong> – If False, user state will not be |
| permitted in the DoFn’s process method. This is disabled by default |
| because user state is potentially unsafe with exception handling |
| since it can be successfully stored or cleared even if the associated |
| element fails and is routed to a dead letter queue. Semantics around |
| state in this kind of failure scenario are not well defined and are |
| subject to change.</p></li> |
| </ul> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.ParDo.with_error_handler"> |
| <span class="sig-name descname"><span class="pre">with_error_handler</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">error_handler</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">exception_handling_kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.with_error_handler"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.with_error_handler" title="Link to this definition"></a></dt> |
| <dd><p>An alias for <cite>with_exception_handling(error_handler=error_handler, …)</cite></p> |
| <p>This is provided to fit the general ErrorHandler conventions.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.ParDo.default_type_hints"> |
| <span class="sig-name descname"><span class="pre">default_type_hints</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.default_type_hints"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.default_type_hints" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.ParDo.infer_output_type"> |
| <span class="sig-name descname"><span class="pre">infer_output_type</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_type</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.infer_output_type"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.infer_output_type" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.ParDo.infer_batch_converters"> |
| <span class="sig-name descname"><span class="pre">infer_batch_converters</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_element_type</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.infer_batch_converters"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.infer_batch_converters" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.ParDo.make_fn"> |
| <span class="sig-name descname"><span class="pre">make_fn</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">has_side_inputs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.make_fn"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.make_fn" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.ParDo.display_data"> |
| <span class="sig-name descname"><span class="pre">display_data</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.display_data"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.display_data" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.ParDo.expand"> |
| <span class="sig-name descname"><span class="pre">expand</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">pcoll</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.expand"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.expand" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.ParDo.with_outputs"> |
| <span class="sig-name descname"><span class="pre">with_outputs</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">tags</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">main</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">allow_unknown_tags</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.with_outputs"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.with_outputs" title="Link to this definition"></a></dt> |
| <dd><p>Returns a tagged tuple allowing access to the outputs of a |
| <a class="reference internal" href="#apache_beam.transforms.core.ParDo" title="apache_beam.transforms.core.ParDo"><code class="xref py py-class docutils literal notranslate"><span class="pre">ParDo</span></code></a>.</p> |
| <p>The resulting object supports access to the |
| <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> associated with a tag |
| (e.g. <code class="docutils literal notranslate"><span class="pre">o.tag</span></code>, <code class="docutils literal notranslate"><span class="pre">o[tag]</span></code>) and iterating over the available tags |
| (e.g. <code class="docutils literal notranslate"><span class="pre">for</span> <span class="pre">tag</span> <span class="pre">in</span> <span class="pre">o:</span> <span class="pre">...</span></code>).</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>*tags</strong> – if non-empty, list of valid tags. If a list of valid tags is given, |
| it will be an error to use an undeclared tag later in the pipeline.</p></li> |
| <li><p><strong>**main_kw</strong> – dictionary empty or with one key <code class="docutils literal notranslate"><span class="pre">'main'</span></code> defining the tag to |
| be used for the main output (which will not have a tag associated with |
| it).</p></li> |
| </ul> |
| </dd> |
| <dt class="field-even">Returns<span class="colon">:</span></dt> |
| <dd class="field-even"><p>An object of type |
| <code class="xref py py-class docutils literal notranslate"><span class="pre">DoOutputsTuple</span></code> that bundles together all |
| the outputs of a <a class="reference internal" href="#apache_beam.transforms.core.ParDo" title="apache_beam.transforms.core.ParDo"><code class="xref py py-class docutils literal notranslate"><span class="pre">ParDo</span></code></a> transform and allows accessing the |
| individual <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> s for each output |
| using an <code class="docutils literal notranslate"><span class="pre">object.tag</span></code> syntax.</p> |
| </dd> |
| <dt class="field-odd">Return type<span class="colon">:</span></dt> |
| <dd class="field-odd"><p><em>DoOutputsTuple</em></p> |
| </dd> |
| <dt class="field-even">Raises<span class="colon">:</span></dt> |
| <dd class="field-even"><ul class="simple"> |
| <li><p><a class="reference external" href="https://docs.python.org/3/library/exceptions.html#TypeError" title="(in Python v3.13)"><strong>TypeError</strong></a> – if the <strong>self</strong> object is not a |
| <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> that is the result of a |
| <a class="reference internal" href="#apache_beam.transforms.core.ParDo" title="apache_beam.transforms.core.ParDo"><code class="xref py py-class docutils literal notranslate"><span class="pre">ParDo</span></code></a> transform.</p></li> |
| <li><p><a class="reference external" href="https://docs.python.org/3/library/exceptions.html#ValueError" title="(in Python v3.13)"><strong>ValueError</strong></a> – if <strong>main_kw</strong> contains any key other than |
| <code class="docutils literal notranslate"><span class="pre">'main'</span></code>.</p></li> |
| </ul> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.ParDo.to_runner_api_parameter"> |
| <span class="sig-name descname"><span class="pre">to_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">context</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="apache_beam.io.iobase.html#apache_beam.io.iobase.Read.PipelineContext" title="apache_beam.io.iobase.Read.PipelineContext"><span class="pre">PipelineContext</span></a></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">extra_kwargs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Any" title="(in Python v3.13)"><span class="pre">Any</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Tuple" title="(in Python v3.13)"><span class="pre">Tuple</span></a><span class="p"><span class="pre">[</span></span><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><span class="pre">str</span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">message.Message</span><span class="p"><span class="pre">]</span></span></span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.to_runner_api_parameter"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.to_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.ParDo.from_runner_api_parameter"> |
| <em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">unused_ptransform</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">pardo_payload</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">context</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.from_runner_api_parameter"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.from_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.ParDo.runner_api_requires_keyed_input"> |
| <span class="sig-name descname"><span class="pre">runner_api_requires_keyed_input</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.runner_api_requires_keyed_input"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.runner_api_requires_keyed_input" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.ParDo.get_restriction_coder"> |
| <span class="sig-name descname"><span class="pre">get_restriction_coder</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#ParDo.get_restriction_coder"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.ParDo.get_restriction_coder" title="Link to this definition"></a></dt> |
| <dd><p>Returns <cite>restriction coder if `DoFn</cite> of this <cite>ParDo</cite> is a SDF.</p> |
| <p>Returns <cite>None</cite> otherwise.</p> |
| </dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py function"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.FlatMap"> |
| <span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">FlatMap</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn=<function</span> <span class="pre">identity></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">*args</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">**kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#FlatMap"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.FlatMap" title="Link to this definition"></a></dt> |
| <dd><p><a class="reference internal" href="#apache_beam.transforms.core.FlatMap" title="apache_beam.transforms.core.FlatMap"><code class="xref py py-func docutils literal notranslate"><span class="pre">FlatMap()</span></code></a> is like <a class="reference internal" href="#apache_beam.transforms.core.ParDo" title="apache_beam.transforms.core.ParDo"><code class="xref py py-class docutils literal notranslate"><span class="pre">ParDo</span></code></a> except it takes a callable to |
| specify the transformation.</p> |
| <p>The callable must return an iterable for each element of the input |
| <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a>. The elements of these iterables will |
| be flattened into the output <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a>. If |
| no callable is given, then all elements of the input PCollection must already |
| be iterables themselves and will be flattened into the output PCollection.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>fn</strong> (<em>callable</em>) – a callable object.</p></li> |
| <li><p><strong>*args</strong> – positional arguments passed to the transform callable.</p></li> |
| <li><p><strong>**kwargs</strong> – keyword arguments passed to the transform callable.</p></li> |
| </ul> |
| </dd> |
| <dt class="field-even">Returns<span class="colon">:</span></dt> |
| <dd class="field-even"><p>A <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> containing the |
| <a class="reference internal" href="#apache_beam.transforms.core.FlatMap" title="apache_beam.transforms.core.FlatMap"><code class="xref py py-func docutils literal notranslate"><span class="pre">FlatMap()</span></code></a> outputs.</p> |
| </dd> |
| <dt class="field-odd">Return type<span class="colon">:</span></dt> |
| <dd class="field-odd"><p><a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><em>PCollection</em></a></p> |
| </dd> |
| <dt class="field-even">Raises<span class="colon">:</span></dt> |
| <dd class="field-even"><p><a class="reference external" href="https://docs.python.org/3/library/exceptions.html#TypeError" title="(in Python v3.13)"><strong>TypeError</strong></a> – If the <strong>fn</strong> passed as argument is not a callable. |
| Typical error is to pass a <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">DoFn</span></code></a> instance which is supported only |
| for <a class="reference internal" href="#apache_beam.transforms.core.ParDo" title="apache_beam.transforms.core.ParDo"><code class="xref py py-class docutils literal notranslate"><span class="pre">ParDo</span></code></a>.</p> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py function"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.FlatMapTuple"> |
| <span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">FlatMapTuple</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#FlatMapTuple"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.FlatMapTuple" title="Link to this definition"></a></dt> |
| <dd><p><a class="reference internal" href="#apache_beam.transforms.core.FlatMapTuple" title="apache_beam.transforms.core.FlatMapTuple"><code class="xref py py-func docutils literal notranslate"><span class="pre">FlatMapTuple()</span></code></a> is like <a class="reference internal" href="#apache_beam.transforms.core.FlatMap" title="apache_beam.transforms.core.FlatMap"><code class="xref py py-func docutils literal notranslate"><span class="pre">FlatMap()</span></code></a> but expects tuple inputs and |
| flattens them into multiple input arguments.</p> |
| <p>In other words</p> |
| <blockquote> |
| <div><p>beam.FlatMap(lambda start_end: range(start_end[0], start_end[1]))</p> |
| </div></blockquote> |
| <p>is equivalent to</p> |
| <blockquote> |
| <div><p>beam.FlatMapTuple(lambda start, end: range(start, end))</p> |
| </div></blockquote> |
| <p>This can be useful when processing a PCollection of tuples |
| (e.g. key-value pairs).</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>fn</strong> (<em>callable</em>) – a callable object.</p></li> |
| <li><p><strong>*args</strong> – positional arguments passed to the transform callable.</p></li> |
| <li><p><strong>**kwargs</strong> – keyword arguments passed to the transform callable.</p></li> |
| </ul> |
| </dd> |
| <dt class="field-even">Returns<span class="colon">:</span></dt> |
| <dd class="field-even"><p>A <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> containing the |
| <a class="reference internal" href="#apache_beam.transforms.core.FlatMapTuple" title="apache_beam.transforms.core.FlatMapTuple"><code class="xref py py-func docutils literal notranslate"><span class="pre">FlatMapTuple()</span></code></a> outputs.</p> |
| </dd> |
| <dt class="field-odd">Return type<span class="colon">:</span></dt> |
| <dd class="field-odd"><p><a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><em>PCollection</em></a></p> |
| </dd> |
| <dt class="field-even">Raises<span class="colon">:</span></dt> |
| <dd class="field-even"><p><a class="reference external" href="https://docs.python.org/3/library/exceptions.html#TypeError" title="(in Python v3.13)"><strong>TypeError</strong></a> – If the <strong>fn</strong> passed as argument is not a callable. |
| Typical error is to pass a <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">DoFn</span></code></a> instance which is supported only |
| for <a class="reference internal" href="#apache_beam.transforms.core.ParDo" title="apache_beam.transforms.core.ParDo"><code class="xref py py-class docutils literal notranslate"><span class="pre">ParDo</span></code></a>.</p> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py function"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Map"> |
| <span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">Map</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Map"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Map" title="Link to this definition"></a></dt> |
| <dd><p><a class="reference internal" href="#apache_beam.transforms.core.Map" title="apache_beam.transforms.core.Map"><code class="xref py py-func docutils literal notranslate"><span class="pre">Map()</span></code></a> is like <a class="reference internal" href="#apache_beam.transforms.core.FlatMap" title="apache_beam.transforms.core.FlatMap"><code class="xref py py-func docutils literal notranslate"><span class="pre">FlatMap()</span></code></a> except its callable returns only a |
| single element.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>fn</strong> (<em>callable</em>) – a callable object.</p></li> |
| <li><p><strong>*args</strong> – positional arguments passed to the transform callable.</p></li> |
| <li><p><strong>**kwargs</strong> – keyword arguments passed to the transform callable.</p></li> |
| </ul> |
| </dd> |
| <dt class="field-even">Returns<span class="colon">:</span></dt> |
| <dd class="field-even"><p>A <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> containing the |
| <a class="reference internal" href="#apache_beam.transforms.core.Map" title="apache_beam.transforms.core.Map"><code class="xref py py-func docutils literal notranslate"><span class="pre">Map()</span></code></a> outputs.</p> |
| </dd> |
| <dt class="field-odd">Return type<span class="colon">:</span></dt> |
| <dd class="field-odd"><p><a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><em>PCollection</em></a></p> |
| </dd> |
| <dt class="field-even">Raises<span class="colon">:</span></dt> |
| <dd class="field-even"><p><a class="reference external" href="https://docs.python.org/3/library/exceptions.html#TypeError" title="(in Python v3.13)"><strong>TypeError</strong></a> – If the <strong>fn</strong> passed as argument is not a callable. |
| Typical error is to pass a <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">DoFn</span></code></a> instance which is supported only |
| for <a class="reference internal" href="#apache_beam.transforms.core.ParDo" title="apache_beam.transforms.core.ParDo"><code class="xref py py-class docutils literal notranslate"><span class="pre">ParDo</span></code></a>.</p> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py function"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.MapTuple"> |
| <span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">MapTuple</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#MapTuple"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.MapTuple" title="Link to this definition"></a></dt> |
| <dd><p><a class="reference internal" href="#apache_beam.transforms.core.MapTuple" title="apache_beam.transforms.core.MapTuple"><code class="xref py py-func docutils literal notranslate"><span class="pre">MapTuple()</span></code></a> is like <a class="reference internal" href="#apache_beam.transforms.core.Map" title="apache_beam.transforms.core.Map"><code class="xref py py-func docutils literal notranslate"><span class="pre">Map()</span></code></a> but expects tuple inputs and |
| flattens them into multiple input arguments.</p> |
| <p>In other words</p> |
| <blockquote> |
| <div><p>“SwapKV” >> beam.Map(lambda kv: (kv[1], kv[0]))</p> |
| </div></blockquote> |
| <p>is equivalent to</p> |
| <blockquote> |
| <div><p>“SwapKV” >> beam.MapTuple(lambda k, v: (v, k))</p> |
| </div></blockquote> |
| <p>This can be useful when processing a PCollection of tuples |
| (e.g. key-value pairs).</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>fn</strong> (<em>callable</em>) – a callable object.</p></li> |
| <li><p><strong>*args</strong> – positional arguments passed to the transform callable.</p></li> |
| <li><p><strong>**kwargs</strong> – keyword arguments passed to the transform callable.</p></li> |
| </ul> |
| </dd> |
| <dt class="field-even">Returns<span class="colon">:</span></dt> |
| <dd class="field-even"><p>A <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> containing the |
| <a class="reference internal" href="#apache_beam.transforms.core.MapTuple" title="apache_beam.transforms.core.MapTuple"><code class="xref py py-func docutils literal notranslate"><span class="pre">MapTuple()</span></code></a> outputs.</p> |
| </dd> |
| <dt class="field-odd">Return type<span class="colon">:</span></dt> |
| <dd class="field-odd"><p><a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><em>PCollection</em></a></p> |
| </dd> |
| <dt class="field-even">Raises<span class="colon">:</span></dt> |
| <dd class="field-even"><p><a class="reference external" href="https://docs.python.org/3/library/exceptions.html#TypeError" title="(in Python v3.13)"><strong>TypeError</strong></a> – If the <strong>fn</strong> passed as argument is not a callable. |
| Typical error is to pass a <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">DoFn</span></code></a> instance which is supported only |
| for <a class="reference internal" href="#apache_beam.transforms.core.ParDo" title="apache_beam.transforms.core.ParDo"><code class="xref py py-class docutils literal notranslate"><span class="pre">ParDo</span></code></a>.</p> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py function"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Filter"> |
| <span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">Filter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Filter"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Filter" title="Link to this definition"></a></dt> |
| <dd><p><a class="reference internal" href="#apache_beam.transforms.core.Filter" title="apache_beam.transforms.core.Filter"><code class="xref py py-func docutils literal notranslate"><span class="pre">Filter()</span></code></a> is a <a class="reference internal" href="#apache_beam.transforms.core.FlatMap" title="apache_beam.transforms.core.FlatMap"><code class="xref py py-func docutils literal notranslate"><span class="pre">FlatMap()</span></code></a> with its callable filtering out |
| elements.</p> |
| <p>Filter accepts a function that keeps elements that return True, and filters |
| out the remaining elements.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>fn</strong> (<code class="docutils literal notranslate"><span class="pre">Callable[...,</span> <span class="pre">bool]</span></code>) – a callable object. First argument will be an |
| element.</p></li> |
| <li><p><strong>*args</strong> – positional arguments passed to the transform callable.</p></li> |
| <li><p><strong>**kwargs</strong> – keyword arguments passed to the transform callable.</p></li> |
| </ul> |
| </dd> |
| <dt class="field-even">Returns<span class="colon">:</span></dt> |
| <dd class="field-even"><p>A <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> containing the |
| <a class="reference internal" href="#apache_beam.transforms.core.Filter" title="apache_beam.transforms.core.Filter"><code class="xref py py-func docutils literal notranslate"><span class="pre">Filter()</span></code></a> outputs.</p> |
| </dd> |
| <dt class="field-odd">Return type<span class="colon">:</span></dt> |
| <dd class="field-odd"><p><a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><em>PCollection</em></a></p> |
| </dd> |
| <dt class="field-even">Raises<span class="colon">:</span></dt> |
| <dd class="field-even"><p><a class="reference external" href="https://docs.python.org/3/library/exceptions.html#TypeError" title="(in Python v3.13)"><strong>TypeError</strong></a> – If the <strong>fn</strong> passed as argument is not a callable. |
| Typical error is to pass a <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">DoFn</span></code></a> instance which is supported only |
| for <a class="reference internal" href="#apache_beam.transforms.core.ParDo" title="apache_beam.transforms.core.ParDo"><code class="xref py py-class docutils literal notranslate"><span class="pre">ParDo</span></code></a>.</p> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineGlobally"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">CombineGlobally</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineGlobally"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference internal" href="apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal notranslate"><span class="pre">PTransform</span></code></a></p> |
| <p>A <a class="reference internal" href="#apache_beam.transforms.core.CombineGlobally" title="apache_beam.transforms.core.CombineGlobally"><code class="xref py py-class docutils literal notranslate"><span class="pre">CombineGlobally</span></code></a> transform.</p> |
| <p>Reduces a <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> to a single value by |
| progressively applying a <a class="reference internal" href="#apache_beam.transforms.core.CombineFn" title="apache_beam.transforms.core.CombineFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">CombineFn</span></code></a> to portions of the |
| <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> (and to intermediate values created |
| thereby). See documentation in <a class="reference internal" href="#apache_beam.transforms.core.CombineFn" title="apache_beam.transforms.core.CombineFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">CombineFn</span></code></a> for details on the specifics |
| on how <a class="reference internal" href="#apache_beam.transforms.core.CombineFn" title="apache_beam.transforms.core.CombineFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">CombineFn</span></code></a> s are applied.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>pcoll</strong> (<a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><em>PCollection</em></a>) – a <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> to be reduced into a single |
| value.</p></li> |
| <li><p><strong>fn</strong> (<em>callable</em>) – a <a class="reference internal" href="#apache_beam.transforms.core.CombineFn" title="apache_beam.transforms.core.CombineFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">CombineFn</span></code></a> object that will be called to |
| progressively reduce the <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> into |
| single values, or a callable suitable for wrapping by |
| <code class="xref py py-class docutils literal notranslate"><span class="pre">CallableWrapperCombineFn</span></code>.</p></li> |
| <li><p><strong>*args</strong> – positional arguments passed to the <a class="reference internal" href="#apache_beam.transforms.core.CombineFn" title="apache_beam.transforms.core.CombineFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">CombineFn</span></code></a> object.</p></li> |
| <li><p><strong>**kwargs</strong> – keyword arguments passed to the <a class="reference internal" href="#apache_beam.transforms.core.CombineFn" title="apache_beam.transforms.core.CombineFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">CombineFn</span></code></a> object.</p></li> |
| </ul> |
| </dd> |
| <dt class="field-even">Raises<span class="colon">:</span></dt> |
| <dd class="field-even"><p><a class="reference external" href="https://docs.python.org/3/library/exceptions.html#TypeError" title="(in Python v3.13)"><strong>TypeError</strong></a> – If the output type of the input |
| <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> is not compatible |
| with <code class="docutils literal notranslate"><span class="pre">Iterable[A]</span></code>.</p> |
| </dd> |
| <dt class="field-odd">Returns<span class="colon">:</span></dt> |
| <dd class="field-odd"><p>A single-element |
| <a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><code class="xref py py-class docutils literal notranslate"><span class="pre">PCollection</span></code></a> containing the main output of |
| the <a class="reference internal" href="#apache_beam.transforms.core.CombineGlobally" title="apache_beam.transforms.core.CombineGlobally"><code class="xref py py-class docutils literal notranslate"><span class="pre">CombineGlobally</span></code></a> transform.</p> |
| </dd> |
| <dt class="field-even">Return type<span class="colon">:</span></dt> |
| <dd class="field-even"><p><a class="reference internal" href="apache_beam.pvalue.html#apache_beam.pvalue.PCollection" title="apache_beam.pvalue.PCollection"><em>PCollection</em></a></p> |
| </dd> |
| </dl> |
| <p>Note that the positional and keyword arguments will be processed in order |
| to detect <code class="xref py py-class docutils literal notranslate"><span class="pre">PValue</span></code> s that will be computed as side |
| inputs to the transform. |
| During pipeline execution whenever the <a class="reference internal" href="#apache_beam.transforms.core.CombineFn" title="apache_beam.transforms.core.CombineFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">CombineFn</span></code></a> object gets executed |
| (i.e. any of the <a class="reference internal" href="#apache_beam.transforms.core.CombineFn" title="apache_beam.transforms.core.CombineFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">CombineFn</span></code></a> methods get called), the |
| <code class="xref py py-class docutils literal notranslate"><span class="pre">PValue</span></code> arguments will be replaced by their |
| actual value in the exact position where they appear in the argument lists.</p> |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineGlobally.has_defaults"> |
| <span class="sig-name descname"><span class="pre">has_defaults</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">True</span></em><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.has_defaults" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineGlobally.as_view"> |
| <span class="sig-name descname"><span class="pre">as_view</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">False</span></em><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.as_view" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py attribute"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineGlobally.fanout"> |
| <span class="sig-name descname"><span class="pre">fanout</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Optional" title="(in Python v3.13)"><span class="pre">Optional</span></a><span class="p"><span class="pre">[</span></span><a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><span class="pre">int</span></a><span class="p"><span class="pre">]</span></span></em><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">None</span></em><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.fanout" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineGlobally.display_data"> |
| <span class="sig-name descname"><span class="pre">display_data</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineGlobally.display_data"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.display_data" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineGlobally.default_label"> |
| <span class="sig-name descname"><span class="pre">default_label</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineGlobally.default_label"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.default_label" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineGlobally.with_fanout"> |
| <span class="sig-name descname"><span class="pre">with_fanout</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fanout</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineGlobally.with_fanout"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.with_fanout" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineGlobally.with_defaults"> |
| <span class="sig-name descname"><span class="pre">with_defaults</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">has_defaults</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">True</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineGlobally.with_defaults"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.with_defaults" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineGlobally.without_defaults"> |
| <span class="sig-name descname"><span class="pre">without_defaults</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineGlobally.without_defaults"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.without_defaults" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineGlobally.as_singleton_view"> |
| <span class="sig-name descname"><span class="pre">as_singleton_view</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineGlobally.as_singleton_view"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.as_singleton_view" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineGlobally.expand"> |
| <span class="sig-name descname"><span class="pre">expand</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">pcoll</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineGlobally.expand"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.expand" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineGlobally.from_runner_api_parameter"> |
| <em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">unused_ptransform</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">combine_payload</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">context</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineGlobally.from_runner_api_parameter"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineGlobally.from_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombinePerKey"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">CombinePerKey</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombinePerKey"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombinePerKey" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">PTransformWithSideInputs</span></code></p> |
| <p>A per-key Combine transform.</p> |
| <p>Identifies sets of values associated with the same key in the input |
| PCollection, then applies a CombineFn to condense those sets to single |
| values. See documentation in CombineFn for details on the specifics on how |
| CombineFns are applied.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>pcoll</strong> – input pcollection.</p></li> |
| <li><p><strong>fn</strong> – instance of CombineFn to apply to all values under the same key in |
| pcoll, or a callable whose signature is <code class="docutils literal notranslate"><span class="pre">f(iterable,</span> <span class="pre">*args,</span> <span class="pre">**kwargs)</span></code> |
| (e.g., sum, max).</p></li> |
| <li><p><strong>*args</strong> – arguments and side inputs, passed directly to the CombineFn.</p></li> |
| <li><p><strong>**kwargs</strong> – arguments and side inputs, passed directly to the CombineFn.</p></li> |
| </ul> |
| </dd> |
| <dt class="field-even">Returns<span class="colon">:</span></dt> |
| <dd class="field-even"><p>A PObject holding the result of the combine operation.</p> |
| </dd> |
| </dl> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombinePerKey.with_hot_key_fanout"> |
| <span class="sig-name descname"><span class="pre">with_hot_key_fanout</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fanout</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombinePerKey.with_hot_key_fanout"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombinePerKey.with_hot_key_fanout" title="Link to this definition"></a></dt> |
| <dd><p>A per-key combine operation like self but with two levels of aggregation.</p> |
| <p>If a given key is produced by too many upstream bundles, the final |
| reduction can become a bottleneck despite partial combining being lifted |
| pre-GroupByKey. In these cases it can be helpful to perform intermediate |
| partial aggregations in parallel and then re-group to peform a final |
| (per-key) combine. This is also useful for high-volume keys in streaming |
| where combiners are not generally lifted for latency reasons.</p> |
| <p>Note that a fanout greater than 1 requires the data to be sent through |
| two GroupByKeys, and a high fanout can also result in more shuffle data |
| due to less per-bundle combining. Setting the fanout for a key at 1 or less |
| places values on the “cold key” path that skip the intermediate level of |
| aggregation.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><p><strong>fanout</strong> – either None, for no fanout, an int, for a constant-degree fanout, |
| or a callable mapping keys to a key-specific degree of fanout.</p> |
| </dd> |
| <dt class="field-even">Returns<span class="colon">:</span></dt> |
| <dd class="field-even"><p>A per-key combining PTransform with the specified fanout.</p> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombinePerKey.display_data"> |
| <span class="sig-name descname"><span class="pre">display_data</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombinePerKey.display_data"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombinePerKey.display_data" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombinePerKey.make_fn"> |
| <span class="sig-name descname"><span class="pre">make_fn</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">has_side_inputs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombinePerKey.make_fn"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombinePerKey.make_fn" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombinePerKey.default_label"> |
| <span class="sig-name descname"><span class="pre">default_label</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombinePerKey.default_label"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombinePerKey.default_label" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombinePerKey.expand"> |
| <span class="sig-name descname"><span class="pre">expand</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">pcoll</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombinePerKey.expand"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombinePerKey.expand" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombinePerKey.default_type_hints"> |
| <span class="sig-name descname"><span class="pre">default_type_hints</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombinePerKey.default_type_hints"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombinePerKey.default_type_hints" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombinePerKey.to_runner_api_parameter"> |
| <span class="sig-name descname"><span class="pre">to_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">context</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="apache_beam.io.iobase.html#apache_beam.io.iobase.Read.PipelineContext" title="apache_beam.io.iobase.Read.PipelineContext"><span class="pre">PipelineContext</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Tuple" title="(in Python v3.13)"><span class="pre">Tuple</span></a><span class="p"><span class="pre">[</span></span><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><span class="pre">str</span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">beam_runner_api_pb2.CombinePayload</span><span class="p"><span class="pre">]</span></span></span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombinePerKey.to_runner_api_parameter"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombinePerKey.to_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombinePerKey.from_runner_api_parameter"> |
| <em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">unused_ptransform</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">combine_payload</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">context</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombinePerKey.from_runner_api_parameter"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombinePerKey.from_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombinePerKey.runner_api_requires_keyed_input"> |
| <span class="sig-name descname"><span class="pre">runner_api_requires_keyed_input</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombinePerKey.runner_api_requires_keyed_input"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombinePerKey.runner_api_requires_keyed_input" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineValues"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">CombineValues</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="apache_beam.typehints.decorators.html#apache_beam.typehints.decorators.WithTypeHints" title="apache_beam.typehints.decorators.WithTypeHints"><span class="pre">WithTypeHints</span></a></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Any" title="(in Python v3.13)"><span class="pre">Any</span></a></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Any" title="(in Python v3.13)"><span class="pre">Any</span></a></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineValues"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineValues" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">PTransformWithSideInputs</span></code></p> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineValues.make_fn"> |
| <span class="sig-name descname"><span class="pre">make_fn</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">has_side_inputs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineValues.make_fn"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineValues.make_fn" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineValues.expand"> |
| <span class="sig-name descname"><span class="pre">expand</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">pcoll</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineValues.expand"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineValues.expand" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineValues.to_runner_api_parameter"> |
| <span class="sig-name descname"><span class="pre">to_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">context</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineValues.to_runner_api_parameter"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineValues.to_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.CombineValues.from_runner_api_parameter"> |
| <em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">unused_ptransform</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">combine_payload</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">context</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#CombineValues.from_runner_api_parameter"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.CombineValues.from_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.GroupBy"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">GroupBy</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">fields</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupBy"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupBy" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference internal" href="apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal notranslate"><span class="pre">PTransform</span></code></a></p> |
| <p>Groups a PCollection by one or more expressions, used to derive the key.</p> |
| <p><cite>GroupBy(expr)</cite> is roughly equivalent to</p> |
| <blockquote> |
| <div><p>beam.Map(lambda v: (expr(v), v)) | beam.GroupByKey()</p> |
| </div></blockquote> |
| <p>but provides several conveniences, e.g.</p> |
| <blockquote> |
| <div><ul class="simple"> |
| <li><p>Several arguments may be provided, as positional or keyword arguments, |
| resulting in a tuple-like key. For example <cite>GroupBy(a=expr1, b=expr2)</cite> |
| groups by a key with attributes <cite>a</cite> and <cite>b</cite> computed by applying |
| <cite>expr1</cite> and <cite>expr2</cite> to each element.</p></li> |
| <li><p>Strings can be used as a shorthand for accessing an attribute, e.g. |
| <cite>GroupBy(‘some_field’)</cite> is equivalent to |
| <cite>GroupBy(lambda v: getattr(v, ‘some_field’))</cite>.</p></li> |
| </ul> |
| </div></blockquote> |
| <p>The GroupBy operation can be made into an aggregating operation by invoking |
| its <cite>aggregate_field</cite> method.</p> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.GroupBy.aggregate_field"> |
| <span class="sig-name descname"><span class="pre">aggregate_field</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">field</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">combine_fn</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">dest</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupBy.aggregate_field"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupBy.aggregate_field" title="Link to this definition"></a></dt> |
| <dd><p>Returns a grouping operation that also aggregates grouped values.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>field</strong> – indicates the field to be aggregated</p></li> |
| <li><p><strong>combine_fn</strong> – indicates the aggregation function to be used</p></li> |
| <li><p><strong>dest</strong> – indicates the name that will be used for the aggregate in the output</p></li> |
| </ul> |
| </dd> |
| </dl> |
| <p>May be called repeatedly to aggregate multiple fields, e.g.</p> |
| <blockquote> |
| <div><dl class="simple"> |
| <dt>GroupBy(‘key’)</dt><dd><p>.aggregate_field(‘some_attr’, sum, ‘sum_attr’) |
| .aggregate_field(lambda v: …, MeanCombineFn, ‘mean’)</p> |
| </dd> |
| </dl> |
| </div></blockquote> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.GroupBy.force_tuple_keys"> |
| <span class="sig-name descname"><span class="pre">force_tuple_keys</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">value</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">True</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupBy.force_tuple_keys"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupBy.force_tuple_keys" title="Link to this definition"></a></dt> |
| <dd><p>Forces the keys to always be tuple-like, even if there is only a single |
| expression.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.GroupBy.default_label"> |
| <span class="sig-name descname"><span class="pre">default_label</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupBy.default_label"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupBy.default_label" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.GroupBy.expand"> |
| <span class="sig-name descname"><span class="pre">expand</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">pcoll</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupBy.expand"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupBy.expand" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.GroupByKey"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">GroupByKey</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">label</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><span class="pre">str</span></a><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><span class="pre">None</span></a></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupByKey"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupByKey" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference internal" href="apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal notranslate"><span class="pre">PTransform</span></code></a></p> |
| <p>A group by key transform.</p> |
| <p>Processes an input PCollection consisting of key/value pairs represented as a |
| tuple pair. The result is a PCollection where values having a common key are |
| grouped together. For example (a, 1), (b, 2), (a, 3) will result into |
| (a, [1, 3]), (b, [2]).</p> |
| <p>The implementation here is used only when run on the local direct runner.</p> |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.GroupByKey.ReifyWindows"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">ReifyWindows</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">unused_args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">unused_kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupByKey.ReifyWindows"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupByKey.ReifyWindows" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">DoFn</span></code></a></p> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.GroupByKey.ReifyWindows.process"> |
| <span class="sig-name descname"><span class="pre">process</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">element</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">window</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">WindowParam</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">timestamp</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">TimestampParam</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupByKey.ReifyWindows.process"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupByKey.ReifyWindows.process" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.GroupByKey.ReifyWindows.infer_output_type"> |
| <span class="sig-name descname"><span class="pre">infer_output_type</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_type</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupByKey.ReifyWindows.infer_output_type"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupByKey.ReifyWindows.infer_output_type" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.GroupByKey.expand"> |
| <span class="sig-name descname"><span class="pre">expand</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">pcoll</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupByKey.expand"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupByKey.expand" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.GroupByKey.infer_output_type"> |
| <span class="sig-name descname"><span class="pre">infer_output_type</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_type</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupByKey.infer_output_type"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupByKey.infer_output_type" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.GroupByKey.to_runner_api_parameter"> |
| <span class="sig-name descname"><span class="pre">to_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">unused_context</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="apache_beam.io.iobase.html#apache_beam.io.iobase.Read.PipelineContext" title="apache_beam.io.iobase.Read.PipelineContext"><span class="pre">PipelineContext</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Tuple" title="(in Python v3.13)"><span class="pre">Tuple</span></a><span class="p"><span class="pre">[</span></span><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><span class="pre">str</span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><span class="pre">None</span></a><span class="p"><span class="pre">]</span></span></span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupByKey.to_runner_api_parameter"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupByKey.to_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.GroupByKey.from_runner_api_parameter"> |
| <em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">unused_ptransform</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">unused_payload</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">unused_context</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupByKey.from_runner_api_parameter"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupByKey.from_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.GroupByKey.runner_api_requires_keyed_input"> |
| <span class="sig-name descname"><span class="pre">runner_api_requires_keyed_input</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#GroupByKey.runner_api_requires_keyed_input"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.GroupByKey.runner_api_requires_keyed_input" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Select"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">Select</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Select"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Select" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference internal" href="apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal notranslate"><span class="pre">PTransform</span></code></a></p> |
| <p>Converts the elements of a PCollection into a schema’d PCollection of Rows.</p> |
| <p><cite>Select(…)</cite> is roughly equivalent to <cite>Map(lambda x: Row(…))</cite> where each |
| argument (which may be a string or callable) of <cite>ToRow</cite> is applied to <cite>x</cite>. |
| For example,</p> |
| <blockquote> |
| <div><p>pcoll | beam.Select(‘a’, b=lambda x: foo(x))</p> |
| </div></blockquote> |
| <p>is the same as</p> |
| <blockquote> |
| <div><p>pcoll | beam.Map(lambda x: beam.Row(a=x.a, b=foo(x)))</p> |
| </div></blockquote> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Select.with_exception_handling"> |
| <span class="sig-name descname"><span class="pre">with_exception_handling</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Select.with_exception_handling"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Select.with_exception_handling" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Select.default_label"> |
| <span class="sig-name descname"><span class="pre">default_label</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Select.default_label"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Select.default_label" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Select.expand"> |
| <span class="sig-name descname"><span class="pre">expand</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">pcoll</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Select.expand"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Select.expand" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Select.infer_output_type"> |
| <span class="sig-name descname"><span class="pre">infer_output_type</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_type</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Select.infer_output_type"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Select.infer_output_type" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Partition"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">Partition</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="apache_beam.typehints.decorators.html#apache_beam.typehints.decorators.WithTypeHints" title="apache_beam.typehints.decorators.WithTypeHints"><span class="pre">WithTypeHints</span></a></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Any" title="(in Python v3.13)"><span class="pre">Any</span></a></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Any" title="(in Python v3.13)"><span class="pre">Any</span></a></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Partition"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Partition" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">PTransformWithSideInputs</span></code></p> |
| <p>Split a PCollection into several partitions.</p> |
| <p>Uses the specified PartitionFn to separate an input PCollection into the |
| specified number of sub-PCollections.</p> |
| <p>When apply()d, a Partition() PTransform requires the following:</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>partitionfn</strong> – a PartitionFn, or a callable with the signature described in |
| CallableWrapperPartitionFn.</p></li> |
| <li><p><strong>n</strong> – number of output partitions.</p></li> |
| </ul> |
| </dd> |
| </dl> |
| <p>The result of this PTransform is a simple list of the output PCollections |
| representing each of n partitions, in order.</p> |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Partition.ApplyPartitionFnFn"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">ApplyPartitionFnFn</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">unused_args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">unused_kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Partition.ApplyPartitionFnFn"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Partition.ApplyPartitionFnFn" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">DoFn</span></code></a></p> |
| <p>A DoFn that applies a PartitionFn.</p> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Partition.ApplyPartitionFnFn.process"> |
| <span class="sig-name descname"><span class="pre">process</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">element</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">partitionfn</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">n</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Partition.ApplyPartitionFnFn.process"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Partition.ApplyPartitionFnFn.process" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Partition.make_fn"> |
| <span class="sig-name descname"><span class="pre">make_fn</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">fn</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">has_side_inputs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Partition.make_fn"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Partition.make_fn" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Partition.expand"> |
| <span class="sig-name descname"><span class="pre">expand</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">pcoll</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Partition.expand"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Partition.expand" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Windowing"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">Windowing</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">windowfn</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">triggerfn</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">accumulation_mode</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">timestamp_combiner</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">allowed_lateness</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">0</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">environment_id</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Windowing"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Windowing" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference external" href="https://docs.python.org/3/library/functions.html#object" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate"><span class="pre">object</span></code></a></p> |
| <p>Class representing the window strategy.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>windowfn</strong> – Window assign function.</p></li> |
| <li><p><strong>triggerfn</strong> – Trigger function.</p></li> |
| <li><p><strong>accumulation_mode</strong> – a AccumulationMode, controls what to do with data |
| when a trigger fires multiple times.</p></li> |
| <li><p><strong>timestamp_combiner</strong> – a TimestampCombiner, determines how output |
| timestamps of grouping operations are assigned.</p></li> |
| <li><p><strong>allowed_lateness</strong> – Maximum delay in seconds after end of window |
| allowed for any late data to be processed without being discarded |
| directly.</p></li> |
| <li><p><strong>environment_id</strong> – Environment where the current window_fn should be |
| applied in.</p></li> |
| </ul> |
| </dd> |
| </dl> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Windowing.is_default"> |
| <span class="sig-name descname"><span class="pre">is_default</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Windowing.is_default"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Windowing.is_default" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Windowing.to_runner_api"> |
| <span class="sig-name descname"><span class="pre">to_runner_api</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">context</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="apache_beam.io.iobase.html#apache_beam.io.iobase.Read.PipelineContext" title="apache_beam.io.iobase.Read.PipelineContext"><span class="pre">PipelineContext</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">beam_runner_api_pb2.WindowingStrategy</span></span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Windowing.to_runner_api"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Windowing.to_runner_api" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Windowing.from_runner_api"> |
| <em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_runner_api</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">proto</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">context</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Windowing.from_runner_api"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Windowing.from_runner_api" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.WindowInto"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">WindowInto</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">windowfn</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">trigger</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">accumulation_mode</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">timestamp_combiner</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">allowed_lateness</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">0</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WindowInto"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.WindowInto" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.core.ParDo" title="apache_beam.transforms.core.ParDo"><code class="xref py py-class docutils literal notranslate"><span class="pre">ParDo</span></code></a></p> |
| <p>A window transform assigning windows to each element of a PCollection.</p> |
| <p>Transforms an input PCollection by applying a windowing function to each |
| element. Each transformed element in the result will be a WindowedValue |
| element with the same input value and timestamp, with its new set of windows |
| determined by the windowing function.</p> |
| <p>Initializes a WindowInto transform.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><ul class="simple"> |
| <li><p><strong>windowfn</strong> (<a class="reference internal" href="#apache_beam.transforms.core.Windowing" title="apache_beam.transforms.core.Windowing"><em>Windowing</em></a><em>, </em><a class="reference internal" href="apache_beam.transforms.window.html#apache_beam.transforms.window.WindowFn" title="apache_beam.transforms.window.WindowFn"><em>WindowFn</em></a>) – Function to be used for windowing.</p></li> |
| <li><p><strong>trigger</strong> – (optional) Trigger used for windowing, or None for default.</p></li> |
| <li><p><strong>accumulation_mode</strong> – (optional) Accumulation mode used for windowing, |
| required for non-trivial triggers.</p></li> |
| <li><p><strong>timestamp_combiner</strong> – (optional) Timestamp combniner used for windowing, |
| or None for default.</p></li> |
| </ul> |
| </dd> |
| </dl> |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.WindowInto.WindowIntoFn"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">WindowIntoFn</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">windowing</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#apache_beam.transforms.core.Windowing" title="apache_beam.transforms.core.Windowing"><span class="pre">Windowing</span></a></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WindowInto.WindowIntoFn"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.WindowInto.WindowIntoFn" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference internal" href="#apache_beam.transforms.core.DoFn" title="apache_beam.transforms.core.DoFn"><code class="xref py py-class docutils literal notranslate"><span class="pre">DoFn</span></code></a></p> |
| <p>A DoFn that applies a WindowInto operation.</p> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.WindowInto.WindowIntoFn.process"> |
| <span class="sig-name descname"><span class="pre">process</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">element</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">timestamp</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">TimestampParam</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">window</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">WindowParam</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WindowInto.WindowIntoFn.process"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.WindowInto.WindowIntoFn.process" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.WindowInto.WindowIntoFn.infer_output_type"> |
| <span class="sig-name descname"><span class="pre">infer_output_type</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_type</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WindowInto.WindowIntoFn.infer_output_type"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.WindowInto.WindowIntoFn.infer_output_type" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.WindowInto.get_windowing"> |
| <span class="sig-name descname"><span class="pre">get_windowing</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">unused_inputs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Any" title="(in Python v3.13)"><span class="pre">Any</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="#apache_beam.transforms.core.Windowing" title="apache_beam.transforms.core.Windowing"><span class="pre">Windowing</span></a></span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WindowInto.get_windowing"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.WindowInto.get_windowing" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.WindowInto.infer_output_type"> |
| <span class="sig-name descname"><span class="pre">infer_output_type</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_type</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WindowInto.infer_output_type"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.WindowInto.infer_output_type" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.WindowInto.expand"> |
| <span class="sig-name descname"><span class="pre">expand</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">pcoll</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WindowInto.expand"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.WindowInto.expand" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.WindowInto.to_runner_api_parameter"> |
| <span class="sig-name descname"><span class="pre">to_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">context</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="apache_beam.io.iobase.html#apache_beam.io.iobase.Read.PipelineContext" title="apache_beam.io.iobase.Read.PipelineContext"><span class="pre">PipelineContext</span></a></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">extra_kwargs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Any" title="(in Python v3.13)"><span class="pre">Any</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Tuple" title="(in Python v3.13)"><span class="pre">Tuple</span></a><span class="p"><span class="pre">[</span></span><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><span class="pre">str</span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">message.Message</span><span class="p"><span class="pre">]</span></span></span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WindowInto.to_runner_api_parameter"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.WindowInto.to_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.WindowInto.from_runner_api_parameter"> |
| <em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">unused_ptransform</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">proto</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">context</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WindowInto.from_runner_api_parameter"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.WindowInto.from_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Flatten"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">Flatten</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Flatten"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Flatten" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference internal" href="apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal notranslate"><span class="pre">PTransform</span></code></a></p> |
| <p>Merges several PCollections into a single PCollection.</p> |
| <p>Copies all elements in 0 or more PCollections into a single output |
| PCollection. If there are no input PCollections, the resulting PCollection |
| will be empty (but see also kwargs below).</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><p><strong>**kwargs</strong> – Accepts a single named argument “pipeline”, which specifies the |
| pipeline that “owns” this PTransform. Ordinarily Flatten can obtain this |
| information from one of the input PCollections, but if there are none (or |
| if there’s a chance there may be none), this argument is the only way to |
| provide pipeline information and should be considered mandatory.</p> |
| </dd> |
| </dl> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Flatten.expand"> |
| <span class="sig-name descname"><span class="pre">expand</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">pcolls</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Flatten.expand"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Flatten.expand" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Flatten.infer_output_type"> |
| <span class="sig-name descname"><span class="pre">infer_output_type</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_type</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Flatten.infer_output_type"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Flatten.infer_output_type" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Flatten.to_runner_api_parameter"> |
| <span class="sig-name descname"><span class="pre">to_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">context</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="apache_beam.io.iobase.html#apache_beam.io.iobase.Read.PipelineContext" title="apache_beam.io.iobase.Read.PipelineContext"><span class="pre">PipelineContext</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Tuple" title="(in Python v3.13)"><span class="pre">Tuple</span></a><span class="p"><span class="pre">[</span></span><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><span class="pre">str</span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><span class="pre">None</span></a><span class="p"><span class="pre">]</span></span></span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Flatten.to_runner_api_parameter"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Flatten.to_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Flatten.from_runner_api_parameter"> |
| <em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">unused_ptransform</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">unused_parameter</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">unused_context</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Flatten.from_runner_api_parameter"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Flatten.from_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.FlattenWith"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">FlattenWith</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">others</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#FlattenWith"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.FlattenWith" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference internal" href="apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal notranslate"><span class="pre">PTransform</span></code></a></p> |
| <p>A PTransform that flattens its input with other PCollections.</p> |
| <p>This is equivalent to creating a tuple containing both the input and the |
| other PCollection(s), but has the advantage that it can be more easily used |
| inline.</p> |
| <p>Root PTransforms can be passed as well as PCollections, in which case their |
| outputs will be flattened.</p> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.FlattenWith.expand"> |
| <span class="sig-name descname"><span class="pre">expand</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">pcoll</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#FlattenWith.expand"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.FlattenWith.expand" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Create"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">Create</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">values</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">reshuffle</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">True</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Create"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Create" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference internal" href="apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal notranslate"><span class="pre">PTransform</span></code></a></p> |
| <p>A transform that creates a PCollection from an iterable.</p> |
| <p>Initializes a Create transform.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><p><strong>values</strong> – An object of values for the PCollection</p> |
| </dd> |
| </dl> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Create.to_runner_api_parameter"> |
| <span class="sig-name descname"><span class="pre">to_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">context</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="apache_beam.io.iobase.html#apache_beam.io.iobase.Read.PipelineContext" title="apache_beam.io.iobase.Read.PipelineContext"><span class="pre">PipelineContext</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Tuple" title="(in Python v3.13)"><span class="pre">Tuple</span></a><span class="p"><span class="pre">[</span></span><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><span class="pre">str</span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#bytes" title="(in Python v3.13)"><span class="pre">bytes</span></a><span class="p"><span class="pre">]</span></span></span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Create.to_runner_api_parameter"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Create.to_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Create.infer_output_type"> |
| <span class="sig-name descname"><span class="pre">infer_output_type</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">unused_input_type</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Create.infer_output_type"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Create.infer_output_type" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Create.get_output_type"> |
| <span class="sig-name descname"><span class="pre">get_output_type</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Create.get_output_type"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Create.get_output_type" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Create.expand"> |
| <span class="sig-name descname"><span class="pre">expand</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">pbegin</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Create.expand"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Create.expand" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Create.as_read"> |
| <span class="sig-name descname"><span class="pre">as_read</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Create.as_read"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Create.as_read" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Create.get_windowing"> |
| <span class="sig-name descname"><span class="pre">get_windowing</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">unused_inputs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Any" title="(in Python v3.13)"><span class="pre">Any</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="#apache_beam.transforms.core.Windowing" title="apache_beam.transforms.core.Windowing"><span class="pre">Windowing</span></a></span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Create.get_windowing"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Create.get_windowing" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Impulse"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">Impulse</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">label</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><span class="pre">str</span></a><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><span class="pre">None</span></a></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Impulse"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Impulse" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference internal" href="apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform" title="apache_beam.transforms.ptransform.PTransform"><code class="xref py py-class docutils literal notranslate"><span class="pre">PTransform</span></code></a></p> |
| <p>Impulse primitive.</p> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Impulse.expand"> |
| <span class="sig-name descname"><span class="pre">expand</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">pbegin</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Impulse.expand"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Impulse.expand" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Impulse.get_windowing"> |
| <span class="sig-name descname"><span class="pre">get_windowing</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">inputs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Any" title="(in Python v3.13)"><span class="pre">Any</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="#apache_beam.transforms.core.Windowing" title="apache_beam.transforms.core.Windowing"><span class="pre">Windowing</span></a></span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Impulse.get_windowing"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Impulse.get_windowing" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Impulse.infer_output_type"> |
| <span class="sig-name descname"><span class="pre">infer_output_type</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">unused_input_type</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Impulse.infer_output_type"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Impulse.infer_output_type" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Impulse.to_runner_api_parameter"> |
| <span class="sig-name descname"><span class="pre">to_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">unused_context</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="apache_beam.io.iobase.html#apache_beam.io.iobase.Read.PipelineContext" title="apache_beam.io.iobase.Read.PipelineContext"><span class="pre">PipelineContext</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.Tuple" title="(in Python v3.13)"><span class="pre">Tuple</span></a><span class="p"><span class="pre">[</span></span><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str" title="(in Python v3.13)"><span class="pre">str</span></a><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference external" href="https://docs.python.org/3/library/constants.html#None" title="(in Python v3.13)"><span class="pre">None</span></a><span class="p"><span class="pre">]</span></span></span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Impulse.to_runner_api_parameter"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Impulse.to_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.Impulse.from_runner_api_parameter"> |
| <em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_runner_api_parameter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">unused_ptransform</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">unused_parameter</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">unused_context</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#Impulse.from_runner_api_parameter"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.Impulse.from_runner_api_parameter" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.RestrictionProvider"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">RestrictionProvider</span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#RestrictionProvider"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.RestrictionProvider" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference external" href="https://docs.python.org/3/library/functions.html#object" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate"><span class="pre">object</span></code></a></p> |
| <p>Provides methods for generating and manipulating restrictions.</p> |
| <p>This class should be implemented to support Splittable <code class="docutils literal notranslate"><span class="pre">DoFn</span></code> in Python |
| SDK. See <a class="reference external" href="https://s.apache.org/splittable-do-fn">https://s.apache.org/splittable-do-fn</a> for more details about |
| Splittable <code class="docutils literal notranslate"><span class="pre">DoFn</span></code>.</p> |
| <p>To denote a <code class="docutils literal notranslate"><span class="pre">DoFn</span></code> class to be Splittable <code class="docutils literal notranslate"><span class="pre">DoFn</span></code>, <code class="docutils literal notranslate"><span class="pre">DoFn.process()</span></code> |
| method of that class should have exactly one parameter whose default value is |
| an instance of <code class="docutils literal notranslate"><span class="pre">RestrictionParam</span></code>. This <code class="docutils literal notranslate"><span class="pre">RestrictionParam</span></code> can either be |
| constructed with an explicit <code class="docutils literal notranslate"><span class="pre">RestrictionProvider</span></code>, or, if no |
| <code class="docutils literal notranslate"><span class="pre">RestrictionProvider</span></code> is provided, the <code class="docutils literal notranslate"><span class="pre">DoFn</span></code> itself must be a |
| <code class="docutils literal notranslate"><span class="pre">RestrictionProvider</span></code>.</p> |
| <p>The provided <code class="docutils literal notranslate"><span class="pre">RestrictionProvider</span></code> instance must provide suitable overrides |
| for the following methods: |
| * create_tracker() |
| * initial_restriction() |
| * restriction_size()</p> |
| <p>Optionally, <code class="docutils literal notranslate"><span class="pre">RestrictionProvider</span></code> may override default implementations of |
| following methods: |
| * restriction_coder() |
| * split() |
| * split_and_size() |
| * truncate()</p> |
| <p>** Pausing and resuming processing of an element **</p> |
| <p>As the last element produced by the iterator returned by the |
| <code class="docutils literal notranslate"><span class="pre">DoFn.process()</span></code> method, a Splittable <code class="docutils literal notranslate"><span class="pre">DoFn</span></code> may return an object of type |
| <code class="docutils literal notranslate"><span class="pre">ProcessContinuation</span></code>.</p> |
| <p>If restriction_tracker.defer_remander is called in the <code class="docutils literal notranslate"><span class="pre">`DoFn.process()</span></code>, it |
| means that runner should later re-invoke <code class="docutils literal notranslate"><span class="pre">DoFn.process()</span></code> method to resume |
| processing the current element and the manner in which the re-invocation |
| should be performed.</p> |
| <p>** Updating output watermark **</p> |
| <p><code class="docutils literal notranslate"><span class="pre">DoFn.process()</span></code> method of Splittable <code class="docutils literal notranslate"><span class="pre">DoFn``s</span> <span class="pre">could</span> <span class="pre">contain</span> <span class="pre">a</span> <span class="pre">parameter</span> |
| <span class="pre">with</span> <span class="pre">default</span> <span class="pre">value</span> <span class="pre">``DoFn.WatermarkReporterParam</span></code>. If specified this asks the |
| runner to provide a function that can be used to give the runner a |
| (best-effort) lower bound about the timestamps of future output associated |
| with the current element processed by the <code class="docutils literal notranslate"><span class="pre">DoFn</span></code>. If the <code class="docutils literal notranslate"><span class="pre">DoFn</span></code> has |
| multiple outputs, the watermark applies to all of them. Provided function must |
| be invoked with a single parameter of type <code class="docutils literal notranslate"><span class="pre">Timestamp</span></code> or as an integer that |
| gives the watermark in number of seconds.</p> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.RestrictionProvider.create_tracker"> |
| <span class="sig-name descname"><span class="pre">create_tracker</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">restriction</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="apache_beam.io.iobase.html#apache_beam.io.iobase.RestrictionTracker" title="apache_beam.io.iobase.RestrictionTracker"><span class="pre">iobase.RestrictionTracker</span></a></span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#RestrictionProvider.create_tracker"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.RestrictionProvider.create_tracker" title="Link to this definition"></a></dt> |
| <dd><p>Produces a new <code class="docutils literal notranslate"><span class="pre">RestrictionTracker</span></code> for the given restriction.</p> |
| <p>This API is required to be implemented.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Parameters<span class="colon">:</span></dt> |
| <dd class="field-odd"><p><strong>restriction</strong> – an object that defines a restriction as identified by a |
| Splittable <code class="docutils literal notranslate"><span class="pre">DoFn</span></code> that utilizes the current <code class="docutils literal notranslate"><span class="pre">RestrictionProvider</span></code>. |
| For example, a tuple that gives a range of positions for a Splittable |
| <code class="docutils literal notranslate"><span class="pre">DoFn</span></code> that reads files based on byte positions.</p> |
| </dd> |
| </dl> |
| <p>Returns: an object of type <code class="docutils literal notranslate"><span class="pre">RestrictionTracker</span></code>.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.RestrictionProvider.initial_restriction"> |
| <span class="sig-name descname"><span class="pre">initial_restriction</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">element</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#RestrictionProvider.initial_restriction"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.RestrictionProvider.initial_restriction" title="Link to this definition"></a></dt> |
| <dd><p>Produces an initial restriction for the given element.</p> |
| <p>This API is required to be implemented.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.RestrictionProvider.split"> |
| <span class="sig-name descname"><span class="pre">split</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">element</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">restriction</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#RestrictionProvider.split"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.RestrictionProvider.split" title="Link to this definition"></a></dt> |
| <dd><p>Splits the given element and restriction initially.</p> |
| <p>This method enables runners to perform bulk splitting initially allowing for |
| a rapid increase in parallelism. Note that initial split is a different |
| concept from the split during element processing time. Please refer to |
| <code class="docutils literal notranslate"><span class="pre">iobase.RestrictionTracker.try_split</span></code> for details about splitting when the |
| current element and restriction are actively being processed.</p> |
| <p>Returns an iterator of restrictions. The total set of elements produced by |
| reading input element for each of the returned restrictions should be the |
| same as the total set of elements produced by reading the input element for |
| the input restriction.</p> |
| <p>This API is optional if <code class="docutils literal notranslate"><span class="pre">split_and_size</span></code> has been implemented.</p> |
| <p>If this method is not override, there is no initial splitting happening on |
| each restriction.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.RestrictionProvider.restriction_coder"> |
| <span class="sig-name descname"><span class="pre">restriction_coder</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#RestrictionProvider.restriction_coder"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.RestrictionProvider.restriction_coder" title="Link to this definition"></a></dt> |
| <dd><p>Returns a <code class="docutils literal notranslate"><span class="pre">Coder</span></code> for restrictions.</p> |
| <p>Returned``Coder`` will be used for the restrictions produced by the current |
| <code class="docutils literal notranslate"><span class="pre">RestrictionProvider</span></code>.</p> |
| <dl class="field-list simple"> |
| <dt class="field-odd">Returns<span class="colon">:</span></dt> |
| <dd class="field-odd"><p>an object of type <code class="docutils literal notranslate"><span class="pre">Coder</span></code>.</p> |
| </dd> |
| </dl> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.RestrictionProvider.restriction_size"> |
| <span class="sig-name descname"><span class="pre">restriction_size</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">element</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">restriction</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#RestrictionProvider.restriction_size"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.RestrictionProvider.restriction_size" title="Link to this definition"></a></dt> |
| <dd><p>Returns the size of a restriction with respect to the given element.</p> |
| <p>By default, asks a newly-created restriction tracker for the default size |
| of the restriction.</p> |
| <p>The return value must be non-negative.</p> |
| <p>Must be thread safe. Will be invoked concurrently during bundle processing |
| due to runner initiated splitting and progress estimation.</p> |
| <p>This API is required to be implemented.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.RestrictionProvider.split_and_size"> |
| <span class="sig-name descname"><span class="pre">split_and_size</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">element</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">restriction</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#RestrictionProvider.split_and_size"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.RestrictionProvider.split_and_size" title="Link to this definition"></a></dt> |
| <dd><p>Like split, but also does sizing, returning (restriction, size) pairs.</p> |
| <p>For each pair, size must be non-negative.</p> |
| <p>This API is optional if <code class="docutils literal notranslate"><span class="pre">split</span></code> and <code class="docutils literal notranslate"><span class="pre">restriction_size</span></code> have been |
| implemented.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.RestrictionProvider.truncate"> |
| <span class="sig-name descname"><span class="pre">truncate</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">element</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">restriction</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#RestrictionProvider.truncate"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.RestrictionProvider.truncate" title="Link to this definition"></a></dt> |
| <dd><p>Truncates the provided restriction into a restriction representing a |
| finite amount of work when the pipeline is |
| <cite>draining <https://docs.google.com/document/d/1NExwHlj-2q2WUGhSO4jTu8XGhDPmm3cllSN8IMmWci8/edit#> for additional details about drain.>_</cite>. # pylint: disable=line-too-long |
| By default, if the restriction is bounded then the restriction will be |
| returned otherwise None will be returned.</p> |
| <p>This API is optional and should only be implemented if more granularity is |
| required.</p> |
| <p>Return a truncated finite restriction if further processing is required |
| otherwise return None to represent that no further processing of this |
| restriction is required.</p> |
| <p>The default behavior when a pipeline is being drained is that bounded |
| restrictions process entirely while unbounded restrictions process till a |
| checkpoint is possible.</p> |
| </dd></dl> |
| |
| </dd></dl> |
| |
| <dl class="py class"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.WatermarkEstimatorProvider"> |
| <em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">apache_beam.transforms.core.</span></span><span class="sig-name descname"><span class="pre">WatermarkEstimatorProvider</span></span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WatermarkEstimatorProvider"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.WatermarkEstimatorProvider" title="Link to this definition"></a></dt> |
| <dd><p>Bases: <a class="reference external" href="https://docs.python.org/3/library/functions.html#object" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate"><span class="pre">object</span></code></a></p> |
| <p>Provides methods for generating WatermarkEstimator.</p> |
| <p>This class should be implemented if wanting to providing output_watermark |
| information within an SDF.</p> |
| <p>In order to make an SDF.process() access to the typical WatermarkEstimator, |
| the SDF author should have an argument whose default value is a |
| DoFn.WatermarkEstimatorParam instance. This DoFn.WatermarkEstimatorParam |
| can either be constructed with an explicit WatermarkEstimatorProvider, |
| or, if no WatermarkEstimatorProvider is provided, the DoFn itself must |
| be a WatermarkEstimatorProvider.</p> |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.WatermarkEstimatorProvider.initial_estimator_state"> |
| <span class="sig-name descname"><span class="pre">initial_estimator_state</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">element</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">restriction</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WatermarkEstimatorProvider.initial_estimator_state"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.WatermarkEstimatorProvider.initial_estimator_state" title="Link to this definition"></a></dt> |
| <dd><p>Returns the initial state of the WatermarkEstimator with given element |
| and restriction. |
| This function is called by the system.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.WatermarkEstimatorProvider.create_watermark_estimator"> |
| <span class="sig-name descname"><span class="pre">create_watermark_estimator</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">estimator_state</span></span></em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WatermarkEstimatorProvider.create_watermark_estimator"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.WatermarkEstimatorProvider.create_watermark_estimator" title="Link to this definition"></a></dt> |
| <dd><p>Create a new WatermarkEstimator based on the state. The state is |
| typically useful when resuming processing an element.</p> |
| </dd></dl> |
| |
| <dl class="py method"> |
| <dt class="sig sig-object py" id="apache_beam.transforms.core.WatermarkEstimatorProvider.estimator_state_coder"> |
| <span class="sig-name descname"><span class="pre">estimator_state_coder</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/transforms/core.html#WatermarkEstimatorProvider.estimator_state_coder"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#apache_beam.transforms.core.WatermarkEstimatorProvider.estimator_state_coder" title="Link to this definition"></a></dt> |
| <dd></dd></dl> |
| |
| </dd></dl> |
| |
| </section> |
| |
| |
| </div> |
| </div> |
| <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer"> |
| <a href="apache_beam.transforms.combiners.html" class="btn btn-neutral float-left" title="apache_beam.transforms.combiners module" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a> |
| <a href="apache_beam.transforms.create_source.html" class="btn btn-neutral float-right" title="apache_beam.transforms.create_source module" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a> |
| </div> |
| |
| <hr/> |
| |
| <div role="contentinfo"> |
| <p>© Copyright %Y, Apache Beam.</p> |
| </div> |
| |
| Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a |
| <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a> |
| provided by <a href="https://readthedocs.org">Read the Docs</a>. |
| |
| |
| </footer> |
| </div> |
| </div> |
| </section> |
| </div> |
| <script> |
| jQuery(function () { |
| SphinxRtdTheme.Navigation.enable(true); |
| }); |
| </script> |
| |
| </body> |
| </html> |
