blob: a1307bd04d2fec38ac125697d6390f2d7152eb6b [file] [log] [blame]
<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>apache_beam.runners.interactive.caching.read_cache module &mdash; Apache Beam 2.38.0 documentation</title>
<script type="text/javascript" src="_static/js/modernizr.min.js"></script>
<script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<script type="text/javascript" src="_static/language_data.js"></script>
<script async="async" type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script type="text/javascript" src="_static/js/theme.js"></script>
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="apache_beam.runners.interactive.caching.reify module" href="apache_beam.runners.interactive.caching.reify.html" />
<link rel="prev" title="apache_beam.runners.interactive.caching.expression_cache module" href="apache_beam.runners.interactive.caching.expression_cache.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="index.html" class="icon icon-home"> Apache Beam
</a>
<div class="version">
2.38.0
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="apache_beam.coders.html">apache_beam.coders package</a></li>
<li class="toctree-l1"><a class="reference internal" href="apache_beam.dataframe.html">apache_beam.dataframe package</a></li>
<li class="toctree-l1"><a class="reference internal" href="apache_beam.io.html">apache_beam.io package</a></li>
<li class="toctree-l1"><a class="reference internal" href="apache_beam.metrics.html">apache_beam.metrics package</a></li>
<li class="toctree-l1"><a class="reference internal" href="apache_beam.ml.html">apache_beam.ml package</a></li>
<li class="toctree-l1"><a class="reference internal" href="apache_beam.options.html">apache_beam.options package</a></li>
<li class="toctree-l1"><a class="reference internal" href="apache_beam.portability.html">apache_beam.portability package</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="apache_beam.runners.html">apache_beam.runners package</a><ul class="current">
<li class="toctree-l2 current"><a class="reference internal" href="apache_beam.runners.html#subpackages">Subpackages</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="apache_beam.runners.dataflow.html">apache_beam.runners.dataflow package</a></li>
<li class="toctree-l3"><a class="reference internal" href="apache_beam.runners.direct.html">apache_beam.runners.direct package</a></li>
<li class="toctree-l3 current"><a class="reference internal" href="apache_beam.runners.interactive.html">apache_beam.runners.interactive package</a><ul class="current">
<li class="toctree-l4 current"><a class="reference internal" href="apache_beam.runners.interactive.html#subpackages">Subpackages</a></li>
<li class="toctree-l4"><a class="reference internal" href="apache_beam.runners.interactive.html#submodules">Submodules</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="apache_beam.runners.job.html">apache_beam.runners.job package</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="apache_beam.runners.html#submodules">Submodules</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="apache_beam.transforms.html">apache_beam.transforms package</a></li>
<li class="toctree-l1"><a class="reference internal" href="apache_beam.typehints.html">apache_beam.typehints package</a></li>
<li class="toctree-l1"><a class="reference internal" href="apache_beam.utils.html">apache_beam.utils package</a></li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="apache_beam.error.html">apache_beam.error module</a></li>
<li class="toctree-l1"><a class="reference internal" href="apache_beam.pipeline.html">apache_beam.pipeline module</a></li>
<li class="toctree-l1"><a class="reference internal" href="apache_beam.pvalue.html">apache_beam.pvalue module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
<nav class="wy-nav-top" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">Apache Beam</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="index.html">Docs</a> &raquo;</li>
<li><a href="apache_beam.runners.html">apache_beam.runners package</a> &raquo;</li>
<li><a href="apache_beam.runners.interactive.html">apache_beam.runners.interactive package</a> &raquo;</li>
<li><a href="apache_beam.runners.interactive.caching.html">apache_beam.runners.interactive.caching package</a> &raquo;</li>
<li>apache_beam.runners.interactive.caching.read_cache module</li>
<li class="wy-breadcrumbs-aside">
<a href="_sources/apache_beam.runners.interactive.caching.read_cache.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<div class="section" id="module-apache_beam.runners.interactive.caching.read_cache">
<span id="apache-beam-runners-interactive-caching-read-cache-module"></span><h1>apache_beam.runners.interactive.caching.read_cache module<a class="headerlink" href="#module-apache_beam.runners.interactive.caching.read_cache" title="Permalink to this headline"></a></h1>
<p>Module to read cache of computed PCollections.</p>
<p>For internal use only; no backward-compatibility guarantees.</p>
<dl class="class">
<dt id="apache_beam.runners.interactive.caching.read_cache.ReadCache">
<em class="property">class </em><code class="descclassname">apache_beam.runners.interactive.caching.read_cache.</code><code class="descname">ReadCache</code><span class="sig-paren">(</span><em>pipeline: beam_runner_api_pb2.Pipeline</em>, <em>context: apache_beam.runners.pipeline_context.PipelineContext</em>, <em>cache_manager: apache_beam.runners.interactive.cache_manager.CacheManager</em>, <em>cacheable: apache_beam.runners.interactive.caching.cacheable.Cacheable</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/apache_beam/runners/interactive/caching/read_cache.html#ReadCache"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.runners.interactive.caching.read_cache.ReadCache" title="Permalink to this definition"></a></dt>
<dd><p>Bases: <a class="reference external" href="https://docs.python.org/3/library/functions.html#object" title="(in Python v3.10)"><code class="xref py py-class docutils literal notranslate"><span class="pre">object</span></code></a></p>
<p>Class that facilitates reading cache of computed PCollections.</p>
<dl class="method">
<dt id="apache_beam.runners.interactive.caching.read_cache.ReadCache.read_cache">
<code class="descname">read_cache</code><span class="sig-paren">(</span><span class="sig-paren">)</span> &#x2192; Tuple[str, str]<a class="reference internal" href="_modules/apache_beam/runners/interactive/caching/read_cache.html#ReadCache.read_cache"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#apache_beam.runners.interactive.caching.read_cache.ReadCache.read_cache" title="Permalink to this definition"></a></dt>
<dd><p>Reads cache of the cacheable PCollection and wires the cache into the
pipeline proto. Returns the pipeline-scoped ids of the cacheable PCollection
and the cache reading output PCollection that replaces it.</p>
<p>First, it creates a temporary pipeline instance on top of the existing
component_id_map from the self._pipeline’s context so that both pipelines
share the context and have no conflict component ids.
Second, it instantiates a _ReadCacheTransform to build the temporary
pipeline with a subgraph under top level transforms that reads the cache of
a cacheable PCollection.
Third, it copies components of the subgraph from the temporary pipeline to
self._pipeline, skipping components that are not in the temporary pipeline
but presents in the component_id_map of self._pipeline. Since to_runner_api
generates components for all entries in the component_id_map, those
component ids from the context shared by self._pipeline need to be ignored.
Last, it replaces inputs of all transforms that consume the cacheable
PCollection with the output PCollection of the _ReadCacheTransform so that
the whole pipeline computes with data from the cache. The pipeline
fragment of reading the cacheable PCollection is now disconnected from the
rest of the pipeline and can be pruned later.</p>
</dd></dl>
</dd></dl>
</div>
</div>
</div>
<footer>
<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
<a href="apache_beam.runners.interactive.caching.reify.html" class="btn btn-neutral float-right" title="apache_beam.runners.interactive.caching.reify module" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
<a href="apache_beam.runners.interactive.caching.expression_cache.html" class="btn btn-neutral float-left" title="apache_beam.runners.interactive.caching.expression_cache module" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
</div>
<hr/>
<div role="contentinfo">
<p>
&copy; Copyright
</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>