Google Git
Sign in
apache / mxnet-site / refs/heads/test-revert / . / api / python / kvstore / kvstore.html
blob: 990bfc4f8b5850b084401767c7294a54d695c67c [file] [log] [blame]
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<meta content="IE=edge" http-equiv="X-UA-Compatible"/>
<meta content="width=device-width, initial-scale=1" name="viewport"/>
<title>KVStore API — mxnet documentation</title>
<link crossorigin="anonymous" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css" integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7" rel="stylesheet"/>
<link href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css" rel="stylesheet"/>
<link href="../../../_static/basic.css" rel="stylesheet" type="text/css">
<link href="../../../_static/pygments.css" rel="stylesheet" type="text/css">
<link href="../../../_static/mxnet.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../../../',
VERSION: '',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: ''
};
</script>
<script src="https://code.jquery.com/jquery-1.11.1.min.js" type="text/javascript"></script>
<script src="../../../_static/underscore.js" type="text/javascript"></script>
<script src="../../../_static/searchtools_custom.js" type="text/javascript"></script>
<script src="../../../_static/doctools.js" type="text/javascript"></script>
<script src="../../../_static/selectlang.js" type="text/javascript"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script>
<script type="text/javascript"> jQuery(function() { Search.loadIndex("/searchindex.js"); Search.init();}); </script>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new
Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-96378503-1', 'auto');
ga('send', 'pageview');
</script>
<!-- -->
<!-- <script type="text/javascript" src="../../../_static/jquery.js"></script> -->
<!-- -->
<!-- <script type="text/javascript" src="../../../_static/underscore.js"></script> -->
<!-- -->
<!-- <script type="text/javascript" src="../../../_static/doctools.js"></script> -->
<!-- -->
<!-- <script type="text/javascript" src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script> -->
<!-- -->
<link href="../index.html" rel="up" title="MXNet - Python API">
<link href="../io/io.html" rel="next" title="Data Loading API"/>
<link href="../gluon/contrib.html" rel="prev" title="Gluon Contrib API"/>
<link href="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet-icon.png" rel="icon" type="image/png"/>
</link></link></link></head>
<body background="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet-background-compressed.jpeg" role="document">
<div class="content-block"><div class="navbar navbar-fixed-top">
<div class="container" id="navContainer">
<div class="innder" id="header-inner">
<h1 id="logo-wrap">
<a href="../../../" id="logo"><img src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet_logo.png"/></a>
</h1>
<nav class="nav-bar" id="main-nav">
<a class="main-nav-link" href="../../../install/index.html">Install</a>
<a class="main-nav-link" href="../../../tutorials/index.html">Tutorials</a>
<span id="dropdown-menu-position-anchor">
<a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Gluon <span class="caret"></span></a>
<ul class="dropdown-menu navbar-menu" id="package-dropdown-menu">
<li><a class="main-nav-link" href="../../../gluon/index.html">About</a></li>
<li><a class="main-nav-link" href="http://gluon.mxnet.io">Tutorials</a></li>
</ul>
</span>
<span id="dropdown-menu-position-anchor">
<a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">API <span class="caret"></span></a>
<ul class="dropdown-menu navbar-menu" id="package-dropdown-menu">
<li><a class="main-nav-link" href="../../../api/python/index.html">Python</a></li>
<li><a class="main-nav-link" href="../../../api/scala/index.html">Scala</a></li>
<li><a class="main-nav-link" href="../../../api/r/index.html">R</a></li>
<li><a class="main-nav-link" href="../../../api/julia/index.html">Julia</a></li>
<li><a class="main-nav-link" href="../../../api/c++/index.html">C++</a></li>
<li><a class="main-nav-link" href="../../../api/perl/index.html">Perl</a></li>
</ul>
</span>
<span id="dropdown-menu-position-anchor-docs">
<a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Docs <span class="caret"></span></a>
<ul class="dropdown-menu navbar-menu" id="package-dropdown-menu-docs">
<li><a class="main-nav-link" href="../../../faq/index.html">FAQ</a></li>
<li><a class="main-nav-link" href="../../../architecture/index.html">Architecture</a></li>
<li><a class="main-nav-link" href="https://github.com/apache/incubator-mxnet/tree/1.0.0/example">Examples</a></li>
<li><a class="main-nav-link" href="../../../model_zoo/index.html">Model Zoo</a></li>
</ul>
</span>
<a class="main-nav-link" href="https://github.com/dmlc/mxnet">Github</a>
<span id="dropdown-menu-position-anchor-community">
<a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Community <span class="caret"></span></a>
<ul class="dropdown-menu navbar-menu" id="package-dropdown-menu-community">
<li><a class="main-nav-link" href="../../../community/index.html">Community</a></li>
<li><a class="main-nav-link" href="../../../community/contribute.html">Contribute</a></li>
<li><a class="main-nav-link" href="../../../community/powered_by.html">Powered By</a></li>
</ul>
</span>
<a class="main-nav-link" href="http://discuss.mxnet.io">Discuss</a>
<span id="dropdown-menu-position-anchor-version" style="position: relative"><a href="#" class="main-nav-link dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="true">Versions(1.0.0)<span class="caret"></span></a><ul id="package-dropdown-menu" class="dropdown-menu"><li><a class="main-nav-link" href=https://mxnet.incubator.apache.org/>1.0.0</a></li><li><a class="main-nav-link" href=https://mxnet.incubator.apache.org/versions/0.12.1/index.html>0.12.1</a></li><li><a class="main-nav-link" href=https://mxnet.incubator.apache.org/versions/0.12.0/index.html>0.12.0</a></li><li><a class="main-nav-link" href=https://mxnet.incubator.apache.org/versions/0.11.0/index.html>0.11.0</a></li><li><a class="main-nav-link" href=https://mxnet.incubator.apache.org/versions/master/index.html>master</a></li></ul></span></nav>
<script> function getRootPath(){ return "../../../" } </script>
<div class="burgerIcon dropdown">
<a class="dropdown-toggle" data-toggle="dropdown" href="#" role="button">☰</a>
<ul class="dropdown-menu" id="burgerMenu">
<li><a href="../../../install/index.html">Install</a></li>
<li><a class="main-nav-link" href="../../../tutorials/index.html">Tutorials</a></li>
<li class="dropdown-submenu">
<a href="#" tabindex="-1">Community</a>
<ul class="dropdown-menu">
<li><a href="../../../community/index.html" tabindex="-1">Community</a></li>
<li><a href="../../../community/contribute.html" tabindex="-1">Contribute</a></li>
<li><a href="../../../community/powered_by.html" tabindex="-1">Powered By</a></li>
</ul>
</li>
<li class="dropdown-submenu">
<a href="#" tabindex="-1">API</a>
<ul class="dropdown-menu">
<li><a href="../../../api/python/index.html" tabindex="-1">Python</a>
</li>
<li><a href="../../../api/scala/index.html" tabindex="-1">Scala</a>
</li>
<li><a href="../../../api/r/index.html" tabindex="-1">R</a>
</li>
<li><a href="../../../api/julia/index.html" tabindex="-1">Julia</a>
</li>
<li><a href="../../../api/c++/index.html" tabindex="-1">C++</a>
</li>
<li><a href="../../../api/perl/index.html" tabindex="-1">Perl</a>
</li>
</ul>
</li>
<li class="dropdown-submenu">
<a href="#" tabindex="-1">Docs</a>
<ul class="dropdown-menu">
<li><a href="../../../tutorials/index.html" tabindex="-1">Tutorials</a></li>
<li><a href="../../../faq/index.html" tabindex="-1">FAQ</a></li>
<li><a href="../../../architecture/index.html" tabindex="-1">Architecture</a></li>
<li><a href="https://github.com/apache/incubator-mxnet/tree/1.0.0/example" tabindex="-1">Examples</a></li>
<li><a href="../../../model_zoo/index.html" tabindex="-1">Model Zoo</a></li>
</ul>
</li>
<li><a href="../../../architecture/index.html">Architecture</a></li>
<li><a class="main-nav-link" href="https://github.com/dmlc/mxnet">Github</a></li>
<li id="dropdown-menu-position-anchor-version-mobile" class="dropdown-submenu" style="position: relative"><a href="#" tabindex="-1">Versions(1.0.0)</a><ul class="dropdown-menu"><li><a tabindex="-1" href=https://mxnet.incubator.apache.org/>1.0.0</a></li><li><a tabindex="-1" href=https://mxnet.incubator.apache.org/versions/0.12.1/index.html>0.12.1</a></li><li><a tabindex="-1" href=https://mxnet.incubator.apache.org/versions/0.12.0/index.html>0.12.0</a></li><li><a tabindex="-1" href=https://mxnet.incubator.apache.org/versions/0.11.0/index.html>0.11.0</a></li><li><a tabindex="-1" href=https://mxnet.incubator.apache.org/versions/master/index.html>master</a></li></ul></li></ul>
</div>
<div class="plusIcon dropdown">
<a class="dropdown-toggle" data-toggle="dropdown" href="#" role="button"><span aria-hidden="true" class="glyphicon glyphicon-plus"></span></a>
<ul class="dropdown-menu dropdown-menu-right" id="plusMenu"></ul>
</div>
<div id="search-input-wrap">
<form action="../../../search.html" autocomplete="off" class="" method="get" role="search">
<div class="form-group inner-addon left-addon">
<i class="glyphicon glyphicon-search"></i>
<input class="form-control" name="q" placeholder="Search" type="text"/>
</div>
<input name="check_keywords" type="hidden" value="yes">
<input name="area" type="hidden" value="default"/>
</input></form>
<div id="search-preview"></div>
</div>
<div id="searchIcon">
<span aria-hidden="true" class="glyphicon glyphicon-search"></span>
</div>
<!-- <div id="lang-select-wrap"> -->
<!-- <label id="lang-select-label"> -->
<!-- <\!-- <i class="fa fa-globe"></i> -\-> -->
<!-- <span></span> -->
<!-- </label> -->
<!-- <select id="lang-select"> -->
<!-- <option value="en">Eng</option> -->
<!-- <option value="zh">中文</option> -->
<!-- </select> -->
<!-- </div> -->
<!-- <a id="mobile-nav-toggle">
<span class="mobile-nav-toggle-bar"></span>
<span class="mobile-nav-toggle-bar"></span>
<span class="mobile-nav-toggle-bar"></span>
</a> -->
</div>
</div>
</div>
<script type="text/javascript">
$('body').css('background', 'white');
</script>
<div class="container">
<div class="row">
<div aria-label="main navigation" class="sphinxsidebar leftsidebar" role="navigation">
<div class="sphinxsidebarwrapper">
<ul class="current">
<li class="toctree-l1 current"><a class="reference internal" href="../index.html">Python Documents</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../index.html#ndarray-api">NDArray API</a></li>
<li class="toctree-l2"><a class="reference internal" href="../index.html#symbol-api">Symbol API</a></li>
<li class="toctree-l2"><a class="reference internal" href="../index.html#module-api">Module API</a></li>
<li class="toctree-l2"><a class="reference internal" href="../index.html#autograd-api">Autograd API</a></li>
<li class="toctree-l2"><a class="reference internal" href="../index.html#gluon-api">Gluon API</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../index.html#kvstore-api">KVStore API</a><ul class="current">
<li class="toctree-l3 current"><a class="current reference internal" href="">KVStore API</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#basic-push-and-pull">Basic Push and Pull</a></li>
<li class="toctree-l4"><a class="reference internal" href="#list-key-value-pairs">List Key-Value Pairs</a></li>
<li class="toctree-l4"><a class="reference internal" href="#api-reference">API Reference</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../index.html#io-api">IO API</a></li>
<li class="toctree-l2"><a class="reference internal" href="../index.html#image-api">Image API</a></li>
<li class="toctree-l2"><a class="reference internal" href="../index.html#optimization-api">Optimization API</a></li>
<li class="toctree-l2"><a class="reference internal" href="../index.html#callback-api">Callback API</a></li>
<li class="toctree-l2"><a class="reference internal" href="../index.html#metric-api">Metric API</a></li>
<li class="toctree-l2"><a class="reference internal" href="../index.html#run-time-compilation-api">Run-Time Compilation API</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../r/index.html">R Documents</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../julia/index.html">Julia Documents</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../c++/index.html">C++ Documents</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../scala/index.html">Scala Documents</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../perl/index.html">Perl Documents</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../faq/index.html">HowTo Documents</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../architecture/index.html">System Documents</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../tutorials/index.html">Tutorials</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../community/index.html">Community</a></li>
</ul>
</div>
</div>
<div class="content">
<div class="page-tracker"></div>
<div class="section" id="kvstore-api">
<span id="kvstore-api"></span><h1>KVStore API<a class="headerlink" href="#kvstore-api" title="Permalink to this headline"></a></h1>
<div class="section" id="basic-push-and-pull">
<span id="basic-push-and-pull"></span><h2>Basic Push and Pull<a class="headerlink" href="#basic-push-and-pull" title="Permalink to this headline"></a></h2>
<p>Provides basic operation over multiple devices (GPUs) on a single device.</p>
<div class="section" id="initialization">
<span id="initialization"></span><h3>Initialization<a class="headerlink" href="#initialization" title="Permalink to this headline"></a></h3>
<p>Let’s consider a simple example. It initializes
a (<code class="docutils literal"><span class="pre">int</span></code>, <code class="docutils literal"><span class="pre">NDArray</span></code>) pair into the store, and then pulls the value out.</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">kv</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">kv</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="s1">'local'</span><span class="p">)</span> <span class="c1"># create a local kv store.</span>
<span class="gp">>>> </span><span class="n">shape</span> <span class="o">=</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">)</span><span class="o">*</span><span class="mi">2</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">a</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">shape</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">out</span> <span class="o">=</span> <span class="n">a</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">a</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 2. 2. 2.]</span>
<span class="go"> [ 2. 2. 2.]]</span>
</pre></div>
</div>
</div>
<div class="section" id="push-aggregation-and-updater">
<span id="push-aggregation-and-updater"></span><h3>Push, Aggregation, and Updater<a class="headerlink" href="#push-aggregation-and-updater" title="Permalink to this headline"></a></h3>
<p>For any key that’s been initialized, you can push a new value with the same shape to the key, as follows:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">)</span><span class="o">*</span><span class="mi">8</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">out</span> <span class="o">=</span> <span class="n">a</span><span class="p">)</span> <span class="c1"># pull out the value</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">a</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 8. 8. 8.]</span>
<span class="go"> [ 8. 8. 8.]]</span>
</pre></div>
</div>
<p>The data that you want to push can be stored on any device. Furthermore, you can push multiple
values into the same key, where KVStore first sums all of these
values, and then pushes the aggregated value, as follows:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">gpus</span> <span class="o">=</span> <span class="p">[</span><span class="n">mx</span><span class="o">.</span><span class="n">gpu</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">4</span><span class="p">)]</span>
<span class="gp">>>> </span><span class="n">b</span> <span class="o">=</span> <span class="p">[</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">,</span> <span class="n">gpu</span><span class="p">)</span> <span class="k">for</span> <span class="n">gpu</span> <span class="ow">in</span> <span class="n">gpus</span><span class="p">]</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">out</span> <span class="o">=</span> <span class="n">a</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">a</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 4. 4. 4.]</span>
<span class="go"> [ 4. 4. 4.]]</span>
</pre></div>
</div>
<p>For each push command, KVStore applies the pushed value to the value stored by an
<code class="docutils literal"><span class="pre">updater</span></code>. The default updater is <code class="docutils literal"><span class="pre">ASSIGN</span></code>. You can replace the default to
control how data is merged.</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="nb">input</span><span class="p">,</span> <span class="n">stored</span><span class="p">):</span>
<span class="gp">>>> </span> <span class="k">print</span> <span class="s2">"update on key: </span><span class="si">%d</span><span class="s2">"</span> <span class="o">%</span> <span class="n">key</span>
<span class="gp">>>> </span> <span class="n">stored</span> <span class="o">+=</span> <span class="nb">input</span> <span class="o">*</span> <span class="mi">2</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">_set_updater</span><span class="p">(</span><span class="n">update</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">a</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">a</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 4. 4. 4.]</span>
<span class="go"> [ 4. 4. 4.]]</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">))</span>
<span class="go">update on key: 3</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">a</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">a</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 6. 6. 6.]</span>
<span class="go"> [ 6. 6. 6.]]</span>
</pre></div>
</div>
</div>
<div class="section" id="pull">
<span id="pull"></span><h3>Pull<a class="headerlink" href="#pull" title="Permalink to this headline"></a></h3>
<p>You’ve already seen how to pull a single key-value pair. Similar to the way that you use the push command, you can
pull the value into several devices with a single call.</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">b</span> <span class="o">=</span> <span class="p">[</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">,</span> <span class="n">gpu</span><span class="p">)</span> <span class="k">for</span> <span class="n">gpu</span> <span class="ow">in</span> <span class="n">gpus</span><span class="p">]</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">out</span> <span class="o">=</span> <span class="n">b</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">b</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 6. 6. 6.]</span>
<span class="go"> [ 6. 6. 6.]]</span>
</pre></div>
</div>
</div>
</div>
<div class="section" id="list-key-value-pairs">
<span id="list-key-value-pairs"></span><h2>List Key-Value Pairs<a class="headerlink" href="#list-key-value-pairs" title="Permalink to this headline"></a></h2>
<p>All of the operations that we’ve discussed so far are performed on a single key. KVStore also provides
the interface for generating a list of key-value pairs. For a single device, use the following:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">keys</span> <span class="o">=</span> <span class="p">[</span><span class="mi">5</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">9</span><span class="p">]</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="p">[</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">)]</span><span class="o">*</span><span class="nb">len</span><span class="p">(</span><span class="n">keys</span><span class="p">))</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="p">[</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">)]</span><span class="o">*</span><span class="nb">len</span><span class="p">(</span><span class="n">keys</span><span class="p">))</span>
<span class="go">update on key: 5</span>
<span class="go">update on key: 7</span>
<span class="go">update on key: 9</span>
<span class="gp">>>> </span><span class="n">b</span> <span class="o">=</span> <span class="p">[</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">shape</span><span class="p">)]</span><span class="o">*</span><span class="nb">len</span><span class="p">(</span><span class="n">keys</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">out</span> <span class="o">=</span> <span class="n">b</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">b</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 3. 3. 3.]</span>
<span class="go"> [ 3. 3. 3.]]</span>
</pre></div>
</div>
<p>For multiple devices:</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">b</span> <span class="o">=</span> <span class="p">[[</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">,</span> <span class="n">gpu</span><span class="p">)</span> <span class="k">for</span> <span class="n">gpu</span> <span class="ow">in</span> <span class="n">gpus</span><span class="p">]]</span> <span class="o">*</span> <span class="nb">len</span><span class="p">(</span><span class="n">keys</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>
<span class="go">update on key: 5</span>
<span class="go">update on key: 7</span>
<span class="go">update on key: 9</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">out</span> <span class="o">=</span> <span class="n">b</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">b</span><span class="p">[</span><span class="mi">1</span><span class="p">][</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 11. 11. 11.]</span>
<span class="go"> [ 11. 11. 11.]]</span>
</pre></div>
</div>
</div>
<div class="section" id="api-reference">
<span id="api-reference"></span><h2>API Reference<a class="headerlink" href="#api-reference" title="Permalink to this headline"></a></h2>
<script src="../../_static/js/auto_module_index.js" type="text/javascript"></script><span class="target" id="module-mxnet.kvstore"></span><p>Key value store interface of MXNet for parameter synchronization.</p>
<dl class="class">
<dt id="mxnet.kvstore.KVStore">
<em class="property">class </em><code class="descclassname">mxnet.kvstore.</code><code class="descname">KVStore</code><span class="sig-paren">(</span><em>handle</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/kvstore.html#KVStore"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.kvstore.KVStore" title="Permalink to this definition"></a></dt>
<dd><p>A key-value store for synchronization of values, over multiple devices.</p>
<dl class="method">
<dt id="mxnet.kvstore.KVStore.init">
<code class="descname">init</code><span class="sig-paren">(</span><em>key</em>, <em>value</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/kvstore.html#KVStore.init"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.kvstore.KVStore.init" title="Permalink to this definition"></a></dt>
<dd><p>Initializes a single or a sequence of key-value pairs into the store.</p>
<p>For each key, one must <cite>init</cite> it before calling <cite>push</cite> or <cite>pull</cite>.
When multiple workers invoke <cite>init</cite> for the same key, only
the value supplied by worker with rank <cite>0</cite> is used. This function returns
after data has been initialized successfully.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name"/>
<col class="field-body"/>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
<li><strong>key</strong> (<em>str, int, or sequence of str or int</em>) – The keys.</li>
<li><strong>value</strong> (<em>NDArray, RowSparseNDArray or sequence of NDArray or RowSparseNDArray</em>) – Values corresponding to the keys.</li>
</ul>
</td>
</tr>
</tbody>
</table>
<p class="rubric">Examples</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="c1"># init a single key-value pair</span>
<span class="gp">>>> </span><span class="n">shape</span> <span class="o">=</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">kv</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="s1">'local'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="s1">'3'</span><span class="p">,</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">)</span><span class="o">*</span><span class="mi">2</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">a</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">shape</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="s1">'3'</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">a</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">a</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 2. 2. 2.]</span>
<span class="go">[ 2. 2. 2.]]</span>
</pre></div>
</div>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="c1"># init a list of key-value pairs</span>
<span class="gp">>>> </span><span class="n">keys</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'5'</span><span class="p">,</span> <span class="s1">'7'</span><span class="p">,</span> <span class="s1">'9'</span><span class="p">]</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="p">[</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">)]</span><span class="o">*</span><span class="nb">len</span><span class="p">(</span><span class="n">keys</span><span class="p">))</span>
</pre></div>
</div>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="c1"># init a row_sparse value</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="s1">'4'</span><span class="p">,</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">)</span><span class="o">.</span><span class="n">tostype</span><span class="p">(</span><span class="s1">'row_sparse'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="n">b</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">sparse</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="s1">'row_sparse'</span><span class="p">,</span> <span class="n">shape</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">row_sparse_pull</span><span class="p">(</span><span class="s1">'4'</span><span class="p">,</span> <span class="n">row_ids</span><span class="o">=</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">]),</span> <span class="n">out</span><span class="o">=</span><span class="n">b</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">b</span>
<span class="go"><RowSparseNDArray 2x3 @cpu(0)></span>
</pre></div>
</div>
</dd></dl>
<dl class="method">
<dt id="mxnet.kvstore.KVStore.push">
<code class="descname">push</code><span class="sig-paren">(</span><em>key</em>, <em>value</em>, <em>priority=0</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/kvstore.html#KVStore.push"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.kvstore.KVStore.push" title="Permalink to this definition"></a></dt>
<dd><p>Pushes a single or a sequence of key-value pairs into the store.</p>
<p>This function returns immediately after adding an operator to the engine.
The actual operation is executed asynchronously. If there are consecutive
pushes to the same key, there is no guarantee on the serialization of pushes.
The execution of a push does not guarantee that all previous pushes are
finished.
There is no synchronization between workers.
One can use <code class="docutils literal"><span class="pre">_barrier()</span></code> to sync all workers.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name"/>
<col class="field-body"/>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
<li><strong>key</strong> (<em>str, int, or sequence of str or int</em>) – Keys.</li>
<li><strong>value</strong> (<em>NDArray, RowSparseNDArray, list of NDArray or RowSparseNDArray,</em>) – or list of list of NDArray or RowSparseNDArray
Values corresponding to the keys.</li>
<li><strong>priority</strong> (<em>int, optional</em>) – The priority of the push operation.
Higher priority push operations are likely to be executed before
other push actions.</li>
</ul>
</td>
</tr>
</tbody>
</table>
<p class="rubric">Examples</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="c1"># push a single key-value pair</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="s1">'3'</span><span class="p">,</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">)</span><span class="o">*</span><span class="mi">8</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="s1">'3'</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">a</span><span class="p">)</span> <span class="c1"># pull out the value</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">a</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 8. 8. 8.]</span>
<span class="go">[ 8. 8. 8.]]</span>
</pre></div>
</div>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="c1"># aggregate the value and the push</span>
<span class="gp">>>> </span><span class="n">gpus</span> <span class="o">=</span> <span class="p">[</span><span class="n">mx</span><span class="o">.</span><span class="n">gpu</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">4</span><span class="p">)]</span>
<span class="gp">>>> </span><span class="n">b</span> <span class="o">=</span> <span class="p">[</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">,</span> <span class="n">gpu</span><span class="p">)</span> <span class="k">for</span> <span class="n">gpu</span> <span class="ow">in</span> <span class="n">gpus</span><span class="p">]</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="s1">'3'</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="s1">'3'</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">a</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">a</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 4. 4. 4.]</span>
<span class="go">[ 4. 4. 4.]]</span>
</pre></div>
</div>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="c1"># push a list of keys.</span>
<span class="gp">>>> </span><span class="c1"># single device</span>
<span class="gp">>>> </span><span class="n">keys</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'4'</span><span class="p">,</span> <span class="s1">'5'</span><span class="p">,</span> <span class="s1">'6'</span><span class="p">]</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="p">[</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">)]</span><span class="o">*</span><span class="nb">len</span><span class="p">(</span><span class="n">keys</span><span class="p">))</span>
<span class="gp">>>> </span><span class="n">b</span> <span class="o">=</span> <span class="p">[</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">shape</span><span class="p">)]</span><span class="o">*</span><span class="nb">len</span><span class="p">(</span><span class="n">keys</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">b</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">b</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 1. 1. 1.]</span>
<span class="go">[ 1. 1. 1.]]</span>
</pre></div>
</div>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="c1"># multiple devices:</span>
<span class="gp">>>> </span><span class="n">keys</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'7'</span><span class="p">,</span> <span class="s1">'8'</span><span class="p">,</span> <span class="s1">'9'</span><span class="p">]</span>
<span class="gp">>>> </span><span class="n">b</span> <span class="o">=</span> <span class="p">[[</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">,</span> <span class="n">gpu</span><span class="p">)</span> <span class="k">for</span> <span class="n">gpu</span> <span class="ow">in</span> <span class="n">gpus</span><span class="p">]]</span> <span class="o">*</span> <span class="nb">len</span><span class="p">(</span><span class="n">keys</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">b</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">b</span><span class="p">[</span><span class="mi">1</span><span class="p">][</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 4. 4. 4.]</span>
<span class="go">[ 4. 4. 4.]]</span>
</pre></div>
</div>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="c1"># push a row_sparse value</span>
<span class="gp">>>> </span><span class="n">b</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">sparse</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="s1">'row_sparse'</span><span class="p">,</span> <span class="n">shape</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="s1">'10'</span><span class="p">,</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">sparse</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="s1">'row_sparse'</span><span class="p">,</span> <span class="n">shape</span><span class="p">))</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="s1">'10'</span><span class="p">,</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">)</span><span class="o">.</span><span class="n">tostype</span><span class="p">(</span><span class="s1">'row_sparse'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="c1"># pull out the value</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">row_sparse_pull</span><span class="p">(</span><span class="s1">'10'</span><span class="p">,</span> <span class="n">row_ids</span><span class="o">=</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">]),</span> <span class="n">out</span><span class="o">=</span><span class="n">b</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">b</span>
<span class="go"><RowSparseNDArray 2x3 @cpu(0)></span>
</pre></div>
</div>
</dd></dl>
<dl class="method">
<dt id="mxnet.kvstore.KVStore.pull">
<code class="descname">pull</code><span class="sig-paren">(</span><em>key</em>, <em>out=None</em>, <em>priority=0</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/kvstore.html#KVStore.pull"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.kvstore.KVStore.pull" title="Permalink to this definition"></a></dt>
<dd><p>Pulls a single value or a sequence of values from the store.</p>
<p>This function returns immediately after adding an operator to the engine.
Subsequent attempts to read from the <cite>out</cite> variable will be blocked until the
pull operation completes.</p>
<p><cite>pull</cite> is executed asynchronously after all previous <cite>pull</cite> calls and only
the last <cite>push</cite> call for the same input key(s) are finished.</p>
<p>The returned values are guaranteed to be the latest values in the store.</p>
<p>For <cite>RowSparseNDArray</cite> values, this call is ignored,
please use <code class="docutils literal"><span class="pre">row_sparse_pull</span></code> instead.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name"/>
<col class="field-body"/>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
<li><strong>key</strong> (<em>str, int, or sequence of str or int</em>) – Keys.</li>
<li><strong>out</strong> (<em>NDArray or list of NDArray or list of list of NDArray</em>) – Values corresponding to the keys.</li>
<li><strong>priority</strong> (<em>int, optional</em>) – The priority of the pull operation.
Higher priority pull operations are likely to be executed before
other pull actions.</li>
</ul>
</td>
</tr>
</tbody>
</table>
<p class="rubric">Examples</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="c1"># pull a single key-value pair</span>
<span class="gp">>>> </span><span class="n">a</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">shape</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="s1">'3'</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">a</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">a</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 2. 2. 2.]</span>
<span class="go">[ 2. 2. 2.]]</span>
</pre></div>
</div>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="c1"># pull into multiple devices</span>
<span class="gp">>>> </span><span class="n">b</span> <span class="o">=</span> <span class="p">[</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">,</span> <span class="n">gpu</span><span class="p">)</span> <span class="k">for</span> <span class="n">gpu</span> <span class="ow">in</span> <span class="n">gpus</span><span class="p">]</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="s1">'3'</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">b</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">b</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 2. 2. 2.]</span>
<span class="go">[ 2. 2. 2.]]</span>
</pre></div>
</div>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="c1"># pull a list of key-value pairs.</span>
<span class="gp">>>> </span><span class="c1"># On single device</span>
<span class="gp">>>> </span><span class="n">keys</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'5'</span><span class="p">,</span> <span class="s1">'7'</span><span class="p">,</span> <span class="s1">'9'</span><span class="p">]</span>
<span class="gp">>>> </span><span class="n">b</span> <span class="o">=</span> <span class="p">[</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">shape</span><span class="p">)]</span><span class="o">*</span><span class="nb">len</span><span class="p">(</span><span class="n">keys</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">b</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">b</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 2. 2. 2.]</span>
<span class="go">[ 2. 2. 2.]]</span>
<span class="gp">>>> </span><span class="c1"># On multiple devices</span>
<span class="gp">>>> </span><span class="n">keys</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'6'</span><span class="p">,</span> <span class="s1">'8'</span><span class="p">,</span> <span class="s1">'10'</span><span class="p">]</span>
<span class="gp">>>> </span><span class="n">b</span> <span class="o">=</span> <span class="p">[[</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">,</span> <span class="n">gpu</span><span class="p">)</span> <span class="k">for</span> <span class="n">gpu</span> <span class="ow">in</span> <span class="n">gpus</span><span class="p">]]</span> <span class="o">*</span> <span class="nb">len</span><span class="p">(</span><span class="n">keys</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="n">keys</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">b</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">b</span><span class="p">[</span><span class="mi">1</span><span class="p">][</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 2. 2. 2.]</span>
<span class="go">[ 2. 2. 2.]]</span>
</pre></div>
</div>
</dd></dl>
<dl class="method">
<dt id="mxnet.kvstore.KVStore.row_sparse_pull">
<code class="descname">row_sparse_pull</code><span class="sig-paren">(</span><em>key</em>, <em>out=None</em>, <em>priority=0</em>, <em>row_ids=None</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/kvstore.html#KVStore.row_sparse_pull"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.kvstore.KVStore.row_sparse_pull" title="Permalink to this definition"></a></dt>
<dd><p>Pulls a single RowSparseNDArray value or a sequence of RowSparseNDArray values from the store with specified row_ids.</p>
<p><cite>row_sparse_pull</cite> is executed asynchronously after all previous
<cite>pull</cite>/<cite>row_sparse_pull</cite> calls and the last <cite>push</cite> call for the
same input key(s) are finished.</p>
<p>The returned values are guaranteed to be the latest values in the store.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name"/>
<col class="field-body"/>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
<li><strong>key</strong> (<em>str, int, or sequence of str or int</em>) – Keys.</li>
<li><strong>out</strong> (<em>RowSparseNDArray or list of RowSparseNDArray or list of list of RowSparseNDArray</em>) – Values corresponding to the keys. The stype is expected to be row_sparse</li>
<li><strong>priority</strong> (<em>int, optional</em>) – The priority of the pull operation.
Higher priority pull operations are likely to be executed before
other pull actions.</li>
<li><strong>row_ids</strong> (<em>NDArray or list of NDArray</em>) – The row_ids for which to pull for each value. Each row_id is an 1D NDArray whose values don’t have to be unique nor sorted.</li>
</ul>
</td>
</tr>
</tbody>
</table>
<p class="rubric">Examples</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">shape</span> <span class="o">=</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="s1">'3'</span><span class="p">,</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">)</span><span class="o">.</span><span class="n">tostype</span><span class="p">(</span><span class="s1">'row_sparse'</span><span class="p">))</span>
<span class="gp">>>> </span><span class="n">a</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">sparse</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="s1">'row_sparse'</span><span class="p">,</span> <span class="n">shape</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">row_ids</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="s1">'int64'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">row_sparse_pull</span><span class="p">(</span><span class="s1">'3'</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">a</span><span class="p">,</span> <span class="n">row_ids</span><span class="o">=</span><span class="n">row_ids</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">a</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 1. 1. 1.]</span>
<span class="go">[ 0. 0. 0.]</span>
<span class="go">[ 1. 1. 1.]]</span>
<span class="gp">>>> </span><span class="n">duplicate_row_ids</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="s1">'int64'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">row_sparse_pull</span><span class="p">(</span><span class="s1">'3'</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">a</span><span class="p">,</span> <span class="n">row_ids</span><span class="o">=</span><span class="n">duplicate_row_ids</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">a</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 0. 0. 0.]</span>
<span class="go">[ 0. 0. 0.]</span>
<span class="go">[ 1. 1. 1.]]</span>
<span class="gp">>>> </span><span class="n">unsorted_row_ids</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="s1">'int64'</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">row_sparse_pull</span><span class="p">(</span><span class="s1">'3'</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">a</span><span class="p">,</span> <span class="n">row_ids</span><span class="o">=</span><span class="n">unsorted_row_ids</span><span class="p">)</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">a</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">[[ 1. 1. 1.]</span>
<span class="go">[ 1. 1. 1.]</span>
<span class="go">[ 0. 0. 0.]]</span>
</pre></div>
</div>
</dd></dl>
<dl class="method">
<dt id="mxnet.kvstore.KVStore.set_gradient_compression">
<code class="descname">set_gradient_compression</code><span class="sig-paren">(</span><em>compression_params</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/kvstore.html#KVStore.set_gradient_compression"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.kvstore.KVStore.set_gradient_compression" title="Permalink to this definition"></a></dt>
<dd><p>Specifies type of low-bit quantization for gradient compression and additional arguments depending on the type of compression being used.</p>
<p>2bit Gradient Compression takes a positive float <cite>threshold</cite>.
The technique works by thresholding values such that positive values in the
gradient above threshold will be set to threshold. Negative values whose absolute
values are higher than threshold, will be set to the negative of threshold.
Values whose absolute values are less than threshold will be set to 0.
By doing so, each value in the gradient is in one of three states. 2bits are
used to represent these states, and every 16 float values in the original
gradient can be represented using one float. This compressed representation
can reduce communication costs. The difference between these thresholded values and
original values is stored at the sender’s end as residual and added to the
gradient in the next iteration.</p>
<p>When kvstore is ‘local’, gradient compression is used to reduce communication
between multiple devices (gpus). Gradient is quantized on each GPU which
computed the gradients, then sent to the GPU which merges the gradients. This
receiving GPU dequantizes the gradients and merges them. Note that this
increases memory usage on each GPU because of the residual array stored.</p>
<p>When kvstore is ‘dist’, gradient compression is used to reduce communication
from worker to sender. Gradient is quantized on each worker which
computed the gradients, then sent to the server which dequantizes
this data and merges the gradients from each worker. Note that this
increases CPU memory usage on each worker because of the residual array stored.
Only worker to server communication is compressed in this setting.
If each machine has multiple GPUs, currently this GPU to GPU or GPU to CPU communication
is not compressed. Server to worker communication (in the case of pull)
is also not compressed.</p>
<p>To use 2bit compression, we need to specify <cite>type</cite> as <cite>2bit</cite>.
Only specifying <cite>type</cite> would use default value for the threshold.
To completely specify the arguments for 2bit compression, we would need to pass
a dictionary which includes <cite>threshold</cite> like:
{‘type’: ‘2bit’, ‘threshold’: 0.5}</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name"/>
<col class="field-body"/>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>compression_params</strong> (<em>dict</em>) – A dictionary specifying the type and parameters for gradient compression.
The key <cite>type</cite> in this dictionary is a
required string argument and specifies the type of gradient compression.
Currently <cite>type</cite> can be only <cite>2bit</cite>
Other keys in this dictionary are optional and specific to the type
of gradient compression.</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="mxnet.kvstore.KVStore.set_optimizer">
<code class="descname">set_optimizer</code><span class="sig-paren">(</span><em>optimizer</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/kvstore.html#KVStore.set_optimizer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.kvstore.KVStore.set_optimizer" title="Permalink to this definition"></a></dt>
<dd><p>Registers an optimizer with the kvstore.</p>
<p>When using a single machine, this function updates the local optimizer.
If using multiple machines and this operation is invoked from a worker node,
it will serialized the optimizer with pickle and send it to all servers.
The function returns after all servers have been updated.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name"/>
<col class="field-body"/>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>optimizer</strong> (<a class="reference internal" href="../optimization/optimization.html#mxnet.optimizer.Optimizer" title="mxnet.optimizer.Optimizer"><em>Optimizer</em></a>) – The new optimizer for the store</td>
</tr>
</tbody>
</table>
<p class="rubric">Examples</p>
<div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">kv</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">kv</span><span class="o">.</span><span class="n">create</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">shape</span> <span class="o">=</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">weight</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">shape</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">weight</span><span class="p">)</span>
<span class="gp">>>> </span><span class="c1"># set the optimizer for kvstore as the default SGD optimizer</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">set_optimizer</span><span class="p">(</span><span class="n">mx</span><span class="o">.</span><span class="n">optimizer</span><span class="o">.</span><span class="n">SGD</span><span class="p">())</span>
<span class="gp">>>> </span><span class="n">grad</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">(</span><span class="n">shape</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">push</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">grad</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">kv</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="n">out</span> <span class="o">=</span> <span class="n">weight</span><span class="p">)</span>
<span class="gp">>>> </span><span class="c1"># weight is updated via gradient descent</span>
<span class="gp">>>> </span><span class="n">weight</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span>
<span class="go">array([[-0.01, -0.01],</span>
<span class="go"> [-0.01, -0.01]], dtype=float32)</span>
</pre></div>
</div>
</dd></dl>
<dl class="attribute">
<dt id="mxnet.kvstore.KVStore.type">
<code class="descname">type</code><a class="headerlink" href="#mxnet.kvstore.KVStore.type" title="Permalink to this definition"></a></dt>
<dd><p>Returns the type of this kvstore.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name"/>
<col class="field-body"/>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body"><strong>type</strong> –
the string type</td>
</tr>
<tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body">str</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="attribute">
<dt id="mxnet.kvstore.KVStore.rank">
<code class="descname">rank</code><a class="headerlink" href="#mxnet.kvstore.KVStore.rank" title="Permalink to this definition"></a></dt>
<dd><p>Returns the rank of this worker node.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name"/>
<col class="field-body"/>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body"><strong>rank</strong> –
The rank of this node, which is in range [0, num_workers())</td>
</tr>
<tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body">int</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="attribute">
<dt id="mxnet.kvstore.KVStore.num_workers">
<code class="descname">num_workers</code><a class="headerlink" href="#mxnet.kvstore.KVStore.num_workers" title="Permalink to this definition"></a></dt>
<dd><p>Returns the number of worker nodes.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name"/>
<col class="field-body"/>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body"><strong>size</strong> –
The number of worker nodes.</td>
</tr>
<tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body">int</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="mxnet.kvstore.KVStore.save_optimizer_states">
<code class="descname">save_optimizer_states</code><span class="sig-paren">(</span><em>fname</em>, <em>dump_optimizer=False</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/kvstore.html#KVStore.save_optimizer_states"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.kvstore.KVStore.save_optimizer_states" title="Permalink to this definition"></a></dt>
<dd><p>Saves the optimizer (updater) state to a file. This is often used when checkpointing
the model during training.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name"/>
<col class="field-body"/>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
<li><strong>fname</strong> (<em>str</em>) – Path to the output states file.</li>
<li><strong>dump_optimizer</strong> (<em>bool, default False</em>) – Whether to also save the optimizer itself. This would also save optimizer
information such as learning rate and weight decay schedules.</li>
</ul>
</td>
</tr>
</tbody>
</table>
</dd></dl>
<dl class="method">
<dt id="mxnet.kvstore.KVStore.load_optimizer_states">
<code class="descname">load_optimizer_states</code><span class="sig-paren">(</span><em>fname</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/kvstore.html#KVStore.load_optimizer_states"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.kvstore.KVStore.load_optimizer_states" title="Permalink to this definition"></a></dt>
<dd><p>Loads the optimizer (updater) state from the file.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name"/>
<col class="field-body"/>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>fname</strong> (<em>str</em>) – Path to input states file.</td>
</tr>
</tbody>
</table>
</dd></dl>
</dd></dl>
<dl class="function">
<dt id="mxnet.kvstore.create">
<code class="descclassname">mxnet.kvstore.</code><code class="descname">create</code><span class="sig-paren">(</span><em>name='local'</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/kvstore.html#create"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.kvstore.create" title="Permalink to this definition"></a></dt>
<dd><p>Creates a new KVStore.</p>
<p>For single machine training, there are two commonly used types:</p>
<p><code class="docutils literal"><span class="pre">local</span></code>: Copies all gradients to CPU memory and updates weights there.</p>
<p><code class="docutils literal"><span class="pre">device</span></code>: Aggregates gradients and updates weights on GPUs. With this setting,
the KVStore also attempts to use GPU peer-to-peer communication,
potentially accelerating the communication.</p>
<p>For distributed training, KVStore also supports a number of types:</p>
<p><code class="docutils literal"><span class="pre">dist_sync</span></code>: Behaves similarly to <code class="docutils literal"><span class="pre">local</span></code> but with one major difference.
With <code class="docutils literal"><span class="pre">dist_sync</span></code>, batch-size now means the batch size used on each machine.
So if there are <code class="docutils literal"><span class="pre">n</span></code> machines and we use batch size <code class="docutils literal"><span class="pre">b</span></code>,
then <code class="docutils literal"><span class="pre">dist_sync</span></code> behaves like <code class="docutils literal"><span class="pre">local</span></code> with batch size <code class="docutils literal"><span class="pre">n</span> <span class="pre">*</span> <span class="pre">b</span></code>.</p>
<p><code class="docutils literal"><span class="pre">dist_device_sync</span></code>: Identical to <code class="docutils literal"><span class="pre">dist_sync</span></code> with the difference similar
to <code class="docutils literal"><span class="pre">device</span></code> vs <code class="docutils literal"><span class="pre">local</span></code>.</p>
<p><code class="docutils literal"><span class="pre">dist_async</span></code>: Performs asynchronous updates.
The weights are updated whenever gradients are received from any machine.
No two updates happen on the same weight at the same time. However, the order is not
guaranteed.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name"/>
<col class="field-body"/>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>name</strong> (<em>{'local', 'device', 'nccl', 'dist_sync', 'dist_device_sync', 'dist_async'}</em>) – The type of KVStore.</td>
</tr>
<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><strong>kv</strong> –
The created KVStore.</td>
</tr>
<tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body"><a class="reference internal" href="#mxnet.kvstore.KVStore" title="mxnet.kvstore.KVStore">KVStore</a></td>
</tr>
</tbody>
</table>
</dd></dl>
<script>auto_index("api-reference");</script></div>
</div>
</div>
</div>
<div aria-label="main navigation" class="sphinxsidebar rightsidebar" role="navigation">
<div class="sphinxsidebarwrapper">
<h3><a href="../../../index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">KVStore API</a><ul>
<li><a class="reference internal" href="#basic-push-and-pull">Basic Push and Pull</a><ul>
<li><a class="reference internal" href="#initialization">Initialization</a></li>
<li><a class="reference internal" href="#push-aggregation-and-updater">Push, Aggregation, and Updater</a></li>
<li><a class="reference internal" href="#pull">Pull</a></li>
</ul>
</li>
<li><a class="reference internal" href="#list-key-value-pairs">List Key-Value Pairs</a></li>
<li><a class="reference internal" href="#api-reference">API Reference</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div><div class="footer">
<div class="section-disclaimer">
<div class="container">
<div>
<img height="60" src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/apache_incubator_logo.png"/>
<p>
Apache MXNet is an effort undergoing incubation at The Apache Software Foundation (ASF), <strong>sponsored by the <i>Apache Incubator</i></strong>. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
</p>
<p>
"Copyright © 2017, The Apache Software Foundation
Apache MXNet, MXNet, Apache, the Apache feather, and the Apache MXNet project logo are either registered trademarks or trademarks of the Apache Software Foundation."
</p>
</div>
</div>
</div>
</div> <!-- pagename != index -->
</div>
<script crossorigin="anonymous" integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS" src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"></script>
<script src="../../../_static/js/sidebar.js" type="text/javascript"></script>
<script src="../../../_static/js/search.js" type="text/javascript"></script>
<script src="../../../_static/js/navbar.js" type="text/javascript"></script>
<script src="../../../_static/js/clipboard.min.js" type="text/javascript"></script>
<script src="../../../_static/js/copycode.js" type="text/javascript"></script>
<script src="../../../_static/js/page.js" type="text/javascript"></script>
<script type="text/javascript">
$('body').ready(function () {
$('body').css('visibility', 'visible');
});
</script>
</body>
</html>