| |
| |
| |
| <!DOCTYPE html> |
| <!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> |
| <!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> |
| <head> |
| <meta charset="utf-8"> |
| |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| <title>Hybrid Parallelism — incubator-singa 0.3.0 documentation</title> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> |
| |
| |
| |
| |
| |
| <link rel="top" title="incubator-singa 0.3.0 documentation" href="../index.html"/> |
| |
| |
| <script src="../_static/js/modernizr.min.js"></script> |
| |
| </head> |
| |
| <body class="wy-body-for-nav" role="document"> |
| |
| <div class="wy-grid-for-nav"> |
| |
| |
| <nav data-toggle="wy-nav-shift" class="wy-nav-side"> |
| <div class="wy-side-scroll"> |
| <div class="wy-side-nav-search"> |
| |
| |
| |
| <a href="../index.html" class="icon icon-home"> incubator-singa |
| |
| |
| |
| |
| <img src="../_static/singa.png" class="logo" /> |
| |
| </a> |
| |
| |
| |
| |
| <div class="version"> |
| 0.3.0 |
| </div> |
| |
| |
| |
| |
| <div role="search"> |
| <form id="rtd-search-form" class="wy-form" action="../search.html" method="get"> |
| <input type="text" name="q" placeholder="Search docs" /> |
| <input type="hidden" name="check_keywords" value="yes" /> |
| <input type="hidden" name="area" value="default" /> |
| </form> |
| </div> |
| |
| |
| </div> |
| |
| <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> |
| |
| |
| |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../downloads.html">Download SINGA</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="index.html">Documentation</a></li> |
| </ul> |
| <p class="caption"><span class="caption-text">Development</span></p> |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../develop/schedule.html">Development Schedule</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../develop/how-contribute.html">How to Contribute to SINGA</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../develop/contribute-code.html">How to Contribute Code</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../develop/contribute-docs.html">How to Contribute Documentation</a></li> |
| </ul> |
| <p class="caption"><span class="caption-text">Community</span></p> |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../community/source-repository.html">Source Repository</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../community/mail-lists.html">Project Mailing Lists</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../community/issue-tracking.html">Issue Tracking</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../community/team-list.html">The SINGA Team</a></li> |
| </ul> |
| |
| |
| |
| </div> |
| </div> |
| </nav> |
| |
| <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> |
| |
| |
| <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> |
| <i data-toggle="wy-nav-top" class="fa fa-bars"></i> |
| <a href="../index.html">incubator-singa</a> |
| </nav> |
| |
| |
| |
| <div class="wy-nav-content"> |
| <div class="rst-content"> |
| |
| |
| |
| |
| |
| |
| <div role="navigation" aria-label="breadcrumbs navigation"> |
| <ul class="wy-breadcrumbs"> |
| <li><a href="../index.html">Docs</a> »</li> |
| |
| <li>Hybrid Parallelism</li> |
| <li class="wy-breadcrumbs-aside"> |
| |
| |
| |
| </li> |
| </ul> |
| <hr/> |
| </div> |
| <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> |
| <div itemprop="articleBody"> |
| |
| <div class="section" id="hybrid-parallelism"> |
| <span id="hybrid-parallelism"></span><h1>Hybrid Parallelism<a class="headerlink" href="#hybrid-parallelism" title="Permalink to this headline">¶</a></h1> |
| <hr class="docutils" /> |
| <div class="section" id="user-guide"> |
| <span id="user-guide"></span><h2>User Guide<a class="headerlink" href="#user-guide" title="Permalink to this headline">¶</a></h2> |
| <p>SINGA supports different parallelism options for distributed training. |
| Users just need to configure it in the job configuration.</p> |
| <p>Both <code class="docutils literal"><span class="pre">NetProto</span></code> and <code class="docutils literal"><span class="pre">LayerProto</span></code> have a field <code class="docutils literal"><span class="pre">partition_dim</span></code> to control the parallelism option:</p> |
| <ul class="simple"> |
| <li><code class="docutils literal"><span class="pre">partition_dim=0</span></code>: neuralnet/layer is partitioned on data dimension, i.e., each worker processes a subset of data records.</li> |
| <li><code class="docutils literal"><span class="pre">partition_dim=1</span></code>: neuralnet/layer is partitioned on feature dimension, i.e., each worker maintains a subset of feature parameters.</li> |
| </ul> |
| <p><code class="docutils literal"><span class="pre">partition_dim</span></code> field in <code class="docutils literal"><span class="pre">NetProto</span></code> will be applied to all layers, unless a layer has its own <code class="docutils literal"><span class="pre">partition_dim</span></code> field set.</p> |
| <p>If we want data parallelism for the whole model, just leave <code class="docutils literal"><span class="pre">partition_dim</span></code> as default (which is 0), or configure the job.conf like:</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">neuralnet</span> <span class="p">{</span> |
| <span class="n">partition_dim</span><span class="p">:</span> <span class="mi">0</span> |
| <span class="n">layer</span> <span class="p">{</span> |
| <span class="n">name</span><span class="p">:</span> <span class="o">...</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="o">...</span> |
| <span class="p">}</span> |
| <span class="o">...</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| <p>With the hybrid parallelism, we can have layers either partitioned on data dimension or feature dimension. |
| For example, if we want a specific layer partitioned on feature dimension, just configure like:</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">neuralnet</span> <span class="p">{</span> |
| <span class="n">partition_dim</span><span class="p">:</span> <span class="mi">0</span> |
| <span class="n">layer</span> <span class="p">{</span> |
| <span class="n">name</span><span class="p">:</span> <span class="s2">"layer1_partition_on_data_dimension"</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="o">...</span> |
| <span class="p">}</span> |
| <span class="n">layer</span> <span class="p">{</span> |
| <span class="n">name</span><span class="p">:</span> <span class="s2">"layer2_partition_on_feature_dimension"</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="o">...</span> |
| <span class="n">partition_dim</span><span class="p">:</span> <span class="mi">1</span> |
| <span class="p">}</span> |
| <span class="o">...</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| </div> |
| <div class="section" id="developer-guide"> |
| <span id="developer-guide"></span><h2>Developer Guide<a class="headerlink" href="#developer-guide" title="Permalink to this headline">¶</a></h2> |
| <p>To support hybrid parallelism, after singa read users’ model and paration configuration, a set of connection layers are automatically added between layers when needed:</p> |
| <ul class="simple"> |
| <li><code class="docutils literal"><span class="pre">BridgeSrcLayer</span></code> & <code class="docutils literal"><span class="pre">BridgeDstLayer</span></code> are added when two connected layers are not in the same machine. They are paired and are responsible for sending data/gradient to the other side during each iteration.</li> |
| <li><code class="docutils literal"><span class="pre">ConcateLayer</span></code> is added when there are multiple source layers. It combines their feature blobs along a given dimension.</li> |
| <li><code class="docutils literal"><span class="pre">SliceLayer</span></code> is added when there are mutliple dest layers, each of which only needs a subset(slice) of this layers’ feature blob.</li> |
| <li><code class="docutils literal"><span class="pre">SplitLayer</span></code> is added when there are multiple dest layers, each of which needs the whole feature blob.</li> |
| </ul> |
| <p>Following is the logic used in our code to add connection layers:</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span>Add Slice, Concate, Split Layers for Hybrid Partition |
| |
| All cases are as follows: |
| src_pdim | dst_pdim | connection_type | Action |
| 0 | 0 | OneToOne | Direct Connection |
| 1 | 1 | OneToOne | Direct Connection |
| 0 | 0 | OneToAll | Direct Connection |
| 1 | 0 | OneToOne | Slice -> Concate |
| 0 | 1 | OneToOne | Slice -> Concate |
| 1 | 0 | OneToAll | Slice -> Concate |
| 0 | 1 | OneToAll | Split -> Concate |
| 1 | 1 | OneToAll | Split -> Concate |
| |
| Logic: |
| dst_pdim = 1 && OneToAll ? |
| (YES) Split -> Concate |
| (NO) src_pdim = dst_pdim ? |
| (YES) Direct Connection |
| (NO) Slice -> Concate |
| </pre></div> |
| </div> |
| </div> |
| </div> |
| |
| |
| </div> |
| </div> |
| <footer> |
| |
| |
| <hr/> |
| |
| <div role="contentinfo"> |
| <p> |
| © Copyright 2016 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.. |
| |
| </p> |
| </div> |
| Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. |
| |
| </footer> |
| |
| </div> |
| </div> |
| |
| </section> |
| |
| </div> |
| |
| |
| |
| |
| |
| <script type="text/javascript"> |
| var DOCUMENTATION_OPTIONS = { |
| URL_ROOT:'../', |
| VERSION:'0.3.0', |
| COLLAPSE_INDEX:false, |
| FILE_SUFFIX:'.html', |
| HAS_SOURCE: true |
| }; |
| </script> |
| <script type="text/javascript" src="../_static/jquery.js"></script> |
| <script type="text/javascript" src="../_static/underscore.js"></script> |
| <script type="text/javascript" src="../_static/doctools.js"></script> |
| |
| |
| |
| |
| |
| <script type="text/javascript" src="../_static/js/theme.js"></script> |
| |
| |
| |
| |
| <script type="text/javascript"> |
| jQuery(function () { |
| SphinxRtdTheme.StickyNav.enable(); |
| }); |
| </script> |
| |
| |
| |
| <div class="rst-versions shift-up" data-toggle="rst-versions" role="note" aria-label="versions"> |
| <img src="../_static/apache.jpg"> |
| |
| <span class="rst-current-version" data-toggle="rst-current-version"> |
| <span class="fa fa-book"> incubator-singa </span> |
| v: 0.3.0 |
| <span class="fa fa-caret-down"></span> |
| </span> |
| <div class="rst-other-versions"> |
| <dl> |
| <dt>Languages</dt> |
| <dd><a href="../../en/index.html">English</a></dd> |
| <dd><a href="../../zh/index.html">中文</a></dd> |
| <dd><a href="../../jp/index.html">日本語</a></dd> |
| <dd><a href="../../kr/index.html">한국어</a></dd> |
| </dl> |
| </div> |
| </div> |
| |
| <a href="https://github.com/apache/incubator-singa"> |
| <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" |
| src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png" |
| alt="Fork me on GitHub"> |
| </a> |
| |
| |
| |
| |
| </body> |
| </html> |
