| |
| |
| |
| <!DOCTYPE html> |
| <!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> |
| <!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> |
| <head> |
| <meta charset="utf-8"> |
| |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| <title>分散トレーニング フレームワーク — incubator-singa 0.3.0 documentation</title> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> |
| |
| |
| |
| |
| |
| <link rel="top" title="incubator-singa 0.3.0 documentation" href="../index.html"/> |
| |
| |
| <script src="../_static/js/modernizr.min.js"></script> |
| |
| </head> |
| |
| <body class="wy-body-for-nav" role="document"> |
| |
| <div class="wy-grid-for-nav"> |
| |
| |
| <nav data-toggle="wy-nav-shift" class="wy-nav-side"> |
| <div class="wy-side-scroll"> |
| <div class="wy-side-nav-search"> |
| |
| |
| |
| <a href="../index.html" class="icon icon-home"> incubator-singa |
| |
| |
| |
| |
| <img src="../_static/singa.png" class="logo" /> |
| |
| </a> |
| |
| |
| |
| |
| <div class="version"> |
| 0.3.0 |
| </div> |
| |
| |
| |
| |
| <div role="search"> |
| <form id="rtd-search-form" class="wy-form" action="../search.html" method="get"> |
| <input type="text" name="q" placeholder="Search docs" /> |
| <input type="hidden" name="check_keywords" value="yes" /> |
| <input type="hidden" name="area" value="default" /> |
| </form> |
| </div> |
| |
| |
| </div> |
| |
| <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> |
| |
| |
| |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="index.html">ドキュメント</a></li> |
| </ul> |
| <p class="caption"><span class="caption-text">Development</span></p> |
| <ul class="simple"> |
| </ul> |
| <p class="caption"><span class="caption-text">Community</span></p> |
| <ul class="simple"> |
| </ul> |
| |
| |
| |
| </div> |
| </div> |
| </nav> |
| |
| <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> |
| |
| |
| <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> |
| <i data-toggle="wy-nav-top" class="fa fa-bars"></i> |
| <a href="../index.html">incubator-singa</a> |
| </nav> |
| |
| |
| |
| <div class="wy-nav-content"> |
| <div class="rst-content"> |
| |
| |
| |
| |
| |
| |
| <div role="navigation" aria-label="breadcrumbs navigation"> |
| <ul class="wy-breadcrumbs"> |
| <li><a href="../index.html">Docs</a> »</li> |
| |
| <li>分散トレーニング フレームワーク</li> |
| <li class="wy-breadcrumbs-aside"> |
| |
| |
| |
| </li> |
| </ul> |
| <hr/> |
| </div> |
| <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> |
| <div itemprop="articleBody"> |
| |
| <div class="section" id=""> |
| <span id="id1"></span><h1>分散トレーニング フレームワーク<a class="headerlink" href="#" title="Permalink to this headline">¶</a></h1> |
| <hr class="docutils" /> |
| <div class="section" id=""> |
| <span id="id2"></span><h2>クラスタトポロジーの設定<a class="headerlink" href="#" title="Permalink to this headline">¶</a></h2> |
| <p>クラスタトポロジーは <code class="docutils literal"><span class="pre">JobProto</span></code> の <code class="docutils literal"><span class="pre">cluster</span></code> フィールドで設定します。 |
| <code class="docutils literal"><span class="pre">cluster</span></code> の type は <code class="docutils literal"><span class="pre">ClusterProto</span></code> です。</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span>message ClusterProto { |
| optional int32 nworker_groups = 1; |
| optional int32 nserver_groups = 2; |
| optional int32 nworkers_per_group = 3 [default = 1]; |
| optional int32 nservers_per_group = 4 [default = 1]; |
| optional int32 nworkers_per_procs = 5 [default = 1]; |
| optional int32 nservers_per_procs = 6 [default = 1]; |
| |
| // servers and workers in different processes? |
| optional bool server_worker_separate = 20 [default = false]; |
| |
| ...... |
| } |
| </pre></div> |
| </div> |
| <p>下記のフィールドを設定して、クラスタトポロジーをカスタマイズします。</p> |
| <ul class="simple"> |
| <li><code class="docutils literal"><span class="pre">nworkers_per_group</span></code> と <code class="docutils literal"><span class="pre">nworkers_per_procs</span></code> |
| decide the partitioning of worker side ParamShard.</li> |
| <li><code class="docutils literal"><span class="pre">nservers_per_group</span></code> と <code class="docutils literal"><span class="pre">nservers_per_procs</span></code> |
| decide the partitioning of server side ParamShard.</li> |
| <li><code class="docutils literal"><span class="pre">server_worker_separate</span></code> |
| separate servers and workers in different processes.</li> |
| </ul> |
| </div> |
| <div class="section" id=""> |
| <span id="id3"></span><h2>トレーニング フレームワーク<a class="headerlink" href="#" title="Permalink to this headline">¶</a></h2> |
| <p>In SINGA, worker groups run asynchronously and workers within one group run synchronously. |
| Users can leverage this general design to run both <strong>synchronous</strong> and <strong>asynchronous</strong> training frameworks. |
| Here we illustrate how to configure popular distributed training frameworks in SINGA.</p> |
| <p><img src="../_static/images/frameworks.png" style="width: 800px"/></p> |
| <p><strong> Fig.1 - Training frameworks in SINGA</strong></p><div class="section" id="sandblaster"> |
| <span id="sandblaster"></span><h3>Sandblaster<a class="headerlink" href="#sandblaster" title="Permalink to this headline">¶</a></h3> |
| <p>Google Brain で使われている <strong>synchronous</strong> フレームワークです。 |
| Fig.2(a)に、SINGA で実装された Sandblaster を示します。 |
| cluster フィールドの設定は次のとおりです。</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">cluster</span> <span class="p">{</span> |
| <span class="n">nworker_groups</span><span class="p">:</span> <span class="mi">1</span> |
| <span class="n">nserver_groups</span><span class="p">:</span> <span class="mi">1</span> |
| <span class="n">nworkers_per_group</span><span class="p">:</span> <span class="mi">3</span> |
| <span class="n">nservers_per_group</span><span class="p">:</span> <span class="mi">2</span> |
| <span class="n">server_worker_separate</span><span class="p">:</span> <span class="n">true</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| <p>A single server group is launched to handle all requests from workers. |
| A worker computes on its partition of the model, |
| and only communicates with servers handling related parameters.</p> |
| </div> |
| <div class="section" id="allreduce"> |
| <span id="allreduce"></span><h3>AllReduce<a class="headerlink" href="#allreduce" title="Permalink to this headline">¶</a></h3> |
| <p>Baidu’s DeepImage で使われている <strong>synchronous</strong> フレームワークです。 |
| Fig.2(b)に、SINGA で実装された AllReduce を示します。 |
| cluster フィールドの設定は次のとおりです。</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">cluster</span> <span class="p">{</span> |
| <span class="n">nworker_groups</span><span class="p">:</span> <span class="mi">1</span> |
| <span class="n">nserver_groups</span><span class="p">:</span> <span class="mi">1</span> |
| <span class="n">nworkers_per_group</span><span class="p">:</span> <span class="mi">3</span> |
| <span class="n">nservers_per_group</span><span class="p">:</span> <span class="mi">3</span> |
| <span class="n">server_worker_separate</span><span class="p">:</span> <span class="n">false</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| <p>We bind each worker with a server on the same node, so that each |
| node is responsible for maintaining a partition of parameters and |
| collecting updates from all other nodes.</p> |
| </div> |
| <div class="section" id="downpour"> |
| <span id="downpour"></span><h3>Downpour<a class="headerlink" href="#downpour" title="Permalink to this headline">¶</a></h3> |
| <p>Google Brain で使われている <strong>asynchronous</strong> フレームワークです。 |
| Fig.2(c)に、SINGA で実装された Downpour を示します。 |
| cluster フィールドの設定は次のとおりです。</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">cluster</span> <span class="p">{</span> |
| <span class="n">nworker_groups</span><span class="p">:</span> <span class="mi">2</span> |
| <span class="n">nserver_groups</span><span class="p">:</span> <span class="mi">1</span> |
| <span class="n">nworkers_per_group</span><span class="p">:</span> <span class="mi">2</span> |
| <span class="n">nservers_per_group</span><span class="p">:</span> <span class="mi">2</span> |
| <span class="n">server_worker_separate</span><span class="p">:</span> <span class="n">true</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| <p>Similar to the synchronous Sandblaster, all workers send |
| requests to a global server group. We divide workers into several |
| worker groups, each running independently and working on parameters |
| from the last <em>update</em> response.</p> |
| <p>###Distributed Hogwild</p> |
| <p>Caffe で使われている <strong>asynchronous</strong> フレームワークです。 |
| Fig.2(d)に、SINGA で実装された Distributed Hogwild を示します。 |
| cluster フィールドの設定は次のとおりです。</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">cluster</span> <span class="p">{</span> |
| <span class="n">nworker_groups</span><span class="p">:</span> <span class="mi">3</span> |
| <span class="n">nserver_groups</span><span class="p">:</span> <span class="mi">3</span> |
| <span class="n">nworkers_per_group</span><span class="p">:</span> <span class="mi">1</span> |
| <span class="n">nservers_per_group</span><span class="p">:</span> <span class="mi">1</span> |
| <span class="n">server_worker_separate</span><span class="p">:</span> <span class="n">false</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| <p>Each node contains a complete server group and a complete worker group. |
| Parameter updates are done locally, so that communication cost |
| during each training step is minimized. |
| However, the server group must periodically synchronize with |
| neighboring groups to improve the training convergence.</p> |
| </div> |
| </div> |
| </div> |
| |
| |
| </div> |
| </div> |
| <footer> |
| |
| |
| <hr/> |
| |
| <div role="contentinfo"> |
| <p> |
| © Copyright 2016 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.. |
| |
| </p> |
| </div> |
| Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. |
| |
| </footer> |
| |
| </div> |
| </div> |
| |
| </section> |
| |
| </div> |
| |
| |
| |
| |
| |
| <script type="text/javascript"> |
| var DOCUMENTATION_OPTIONS = { |
| URL_ROOT:'../', |
| VERSION:'0.3.0', |
| COLLAPSE_INDEX:false, |
| FILE_SUFFIX:'.html', |
| HAS_SOURCE: true |
| }; |
| </script> |
| <script type="text/javascript" src="../_static/jquery.js"></script> |
| <script type="text/javascript" src="../_static/underscore.js"></script> |
| <script type="text/javascript" src="../_static/doctools.js"></script> |
| |
| |
| |
| |
| |
| <script type="text/javascript" src="../_static/js/theme.js"></script> |
| |
| |
| |
| |
| <script type="text/javascript"> |
| jQuery(function () { |
| SphinxRtdTheme.StickyNav.enable(); |
| }); |
| </script> |
| |
| |
| |
| <div class="rst-versions shift-up" data-toggle="rst-versions" role="note" aria-label="versions"> |
| <img src="../_static/apache.jpg"> |
| |
| <span class="rst-current-version" data-toggle="rst-current-version"> |
| <span class="fa fa-book"> incubator-singa </span> |
| v: 0.3.0 |
| <span class="fa fa-caret-down"></span> |
| </span> |
| <div class="rst-other-versions"> |
| <dl> |
| <dt>Languages</dt> |
| <dd><a href="../../en/index.html">English</a></dd> |
| <dd><a href="../../zh/index.html">中文</a></dd> |
| <dd><a href="../../jp/index.html">日本語</a></dd> |
| <dd><a href="../../kr/index.html">한국어</a></dd> |
| </dl> |
| </div> |
| </div> |
| |
| <a href="https://github.com/apache/incubator-singa"> |
| <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" |
| src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png" |
| alt="Fork me on GitHub"> |
| </a> |
| |
| |
| |
| |
| </body> |
| </html> |
