| |
| |
| |
| <!DOCTYPE html> |
| <!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> |
| <!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> |
| <head> |
| <meta charset="utf-8"> |
| |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| <title>Updater — incubator-singa 0.3.0 documentation</title> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> |
| |
| |
| |
| |
| |
| <link rel="top" title="incubator-singa 0.3.0 documentation" href="../index.html"/> |
| |
| |
| <script src="../_static/js/modernizr.min.js"></script> |
| |
| </head> |
| |
| <body class="wy-body-for-nav" role="document"> |
| |
| <div class="wy-grid-for-nav"> |
| |
| |
| <nav data-toggle="wy-nav-shift" class="wy-nav-side"> |
| <div class="wy-side-scroll"> |
| <div class="wy-side-nav-search"> |
| |
| |
| |
| <a href="../index.html" class="icon icon-home"> incubator-singa |
| |
| |
| |
| |
| <img src="../_static/singa.png" class="logo" /> |
| |
| </a> |
| |
| |
| |
| |
| <div class="version"> |
| 0.3.0 |
| </div> |
| |
| |
| |
| |
| <div role="search"> |
| <form id="rtd-search-form" class="wy-form" action="../search.html" method="get"> |
| <input type="text" name="q" placeholder="Search docs" /> |
| <input type="hidden" name="check_keywords" value="yes" /> |
| <input type="hidden" name="area" value="default" /> |
| </form> |
| </div> |
| |
| |
| </div> |
| |
| <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> |
| |
| |
| |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../downloads.html">Download SINGA</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="index.html">Documentation</a></li> |
| </ul> |
| <p class="caption"><span class="caption-text">Development</span></p> |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../develop/schedule.html">Development Schedule</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../develop/how-contribute.html">How to Contribute to SINGA</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../develop/contribute-code.html">How to Contribute Code</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../develop/contribute-docs.html">How to Contribute Documentation</a></li> |
| </ul> |
| <p class="caption"><span class="caption-text">Community</span></p> |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../community/source-repository.html">Source Repository</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../community/mail-lists.html">Project Mailing Lists</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../community/issue-tracking.html">Issue Tracking</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../community/team-list.html">The SINGA Team</a></li> |
| </ul> |
| |
| |
| |
| </div> |
| </div> |
| </nav> |
| |
| <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> |
| |
| |
| <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> |
| <i data-toggle="wy-nav-top" class="fa fa-bars"></i> |
| <a href="../index.html">incubator-singa</a> |
| </nav> |
| |
| |
| |
| <div class="wy-nav-content"> |
| <div class="rst-content"> |
| |
| |
| |
| |
| |
| |
| <div role="navigation" aria-label="breadcrumbs navigation"> |
| <ul class="wy-breadcrumbs"> |
| <li><a href="../index.html">Docs</a> »</li> |
| |
| <li>Updater</li> |
| <li class="wy-breadcrumbs-aside"> |
| |
| |
| |
| </li> |
| </ul> |
| <hr/> |
| </div> |
| <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> |
| <div itemprop="articleBody"> |
| |
| <div class="section" id="updater"> |
| <span id="updater"></span><h1>Updater<a class="headerlink" href="#updater" title="Permalink to this headline">¶</a></h1> |
| <hr class="docutils" /> |
| <p>Every server in SINGA has an <a class="reference external" href="../api/classsinga_1_1Updater.html">Updater</a> |
| instance that updates parameters based on gradients. |
| In this page, the <em>Basic user guide</em> describes the configuration of an updater. |
| The <em>Advanced user guide</em> present details on how to implement a new updater and a new |
| learning rate changing method.</p> |
| <div class="section" id="basic-user-guide"> |
| <span id="basic-user-guide"></span><h2>Basic user guide<a class="headerlink" href="#basic-user-guide" title="Permalink to this headline">¶</a></h2> |
| <p>There are many different parameter updating protocols (i.e., subclasses of |
| <code class="docutils literal"><span class="pre">Updater</span></code>). They share some configuration fields like</p> |
| <ul class="simple"> |
| <li><code class="docutils literal"><span class="pre">type</span></code>, an integer for identifying an updater;</li> |
| <li><code class="docutils literal"><span class="pre">learning_rate</span></code>, configuration for the |
| <a class="reference external" href="../api/classsinga_1_1LRGenerator.html">LRGenerator</a> which controls the learning rate.</li> |
| <li><code class="docutils literal"><span class="pre">weight_decay</span></code>, the co-efficient for <a class="reference external" href="http://deeplearning.net/tutorial/gettingstarted.html#regularization">L2 * regularization</a>.</li> |
| <li><a class="reference external" href="http://ufldl.stanford.edu/tutorial/supervised/OptimizationStochasticGradientDescent/">momentum</a>.</li> |
| </ul> |
| <p>If you are not familiar with the above terms, you can get their meanings in |
| <a class="reference external" href="http://cs231n.github.io/neural-networks-3/#update">this page provided by Karpathy</a>.</p> |
| <div class="section" id="configuration-of-built-in-updater-classes"> |
| <span id="configuration-of-built-in-updater-classes"></span><h3>Configuration of built-in updater classes<a class="headerlink" href="#configuration-of-built-in-updater-classes" title="Permalink to this headline">¶</a></h3> |
| <div class="section" id="updater"> |
| <span id="id1"></span><h4>Updater<a class="headerlink" href="#updater" title="Permalink to this headline">¶</a></h4> |
| <p>The base <code class="docutils literal"><span class="pre">Updater</span></code> implements the <a class="reference external" href="http://cs231n.github.io/neural-networks-3/#sgd">vanilla SGD algorithm</a>. |
| Its configuration type is <code class="docutils literal"><span class="pre">kSGD</span></code>. |
| Users need to configure at least the <code class="docutils literal"><span class="pre">learning_rate</span></code> field. |
| <code class="docutils literal"><span class="pre">momentum</span></code> and <code class="docutils literal"><span class="pre">weight_decay</span></code> are optional fields.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">updater</span><span class="p">{</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="n">kSGD</span> |
| <span class="n">momentum</span><span class="p">:</span> <span class="nb">float</span> |
| <span class="n">weight_decay</span><span class="p">:</span> <span class="nb">float</span> |
| <span class="n">learning_rate</span> <span class="p">{</span> |
| <span class="o">...</span> |
| <span class="p">}</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| </div> |
| <div class="section" id="adagradupdater"> |
| <span id="adagradupdater"></span><h4>AdaGradUpdater<a class="headerlink" href="#adagradupdater" title="Permalink to this headline">¶</a></h4> |
| <p>It inherits the base <code class="docutils literal"><span class="pre">Updater</span></code> to implement the |
| <a class="reference external" href="http://www.magicbroom.info/Papers/DuchiHaSi10.pdf">AdaGrad</a> algorithm. |
| Its type is <code class="docutils literal"><span class="pre">kAdaGrad</span></code>. |
| <code class="docutils literal"><span class="pre">AdaGradUpdater</span></code> is configured similar to <code class="docutils literal"><span class="pre">Updater</span></code> except |
| that <code class="docutils literal"><span class="pre">momentum</span></code> is not used.</p> |
| </div> |
| <div class="section" id="nesterovupdater"> |
| <span id="nesterovupdater"></span><h4>NesterovUpdater<a class="headerlink" href="#nesterovupdater" title="Permalink to this headline">¶</a></h4> |
| <p>It inherits the base <code class="docutils literal"><span class="pre">Updater</span></code> to implements the |
| <a class="reference external" href="http://arxiv.org/pdf/1212.0901v2.pdf">Nesterov</a> (section 3.5) updating protocol. |
| Its type is <code class="docutils literal"><span class="pre">kNesterov</span></code>. |
| <code class="docutils literal"><span class="pre">learning_rate</span></code> and <code class="docutils literal"><span class="pre">momentum</span></code> must be configured. <code class="docutils literal"><span class="pre">weight_decay</span></code> is an |
| optional configuration field.</p> |
| </div> |
| <div class="section" id="rmspropupdater"> |
| <span id="rmspropupdater"></span><h4>RMSPropUpdater<a class="headerlink" href="#rmspropupdater" title="Permalink to this headline">¶</a></h4> |
| <p>It inherits the base <code class="docutils literal"><span class="pre">Updater</span></code> to implements the |
| <a class="reference external" href="http://cs231n.github.io/neural-networks-3/#sgd">RMSProp algorithm</a> proposed by |
| <a class="reference external" href="http://www.cs.toronto.edu/%7Etijmen/csc321/slides/lecture_slides_lec6.pdf">Hinton</a>(slide 29). |
| Its type is <code class="docutils literal"><span class="pre">kRMSProp</span></code>.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">updater</span> <span class="p">{</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="n">kRMSProp</span> |
| <span class="n">rmsprop_conf</span> <span class="p">{</span> |
| <span class="n">rho</span><span class="p">:</span> <span class="nb">float</span> <span class="c1"># [0,1]</span> |
| <span class="p">}</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| </div> |
| <div class="section" id="adadeltaupdater"> |
| <span id="adadeltaupdater"></span><h4>AdaDeltaUpdater<a class="headerlink" href="#adadeltaupdater" title="Permalink to this headline">¶</a></h4> |
| <p>It inherits the base <code class="docutils literal"><span class="pre">Updater</span></code> to implements the |
| <a class="reference external" href="http://arxiv.org/abs/1212.5701">AdaDelta</a> updating algorithm. |
| Its type is <code class="docutils literal"><span class="pre">kAdaDelta</span></code>.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">updater</span> <span class="p">{</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="n">kAdaDelta</span> |
| <span class="n">adadelta_conf</span> <span class="p">{</span> |
| <span class="n">rho</span><span class="p">:</span> <span class="nb">float</span> <span class="c1"># [0,1]</span> |
| <span class="p">}</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| </div> |
| <div class="section" id="adam"> |
| <span id="adam"></span><h4>Adam<a class="headerlink" href="#adam" title="Permalink to this headline">¶</a></h4> |
| <p>It inherits the base <code class="docutils literal"><span class="pre">Updater</span></code> to implements the |
| <a class="reference external" href="http://arxiv.org/pdf/1412.6980.pdf">Adam</a> updating algorithm. |
| Its type is <code class="docutils literal"><span class="pre">kAdam</span></code>. |
| <code class="docutils literal"><span class="pre">beta1</span></code> and <code class="docutils literal"><span class="pre">beta2</span></code> is floats, 0 < <code class="docutils literal"><span class="pre">beta</span></code> < 1, generally close to 1.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">updater</span> <span class="p">{</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="n">kAdam</span> |
| <span class="n">adam_conf</span> <span class="p">{</span> |
| <span class="n">beta1</span><span class="p">:</span> <span class="nb">float</span> <span class="c1"># [0,1]</span> |
| <span class="n">beta2</span><span class="p">:</span> <span class="nb">float</span> <span class="c1"># [0,1]</span> |
| <span class="p">}</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| </div> |
| <div class="section" id="adamax"> |
| <span id="adamax"></span><h4>AdaMax<a class="headerlink" href="#adamax" title="Permalink to this headline">¶</a></h4> |
| <p>It inherits the base <code class="docutils literal"><span class="pre">Updater</span></code> to implements the |
| <a class="reference external" href="http://arxiv.org/pdf/1412.6980.pdf">AdaMax</a> updating algorithm. |
| Its type is <code class="docutils literal"><span class="pre">kAdamMax</span></code>. |
| <code class="docutils literal"><span class="pre">beta1</span></code> and <code class="docutils literal"><span class="pre">beta2</span></code> is floats, 0 < <code class="docutils literal"><span class="pre">beta</span></code> < 1, generally close to 1.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">updater</span> <span class="p">{</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="n">kAdamMax</span> |
| <span class="n">adammax_conf</span> <span class="p">{</span> |
| <span class="n">beta1</span><span class="p">:</span> <span class="nb">float</span> <span class="c1"># [0,1]</span> |
| <span class="n">beta2</span><span class="p">:</span> <span class="nb">float</span> <span class="c1"># [0,1]</span> |
| <span class="p">}</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| </div> |
| </div> |
| <div class="section" id="configuration-of-learning-rate"> |
| <span id="configuration-of-learning-rate"></span><h3>Configuration of learning rate<a class="headerlink" href="#configuration-of-learning-rate" title="Permalink to this headline">¶</a></h3> |
| <p>The <code class="docutils literal"><span class="pre">learning_rate</span></code> field is configured as,</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">learning_rate</span> <span class="p">{</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="n">ChangeMethod</span> |
| <span class="n">base_lr</span><span class="p">:</span> <span class="nb">float</span> <span class="c1"># base/initial learning rate</span> |
| <span class="o">...</span> <span class="c1"># fields to a specific changing method</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| <p>The common fields include <code class="docutils literal"><span class="pre">type</span></code> and <code class="docutils literal"><span class="pre">base_lr</span></code>. SINGA provides the following |
| <code class="docutils literal"><span class="pre">ChangeMethod</span></code>s.</p> |
| <div class="section" id="kfixed"> |
| <span id="kfixed"></span><h4>kFixed<a class="headerlink" href="#kfixed" title="Permalink to this headline">¶</a></h4> |
| <p>The <code class="docutils literal"><span class="pre">base_lr</span></code> is used for all steps.</p> |
| </div> |
| <div class="section" id="klinear"> |
| <span id="klinear"></span><h4>kLinear<a class="headerlink" href="#klinear" title="Permalink to this headline">¶</a></h4> |
| <p>The updater should be configured like</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">learning_rate</span> <span class="p">{</span> |
| <span class="n">base_lr</span><span class="p">:</span> <span class="nb">float</span> |
| <span class="n">linear_conf</span> <span class="p">{</span> |
| <span class="n">freq</span><span class="p">:</span> <span class="nb">int</span> |
| <span class="n">final_lr</span><span class="p">:</span> <span class="nb">float</span> |
| <span class="p">}</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| <p>Linear interpolation is used to change the learning rate,</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">lr</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">step</span> <span class="o">/</span> <span class="n">freq</span><span class="p">)</span> <span class="o">*</span> <span class="n">base_lr</span> <span class="o">+</span> <span class="p">(</span><span class="n">step</span> <span class="o">/</span> <span class="n">freq</span><span class="p">)</span> <span class="o">*</span> <span class="n">final_lr</span> |
| </pre></div> |
| </div> |
| </div> |
| <div class="section" id="kexponential"> |
| <span id="kexponential"></span><h4>kExponential<a class="headerlink" href="#kexponential" title="Permalink to this headline">¶</a></h4> |
| <p>The udapter should be configured like</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">learning_rate</span> <span class="p">{</span> |
| <span class="n">base_lr</span><span class="p">:</span> <span class="nb">float</span> |
| <span class="n">exponential_conf</span> <span class="p">{</span> |
| <span class="n">freq</span><span class="p">:</span> <span class="nb">int</span> |
| <span class="p">}</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| <p>The learning rate for <code class="docutils literal"><span class="pre">step</span></code> is</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">lr</span> <span class="o">=</span> <span class="n">base_lr</span> <span class="o">/</span> <span class="mi">2</span><span class="o">^</span><span class="p">(</span><span class="n">step</span> <span class="o">/</span> <span class="n">freq</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| </div> |
| <div class="section" id="kinverset"> |
| <span id="kinverset"></span><h4>kInverseT<a class="headerlink" href="#kinverset" title="Permalink to this headline">¶</a></h4> |
| <p>The updater should be configured like</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">learning_rate</span> <span class="p">{</span> |
| <span class="n">base_lr</span><span class="p">:</span> <span class="nb">float</span> |
| <span class="n">inverset_conf</span> <span class="p">{</span> |
| <span class="n">final_lr</span><span class="p">:</span> <span class="nb">float</span> |
| <span class="p">}</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| <p>The learning rate for <code class="docutils literal"><span class="pre">step</span></code> is</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">lr</span> <span class="o">=</span> <span class="n">base_lr</span> <span class="o">/</span> <span class="p">(</span><span class="mi">1</span> <span class="o">+</span> <span class="n">step</span> <span class="o">/</span> <span class="n">final_lr</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| </div> |
| <div class="section" id="kinverse"> |
| <span id="kinverse"></span><h4>kInverse<a class="headerlink" href="#kinverse" title="Permalink to this headline">¶</a></h4> |
| <p>The updater should be configured like</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">learning_rate</span> <span class="p">{</span> |
| <span class="n">base_lr</span><span class="p">:</span> <span class="nb">float</span> |
| <span class="n">inverse_conf</span> <span class="p">{</span> |
| <span class="n">gamma</span><span class="p">:</span> <span class="nb">float</span> |
| <span class="nb">pow</span><span class="p">:</span> <span class="nb">float</span> |
| <span class="p">}</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| <p>The learning rate for <code class="docutils literal"><span class="pre">step</span></code> is</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">lr</span> <span class="o">=</span> <span class="n">base_lr</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">+</span> <span class="n">gamma</span> <span class="o">*</span> <span class="n">setp</span><span class="p">)</span><span class="o">^</span><span class="p">(</span><span class="o">-</span><span class="nb">pow</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| </div> |
| <div class="section" id="kstep"> |
| <span id="kstep"></span><h4>kStep<a class="headerlink" href="#kstep" title="Permalink to this headline">¶</a></h4> |
| <p>The updater should be configured like</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">learning_rate</span> <span class="p">{</span> |
| <span class="n">base_lr</span> <span class="p">:</span> <span class="nb">float</span> |
| <span class="n">step_conf</span> <span class="p">{</span> |
| <span class="n">change_freq</span><span class="p">:</span> <span class="nb">int</span> |
| <span class="n">gamma</span><span class="p">:</span> <span class="nb">float</span> |
| <span class="p">}</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| <p>The learning rate for <code class="docutils literal"><span class="pre">step</span></code> is</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">lr</span> <span class="o">=</span> <span class="n">base_lr</span> <span class="o">*</span> <span class="n">gamma</span><span class="o">^</span> <span class="p">(</span><span class="n">step</span> <span class="o">/</span> <span class="n">change_freq</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| </div> |
| <div class="section" id="kfixedstep"> |
| <span id="kfixedstep"></span><h4>kFixedStep<a class="headerlink" href="#kfixedstep" title="Permalink to this headline">¶</a></h4> |
| <p>The updater should be configured like</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">learning_rate</span> <span class="p">{</span> |
| <span class="n">fixedstep_conf</span> <span class="p">{</span> |
| <span class="n">step</span><span class="p">:</span> <span class="nb">int</span> |
| <span class="n">step_lr</span><span class="p">:</span> <span class="nb">float</span> |
| |
| <span class="n">step</span><span class="p">:</span> <span class="nb">int</span> |
| <span class="n">step_lr</span><span class="p">:</span> <span class="nb">float</span> |
| |
| <span class="o">...</span> |
| <span class="p">}</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| <p>Denote the i-th tuple as (step[i], step_lr[i]), then the learning rate for |
| <code class="docutils literal"><span class="pre">step</span></code> is,</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">step_lr</span><span class="p">[</span><span class="n">k</span><span class="p">]</span> |
| </pre></div> |
| </div> |
| <p>where step[k] is the smallest number that is larger than <code class="docutils literal"><span class="pre">step</span></code>.</p> |
| </div> |
| </div> |
| </div> |
| <div class="section" id="advanced-user-guide"> |
| <span id="advanced-user-guide"></span><h2>Advanced user guide<a class="headerlink" href="#advanced-user-guide" title="Permalink to this headline">¶</a></h2> |
| <div class="section" id="implementing-a-new-updater-subclass"> |
| <span id="implementing-a-new-updater-subclass"></span><h3>Implementing a new Updater subclass<a class="headerlink" href="#implementing-a-new-updater-subclass" title="Permalink to this headline">¶</a></h3> |
| <p>The base Updater class has one virtual function,</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Updater</span><span class="p">{</span> |
| <span class="n">public</span><span class="p">:</span> |
| <span class="n">virtual</span> <span class="n">void</span> <span class="n">Update</span><span class="p">(</span><span class="nb">int</span> <span class="n">step</span><span class="p">,</span> <span class="n">Param</span><span class="o">*</span> <span class="n">param</span><span class="p">,</span> <span class="nb">float</span> <span class="n">grad_scale</span> <span class="o">=</span> <span class="mf">1.0</span><span class="n">f</span><span class="p">)</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> |
| |
| <span class="n">protected</span><span class="p">:</span> |
| <span class="n">UpdaterProto</span> <span class="n">proto_</span><span class="p">;</span> |
| <span class="n">LRGenerator</span> <span class="n">lr_gen_</span><span class="p">;</span> |
| <span class="p">};</span> |
| </pre></div> |
| </div> |
| <p>It updates the values of the <code class="docutils literal"><span class="pre">param</span></code> based on its gradients. The <code class="docutils literal"><span class="pre">step</span></code> |
| argument is for deciding the learning rate which may change through time |
| (step). <code class="docutils literal"><span class="pre">grad_scale</span></code> scales the original gradient values. This function is |
| called by servers once it receives all gradients for the same <code class="docutils literal"><span class="pre">Param</span></code> object.</p> |
| <p>To implement a new Updater subclass, users must override the <code class="docutils literal"><span class="pre">Update</span></code> function.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">FooUpdater</span> <span class="p">:</span> <span class="n">public</span> <span class="n">Updater</span> <span class="p">{</span> |
| <span class="n">void</span> <span class="n">Update</span><span class="p">(</span><span class="nb">int</span> <span class="n">step</span><span class="p">,</span> <span class="n">Param</span><span class="o">*</span> <span class="n">param</span><span class="p">,</span> <span class="nb">float</span> <span class="n">grad_scale</span> <span class="o">=</span> <span class="mf">1.0</span><span class="n">f</span><span class="p">)</span> <span class="n">override</span><span class="p">;</span> |
| <span class="p">};</span> |
| </pre></div> |
| </div> |
| <p>Configuration of this new updater can be declared similar to that of a new |
| layer,</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># in user.proto</span> |
| <span class="n">FooUpdaterProto</span> <span class="p">{</span> |
| <span class="n">optional</span> <span class="n">int32</span> <span class="n">c</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> |
| <span class="p">}</span> |
| |
| <span class="n">extend</span> <span class="n">UpdaterProto</span> <span class="p">{</span> |
| <span class="n">optional</span> <span class="n">FooUpdaterProto</span> <span class="n">fooupdater_conf</span><span class="o">=</span> <span class="mi">101</span><span class="p">;</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| <p>The new updater should be registered in the |
| <a class="reference external" href="programming-guide.html">main function</a></p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">driver</span><span class="o">.</span><span class="n">RegisterUpdater</span><span class="o"><</span><span class="n">FooUpdater</span><span class="o">></span><span class="p">(</span><span class="s2">"FooUpdater"</span><span class="p">);</span> |
| </pre></div> |
| </div> |
| <p>Users can then configure the job as</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># in job.conf</span> |
| <span class="n">updater</span> <span class="p">{</span> |
| <span class="n">user_type</span><span class="p">:</span> <span class="s2">"FooUpdater"</span> <span class="c1"># must use user_type with the same string identifier as the one used for registration</span> |
| <span class="n">fooupdater_conf</span> <span class="p">{</span> |
| <span class="n">c</span> <span class="p">:</span> <span class="mi">20</span><span class="p">;</span> |
| <span class="p">}</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| </div> |
| <div class="section" id="implementing-a-new-lrgenerator-subclass"> |
| <span id="implementing-a-new-lrgenerator-subclass"></span><h3>Implementing a new LRGenerator subclass<a class="headerlink" href="#implementing-a-new-lrgenerator-subclass" title="Permalink to this headline">¶</a></h3> |
| <p>The base <code class="docutils literal"><span class="pre">LRGenerator</span></code> is declared as,</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">virtual</span> <span class="nb">float</span> <span class="n">Get</span><span class="p">(</span><span class="nb">int</span> <span class="n">step</span><span class="p">);</span> |
| </pre></div> |
| </div> |
| <p>To implement a subclass, e.g., <code class="docutils literal"><span class="pre">FooLRGen</span></code>, users should declare it like</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">FooLRGen</span> <span class="p">:</span> <span class="n">public</span> <span class="n">LRGenerator</span> <span class="p">{</span> |
| <span class="n">public</span><span class="p">:</span> |
| <span class="nb">float</span> <span class="n">Get</span><span class="p">(</span><span class="nb">int</span> <span class="n">step</span><span class="p">)</span> <span class="n">override</span><span class="p">;</span> |
| <span class="p">};</span> |
| </pre></div> |
| </div> |
| <p>Configuration of <code class="docutils literal"><span class="pre">FooLRGen</span></code> can be defined using a protocol message,</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># in user.proto</span> |
| <span class="n">message</span> <span class="n">FooLRProto</span> <span class="p">{</span> |
| <span class="o">...</span> |
| <span class="p">}</span> |
| |
| <span class="n">extend</span> <span class="n">LRGenProto</span> <span class="p">{</span> |
| <span class="n">optional</span> <span class="n">FooLRProto</span> <span class="n">foolr_conf</span> <span class="o">=</span> <span class="mi">101</span><span class="p">;</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| <p>The configuration is then like,</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">learning_rate</span> <span class="p">{</span> |
| <span class="n">user_type</span> <span class="p">:</span> <span class="s2">"FooLR"</span> <span class="c1"># must use user_type with the same string identifier as the one used for registration</span> |
| <span class="n">base_lr</span><span class="p">:</span> <span class="nb">float</span> |
| <span class="n">foolr_conf</span> <span class="p">{</span> |
| <span class="o">...</span> |
| <span class="p">}</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| <p>Users have to register this subclass in the main function,</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span> <span class="n">driver</span><span class="o">.</span><span class="n">RegisterLRGenerator</span><span class="o"><</span><span class="n">FooLRGen</span><span class="p">,</span> <span class="n">std</span><span class="p">::</span><span class="n">string</span><span class="o">></span><span class="p">(</span><span class="s2">"FooLR"</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| </div> |
| </div> |
| </div> |
| |
| |
| </div> |
| </div> |
| <footer> |
| |
| |
| <hr/> |
| |
| <div role="contentinfo"> |
| <p> |
| © Copyright 2016 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.. |
| |
| </p> |
| </div> |
| Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. |
| |
| </footer> |
| |
| </div> |
| </div> |
| |
| </section> |
| |
| </div> |
| |
| |
| |
| |
| |
| <script type="text/javascript"> |
| var DOCUMENTATION_OPTIONS = { |
| URL_ROOT:'../', |
| VERSION:'0.3.0', |
| COLLAPSE_INDEX:false, |
| FILE_SUFFIX:'.html', |
| HAS_SOURCE: true |
| }; |
| </script> |
| <script type="text/javascript" src="../_static/jquery.js"></script> |
| <script type="text/javascript" src="../_static/underscore.js"></script> |
| <script type="text/javascript" src="../_static/doctools.js"></script> |
| |
| |
| |
| |
| |
| <script type="text/javascript" src="../_static/js/theme.js"></script> |
| |
| |
| |
| |
| <script type="text/javascript"> |
| jQuery(function () { |
| SphinxRtdTheme.StickyNav.enable(); |
| }); |
| </script> |
| |
| |
| |
| <div class="rst-versions shift-up" data-toggle="rst-versions" role="note" aria-label="versions"> |
| <img src="../_static/apache.jpg"> |
| |
| <span class="rst-current-version" data-toggle="rst-current-version"> |
| <span class="fa fa-book"> incubator-singa </span> |
| v: 0.3.0 |
| <span class="fa fa-caret-down"></span> |
| </span> |
| <div class="rst-other-versions"> |
| <dl> |
| <dt>Languages</dt> |
| <dd><a href="../../en/index.html">English</a></dd> |
| <dd><a href="../../zh/index.html">中文</a></dd> |
| <dd><a href="../../jp/index.html">日本語</a></dd> |
| <dd><a href="../../kr/index.html">한국어</a></dd> |
| </dl> |
| </div> |
| </div> |
| |
| <a href="https://github.com/apache/incubator-singa"> |
| <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" |
| src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png" |
| alt="Fork me on GitHub"> |
| </a> |
| |
| |
| |
| |
| </body> |
| </html> |
