Google Git
Sign in
apache / singa-site / be983bc44db8673b80f36c133d1ceb12926645fd / . / content / v0.3.0 / en / docs / general-rnn.html
blob: 12c456d73dd9b9fad16f30751b98b141969aa84d [file] [log] [blame]
<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>RNN in SINGA &mdash; incubator-singa 0.3.0 documentation</title>
<link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="top" title="incubator-singa 0.3.0 documentation" href="../index.html"/>
<script src="../_static/js/modernizr.min.js"></script>
</head>
<body class="wy-body-for-nav" role="document">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search">
<a href="../index.html" class="icon icon-home"> incubator-singa
<img src="../_static/singa.png" class="logo" />
</a>
<div class="version">
0.3.0
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
<ul>
<li class="toctree-l1"><a class="reference internal" href="../downloads.html">Download SINGA</a></li>
<li class="toctree-l1"><a class="reference internal" href="index.html">Documentation</a></li>
</ul>
<p class="caption"><span class="caption-text">Development</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../develop/schedule.html">Development Schedule</a></li>
<li class="toctree-l1"><a class="reference internal" href="../develop/how-contribute.html">How to Contribute to SINGA</a></li>
<li class="toctree-l1"><a class="reference internal" href="../develop/contribute-code.html">How to Contribute Code</a></li>
<li class="toctree-l1"><a class="reference internal" href="../develop/contribute-docs.html">How to Contribute Documentation</a></li>
</ul>
<p class="caption"><span class="caption-text">Community</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../community/source-repository.html">Source Repository</a></li>
<li class="toctree-l1"><a class="reference internal" href="../community/mail-lists.html">Project Mailing Lists</a></li>
<li class="toctree-l1"><a class="reference internal" href="../community/issue-tracking.html">Issue Tracking</a></li>
<li class="toctree-l1"><a class="reference internal" href="../community/team-list.html">The SINGA Team</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">incubator-singa</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html">Docs</a> &raquo;</li>
<li>RNN in SINGA</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<div class="section" id="rnn-in-singa">
<span id="rnn-in-singa"></span><h1>RNN in SINGA<a class="headerlink" href="#rnn-in-singa" title="Permalink to this headline"></a></h1>
<hr class="docutils" />
<p>Recurrent neural networks (RNN) are widely used for modelling sequential data,
e.g., natural language sentences. In this page, we describe how to implement a
RNN application (or model) using SINGA built-in RNN layers. We will
use the <a class="reference external" href="https://github.com/karpathy/char-rnn">char-rnn modle</a> as an example,
which trains over setences or source code, with each character as an input
unit. Particularly, we will train a RNN using GRU over
<a class="reference external" href="http://cs.stanford.edu/people/karpathy/char-rnn/">Linux kernel source code</a>.
After training, we expect to generate meaningful code from the model, like the
one shown by <a class="reference external" href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">Karpathy</a>.
There is a <a class="reference external" href="rnn.html">vanilla RNN example</a> for language modelling using user
defined RNN layers, which is different to using built-in RNN layers discribed
in this page.</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="o">/*</span>
<span class="o">*</span> <span class="n">If</span> <span class="n">this</span> <span class="n">error</span> <span class="ow">is</span> <span class="nb">set</span><span class="p">,</span> <span class="n">we</span> <span class="n">will</span> <span class="n">need</span> <span class="n">anything</span> <span class="n">right</span> <span class="n">after</span> <span class="n">that</span> <span class="n">BSD</span><span class="o">.</span>
<span class="o">*/</span>
<span class="n">static</span> <span class="n">void</span> <span class="n">action_new_function</span><span class="p">(</span><span class="n">struct</span> <span class="n">s_stat_info</span> <span class="o">*</span><span class="n">wb</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">unsigned</span> <span class="n">long</span> <span class="n">flags</span><span class="p">;</span>
<span class="nb">int</span> <span class="n">lel_idx_bit</span> <span class="o">=</span> <span class="n">e</span><span class="o">-&gt;</span><span class="n">edd</span><span class="p">,</span> <span class="o">*</span><span class="n">sys</span> <span class="o">&amp;</span> <span class="o">~</span><span class="p">((</span><span class="n">unsigned</span> <span class="n">long</span><span class="p">)</span> <span class="o">*</span><span class="n">FIRST_COMPAT</span><span class="p">);</span>
<span class="n">buf</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mh">0xFFFFFFFF</span> <span class="o">&amp;</span> <span class="p">(</span><span class="n">bit</span> <span class="o">&lt;&lt;</span> <span class="mi">4</span><span class="p">);</span>
<span class="nb">min</span><span class="p">(</span><span class="n">inc</span><span class="p">,</span> <span class="n">slist</span><span class="o">-&gt;</span><span class="nb">bytes</span><span class="p">);</span>
<span class="n">printk</span><span class="p">(</span><span class="n">KERN_WARNING</span> <span class="s2">&quot;Memory allocated </span><span class="si">%02x</span><span class="s2">/</span><span class="si">%02x</span><span class="s2">, &quot;</span>
<span class="s2">&quot;original MLL instead</span><span class="se">\n</span><span class="s2">&quot;</span><span class="p">),</span>
<span class="nb">min</span><span class="p">(</span><span class="nb">min</span><span class="p">(</span><span class="n">multi_run</span> <span class="o">-</span> <span class="n">s</span><span class="o">-&gt;</span><span class="nb">len</span><span class="p">,</span> <span class="nb">max</span><span class="p">)</span> <span class="o">*</span> <span class="n">num_data_in</span><span class="p">),</span>
<span class="n">frame_pos</span><span class="p">,</span> <span class="n">sz</span> <span class="o">+</span> <span class="n">first_seg</span><span class="p">);</span>
<span class="n">div_u64_w</span><span class="p">(</span><span class="n">val</span><span class="p">,</span> <span class="n">inb_p</span><span class="p">);</span>
<span class="n">spin_unlock</span><span class="p">(</span><span class="o">&amp;</span><span class="n">disk</span><span class="o">-&gt;</span><span class="n">queue_lock</span><span class="p">);</span>
<span class="n">mutex_unlock</span><span class="p">(</span><span class="o">&amp;</span><span class="n">s</span><span class="o">-&gt;</span><span class="n">sock</span><span class="o">-&gt;</span><span class="n">mutex</span><span class="p">);</span>
<span class="n">mutex_unlock</span><span class="p">(</span><span class="o">&amp;</span><span class="n">func</span><span class="o">-&gt;</span><span class="n">mutex</span><span class="p">);</span>
<span class="k">return</span> <span class="n">disassemble</span><span class="p">(</span><span class="n">info</span><span class="o">-&gt;</span><span class="n">pending_bh</span><span class="p">);</span>
<span class="p">}</span>
</pre></div>
</div>
<div class="section" id="user-configuration">
<span id="user-configuration"></span><h2>User configuration<a class="headerlink" href="#user-configuration" title="Permalink to this headline"></a></h2>
<p>The major diffences to the configuration of other models, e.g., feed-forward
models include,</p>
<ol class="simple">
<li>the training algorithm should be changed to BPTT (back-propagation through time).</li>
<li>the layers and their connections should be configured differently.</li>
</ol>
<p>The train one batch algorithm can be simply configured as</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">train_one_batch</span> <span class="p">{</span>
<span class="n">alg</span><span class="p">:</span> <span class="n">kBPTT</span>
<span class="p">}</span>
</pre></div>
</div>
<p>Next, we introduce the configuration of the neural net.</p>
<p><img src="../_static/images/char-rnn.png" style="width: 550px"/></p>
<p><strong> Fig.1 - Illustration of the structure of the Char-RNN model</strong></p><p>Fig.1 illustrates the net structure of the char-rnn model. The input layer
buffers all training data (the Linux kernel code is about 6MB). For each
iteration, it reads <code class="docutils literal"><span class="pre">unroll_len</span> <span class="pre">+1</span></code> (<code class="docutils literal"><span class="pre">unroll_len</span></code> is configured by users)
successive characters, e.g., &#8220;int a;&#8221;, and passes the first <code class="docutils literal"><span class="pre">unroll_len</span></code>
characters to <code class="docutils literal"><span class="pre">OneHotLayer</span></code>s (one per layer). Every <code class="docutils literal"><span class="pre">OneHotLayer</span></code> converts its
character into the one-hot vector representation. The input layer passes the
last <code class="docutils literal"><span class="pre">unroll_len</span></code> characters as labels to the <code class="docutils literal"><span class="pre">RNNLabelLayer</span></code> (the label of the
i-th character is the i+1 character, i.e., the objective is to predict the next
character). Each <code class="docutils literal"><span class="pre">GRULayer</span></code> receives an one-hot vector and the hidden feature
vector from its precedent layer. After some feature transformation, its own
feature vector is passed to an inner-product layer and its successive
<code class="docutils literal"><span class="pre">GRULayer</span></code>. The i-th SoftmaxLossLayer measures the cross-entropy loss for
predicting the i-th character. According to Karpathy, there could be another
stack of <code class="docutils literal"><span class="pre">GRULayer</span></code>s connecting the first stack of <code class="docutils literal"><span class="pre">GRULayer</span></code>s, which improves
the performance if there is enough training data. The layer configuration is
similar to that for other models, e.g., feed-forward models. The major
difference is on the connection configuration.</p>
<div class="section" id="unrolling-length">
<span id="unrolling-length"></span><h3>Unrolling length<a class="headerlink" href="#unrolling-length" title="Permalink to this headline"></a></h3>
<p>To model the long dependency, recurrent layers need to be unrolled many times,
denoted as <code class="docutils literal"><span class="pre">unroll_len</span></code> (i.e., 50). According to our unified neural net
representation, the neural net should have configurations for <code class="docutils literal"><span class="pre">unroll_len</span></code>
recurrent layers. It is tedious
to let users configure these layers manually. Hence, SINGA makes it a
configuration field for each layer. For example, to unroll the <code class="docutils literal"><span class="pre">GRULayer</span></code>,
users just configure it as,</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">layer</span> <span class="p">{</span>
<span class="nb">type</span><span class="p">:</span> <span class="n">kGRU</span>
<span class="n">unroll_len</span><span class="p">:</span> <span class="mi">50</span>
<span class="p">}</span>
</pre></div>
</div>
<p>Not only the <code class="docutils literal"><span class="pre">GRULayer</span></code> is unrolled, other layers like <code class="docutils literal"><span class="pre">InnerProductLayer</span></code> and
<code class="docutils literal"><span class="pre">SoftmaxLossLayer</span></code>, are also unrolled. To simplify the configuration, SINGA
provides a <code class="docutils literal"><span class="pre">unroll_len</span></code> field in the net configuration, which sets the
<code class="docutils literal"><span class="pre">unroll_len</span></code> of each layer configuration if the <code class="docutils literal"><span class="pre">unroll_len</span></code> is not configured
explicitly for that layer. For instance, SINGA would set the <code class="docutils literal"><span class="pre">unroll_len</span></code> of
the <code class="docutils literal"><span class="pre">GRULayer</span></code> to 50 implicitly for the following configuration.</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">net</span> <span class="p">{</span>
<span class="n">unroll_len</span><span class="p">:</span> <span class="mi">50</span>
<span class="n">layer</span> <span class="p">{</span>
<span class="nb">type</span><span class="p">:</span> <span class="n">kCharRNNInput</span>
<span class="n">unroll_len</span><span class="p">:</span> <span class="mi">1</span> <span class="o">//</span> <span class="n">configure</span> <span class="n">it</span> <span class="n">explicitly</span>
<span class="p">}</span>
<span class="n">layer</span> <span class="p">{</span>
<span class="nb">type</span><span class="p">:</span> <span class="n">kGRU</span>
<span class="o">//</span> <span class="n">no</span> <span class="n">configuration</span> <span class="k">for</span> <span class="n">unroll_len</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
</div>
</div>
<div class="section" id="connectiontype">
<span id="connectiontype"></span><h3>ConnectionType<a class="headerlink" href="#connectiontype" title="Permalink to this headline"></a></h3>
<p><img src="http://karpathy.github.io/assets/rnn/diags.jpeg" style="width: 550px"/></p>
<p><strong> Fig.1 - Different RNN structures from [Karpathy](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)</strong></p><p>There would be many types of connections between layers in RNN models as shown
by Karpathy in Fig.2. For each <code class="docutils literal"><span class="pre">srclayer</span></code>, there is a connection_type for it.
Taking the i-th <code class="docutils literal"><span class="pre">srclayer</span></code> as an example, if its connection type is,</p>
<ul class="simple">
<li>kOneToOne, then each unrolled layer is connected with one unrolled layer from the i-th <code class="docutils literal"><span class="pre">srclayer</span></code>.</li>
<li>kOneToALL, then each unrolled layer is connected with all unrolled layers from the i-th <code class="docutils literal"><span class="pre">srclayer</span></code>.</li>
</ul>
</div>
</div>
<div class="section" id="implementation">
<span id="implementation"></span><h2>Implementation<a class="headerlink" href="#implementation" title="Permalink to this headline"></a></h2>
<div class="section" id="neural-net-configuration-preprocessing">
<span id="neural-net-configuration-preprocessing"></span><h3>Neural net configuration preprocessing<a class="headerlink" href="#neural-net-configuration-preprocessing" title="Permalink to this headline"></a></h3>
<p>User configured neural net is preprocessed to unroll the recurrent layers,
i.e., duplicating the configuration of the <code class="docutils literal"><span class="pre">GRULayer</span></code>s, renaming the name of
each layer with unrolling index, and re-configuring the <code class="docutils literal"><span class="pre">srclayer</span></code> field. After
preprocessing, each layer&#8217;s name is changed to
<code class="docutils literal"><span class="pre">&lt;unrolling_index&gt;#&lt;user_configured_name&gt;.</span></code> Consequently, the (unrolled) neural
net configuration passed to NeuralNet class includes all layers and their
connections. The NeuralNet class creates and setup each layer in the same way
as for other models. For example, after partitioning, each layer&#8217;s name is
changed to <code class="docutils literal"><span class="pre">&lt;layer_name&gt;&#64;&lt;partition_index&gt;</span></code>. One difference is that it has some
special code for sharing Param data and grad Blobs for layers unrolled from the
same original layer.</p>
<p>Users can visualize the neural net structure using the Python script <code class="docutils literal"><span class="pre">tool/graph.py</span></code>
and the files in <em>WORKSPACE/visualization/</em>. For example, after the training program
is started,</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">python</span> <span class="n">tool</span><span class="o">/</span><span class="n">graph</span><span class="o">.</span><span class="n">py</span> <span class="n">examples</span><span class="o">/</span><span class="n">char</span><span class="o">-</span><span class="n">rnn</span><span class="o">/</span><span class="n">visualization</span><span class="o">/</span><span class="n">train_net</span><span class="o">.</span><span class="n">json</span>
</pre></div>
</div>
<p>The generated image file is shown in Fig.3 for <code class="docutils literal"><span class="pre">unroll_len=5</span></code>,</p>
<p><img src="../_static/images/char-rnn-net.jpg" style="width: 550px"/></p>
<p><strong> Fig.3 - Net structure generated by SINGA</strong></p></div>
<div class="section" id="bpttworker">
<span id="bpttworker"></span><h3>BPTTWorker<a class="headerlink" href="#bpttworker" title="Permalink to this headline"></a></h3>
<p>The BPTT (back-propagation through time) algorithm is typically used to compute
gradients of the objective loss w.r.t. parameters for RNN models. It forwards
propagates through all unrolled layers (i.e., timepoints) to compute features
of each layer, and backwards propagates to compute gradients of parameters. It
is the same as the BP algorithm for feed-forward models if the recurrent layers
are unrolled infinite times. In practice, due to the constraint of memory, the
truncated BPTT is widely used. It unrolls the recurrent layers a fixed
(truncated) times (controlled by <code class="docutils literal"><span class="pre">unroll_len</span></code>). In SINGA, a BPTTWorker is
provided to run the truncated BPTT algorithm for each mini-batch (i.e.,
iteration). The pseudo code is</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">BPTTWorker</span><span class="p">::</span><span class="n">Forward</span><span class="p">(</span><span class="n">phase</span><span class="p">,</span> <span class="n">net</span><span class="p">)</span> <span class="p">{</span>
<span class="k">for</span> <span class="n">each</span> <span class="n">layer</span> <span class="ow">in</span> <span class="n">net</span>
<span class="k">if</span> <span class="n">layer</span><span class="o">.</span><span class="n">unroll_index</span><span class="p">()</span> <span class="o">==</span> <span class="mi">0</span>
<span class="n">Get</span><span class="p">(</span><span class="n">layer</span><span class="o">.</span><span class="n">params</span><span class="p">());</span> <span class="o">//</span> <span class="n">fetch</span> <span class="n">params</span> <span class="n">values</span> <span class="kn">from</span> <span class="nn">servers</span>
<span class="n">srclayers</span> <span class="o">=</span> <span class="n">layer</span><span class="o">.</span><span class="n">srclayer</span><span class="p">();</span>
<span class="k">if</span> <span class="n">phase</span> <span class="o">&amp;</span> <span class="n">kTest</span>
<span class="n">srclayers</span><span class="o">.</span><span class="n">push_back</span><span class="p">(</span><span class="n">net</span><span class="o">-&gt;</span><span class="n">GetConextLayer</span><span class="p">(</span><span class="n">layer</span><span class="p">))</span>
<span class="n">layer</span><span class="o">.</span><span class="n">ComputeFeature</span><span class="p">(</span><span class="n">phase</span><span class="p">,</span> <span class="n">srclayers</span><span class="p">)</span>
<span class="p">}</span>
<span class="n">BPTTWorker</span><span class="p">::</span><span class="n">Backward</span><span class="p">(</span><span class="n">phase</span><span class="p">,</span> <span class="n">net</span><span class="p">)</span> <span class="p">{</span>
<span class="k">for</span> <span class="n">each</span> <span class="n">layer</span> <span class="ow">in</span> <span class="n">reverse</span><span class="p">(</span><span class="n">net</span><span class="o">.</span><span class="n">layers</span><span class="p">())</span>
<span class="n">layer</span><span class="o">.</span><span class="n">ComputeGradient</span><span class="p">(</span><span class="n">layer</span><span class="o">.</span><span class="n">srclayers</span><span class="p">())</span>
<span class="k">if</span> <span class="n">layer</span><span class="o">.</span><span class="n">unroll_index</span><span class="p">()</span> <span class="o">==</span> <span class="mi">0</span>
<span class="n">Update</span><span class="p">(</span><span class="n">layer</span><span class="o">.</span><span class="n">params</span><span class="p">());</span> <span class="o">//</span> <span class="n">send</span> <span class="n">params</span> <span class="n">gradients</span> <span class="n">to</span> <span class="n">servers</span>
<span class="p">}</span>
</pre></div>
</div>
<p>The testing phase is processed specially. Because the test phase may sample a
long sequence of data (e.g., sampling a piece of Linux kernel code), which
requires many unrolled layers (e.g., more than 1000 characters/layers). But we
cannot unroll the recurrent layers too many times due to memory constraint.
The special line add the 0-th unrolled layer as one of its own source layer.
Consequently, it dynamically adds a recurrent connection to the recurrent layer
(e.g., GRULayer). Then we can sample from the model for infinite times. Taking
the char-rnn model as an example, the test job can be configured as</p>
<div class="highlight-default"><div class="highlight"><pre><span></span>test_steps: 10000
train_one_batch {
Alg: kBPTT
}
net {
// do not set the unroll_len
layer {
// do not set the unroll_len
}
…
}
</pre></div>
</div>
<p>The instructions for <a class="reference external" href="test.html">running test</a> is the same for feed-forward
models.</p>
</div>
</div>
</div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>
&copy; Copyright 2016 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners..
</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT:'../',
VERSION:'0.3.0',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<script type="text/javascript" src="../_static/js/theme.js"></script>
<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.StickyNav.enable();
});
</script>
<div class="rst-versions shift-up" data-toggle="rst-versions" role="note" aria-label="versions">
<img src="../_static/apache.jpg">
<span class="rst-current-version" data-toggle="rst-current-version">
<span class="fa fa-book"> incubator-singa </span>
v: 0.3.0
<span class="fa fa-caret-down"></span>
</span>
<div class="rst-other-versions">
<dl>
<dt>Languages</dt>
<dd><a href="../../en/index.html">English</a></dd>
<dd><a href="../../zh/index.html">中文</a></dd>
<dd><a href="../../jp/index.html">日本語</a></dd>
<dd><a href="../../kr/index.html">한국어</a></dd>
</dl>
</div>
</div>
<a href="https://github.com/apache/incubator-singa">
<img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"
alt="Fork me on GitHub">
</a>
</body>
</html>