Google Git
Sign in
apache / singa-site / be983bc44db8673b80f36c133d1ceb12926645fd / . / content / v1.1.0 / docs / neural-net.html
blob: 4b967496c0bd015948c1b92799ca72b34416db76 [file] [log] [blame]
<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Neural Net &mdash; incubator-singa 1.1.0 documentation</title>
<link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="top" title="incubator-singa 1.1.0 documentation" href="../index.html"/>
<link href="../_static/style.css" rel="stylesheet" type="text/css">
<script src="../_static/js/modernizr.min.js"></script>
</head>
<body class="wy-body-for-nav" role="document">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search">
<a href="../index.html" class="icon icon-home"> incubator-singa
<img src="../_static/singa.png" class="logo" />
</a>
<div class="version">
latest
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
<ul>
<li class="toctree-l1"><a class="reference internal" href="../downloads.html">Download SINGA</a></li>
<li class="toctree-l1"><a class="reference internal" href="index.html">Documentation</a></li>
</ul>
<p class="caption"><span class="caption-text">Development</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../develop/schedule.html">Development Schedule</a></li>
<li class="toctree-l1"><a class="reference internal" href="../develop/how-contribute.html">How to Contribute to SINGA</a></li>
<li class="toctree-l1"><a class="reference internal" href="../develop/contribute-code.html">How to Contribute Code</a></li>
<li class="toctree-l1"><a class="reference internal" href="../develop/contribute-docs.html">How to Contribute to Documentation</a></li>
</ul>
<p class="caption"><span class="caption-text">Community</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../community/source-repository.html">Source Repository</a></li>
<li class="toctree-l1"><a class="reference internal" href="../community/mail-lists.html">Project Mailing Lists</a></li>
<li class="toctree-l1"><a class="reference internal" href="../community/issue-tracking.html">Issue Tracking</a></li>
<li class="toctree-l1"><a class="reference internal" href="../community/team-list.html">The SINGA Team</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">incubator-singa</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html">Docs</a> &raquo;</li>
<li>Neural Net</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<div class="section" id="neural-net">
<span id="neural-net"></span><h1>Neural Net<a class="headerlink" href="#neural-net" title="Permalink to this headline"></a></h1>
<p><code class="docutils literal"><span class="pre">NeuralNet</span></code> in SINGA represents an instance of user&#8217;s neural net model. As the
neural net typically consists of a set of layers, <code class="docutils literal"><span class="pre">NeuralNet</span></code> comprises
a set of unidirectionally connected <a class="reference external" href="layer.html">Layer</a>s.
This page describes how to convert an user&#8217;s neural net into
the configuration of <code class="docutils literal"><span class="pre">NeuralNet</span></code>.</p>
<p><img src="../_static/images/model-category.png" align="center" width="200px"/>
<span><strong>Figure 1 - Categorization of popular deep learning models.</strong></span></p>
<div class="section" id="net-structure-configuration">
<span id="net-structure-configuration"></span><h2>Net structure configuration<a class="headerlink" href="#net-structure-configuration" title="Permalink to this headline"></a></h2>
<p>Users configure the <code class="docutils literal"><span class="pre">NeuralNet</span></code> by listing all layers of the neural net and
specifying each layer&#8217;s source layer names. Popular deep learning models can be
categorized as Figure 1. The subsequent sections give details for each
category.</p>
<div class="section" id="feed-forward-models">
<span id="feed-forward-models"></span><h3>Feed-forward models<a class="headerlink" href="#feed-forward-models" title="Permalink to this headline"></a></h3>
<div align = "left">
<img src="../_static/images/mlp-net.png" align="center" width="200px"/>
<span><strong>Figure 2 - Net structure of a MLP model.</strong></span>
</div><p>Feed-forward models, e.g., CNN and MLP, can easily get configured as their layer
connections are undirected without circles. The
configuration for the MLP model shown in Figure 1 is as follows,</p>
<div class="highlight-python"><div class="highlight"><pre>net {
layer {
name : &#39;data&quot;
type : kData
}
layer {
name : &#39;image&quot;
type : kImage
srclayer: &#39;data&#39;
}
layer {
name : &#39;label&quot;
type : kLabel
srclayer: &#39;data&#39;
}
layer {
name : &#39;hidden&quot;
type : kHidden
srclayer: &#39;image&#39;
}
layer {
name : &#39;softmax&quot;
type : kSoftmaxLoss
srclayer: &#39;hidden&#39;
srclayer: &#39;label&#39;
}
}
</pre></div>
</div>
</div>
<div class="section" id="energy-models">
<span id="energy-models"></span><h3>Energy models<a class="headerlink" href="#energy-models" title="Permalink to this headline"></a></h3>
<p><img src="../_static/images/rbm-rnn.png" align="center" width="500px"/>
<span><strong>Figure 3 - Convert connections in RBM and RNN.</strong></span></p>
<p>For energy models including RBM, DBM,
etc., their connections are undirected (i.e., Category B). To represent these models using
<code class="docutils literal"><span class="pre">NeuralNet</span></code>, users can simply replace each connection with two directed
connections, as shown in Figure 3a. In other words, for each pair of connected layers, their source
layer field should include each other&#8217;s name.
The full <a class="reference external" href="rbm.html">RBM example</a> has
detailed neural net configuration for a RBM model, which looks like</p>
<div class="highlight-python"><div class="highlight"><pre>net {
layer {
name : &quot;vis&quot;
type : kVisLayer
param {
name : &quot;w1&quot;
}
srclayer: &quot;hid&quot;
}
layer {
name : &quot;hid&quot;
type : kHidLayer
param {
name : &quot;w2&quot;
share_from: &quot;w1&quot;
}
srclayer: &quot;vis&quot;
}
}
</pre></div>
</div>
</div>
<div class="section" id="rnn-models">
<span id="rnn-models"></span><h3>RNN models<a class="headerlink" href="#rnn-models" title="Permalink to this headline"></a></h3>
<p>For recurrent neural networks (RNN), users can remove the recurrent connections
by unrolling the recurrent layer. For example, in Figure 3b, the original
layer is unrolled into a new layer with 4 internal layers. In this way, the
model is like a normal feed-forward model, thus can be configured similarly.
The <a class="reference external" href="rnn.html">RNN example</a> has a full neural net
configuration for a RNN model.</p>
</div>
</div>
<div class="section" id="configuration-for-multiple-nets">
<span id="configuration-for-multiple-nets"></span><h2>Configuration for multiple nets<a class="headerlink" href="#configuration-for-multiple-nets" title="Permalink to this headline"></a></h2>
<p>Typically, a training job includes three neural nets for
training, validation and test phase respectively. The three neural nets share most
layers except the data layer, loss layer or output layer, etc.. To avoid
redundant configurations for the shared layers, users can uses the <code class="docutils literal"><span class="pre">exclude</span></code>
filed to filter a layer in the neural net, e.g., the following layer will be
filtered when creating the testing <code class="docutils literal"><span class="pre">NeuralNet</span></code>.</p>
<div class="highlight-python"><div class="highlight"><pre>layer {
...
exclude : kTest # filter this layer for creating test net
}
</pre></div>
</div>
</div>
<div class="section" id="neural-net-partitioning">
<span id="neural-net-partitioning"></span><h2>Neural net partitioning<a class="headerlink" href="#neural-net-partitioning" title="Permalink to this headline"></a></h2>
<p>A neural net can be partitioned in different ways to distribute the training
over multiple workers.</p>
<div class="section" id="batch-and-feature-dimension">
<span id="batch-and-feature-dimension"></span><h3>Batch and feature dimension<a class="headerlink" href="#batch-and-feature-dimension" title="Permalink to this headline"></a></h3>
<p><img src="../_static/images/partition_fc.png" align="center" width="400px"/>
<span><strong>Figure 4 - Partitioning of a fully connected layer.</strong></span></p>
<p>Every layer&#8217;s feature blob is considered a matrix whose rows are feature
vectors. Thus, one layer can be split on two dimensions. Partitioning on
dimension 0 (also called batch dimension) slices the feature matrix by rows.
For instance, if the mini-batch size is 256 and the layer is partitioned into 2
sub-layers, each sub-layer would have 128 feature vectors in its feature blob.
Partitioning on this dimension has no effect on the parameters, as every
<a class="reference external" href="param.html">Param</a> object is replicated in the sub-layers. Partitioning on dimension
1 (also called feature dimension) slices the feature matrix by columns. For
example, suppose the original feature vector has 50 units, after partitioning
into 2 sub-layers, each sub-layer would have 25 units. This partitioning may
result in <a class="reference external" href="param.html">Param</a> object being split, as shown in
Figure 4. Both the bias vector and weight matrix are
partitioned into two sub-layers.</p>
</div>
<div class="section" id="partitioning-configuration">
<span id="partitioning-configuration"></span><h3>Partitioning configuration<a class="headerlink" href="#partitioning-configuration" title="Permalink to this headline"></a></h3>
<p>There are 4 partitioning schemes, whose configurations are give below,</p>
<ol>
<li><p class="first">Partitioning each singe layer into sub-layers on batch dimension (see
below). It is enabled by configuring the partition dimension of the layer to
0, e.g.,</p>
<div class="highlight-python"><div class="highlight"><pre> # with other fields omitted
layer {
partition_dim: 0
}
</pre></div>
</div>
</li>
<li><p class="first">Partitioning each singe layer into sub-layers on feature dimension (see
below). It is enabled by configuring the partition dimension of the layer to
1, e.g.,</p>
<div class="highlight-python"><div class="highlight"><pre> # with other fields omitted
layer {
partition_dim: 1
}
</pre></div>
</div>
</li>
<li><p class="first">Partitioning all layers into different subsets. It is enabled by
configuring the location ID of a layer, e.g.,</p>
<div class="highlight-python"><div class="highlight"><pre> # with other fields omitted
layer {
location: 1
}
layer {
location: 0
}
</pre></div>
</div>
</li>
</ol>
<ol>
<li><p class="first">Hybrid partitioning of strategy 1, 2 and 3. The hybrid partitioning is
useful for large models. An example application is to implement the
<a class="reference external" href="http://arxiv.org/abs/1404.5997">idea proposed by Alex</a>.
Hybrid partitioning is configured like,</p>
<div class="highlight-python"><div class="highlight"><pre> # with other fields omitted
layer {
location: 1
}
layer {
location: 0
}
layer {
partition_dim: 0
location: 0
}
layer {
partition_dim: 1
location: 0
}
</pre></div>
</div>
</li>
</ol>
<p>Currently SINGA supports strategy-2 well. Other partitioning strategies are
are under test and will be released in later version.</p>
</div>
</div>
<div class="section" id="parameter-sharing">
<span id="parameter-sharing"></span><h2>Parameter sharing<a class="headerlink" href="#parameter-sharing" title="Permalink to this headline"></a></h2>
<p>Parameters can be shared in two cases,</p>
<ul class="simple">
<li>sharing parameters among layers via user configuration. For example, the
visible layer and hidden layer of a RBM shares the weight matrix, which is configured through
the <code class="docutils literal"><span class="pre">share_from</span></code> field as shown in the above RBM configuration. The
configurations must be the same (except name) for shared parameters.</li>
<li>due to neural net partitioning, some <code class="docutils literal"><span class="pre">Param</span></code> objects are replicated into
different workers, e.g., partitioning one layer on batch dimension. These
workers share parameter values. SINGA controls this kind of parameter
sharing automatically, users do not need to do any configuration.</li>
<li>the <code class="docutils literal"><span class="pre">NeuralNet</span></code> for training and testing (and validation) share most layers
, thus share <code class="docutils literal"><span class="pre">Param</span></code> values.</li>
</ul>
<p>If the shared <code class="docutils literal"><span class="pre">Param</span></code> instances resident in the same process (may in different
threads), they use the same chunk of memory space for their values. But they
would have different memory spaces for their gradients. In fact, their
gradients will be averaged by the stub or server.</p>
</div>
<div class="section" id="advanced-user-guide">
<span id="advanced-user-guide"></span><h2>Advanced user guide<a class="headerlink" href="#advanced-user-guide" title="Permalink to this headline"></a></h2>
<div class="section" id="creation">
<span id="creation"></span><h3>Creation<a class="headerlink" href="#creation" title="Permalink to this headline"></a></h3>
<div class="highlight-python"><div class="highlight"><pre>static NeuralNet* NeuralNet::Create(const NetProto&amp; np, Phase phase, int num);
</pre></div>
</div>
<p>The above function creates a <code class="docutils literal"><span class="pre">NeuralNet</span></code> for a given phase, and returns a
pointer to the <code class="docutils literal"><span class="pre">NeuralNet</span></code> instance. The phase is in {kTrain,
kValidation, kTest}. <code class="docutils literal"><span class="pre">num</span></code> is used for net partitioning which indicates the
number of partitions. Typically, a training job includes three neural nets for
training, validation and test phase respectively. The three neural nets share most
layers except the data layer, loss layer or output layer, etc.. The <code class="docutils literal"><span class="pre">Create</span></code>
function takes in the full net configuration including layers for training,
validation and test. It removes layers for phases other than the specified
phase based on the <code class="docutils literal"><span class="pre">exclude</span></code> field in
<a class="reference external" href="layer.html">layer configuration</a>:</p>
<div class="highlight-python"><div class="highlight"><pre>layer {
...
exclude : kTest # filter this layer for creating test net
}
</pre></div>
</div>
<p>The filtered net configuration is passed to the constructor of <code class="docutils literal"><span class="pre">NeuralNet</span></code>:</p>
<div class="highlight-python"><div class="highlight"><pre>NeuralNet::NeuralNet(NetProto netproto, int npartitions);
</pre></div>
</div>
<p>The constructor creates a graph representing the net structure firstly in</p>
<div class="highlight-python"><div class="highlight"><pre>Graph* NeuralNet::CreateGraph(const NetProto&amp; netproto, int npartitions);
</pre></div>
</div>
<p>Next, it creates a layer for each node and connects layers if their nodes are
connected.</p>
<div class="highlight-python"><div class="highlight"><pre>void NeuralNet::CreateNetFromGraph(Graph* graph, int npartitions);
</pre></div>
</div>
<p>Since the <code class="docutils literal"><span class="pre">NeuralNet</span></code> instance may be shared among multiple workers, the
<code class="docutils literal"><span class="pre">Create</span></code> function returns a pointer to the <code class="docutils literal"><span class="pre">NeuralNet</span></code> instance .</p>
</div>
<div class="section" id="parameter-sharing">
<span id="id1"></span><h3>Parameter sharing<a class="headerlink" href="#parameter-sharing" title="Permalink to this headline"></a></h3>
<p><code class="docutils literal"><span class="pre">Param</span></code> sharing
is enabled by first sharing the Param configuration (in <code class="docutils literal"><span class="pre">NeuralNet::Create</span></code>)
to create two similar (e.g., the same shape) Param objects, and then calling
(in <code class="docutils literal"><span class="pre">NeuralNet::CreateNetFromGraph</span></code>),</p>
<div class="highlight-python"><div class="highlight"><pre>void Param::ShareFrom(const Param&amp; from);
</pre></div>
</div>
<p>It is also possible to share <code class="docutils literal"><span class="pre">Param</span></code>s of two nets, e.g., sharing parameters of
the training net and the test net,</p>
<div class="highlight-python"><div class="highlight"><pre>void NeuralNet:ShareParamsFrom(NeuralNet* other);
</pre></div>
</div>
<p>It will call <code class="docutils literal"><span class="pre">Param::ShareFrom</span></code> for each Param object.</p>
</div>
<div class="section" id="access-functions">
<span id="access-functions"></span><h3>Access functions<a class="headerlink" href="#access-functions" title="Permalink to this headline"></a></h3>
<p><code class="docutils literal"><span class="pre">NeuralNet</span></code> provides a couple of access function to get the layers and params
of the net:</p>
<div class="highlight-python"><div class="highlight"><pre>const std::vector&lt;Layer*&gt;&amp; layers() const;
const std::vector&lt;Param*&gt;&amp; params() const ;
Layer* name2layer(string name) const;
Param* paramid2param(int id) const;
</pre></div>
</div>
</div>
<div class="section" id="partitioning">
<span id="partitioning"></span><h3>Partitioning<a class="headerlink" href="#partitioning" title="Permalink to this headline"></a></h3>
<div class="section" id="implementation">
<span id="implementation"></span><h4>Implementation<a class="headerlink" href="#implementation" title="Permalink to this headline"></a></h4>
<p>SINGA partitions the neural net in <code class="docutils literal"><span class="pre">CreateGraph</span></code> function, which creates one
node for each (partitioned) layer. For example, if one layer&#8217;s partition
dimension is 0 or 1, then it creates <code class="docutils literal"><span class="pre">npartition</span></code> nodes for it; if the
partition dimension is -1, a single node is created, i.e., no partitioning.
Each node is assigned a partition (or location) ID. If the original layer is
configured with a location ID, then the ID is assigned to each newly created node.
These nodes are connected according to the connections of the original layers.
Some connection layers will be added automatically.
For instance, if two connected sub-layers are located at two
different workers, then a pair of bridge layers is inserted to transfer the
feature (and gradient) blob between them. When two layers are partitioned on
different dimensions, a concatenation layer which concatenates feature rows (or
columns) and a slice layer which slices feature rows (or columns) would be
inserted. These connection layers help making the network communication and
synchronization transparent to the users.</p>
</div>
<div class="section" id="dispatching-partitions-to-workers">
<span id="dispatching-partitions-to-workers"></span><h4>Dispatching partitions to workers<a class="headerlink" href="#dispatching-partitions-to-workers" title="Permalink to this headline"></a></h4>
<p>Each (partitioned) layer is assigned a location ID, based on which it is dispatched to one
worker. Particularly, the pointer to the <code class="docutils literal"><span class="pre">NeuralNet</span></code> instance is passed
to every worker within the same group, but each worker only computes over the
layers that have the same partition (or location) ID as the worker&#8217;s ID. When
every worker computes the gradients of the entire model parameters
(strategy-2), we refer to this process as data parallelism. When different
workers compute the gradients of different parameters (strategy-3 or
strategy-1), we call this process model parallelism. The hybrid partitioning
leads to hybrid parallelism where some workers compute the gradients of the
same subset of model parameters while other workers compute on different model
parameters. For example, to implement the hybrid parallelism in for the
<a class="reference external" href="http://arxiv.org/abs/1404.5997">DCNN model</a>, we set <code class="docutils literal"><span class="pre">partition_dim</span> <span class="pre">=</span> <span class="pre">0</span></code> for
lower layers and <code class="docutils literal"><span class="pre">partition_dim</span> <span class="pre">=</span> <span class="pre">1</span></code> for higher layers.</p>
</div>
</div>
</div>
</div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>
&copy; Copyright 2017 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners..
</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT:'../',
VERSION:'1.1.0',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<script type="text/javascript" src="../_static/js/theme.js"></script>
<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.StickyNav.enable();
});
</script>
<div class="rst-versions shift-up" data-toggle="rst-versions" role="note" aria-label="versions">
<span class="rst-current-version" data-toggle="rst-current-version">
<span class="fa fa-book"> incubator-singa </span>
v: latest
<span class="fa fa-caret-down"></span>
</span>
<div class="rst-other-versions">
<dl>
<dt>Languages</dt>
<dd><a href="../../en/index.html">English</a></dd>
<dd><a href="../../zh/index.html">中文</a></dd>
</dl>
<dl>
<dt>Versions</dt>
<dd><a href="http://singa.apache.org/v0.3.0/">0.3</a></dd>
<dd><a href="http://singa.apache.org/v1.1.0/">1.1</a></dd>
</dl>
</div>
<a href="http://incubator.apache.org/"> <img src= "../_static/apache.jpg"> </a>
</div>
<a href="https://github.com/apache/incubator-singa">
<img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"
alt="Fork me on GitHub">
</a>
</body>
</html>