content/v0.3.0/en/docs/neuralnet-partition.html - singa-site - Git at Google



 <!DOCTYPE html>
 <!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
 <!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
 <head>
   <meta charset="utf-8">

   <meta name="viewport" content="width=device-width, initial-scale=1.0">

   <title>Neural Net Partition &mdash; incubator-singa 0.3.0 documentation</title>


     <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />


     <link rel="top" title="incubator-singa 0.3.0 documentation" href="../index.html"/>


   <script src="../_static/js/modernizr.min.js"></script>

 </head>

 <body class="wy-body-for-nav" role="document">

   <div class="wy-grid-for-nav">


     <nav data-toggle="wy-nav-shift" class="wy-nav-side">
       <div class="wy-side-scroll">
         <div class="wy-side-nav-search">


             <a href="../index.html" class="icon icon-home"> incubator-singa


             <img src="../_static/singa.png" class="logo" />

           </a>


               <div class="version">
                 0.3.0
               </div>


 <div role="search">
   <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
     <input type="text" name="q" placeholder="Search docs" />
     <input type="hidden" name="check_keywords" value="yes" />
     <input type="hidden" name="area" value="default" />
   </form>
 </div>


         </div>

         <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">


                 <ul>
 <li class="toctree-l1"><a class="reference internal" href="../downloads.html">Download SINGA</a></li>
 <li class="toctree-l1"><a class="reference internal" href="index.html">Documentation</a></li>
 </ul>
 <p class="caption"><span class="caption-text">Development</span></p>
 <ul>
 <li class="toctree-l1"><a class="reference internal" href="../develop/schedule.html">Development Schedule</a></li>
 <li class="toctree-l1"><a class="reference internal" href="../develop/how-contribute.html">How to Contribute to SINGA</a></li>
 <li class="toctree-l1"><a class="reference internal" href="../develop/contribute-code.html">How to Contribute Code</a></li>
 <li class="toctree-l1"><a class="reference internal" href="../develop/contribute-docs.html">How to Contribute Documentation</a></li>
 </ul>
 <p class="caption"><span class="caption-text">Community</span></p>
 <ul>
 <li class="toctree-l1"><a class="reference internal" href="../community/source-repository.html">Source Repository</a></li>
 <li class="toctree-l1"><a class="reference internal" href="../community/mail-lists.html">Project Mailing Lists</a></li>
 <li class="toctree-l1"><a class="reference internal" href="../community/issue-tracking.html">Issue Tracking</a></li>
 <li class="toctree-l1"><a class="reference internal" href="../community/team-list.html">The SINGA Team</a></li>
 </ul>


         </div>
       </div>
     </nav>

     <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">


       <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
         <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
         <a href="../index.html">incubator-singa</a>
       </nav>


       <div class="wy-nav-content">
         <div class="rst-content">


 <div role="navigation" aria-label="breadcrumbs navigation">
   <ul class="wy-breadcrumbs">
     <li><a href="../index.html">Docs</a> &raquo;</li>

     <li>Neural Net Partition</li>
       <li class="wy-breadcrumbs-aside">


       </li>
   </ul>
   <hr/>
 </div>
           <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
            <div itemprop="articleBody">

   <div class="section" id="neural-net-partition">
 <span id="neural-net-partition"></span><h1>Neural Net Partition<a class="headerlink" href="#neural-net-partition" title="Permalink to this headline">¶</a></h1>
 <hr class="docutils" />
 <p>The purposes of partitioning neural network is to distribute the partitions onto
 different working units (e.g., threads or nodes, called workers in this article)
 and parallelize the processing.
 Another reason for partition is to handle large neural network which cannot be
 hold in a single node. For instance, to train models against images with high
 resolution we need large neural networks (in terms of training parameters).</p>
 <p>Since <em>Layer</em> is the first class citizen in SIGNA, we do the partition against
 layers. Specifically, we support partitions at two levels. First, users can configure
 the location (i.e., worker ID) of each layer. In this way, users assign one worker
 for each layer. Secondly, for one layer, we can partition its neurons or partition
 the instances (e.g, images). They are called layer partition and data partition
 respectively. We illustrate the two types of partitions using an simple convolutional neural network.</p>
 <p><img src="../_static/images/conv-mnist.png" style="width: 220px"/></p>
 <p>The above figure shows a convolutional neural network without any partition. It
 has 8 layers in total (one rectangular represents one layer). The first layer is
 DataLayer (data) which reads data from local disk files/databases (or HDFS). The second layer
 is a MnistLayer which parses the records from MNIST data to get the pixels of a batch
 of 8 images (each image is of size 28x28). The LabelLayer (label) parses the records to get the label
 of each image in the batch. The ConvolutionalLayer (conv1) transforms the input image to the
 shape of 8x27x27. The ReLULayer (relu1) conducts elementwise transformations. The PoolingLayer (pool1)
 sub-samples the images. The fc1 layer is fully connected with pool1 layer. It
 mulitplies each image with a weight matrix to generate a 10 dimension hidden feature which
 is then normalized by a SoftmaxLossLayer to get the prediction.</p>
 <p><img src="../_static/images/conv-mnist-datap.png" style="width: 1000px"/></p>
 <p>The above figure shows the convolutional neural network after partitioning all layers
 except the DataLayer and ParserLayers, into 3 partitions using data partition.
 The read layers process 4 images of the batch, the black and blue layers process 2 images
 respectively. Some helper layers, i.e., SliceLayer, ConcateLayer, BridgeSrcLayer,
 BridgeDstLayer and SplitLayer, are added automatically by our partition algorithm.
 Layers of the same color resident in the same worker. There would be data transferring
 across different workers at the boundary layers (i.e., BridgeSrcLayer and BridgeDstLayer),
 e.g., between s-slice-mnist-conv1 and d-slice-mnist-conv1.</p>
 <p><img src="../_static/images/conv-mnist-layerp.png" style="width: 1000px"/></p>
 <p>The above figure shows the convolutional neural network after partitioning all layers
 except the DataLayer and ParserLayers, into 2 partitions using layer partition. We can
 see that each layer processes all 8 images from the batch. But different partitions process
 different part of one image. For instance, the layer conv1-00 process only 4 channels. The other
 4 channels are processed by conv1-01 which residents in another worker.</p>
 <p>Since the partition is done at the layer level, we can apply different partitions for
 different layers to get a hybrid partition for the whole neural network. Moreover,
 we can also specify the layer locations to locate different layers to different workers.</p>
 </div>


            </div>
           </div>
           <footer>


   <hr/>

   <div role="contentinfo">
     <p>
         &copy; Copyright 2016 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners..

     </p>
   </div>
   Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.

 </footer>

         </div>
       </div>

     </section>

   </div>


     <script type="text/javascript">
         var DOCUMENTATION_OPTIONS = {
             URL_ROOT:'../',
             VERSION:'0.3.0',
             COLLAPSE_INDEX:false,
             FILE_SUFFIX:'.html',
             HAS_SOURCE:  true
         };
     </script>
       <script type="text/javascript" src="../_static/jquery.js"></script>
       <script type="text/javascript" src="../_static/underscore.js"></script>
       <script type="text/javascript" src="../_static/doctools.js"></script>


     <script type="text/javascript" src="../_static/js/theme.js"></script>


   <script type="text/javascript">
       jQuery(function () {
           SphinxRtdTheme.StickyNav.enable();
       });
   </script>


 <div class="rst-versions shift-up" data-toggle="rst-versions" role="note" aria-label="versions">
     <img src="../_static/apache.jpg">

     <span class="rst-current-version" data-toggle="rst-current-version">
     <span class="fa fa-book"> incubator-singa </span>
     v: 0.3.0
     <span class="fa fa-caret-down"></span>
   </span>
     <div class="rst-other-versions">
         <dl>
             <dt>Languages</dt>
             <dd><a href="../../en/index.html">English</a></dd>
             <dd><a href="../../zh/index.html">中文</a></dd>
             <dd><a href="../../jp/index.html">日本語</a></dd>
             <dd><a href="../../kr/index.html">한국어</a></dd>
         </dl>
     </div>
 </div>

 <a href="https://github.com/apache/incubator-singa">
     <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
          src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"
          alt="Fork me on GitHub">
 </a>


 </body>
 </html>



	<!DOCTYPE html>
	<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
	<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
	<head>
	<meta charset="utf-8">

	<meta name="viewport" content="width=device-width, initial-scale=1.0">

	<title>Neural Net Partition — incubator-singa 0.3.0 documentation</title>















	<link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />





	<link rel="top" title="incubator-singa 0.3.0 documentation" href="../index.html"/>


	<script src="../_static/js/modernizr.min.js"></script>

	</head>

	<body class="wy-body-for-nav" role="document">

	<div class="wy-grid-for-nav">


	<nav data-toggle="wy-nav-shift" class="wy-nav-side">
	<div class="wy-side-scroll">
	<div class="wy-side-nav-search">



	<a href="../index.html" class="icon icon-home"> incubator-singa




	<img src="../_static/singa.png" class="logo" />

	</a>




	<div class="version">
	0.3.0
	</div>




	<div role="search">
	<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
	<input type="text" name="q" placeholder="Search docs" />
	<input type="hidden" name="check_keywords" value="yes" />
	<input type="hidden" name="area" value="default" />
	</form>
	</div>


	</div>

	<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">



	<ul>
	<li class="toctree-l1"><a class="reference internal" href="../downloads.html">Download SINGA</a></li>
	<li class="toctree-l1"><a class="reference internal" href="index.html">Documentation</a></li>
	</ul>
	<p class="caption"><span class="caption-text">Development</span></p>
	<ul>
	<li class="toctree-l1"><a class="reference internal" href="../develop/schedule.html">Development Schedule</a></li>
	<li class="toctree-l1"><a class="reference internal" href="../develop/how-contribute.html">How to Contribute to SINGA</a></li>
	<li class="toctree-l1"><a class="reference internal" href="../develop/contribute-code.html">How to Contribute Code</a></li>
	<li class="toctree-l1"><a class="reference internal" href="../develop/contribute-docs.html">How to Contribute Documentation</a></li>
	</ul>
	<p class="caption"><span class="caption-text">Community</span></p>
	<ul>
	<li class="toctree-l1"><a class="reference internal" href="../community/source-repository.html">Source Repository</a></li>
	<li class="toctree-l1"><a class="reference internal" href="../community/mail-lists.html">Project Mailing Lists</a></li>
	<li class="toctree-l1"><a class="reference internal" href="../community/issue-tracking.html">Issue Tracking</a></li>
	<li class="toctree-l1"><a class="reference internal" href="../community/team-list.html">The SINGA Team</a></li>
	</ul>



	</div>
	</div>
	</nav>

	<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">


	<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
	<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
	<a href="../index.html">incubator-singa</a>
	</nav>



	<div class="wy-nav-content">
	<div class="rst-content">






	<div role="navigation" aria-label="breadcrumbs navigation">
	<ul class="wy-breadcrumbs">
	<li><a href="../index.html">Docs</a> »</li>

	<li>Neural Net Partition</li>
	<li class="wy-breadcrumbs-aside">



	</li>
	</ul>
	<hr/>
	</div>
	<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
	<div itemprop="articleBody">

	<div class="section" id="neural-net-partition">
	<span id="neural-net-partition"></span><h1>Neural Net Partition<a class="headerlink" href="#neural-net-partition" title="Permalink to this headline">¶</a></h1>
	<hr class="docutils" />
	<p>The purposes of partitioning neural network is to distribute the partitions onto
	different working units (e.g., threads or nodes, called workers in this article)
	and parallelize the processing.
	Another reason for partition is to handle large neural network which cannot be
	hold in a single node. For instance, to train models against images with high
	resolution we need large neural networks (in terms of training parameters).</p>
	<p>Since <em>Layer</em> is the first class citizen in SIGNA, we do the partition against
	layers. Specifically, we support partitions at two levels. First, users can configure
	the location (i.e., worker ID) of each layer. In this way, users assign one worker
	for each layer. Secondly, for one layer, we can partition its neurons or partition
	the instances (e.g, images). They are called layer partition and data partition
	respectively. We illustrate the two types of partitions using an simple convolutional neural network.</p>
	<p><img src="../_static/images/conv-mnist.png" style="width: 220px"/></p>
	<p>The above figure shows a convolutional neural network without any partition. It
	has 8 layers in total (one rectangular represents one layer). The first layer is
	DataLayer (data) which reads data from local disk files/databases (or HDFS). The second layer
	is a MnistLayer which parses the records from MNIST data to get the pixels of a batch
	of 8 images (each image is of size 28x28). The LabelLayer (label) parses the records to get the label
	of each image in the batch. The ConvolutionalLayer (conv1) transforms the input image to the
	shape of 8x27x27. The ReLULayer (relu1) conducts elementwise transformations. The PoolingLayer (pool1)
	sub-samples the images. The fc1 layer is fully connected with pool1 layer. It
	mulitplies each image with a weight matrix to generate a 10 dimension hidden feature which
	is then normalized by a SoftmaxLossLayer to get the prediction.</p>
	<p><img src="../_static/images/conv-mnist-datap.png" style="width: 1000px"/></p>
	<p>The above figure shows the convolutional neural network after partitioning all layers
	except the DataLayer and ParserLayers, into 3 partitions using data partition.
	The read layers process 4 images of the batch, the black and blue layers process 2 images
	respectively. Some helper layers, i.e., SliceLayer, ConcateLayer, BridgeSrcLayer,
	BridgeDstLayer and SplitLayer, are added automatically by our partition algorithm.
	Layers of the same color resident in the same worker. There would be data transferring
	across different workers at the boundary layers (i.e., BridgeSrcLayer and BridgeDstLayer),
	e.g., between s-slice-mnist-conv1 and d-slice-mnist-conv1.</p>
	<p><img src="../_static/images/conv-mnist-layerp.png" style="width: 1000px"/></p>
	<p>The above figure shows the convolutional neural network after partitioning all layers
	except the DataLayer and ParserLayers, into 2 partitions using layer partition. We can
	see that each layer processes all 8 images from the batch. But different partitions process
	different part of one image. For instance, the layer conv1-00 process only 4 channels. The other
	4 channels are processed by conv1-01 which residents in another worker.</p>
	<p>Since the partition is done at the layer level, we can apply different partitions for
	different layers to get a hybrid partition for the whole neural network. Moreover,
	we can also specify the layer locations to locate different layers to different workers.</p>
	</div>


	</div>
	</div>
	<footer>


	<hr/>

	<div role="contentinfo">
	<p>
	© Copyright 2016 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners..

	</p>
	</div>
	Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.

	</footer>

	</div>
	</div>

	</section>

	</div>





	<script type="text/javascript">
	var DOCUMENTATION_OPTIONS = {
	URL_ROOT:'../',
	VERSION:'0.3.0',
	COLLAPSE_INDEX:false,
	FILE_SUFFIX:'.html',
	HAS_SOURCE: true
	};
	</script>
	<script type="text/javascript" src="../_static/jquery.js"></script>
	<script type="text/javascript" src="../_static/underscore.js"></script>
	<script type="text/javascript" src="../_static/doctools.js"></script>





	<script type="text/javascript" src="../_static/js/theme.js"></script>




	<script type="text/javascript">
	jQuery(function () {
	SphinxRtdTheme.StickyNav.enable();
	});
	</script>



	<div class="rst-versions shift-up" data-toggle="rst-versions" role="note" aria-label="versions">
	<img src="../_static/apache.jpg">

	<span class="rst-current-version" data-toggle="rst-current-version">
	<span class="fa fa-book"> incubator-singa </span>
	v: 0.3.0
	<span class="fa fa-caret-down"></span>
	</span>
	<div class="rst-other-versions">
	<dl>
	<dt>Languages</dt>
	<dd><a href="../../en/index.html">English</a></dd>
	<dd><a href="../../zh/index.html">中文</a></dd>
	<dd><a href="../../jp/index.html">日本語</a></dd>
	<dd><a href="../../kr/index.html">한국어</a></dd>
	</dl>
	</div>
	</div>

	<a href="https://github.com/apache/incubator-singa">
	<img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
	src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"
	alt="Fork me on GitHub">
	</a>




	</body>
	</html>