content/v0.3.0/en/docs/overview.html - singa-site - Git at Google



 <!DOCTYPE html>
 <!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
 <!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
 <head>
   <meta charset="utf-8">

   <meta name="viewport" content="width=device-width, initial-scale=1.0">

   <title>Introduction &mdash; incubator-singa 0.3.0 documentation</title>


     <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />


     <link rel="top" title="incubator-singa 0.3.0 documentation" href="../index.html"/>


   <script src="../_static/js/modernizr.min.js"></script>

 </head>

 <body class="wy-body-for-nav" role="document">

   <div class="wy-grid-for-nav">


     <nav data-toggle="wy-nav-shift" class="wy-nav-side">
       <div class="wy-side-scroll">
         <div class="wy-side-nav-search">


             <a href="../index.html" class="icon icon-home"> incubator-singa


             <img src="../_static/singa.png" class="logo" />

           </a>


               <div class="version">
                 0.3.0
               </div>


 <div role="search">
   <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
     <input type="text" name="q" placeholder="Search docs" />
     <input type="hidden" name="check_keywords" value="yes" />
     <input type="hidden" name="area" value="default" />
   </form>
 </div>


         </div>

         <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">


                 <ul>
 <li class="toctree-l1"><a class="reference internal" href="../downloads.html">Download SINGA</a></li>
 <li class="toctree-l1"><a class="reference internal" href="index.html">Documentation</a></li>
 </ul>
 <p class="caption"><span class="caption-text">Development</span></p>
 <ul>
 <li class="toctree-l1"><a class="reference internal" href="../develop/schedule.html">Development Schedule</a></li>
 <li class="toctree-l1"><a class="reference internal" href="../develop/how-contribute.html">How to Contribute to SINGA</a></li>
 <li class="toctree-l1"><a class="reference internal" href="../develop/contribute-code.html">How to Contribute Code</a></li>
 <li class="toctree-l1"><a class="reference internal" href="../develop/contribute-docs.html">How to Contribute Documentation</a></li>
 </ul>
 <p class="caption"><span class="caption-text">Community</span></p>
 <ul>
 <li class="toctree-l1"><a class="reference internal" href="../community/source-repository.html">Source Repository</a></li>
 <li class="toctree-l1"><a class="reference internal" href="../community/mail-lists.html">Project Mailing Lists</a></li>
 <li class="toctree-l1"><a class="reference internal" href="../community/issue-tracking.html">Issue Tracking</a></li>
 <li class="toctree-l1"><a class="reference internal" href="../community/team-list.html">The SINGA Team</a></li>
 </ul>


         </div>
       </div>
     </nav>

     <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">


       <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
         <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
         <a href="../index.html">incubator-singa</a>
       </nav>


       <div class="wy-nav-content">
         <div class="rst-content">


 <div role="navigation" aria-label="breadcrumbs navigation">
   <ul class="wy-breadcrumbs">
     <li><a href="../index.html">Docs</a> &raquo;</li>

     <li>Introduction</li>
       <li class="wy-breadcrumbs-aside">


       </li>
   </ul>
   <hr/>
 </div>
           <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
            <div itemprop="articleBody">

   <div class="section" id="introduction">
 <span id="introduction"></span><h1>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h1>
 <hr class="docutils" />
 <p>SINGA is a general distributed deep learning platform for training big deep
 learning models over large datasets. It is designed with an intuitive
 programming model based on the layer abstraction. A variety
 of popular deep learning models are supported, namely feed-forward models including
 convolutional neural networks (CNN), energy models like restricted Boltzmann
 machine (RBM), and recurrent neural networks (RNN). Many built-in layers are
 provided for users. SINGA architecture is
 sufficiently flexible to run synchronous, asynchronous and hybrid training
 frameworks.  SINGA
 also supports different neural net partitioning schemes to parallelize the
 training of large models, namely partitioning on batch dimension, feature
 dimension or hybrid partitioning.</p>
 <div class="section" id="goals">
 <span id="goals"></span><h2>Goals<a class="headerlink" href="#goals" title="Permalink to this headline">¶</a></h2>
 <p>As a distributed system, the first goal of SINGA is to have good scalability. In other
 words, SINGA is expected to reduce the total training time to achieve certain
 accuracy with more computing resources (i.e., machines).</p>
 <p>The second goal is to make SINGA easy to use.
 It is non-trivial for programmers to develop and train models with deep and
 complex model structures.  Distributed training further increases the burden of
 programmers, e.g., data and model partitioning, and network communication.  Hence it is essential to
 provide an easy to use programming model so that users can implement their deep
 learning models/algorithms without much awareness of the underlying distributed
 platform.</p>
 </div>
 <div class="section" id="principles">
 <span id="principles"></span><h2>Principles<a class="headerlink" href="#principles" title="Permalink to this headline">¶</a></h2>
 <p>Scalability is a challenging research problem for distributed deep learning
 training. SINGA provides a general architecture to exploit the scalability of
 different training frameworks. Synchronous training frameworks improve the
 efficiency of one training iteration, and
 asynchronous training frameworks improve the convergence rate. Given a fixed budget
 (e.g., cluster size), users can run a hybrid framework that maximizes the
 scalability by trading off between efficiency and convergence rate.</p>
 <p>SINGA comes with a programming model designed based on the layer abstraction, which
 is intuitive for deep learning models.  A variety of
 popular deep learning models can be expressed and trained using this programming model.</p>
 </div>
 <div class="section" id="system-overview">
 <span id="system-overview"></span><h2>System overview<a class="headerlink" href="#system-overview" title="Permalink to this headline">¶</a></h2>
 <p><img src="../_static/images/sgd.png" align="center" width="400px"/>
 <span><strong>Figure 1 - SGD flow.</strong></span></p>
 <p>Training a deep learning model is to find the optimal parameters involved in
 the transformation functions that generate good features for specific tasks.
 The goodness of a set of parameters is measured by a loss function, e.g.,
 <a class="reference external" href="https://en.wikipedia.org/wiki/Cross_entropy">Cross-Entropy Loss</a>. Since the
 loss functions are usually non-linear and non-convex, it is difficult to get a
 closed form solution. Typically, people use the stochastic gradient descent
 (SGD) algorithm, which randomly
 initializes the parameters and then iteratively updates them to reduce the loss
 as shown in Figure 1.</p>
 <p><img src="../_static/images/overview.png" align="center" width="400px"/>
 <span><strong>Figure 2 - SINGA overview.</strong></span></p>
 <p>SGD is used in SINGA to train
 parameters of deep learning models. The training workload is distributed over
 worker and server units as shown in Figure 2. In each
 iteration, every worker calls <em>TrainOneBatch</em> function to compute
 parameter gradients. <em>TrainOneBatch</em> takes a <em>NeuralNet</em> object
 representing the neural net, and visits layers of the <em>NeuralNet</em> in
 certain order. The resultant gradients are sent to the local stub that
 aggregates the requests and forwards them to corresponding servers for
 updating. Servers reply to workers with the updated parameters for the next
 iteration.</p>
 </div>
 <div class="section" id="job-submission">
 <span id="job-submission"></span><h2>Job submission<a class="headerlink" href="#job-submission" title="Permalink to this headline">¶</a></h2>
 <p>To submit a job in SINGA (i.e., training a deep learning model),
 users pass the job configuration to SINGA driver in the
 <a class="reference external" href="programming-guide.html">main function</a>. The job configuration
 specifies the four major components in Figure 2,</p>
 <ul class="simple">
 <li>a <a class="reference external" href="neural-net.html">NeuralNet</a> describing the neural net structure with the detailed layer setting and their connections;</li>
 <li>a <a class="reference external" href="train-one-batch.html">TrainOneBatch</a> algorithm which is tailored for different model categories;</li>
 <li>an <a class="reference external" href="updater.html">Updater</a> defining the protocol for updating parameters at the server side;</li>
 <li>a <a class="reference external" href="distributed-training.html">Cluster Topology</a> specifying the distributed architecture of workers and servers.</li>
 </ul>
 <p>This process is like the job submission in Hadoop, where users configure their
 jobs in the main function to set the mapper, reducer, etc.
 In Hadoop, users can configure their jobs with their own (or built-in) mapper and reducer; in SINGA, users
 can configure their jobs with their own (or built-in) layer, updater, etc.</p>
 </div>
 </div>


            </div>
           </div>
           <footer>


   <hr/>

   <div role="contentinfo">
     <p>
         &copy; Copyright 2016 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners..

     </p>
   </div>
   Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.

 </footer>

         </div>
       </div>

     </section>

   </div>


     <script type="text/javascript">
         var DOCUMENTATION_OPTIONS = {
             URL_ROOT:'../',
             VERSION:'0.3.0',
             COLLAPSE_INDEX:false,
             FILE_SUFFIX:'.html',
             HAS_SOURCE:  true
         };
     </script>
       <script type="text/javascript" src="../_static/jquery.js"></script>
       <script type="text/javascript" src="../_static/underscore.js"></script>
       <script type="text/javascript" src="../_static/doctools.js"></script>


     <script type="text/javascript" src="../_static/js/theme.js"></script>


   <script type="text/javascript">
       jQuery(function () {
           SphinxRtdTheme.StickyNav.enable();
       });
   </script>


 <div class="rst-versions shift-up" data-toggle="rst-versions" role="note" aria-label="versions">
     <img src="../_static/apache.jpg">

     <span class="rst-current-version" data-toggle="rst-current-version">
     <span class="fa fa-book"> incubator-singa </span>
     v: 0.3.0
     <span class="fa fa-caret-down"></span>
   </span>
     <div class="rst-other-versions">
         <dl>
             <dt>Languages</dt>
             <dd><a href="../../en/index.html">English</a></dd>
             <dd><a href="../../zh/index.html">中文</a></dd>
             <dd><a href="../../jp/index.html">日本語</a></dd>
             <dd><a href="../../kr/index.html">한국어</a></dd>
         </dl>
     </div>
 </div>

 <a href="https://github.com/apache/incubator-singa">
     <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
          src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"
          alt="Fork me on GitHub">
 </a>


 </body>
 </html>



	<!DOCTYPE html>
	<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
	<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
	<head>
	<meta charset="utf-8">

	<meta name="viewport" content="width=device-width, initial-scale=1.0">

	<title>Introduction — incubator-singa 0.3.0 documentation</title>















	<link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />





	<link rel="top" title="incubator-singa 0.3.0 documentation" href="../index.html"/>


	<script src="../_static/js/modernizr.min.js"></script>

	</head>

	<body class="wy-body-for-nav" role="document">

	<div class="wy-grid-for-nav">


	<nav data-toggle="wy-nav-shift" class="wy-nav-side">
	<div class="wy-side-scroll">
	<div class="wy-side-nav-search">



	<a href="../index.html" class="icon icon-home"> incubator-singa




	<img src="../_static/singa.png" class="logo" />

	</a>




	<div class="version">
	0.3.0
	</div>




	<div role="search">
	<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
	<input type="text" name="q" placeholder="Search docs" />
	<input type="hidden" name="check_keywords" value="yes" />
	<input type="hidden" name="area" value="default" />
	</form>
	</div>


	</div>

	<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">



	<ul>
	<li class="toctree-l1"><a class="reference internal" href="../downloads.html">Download SINGA</a></li>
	<li class="toctree-l1"><a class="reference internal" href="index.html">Documentation</a></li>
	</ul>
	<p class="caption"><span class="caption-text">Development</span></p>
	<ul>
	<li class="toctree-l1"><a class="reference internal" href="../develop/schedule.html">Development Schedule</a></li>
	<li class="toctree-l1"><a class="reference internal" href="../develop/how-contribute.html">How to Contribute to SINGA</a></li>
	<li class="toctree-l1"><a class="reference internal" href="../develop/contribute-code.html">How to Contribute Code</a></li>
	<li class="toctree-l1"><a class="reference internal" href="../develop/contribute-docs.html">How to Contribute Documentation</a></li>
	</ul>
	<p class="caption"><span class="caption-text">Community</span></p>
	<ul>
	<li class="toctree-l1"><a class="reference internal" href="../community/source-repository.html">Source Repository</a></li>
	<li class="toctree-l1"><a class="reference internal" href="../community/mail-lists.html">Project Mailing Lists</a></li>
	<li class="toctree-l1"><a class="reference internal" href="../community/issue-tracking.html">Issue Tracking</a></li>
	<li class="toctree-l1"><a class="reference internal" href="../community/team-list.html">The SINGA Team</a></li>
	</ul>



	</div>
	</div>
	</nav>

	<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">


	<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
	<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
	<a href="../index.html">incubator-singa</a>
	</nav>



	<div class="wy-nav-content">
	<div class="rst-content">






	<div role="navigation" aria-label="breadcrumbs navigation">
	<ul class="wy-breadcrumbs">
	<li><a href="../index.html">Docs</a> »</li>

	<li>Introduction</li>
	<li class="wy-breadcrumbs-aside">



	</li>
	</ul>
	<hr/>
	</div>
	<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
	<div itemprop="articleBody">

	<div class="section" id="introduction">
	<span id="introduction"></span><h1>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h1>
	<hr class="docutils" />
	<p>SINGA is a general distributed deep learning platform for training big deep
	learning models over large datasets. It is designed with an intuitive
	programming model based on the layer abstraction. A variety
	of popular deep learning models are supported, namely feed-forward models including
	convolutional neural networks (CNN), energy models like restricted Boltzmann
	machine (RBM), and recurrent neural networks (RNN). Many built-in layers are
	provided for users. SINGA architecture is
	sufficiently flexible to run synchronous, asynchronous and hybrid training
	frameworks. SINGA
	also supports different neural net partitioning schemes to parallelize the
	training of large models, namely partitioning on batch dimension, feature
	dimension or hybrid partitioning.</p>
	<div class="section" id="goals">
	<span id="goals"></span><h2>Goals<a class="headerlink" href="#goals" title="Permalink to this headline">¶</a></h2>
	<p>As a distributed system, the first goal of SINGA is to have good scalability. In other
	words, SINGA is expected to reduce the total training time to achieve certain
	accuracy with more computing resources (i.e., machines).</p>
	<p>The second goal is to make SINGA easy to use.
	It is non-trivial for programmers to develop and train models with deep and
	complex model structures. Distributed training further increases the burden of
	programmers, e.g., data and model partitioning, and network communication. Hence it is essential to
	provide an easy to use programming model so that users can implement their deep
	learning models/algorithms without much awareness of the underlying distributed
	platform.</p>
	</div>
	<div class="section" id="principles">
	<span id="principles"></span><h2>Principles<a class="headerlink" href="#principles" title="Permalink to this headline">¶</a></h2>
	<p>Scalability is a challenging research problem for distributed deep learning
	training. SINGA provides a general architecture to exploit the scalability of
	different training frameworks. Synchronous training frameworks improve the
	efficiency of one training iteration, and
	asynchronous training frameworks improve the convergence rate. Given a fixed budget
	(e.g., cluster size), users can run a hybrid framework that maximizes the
	scalability by trading off between efficiency and convergence rate.</p>
	<p>SINGA comes with a programming model designed based on the layer abstraction, which
	is intuitive for deep learning models. A variety of
	popular deep learning models can be expressed and trained using this programming model.</p>
	</div>
	<div class="section" id="system-overview">
	<span id="system-overview"></span><h2>System overview<a class="headerlink" href="#system-overview" title="Permalink to this headline">¶</a></h2>
	<p><img src="../_static/images/sgd.png" align="center" width="400px"/>
	<span><strong>Figure 1 - SGD flow.</strong></span></p>
	<p>Training a deep learning model is to find the optimal parameters involved in
	the transformation functions that generate good features for specific tasks.
	The goodness of a set of parameters is measured by a loss function, e.g.,
	<a class="reference external" href="https://en.wikipedia.org/wiki/Cross_entropy">Cross-Entropy Loss</a>. Since the
	loss functions are usually non-linear and non-convex, it is difficult to get a
	closed form solution. Typically, people use the stochastic gradient descent
	(SGD) algorithm, which randomly
	initializes the parameters and then iteratively updates them to reduce the loss
	as shown in Figure 1.</p>
	<p><img src="../_static/images/overview.png" align="center" width="400px"/>
	<span><strong>Figure 2 - SINGA overview.</strong></span></p>
	<p>SGD is used in SINGA to train
	parameters of deep learning models. The training workload is distributed over
	worker and server units as shown in Figure 2. In each
	iteration, every worker calls <em>TrainOneBatch</em> function to compute
	parameter gradients. <em>TrainOneBatch</em> takes a <em>NeuralNet</em> object
	representing the neural net, and visits layers of the <em>NeuralNet</em> in
	certain order. The resultant gradients are sent to the local stub that
	aggregates the requests and forwards them to corresponding servers for
	updating. Servers reply to workers with the updated parameters for the next
	iteration.</p>
	</div>
	<div class="section" id="job-submission">
	<span id="job-submission"></span><h2>Job submission<a class="headerlink" href="#job-submission" title="Permalink to this headline">¶</a></h2>
	<p>To submit a job in SINGA (i.e., training a deep learning model),
	users pass the job configuration to SINGA driver in the
	<a class="reference external" href="programming-guide.html">main function</a>. The job configuration
	specifies the four major components in Figure 2,</p>
	<ul class="simple">
	<li>a <a class="reference external" href="neural-net.html">NeuralNet</a> describing the neural net structure with the detailed layer setting and their connections;</li>
	<li>a <a class="reference external" href="train-one-batch.html">TrainOneBatch</a> algorithm which is tailored for different model categories;</li>
	<li>an <a class="reference external" href="updater.html">Updater</a> defining the protocol for updating parameters at the server side;</li>
	<li>a <a class="reference external" href="distributed-training.html">Cluster Topology</a> specifying the distributed architecture of workers and servers.</li>
	</ul>
	<p>This process is like the job submission in Hadoop, where users configure their
	jobs in the main function to set the mapper, reducer, etc.
	In Hadoop, users can configure their jobs with their own (or built-in) mapper and reducer; in SINGA, users
	can configure their jobs with their own (or built-in) layer, updater, etc.</p>
	</div>
	</div>


	</div>
	</div>
	<footer>


	<hr/>

	<div role="contentinfo">
	<p>
	© Copyright 2016 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners..

	</p>
	</div>
	Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.

	</footer>

	</div>
	</div>

	</section>

	</div>





	<script type="text/javascript">
	var DOCUMENTATION_OPTIONS = {
	URL_ROOT:'../',
	VERSION:'0.3.0',
	COLLAPSE_INDEX:false,
	FILE_SUFFIX:'.html',
	HAS_SOURCE: true
	};
	</script>
	<script type="text/javascript" src="../_static/jquery.js"></script>
	<script type="text/javascript" src="../_static/underscore.js"></script>
	<script type="text/javascript" src="../_static/doctools.js"></script>





	<script type="text/javascript" src="../_static/js/theme.js"></script>




	<script type="text/javascript">
	jQuery(function () {
	SphinxRtdTheme.StickyNav.enable();
	});
	</script>



	<div class="rst-versions shift-up" data-toggle="rst-versions" role="note" aria-label="versions">
	<img src="../_static/apache.jpg">

	<span class="rst-current-version" data-toggle="rst-current-version">
	<span class="fa fa-book"> incubator-singa </span>
	v: 0.3.0
	<span class="fa fa-caret-down"></span>
	</span>
	<div class="rst-other-versions">
	<dl>
	<dt>Languages</dt>
	<dd><a href="../../en/index.html">English</a></dd>
	<dd><a href="../../zh/index.html">中文</a></dd>
	<dd><a href="../../jp/index.html">日本語</a></dd>
	<dd><a href="../../kr/index.html">한국어</a></dd>
	</dl>
	</div>
	</div>

	<a href="https://github.com/apache/incubator-singa">
	<img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;"
	src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png"
	alt="Fork me on GitHub">
	</a>




	</body>
	</html>