| |
| |
| |
| <!DOCTYPE html> |
| <!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> |
| <!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> |
| <head> |
| <meta charset="utf-8"> |
| |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| <title>CNN Example — incubator-singa 0.3.0 documentation</title> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> |
| |
| |
| |
| |
| |
| <link rel="top" title="incubator-singa 0.3.0 documentation" href="../index.html"/> |
| |
| |
| <script src="../_static/js/modernizr.min.js"></script> |
| |
| </head> |
| |
| <body class="wy-body-for-nav" role="document"> |
| |
| <div class="wy-grid-for-nav"> |
| |
| |
| <nav data-toggle="wy-nav-shift" class="wy-nav-side"> |
| <div class="wy-side-scroll"> |
| <div class="wy-side-nav-search"> |
| |
| |
| |
| <a href="../index.html" class="icon icon-home"> incubator-singa |
| |
| |
| |
| |
| <img src="../_static/singa.png" class="logo" /> |
| |
| </a> |
| |
| |
| |
| |
| <div class="version"> |
| 0.3.0 |
| </div> |
| |
| |
| |
| |
| <div role="search"> |
| <form id="rtd-search-form" class="wy-form" action="../search.html" method="get"> |
| <input type="text" name="q" placeholder="Search docs" /> |
| <input type="hidden" name="check_keywords" value="yes" /> |
| <input type="hidden" name="area" value="default" /> |
| </form> |
| </div> |
| |
| |
| </div> |
| |
| <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> |
| |
| |
| |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../downloads.html">Download SINGA</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="index.html">Documentation</a></li> |
| </ul> |
| <p class="caption"><span class="caption-text">Development</span></p> |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../develop/schedule.html">Development Schedule</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../develop/how-contribute.html">How to Contribute to SINGA</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../develop/contribute-code.html">How to Contribute Code</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../develop/contribute-docs.html">How to Contribute Documentation</a></li> |
| </ul> |
| <p class="caption"><span class="caption-text">Community</span></p> |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../community/source-repository.html">Source Repository</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../community/mail-lists.html">Project Mailing Lists</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../community/issue-tracking.html">Issue Tracking</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../community/team-list.html">The SINGA Team</a></li> |
| </ul> |
| |
| |
| |
| </div> |
| </div> |
| </nav> |
| |
| <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> |
| |
| |
| <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> |
| <i data-toggle="wy-nav-top" class="fa fa-bars"></i> |
| <a href="../index.html">incubator-singa</a> |
| </nav> |
| |
| |
| |
| <div class="wy-nav-content"> |
| <div class="rst-content"> |
| |
| |
| |
| |
| |
| |
| <div role="navigation" aria-label="breadcrumbs navigation"> |
| <ul class="wy-breadcrumbs"> |
| <li><a href="../index.html">Docs</a> »</li> |
| |
| <li>CNN Example</li> |
| <li class="wy-breadcrumbs-aside"> |
| |
| |
| |
| </li> |
| </ul> |
| <hr/> |
| </div> |
| <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> |
| <div itemprop="articleBody"> |
| |
| <div class="section" id="cnn-example"> |
| <span id="cnn-example"></span><h1>CNN Example<a class="headerlink" href="#cnn-example" title="Permalink to this headline">¶</a></h1> |
| <hr class="docutils" /> |
| <p>Convolutional neural network (CNN) is a type of feed-forward artificial neural |
| network widely used for image and video classification. In this example, we will |
| use a deep CNN model to do image classification for the |
| <a class="reference external" href="http://www.cs.toronto.edu/~kriz/cifar.html">CIFAR10 dataset</a>.</p> |
| <div class="section" id="running-instructions"> |
| <span id="running-instructions"></span><h2>Running instructions<a class="headerlink" href="#running-instructions" title="Permalink to this headline">¶</a></h2> |
| <p>Please refer to the <a class="reference external" href="installation.html">installation</a> page for |
| instructions on building SINGA, and the <a class="reference external" href="quick-start.html">quick start</a> |
| for instructions on starting zookeeper.</p> |
| <p>We have provided scripts for preparing the training and test dataset in <em>examples/cifar10/</em>.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span># in examples/cifar10 |
| $ cp Makefile.example Makefile |
| $ make download |
| $ make create |
| </pre></div> |
| </div> |
| <div class="section" id="training-on-cpu"> |
| <span id="training-on-cpu"></span><h3>Training on CPU<a class="headerlink" href="#training-on-cpu" title="Permalink to this headline">¶</a></h3> |
| <p>We can start the training by</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="o">./</span><span class="nb">bin</span><span class="o">/</span><span class="n">singa</span><span class="o">-</span><span class="n">run</span><span class="o">.</span><span class="n">sh</span> <span class="o">-</span><span class="n">conf</span> <span class="n">examples</span><span class="o">/</span><span class="n">cifar10</span><span class="o">/</span><span class="n">job</span><span class="o">.</span><span class="n">conf</span> |
| </pre></div> |
| </div> |
| <p>You should see output like</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">Record</span> <span class="n">job</span> <span class="n">information</span> <span class="n">to</span> <span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">singa</span><span class="o">-</span><span class="n">log</span><span class="o">/</span><span class="n">job</span><span class="o">-</span><span class="n">info</span><span class="o">/</span><span class="n">job</span><span class="o">-</span><span class="mi">2</span><span class="o">-</span><span class="mi">20150817</span><span class="o">-</span><span class="mi">055601</span> |
| <span class="n">Executing</span> <span class="p">:</span> <span class="o">./</span><span class="n">singa</span> <span class="o">-</span><span class="n">conf</span> <span class="o">/</span><span class="n">xxx</span><span class="o">/</span><span class="n">incubator</span><span class="o">-</span><span class="n">singa</span><span class="o">/</span><span class="n">examples</span><span class="o">/</span><span class="n">cifar10</span><span class="o">/</span><span class="n">job</span><span class="o">.</span><span class="n">conf</span> <span class="o">-</span><span class="n">singa_conf</span> <span class="o">/</span><span class="n">xxx</span><span class="o">/</span><span class="n">incubator</span><span class="o">-</span><span class="n">singa</span><span class="o">/</span><span class="n">conf</span><span class="o">/</span><span class="n">singa</span><span class="o">.</span><span class="n">conf</span> <span class="o">-</span><span class="n">singa_job</span> <span class="mi">2</span> |
| <span class="n">E0817</span> <span class="mi">06</span><span class="p">:</span><span class="mi">56</span><span class="p">:</span><span class="mf">18.868259</span> <span class="mi">33849</span> <span class="n">cluster</span><span class="o">.</span><span class="n">cc</span><span class="p">:</span><span class="mi">51</span><span class="p">]</span> <span class="n">proc</span> <span class="c1">#0 -> 192.168.5.128:49152 (pid = 33849)</span> |
| <span class="n">E0817</span> <span class="mi">06</span><span class="p">:</span><span class="mi">56</span><span class="p">:</span><span class="mf">18.928452</span> <span class="mi">33871</span> <span class="n">server</span><span class="o">.</span><span class="n">cc</span><span class="p">:</span><span class="mi">36</span><span class="p">]</span> <span class="n">Server</span> <span class="p">(</span><span class="n">group</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="nb">id</span> <span class="o">=</span> <span class="mi">0</span><span class="p">)</span> <span class="n">start</span> |
| <span class="n">E0817</span> <span class="mi">06</span><span class="p">:</span><span class="mi">56</span><span class="p">:</span><span class="mf">18.928469</span> <span class="mi">33872</span> <span class="n">worker</span><span class="o">.</span><span class="n">cc</span><span class="p">:</span><span class="mi">134</span><span class="p">]</span> <span class="n">Worker</span> <span class="p">(</span><span class="n">group</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="nb">id</span> <span class="o">=</span> <span class="mi">0</span><span class="p">)</span> <span class="n">start</span> |
| <span class="n">E0817</span> <span class="mi">06</span><span class="p">:</span><span class="mi">57</span><span class="p">:</span><span class="mf">13.657302</span> <span class="mi">33849</span> <span class="n">trainer</span><span class="o">.</span><span class="n">cc</span><span class="p">:</span><span class="mi">373</span><span class="p">]</span> <span class="n">Test</span> <span class="n">step</span><span class="o">-</span><span class="mi">0</span><span class="p">,</span> <span class="n">loss</span> <span class="p">:</span> <span class="mf">2.302588</span><span class="p">,</span> <span class="n">accuracy</span> <span class="p">:</span> <span class="mf">0.077900</span> |
| <span class="n">E0817</span> <span class="mi">06</span><span class="p">:</span><span class="mi">57</span><span class="p">:</span><span class="mf">17.626708</span> <span class="mi">33849</span> <span class="n">trainer</span><span class="o">.</span><span class="n">cc</span><span class="p">:</span><span class="mi">373</span><span class="p">]</span> <span class="n">Train</span> <span class="n">step</span><span class="o">-</span><span class="mi">0</span><span class="p">,</span> <span class="n">loss</span> <span class="p">:</span> <span class="mf">2.302578</span><span class="p">,</span> <span class="n">accuracy</span> <span class="p">:</span> <span class="mf">0.062500</span> |
| <span class="n">E0817</span> <span class="mi">06</span><span class="p">:</span><span class="mi">57</span><span class="p">:</span><span class="mf">24.142645</span> <span class="mi">33849</span> <span class="n">trainer</span><span class="o">.</span><span class="n">cc</span><span class="p">:</span><span class="mi">373</span><span class="p">]</span> <span class="n">Train</span> <span class="n">step</span><span class="o">-</span><span class="mi">30</span><span class="p">,</span> <span class="n">loss</span> <span class="p">:</span> <span class="mf">2.302404</span><span class="p">,</span> <span class="n">accuracy</span> <span class="p">:</span> <span class="mf">0.131250</span> |
| <span class="n">E0817</span> <span class="mi">06</span><span class="p">:</span><span class="mi">57</span><span class="p">:</span><span class="mf">30.813354</span> <span class="mi">33849</span> <span class="n">trainer</span><span class="o">.</span><span class="n">cc</span><span class="p">:</span><span class="mi">373</span><span class="p">]</span> <span class="n">Train</span> <span class="n">step</span><span class="o">-</span><span class="mi">60</span><span class="p">,</span> <span class="n">loss</span> <span class="p">:</span> <span class="mf">2.302248</span><span class="p">,</span> <span class="n">accuracy</span> <span class="p">:</span> <span class="mf">0.156250</span> |
| <span class="n">E0817</span> <span class="mi">06</span><span class="p">:</span><span class="mi">57</span><span class="p">:</span><span class="mf">37.556655</span> <span class="mi">33849</span> <span class="n">trainer</span><span class="o">.</span><span class="n">cc</span><span class="p">:</span><span class="mi">373</span><span class="p">]</span> <span class="n">Train</span> <span class="n">step</span><span class="o">-</span><span class="mi">90</span><span class="p">,</span> <span class="n">loss</span> <span class="p">:</span> <span class="mf">2.301849</span><span class="p">,</span> <span class="n">accuracy</span> <span class="p">:</span> <span class="mf">0.175000</span> |
| <span class="n">E0817</span> <span class="mi">06</span><span class="p">:</span><span class="mi">57</span><span class="p">:</span><span class="mf">44.971276</span> <span class="mi">33849</span> <span class="n">trainer</span><span class="o">.</span><span class="n">cc</span><span class="p">:</span><span class="mi">373</span><span class="p">]</span> <span class="n">Train</span> <span class="n">step</span><span class="o">-</span><span class="mi">120</span><span class="p">,</span> <span class="n">loss</span> <span class="p">:</span> <span class="mf">2.301077</span><span class="p">,</span> <span class="n">accuracy</span> <span class="p">:</span> <span class="mf">0.137500</span> |
| <span class="n">E0817</span> <span class="mi">06</span><span class="p">:</span><span class="mi">57</span><span class="p">:</span><span class="mf">51.801949</span> <span class="mi">33849</span> <span class="n">trainer</span><span class="o">.</span><span class="n">cc</span><span class="p">:</span><span class="mi">373</span><span class="p">]</span> <span class="n">Train</span> <span class="n">step</span><span class="o">-</span><span class="mi">150</span><span class="p">,</span> <span class="n">loss</span> <span class="p">:</span> <span class="mf">2.300410</span><span class="p">,</span> <span class="n">accuracy</span> <span class="p">:</span> <span class="mf">0.135417</span> |
| <span class="n">E0817</span> <span class="mi">06</span><span class="p">:</span><span class="mi">57</span><span class="p">:</span><span class="mf">58.682281</span> <span class="mi">33849</span> <span class="n">trainer</span><span class="o">.</span><span class="n">cc</span><span class="p">:</span><span class="mi">373</span><span class="p">]</span> <span class="n">Train</span> <span class="n">step</span><span class="o">-</span><span class="mi">180</span><span class="p">,</span> <span class="n">loss</span> <span class="p">:</span> <span class="mf">2.300067</span><span class="p">,</span> <span class="n">accuracy</span> <span class="p">:</span> <span class="mf">0.127083</span> |
| <span class="n">E0817</span> <span class="mi">06</span><span class="p">:</span><span class="mi">58</span><span class="p">:</span><span class="mf">05.578366</span> <span class="mi">33849</span> <span class="n">trainer</span><span class="o">.</span><span class="n">cc</span><span class="p">:</span><span class="mi">373</span><span class="p">]</span> <span class="n">Train</span> <span class="n">step</span><span class="o">-</span><span class="mi">210</span><span class="p">,</span> <span class="n">loss</span> <span class="p">:</span> <span class="mf">2.300143</span><span class="p">,</span> <span class="n">accuracy</span> <span class="p">:</span> <span class="mf">0.154167</span> |
| <span class="n">E0817</span> <span class="mi">06</span><span class="p">:</span><span class="mi">58</span><span class="p">:</span><span class="mf">12.518497</span> <span class="mi">33849</span> <span class="n">trainer</span><span class="o">.</span><span class="n">cc</span><span class="p">:</span><span class="mi">373</span><span class="p">]</span> <span class="n">Train</span> <span class="n">step</span><span class="o">-</span><span class="mi">240</span><span class="p">,</span> <span class="n">loss</span> <span class="p">:</span> <span class="mf">2.295912</span><span class="p">,</span> <span class="n">accuracy</span> <span class="p">:</span> <span class="mf">0.185417</span> |
| </pre></div> |
| </div> |
| <p>After training some steps (depends on the setting) or the job is |
| finished, SINGA will <a class="reference external" href="checkpoint.html">checkpoint</a> the model parameters.</p> |
| </div> |
| <div class="section" id="training-on-gpu"> |
| <span id="training-on-gpu"></span><h3>Training on GPU<a class="headerlink" href="#training-on-gpu" title="Permalink to this headline">¶</a></h3> |
| <p>Since version 0.2, we can train CNN models on GPU using cuDNN. Please refer to |
| the <a class="reference external" href="gpu.html">GPU page</a> for details on compiling SINGA with GPU and cuDNN. |
| The configuration file is similar to that for CPU training, except that the |
| cuDNN layers are used and the GPU device is configured.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="o">./</span><span class="nb">bin</span><span class="o">/</span><span class="n">singa</span><span class="o">-</span><span class="n">run</span><span class="o">.</span><span class="n">sh</span> <span class="o">-</span><span class="n">conf</span> <span class="n">examples</span><span class="o">/</span><span class="n">cifar10</span><span class="o">/</span><span class="n">cudnn</span><span class="o">.</span><span class="n">conf</span> |
| </pre></div> |
| </div> |
| </div> |
| <div class="section" id="training-using-python-script"> |
| <span id="training-using-python-script"></span><h3>Training using Python script<a class="headerlink" href="#training-using-python-script" title="Permalink to this headline">¶</a></h3> |
| <p>The python helpers coming with SINGA 0.2 make it easy to configure a training |
| job. For example the <em>job.conf</em> is replaced with a simple python script |
| <em>mnist_mlp.py</em> which has about 30 lines of code following the <a class="reference external" href="http://keras.io/">Keras API</a>.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span> <span class="c1"># on CPU</span> |
| <span class="o">./</span><span class="nb">bin</span><span class="o">/</span><span class="n">singa</span><span class="o">-</span><span class="n">run</span><span class="o">.</span><span class="n">sh</span> <span class="o">-</span><span class="n">exec</span> <span class="n">tool</span><span class="o">/</span><span class="n">python</span><span class="o">/</span><span class="n">examples</span><span class="o">/</span><span class="n">cifar10_cnn</span><span class="o">.</span><span class="n">py</span> |
| <span class="c1"># on GPU</span> |
| <span class="o">./</span><span class="nb">bin</span><span class="o">/</span><span class="n">singa</span><span class="o">-</span><span class="n">run</span><span class="o">.</span><span class="n">sh</span> <span class="o">-</span><span class="n">exec</span> <span class="n">tool</span><span class="o">/</span><span class="n">python</span><span class="o">/</span><span class="n">examples</span><span class="o">/</span><span class="n">cifar10_cnn_cudnn</span><span class="o">.</span><span class="n">py</span> |
| </pre></div> |
| </div> |
| </div> |
| </div> |
| <div class="section" id="details"> |
| <span id="details"></span><h2>Details<a class="headerlink" href="#details" title="Permalink to this headline">¶</a></h2> |
| <p>To train a model in SINGA, you need to prepare the datasets, |
| and a job configuration which specifies the neural net structure, training |
| algorithm (BP or CD), SGD update algorithm (e.g. Adagrad), |
| number of training/test steps, etc.</p> |
| <div class="section" id="data-preparation"> |
| <span id="data-preparation"></span><h3>Data preparation<a class="headerlink" href="#data-preparation" title="Permalink to this headline">¶</a></h3> |
| <p>Before using SINGA, you need to write a program to convert the dataset |
| into a format that SINGA can read. Please refer to the |
| <a class="reference external" href="data.html#example---cifar-dataset">Data Preparation</a> to get details about |
| preparing this CIFAR10 dataset.</p> |
| </div> |
| <div class="section" id="neural-net"> |
| <span id="neural-net"></span><h3>Neural net<a class="headerlink" href="#neural-net" title="Permalink to this headline">¶</a></h3> |
| <p>Figure 1 shows the net structure of the CNN model we used in this example, which is |
| set following <a class="reference external" href="https://code.google.com/p/cuda-convnet/source/browse/trunk/example-layers/layers-18pct.cfg.">Alex</a> |
| The dashed circle represents one feature transformation stage, which generally |
| has four layers as shown in the figure. Sometimes the rectifier layer and normalization layer |
| are omitted or swapped in one stage. For this example, there are 3 such stages.</p> |
| <p>Next we follow the guide in <a class="reference external" href="neural-net.html">neural net page</a> |
| and <a class="reference external" href="layer.html">layer page</a> to write the neural net configuration.</p> |
| <div style = "text-align: center"> |
| <img src = "../_static/images/example-cnn.png" style = "width: 200px"> <br/> |
| <strong>Figure 1 - Net structure of the CNN example.</strong></img> |
| </div><ul> |
| <li><p class="first">We configure an input layer to read the training/testing records from a disk file.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span> <span class="n">layer</span><span class="p">{</span> |
| <span class="n">name</span><span class="p">:</span> <span class="s2">"data"</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="n">kRecordInput</span> |
| <span class="n">store_conf</span> <span class="p">{</span> |
| <span class="n">backend</span><span class="p">:</span> <span class="s2">"kvfile"</span> |
| <span class="n">path</span><span class="p">:</span> <span class="s2">"examples/cifar10/train_data.bin"</span> |
| <span class="n">mean_file</span><span class="p">:</span> <span class="s2">"examples/cifar10/image_mean.bin"</span> |
| <span class="n">batchsize</span><span class="p">:</span> <span class="mi">64</span> |
| <span class="n">random_skip</span><span class="p">:</span> <span class="mi">5000</span> |
| <span class="n">shape</span><span class="p">:</span> <span class="mi">3</span> |
| <span class="n">shape</span><span class="p">:</span> <span class="mi">32</span> |
| <span class="n">shape</span><span class="p">:</span> <span class="mi">32</span> |
| <span class="p">}</span> |
| <span class="n">exclude</span><span class="p">:</span> <span class="n">kTest</span> <span class="c1"># exclude this layer for the testing net</span> |
| <span class="p">}</span> |
| <span class="n">layer</span><span class="p">{</span> |
| <span class="n">name</span><span class="p">:</span> <span class="s2">"data"</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="n">kRecordInput</span> |
| <span class="n">store_conf</span> <span class="p">{</span> |
| <span class="n">backend</span><span class="p">:</span> <span class="s2">"kvfile"</span> |
| <span class="n">path</span><span class="p">:</span> <span class="s2">"examples/cifar10/test_data.bin"</span> |
| <span class="n">mean_file</span><span class="p">:</span> <span class="s2">"examples/cifar10/image_mean.bin"</span> |
| <span class="n">batchsize</span><span class="p">:</span> <span class="mi">100</span> |
| <span class="n">shape</span><span class="p">:</span> <span class="mi">3</span> |
| <span class="n">shape</span><span class="p">:</span> <span class="mi">32</span> |
| <span class="n">shape</span><span class="p">:</span> <span class="mi">32</span> |
| <span class="p">}</span> |
| <span class="n">exclude</span><span class="p">:</span> <span class="n">kTrain</span> <span class="c1"># exclude this layer for the training net</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| </li> |
| </ul> |
| <ul> |
| <li><p class="first">We configure layers for the feature transformation as follows |
| (all layers are built-in layers in SINGA; hyper-parameters of these layers are set according to |
| <a class="reference external" href="https://code.google.com/p/cuda-convnet/source/browse/trunk/example-layers/layers-18pct.cfg">Alex’s setting</a>).</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span> <span class="n">layer</span> <span class="p">{</span> |
| <span class="n">name</span><span class="p">:</span> <span class="s2">"conv1"</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="n">kConvolution</span> |
| <span class="n">srclayers</span><span class="p">:</span> <span class="s2">"data"</span> |
| <span class="n">convolution_conf</span> <span class="p">{</span><span class="o">...</span> <span class="p">}</span> |
| <span class="o">...</span> |
| <span class="p">}</span> |
| <span class="n">layer</span> <span class="p">{</span> |
| <span class="n">name</span><span class="p">:</span> <span class="s2">"pool1"</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="n">kPooling</span> |
| <span class="n">srclayers</span><span class="p">:</span> <span class="s2">"conv1"</span> |
| <span class="n">pooling_conf</span> <span class="p">{</span><span class="o">...</span> <span class="p">}</span> |
| <span class="p">}</span> |
| <span class="n">layer</span> <span class="p">{</span> |
| <span class="n">name</span><span class="p">:</span> <span class="s2">"relu1"</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="n">kReLU</span> |
| <span class="n">srclayers</span><span class="p">:</span><span class="s2">"pool1"</span> |
| <span class="p">}</span> |
| <span class="n">layer</span> <span class="p">{</span> |
| <span class="n">name</span><span class="p">:</span> <span class="s2">"norm1"</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="n">kLRN</span> |
| <span class="n">lrn_conf</span> <span class="p">{</span><span class="o">...</span> <span class="p">}</span> |
| <span class="n">srclayers</span><span class="p">:</span><span class="s2">"relu1"</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| <p>The configurations for another 2 stages are omitted here.</p> |
| </li> |
| <li><p class="first">There is an <a class="reference external" href="layer.html#innerproductlayer">inner product layer</a> |
| after the 3 transformation stages, which is |
| configured with 10 output units, i.e., the number of total labels. The weight |
| matrix Param is configured with a large weight decay scale to reduce the over-fitting.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span> <span class="n">layer</span> <span class="p">{</span> |
| <span class="n">name</span><span class="p">:</span> <span class="s2">"ip1"</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="n">kInnerProduct</span> |
| <span class="n">srclayers</span><span class="p">:</span><span class="s2">"pool3"</span> |
| <span class="n">innerproduct_conf</span> <span class="p">{</span> |
| <span class="n">num_output</span><span class="p">:</span> <span class="mi">10</span> |
| <span class="p">}</span> |
| <span class="n">param</span> <span class="p">{</span> |
| <span class="n">name</span><span class="p">:</span> <span class="s2">"w4"</span> |
| <span class="n">wd_scale</span><span class="p">:</span><span class="mi">250</span> |
| <span class="o">...</span> |
| <span class="p">}</span> |
| <span class="n">param</span> <span class="p">{</span> |
| <span class="n">name</span><span class="p">:</span> <span class="s2">"b4"</span> |
| <span class="o">...</span> |
| <span class="p">}</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| </li> |
| <li><p class="first">The last layer is a <a class="reference external" href="layer.html#softmaxloss">Softmax loss layer</a></p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span> <span class="n">layer</span><span class="p">{</span> |
| <span class="n">name</span><span class="p">:</span> <span class="s2">"loss"</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="n">kSoftmaxLoss</span> |
| <span class="n">softmaxloss_conf</span><span class="p">{</span> <span class="n">topk</span><span class="p">:</span><span class="mi">1</span> <span class="p">}</span> |
| <span class="n">srclayers</span><span class="p">:</span><span class="s2">"ip1"</span> |
| <span class="n">srclayers</span><span class="p">:</span> <span class="s2">"data"</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| </li> |
| </ul> |
| </div> |
| <div class="section" id="updater"> |
| <span id="updater"></span><h3>Updater<a class="headerlink" href="#updater" title="Permalink to this headline">¶</a></h3> |
| <p>The <a class="reference external" href="updater.html#updater">normal SGD updater</a> is selected. |
| The learning rate is changed like going down stairs, and is configured using the |
| <a class="reference external" href="updater.html#kfixedstep">kFixedStep</a> type.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span> <span class="n">updater</span><span class="p">{</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="n">kSGD</span> |
| <span class="n">weight_decay</span><span class="p">:</span><span class="mf">0.004</span> |
| <span class="n">learning_rate</span> <span class="p">{</span> |
| <span class="nb">type</span><span class="p">:</span> <span class="n">kFixedStep</span> |
| <span class="n">fixedstep_conf</span><span class="p">:{</span> |
| <span class="n">step</span><span class="p">:</span><span class="mi">0</span> <span class="c1"># lr for step 0-60000 is 0.001</span> |
| <span class="n">step</span><span class="p">:</span><span class="mi">60000</span> <span class="c1"># lr for step 60000-65000 is 0.0001</span> |
| <span class="n">step</span><span class="p">:</span><span class="mi">65000</span> <span class="c1"># lr for step 650000- is 0.00001</span> |
| <span class="n">step_lr</span><span class="p">:</span><span class="mf">0.001</span> |
| <span class="n">step_lr</span><span class="p">:</span><span class="mf">0.0001</span> |
| <span class="n">step_lr</span><span class="p">:</span><span class="mf">0.00001</span> |
| <span class="p">}</span> |
| <span class="p">}</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| </div> |
| <div class="section" id="trainonebatch-algorithm"> |
| <span id="trainonebatch-algorithm"></span><h3>TrainOneBatch algorithm<a class="headerlink" href="#trainonebatch-algorithm" title="Permalink to this headline">¶</a></h3> |
| <p>The CNN model is a feed forward model, thus should be configured to use the |
| <a class="reference external" href="train-one-batch.html#back-propagation">Back-propagation algorithm</a>.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">train_one_batch</span> <span class="p">{</span> |
| <span class="n">alg</span><span class="p">:</span> <span class="n">kBP</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| </div> |
| <div class="section" id="cluster-setting"> |
| <span id="cluster-setting"></span><h3>Cluster setting<a class="headerlink" href="#cluster-setting" title="Permalink to this headline">¶</a></h3> |
| <p>The following configuration set a single worker and server for training. |
| <a class="reference external" href="frameworks.html">Training frameworks</a> page introduces configurations of a couple of distributed |
| training frameworks.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">cluster</span> <span class="p">{</span> |
| <span class="n">nworker_groups</span><span class="p">:</span> <span class="mi">1</span> |
| <span class="n">nserver_groups</span><span class="p">:</span> <span class="mi">1</span> |
| <span class="p">}</span> |
| </pre></div> |
| </div> |
| </div> |
| </div> |
| </div> |
| |
| |
| </div> |
| </div> |
| <footer> |
| |
| |
| <hr/> |
| |
| <div role="contentinfo"> |
| <p> |
| © Copyright 2016 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.. |
| |
| </p> |
| </div> |
| Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. |
| |
| </footer> |
| |
| </div> |
| </div> |
| |
| </section> |
| |
| </div> |
| |
| |
| |
| |
| |
| <script type="text/javascript"> |
| var DOCUMENTATION_OPTIONS = { |
| URL_ROOT:'../', |
| VERSION:'0.3.0', |
| COLLAPSE_INDEX:false, |
| FILE_SUFFIX:'.html', |
| HAS_SOURCE: true |
| }; |
| </script> |
| <script type="text/javascript" src="../_static/jquery.js"></script> |
| <script type="text/javascript" src="../_static/underscore.js"></script> |
| <script type="text/javascript" src="../_static/doctools.js"></script> |
| |
| |
| |
| |
| |
| <script type="text/javascript" src="../_static/js/theme.js"></script> |
| |
| |
| |
| |
| <script type="text/javascript"> |
| jQuery(function () { |
| SphinxRtdTheme.StickyNav.enable(); |
| }); |
| </script> |
| |
| |
| |
| <div class="rst-versions shift-up" data-toggle="rst-versions" role="note" aria-label="versions"> |
| <img src="../_static/apache.jpg"> |
| |
| <span class="rst-current-version" data-toggle="rst-current-version"> |
| <span class="fa fa-book"> incubator-singa </span> |
| v: 0.3.0 |
| <span class="fa fa-caret-down"></span> |
| </span> |
| <div class="rst-other-versions"> |
| <dl> |
| <dt>Languages</dt> |
| <dd><a href="../../en/index.html">English</a></dd> |
| <dd><a href="../../zh/index.html">中文</a></dd> |
| <dd><a href="../../jp/index.html">日本語</a></dd> |
| <dd><a href="../../kr/index.html">한국어</a></dd> |
| </dl> |
| </div> |
| </div> |
| |
| <a href="https://github.com/apache/incubator-singa"> |
| <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" |
| src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png" |
| alt="Fork me on GitHub"> |
| </a> |
| |
| |
| |
| |
| </body> |
| </html> |
