| |
| |
| |
| <!DOCTYPE html> |
| <!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> |
| <!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> |
| <head> |
| <meta charset="utf-8"> |
| |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| <title>Using HDFS with SINGA — incubator-singa 0.3.0 documentation</title> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> |
| |
| |
| |
| |
| |
| <link rel="top" title="incubator-singa 0.3.0 documentation" href="../index.html"/> |
| |
| |
| <script src="../_static/js/modernizr.min.js"></script> |
| |
| </head> |
| |
| <body class="wy-body-for-nav" role="document"> |
| |
| <div class="wy-grid-for-nav"> |
| |
| |
| <nav data-toggle="wy-nav-shift" class="wy-nav-side"> |
| <div class="wy-side-scroll"> |
| <div class="wy-side-nav-search"> |
| |
| |
| |
| <a href="../index.html" class="icon icon-home"> incubator-singa |
| |
| |
| |
| |
| <img src="../_static/singa.png" class="logo" /> |
| |
| </a> |
| |
| |
| |
| |
| <div class="version"> |
| 0.3.0 |
| </div> |
| |
| |
| |
| |
| <div role="search"> |
| <form id="rtd-search-form" class="wy-form" action="../search.html" method="get"> |
| <input type="text" name="q" placeholder="Search docs" /> |
| <input type="hidden" name="check_keywords" value="yes" /> |
| <input type="hidden" name="area" value="default" /> |
| </form> |
| </div> |
| |
| |
| </div> |
| |
| <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> |
| |
| |
| |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../downloads.html">Download SINGA</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="index.html">Documentation</a></li> |
| </ul> |
| <p class="caption"><span class="caption-text">Development</span></p> |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../develop/schedule.html">Development Schedule</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../develop/how-contribute.html">How to Contribute to SINGA</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../develop/contribute-code.html">How to Contribute Code</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../develop/contribute-docs.html">How to Contribute Documentation</a></li> |
| </ul> |
| <p class="caption"><span class="caption-text">Community</span></p> |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../community/source-repository.html">Source Repository</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../community/mail-lists.html">Project Mailing Lists</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../community/issue-tracking.html">Issue Tracking</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../community/team-list.html">The SINGA Team</a></li> |
| </ul> |
| |
| |
| |
| </div> |
| </div> |
| </nav> |
| |
| <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> |
| |
| |
| <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> |
| <i data-toggle="wy-nav-top" class="fa fa-bars"></i> |
| <a href="../index.html">incubator-singa</a> |
| </nav> |
| |
| |
| |
| <div class="wy-nav-content"> |
| <div class="rst-content"> |
| |
| |
| |
| |
| |
| |
| <div role="navigation" aria-label="breadcrumbs navigation"> |
| <ul class="wy-breadcrumbs"> |
| <li><a href="../index.html">Docs</a> »</li> |
| |
| <li>Using HDFS with SINGA</li> |
| <li class="wy-breadcrumbs-aside"> |
| |
| |
| |
| </li> |
| </ul> |
| <hr/> |
| </div> |
| <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> |
| <div itemprop="articleBody"> |
| |
| <div class="section" id="using-hdfs-with-singa"> |
| <span id="using-hdfs-with-singa"></span><h1>Using HDFS with SINGA<a class="headerlink" href="#using-hdfs-with-singa" title="Permalink to this headline">¶</a></h1> |
| <p>This guide explains how to make use of HDFS as the data store for SINGA jobs.</p> |
| <ol class="simple"> |
| <li><a class="reference external" href="#quickstart">Quick start using Docker</a></li> |
| <li><a class="reference external" href="#hdfs">Setup HDFS</a></li> |
| <li><a class="reference external" href="#examples">Examples</a></li> |
| </ol> |
| <p>– |
| <a name="quickstart"></a></p> |
| <div class="section" id="quick-start-using-docker"> |
| <span id="quick-start-using-docker"></span><h2>Quick start using Docker<a class="headerlink" href="#quick-start-using-docker" title="Permalink to this headline">¶</a></h2> |
| <p>We provide a Docker container built on top of <code class="docutils literal"><span class="pre">singa/mesos</span></code> (see the <a href="http://singa.incubator.apache.org/docs/docker.html">guide on building SINGA on Docker</a>).</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">git</span> <span class="n">clone</span> <span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">github</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">ug93tad</span><span class="o">/</span><span class="n">incubator</span><span class="o">-</span><span class="n">singa</span> |
| <span class="n">cd</span> <span class="n">incubator</span><span class="o">-</span><span class="n">singa</span> |
| <span class="n">git</span> <span class="n">checkout</span> <span class="n">SINGA</span><span class="o">-</span><span class="mi">97</span><span class="o">-</span><span class="n">docker</span> |
| <span class="n">cd</span> <span class="n">tool</span><span class="o">/</span><span class="n">docker</span><span class="o">/</span><span class="n">hdfs</span> |
| <span class="n">sudo</span> <span class="n">docker</span> <span class="n">build</span> <span class="o">-</span><span class="n">t</span> <span class="n">singa</span><span class="o">/</span><span class="n">hdfs</span> <span class="o">.</span> |
| </pre></div> |
| </div> |
| <p>Once built, the container image <code class="docutils literal"><span class="pre">singa/hdfs</span></code> contains the installation of HDFS C++ library (<code class="docutils literal"><span class="pre">libhdfs3</span></code>) and the latest SINGA code. Many distributed nodes can be launched, and HDFS be set up, by following the <a href="http://singa.incubator.apache.org/docs/mesos.html">guide for running distributed SINGA on Mesos</a>.</p> |
| <p>In the following, we assume the HDFS setup with <code class="docutils literal"><span class="pre">node0</span></code> being the namenode, and <code class="docutils literal"><span class="pre">nodei</span> <span class="pre">(i>0)</span></code> being the datanodes.</p> |
| <p><a name="hdfs"></a></p> |
| </div> |
| <div class="section" id="setup-hdfs"> |
| <span id="setup-hdfs"></span><h2>Setup HDFS<a class="headerlink" href="#setup-hdfs" title="Permalink to this headline">¶</a></h2> |
| <p>There are at least 2 C/C++ client libraries for interacting with HDFS. One is from Hadoop (<code class="docutils literal"><span class="pre">libhdfs</span></code>), which is a <a href="https://wiki.apache.org/hadoop/LibHDFS">JNI-based library</a>, meaning that communication will go through JVM. The other is <code class="docutils literal"><span class="pre">libhdfs3</span></code> which is a <a href="https://github.com/PivotalRD/libhdfs3">native C++ library developed by Pivotal</a>, in which the client communicate directly with HDFS via RPC. The current implementation uses the second one.</p> |
| <ol> |
| <li><p class="first">Install <code class="docutils literal"><span class="pre">libhdfs3</span></code>: follow the <a href="https://github.com/PivotalRD/libhdfs3#installation">official guide</a>.</p> |
| </li> |
| <li><p class="first"><strong>Additional setup</strong>: recent versions of Hadoop (>2.4.x) support short-circuit local reads which bypass network communications (TCP sockets) when retrieving data at the local nodes. <code class="docutils literal"><span class="pre">libhdfs3</span></code> will throws errors (but will still continue to work) when it finds that short-circuit read is not set. To deal with this complaints, and improve performance, add the following configuration to <code class="docutils literal"><span class="pre">hdfs-site.xml</span></code> <strong>and to <code class="docutils literal"><span class="pre">hdfs-client.xml</span></code></strong></p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span> |
| </pre></div> |
| </div> |
| </li> |
| </ol> |
| <p><property> |
| <name>dfs.client.read.shortcircuit</name> |
| <value>true</value> |
| </property> |
| <property> |
| <name>dfs.domain.socket.path</name> |
| <value>/var/lib/hadoop-hdfs/dn_socket</value> |
| </property> |
| ``` |
| Next, at each client, set <code class="docutils literal"><span class="pre">LIBHDFS3_CONF</span></code> variable to point to <code class="docutils literal"><span class="pre">hdfs-client.xml</span></code> file:</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span>``` |
| </pre></div> |
| </div> |
| <p>export LIBHDFS3_CONF=$HADOOP_HOME/etc/hadoop/hdfs-client.xml |
| ```</p> |
| <p><a name="examples"></a></p> |
| </div> |
| <div class="section" id="examples"> |
| <span id="examples"></span><h2>Examples<a class="headerlink" href="#examples" title="Permalink to this headline">¶</a></h2> |
| <p>We explain how to run CIFAR10 and MNIST examples. Before training, the data must be uploaded to HDFS.</p> |
| <div class="section" id="cifar10"> |
| <span id="cifar10"></span><h3>CIFAR10<a class="headerlink" href="#cifar10" title="Permalink to this headline">¶</a></h3> |
| <ol> |
| <li><p class="first">Upload the data to HDFS (done at any of the HDFS nodes)</p> |
| <ul class="simple"> |
| <li>Change <code class="docutils literal"><span class="pre">job.conf</span></code> to use HDFS: in <code class="docutils literal"><span class="pre">examples/cifar10/job.conf</span></code>, set <code class="docutils literal"><span class="pre">backend</span></code> property to <code class="docutils literal"><span class="pre">hdfsfile</span></code></li> |
| <li>Create and upload data:</li> |
| </ul> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">cd</span> <span class="n">examples</span><span class="o">/</span><span class="n">cifar10</span> |
| <span class="n">cp</span> <span class="n">Makefile</span><span class="o">.</span><span class="n">example</span> <span class="n">Makefile</span> |
| <span class="n">make</span> <span class="n">create</span> |
| <span class="n">hadoop</span> <span class="n">dfs</span> <span class="o">-</span><span class="n">mkdir</span> <span class="o">/</span><span class="n">examples</span><span class="o">/</span><span class="n">cifar10</span> |
| <span class="n">hadoop</span> <span class="n">dfs</span> <span class="o">-</span><span class="n">copyFromLocal</span> <span class="n">cifar</span><span class="o">-</span><span class="mi">10</span><span class="o">-</span><span class="n">batches</span><span class="o">-</span><span class="nb">bin</span> <span class="o">/</span><span class="n">examples</span><span class="o">/</span><span class="n">cifar10</span><span class="o">/</span> |
| </pre></div> |
| </div> |
| <p>If successful, the files should be seen in HDFS via <code class="docutils literal"><span class="pre">hadoop</span> <span class="pre">dfs</span> <span class="pre">-ls</span> <span class="pre">/examples/cifar10</span></code></p> |
| </li> |
| <li><p class="first">Training:</p> |
| <ul class="simple"> |
| <li>Make sure <code class="docutils literal"><span class="pre">conf/singa.conf</span></code> has correct path to Zookeeper service:</li> |
| </ul> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">zookeeper_host</span><span class="p">:</span> <span class="s2">"node0:2181"</span> |
| </pre></div> |
| </div> |
| <ul class="simple"> |
| <li>Make sure <code class="docutils literal"><span class="pre">job.conf</span></code> has correct paths to the train and test datasets:</li> |
| </ul> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="o">//</span> <span class="n">train</span> <span class="n">layer</span> |
| <span class="n">path</span><span class="p">:</span> <span class="s2">"hdfs://node0:9000/examples/cifar10/train_data.bin"</span> |
| <span class="n">mean_file</span><span class="p">:</span> <span class="s2">"hdfs://node0:9000/examples/cifar10/image_mean.bin"</span> |
| <span class="o">//</span> <span class="n">test</span> <span class="n">layer</span> |
| <span class="n">path</span><span class="p">:</span> <span class="s2">"hdfs://node0:9000/examples/cifar10/test_data.bin"</span> |
| <span class="n">mean_file</span><span class="p">:</span> <span class="s2">"hdfs://node0:9000/examples/cifar10/image_mean.bin"</span> |
| </pre></div> |
| </div> |
| <ul class="simple"> |
| <li>Start training: execute the following command at every node</li> |
| </ul> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="o">./</span><span class="n">singa</span> <span class="o">-</span><span class="n">conf</span> <span class="n">examples</span><span class="o">/</span><span class="n">cifar10</span><span class="o">/</span><span class="n">job</span><span class="o">.</span><span class="n">conf</span> <span class="o">-</span><span class="n">singa_conf</span> <span class="n">singa</span><span class="o">.</span><span class="n">conf</span> <span class="o">-</span><span class="n">singa_job</span> <span class="mi">0</span> |
| </pre></div> |
| </div> |
| </li> |
| </ol> |
| </div> |
| <div class="section" id="mnist"> |
| <span id="mnist"></span><h3>MNIST<a class="headerlink" href="#mnist" title="Permalink to this headline">¶</a></h3> |
| <ol> |
| <li><p class="first">Upload the data to HDFS (done at any of the HDFS nodes)</p> |
| <ul class="simple"> |
| <li>Change <code class="docutils literal"><span class="pre">job.conf</span></code> to use HDFS: in <code class="docutils literal"><span class="pre">examples/mnist/job.conf</span></code>, set <code class="docutils literal"><span class="pre">backend</span></code> property to <code class="docutils literal"><span class="pre">hdfsfile</span></code></li> |
| <li>Create and upload data:</li> |
| </ul> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">cd</span> <span class="n">examples</span><span class="o">/</span><span class="n">mnist</span> |
| <span class="n">cp</span> <span class="n">Makefile</span><span class="o">.</span><span class="n">example</span> <span class="n">Makefile</span> |
| <span class="n">make</span> <span class="n">create</span> |
| <span class="n">make</span> <span class="nb">compile</span> |
| <span class="o">./</span><span class="n">create_data</span><span class="o">.</span><span class="n">bin</span> <span class="n">train</span><span class="o">-</span><span class="n">images</span><span class="o">-</span><span class="n">idx3</span><span class="o">-</span><span class="n">ubyte</span> <span class="n">train</span><span class="o">-</span><span class="n">labels</span><span class="o">-</span><span class="n">idx1</span><span class="o">-</span><span class="n">ubyte</span> <span class="n">hdfs</span><span class="p">:</span><span class="o">//</span><span class="n">node0</span><span class="p">:</span><span class="mi">9000</span><span class="o">/</span><span class="n">examples</span><span class="o">/</span><span class="n">mnist</span><span class="o">/</span><span class="n">train_data</span><span class="o">.</span><span class="n">bin</span> |
| <span class="o">./</span><span class="n">create_data</span><span class="o">.</span><span class="n">bin</span> <span class="n">t10k</span><span class="o">-</span><span class="n">images</span><span class="o">-</span><span class="n">idx3</span><span class="o">-</span><span class="n">ubyte</span> <span class="n">t10k</span><span class="o">-</span><span class="n">labels</span><span class="o">-</span><span class="n">idx1</span><span class="o">-</span><span class="n">ubyte</span> <span class="n">hdfs</span><span class="p">:</span><span class="o">//</span><span class="n">node0</span><span class="p">:</span><span class="mi">9000</span><span class="o">/</span><span class="n">examples</span><span class="o">/</span><span class="n">mnist</span><span class="o">/</span><span class="n">test_data</span><span class="o">.</span><span class="n">bin</span> |
| </pre></div> |
| </div> |
| <p>If successful, the files should be seen in HDFS via <code class="docutils literal"><span class="pre">hadoop</span> <span class="pre">dfs</span> <span class="pre">-ls</span> <span class="pre">/examples/mnist</span></code></p> |
| </li> |
| <li><p class="first">Training:</p> |
| <ul class="simple"> |
| <li>Make sure <code class="docutils literal"><span class="pre">conf/singa.conf</span></code> has correct path to Zookeeper service:</li> |
| </ul> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">zookeeper_host</span><span class="p">:</span> <span class="s2">"node0:2181"</span> |
| </pre></div> |
| </div> |
| <ul class="simple"> |
| <li>Make sure <code class="docutils literal"><span class="pre">job.conf</span></code> has correct paths to the train and test datasets:</li> |
| </ul> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="o">//</span> <span class="n">train</span> <span class="n">layer</span> |
| <span class="n">path</span><span class="p">:</span> <span class="s2">"hdfs://node0:9000/examples/mnist/train_data.bin"</span> |
| <span class="o">//</span> <span class="n">test</span> <span class="n">layer</span> |
| <span class="n">path</span><span class="p">:</span> <span class="s2">"hdfs://node0:9000/examples/mnist/test_data.bin"</span> |
| </pre></div> |
| </div> |
| <ul class="simple"> |
| <li>Start training: execute the following command at every node</li> |
| </ul> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="o">./</span><span class="n">singa</span> <span class="o">-</span><span class="n">conf</span> <span class="n">examples</span><span class="o">/</span><span class="n">mnist</span><span class="o">/</span><span class="n">job</span><span class="o">.</span><span class="n">conf</span> <span class="o">-</span><span class="n">singa_conf</span> <span class="n">singa</span><span class="o">.</span><span class="n">conf</span> <span class="o">-</span><span class="n">singa_job</span> <span class="mi">0</span> |
| </pre></div> |
| </div> |
| </li> |
| </ol> |
| </div> |
| </div> |
| </div> |
| |
| |
| </div> |
| </div> |
| <footer> |
| |
| |
| <hr/> |
| |
| <div role="contentinfo"> |
| <p> |
| © Copyright 2016 The Apache Software Foundation. All rights reserved. Apache Singa, Apache, the Apache feather logo, and the Apache Singa project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.. |
| |
| </p> |
| </div> |
| Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. |
| |
| </footer> |
| |
| </div> |
| </div> |
| |
| </section> |
| |
| </div> |
| |
| |
| |
| |
| |
| <script type="text/javascript"> |
| var DOCUMENTATION_OPTIONS = { |
| URL_ROOT:'../', |
| VERSION:'0.3.0', |
| COLLAPSE_INDEX:false, |
| FILE_SUFFIX:'.html', |
| HAS_SOURCE: true |
| }; |
| </script> |
| <script type="text/javascript" src="../_static/jquery.js"></script> |
| <script type="text/javascript" src="../_static/underscore.js"></script> |
| <script type="text/javascript" src="../_static/doctools.js"></script> |
| |
| |
| |
| |
| |
| <script type="text/javascript" src="../_static/js/theme.js"></script> |
| |
| |
| |
| |
| <script type="text/javascript"> |
| jQuery(function () { |
| SphinxRtdTheme.StickyNav.enable(); |
| }); |
| </script> |
| |
| |
| |
| <div class="rst-versions shift-up" data-toggle="rst-versions" role="note" aria-label="versions"> |
| <img src="../_static/apache.jpg"> |
| |
| <span class="rst-current-version" data-toggle="rst-current-version"> |
| <span class="fa fa-book"> incubator-singa </span> |
| v: 0.3.0 |
| <span class="fa fa-caret-down"></span> |
| </span> |
| <div class="rst-other-versions"> |
| <dl> |
| <dt>Languages</dt> |
| <dd><a href="../../en/index.html">English</a></dd> |
| <dd><a href="../../zh/index.html">中文</a></dd> |
| <dd><a href="../../jp/index.html">日本語</a></dd> |
| <dd><a href="../../kr/index.html">한국어</a></dd> |
| </dl> |
| </div> |
| </div> |
| |
| <a href="https://github.com/apache/incubator-singa"> |
| <img style="position: absolute; top: 0; right: 0; border: 0; z-index: 10000;" |
| src="https://s3.amazonaws.com/github/ribbons/forkme_right_orange_ff7600.png" |
| alt="Fork me on GitHub"> |
| </a> |
| |
| |
| |
| |
| </body> |
| </html> |
