| <!DOCTYPE html> |
| |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"/> |
| <meta content="IE=edge" http-equiv="X-UA-Compatible"/> |
| <meta content="width=device-width, initial-scale=1" name="viewport"/> |
| <meta content="Data Loading API" property="og:title"> |
| <meta content="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/og-logo.png" property="og:image"> |
| <meta content="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/og-logo.png" property="og:image:secure_url"> |
| <meta content="Data Loading API" property="og:description"/> |
| <title>Data Loading API — mxnet documentation</title> |
| <link crossorigin="anonymous" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css" integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7" rel="stylesheet"/> |
| <link href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css" rel="stylesheet"/> |
| <link href="../../../_static/basic.css" rel="stylesheet" type="text/css"> |
| <link href="../../../_static/pygments.css" rel="stylesheet" type="text/css"> |
| <link href="../../../_static/mxnet.css" rel="stylesheet" type="text/css"/> |
| <script type="text/javascript"> |
| var DOCUMENTATION_OPTIONS = { |
| URL_ROOT: '../../../', |
| VERSION: '', |
| COLLAPSE_INDEX: false, |
| FILE_SUFFIX: '.html', |
| HAS_SOURCE: true, |
| SOURCELINK_SUFFIX: '.txt' |
| }; |
| </script> |
| <script src="https://code.jquery.com/jquery-1.11.1.min.js" type="text/javascript"></script> |
| <script src="../../../_static/underscore.js" type="text/javascript"></script> |
| <script src="../../../_static/searchtools_custom.js" type="text/javascript"></script> |
| <script src="../../../_static/doctools.js" type="text/javascript"></script> |
| <script src="../../../_static/selectlang.js" type="text/javascript"></script> |
| <script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script> |
| <script type="text/javascript"> jQuery(function() { Search.loadIndex("/versions/1.3.1/searchindex.js"); Search.init();}); </script> |
| <script> |
| (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ |
| (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new |
| Date();a=s.createElement(o), |
| m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) |
| })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); |
| |
| ga('create', 'UA-96378503-1', 'auto'); |
| ga('send', 'pageview'); |
| |
| </script> |
| <!-- --> |
| <!-- <script type="text/javascript" src="../../../_static/jquery.js"></script> --> |
| <!-- --> |
| <!-- <script type="text/javascript" src="../../../_static/underscore.js"></script> --> |
| <!-- --> |
| <!-- <script type="text/javascript" src="../../../_static/doctools.js"></script> --> |
| <!-- --> |
| <!-- <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script> --> |
| <!-- --> |
| <link href="../../../genindex.html" rel="index" title="Index"> |
| <link href="../../../search.html" rel="search" title="Search"/> |
| <link href="../index.html" rel="up" title="MXNet - Python API"/> |
| <link href="../image/image.html" rel="next" title="Image API"/> |
| <link href="../gluon/contrib.html" rel="prev" title="Gluon Contrib API"/> |
| <link href="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet-icon.png" rel="icon" type="image/png"/> |
| </link></link></link></meta></meta></meta></head> |
| <body background="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet-background-compressed.jpeg" role="document"> |
| <div class="content-block"><div class="navbar navbar-fixed-top"> |
| <div class="container" id="navContainer"> |
| <div class="innder" id="header-inner"> |
| <h1 id="logo-wrap"> |
| <a href="../../../" id="logo"><img src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet_logo.png"/></a> |
| </h1> |
| <nav class="nav-bar" id="main-nav"> |
| <a class="main-nav-link" href="/versions/1.3.1/install/index.html">Install</a> |
| <span id="dropdown-menu-position-anchor"> |
| <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Gluon <span class="caret"></span></a> |
| <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu"> |
| <li><a class="main-nav-link" href="/versions/1.3.1/tutorials/gluon/gluon.html">About</a></li> |
| <li><a class="main-nav-link" href="https://www.d2l.ai/">Dive into Deep Learning</a></li> |
| <li><a class="main-nav-link" href="https://gluon-cv.mxnet.io">GluonCV Toolkit</a></li> |
| <li><a class="main-nav-link" href="https://gluon-nlp.mxnet.io/">GluonNLP Toolkit</a></li> |
| </ul> |
| </span> |
| <span id="dropdown-menu-position-anchor"> |
| <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">API <span class="caret"></span></a> |
| <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu"> |
| <li><a class="main-nav-link" href="/versions/1.3.1/api/python/index.html">Python</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/api/c++/index.html">C++</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/api/clojure/index.html">Clojure</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/api/julia/index.html">Julia</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/api/perl/index.html">Perl</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/api/r/index.html">R</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/api/scala/index.html">Scala</a></li> |
| </ul> |
| </span> |
| <span id="dropdown-menu-position-anchor-docs"> |
| <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Docs <span class="caret"></span></a> |
| <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu-docs"> |
| <li><a class="main-nav-link" href="/versions/1.3.1/faq/index.html">FAQ</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/tutorials/index.html">Tutorials</a> |
| <li><a class="main-nav-link" href="https://github.com/apache/incubator-mxnet/tree/1.3.1/example">Examples</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/architecture/index.html">Architecture</a></li> |
| <li><a class="main-nav-link" href="https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+Home">Developer Wiki</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/model_zoo/index.html">Model Zoo</a></li> |
| <li><a class="main-nav-link" href="https://github.com/onnx/onnx-mxnet">ONNX</a></li> |
| </li></ul> |
| </span> |
| <span id="dropdown-menu-position-anchor-community"> |
| <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Community <span class="caret"></span></a> |
| <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu-community"> |
| <li><a class="main-nav-link" href="http://discuss.mxnet.io">Forum</a></li> |
| <li><a class="main-nav-link" href="https://github.com/apache/incubator-mxnet/tree/1.3.1">Github</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/community/contribute.html">Contribute</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/community/ecosystem.html">Ecosystem</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/community/powered_by.html">Powered By</a></li> |
| </ul> |
| </span> |
| <span id="dropdown-menu-position-anchor-version" style="position: relative"><a href="#" class="main-nav-link dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="true">1.3.1<span class="caret"></span></a><ul id="package-dropdown-menu" class="dropdown-menu"><li><a href="/">master</a></li><li><a href="/versions/1.7/">1.7</a></li><li><a href=/versions/1.6/>1.6</a></li><li><a href=/versions/1.5.0/>1.5.0</a></li><li><a href=/versions/1.4.1/>1.4.1</a></li><li><a href=/versions/1.3.1/>1.3.1</a></li><li><a href=/versions/1.2.1/>1.2.1</a></li><li><a href=/versions/1.1.0/>1.1.0</a></li><li><a href=/versions/1.0.0/>1.0.0</a></li><li><a href=/versions/0.12.1/>0.12.1</a></li><li><a href=/versions/0.11.0/>0.11.0</a></li></ul></span></nav> |
| <script> function getRootPath(){ return "../../../" } </script> |
| <div class="burgerIcon dropdown"> |
| <a class="dropdown-toggle" data-toggle="dropdown" href="#" role="button">☰</a> |
| <ul class="dropdown-menu" id="burgerMenu"> |
| <li><a href="/versions/1.3.1/install/index.html">Install</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/tutorials/index.html">Tutorials</a></li> |
| <li class="dropdown-submenu dropdown"> |
| <a aria-expanded="true" aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" tabindex="-1">Gluon</a> |
| <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu"> |
| <li><a class="main-nav-link" href="/versions/1.3.1/tutorials/gluon/gluon.html">About</a></li> |
| <li><a class="main-nav-link" href="http://gluon.mxnet.io">The Straight Dope (Tutorials)</a></li> |
| <li><a class="main-nav-link" href="https://gluon-cv.mxnet.io">GluonCV Toolkit</a></li> |
| <li><a class="main-nav-link" href="https://gluon-nlp.mxnet.io/">GluonNLP Toolkit</a></li> |
| </ul> |
| </li> |
| <li class="dropdown-submenu"> |
| <a aria-expanded="true" aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" tabindex="-1">API</a> |
| <ul class="dropdown-menu"> |
| <li><a class="main-nav-link" href="/versions/1.3.1/api/python/index.html">Python</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/api/c++/index.html">C++</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/api/clojure/index.html">Clojure</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/api/julia/index.html">Julia</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/api/perl/index.html">Perl</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/api/r/index.html">R</a></li> |
| <li><a class="main-nav-link" href="/versions/1.3.1/api/scala/index.html">Scala</a></li> |
| </ul> |
| </li> |
| <li class="dropdown-submenu"> |
| <a aria-expanded="true" aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" tabindex="-1">Docs</a> |
| <ul class="dropdown-menu"> |
| <li><a href="/versions/1.3.1/faq/index.html" tabindex="-1">FAQ</a></li> |
| <li><a href="/versions/1.3.1/tutorials/index.html" tabindex="-1">Tutorials</a></li> |
| <li><a href="https://github.com/apache/incubator-mxnet/tree/1.3.1/example" tabindex="-1">Examples</a></li> |
| <li><a href="/versions/1.3.1/architecture/index.html" tabindex="-1">Architecture</a></li> |
| <li><a href="https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+Home" tabindex="-1">Developer Wiki</a></li> |
| <li><a href="/versions/1.3.1/model_zoo/index.html" tabindex="-1">Gluon Model Zoo</a></li> |
| <li><a href="https://github.com/onnx/onnx-mxnet" tabindex="-1">ONNX</a></li> |
| </ul> |
| </li> |
| <li class="dropdown-submenu dropdown"> |
| <a aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" role="button" tabindex="-1">Community</a> |
| <ul class="dropdown-menu"> |
| <li><a href="http://discuss.mxnet.io" tabindex="-1">Forum</a></li> |
| <li><a href="https://github.com/apache/incubator-mxnet/tree/1.3.1" tabindex="-1">Github</a></li> |
| <li><a href="/versions/1.3.1/community/contribute.html" tabindex="-1">Contribute</a></li> |
| <li><a href="/versions/1.3.1/community/ecosystem.html" tabindex="-1">Ecosystem</a></li> |
| <li><a href="/versions/1.3.1/community/powered_by.html" tabindex="-1">Powered By</a></li> |
| </ul> |
| </li> |
| <li id="dropdown-menu-position-anchor-version-mobile" class="dropdown-submenu" style="position: relative"><a href="#" tabindex="-1">1.3.1</a><ul class="dropdown-menu"><li><a tabindex="-1" href=/>master</a></li><li><a tabindex="-1" href=/versions/1.6/>1.6</a></li><li><a tabindex="-1" href=/versions/1.5.0/>1.5.0</a></li><li><a tabindex="-1" href=/versions/1.4.1/>1.4.1</a></li><li><a tabindex="-1" href=/versions/1.3.1/>1.3.1</a></li><li><a tabindex="-1" href=/versions/1.2.1/>1.2.1</a></li><li><a tabindex="-1" href=/versions/1.1.0/>1.1.0</a></li><li><a tabindex="-1" href=/versions/1.0.0/>1.0.0</a></li><li><a tabindex="-1" href=/versions/0.12.1/>0.12.1</a></li><li><a tabindex="-1" href=/versions/0.11.0/>0.11.0</a></li></ul></li></ul> |
| </div> |
| <div class="plusIcon dropdown"> |
| <a class="dropdown-toggle" data-toggle="dropdown" href="#" role="button"><span aria-hidden="true" class="glyphicon glyphicon-plus"></span></a> |
| <ul class="dropdown-menu dropdown-menu-right" id="plusMenu"></ul> |
| </div> |
| <div id="search-input-wrap"> |
| <form action="../../../search.html" autocomplete="off" class="" method="get" role="search"> |
| <div class="form-group inner-addon left-addon"> |
| <i class="glyphicon glyphicon-search"></i> |
| <input class="form-control" name="q" placeholder="Search" type="text"/> |
| </div> |
| <input name="check_keywords" type="hidden" value="yes"> |
| <input name="area" type="hidden" value="default"/> |
| </input></form> |
| <div id="search-preview"></div> |
| </div> |
| <div id="searchIcon"> |
| <span aria-hidden="true" class="glyphicon glyphicon-search"></span> |
| </div> |
| <!-- <div id="lang-select-wrap"> --> |
| <!-- <label id="lang-select-label"> --> |
| <!-- <\!-- <i class="fa fa-globe"></i> -\-> --> |
| <!-- <span></span> --> |
| <!-- </label> --> |
| <!-- <select id="lang-select"> --> |
| <!-- <option value="en">Eng</option> --> |
| <!-- <option value="zh">中文</option> --> |
| <!-- </select> --> |
| <!-- </div> --> |
| <!-- <a id="mobile-nav-toggle"> |
| <span class="mobile-nav-toggle-bar"></span> |
| <span class="mobile-nav-toggle-bar"></span> |
| <span class="mobile-nav-toggle-bar"></span> |
| </a> --> |
| </div> |
| </div> |
| </div> |
| <script type="text/javascript"> |
| $('body').css('background', 'white'); |
| </script> |
| <div class="container"> |
| <div class="row"> |
| <div aria-label="main navigation" class="sphinxsidebar leftsidebar" role="navigation"> |
| <div class="sphinxsidebarwrapper"> |
| <ul class="current"> |
| <li class="toctree-l1 current"><a class="reference internal" href="../index.html">Python Documents</a><ul class="current"> |
| <li class="toctree-l2"><a class="reference internal" href="../index.html#ndarray-api">NDArray API</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../index.html#symbol-api">Symbol API</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../index.html#module-api">Module API</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../index.html#autograd-api">Autograd API</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../index.html#gluon-api">Gluon API</a></li> |
| <li class="toctree-l2 current"><a class="reference internal" href="../index.html#io-api">IO API</a><ul class="current"> |
| <li class="toctree-l3 current"><a class="current reference internal" href="#">Data Loading API</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="#overview">Overview</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#data-iterators">Data iterators</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#helper-classes-and-functions">Helper classes and functions</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#how-to-develop-a-new-iterator">How to develop a new iterator</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="#api-reference">API Reference</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="toctree-l2"><a class="reference internal" href="../index.html#image-api">Image API</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../index.html#optimization-api">Optimization API</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../index.html#callback-api">Callback API</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../index.html#metric-api">Metric API</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../index.html#profiler-api">Profiler API</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../index.html#run-time-compilation-api">Run-Time Compilation API</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../index.html#contrib-package">Contrib Package</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l1"><a class="reference internal" href="../../r/index.html">R Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../julia/index.html">Julia Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../c++/index.html">C++ Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../scala/index.html">Scala Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../perl/index.html">Perl Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../../faq/index.html">HowTo Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../../architecture/index.html">System Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../../tutorials/index.html">Tutorials</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../../community/contribute.html">Community</a></li> |
| </ul> |
| </div> |
| </div> |
| <div class="content"> |
| <div class="page-tracker"></div> |
| <div class="section" id="data-loading-api"> |
| <span id="data-loading-api"></span><h1>Data Loading API<a class="headerlink" href="#data-loading-api" title="Permalink to this headline">¶</a></h1> |
| <div class="section" id="overview"> |
| <span id="overview"></span><h2>Overview<a class="headerlink" href="#overview" title="Permalink to this headline">¶</a></h2> |
| <p>This document summarizes supported data formats and iterator APIs to read the |
| data including:</p> |
| <table border="1" class="longtable docutils"> |
| <colgroup> |
| <col width="10%"/> |
| <col width="90%"/> |
| </colgroup> |
| <tbody valign="top"> |
| <tr class="row-odd"><td><a class="reference internal" href="#module-mxnet.io" title="mxnet.io"><code class="xref py py-obj docutils literal"><span class="pre">mxnet.io</span></code></a></td> |
| <td>Data iterators for common data formats.</td> |
| </tr> |
| <tr class="row-even"><td><a class="reference internal" href="#module-mxnet.recordio" title="mxnet.recordio"><code class="xref py py-obj docutils literal"><span class="pre">mxnet.recordio</span></code></a></td> |
| <td>Read and write for the RecordIO data format.</td> |
| </tr> |
| <tr class="row-odd"><td><a class="reference internal" href="../image/image.html#module-mxnet.image" title="mxnet.image"><code class="xref py py-obj docutils literal"><span class="pre">mxnet.image</span></code></a></td> |
| <td>Image Iterators and image augmentation functions</td> |
| </tr> |
| </tbody> |
| </table> |
| <p>First, let’s see how to write an iterator for a new data format. |
| The following iterator can be used to train a symbol whose input data variable has |
| name <code class="docutils literal"><span class="pre">data</span></code> and input label variable has name <code class="docutils literal"><span class="pre">softmax_label</span></code>. |
| The iterator also provides information about the batch, including the |
| shapes and name.</p> |
| <div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">nd_iter</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">NDArrayIter</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="p">{</span><span class="s1">'data'</span><span class="p">:</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="mi">100</span><span class="p">,</span><span class="mi">10</span><span class="p">))},</span> |
| <span class="gp">... </span> <span class="n">label</span><span class="o">=</span><span class="p">{</span><span class="s1">'softmax_label'</span><span class="p">:</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="mi">100</span><span class="p">,))},</span> |
| <span class="gp">... </span> <span class="n">batch_size</span><span class="o">=</span><span class="mi">25</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="k">print</span><span class="p">(</span><span class="n">nd_iter</span><span class="o">.</span><span class="n">provide_data</span><span class="p">)</span> |
| <span class="go">[DataDesc[data,(25, 10L),<type 'numpy.float32'>,NCHW]]</span> |
| <span class="gp">>>> </span><span class="k">print</span><span class="p">(</span><span class="n">nd_iter</span><span class="o">.</span><span class="n">provide_label</span><span class="p">)</span> |
| <span class="go">[DataDesc[softmax_label,(25,),<type 'numpy.float32'>,NCHW]]</span> |
| </pre></div> |
| </div> |
| <p>Let’s see a complete example of how to use data iterator in model training.</p> |
| <div class="highlight-python"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">data</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">sym</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="s1">'data'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">label</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">sym</span><span class="o">.</span><span class="n">Variable</span><span class="p">(</span><span class="s1">'softmax_label'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">fullc</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">sym</span><span class="o">.</span><span class="n">FullyConnected</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">data</span><span class="p">,</span> <span class="n">num_hidden</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">loss</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">sym</span><span class="o">.</span><span class="n">SoftmaxOutput</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">fullc</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="n">label</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">mod</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">mod</span><span class="o">.</span><span class="n">Module</span><span class="p">(</span><span class="n">loss</span><span class="p">,</span> <span class="n">data_names</span><span class="o">=</span><span class="p">[</span><span class="s1">'data'</span><span class="p">],</span> <span class="n">label_names</span><span class="o">=</span><span class="p">[</span><span class="s1">'softmax_label'</span><span class="p">])</span> |
| <span class="gp">>>> </span><span class="n">mod</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="n">data_shapes</span><span class="o">=</span><span class="n">nd_iter</span><span class="o">.</span><span class="n">provide_data</span><span class="p">,</span> <span class="n">label_shapes</span><span class="o">=</span><span class="n">nd_iter</span><span class="o">.</span><span class="n">provide_label</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">mod</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">nd_iter</span><span class="p">,</span> <span class="n">num_epoch</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <p>A detailed tutorial is available at |
| <a class="reference external" href="/versions/1.3.1/tutorials/basic/data.html">Iterators - Loading data</a>.</p> |
| </div> |
| <div class="section" id="data-iterators"> |
| <span id="data-iterators"></span><h2>Data iterators<a class="headerlink" href="#data-iterators" title="Permalink to this headline">¶</a></h2> |
| <table border="1" class="longtable docutils"> |
| <colgroup> |
| <col width="10%"/> |
| <col width="90%"/> |
| </colgroup> |
| <tbody valign="top"> |
| <tr class="row-odd"><td><a class="reference internal" href="#mxnet.io.NDArrayIter" title="mxnet.io.NDArrayIter"><code class="xref py py-obj docutils literal"><span class="pre">io.NDArrayIter</span></code></a></td> |
| <td>Returns an iterator for <code class="docutils literal"><span class="pre">mx.nd.NDArray</span></code>, <code class="docutils literal"><span class="pre">numpy.ndarray</span></code>, <code class="docutils literal"><span class="pre">h5py.Dataset</span></code> <code class="docutils literal"><span class="pre">mx.nd.sparse.CSRNDArray</span></code> or <code class="docutils literal"><span class="pre">scipy.sparse.csr_matrix</span></code>.</td> |
| </tr> |
| <tr class="row-even"><td><a class="reference internal" href="#mxnet.io.CSVIter" title="mxnet.io.CSVIter"><code class="xref py py-obj docutils literal"><span class="pre">io.CSVIter</span></code></a></td> |
| <td>Returns the CSV file iterator.</td> |
| </tr> |
| <tr class="row-odd"><td><a class="reference internal" href="#mxnet.io.LibSVMIter" title="mxnet.io.LibSVMIter"><code class="xref py py-obj docutils literal"><span class="pre">io.LibSVMIter</span></code></a></td> |
| <td>Returns the LibSVM iterator which returns data with <cite>csr</cite> storage type.</td> |
| </tr> |
| <tr class="row-even"><td><a class="reference internal" href="#mxnet.io.ImageRecordIter" title="mxnet.io.ImageRecordIter"><code class="xref py py-obj docutils literal"><span class="pre">io.ImageRecordIter</span></code></a></td> |
| <td>Iterates on image RecordIO files</td> |
| </tr> |
| <tr class="row-odd"><td><a class="reference internal" href="#mxnet.io.ImageRecordUInt8Iter" title="mxnet.io.ImageRecordUInt8Iter"><code class="xref py py-obj docutils literal"><span class="pre">io.ImageRecordUInt8Iter</span></code></a></td> |
| <td>Iterating on image RecordIO files</td> |
| </tr> |
| <tr class="row-even"><td><a class="reference internal" href="#mxnet.io.MNISTIter" title="mxnet.io.MNISTIter"><code class="xref py py-obj docutils literal"><span class="pre">io.MNISTIter</span></code></a></td> |
| <td>Iterating on the MNIST dataset.</td> |
| </tr> |
| <tr class="row-odd"><td><a class="reference internal" href="#mxnet.recordio.MXRecordIO" title="mxnet.recordio.MXRecordIO"><code class="xref py py-obj docutils literal"><span class="pre">recordio.MXRecordIO</span></code></a></td> |
| <td>Reads/writes <cite>RecordIO</cite> data format, supporting sequential read and write.</td> |
| </tr> |
| <tr class="row-even"><td><a class="reference internal" href="#mxnet.recordio.MXIndexedRecordIO" title="mxnet.recordio.MXIndexedRecordIO"><code class="xref py py-obj docutils literal"><span class="pre">recordio.MXIndexedRecordIO</span></code></a></td> |
| <td>Reads/writes <cite>RecordIO</cite> data format, supporting random access.</td> |
| </tr> |
| <tr class="row-odd"><td><a class="reference internal" href="../image/image.html#mxnet.image.ImageIter" title="mxnet.image.ImageIter"><code class="xref py py-obj docutils literal"><span class="pre">image.ImageIter</span></code></a></td> |
| <td>Image data iterator with a large number of augmentation choices.</td> |
| </tr> |
| <tr class="row-even"><td><a class="reference internal" href="../image/image.html#mxnet.image.ImageDetIter" title="mxnet.image.ImageDetIter"><code class="xref py py-obj docutils literal"><span class="pre">image.ImageDetIter</span></code></a></td> |
| <td>Image iterator with a large number of augmentation choices for detection.</td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
| <div class="section" id="helper-classes-and-functions"> |
| <span id="helper-classes-and-functions"></span><h2>Helper classes and functions<a class="headerlink" href="#helper-classes-and-functions" title="Permalink to this headline">¶</a></h2> |
| <div class="section" id="data-structures-and-other-iterators"> |
| <span id="data-structures-and-other-iterators"></span><h3>Data structures and other iterators<a class="headerlink" href="#data-structures-and-other-iterators" title="Permalink to this headline">¶</a></h3> |
| <table border="1" class="longtable docutils"> |
| <colgroup> |
| <col width="10%"/> |
| <col width="90%"/> |
| </colgroup> |
| <tbody valign="top"> |
| <tr class="row-odd"><td><a class="reference internal" href="#mxnet.io.DataDesc" title="mxnet.io.DataDesc"><code class="xref py py-obj docutils literal"><span class="pre">io.DataDesc</span></code></a></td> |
| <td>DataDesc is used to store name, shape, type and layout information of the data or the label.</td> |
| </tr> |
| <tr class="row-even"><td><a class="reference internal" href="#mxnet.io.DataBatch" title="mxnet.io.DataBatch"><code class="xref py py-obj docutils literal"><span class="pre">io.DataBatch</span></code></a></td> |
| <td>A data batch.</td> |
| </tr> |
| <tr class="row-odd"><td><a class="reference internal" href="#mxnet.io.DataIter" title="mxnet.io.DataIter"><code class="xref py py-obj docutils literal"><span class="pre">io.DataIter</span></code></a></td> |
| <td>The base class for an MXNet data iterator.</td> |
| </tr> |
| <tr class="row-even"><td><a class="reference internal" href="#mxnet.io.ResizeIter" title="mxnet.io.ResizeIter"><code class="xref py py-obj docutils literal"><span class="pre">io.ResizeIter</span></code></a></td> |
| <td>Resize a data iterator to a given number of batches.</td> |
| </tr> |
| <tr class="row-odd"><td><a class="reference internal" href="#mxnet.io.PrefetchingIter" title="mxnet.io.PrefetchingIter"><code class="xref py py-obj docutils literal"><span class="pre">io.PrefetchingIter</span></code></a></td> |
| <td>Performs pre-fetch for other data iterators.</td> |
| </tr> |
| <tr class="row-even"><td><a class="reference internal" href="#mxnet.io.MXDataIter" title="mxnet.io.MXDataIter"><code class="xref py py-obj docutils literal"><span class="pre">io.MXDataIter</span></code></a></td> |
| <td>A python wrapper a C++ data iterator.</td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
| <div class="section" id="functions-to-read-and-write-recordio-files"> |
| <span id="functions-to-read-and-write-recordio-files"></span><h3>Functions to read and write RecordIO files<a class="headerlink" href="#functions-to-read-and-write-recordio-files" title="Permalink to this headline">¶</a></h3> |
| <table border="1" class="longtable docutils"> |
| <colgroup> |
| <col width="10%"/> |
| <col width="90%"/> |
| </colgroup> |
| <tbody valign="top"> |
| <tr class="row-odd"><td><a class="reference internal" href="#mxnet.recordio.pack" title="mxnet.recordio.pack"><code class="xref py py-obj docutils literal"><span class="pre">recordio.pack</span></code></a></td> |
| <td>Pack a string into MXImageRecord.</td> |
| </tr> |
| <tr class="row-even"><td><a class="reference internal" href="#mxnet.recordio.unpack" title="mxnet.recordio.unpack"><code class="xref py py-obj docutils literal"><span class="pre">recordio.unpack</span></code></a></td> |
| <td>Unpack a MXImageRecord to string.</td> |
| </tr> |
| <tr class="row-odd"><td><a class="reference internal" href="#mxnet.recordio.unpack_img" title="mxnet.recordio.unpack_img"><code class="xref py py-obj docutils literal"><span class="pre">recordio.unpack_img</span></code></a></td> |
| <td>Unpack a MXImageRecord to image.</td> |
| </tr> |
| <tr class="row-even"><td><a class="reference internal" href="#mxnet.recordio.pack_img" title="mxnet.recordio.pack_img"><code class="xref py py-obj docutils literal"><span class="pre">recordio.pack_img</span></code></a></td> |
| <td>Pack an image into <code class="docutils literal"><span class="pre">MXImageRecord</span></code>.</td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
| </div> |
| <div class="section" id="how-to-develop-a-new-iterator"> |
| <span id="how-to-develop-a-new-iterator"></span><h2>How to develop a new iterator<a class="headerlink" href="#how-to-develop-a-new-iterator" title="Permalink to this headline">¶</a></h2> |
| <p>Writing a new data iterator in Python is straightforward. Most MXNet |
| training/inference programs accept an iterable object with <code class="docutils literal"><span class="pre">provide_data</span></code> |
| and <code class="docutils literal"><span class="pre">provide_label</span></code> properties. |
| This <a class="reference external" href="/versions/1.3.1/tutorials/basic/data.html">tutorial</a> explains how to |
| write an iterator from scratch.</p> |
| <p>The following example demonstrates how to combine |
| multiple data iterators into a single one. It can be used for multiple |
| modality training such as image captioning, in which images are read by |
| <code class="docutils literal"><span class="pre">ImageRecordIter</span></code> while documents are read by <code class="docutils literal"><span class="pre">CSVIter</span></code></p> |
| <div class="highlight-python"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">MultiIter</span><span class="p">:</span> |
| <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">iter_list</span><span class="p">):</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">iters</span> <span class="o">=</span> <span class="n">iter_list</span> |
| <span class="k">def</span> <span class="nf">next</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| <span class="n">batches</span> <span class="o">=</span> <span class="p">[</span><span class="n">i</span><span class="o">.</span><span class="n">next</span><span class="p">()</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">iters</span><span class="p">]</span> |
| <span class="k">return</span> <span class="n">DataBatch</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="p">[</span><span class="o">*</span><span class="n">b</span><span class="o">.</span><span class="n">data</span> <span class="k">for</span> <span class="n">b</span> <span class="ow">in</span> <span class="n">batches</span><span class="p">],</span> |
| <span class="n">label</span><span class="o">=</span><span class="p">[</span><span class="o">*</span><span class="n">b</span><span class="o">.</span><span class="n">label</span> <span class="k">for</span> <span class="n">b</span> <span class="ow">in</span> <span class="n">batches</span><span class="p">])</span> |
| <span class="k">def</span> <span class="nf">reset</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">iters</span><span class="p">:</span> |
| <span class="n">i</span><span class="o">.</span><span class="n">reset</span><span class="p">()</span> |
| <span class="nd">@property</span> |
| <span class="k">def</span> <span class="nf">provide_data</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| <span class="k">return</span> <span class="p">[</span><span class="o">*</span><span class="n">i</span><span class="o">.</span><span class="n">provide_data</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">iters</span><span class="p">]</span> |
| <span class="nd">@property</span> |
| <span class="k">def</span> <span class="nf">provide_label</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| <span class="k">return</span> <span class="p">[</span><span class="o">*</span><span class="n">i</span><span class="o">.</span><span class="n">provide_label</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">iters</span><span class="p">]</span> |
| |
| <span class="nb">iter</span> <span class="o">=</span> <span class="n">MultiIter</span><span class="p">([</span><span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ImageRecordIter</span><span class="p">(</span><span class="s1">'image.rec'</span><span class="p">),</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">CSVIter</span><span class="p">(</span><span class="s1">'txt.csv'</span><span class="p">)])</span> |
| </pre></div> |
| </div> |
| <p>Parsing and performing another pre-processing such as augmentation may be expensive. |
| If performance is critical, we can implement a data iterator in C++. Refer to |
| <a class="reference external" href="https://github.com/dmlc/mxnet/tree/master/src/io">src/io</a> for examples.</p> |
| <div class="section" id="how-to-change-the-batch-layout"> |
| <span id="how-to-change-the-batch-layout"></span><h3>How to change the batch layout<a class="headerlink" href="#how-to-change-the-batch-layout" title="Permalink to this headline">¶</a></h3> |
| <p>By default, the backend engine treats the first dimension of each data and label variable in data |
| iterators as the batch size (i.e. <code class="docutils literal"><span class="pre">NCHW</span></code> or <code class="docutils literal"><span class="pre">NT</span></code> layout). In order to override the axis for batch size, |
| the <code class="docutils literal"><span class="pre">provide_data</span></code> (and <code class="docutils literal"><span class="pre">provide_label</span></code> if there is label) properties should include the layouts. This |
| is especially useful in RNN since <code class="docutils literal"><span class="pre">TNC</span></code> layouts are often more efficient. For example:</p> |
| <div class="highlight-python"><div class="highlight"><pre><span></span><span class="nd">@property</span> |
| <span class="k">def</span> <span class="nf">provide_data</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| <span class="k">return</span> <span class="p">[</span><span class="n">DataDesc</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s1">'seq_var'</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="n">seq_length</span><span class="p">,</span> <span class="n">batch_size</span><span class="p">),</span> <span class="n">layout</span><span class="o">=</span><span class="s1">'TN'</span><span class="p">)]</span> |
| </pre></div> |
| </div> |
| <p>The backend engine will recognize the index of <code class="docutils literal"><span class="pre">N</span></code> in the <code class="docutils literal"><span class="pre">layout</span></code> as the axis for batch size.</p> |
| </div> |
| </div> |
| <div class="section" id="api-reference"> |
| <span id="api-reference"></span><h2>API Reference<a class="headerlink" href="#api-reference" title="Permalink to this headline">¶</a></h2> |
| <script src="../../../_static/js/auto_module_index.js" type="text/javascript"></script><div class="section" id="module-mxnet.io"> |
| <span id="mxnet-io-data-iterators"></span><span id="mxnet-io-data-iterators"></span><h3>mxnet.io - Data Iterators<a class="headerlink" href="#module-mxnet.io" title="Permalink to this headline">¶</a></h3> |
| <p>Data iterators for common data formats.</p> |
| <dl class="class"> |
| <dt id="mxnet.io.NDArrayIter"> |
| <em class="property">class </em><code class="descclassname">mxnet.io.</code><code class="descname">NDArrayIter</code><span class="sig-paren">(</span><em>data</em>, <em>label=None</em>, <em>batch_size=1</em>, <em>shuffle=False</em>, <em>last_batch_handle='pad'</em>, <em>data_name='data'</em>, <em>label_name='softmax_label'</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/io.html#NDArrayIter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.io.NDArrayIter" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Returns an iterator for <code class="docutils literal"><span class="pre">mx.nd.NDArray</span></code>, <code class="docutils literal"><span class="pre">numpy.ndarray</span></code>, <code class="docutils literal"><span class="pre">h5py.Dataset</span></code> |
| <code class="docutils literal"><span class="pre">mx.nd.sparse.CSRNDArray</span></code> or <code class="docutils literal"><span class="pre">scipy.sparse.csr_matrix</span></code>.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">data</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">40</span><span class="p">)</span><span class="o">.</span><span class="n">reshape</span><span class="p">((</span><span class="mi">10</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">))</span> |
| <span class="gp">>>> </span><span class="n">labels</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">([</span><span class="mi">10</span><span class="p">,</span> <span class="mi">1</span><span class="p">])</span> |
| <span class="gp">>>> </span><span class="n">dataiter</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">NDArrayIter</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">labels</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="kc">True</span><span class="p">,</span> <span class="n">last_batch_handle</span><span class="o">=</span><span class="s1">'discard'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">dataiter</span><span class="p">:</span> |
| <span class="gp">... </span> <span class="nb">print</span> <span class="n">batch</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span> |
| <span class="gp">... </span> <span class="n">batch</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">shape</span> |
| <span class="gp">...</span> |
| <span class="go">[[[ 36. 37.]</span> |
| <span class="go"> [ 38. 39.]]</span> |
| <span class="go"> [[ 16. 17.]</span> |
| <span class="go"> [ 18. 19.]]</span> |
| <span class="go"> [[ 12. 13.]</span> |
| <span class="go"> [ 14. 15.]]]</span> |
| <span class="go">(3L, 2L, 2L)</span> |
| <span class="go">[[[ 32. 33.]</span> |
| <span class="go"> [ 34. 35.]]</span> |
| <span class="go"> [[ 4. 5.]</span> |
| <span class="go"> [ 6. 7.]]</span> |
| <span class="go"> [[ 24. 25.]</span> |
| <span class="go"> [ 26. 27.]]]</span> |
| <span class="go">(3L, 2L, 2L)</span> |
| <span class="go">[[[ 8. 9.]</span> |
| <span class="go"> [ 10. 11.]]</span> |
| <span class="go"> [[ 20. 21.]</span> |
| <span class="go"> [ 22. 23.]]</span> |
| <span class="go"> [[ 28. 29.]</span> |
| <span class="go"> [ 30. 31.]]]</span> |
| <span class="go">(3L, 2L, 2L)</span> |
| <span class="gp">>>> </span><span class="n">dataiter</span><span class="o">.</span><span class="n">provide_data</span> <span class="c1"># Returns a list of `DataDesc`</span> |
| <span class="go">[DataDesc[data,(3, 2L, 2L),<type 'numpy.float32'>,NCHW]]</span> |
| <span class="gp">>>> </span><span class="n">dataiter</span><span class="o">.</span><span class="n">provide_label</span> <span class="c1"># Returns a list of `DataDesc`</span> |
| <span class="go">[DataDesc[softmax_label,(3, 1L),<type 'numpy.float32'>,NCHW]]</span> |
| </pre></div> |
| </div> |
| <p>In the above example, data is shuffled as <cite>shuffle</cite> parameter is set to <cite>True</cite> |
| and remaining examples are discarded as <cite>last_batch_handle</cite> parameter is set to <cite>discard</cite>.</p> |
| <p>Usage of <cite>last_batch_handle</cite> parameter:</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">dataiter</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">NDArrayIter</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">labels</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="kc">True</span><span class="p">,</span> <span class="n">last_batch_handle</span><span class="o">=</span><span class="s1">'pad'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">batchidx</span> <span class="o">=</span> <span class="mi">0</span> |
| <span class="gp">>>> </span><span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">dataiter</span><span class="p">:</span> |
| <span class="gp">... </span> <span class="n">batchidx</span> <span class="o">+=</span> <span class="mi">1</span> |
| <span class="gp">...</span> |
| <span class="gp">>>> </span><span class="n">batchidx</span> <span class="c1"># Padding added after the examples read are over. So, 10/3+1 batches are created.</span> |
| <span class="go">4</span> |
| <span class="gp">>>> </span><span class="n">dataiter</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">NDArrayIter</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">labels</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="kc">True</span><span class="p">,</span> <span class="n">last_batch_handle</span><span class="o">=</span><span class="s1">'discard'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">batchidx</span> <span class="o">=</span> <span class="mi">0</span> |
| <span class="gp">>>> </span><span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">dataiter</span><span class="p">:</span> |
| <span class="gp">... </span> <span class="n">batchidx</span> <span class="o">+=</span> <span class="mi">1</span> |
| <span class="gp">...</span> |
| <span class="gp">>>> </span><span class="n">batchidx</span> <span class="c1"># Remaining examples are discarded. So, 10/3 batches are created.</span> |
| <span class="go">3</span> |
| </pre></div> |
| </div> |
| <p><cite>NDArrayIter</cite> also supports multiple input and labels.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">data</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'data1'</span><span class="p">:</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">)),</span> <span class="s1">'data2'</span><span class="p">:</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">2</span><span class="p">))}</span> |
| <span class="gp">>>> </span><span class="n">label</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'label1'</span><span class="p">:</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">1</span><span class="p">)),</span> <span class="s1">'label2'</span><span class="p">:</span><span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="mi">20</span><span class="p">,</span><span class="mi">1</span><span class="p">))}</span> |
| <span class="gp">>>> </span><span class="n">dataiter</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">NDArrayIter</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">label</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="kc">True</span><span class="p">,</span> <span class="n">last_batch_handle</span><span class="o">=</span><span class="s1">'discard'</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <p><cite>NDArrayIter</cite> also supports <code class="docutils literal"><span class="pre">mx.nd.sparse.CSRNDArray</span></code> |
| with <cite>last_batch_handle</cite> set to <cite>discard</cite>.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">csr_data</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">40</span><span class="p">)</span><span class="o">.</span><span class="n">reshape</span><span class="p">((</span><span class="mi">10</span><span class="p">,</span><span class="mi">4</span><span class="p">)))</span><span class="o">.</span><span class="n">tostype</span><span class="p">(</span><span class="s1">'csr'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">labels</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">([</span><span class="mi">10</span><span class="p">,</span> <span class="mi">1</span><span class="p">])</span> |
| <span class="gp">>>> </span><span class="n">dataiter</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">NDArrayIter</span><span class="p">(</span><span class="n">csr_data</span><span class="p">,</span> <span class="n">labels</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="n">last_batch_handle</span><span class="o">=</span><span class="s1">'discard'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="p">[</span><span class="n">batch</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">dataiter</span><span class="p">]</span> |
| <span class="go">[</span> |
| <span class="go"><CSRNDArray 3x4 @cpu(0)>,</span> |
| <span class="go"><CSRNDArray 3x4 @cpu(0)>,</span> |
| <span class="go"><CSRNDArray 3x4 @cpu(0)>]</span> |
| </pre></div> |
| </div> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>data</strong> (<a class="reference internal" href="../ndarray/ndarray.html#mxnet.ndarray.array" title="mxnet.ndarray.array"><em>array</em></a><em> or </em><em>list of array</em><em> or </em><em>dict of string to array</em>) – The input data.</li> |
| <li><strong>label</strong> (<a class="reference internal" href="../ndarray/ndarray.html#mxnet.ndarray.array" title="mxnet.ndarray.array"><em>array</em></a><em> or </em><em>list of array</em><em> or </em><em>dict of string to array</em><em>, </em><em>optional</em>) – The input label.</li> |
| <li><strong>batch_size</strong> (<em>int</em>) – Batch size of data.</li> |
| <li><strong>shuffle</strong> (<em>bool</em><em>, </em><em>optional</em>) – Whether to shuffle the data. |
| Only supported if no h5py.Dataset inputs are used.</li> |
| <li><strong>last_batch_handle</strong> (<em>str</em><em>, </em><em>optional</em>) – How to handle the last batch. This parameter can be ‘pad’, ‘discard’ or |
| ‘roll_over’. ‘roll_over’ is intended for training and can cause problems |
| if used for prediction.</li> |
| <li><strong>data_name</strong> (<em>str</em><em>, </em><em>optional</em>) – The data name.</li> |
| <li><strong>label_name</strong> (<em>str</em><em>, </em><em>optional</em>) – The label name.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <dl class="attribute"> |
| <dt id="mxnet.io.NDArrayIter.provide_data"> |
| <code class="descname">provide_data</code><a class="headerlink" href="#mxnet.io.NDArrayIter.provide_data" title="Permalink to this definition">¶</a></dt> |
| <dd><p>The name and shape of data provided by this iterator.</p> |
| </dd></dl> |
| <dl class="attribute"> |
| <dt id="mxnet.io.NDArrayIter.provide_label"> |
| <code class="descname">provide_label</code><a class="headerlink" href="#mxnet.io.NDArrayIter.provide_label" title="Permalink to this definition">¶</a></dt> |
| <dd><p>The name and shape of label provided by this iterator.</p> |
| </dd></dl> |
| <dl class="method"> |
| <dt id="mxnet.io.NDArrayIter.hard_reset"> |
| <code class="descname">hard_reset</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/io.html#NDArrayIter.hard_reset"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.io.NDArrayIter.hard_reset" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Ignore roll over data and set to start.</p> |
| </dd></dl> |
| </dd></dl> |
| <dl class="function"> |
| <dt id="mxnet.io.CSVIter"> |
| <code class="descclassname">mxnet.io.</code><code class="descname">CSVIter</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="headerlink" href="#mxnet.io.CSVIter" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Returns the CSV file iterator.</p> |
| <p>In this function, the <cite>data_shape</cite> parameter is used to set the shape of each line of the input data. |
| If a row in an input file is <cite>1,2,3,4,5,6`</cite> and <cite>data_shape</cite> is (3,2), that row |
| will be reshaped, yielding the array [[1,2],[3,4],[5,6]] of shape (3,2).</p> |
| <p>By default, the <cite>CSVIter</cite> has <cite>round_batch</cite> parameter set to <code class="docutils literal"><span class="pre">True</span></code>. So, if <cite>batch_size</cite> |
| is 3 and there are 4 total rows in CSV file, 2 more examples |
| are consumed at the first round. If <cite>reset</cite> function is called after first round, |
| the call is ignored and remaining examples are returned in the second round.</p> |
| <p>If one wants all the instances in the second round after calling <cite>reset</cite>, make sure |
| to set <cite>round_batch</cite> to False.</p> |
| <p>If <code class="docutils literal"><span class="pre">data_csv</span> <span class="pre">=</span> <span class="pre">'data/'</span></code> is set, then all the files in this directory will be read.</p> |
| <p><code class="docutils literal"><span class="pre">reset()</span></code> is expected to be called only after a complete pass of data.</p> |
| <p>By default, the CSVIter parses all entries in the data file as float32 data type, |
| if <cite>dtype</cite> argument is set to be ‘int32’ or ‘int64’ then CSVIter will parse all entries in the file |
| as int32 or int64 data type accordingly.</p> |
| <p>Examples:</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span>// Contents of CSV file ``data/data.csv``. |
| 1,2,3 |
| 2,3,4 |
| 3,4,5 |
| 4,5,6 |
| |
| // Creates a `CSVIter` with `batch_size`=2 and default `round_batch`=True. |
| CSVIter = mx.io.CSVIter(data_csv = 'data/data.csv', data_shape = (3,), |
| batch_size = 2) |
| |
| // Two batches read from the above iterator are as follows: |
| [[ 1. 2. 3.] |
| [ 2. 3. 4.]] |
| [[ 3. 4. 5.] |
| [ 4. 5. 6.]] |
| |
| // Creates a `CSVIter` with default `round_batch` set to True. |
| CSVIter = mx.io.CSVIter(data_csv = 'data/data.csv', data_shape = (3,), |
| batch_size = 3) |
| |
| // Two batches read from the above iterator in the first pass are as follows: |
| [[1. 2. 3.] |
| [2. 3. 4.] |
| [3. 4. 5.]] |
| |
| [[4. 5. 6.] |
| [1. 2. 3.] |
| [2. 3. 4.]] |
| |
| // Now, `reset` method is called. |
| CSVIter.reset() |
| |
| // Batch read from the above iterator in the second pass is as follows: |
| [[ 3. 4. 5.] |
| [ 4. 5. 6.] |
| [ 1. 2. 3.]] |
| |
| // Creates a `CSVIter` with `round_batch`=False. |
| CSVIter = mx.io.CSVIter(data_csv = 'data/data.csv', data_shape = (3,), |
| batch_size = 3, round_batch=False) |
| |
| // Contents of two batches read from the above iterator in both passes, after calling |
| // `reset` method before second pass, is as follows: |
| [[1. 2. 3.] |
| [2. 3. 4.] |
| [3. 4. 5.]] |
| |
| [[4. 5. 6.] |
| [2. 3. 4.] |
| [3. 4. 5.]] |
| |
| // Creates a 'CSVIter' with `dtype`='int32' |
| CSVIter = mx.io.CSVIter(data_csv = 'data/data.csv', data_shape = (3,), |
| batch_size = 3, round_batch=False, dtype='int32') |
| |
| // Contents of two batches read from the above iterator in both passes, after calling |
| // `reset` method before second pass, is as follows: |
| [[1 2 3] |
| [2 3 4] |
| [3 4 5]] |
| |
| [[4 5 6] |
| [2 3 4] |
| [3 4 5]] |
| </pre></div> |
| </div> |
| <p>Defined in src/io/iter_csv.cc:L308</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>data_csv</strong> (<em>string</em><em>, </em><em>required</em>) – The input CSV file or a directory path.</li> |
| <li><strong>data_shape</strong> (<em>Shape</em><em>(</em><em>tuple</em><em>)</em><em>, </em><em>required</em>) – The shape of one example.</li> |
| <li><strong>label_csv</strong> (<em>string</em><em>, </em><em>optional</em><em>, </em><em>default='NULL'</em>) – The input CSV file or a directory path. If NULL, all labels will be returned as 0.</li> |
| <li><strong>label_shape</strong> (<em>Shape</em><em>(</em><em>tuple</em><em>)</em><em>, </em><em>optional</em><em>, </em><em>default=</em><em>[</em><em>1</em><em>]</em>) – The shape of one label.</li> |
| <li><strong>batch_size</strong> (<em>int</em><em> (</em><em>non-negative</em><em>)</em><em>, </em><em>required</em>) – Batch size.</li> |
| <li><strong>round_batch</strong> (<em>boolean</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Whether to use round robin to handle overflow batch or not.</li> |
| <li><strong>prefetch_buffer</strong> (<em>long</em><em> (</em><em>non-negative</em><em>)</em><em>, </em><em>optional</em><em>, </em><em>default=4</em>) – Maximum number of batches to prefetch.</li> |
| <li><strong>dtype</strong> (<em>{None</em><em>, </em><em>'float16'</em><em>, </em><em>'float32'</em><em>, </em><em>'float64'</em><em>, </em><em>'int32'</em><em>, </em><em>'int64'</em><em>, </em><em>'uint8'}</em><em>,</em><em>optional</em><em>, </em><em>default='None'</em>) – Output data type. <code class="docutils literal"><span class="pre">None</span></code> means no change.</li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first">The result iterator.</p> |
| </td> |
| </tr> |
| <tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body"><p class="first last"><a class="reference internal" href="#mxnet.io.MXDataIter" title="mxnet.io.MXDataIter">MXDataIter</a></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| <dl class="function"> |
| <dt id="mxnet.io.LibSVMIter"> |
| <code class="descclassname">mxnet.io.</code><code class="descname">LibSVMIter</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="headerlink" href="#mxnet.io.LibSVMIter" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Returns the LibSVM iterator which returns data with <cite>csr</cite> |
| storage type. This iterator is experimental and should be used with care.</p> |
| <p>The input data is stored in a format similar to LibSVM file format, except that the <strong>indices |
| are expected to be zero-based instead of one-based, and the column indices for each row are |
| expected to be sorted in ascending order</strong>. Details of the LibSVM format are available |
| <a class="reference external" href="https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/">here.</a></p> |
| <p>The <cite>data_shape</cite> parameter is used to set the shape of each line of the data. |
| The dimension of both <cite>data_shape</cite> and <cite>label_shape</cite> are expected to be 1.</p> |
| <p>The <cite>data_libsvm</cite> parameter is used to set the path input LibSVM file. |
| When it is set to a directory, all the files in the directory will be read.</p> |
| <p>When <cite>label_libsvm</cite> is set to <code class="docutils literal"><span class="pre">NULL</span></code>, both data and label are read from the file specified |
| by <cite>data_libsvm</cite>. In this case, the data is stored in <cite>csr</cite> storage type, while the label is a 1D |
| dense array.</p> |
| <p>The <cite>LibSVMIter</cite> only support <cite>round_batch</cite> parameter set to <code class="docutils literal"><span class="pre">True</span></code>. Therefore, if <cite>batch_size</cite> |
| is 3 and there are 4 total rows in libsvm file, 2 more examples are consumed at the first round.</p> |
| <p>When <cite>num_parts</cite> and <cite>part_index</cite> are provided, the data is split into <cite>num_parts</cite> partitions, |
| and the iterator only reads the <cite>part_index</cite>-th partition. However, the partitions are not |
| guaranteed to be even.</p> |
| <p><code class="docutils literal"><span class="pre">reset()</span></code> is expected to be called only after a complete pass of data.</p> |
| <p>Example:</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Contents of libsvm file ``data.t``.</span> |
| <span class="mf">1.0</span> <span class="mi">0</span><span class="p">:</span><span class="mf">0.5</span> <span class="mi">2</span><span class="p">:</span><span class="mf">1.2</span> |
| <span class="o">-</span><span class="mf">2.0</span> |
| <span class="o">-</span><span class="mf">3.0</span> <span class="mi">0</span><span class="p">:</span><span class="mf">0.6</span> <span class="mi">1</span><span class="p">:</span><span class="mf">2.4</span> <span class="mi">2</span><span class="p">:</span><span class="mf">1.2</span> |
| <span class="mi">4</span> <span class="mi">2</span><span class="p">:</span><span class="o">-</span><span class="mf">1.2</span> |
| |
| <span class="c1"># Creates a `LibSVMIter` with `batch_size`=3.</span> |
| <span class="o">>>></span> <span class="n">data_iter</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">LibSVMIter</span><span class="p">(</span><span class="n">data_libsvm</span> <span class="o">=</span> <span class="s1">'data.t'</span><span class="p">,</span> <span class="n">data_shape</span> <span class="o">=</span> <span class="p">(</span><span class="mi">3</span><span class="p">,),</span> <span class="n">batch_size</span> <span class="o">=</span> <span class="mi">3</span><span class="p">)</span> |
| <span class="c1"># The data of the first batch is stored in csr storage type</span> |
| <span class="o">>>></span> <span class="n">batch</span> <span class="o">=</span> <span class="n">data_iter</span><span class="o">.</span><span class="n">next</span><span class="p">()</span> |
| <span class="o">>>></span> <span class="n">csr</span> <span class="o">=</span> <span class="n">batch</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> |
| <span class="o"><</span><span class="n">CSRNDArray</span> <span class="mi">3</span><span class="n">x3</span> <span class="nd">@cpu</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">></span> |
| <span class="o">>>></span> <span class="n">csr</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span> |
| <span class="p">[[</span> <span class="mf">0.5</span> <span class="mf">0.</span> <span class="mf">1.2</span> <span class="p">]</span> |
| <span class="p">[</span> <span class="mf">0.</span> <span class="mf">0.</span> <span class="mf">0.</span> <span class="p">]</span> |
| <span class="p">[</span> <span class="mf">0.6</span> <span class="mf">2.4</span> <span class="mf">1.2</span><span class="p">]]</span> |
| <span class="c1"># The label of first batch</span> |
| <span class="o">>>></span> <span class="n">label</span> <span class="o">=</span> <span class="n">batch</span><span class="o">.</span><span class="n">label</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> |
| <span class="o">>>></span> <span class="n">label</span> |
| <span class="p">[</span> <span class="mf">1.</span> <span class="o">-</span><span class="mf">2.</span> <span class="o">-</span><span class="mf">3.</span><span class="p">]</span> |
| <span class="o"><</span><span class="n">NDArray</span> <span class="mi">3</span> <span class="nd">@cpu</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">></span> |
| |
| <span class="o">>>></span> <span class="n">second_batch</span> <span class="o">=</span> <span class="n">data_iter</span><span class="o">.</span><span class="n">next</span><span class="p">()</span> |
| <span class="c1"># The data of the second batch</span> |
| <span class="o">>>></span> <span class="n">second_batch</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span> |
| <span class="p">[[</span> <span class="mf">0.</span> <span class="mf">0.</span> <span class="o">-</span><span class="mf">1.2</span> <span class="p">]</span> |
| <span class="p">[</span> <span class="mf">0.5</span> <span class="mf">0.</span> <span class="mf">1.2</span> <span class="p">]</span> |
| <span class="p">[</span> <span class="mf">0.</span> <span class="mf">0.</span> <span class="mf">0.</span> <span class="p">]]</span> |
| <span class="c1"># The label of the second batch</span> |
| <span class="o">>>></span> <span class="n">second_batch</span><span class="o">.</span><span class="n">label</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span> |
| <span class="p">[</span> <span class="mf">4.</span> <span class="mf">1.</span> <span class="o">-</span><span class="mf">2.</span><span class="p">]</span> |
| |
| <span class="o">>>></span> <span class="n">data_iter</span><span class="o">.</span><span class="n">reset</span><span class="p">()</span> |
| <span class="c1"># To restart the iterator for the second pass of the data</span> |
| </pre></div> |
| </div> |
| <p>When <cite>label_libsvm</cite> is set to the path to another LibSVM file, |
| data is read from <cite>data_libsvm</cite> and label from <cite>label_libsvm</cite>. |
| In this case, both data and label are stored in the csr format. |
| If the label column in the <cite>data_libsvm</cite> file is ignored.</p> |
| <p>Example:</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Contents of libsvm file ``label.t``</span> |
| <span class="mf">1.0</span> |
| <span class="o">-</span><span class="mf">2.0</span> <span class="mi">0</span><span class="p">:</span><span class="mf">0.125</span> |
| <span class="o">-</span><span class="mf">3.0</span> <span class="mi">2</span><span class="p">:</span><span class="mf">1.2</span> |
| <span class="mi">4</span> <span class="mi">1</span><span class="p">:</span><span class="mf">1.0</span> <span class="mi">2</span><span class="p">:</span><span class="o">-</span><span class="mf">1.2</span> |
| |
| <span class="c1"># Creates a `LibSVMIter` with specified label file</span> |
| <span class="o">>>></span> <span class="n">data_iter</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">LibSVMIter</span><span class="p">(</span><span class="n">data_libsvm</span> <span class="o">=</span> <span class="s1">'data.t'</span><span class="p">,</span> <span class="n">data_shape</span> <span class="o">=</span> <span class="p">(</span><span class="mi">3</span><span class="p">,),</span> |
| <span class="n">label_libsvm</span> <span class="o">=</span> <span class="s1">'label.t'</span><span class="p">,</span> <span class="n">label_shape</span> <span class="o">=</span> <span class="p">(</span><span class="mi">3</span><span class="p">,),</span> <span class="n">batch_size</span> <span class="o">=</span> <span class="mi">3</span><span class="p">)</span> |
| |
| <span class="c1"># Both data and label are in csr storage type</span> |
| <span class="o">>>></span> <span class="n">batch</span> <span class="o">=</span> <span class="n">data_iter</span><span class="o">.</span><span class="n">next</span><span class="p">()</span> |
| <span class="o">>>></span> <span class="n">csr_data</span> <span class="o">=</span> <span class="n">batch</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> |
| <span class="o"><</span><span class="n">CSRNDArray</span> <span class="mi">3</span><span class="n">x3</span> <span class="nd">@cpu</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">></span> |
| <span class="o">>>></span> <span class="n">csr_data</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span> |
| <span class="p">[[</span> <span class="mf">0.5</span> <span class="mf">0.</span> <span class="mf">1.2</span> <span class="p">]</span> |
| <span class="p">[</span> <span class="mf">0.</span> <span class="mf">0.</span> <span class="mf">0.</span> <span class="p">]</span> |
| <span class="p">[</span> <span class="mf">0.6</span> <span class="mf">2.4</span> <span class="mf">1.2</span> <span class="p">]]</span> |
| <span class="o">>>></span> <span class="n">csr_label</span> <span class="o">=</span> <span class="n">batch</span><span class="o">.</span><span class="n">label</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> |
| <span class="o"><</span><span class="n">CSRNDArray</span> <span class="mi">3</span><span class="n">x3</span> <span class="nd">@cpu</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">></span> |
| <span class="o">>>></span> <span class="n">csr_label</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span> |
| <span class="p">[[</span> <span class="mf">0.</span> <span class="mf">0.</span> <span class="mf">0.</span> <span class="p">]</span> |
| <span class="p">[</span> <span class="mf">0.125</span> <span class="mf">0.</span> <span class="mf">0.</span> <span class="p">]</span> |
| <span class="p">[</span> <span class="mf">0.</span> <span class="mf">0.</span> <span class="mf">1.2</span> <span class="p">]]</span> |
| </pre></div> |
| </div> |
| <p>Defined in src/io/iter_libsvm.cc:L298</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>data_libsvm</strong> (<em>string</em><em>, </em><em>required</em>) – The input zero-base indexed LibSVM data file or a directory path.</li> |
| <li><strong>data_shape</strong> (<em>Shape</em><em>(</em><em>tuple</em><em>)</em><em>, </em><em>required</em>) – The shape of one example.</li> |
| <li><strong>label_libsvm</strong> (<em>string</em><em>, </em><em>optional</em><em>, </em><em>default='NULL'</em>) – The input LibSVM label file or a directory path. If NULL, all labels will be read from <code class="docutils literal"><span class="pre">data_libsvm</span></code>.</li> |
| <li><strong>label_shape</strong> (<em>Shape</em><em>(</em><em>tuple</em><em>)</em><em>, </em><em>optional</em><em>, </em><em>default=</em><em>[</em><em>1</em><em>]</em>) – The shape of one label.</li> |
| <li><strong>num_parts</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='1'</em>) – partition the data into multiple parts</li> |
| <li><strong>part_index</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – the index of the part will read</li> |
| <li><strong>batch_size</strong> (<em>int</em><em> (</em><em>non-negative</em><em>)</em><em>, </em><em>required</em>) – Batch size.</li> |
| <li><strong>round_batch</strong> (<em>boolean</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Whether to use round robin to handle overflow batch or not.</li> |
| <li><strong>prefetch_buffer</strong> (<em>long</em><em> (</em><em>non-negative</em><em>)</em><em>, </em><em>optional</em><em>, </em><em>default=4</em>) – Maximum number of batches to prefetch.</li> |
| <li><strong>dtype</strong> (<em>{None</em><em>, </em><em>'float16'</em><em>, </em><em>'float32'</em><em>, </em><em>'float64'</em><em>, </em><em>'int32'</em><em>, </em><em>'int64'</em><em>, </em><em>'uint8'}</em><em>,</em><em>optional</em><em>, </em><em>default='None'</em>) – Output data type. <code class="docutils literal"><span class="pre">None</span></code> means no change.</li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first">The result iterator.</p> |
| </td> |
| </tr> |
| <tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body"><p class="first last"><a class="reference internal" href="#mxnet.io.MXDataIter" title="mxnet.io.MXDataIter">MXDataIter</a></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| <dl class="function"> |
| <dt id="mxnet.io.ImageRecordIter"> |
| <code class="descclassname">mxnet.io.</code><code class="descname">ImageRecordIter</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="headerlink" href="#mxnet.io.ImageRecordIter" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Iterates on image RecordIO files</p> |
| <p>Reads batches of images from .rec RecordIO files. One can use <code class="docutils literal"><span class="pre">im2rec.py</span></code> tool |
| (in tools/) to pack raw image files into RecordIO files. This iterator is less |
| flexible to customization but is fast and has lot of language bindings. To |
| iterate over raw images directly use <code class="docutils literal"><span class="pre">ImageIter</span></code> instead (in Python).</p> |
| <p>Example:</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">data_iter</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ImageRecordIter</span><span class="p">(</span> |
| <span class="n">path_imgrec</span><span class="o">=</span><span class="s2">"./sample.rec"</span><span class="p">,</span> <span class="c1"># The target record file.</span> |
| <span class="n">data_shape</span><span class="o">=</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">227</span><span class="p">,</span> <span class="mi">227</span><span class="p">),</span> <span class="c1"># Output data shape; 227x227 region will be cropped from the original image.</span> |
| <span class="n">batch_size</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="c1"># Number of items per batch.</span> |
| <span class="n">resize</span><span class="o">=</span><span class="mi">256</span> <span class="c1"># Resize the shorter edge to 256 before cropping.</span> |
| <span class="c1"># You can specify more augmentation options. Use help(mx.io.ImageRecordIter) to see all the options.</span> |
| <span class="p">)</span> |
| <span class="c1"># You can now use the data_iter to access batches of images.</span> |
| <span class="n">batch</span> <span class="o">=</span> <span class="n">data_iter</span><span class="o">.</span><span class="n">next</span><span class="p">()</span> <span class="c1"># first batch.</span> |
| <span class="n">images</span> <span class="o">=</span> <span class="n">batch</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="c1"># This will contain 4 (=batch_size) images each of 3x227x227.</span> |
| <span class="c1"># process the images</span> |
| <span class="o">...</span> |
| <span class="n">data_iter</span><span class="o">.</span><span class="n">reset</span><span class="p">()</span> <span class="c1"># To restart the iterator from the beginning.</span> |
| </pre></div> |
| </div> |
| <p>Defined in src/io/iter_image_recordio_2.cc:L751</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>path_imglist</strong> (<em>string</em><em>, </em><em>optional</em><em>, </em><em>default=''</em>) – Path to the image list (.lst) file. Generally created with tools/im2rec.py. Format (Tab separated): <index of record> <one or more labels> <relative path from root folder>.</li> |
| <li><strong>path_imgrec</strong> (<em>string</em><em>, </em><em>optional</em><em>, </em><em>default=''</em>) – Path to the image RecordIO (.rec) file or a directory path. Created with tools/im2rec.py.</li> |
| <li><strong>path_imgidx</strong> (<em>string</em><em>, </em><em>optional</em><em>, </em><em>default=''</em>) – Path to the image RecordIO index (.idx) file. Created with tools/im2rec.py.</li> |
| <li><strong>aug_seq</strong> (<em>string</em><em>, </em><em>optional</em><em>, </em><em>default='aug_default'</em>) – The augmenter names to represent sequence of augmenters to be applied, seperated by comma. Additional keyword parameters will be seen by these augmenters.</li> |
| <li><strong>label_width</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='1'</em>) – The number of labels per image.</li> |
| <li><strong>data_shape</strong> (<em>Shape</em><em>(</em><em>tuple</em><em>)</em><em>, </em><em>required</em>) – The shape of one output image in (channels, height, width) format.</li> |
| <li><strong>preprocess_threads</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='4'</em>) – The number of threads to do preprocessing.</li> |
| <li><strong>verbose</strong> (<em>boolean</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – If or not output verbose information.</li> |
| <li><strong>num_parts</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='1'</em>) – Virtually partition the data into these many parts.</li> |
| <li><strong>part_index</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – The <em>i</em>-th virtual partition to be read.</li> |
| <li><strong>shuffle_chunk_size</strong> (<em>long</em><em> (</em><em>non-negative</em><em>)</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – The data shuffle buffer size in MB. Only valid if shuffle is true.</li> |
| <li><strong>shuffle_chunk_seed</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – The random seed for shuffling</li> |
| <li><strong>shuffle</strong> (<em>boolean</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Whether to shuffle data randomly or not.</li> |
| <li><strong>seed</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – The random seed.</li> |
| <li><strong>batch_size</strong> (<em>int</em><em> (</em><em>non-negative</em><em>)</em><em>, </em><em>required</em>) – Batch size.</li> |
| <li><strong>round_batch</strong> (<em>boolean</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Whether to use round robin to handle overflow batch or not.</li> |
| <li><strong>prefetch_buffer</strong> (<em>long</em><em> (</em><em>non-negative</em><em>)</em><em>, </em><em>optional</em><em>, </em><em>default=4</em>) – Maximum number of batches to prefetch.</li> |
| <li><strong>dtype</strong> (<em>{None</em><em>, </em><em>'float16'</em><em>, </em><em>'float32'</em><em>, </em><em>'float64'</em><em>, </em><em>'int32'</em><em>, </em><em>'int64'</em><em>, </em><em>'uint8'}</em><em>,</em><em>optional</em><em>, </em><em>default='None'</em>) – Output data type. <code class="docutils literal"><span class="pre">None</span></code> means no change.</li> |
| <li><strong>resize</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='-1'</em>) – Down scale the shorter edge to a new size before applying other augmentations.</li> |
| <li><strong>rand_crop</strong> (<em>boolean</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – If or not randomly crop the image</li> |
| <li><strong>random_resized_crop</strong> (<em>boolean</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – If or not perform random resized cropping on the image, as a standard preprocessing for resnet training on ImageNet data.</li> |
| <li><strong>max_rotate_angle</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – Rotate by a random degree in <code class="docutils literal"><span class="pre">[-v,</span> <span class="pre">v]</span></code></li> |
| <li><strong>max_aspect_ratio</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Change the aspect (namely width/height) to a random value. If min_aspect_ratio is None then the aspect ratio ins sampled from [1 - max_aspect_ratio, 1 + max_aspect_ratio], else it is in <code class="docutils literal"><span class="pre">[min_aspect_ratio,</span> <span class="pre">max_aspect_ratio]</span></code></li> |
| <li><strong>min_aspect_ratio</strong> (<em>float</em><em> or </em><em>None</em><em>, </em><em>optional</em><em>, </em><em>default=None</em>) – Change the aspect (namely width/height) to a random value in <code class="docutils literal"><span class="pre">[min_aspect_ratio,</span> <span class="pre">max_aspect_ratio]</span></code></li> |
| <li><strong>max_shear_ratio</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Apply a shear transformation (namely <code class="docutils literal"><span class="pre">(x,y)->(x+my,y)</span></code>) with <code class="docutils literal"><span class="pre">m</span></code> randomly chose from <code class="docutils literal"><span class="pre">[-max_shear_ratio,</span> <span class="pre">max_shear_ratio]</span></code></li> |
| <li><strong>max_crop_size</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='-1'</em>) – Crop both width and height into a random size in <code class="docutils literal"><span class="pre">[min_crop_size,</span> <span class="pre">max_crop_size].``Ignored</span> <span class="pre">if</span> <span class="pre">``random_resized_crop</span></code> is True.</li> |
| <li><strong>min_crop_size</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='-1'</em>) – Crop both width and height into a random size in <code class="docutils literal"><span class="pre">[min_crop_size,</span> <span class="pre">max_crop_size].``Ignored</span> <span class="pre">if</span> <span class="pre">``random_resized_crop</span></code> is True.</li> |
| <li><strong>max_random_scale</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Resize into <code class="docutils literal"><span class="pre">[width*s,</span> <span class="pre">height*s]</span></code> with <code class="docutils literal"><span class="pre">s</span></code> randomly chosen from <code class="docutils literal"><span class="pre">[min_random_scale,</span> <span class="pre">max_random_scale]</span></code>. Ignored if <code class="docutils literal"><span class="pre">random_resized_crop</span></code> is True.</li> |
| <li><strong>min_random_scale</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Resize into <code class="docutils literal"><span class="pre">[width*s,</span> <span class="pre">height*s]</span></code> with <code class="docutils literal"><span class="pre">s</span></code> randomly chosen from <code class="docutils literal"><span class="pre">[min_random_scale,</span> <span class="pre">max_random_scale]``Ignored</span> <span class="pre">if</span> <span class="pre">``random_resized_crop</span></code> is True.</li> |
| <li><strong>max_random_area</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Change the area (namely width * height) to a random value in <code class="docutils literal"><span class="pre">[min_random_area,</span> <span class="pre">max_random_area]</span></code>. Ignored if <code class="docutils literal"><span class="pre">random_resized_crop</span></code> is False.</li> |
| <li><strong>min_random_area</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Change the area (namely width * height) to a random value in <code class="docutils literal"><span class="pre">[min_random_area,</span> <span class="pre">max_random_area]</span></code>. Ignored if <code class="docutils literal"><span class="pre">random_resized_crop</span></code> is False.</li> |
| <li><strong>max_img_size</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=1e+10</em>) – Set the maximal width and height after all resize and rotate argumentation are applied</li> |
| <li><strong>min_img_size</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Set the minimal width and height after all resize and rotate argumentation are applied</li> |
| <li><strong>brightness</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Add a random value in <code class="docutils literal"><span class="pre">[-brightness,</span> <span class="pre">brightness]</span></code> to the brightness of image.</li> |
| <li><strong>contrast</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Add a random value in <code class="docutils literal"><span class="pre">[-contrast,</span> <span class="pre">contrast]</span></code> to the contrast of image.</li> |
| <li><strong>saturation</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Add a random value in <code class="docutils literal"><span class="pre">[-saturation,</span> <span class="pre">saturation]</span></code> to the saturation of image.</li> |
| <li><strong>pca_noise</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Add PCA based noise to the image.</li> |
| <li><strong>random_h</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – Add a random value in <code class="docutils literal"><span class="pre">[-random_h,</span> <span class="pre">random_h]</span></code> to the H channel in HSL color space.</li> |
| <li><strong>random_s</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – Add a random value in <code class="docutils literal"><span class="pre">[-random_s,</span> <span class="pre">random_s]</span></code> to the S channel in HSL color space.</li> |
| <li><strong>random_l</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – Add a random value in <code class="docutils literal"><span class="pre">[-random_l,</span> <span class="pre">random_l]</span></code> to the L channel in HSL color space.</li> |
| <li><strong>rotate</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='-1'</em>) – Rotate by an angle. If set, it overwrites the <code class="docutils literal"><span class="pre">max_rotate_angle</span></code> option.</li> |
| <li><strong>fill_value</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='255'</em>) – Set the padding pixels value to <code class="docutils literal"><span class="pre">fill_value</span></code>.</li> |
| <li><strong>inter_method</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='1'</em>) – The interpolation method: 0-NN 1-bilinear 2-cubic 3-area 4-lanczos4 9-auto 10-rand.</li> |
| <li><strong>pad</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – Change size from <code class="docutils literal"><span class="pre">[width,</span> <span class="pre">height]</span></code> into <code class="docutils literal"><span class="pre">[pad</span> <span class="pre">+</span> <span class="pre">width</span> <span class="pre">+</span> <span class="pre">pad,</span> <span class="pre">pad</span> <span class="pre">+</span> <span class="pre">height</span> <span class="pre">+</span> <span class="pre">pad]</span></code> by padding pixes</li> |
| <li><strong>seed_aug</strong> (<em>int</em><em> or </em><em>None</em><em>, </em><em>optional</em><em>, </em><em>default='None'</em>) – Random seed for augmentations.</li> |
| <li><strong>mirror</strong> (<em>boolean</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Whether to mirror the image or not. If true, images are flipped along the horizontal axis.</li> |
| <li><strong>rand_mirror</strong> (<em>boolean</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Whether to randomly mirror images or not. If true, 50% of the images will be randomly mirrored (flipped along the horizontal axis)</li> |
| <li><strong>mean_img</strong> (<em>string</em><em>, </em><em>optional</em><em>, </em><em>default=''</em>) – Filename of the mean image.</li> |
| <li><strong>mean_r</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – The mean value to be subtracted on the R channel</li> |
| <li><strong>mean_g</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – The mean value to be subtracted on the G channel</li> |
| <li><strong>mean_b</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – The mean value to be subtracted on the B channel</li> |
| <li><strong>mean_a</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – The mean value to be subtracted on the alpha channel</li> |
| <li><strong>std_r</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Augmentation Param: Standard deviation on R channel.</li> |
| <li><strong>std_g</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Augmentation Param: Standard deviation on G channel.</li> |
| <li><strong>std_b</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Augmentation Param: Standard deviation on B channel.</li> |
| <li><strong>std_a</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Augmentation Param: Standard deviation on Alpha channel.</li> |
| <li><strong>scale</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Multiply the image with a scale value.</li> |
| <li><strong>max_random_contrast</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Change the contrast with a value randomly chosen from <code class="docutils literal"><span class="pre">[-max_random_contrast,</span> <span class="pre">max_random_contrast]</span></code></li> |
| <li><strong>max_random_illumination</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Change the illumination with a value randomly chosen from <code class="docutils literal"><span class="pre">[-max_random_illumination,</span> <span class="pre">max_random_illumination]</span></code></li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first">The result iterator.</p> |
| </td> |
| </tr> |
| <tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body"><p class="first last"><a class="reference internal" href="#mxnet.io.MXDataIter" title="mxnet.io.MXDataIter">MXDataIter</a></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| <dl class="function"> |
| <dt id="mxnet.io.ImageRecordUInt8Iter"> |
| <code class="descclassname">mxnet.io.</code><code class="descname">ImageRecordUInt8Iter</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="headerlink" href="#mxnet.io.ImageRecordUInt8Iter" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Iterating on image RecordIO files</p> |
| <p>This iterator is identical to <code class="docutils literal"><span class="pre">ImageRecordIter</span></code> except for using <code class="docutils literal"><span class="pre">uint8</span></code> as |
| the data type instead of <code class="docutils literal"><span class="pre">float</span></code>.</p> |
| <p>Defined in src/io/iter_image_recordio_2.cc:L768</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>path_imglist</strong> (<em>string</em><em>, </em><em>optional</em><em>, </em><em>default=''</em>) – Path to the image list (.lst) file. Generally created with tools/im2rec.py. Format (Tab separated): <index of record> <one or more labels> <relative path from root folder>.</li> |
| <li><strong>path_imgrec</strong> (<em>string</em><em>, </em><em>optional</em><em>, </em><em>default=''</em>) – Path to the image RecordIO (.rec) file or a directory path. Created with tools/im2rec.py.</li> |
| <li><strong>path_imgidx</strong> (<em>string</em><em>, </em><em>optional</em><em>, </em><em>default=''</em>) – Path to the image RecordIO index (.idx) file. Created with tools/im2rec.py.</li> |
| <li><strong>aug_seq</strong> (<em>string</em><em>, </em><em>optional</em><em>, </em><em>default='aug_default'</em>) – The augmenter names to represent sequence of augmenters to be applied, seperated by comma. Additional keyword parameters will be seen by these augmenters.</li> |
| <li><strong>label_width</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='1'</em>) – The number of labels per image.</li> |
| <li><strong>data_shape</strong> (<em>Shape</em><em>(</em><em>tuple</em><em>)</em><em>, </em><em>required</em>) – The shape of one output image in (channels, height, width) format.</li> |
| <li><strong>preprocess_threads</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='4'</em>) – The number of threads to do preprocessing.</li> |
| <li><strong>verbose</strong> (<em>boolean</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – If or not output verbose information.</li> |
| <li><strong>num_parts</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='1'</em>) – Virtually partition the data into these many parts.</li> |
| <li><strong>part_index</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – The <em>i</em>-th virtual partition to be read.</li> |
| <li><strong>shuffle_chunk_size</strong> (<em>long</em><em> (</em><em>non-negative</em><em>)</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – The data shuffle buffer size in MB. Only valid if shuffle is true.</li> |
| <li><strong>shuffle_chunk_seed</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – The random seed for shuffling</li> |
| <li><strong>shuffle</strong> (<em>boolean</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Whether to shuffle data randomly or not.</li> |
| <li><strong>seed</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – The random seed.</li> |
| <li><strong>batch_size</strong> (<em>int</em><em> (</em><em>non-negative</em><em>)</em><em>, </em><em>required</em>) – Batch size.</li> |
| <li><strong>round_batch</strong> (<em>boolean</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Whether to use round robin to handle overflow batch or not.</li> |
| <li><strong>prefetch_buffer</strong> (<em>long</em><em> (</em><em>non-negative</em><em>)</em><em>, </em><em>optional</em><em>, </em><em>default=4</em>) – Maximum number of batches to prefetch.</li> |
| <li><strong>dtype</strong> (<em>{None</em><em>, </em><em>'float16'</em><em>, </em><em>'float32'</em><em>, </em><em>'float64'</em><em>, </em><em>'int32'</em><em>, </em><em>'int64'</em><em>, </em><em>'uint8'}</em><em>,</em><em>optional</em><em>, </em><em>default='None'</em>) – Output data type. <code class="docutils literal"><span class="pre">None</span></code> means no change.</li> |
| <li><strong>resize</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='-1'</em>) – Down scale the shorter edge to a new size before applying other augmentations.</li> |
| <li><strong>rand_crop</strong> (<em>boolean</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – If or not randomly crop the image</li> |
| <li><strong>random_resized_crop</strong> (<em>boolean</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – If or not perform random resized cropping on the image, as a standard preprocessing for resnet training on ImageNet data.</li> |
| <li><strong>max_rotate_angle</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – Rotate by a random degree in <code class="docutils literal"><span class="pre">[-v,</span> <span class="pre">v]</span></code></li> |
| <li><strong>max_aspect_ratio</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Change the aspect (namely width/height) to a random value. If min_aspect_ratio is None then the aspect ratio ins sampled from [1 - max_aspect_ratio, 1 + max_aspect_ratio], else it is in <code class="docutils literal"><span class="pre">[min_aspect_ratio,</span> <span class="pre">max_aspect_ratio]</span></code></li> |
| <li><strong>min_aspect_ratio</strong> (<em>float</em><em> or </em><em>None</em><em>, </em><em>optional</em><em>, </em><em>default=None</em>) – Change the aspect (namely width/height) to a random value in <code class="docutils literal"><span class="pre">[min_aspect_ratio,</span> <span class="pre">max_aspect_ratio]</span></code></li> |
| <li><strong>max_shear_ratio</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Apply a shear transformation (namely <code class="docutils literal"><span class="pre">(x,y)->(x+my,y)</span></code>) with <code class="docutils literal"><span class="pre">m</span></code> randomly chose from <code class="docutils literal"><span class="pre">[-max_shear_ratio,</span> <span class="pre">max_shear_ratio]</span></code></li> |
| <li><strong>max_crop_size</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='-1'</em>) – Crop both width and height into a random size in <code class="docutils literal"><span class="pre">[min_crop_size,</span> <span class="pre">max_crop_size].``Ignored</span> <span class="pre">if</span> <span class="pre">``random_resized_crop</span></code> is True.</li> |
| <li><strong>min_crop_size</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='-1'</em>) – Crop both width and height into a random size in <code class="docutils literal"><span class="pre">[min_crop_size,</span> <span class="pre">max_crop_size].``Ignored</span> <span class="pre">if</span> <span class="pre">``random_resized_crop</span></code> is True.</li> |
| <li><strong>max_random_scale</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Resize into <code class="docutils literal"><span class="pre">[width*s,</span> <span class="pre">height*s]</span></code> with <code class="docutils literal"><span class="pre">s</span></code> randomly chosen from <code class="docutils literal"><span class="pre">[min_random_scale,</span> <span class="pre">max_random_scale]</span></code>. Ignored if <code class="docutils literal"><span class="pre">random_resized_crop</span></code> is True.</li> |
| <li><strong>min_random_scale</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Resize into <code class="docutils literal"><span class="pre">[width*s,</span> <span class="pre">height*s]</span></code> with <code class="docutils literal"><span class="pre">s</span></code> randomly chosen from <code class="docutils literal"><span class="pre">[min_random_scale,</span> <span class="pre">max_random_scale]``Ignored</span> <span class="pre">if</span> <span class="pre">``random_resized_crop</span></code> is True.</li> |
| <li><strong>max_random_area</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Change the area (namely width * height) to a random value in <code class="docutils literal"><span class="pre">[min_random_area,</span> <span class="pre">max_random_area]</span></code>. Ignored if <code class="docutils literal"><span class="pre">random_resized_crop</span></code> is False.</li> |
| <li><strong>min_random_area</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Change the area (namely width * height) to a random value in <code class="docutils literal"><span class="pre">[min_random_area,</span> <span class="pre">max_random_area]</span></code>. Ignored if <code class="docutils literal"><span class="pre">random_resized_crop</span></code> is False.</li> |
| <li><strong>max_img_size</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=1e+10</em>) – Set the maximal width and height after all resize and rotate argumentation are applied</li> |
| <li><strong>min_img_size</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Set the minimal width and height after all resize and rotate argumentation are applied</li> |
| <li><strong>brightness</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Add a random value in <code class="docutils literal"><span class="pre">[-brightness,</span> <span class="pre">brightness]</span></code> to the brightness of image.</li> |
| <li><strong>contrast</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Add a random value in <code class="docutils literal"><span class="pre">[-contrast,</span> <span class="pre">contrast]</span></code> to the contrast of image.</li> |
| <li><strong>saturation</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Add a random value in <code class="docutils literal"><span class="pre">[-saturation,</span> <span class="pre">saturation]</span></code> to the saturation of image.</li> |
| <li><strong>pca_noise</strong> (<em>float</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Add PCA based noise to the image.</li> |
| <li><strong>random_h</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – Add a random value in <code class="docutils literal"><span class="pre">[-random_h,</span> <span class="pre">random_h]</span></code> to the H channel in HSL color space.</li> |
| <li><strong>random_s</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – Add a random value in <code class="docutils literal"><span class="pre">[-random_s,</span> <span class="pre">random_s]</span></code> to the S channel in HSL color space.</li> |
| <li><strong>random_l</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – Add a random value in <code class="docutils literal"><span class="pre">[-random_l,</span> <span class="pre">random_l]</span></code> to the L channel in HSL color space.</li> |
| <li><strong>rotate</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='-1'</em>) – Rotate by an angle. If set, it overwrites the <code class="docutils literal"><span class="pre">max_rotate_angle</span></code> option.</li> |
| <li><strong>fill_value</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='255'</em>) – Set the padding pixels value to <code class="docutils literal"><span class="pre">fill_value</span></code>.</li> |
| <li><strong>inter_method</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='1'</em>) – The interpolation method: 0-NN 1-bilinear 2-cubic 3-area 4-lanczos4 9-auto 10-rand.</li> |
| <li><strong>pad</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – Change size from <code class="docutils literal"><span class="pre">[width,</span> <span class="pre">height]</span></code> into <code class="docutils literal"><span class="pre">[pad</span> <span class="pre">+</span> <span class="pre">width</span> <span class="pre">+</span> <span class="pre">pad,</span> <span class="pre">pad</span> <span class="pre">+</span> <span class="pre">height</span> <span class="pre">+</span> <span class="pre">pad]</span></code> by padding pixes</li> |
| <li><strong>seed_aug</strong> (<em>int</em><em> or </em><em>None</em><em>, </em><em>optional</em><em>, </em><em>default='None'</em>) – Random seed for augmentations.</li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first">The result iterator.</p> |
| </td> |
| </tr> |
| <tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body"><p class="first last"><a class="reference internal" href="#mxnet.io.MXDataIter" title="mxnet.io.MXDataIter">MXDataIter</a></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| <dl class="function"> |
| <dt id="mxnet.io.MNISTIter"> |
| <code class="descclassname">mxnet.io.</code><code class="descname">MNISTIter</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="headerlink" href="#mxnet.io.MNISTIter" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Iterating on the MNIST dataset.</p> |
| <p>One can download the dataset from <a class="reference external" href="http://yann.lecun.com/exdb/mnist/">http://yann.lecun.com/exdb/mnist/</a></p> |
| <p>Defined in src/io/iter_mnist.cc:L265</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>image</strong> (<em>string</em><em>, </em><em>optional</em><em>, </em><em>default='./train-images-idx3-ubyte'</em>) – Dataset Param: Mnist image path.</li> |
| <li><strong>label</strong> (<em>string</em><em>, </em><em>optional</em><em>, </em><em>default='./train-labels-idx1-ubyte'</em>) – Dataset Param: Mnist label path.</li> |
| <li><strong>batch_size</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='128'</em>) – Batch Param: Batch Size.</li> |
| <li><strong>shuffle</strong> (<em>boolean</em><em>, </em><em>optional</em><em>, </em><em>default=1</em>) – Augmentation Param: Whether to shuffle data.</li> |
| <li><strong>flat</strong> (<em>boolean</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Augmentation Param: Whether to flat the data into 1D.</li> |
| <li><strong>seed</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – Augmentation Param: Random Seed.</li> |
| <li><strong>silent</strong> (<em>boolean</em><em>, </em><em>optional</em><em>, </em><em>default=0</em>) – Auxiliary Param: Whether to print out data info.</li> |
| <li><strong>num_parts</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='1'</em>) – partition the data into multiple parts</li> |
| <li><strong>part_index</strong> (<em>int</em><em>, </em><em>optional</em><em>, </em><em>default='0'</em>) – the index of the part will read</li> |
| <li><strong>prefetch_buffer</strong> (<em>long</em><em> (</em><em>non-negative</em><em>)</em><em>, </em><em>optional</em><em>, </em><em>default=4</em>) – Maximum number of batches to prefetch.</li> |
| <li><strong>dtype</strong> (<em>{None</em><em>, </em><em>'float16'</em><em>, </em><em>'float32'</em><em>, </em><em>'float64'</em><em>, </em><em>'int32'</em><em>, </em><em>'int64'</em><em>, </em><em>'uint8'}</em><em>,</em><em>optional</em><em>, </em><em>default='None'</em>) – Output data type. <code class="docutils literal"><span class="pre">None</span></code> means no change.</li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first">The result iterator.</p> |
| </td> |
| </tr> |
| <tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body"><p class="first last"><a class="reference internal" href="#mxnet.io.MXDataIter" title="mxnet.io.MXDataIter">MXDataIter</a></p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| </div> |
| <div class="section" id="module-mxnet.io"> |
| <span id="mxnet-io-helper-classes-functions"></span><span id="mxnet-io-helper-classes-functions"></span><h3>mxnet.io - Helper Classes & Functions<a class="headerlink" href="#module-mxnet.io" title="Permalink to this headline">¶</a></h3> |
| <p>Data iterators for common data formats.</p> |
| <dl class="class"> |
| <dt id="mxnet.io.DataDesc"> |
| <em class="property">class </em><code class="descclassname">mxnet.io.</code><code class="descname">DataDesc</code><a class="reference internal" href="../../../_modules/mxnet/io.html#DataDesc"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.io.DataDesc" title="Permalink to this definition">¶</a></dt> |
| <dd><p>DataDesc is used to store name, shape, type and layout |
| information of the data or the label.</p> |
| <p>The <cite>layout</cite> describes how the axes in <cite>shape</cite> should be interpreted, |
| for example for image data setting <cite>layout=NCHW</cite> indicates |
| that the first axis is number of examples in the batch(N), |
| C is number of channels, H is the height and W is the width of the image.</p> |
| <p>For sequential data, by default <cite>layout</cite> is set to <code class="docutils literal"><span class="pre">NTC</span></code>, where |
| N is number of examples in the batch, T the temporal axis representing time |
| and C is the number of channels.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>cls</strong> (<a class="reference internal" href="#mxnet.io.DataDesc" title="mxnet.io.DataDesc"><em>DataDesc</em></a>) – The class.</li> |
| <li><strong>name</strong> (<em>str</em>) – Data name.</li> |
| <li><strong>shape</strong> (<em>tuple of int</em>) – Data shape.</li> |
| <li><strong>dtype</strong> (<em>np.dtype</em><em>, </em><em>optional</em>) – Data type.</li> |
| <li><strong>layout</strong> (<em>str</em><em>, </em><em>optional</em>) – Data layout.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <dl class="staticmethod"> |
| <dt id="mxnet.io.DataDesc.get_batch_axis"> |
| <em class="property">static </em><code class="descname">get_batch_axis</code><span class="sig-paren">(</span><em>layout</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/io.html#DataDesc.get_batch_axis"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.io.DataDesc.get_batch_axis" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Get the dimension that corresponds to the batch size.</p> |
| <p>When data parallelism is used, the data will be automatically split and |
| concatenated along the batch-size dimension. Axis can be -1, which means |
| the whole array will be copied for each data-parallelism device.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>layout</strong> (<em>str</em>) – layout string. For example, “NCHW”.</td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body">An axis indicating the batch_size dimension.</td> |
| </tr> |
| <tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body">int</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| <dl class="staticmethod"> |
| <dt id="mxnet.io.DataDesc.get_list"> |
| <em class="property">static </em><code class="descname">get_list</code><span class="sig-paren">(</span><em>shapes</em>, <em>types</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/io.html#DataDesc.get_list"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.io.DataDesc.get_list" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Get DataDesc list from attribute lists.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>shapes</strong> (a tuple of (<a class="reference external" href="mxnet.symbol.Symbol.name">name</a>, <a class="reference external" href="mxnet.ndarray.NDArray.shape">shape</a>)) – </li> |
| <li><strong>types</strong> (a tuple of (<a class="reference external" href="mxnet.symbol.Symbol.name">name</a>, np.dtype)) – </li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| </dd></dl> |
| <dl class="class"> |
| <dt id="mxnet.io.DataBatch"> |
| <em class="property">class </em><code class="descclassname">mxnet.io.</code><code class="descname">DataBatch</code><span class="sig-paren">(</span><em>data</em>, <em>label=None</em>, <em>pad=None</em>, <em>index=None</em>, <em>bucket_key=None</em>, <em>provide_data=None</em>, <em>provide_label=None</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/io.html#DataBatch"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.io.DataBatch" title="Permalink to this definition">¶</a></dt> |
| <dd><p>A data batch.</p> |
| <p>MXNet’s data iterator returns a batch of data for each <cite>next</cite> call. |
| This data contains <cite>batch_size</cite> number of examples.</p> |
| <p>If the input data consists of images, then shape of these images depend on |
| the <cite>layout</cite> attribute of <cite>DataDesc</cite> object in <cite>provide_data</cite> parameter.</p> |
| <p>If <cite>layout</cite> is set to ‘NCHW’ then, images should be stored in a 4-D matrix |
| of shape <code class="docutils literal"><span class="pre">(batch_size,</span> <span class="pre">num_channel,</span> <span class="pre">height,</span> <span class="pre">width)</span></code>. |
| If <cite>layout</cite> is set to ‘NHWC’ then, images should be stored in a 4-D matrix |
| of shape <code class="docutils literal"><span class="pre">(batch_size,</span> <span class="pre">height,</span> <span class="pre">width,</span> <span class="pre">num_channel)</span></code>. |
| The channels are often in RGB order.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>data</strong> (list of <cite>NDArray</cite>, each array containing <cite>batch_size</cite> examples.) – A list of input data.</li> |
| <li><strong>label</strong> (list of <cite>NDArray</cite>, each array often containing a 1-dimensional array. optional) – A list of input labels.</li> |
| <li><strong>pad</strong> (<em>int</em><em>, </em><em>optional</em>) – The number of examples padded at the end of a batch. It is used when the |
| total number of examples read is not divisible by the <cite>batch_size</cite>. |
| These extra padded examples are ignored in prediction.</li> |
| <li><strong>index</strong> (<em>numpy.array</em><em>, </em><em>optional</em>) – The example indices in this batch.</li> |
| <li><strong>bucket_key</strong> (<em>int</em><em>, </em><em>optional</em>) – The bucket key, used for bucketing module.</li> |
| <li><strong>provide_data</strong> (list of <cite>DataDesc</cite>, optional) – A list of <cite>DataDesc</cite> objects. <cite>DataDesc</cite> is used to store |
| name, shape, type and layout information of the data. |
| The <em>i</em>-th element describes the name and shape of <code class="docutils literal"><span class="pre">data[i]</span></code>.</li> |
| <li><strong>provide_label</strong> (list of <cite>DataDesc</cite>, optional) – A list of <cite>DataDesc</cite> objects. <cite>DataDesc</cite> is used to store |
| name, shape, type and layout information of the label. |
| The <em>i</em>-th element describes the name and shape of <code class="docutils literal"><span class="pre">label[i]</span></code>.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| <dl class="class"> |
| <dt id="mxnet.io.DataIter"> |
| <em class="property">class </em><code class="descclassname">mxnet.io.</code><code class="descname">DataIter</code><span class="sig-paren">(</span><em>batch_size=0</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/io.html#DataIter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.io.DataIter" title="Permalink to this definition">¶</a></dt> |
| <dd><p>The base class for an MXNet data iterator.</p> |
| <p>All I/O in MXNet is handled by specializations of this class. Data iterators |
| in MXNet are similar to standard-iterators in Python. On each call to <cite>next</cite> |
| they return a <cite>DataBatch</cite> which represents the next batch of data. When |
| there is no more data to return, it raises a <cite>StopIteration</cite> exception.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>batch_size</strong> (<em>int</em><em>, </em><em>optional</em>) – The batch size, namely the number of items in the batch.</td> |
| </tr> |
| </tbody> |
| </table> |
| <div class="admonition seealso"> |
| <p class="first admonition-title">See also</p> |
| <dl class="last docutils"> |
| <dt><a class="reference internal" href="#mxnet.io.NDArrayIter" title="mxnet.io.NDArrayIter"><code class="xref py py-class docutils literal"><span class="pre">NDArrayIter</span></code></a></dt> |
| <dd>Data-iterator for MXNet NDArray or numpy-ndarray objects.</dd> |
| <dt><a class="reference internal" href="#mxnet.io.CSVIter" title="mxnet.io.CSVIter"><code class="xref py py-class docutils literal"><span class="pre">CSVIter</span></code></a></dt> |
| <dd>Data-iterator for csv data.</dd> |
| <dt><a class="reference internal" href="#mxnet.io.LibSVMIter" title="mxnet.io.LibSVMIter"><code class="xref py py-class docutils literal"><span class="pre">LibSVMIter</span></code></a></dt> |
| <dd>Data-iterator for libsvm data.</dd> |
| <dt><code class="xref py py-class docutils literal"><span class="pre">ImageIter</span></code></dt> |
| <dd>Data-iterator for images.</dd> |
| </dl> |
| </div> |
| <dl class="method"> |
| <dt id="mxnet.io.DataIter.reset"> |
| <code class="descname">reset</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/io.html#DataIter.reset"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.io.DataIter.reset" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Reset the iterator to the begin of the data.</p> |
| </dd></dl> |
| <dl class="method"> |
| <dt id="mxnet.io.DataIter.next"> |
| <code class="descname">next</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/io.html#DataIter.next"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.io.DataIter.next" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Get next data batch from iterator.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">The data of next batch.</td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body"><a class="reference internal" href="#mxnet.io.DataBatch" title="mxnet.io.DataBatch">DataBatch</a></td> |
| </tr> |
| <tr class="field-odd field"><th class="field-name">Raises:</th><td class="field-body"><code class="xref py py-exc docutils literal"><span class="pre">StopIteration</span></code> – If the end of the data is reached.</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| <dl class="method"> |
| <dt id="mxnet.io.DataIter.iter_next"> |
| <code class="descname">iter_next</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/io.html#DataIter.iter_next"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.io.DataIter.iter_next" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Move to the next batch.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">Whether the move is successful.</td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body">boolean</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| <dl class="method"> |
| <dt id="mxnet.io.DataIter.getdata"> |
| <code class="descname">getdata</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/io.html#DataIter.getdata"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.io.DataIter.getdata" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Get data of current batch.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">The data of the current batch.</td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body">list of NDArray</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| <dl class="method"> |
| <dt id="mxnet.io.DataIter.getlabel"> |
| <code class="descname">getlabel</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/io.html#DataIter.getlabel"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.io.DataIter.getlabel" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Get label of the current batch.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">The label of the current batch.</td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body">list of NDArray</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| <dl class="method"> |
| <dt id="mxnet.io.DataIter.getindex"> |
| <code class="descname">getindex</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/io.html#DataIter.getindex"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.io.DataIter.getindex" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Get index of the current batch.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body"><strong>index</strong> – The indices of examples in the current batch.</td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body">numpy.array</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| <dl class="method"> |
| <dt id="mxnet.io.DataIter.getpad"> |
| <code class="descname">getpad</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/io.html#DataIter.getpad"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.io.DataIter.getpad" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Get the number of padding examples in the current batch.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body">Number of padding examples in the current batch.</td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body">int</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| </dd></dl> |
| <dl class="class"> |
| <dt id="mxnet.io.ResizeIter"> |
| <em class="property">class </em><code class="descclassname">mxnet.io.</code><code class="descname">ResizeIter</code><span class="sig-paren">(</span><em>data_iter</em>, <em>size</em>, <em>reset_internal=True</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/io.html#ResizeIter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.io.ResizeIter" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Resize a data iterator to a given number of batches.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>data_iter</strong> (<a class="reference internal" href="#mxnet.io.DataIter" title="mxnet.io.DataIter"><em>DataIter</em></a>) – The data iterator to be resized.</li> |
| <li><strong>size</strong> (<em>int</em>) – The number of batches per epoch to resize to.</li> |
| <li><strong>reset_internal</strong> (<em>bool</em>) – Whether to reset internal iterator on ResizeIter.reset.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p class="rubric">Examples</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">nd_iter</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">NDArrayIter</span><span class="p">(</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="mi">100</span><span class="p">,</span><span class="mi">10</span><span class="p">)),</span> <span class="n">batch_size</span><span class="o">=</span><span class="mi">25</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">resize_iter</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ResizeIter</span><span class="p">(</span><span class="n">nd_iter</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">resize_iter</span><span class="p">:</span> |
| <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">batch</span><span class="o">.</span><span class="n">data</span><span class="p">)</span> |
| <span class="go">[<NDArray 25x10 @cpu(0)>]</span> |
| <span class="go">[<NDArray 25x10 @cpu(0)>]</span> |
| </pre></div> |
| </div> |
| </dd></dl> |
| <dl class="class"> |
| <dt id="mxnet.io.PrefetchingIter"> |
| <em class="property">class </em><code class="descclassname">mxnet.io.</code><code class="descname">PrefetchingIter</code><span class="sig-paren">(</span><em>iters</em>, <em>rename_data=None</em>, <em>rename_label=None</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/io.html#PrefetchingIter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.io.PrefetchingIter" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Performs pre-fetch for other data iterators.</p> |
| <p>This iterator will create another thread to perform <code class="docutils literal"><span class="pre">iter_next</span></code> and then |
| store the data in memory. It potentially accelerates the data read, at the |
| cost of more memory usage.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>iters</strong> (<a class="reference internal" href="#mxnet.io.DataIter" title="mxnet.io.DataIter"><em>DataIter</em></a><em> or </em><em>list of DataIter</em>) – The data iterators to be pre-fetched.</li> |
| <li><strong>rename_data</strong> (<em>None</em><em> or </em><em>list of dict</em>) – The <em>i</em>-th element is a renaming map for the <em>i</em>-th iter, in the form of |
| {‘original_name’ : ‘new_name’}. Should have one entry for each entry |
| in iter[i].provide_data.</li> |
| <li><strong>rename_label</strong> (<em>None</em><em> or </em><em>list of dict</em>) – Similar to <code class="docutils literal"><span class="pre">rename_data</span></code>.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p class="rubric">Examples</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">iter1</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">NDArrayIter</span><span class="p">({</span><span class="s1">'data'</span><span class="p">:</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="mi">100</span><span class="p">,</span><span class="mi">10</span><span class="p">))},</span> <span class="n">batch_size</span><span class="o">=</span><span class="mi">25</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">iter2</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">NDArrayIter</span><span class="p">({</span><span class="s1">'data'</span><span class="p">:</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="mi">100</span><span class="p">,</span><span class="mi">10</span><span class="p">))},</span> <span class="n">batch_size</span><span class="o">=</span><span class="mi">25</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">piter</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">PrefetchingIter</span><span class="p">([</span><span class="n">iter1</span><span class="p">,</span> <span class="n">iter2</span><span class="p">],</span> |
| <span class="gp">... </span> <span class="n">rename_data</span><span class="o">=</span><span class="p">[{</span><span class="s1">'data'</span><span class="p">:</span> <span class="s1">'data_1'</span><span class="p">},</span> <span class="p">{</span><span class="s1">'data'</span><span class="p">:</span> <span class="s1">'data_2'</span><span class="p">}])</span> |
| <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">piter</span><span class="o">.</span><span class="n">provide_data</span><span class="p">)</span> |
| <span class="go">[DataDesc[data_1,(25, 10L),<type 'numpy.float32'>,NCHW],</span> |
| <span class="go"> DataDesc[data_2,(25, 10L),<type 'numpy.float32'>,NCHW]]</span> |
| </pre></div> |
| </div> |
| </dd></dl> |
| <dl class="class"> |
| <dt id="mxnet.io.MXDataIter"> |
| <em class="property">class </em><code class="descclassname">mxnet.io.</code><code class="descname">MXDataIter</code><span class="sig-paren">(</span><em>handle</em>, <em>data_name='data'</em>, <em>label_name='softmax_label'</em>, <em>**_</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/io.html#MXDataIter"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.io.MXDataIter" title="Permalink to this definition">¶</a></dt> |
| <dd><p>A python wrapper a C++ data iterator.</p> |
| <p>This iterator is the Python wrapper to all native C++ data iterators, such |
| as <cite>CSVIter</cite>, <cite>ImageRecordIter</cite>, <cite>MNISTIter</cite>, etc. When initializing |
| <cite>CSVIter</cite> for example, you will get an <cite>MXDataIter</cite> instance to use in your |
| Python code. Calls to <cite>next</cite>, <cite>reset</cite>, etc will be delegated to the |
| underlying C++ data iterators.</p> |
| <p>Usually you don’t need to interact with <cite>MXDataIter</cite> directly unless you are |
| implementing your own data iterators in C++. To do that, please refer to |
| examples under the <cite>src/io</cite> folder.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>handle</strong> (<em>DataIterHandle</em><em>, </em><em>required</em>) – The handle to the underlying C++ Data Iterator.</li> |
| <li><strong>data_name</strong> (<em>str</em><em>, </em><em>optional</em>) – Data name. Default to “data”.</li> |
| <li><strong>label_name</strong> (<em>str</em><em>, </em><em>optional</em>) – Label name. Default to “softmax_label”.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <div class="admonition seealso"> |
| <p class="first admonition-title">See also</p> |
| <p class="last">src/io : The underlying C++ data iterator implementation, e.g., <cite>CSVIter</cite>.</p> |
| </div> |
| </dd></dl> |
| </div> |
| <div class="section" id="module-mxnet.recordio"> |
| <span id="mxnet-recordio"></span><span id="mxnet-recordio"></span><h3>mxnet.recordio<a class="headerlink" href="#module-mxnet.recordio" title="Permalink to this headline">¶</a></h3> |
| <p>Read and write for the RecordIO data format.</p> |
| <dl class="class"> |
| <dt id="mxnet.recordio.MXRecordIO"> |
| <em class="property">class </em><code class="descclassname">mxnet.recordio.</code><code class="descname">MXRecordIO</code><span class="sig-paren">(</span><em>uri</em>, <em>flag</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXRecordIO"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.recordio.MXRecordIO" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Reads/writes <cite>RecordIO</cite> data format, supporting sequential read and write.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">record</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">recordio</span><span class="o">.</span><span class="n">MXRecordIO</span><span class="p">(</span><span class="s1">'tmp.rec'</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> |
| <span class="go"><mxnet.recordio.MXRecordIO object at 0x10ef40ed0></span> |
| <span class="gp">>>> </span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span> |
| <span class="gp">... </span> <span class="n">record</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'record_</span><span class="si">%d</span><span class="s1">'</span><span class="o">%</span><span class="n">i</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">record</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> |
| <span class="gp">>>> </span><span class="n">record</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">recordio</span><span class="o">.</span><span class="n">MXRecordIO</span><span class="p">(</span><span class="s1">'tmp.rec'</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span> |
| <span class="gp">... </span> <span class="n">item</span> <span class="o">=</span> <span class="n">record</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> |
| <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">item</span><span class="p">)</span> |
| <span class="go">record_0</span> |
| <span class="go">record_1</span> |
| <span class="go">record_2</span> |
| <span class="go">record_3</span> |
| <span class="go">record_4</span> |
| <span class="gp">>>> </span><span class="n">record</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> |
| </pre></div> |
| </div> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>uri</strong> (<em>string</em>) – Path to the record file.</li> |
| <li><strong>flag</strong> (<em>string</em>) – ‘w’ for write or ‘r’ for read.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <dl class="method"> |
| <dt id="mxnet.recordio.MXRecordIO.open"> |
| <code class="descname">open</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXRecordIO.open"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.recordio.MXRecordIO.open" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Opens the record file.</p> |
| </dd></dl> |
| <dl class="method"> |
| <dt id="mxnet.recordio.MXRecordIO.close"> |
| <code class="descname">close</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXRecordIO.close"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.recordio.MXRecordIO.close" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Closes the record file.</p> |
| </dd></dl> |
| <dl class="method"> |
| <dt id="mxnet.recordio.MXRecordIO.reset"> |
| <code class="descname">reset</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXRecordIO.reset"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.recordio.MXRecordIO.reset" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Resets the pointer to first item.</p> |
| <p>If the record is opened with ‘w’, this function will truncate the file to empty.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">record</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">recordio</span><span class="o">.</span><span class="n">MXRecordIO</span><span class="p">(</span><span class="s1">'tmp.rec'</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">2</span><span class="p">):</span> |
| <span class="gp">... </span> <span class="n">item</span> <span class="o">=</span> <span class="n">record</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> |
| <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">item</span><span class="p">)</span> |
| <span class="go">record_0</span> |
| <span class="go">record_1</span> |
| <span class="gp">>>> </span><span class="n">record</span><span class="o">.</span><span class="n">reset</span><span class="p">()</span> <span class="c1"># Pointer is reset.</span> |
| <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">record</span><span class="o">.</span><span class="n">read</span><span class="p">())</span> <span class="c1"># Started reading from start again.</span> |
| <span class="go">record_0</span> |
| <span class="gp">>>> </span><span class="n">record</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> |
| </pre></div> |
| </div> |
| </dd></dl> |
| <dl class="method"> |
| <dt id="mxnet.recordio.MXRecordIO.write"> |
| <code class="descname">write</code><span class="sig-paren">(</span><em>buf</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXRecordIO.write"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.recordio.MXRecordIO.write" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Inserts a string buffer as a record.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">record</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">recordio</span><span class="o">.</span><span class="n">MXRecordIO</span><span class="p">(</span><span class="s1">'tmp.rec'</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span> |
| <span class="gp">... </span> <span class="n">record</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'record_</span><span class="si">%d</span><span class="s1">'</span><span class="o">%</span><span class="n">i</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">record</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> |
| </pre></div> |
| </div> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>buf</strong> (<em>string</em><em> (</em><em>python2</em><em>)</em><em>, </em><em>bytes</em><em> (</em><em>python3</em><em>)</em>) – Buffer to write.</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| <dl class="method"> |
| <dt id="mxnet.recordio.MXRecordIO.read"> |
| <code class="descname">read</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXRecordIO.read"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.recordio.MXRecordIO.read" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Returns record as a string.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">record</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">recordio</span><span class="o">.</span><span class="n">MXRecordIO</span><span class="p">(</span><span class="s1">'tmp.rec'</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span> |
| <span class="gp">... </span> <span class="n">item</span> <span class="o">=</span> <span class="n">record</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> |
| <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">item</span><span class="p">)</span> |
| <span class="go">record_0</span> |
| <span class="go">record_1</span> |
| <span class="go">record_2</span> |
| <span class="go">record_3</span> |
| <span class="go">record_4</span> |
| <span class="gp">>>> </span><span class="n">record</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> |
| </pre></div> |
| </div> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Returns:</th><td class="field-body"><strong>buf</strong> – Buffer read.</td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Return type:</th><td class="field-body">string</td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| </dd></dl> |
| <dl class="class"> |
| <dt id="mxnet.recordio.MXIndexedRecordIO"> |
| <em class="property">class </em><code class="descclassname">mxnet.recordio.</code><code class="descname">MXIndexedRecordIO</code><span class="sig-paren">(</span><em>idx_path</em>, <em>uri</em>, <em>flag</em>, <em>key_type=<type 'int'></em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXIndexedRecordIO"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.recordio.MXIndexedRecordIO" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Reads/writes <cite>RecordIO</cite> data format, supporting random access.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span> |
| <span class="gp">... </span> <span class="n">record</span><span class="o">.</span><span class="n">write_idx</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="s1">'record_</span><span class="si">%d</span><span class="s1">'</span><span class="o">%</span><span class="n">i</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">record</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> |
| <span class="gp">>>> </span><span class="n">record</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">recordio</span><span class="o">.</span><span class="n">MXIndexedRecordIO</span><span class="p">(</span><span class="s1">'tmp.idx'</span><span class="p">,</span> <span class="s1">'tmp.rec'</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">record</span><span class="o">.</span><span class="n">read_idx</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> |
| <span class="go">record_3</span> |
| </pre></div> |
| </div> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>idx_path</strong> (<em>str</em>) – Path to the index file.</li> |
| <li><strong>uri</strong> (<em>str</em>) – Path to the record file. Only supports seekable file types.</li> |
| <li><strong>flag</strong> (<em>str</em>) – ‘w’ for write or ‘r’ for read.</li> |
| <li><strong>key_type</strong> (<a class="reference internal" href="../kvstore/kvstore.html#mxnet.kvstore.KVStore.type" title="mxnet.kvstore.KVStore.type"><em>type</em></a>) – Data type for keys.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <dl class="method"> |
| <dt id="mxnet.recordio.MXIndexedRecordIO.close"> |
| <code class="descname">close</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXIndexedRecordIO.close"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.recordio.MXIndexedRecordIO.close" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Closes the record file.</p> |
| </dd></dl> |
| <dl class="method"> |
| <dt id="mxnet.recordio.MXIndexedRecordIO.seek"> |
| <code class="descname">seek</code><span class="sig-paren">(</span><em>idx</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXIndexedRecordIO.seek"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.recordio.MXIndexedRecordIO.seek" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Sets the current read pointer position.</p> |
| <p>This function is internally called by <cite>read_idx(idx)</cite> to find the current |
| reader pointer position. It doesn’t return anything.</p> |
| </dd></dl> |
| <dl class="method"> |
| <dt id="mxnet.recordio.MXIndexedRecordIO.tell"> |
| <code class="descname">tell</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXIndexedRecordIO.tell"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.recordio.MXIndexedRecordIO.tell" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Returns the current position of write head.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">record</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">recordio</span><span class="o">.</span><span class="n">MXIndexedRecordIO</span><span class="p">(</span><span class="s1">'tmp.idx'</span><span class="p">,</span> <span class="s1">'tmp.rec'</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">record</span><span class="o">.</span><span class="n">tell</span><span class="p">())</span> |
| <span class="go">0</span> |
| <span class="gp">>>> </span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span> |
| <span class="gp">... </span> <span class="n">record</span><span class="o">.</span><span class="n">write_idx</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="s1">'record_</span><span class="si">%d</span><span class="s1">'</span><span class="o">%</span><span class="n">i</span><span class="p">)</span> |
| <span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="n">record</span><span class="o">.</span><span class="n">tell</span><span class="p">())</span> |
| <span class="go">16</span> |
| <span class="go">32</span> |
| <span class="go">48</span> |
| <span class="go">64</span> |
| <span class="go">80</span> |
| </pre></div> |
| </div> |
| </dd></dl> |
| <dl class="method"> |
| <dt id="mxnet.recordio.MXIndexedRecordIO.read_idx"> |
| <code class="descname">read_idx</code><span class="sig-paren">(</span><em>idx</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXIndexedRecordIO.read_idx"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.recordio.MXIndexedRecordIO.read_idx" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Returns the record at given index.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">record</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">recordio</span><span class="o">.</span><span class="n">MXIndexedRecordIO</span><span class="p">(</span><span class="s1">'tmp.idx'</span><span class="p">,</span> <span class="s1">'tmp.rec'</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span> |
| <span class="gp">... </span> <span class="n">record</span><span class="o">.</span><span class="n">write_idx</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="s1">'record_</span><span class="si">%d</span><span class="s1">'</span><span class="o">%</span><span class="n">i</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">record</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> |
| <span class="gp">>>> </span><span class="n">record</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">recordio</span><span class="o">.</span><span class="n">MXIndexedRecordIO</span><span class="p">(</span><span class="s1">'tmp.idx'</span><span class="p">,</span> <span class="s1">'tmp.rec'</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">record</span><span class="o">.</span><span class="n">read_idx</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> |
| <span class="go">record_3</span> |
| </pre></div> |
| </div> |
| </dd></dl> |
| <dl class="method"> |
| <dt id="mxnet.recordio.MXIndexedRecordIO.write_idx"> |
| <code class="descname">write_idx</code><span class="sig-paren">(</span><em>idx</em>, <em>buf</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/recordio.html#MXIndexedRecordIO.write_idx"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.recordio.MXIndexedRecordIO.write_idx" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Inserts input record at given index.</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span> |
| <span class="gp">... </span> <span class="n">record</span><span class="o">.</span><span class="n">write_idx</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="s1">'record_</span><span class="si">%d</span><span class="s1">'</span><span class="o">%</span><span class="n">i</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">record</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> |
| </pre></div> |
| </div> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>idx</strong> (<em>int</em>) – Index of a file.</li> |
| <li><strong>buf</strong> – Record to write.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| </dd></dl> |
| </dd></dl> |
| <dl class="attribute"> |
| <dt id="mxnet.recordio.IRHeader"> |
| <code class="descclassname">mxnet.recordio.</code><code class="descname">IRHeader</code><a class="headerlink" href="#mxnet.recordio.IRHeader" title="Permalink to this definition">¶</a></dt> |
| <dd><p>An alias for HEADER. Used to store metadata (e.g. labels) accompanying a record. |
| See mxnet.recordio.pack and mxnet.recordio.pack_img for example uses.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple"> |
| <li><strong>flag</strong> (<em>int</em>) – Available for convenience, can be set arbitrarily.</li> |
| <li><strong>label</strong> (<em>float</em><em> or </em><em>an array of float</em>) – Typically used to store label(s) for a record.</li> |
| <li><strong>id</strong> (<em>int</em>) – Usually a unique id representing record.</li> |
| <li><strong>id2</strong> (<em>int</em>) – Higher order bits of the unique id, should be set to 0 (in most cases).</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p>alias of <code class="xref py py-class docutils literal"><span class="pre">HEADER</span></code></p> |
| </dd></dl> |
| <dl class="function"> |
| <dt id="mxnet.recordio.pack"> |
| <code class="descclassname">mxnet.recordio.</code><code class="descname">pack</code><span class="sig-paren">(</span><em>header</em>, <em>s</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/recordio.html#pack"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.recordio.pack" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Pack a string into MXImageRecord.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>header</strong> (<a class="reference internal" href="#mxnet.recordio.IRHeader" title="mxnet.recordio.IRHeader"><em>IRHeader</em></a>) – Header of the image record. |
| <code class="docutils literal"><span class="pre">header.label</span></code> can be a number or an array. See more detail in <code class="docutils literal"><span class="pre">IRHeader</span></code>.</li> |
| <li><strong>s</strong> (<em>str</em>) – Raw image string to be packed.</li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first"><strong>s</strong> – The packed string.</p> |
| </td> |
| </tr> |
| <tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body"><p class="first last">str</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p class="rubric">Examples</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">label</span> <span class="o">=</span> <span class="mi">4</span> <span class="c1"># label can also be a 1-D array, for example: label = [1,2,3]</span> |
| <span class="gp">>>> </span><span class="nb">id</span> <span class="o">=</span> <span class="mi">2574</span> |
| <span class="gp">>>> </span><span class="n">header</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">recordio</span><span class="o">.</span><span class="n">IRHeader</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">label</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span> <span class="k">as</span> <span class="n">file</span><span class="p">:</span> |
| <span class="gp">... </span> <span class="n">s</span> <span class="o">=</span> <span class="n">file</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> |
| <span class="gp">>>> </span><span class="n">packed_s</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">recordio</span><span class="o">.</span><span class="n">pack</span><span class="p">(</span><span class="n">header</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| </dd></dl> |
| <dl class="function"> |
| <dt id="mxnet.recordio.unpack"> |
| <code class="descclassname">mxnet.recordio.</code><code class="descname">unpack</code><span class="sig-paren">(</span><em>s</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/recordio.html#unpack"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.recordio.unpack" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Unpack a MXImageRecord to string.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>s</strong> (<em>str</em>) – String buffer from <code class="docutils literal"><span class="pre">MXRecordIO.read</span></code>.</td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><ul class="simple"> |
| <li><strong>header</strong> (<em>IRHeader</em>) – Header of the image record.</li> |
| <li><strong>s</strong> (<em>str</em>) – Unpacked string.</li> |
| </ul> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p class="rubric">Examples</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">record</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">recordio</span><span class="o">.</span><span class="n">MXRecordIO</span><span class="p">(</span><span class="s1">'test.rec'</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">item</span> <span class="o">=</span> <span class="n">record</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> |
| <span class="gp">>>> </span><span class="n">header</span><span class="p">,</span> <span class="n">s</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">recordio</span><span class="o">.</span><span class="n">unpack</span><span class="p">(</span><span class="n">item</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">header</span> |
| <span class="go">HEADER(flag=0, label=14.0, id=20129312, id2=0)</span> |
| </pre></div> |
| </div> |
| </dd></dl> |
| <dl class="function"> |
| <dt id="mxnet.recordio.unpack_img"> |
| <code class="descclassname">mxnet.recordio.</code><code class="descname">unpack_img</code><span class="sig-paren">(</span><em>s</em>, <em>iscolor=-1</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/recordio.html#unpack_img"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.recordio.unpack_img" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Unpack a MXImageRecord to image.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>s</strong> (<em>str</em>) – String buffer from <code class="docutils literal"><span class="pre">MXRecordIO.read</span></code>.</li> |
| <li><strong>iscolor</strong> (<em>int</em>) – Image format option for <code class="docutils literal"><span class="pre">cv2.imdecode</span></code>.</li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last"><ul class="simple"> |
| <li><strong>header</strong> (<em>IRHeader</em>) – Header of the image record.</li> |
| <li><strong>img</strong> (<em>numpy.ndarray</em>) – Unpacked image.</li> |
| </ul> |
| </p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p class="rubric">Examples</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">record</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">recordio</span><span class="o">.</span><span class="n">MXRecordIO</span><span class="p">(</span><span class="s1">'test.rec'</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">item</span> <span class="o">=</span> <span class="n">record</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> |
| <span class="gp">>>> </span><span class="n">header</span><span class="p">,</span> <span class="n">img</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">recordio</span><span class="o">.</span><span class="n">unpack_img</span><span class="p">(</span><span class="n">item</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">header</span> |
| <span class="go">HEADER(flag=0, label=14.0, id=20129312, id2=0)</span> |
| <span class="gp">>>> </span><span class="n">img</span> |
| <span class="go">array([[[ 23, 27, 45],</span> |
| <span class="go"> [ 28, 32, 50],</span> |
| <span class="go"> ...,</span> |
| <span class="go"> [ 36, 40, 59],</span> |
| <span class="go"> [ 35, 39, 58]],</span> |
| <span class="go"> ...,</span> |
| <span class="go"> [[ 91, 92, 113],</span> |
| <span class="go"> [ 97, 98, 119],</span> |
| <span class="go"> ...,</span> |
| <span class="go"> [168, 169, 167],</span> |
| <span class="go"> [166, 167, 165]]], dtype=uint8)</span> |
| </pre></div> |
| </div> |
| </dd></dl> |
| <dl class="function"> |
| <dt id="mxnet.recordio.pack_img"> |
| <code class="descclassname">mxnet.recordio.</code><code class="descname">pack_img</code><span class="sig-paren">(</span><em>header</em>, <em>img</em>, <em>quality=95</em>, <em>img_fmt='.jpg'</em><span class="sig-paren">)</span><a class="reference internal" href="../../../_modules/mxnet/recordio.html#pack_img"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#mxnet.recordio.pack_img" title="Permalink to this definition">¶</a></dt> |
| <dd><p>Pack an image into <code class="docutils literal"><span class="pre">MXImageRecord</span></code>.</p> |
| <table class="docutils field-list" frame="void" rules="none"> |
| <col class="field-name"/> |
| <col class="field-body"/> |
| <tbody valign="top"> |
| <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple"> |
| <li><strong>header</strong> (<a class="reference internal" href="#mxnet.recordio.IRHeader" title="mxnet.recordio.IRHeader"><em>IRHeader</em></a>) – Header of the image record. |
| <code class="docutils literal"><span class="pre">header.label</span></code> can be a number or an array. See more detail in <code class="docutils literal"><span class="pre">IRHeader</span></code>.</li> |
| <li><strong>img</strong> (<em>numpy.ndarray</em>) – Image to be packed.</li> |
| <li><strong>quality</strong> (<em>int</em>) – Quality for JPEG encoding in range 1-100, or compression for PNG encoding in range 1-9.</li> |
| <li><strong>img_fmt</strong> (<em>str</em>) – Encoding of the image (.jpg for JPEG, .png for PNG).</li> |
| </ul> |
| </td> |
| </tr> |
| <tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first"><strong>s</strong> – The packed string.</p> |
| </td> |
| </tr> |
| <tr class="field-odd field"><th class="field-name">Return type:</th><td class="field-body"><p class="first last">str</p> |
| </td> |
| </tr> |
| </tbody> |
| </table> |
| <p class="rubric">Examples</p> |
| <div class="highlight-default"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">label</span> <span class="o">=</span> <span class="mi">4</span> <span class="c1"># label can also be a 1-D array, for example: label = [1,2,3]</span> |
| <span class="gp">>>> </span><span class="nb">id</span> <span class="o">=</span> <span class="mi">2574</span> |
| <span class="gp">>>> </span><span class="n">header</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">recordio</span><span class="o">.</span><span class="n">IRHeader</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">label</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">img</span> <span class="o">=</span> <span class="n">cv2</span><span class="o">.</span><span class="n">imread</span><span class="p">(</span><span class="s1">'test.jpg'</span><span class="p">)</span> |
| <span class="gp">>>> </span><span class="n">packed_s</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">recordio</span><span class="o">.</span><span class="n">pack_img</span><span class="p">(</span><span class="n">header</span><span class="p">,</span> <span class="n">img</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| </dd></dl> |
| <script>auto_index("api-reference");</script></div> |
| </div> |
| </div> |
| </div> |
| </div> |
| <div aria-label="main navigation" class="sphinxsidebar rightsidebar" role="navigation"> |
| <div class="sphinxsidebarwrapper"> |
| <h3><a href="../../../index.html">Table Of Contents</a></h3> |
| <ul> |
| <li><a class="reference internal" href="#">Data Loading API</a><ul> |
| <li><a class="reference internal" href="#overview">Overview</a></li> |
| <li><a class="reference internal" href="#data-iterators">Data iterators</a></li> |
| <li><a class="reference internal" href="#helper-classes-and-functions">Helper classes and functions</a><ul> |
| <li><a class="reference internal" href="#data-structures-and-other-iterators">Data structures and other iterators</a></li> |
| <li><a class="reference internal" href="#functions-to-read-and-write-recordio-files">Functions to read and write RecordIO files</a></li> |
| </ul> |
| </li> |
| <li><a class="reference internal" href="#how-to-develop-a-new-iterator">How to develop a new iterator</a><ul> |
| <li><a class="reference internal" href="#how-to-change-the-batch-layout">How to change the batch layout</a></li> |
| </ul> |
| </li> |
| <li><a class="reference internal" href="#api-reference">API Reference</a><ul> |
| <li><a class="reference internal" href="#module-mxnet.io">mxnet.io - Data Iterators</a></li> |
| <li><a class="reference internal" href="#module-mxnet.io">mxnet.io - Helper Classes & Functions</a></li> |
| <li><a class="reference internal" href="#module-mxnet.recordio">mxnet.recordio</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </div><div class="footer"> |
| <div class="section-disclaimer"> |
| <div class="container"> |
| <div> |
| <img height="60" src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/apache_incubator_logo.png"/> |
| <p> |
| Apache MXNet is an effort undergoing incubation at The Apache Software Foundation (ASF), <strong>sponsored by the <i>Apache Incubator</i></strong>. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. |
| </p> |
| <p> |
| "Copyright © 2017-2018, The Apache Software Foundation |
| Apache MXNet, MXNet, Apache, the Apache feather, and the Apache MXNet project logo are either registered trademarks or trademarks of the Apache Software Foundation." |
| </p> |
| </div> |
| </div> |
| </div> |
| </div> <!-- pagename != index --> |
| </div> |
| <script crossorigin="anonymous" integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS" src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"></script> |
| <script src="../../../_static/js/sidebar.js" type="text/javascript"></script> |
| <script src="../../../_static/js/search.js" type="text/javascript"></script> |
| <script src="../../../_static/js/navbar.js" type="text/javascript"></script> |
| <script src="../../../_static/js/clipboard.min.js" type="text/javascript"></script> |
| <script src="../../../_static/js/copycode.js" type="text/javascript"></script> |
| <script src="../../../_static/js/page.js" type="text/javascript"></script> |
| <script src="../../../_static/js/docversion.js" type="text/javascript"></script> |
| <script type="text/javascript"> |
| $('body').ready(function () { |
| $('body').css('visibility', 'visible'); |
| }); |
| </script> |
| </body> |
| </html> |
