Google Git
Sign in
apache / mxnet-site / b29f64d55499f1854ba5174ff443cf921bb86ea8 / . / versions / 1.1.0 / _modules / mxnet / io.html
blob: a4bb04a4764c802a421e364b2729efcf1acbb815 [file] [log] [blame]
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<meta content="IE=edge" http-equiv="X-UA-Compatible"/>
<meta content="width=device-width, initial-scale=1" name="viewport"/>
<meta content="mxnet.io" property="og:title">
<meta content="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/og-logo.png" property="og:image">
<meta content="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/og-logo.png" property="og:image:secure_url">
<meta content="mxnet.io" property="og:description"/>
<title>mxnet.io — mxnet documentation</title>
<link crossorigin="anonymous" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css" integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7" rel="stylesheet"/>
<link href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css" rel="stylesheet"/>
<link href="../../_static/basic.css" rel="stylesheet" type="text/css">
<link href="../../_static/pygments.css" rel="stylesheet" type="text/css">
<link href="../../_static/mxnet.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../../',
VERSION: '',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: '.txt'
};
</script>
<script src="https://code.jquery.com/jquery-1.11.1.min.js" type="text/javascript"></script>
<script src="../../_static/underscore.js" type="text/javascript"></script>
<script src="../../_static/searchtools_custom.js" type="text/javascript"></script>
<script src="../../_static/doctools.js" type="text/javascript"></script>
<script src="../../_static/selectlang.js" type="text/javascript"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script>
<script type="text/javascript"> jQuery(function() { Search.loadIndex("/versions/1.1.0/searchindex.js"); Search.init();}); </script>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new
Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-96378503-1', 'auto');
ga('send', 'pageview');
</script>
<!-- -->
<!-- <script type="text/javascript" src="../../_static/jquery.js"></script> -->
<!-- -->
<!-- <script type="text/javascript" src="../../_static/underscore.js"></script> -->
<!-- -->
<!-- <script type="text/javascript" src="../../_static/doctools.js"></script> -->
<!-- -->
<!-- <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script> -->
<!-- -->
<link href="../../genindex.html" rel="index" title="Index">
<link href="../../search.html" rel="search" title="Search"/>
<link href="../index.html" rel="up" title="Module code"/>
<link href="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet-icon.png" rel="icon" type="image/png"/>
</link></link></link></meta></meta></meta></head>
<body background="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet-background-compressed.jpeg" role="document">
<div class="content-block"><div class="navbar navbar-fixed-top">
<div class="container" id="navContainer">
<div class="innder" id="header-inner">
<h1 id="logo-wrap">
<a href="../../" id="logo"><img src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet_logo.png"/></a>
</h1>
<nav class="nav-bar" id="main-nav">
<a class="main-nav-link" href="/versions/1.1.0/install/index.html">Install</a>
<span id="dropdown-menu-position-anchor">
<a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Gluon <span class="caret"></span></a>
<ul class="dropdown-menu navbar-menu" id="package-dropdown-menu">
<li><a class="main-nav-link" href="/versions/1.1.0/tutorials/gluon/gluon.html">About</a></li>
<li><a class="main-nav-link" href="https://www.d2l.ai/">Dive into Deep Learning</a></li>
<li><a class="main-nav-link" href="https://gluon-cv.mxnet.io">GluonCV Toolkit</a></li>
<li><a class="main-nav-link" href="https://gluon-nlp.mxnet.io/">GluonNLP Toolkit</a></li>
</ul>
</span>
<span id="dropdown-menu-position-anchor">
<a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">API <span class="caret"></span></a>
<ul class="dropdown-menu navbar-menu" id="package-dropdown-menu">
<li><a class="main-nav-link" href="/versions/1.1.0/api/python/index.html">Python</a></li>
<li><a class="main-nav-link" href="/versions/1.1.0/api/c++/index.html">C++</a></li>
<li><a class="main-nav-link" href="/versions/1.1.0/api/julia/index.html">Julia</a></li>
<li><a class="main-nav-link" href="/versions/1.1.0/api/perl/index.html">Perl</a></li>
<li><a class="main-nav-link" href="/versions/1.1.0/api/r/index.html">R</a></li>
<li><a class="main-nav-link" href="/versions/1.1.0/api/scala/index.html">Scala</a></li>
</ul>
</span>
<span id="dropdown-menu-position-anchor-docs">
<a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Docs <span class="caret"></span></a>
<ul class="dropdown-menu navbar-menu" id="package-dropdown-menu-docs">
<li><a class="main-nav-link" href="/versions/1.1.0/faq/index.html">FAQ</a></li>
<li><a class="main-nav-link" href="/versions/1.1.0/tutorials/index.html">Tutorials</a>
<li><a class="main-nav-link" href="https://github.com/apache/incubator-mxnet/tree/v1.1.0/example">Examples</a></li>
<li><a class="main-nav-link" href="/versions/1.1.0/architecture/index.html">Architecture</a></li>
<li><a class="main-nav-link" href="https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+Home">Developer Wiki</a></li>
<li><a class="main-nav-link" href="/versions/1.1.0/model_zoo/index.html">Model Zoo</a></li>
<li><a class="main-nav-link" href="https://github.com/onnx/onnx-mxnet">ONNX</a></li>
</li></ul>
</span>
<span id="dropdown-menu-position-anchor-community">
<a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Community <span class="caret"></span></a>
<ul class="dropdown-menu navbar-menu" id="package-dropdown-menu-community">
<li><a class="main-nav-link" href="http://discuss.mxnet.io">Forum</a></li>
<li><a class="main-nav-link" href="https://github.com/apache/incubator-mxnet/tree/v1.1.0">Github</a></li>
<li><a class="main-nav-link" href="/versions/1.1.0/community/contribute.html">Contribute</a></li>
<li><a class="main-nav-link" href="/versions/1.1.0/community/powered_by.html">Powered By</a></li>
</ul>
</span>
<span id="dropdown-menu-position-anchor-version" style="position: relative"><a href="#" class="main-nav-link dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="true">1.1.0<span class="caret"></span></a><ul id="package-dropdown-menu" class="dropdown-menu"><li><a href="/">master</a></li><li><a href="/versions/1.7.0/">1.7.0</a></li><li><a href=/versions/1.6.0/>1.6.0</a></li><li><a href=/versions/1.5.0/>1.5.0</a></li><li><a href=/versions/1.4.1/>1.4.1</a></li><li><a href=/versions/1.3.1/>1.3.1</a></li><li><a href=/versions/1.2.1/>1.2.1</a></li><li><a href=/versions/1.1.0/>1.1.0</a></li><li><a href=/versions/1.0.0/>1.0.0</a></li><li><a href=/versions/0.12.1/>0.12.1</a></li><li><a href=/versions/0.11.0/>0.11.0</a></li></ul></span></nav>
<script> function getRootPath(){ return "../../" } </script>
<div class="burgerIcon dropdown">
<a class="dropdown-toggle" data-toggle="dropdown" href="#" role="button">☰</a>
<ul class="dropdown-menu" id="burgerMenu">
<li><a href="/versions/1.1.0/install/index.html">Install</a></li>
<li><a class="main-nav-link" href="/versions/1.1.0/tutorials/index.html">Tutorials</a></li>
<li class="dropdown-submenu dropdown">
<a aria-expanded="true" aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" tabindex="-1">Gluon</a>
<ul class="dropdown-menu navbar-menu" id="package-dropdown-menu">
<li><a class="main-nav-link" href="/versions/1.1.0/tutorials/gluon/gluon.html">About</a></li>
<li><a class="main-nav-link" href="http://gluon.mxnet.io">The Straight Dope (Tutorials)</a></li>
<li><a class="main-nav-link" href="https://gluon-cv.mxnet.io">GluonCV Toolkit</a></li>
<li><a class="main-nav-link" href="https://gluon-nlp.mxnet.io/">GluonNLP Toolkit</a></li>
</ul>
</li>
<li class="dropdown-submenu">
<a aria-expanded="true" aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" tabindex="-1">API</a>
<ul class="dropdown-menu">
<li><a class="main-nav-link" href="/versions/1.1.0/api/python/index.html">Python</a></li>
<li><a class="main-nav-link" href="/versions/1.1.0/api/c++/index.html">C++</a></li>
<li><a class="main-nav-link" href="/versions/1.1.0/api/julia/index.html">Julia</a></li>
<li><a class="main-nav-link" href="/versions/1.1.0/api/perl/index.html">Perl</a></li>
<li><a class="main-nav-link" href="/versions/1.1.0/api/r/index.html">R</a></li>
<li><a class="main-nav-link" href="/versions/1.1.0/api/scala/index.html">Scala</a></li>
</ul>
</li>
<li class="dropdown-submenu">
<a aria-expanded="true" aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" tabindex="-1">Docs</a>
<ul class="dropdown-menu">
<li><a href="/versions/1.1.0/faq/index.html" tabindex="-1">FAQ</a></li>
<li><a href="/versions/1.1.0/tutorials/index.html" tabindex="-1">Tutorials</a></li>
<li><a href="https://github.com/apache/incubator-mxnet/tree/v1.1.0/example" tabindex="-1">Examples</a></li>
<li><a href="/versions/1.1.0/architecture/index.html" tabindex="-1">Architecture</a></li>
<li><a href="https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+Home" tabindex="-1">Developer Wiki</a></li>
<li><a href="/versions/1.1.0/model_zoo/index.html" tabindex="-1">Gluon Model Zoo</a></li>
<li><a href="https://github.com/onnx/onnx-mxnet" tabindex="-1">ONNX</a></li>
</ul>
</li>
<li class="dropdown-submenu dropdown">
<a aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" role="button" tabindex="-1">Community</a>
<ul class="dropdown-menu">
<li><a href="http://discuss.mxnet.io" tabindex="-1">Forum</a></li>
<li><a href="https://github.com/apache/incubator-mxnet/tree/v1.1.0" tabindex="-1">Github</a></li>
<li><a href="/versions/1.1.0/community/contribute.html" tabindex="-1">Contribute</a></li>
<li><a href="/versions/1.1.0/community/powered_by.html" tabindex="-1">Powered By</a></li>
</ul>
</li>
<li id="dropdown-menu-position-anchor-version-mobile" class="dropdown-submenu" style="position: relative"><a href="#" tabindex="-1">1.1.0</a><ul class="dropdown-menu"><li><a tabindex="-1" href=/>master</a></li><li><a tabindex="-1" href=/versions/1.6.0/>1.6.0</a></li><li><a tabindex="-1" href=/versions/1.5.0/>1.5.0</a></li><li><a tabindex="-1" href=/versions/1.4.1/>1.4.1</a></li><li><a tabindex="-1" href=/versions/1.3.1/>1.3.1</a></li><li><a tabindex="-1" href=/versions/1.2.1/>1.2.1</a></li><li><a tabindex="-1" href=/versions/1.1.0/>1.1.0</a></li><li><a tabindex="-1" href=/versions/1.0.0/>1.0.0</a></li><li><a tabindex="-1" href=/versions/0.12.1/>0.12.1</a></li><li><a tabindex="-1" href=/versions/0.11.0/>0.11.0</a></li></ul></li></ul>
</div>
<div class="plusIcon dropdown">
<a class="dropdown-toggle" data-toggle="dropdown" href="#" role="button"><span aria-hidden="true" class="glyphicon glyphicon-plus"></span></a>
<ul class="dropdown-menu dropdown-menu-right" id="plusMenu"></ul>
</div>
<div id="search-input-wrap">
<form action="../../search.html" autocomplete="off" class="" method="get" role="search">
<div class="form-group inner-addon left-addon">
<i class="glyphicon glyphicon-search"></i>
<input class="form-control" name="q" placeholder="Search" type="text"/>
</div>
<input name="check_keywords" type="hidden" value="yes">
<input name="area" type="hidden" value="default"/>
</input></form>
<div id="search-preview"></div>
</div>
<div id="searchIcon">
<span aria-hidden="true" class="glyphicon glyphicon-search"></span>
</div>
<!-- <div id="lang-select-wrap"> -->
<!-- <label id="lang-select-label"> -->
<!-- <\!-- <i class="fa fa-globe"></i> -\-> -->
<!-- <span></span> -->
<!-- </label> -->
<!-- <select id="lang-select"> -->
<!-- <option value="en">Eng</option> -->
<!-- <option value="zh">中文</option> -->
<!-- </select> -->
<!-- </div> -->
<!-- <a id="mobile-nav-toggle">
<span class="mobile-nav-toggle-bar"></span>
<span class="mobile-nav-toggle-bar"></span>
<span class="mobile-nav-toggle-bar"></span>
</a> -->
</div>
</div>
</div>
<script type="text/javascript">
$('body').css('background', 'white');
</script>
<div class="container">
<div class="row">
<div aria-label="main navigation" class="sphinxsidebar leftsidebar" role="navigation">
<div class="sphinxsidebarwrapper">
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../api/python/index.html">Python Documents</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/r/index.html">R Documents</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/julia/index.html">Julia Documents</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/c++/index.html">C++ Documents</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/scala/index.html">Scala Documents</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/perl/index.html">Perl Documents</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../faq/index.html">HowTo Documents</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../architecture/index.html">System Documents</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../tutorials/index.html">Tutorials</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../community/index.html">Community</a></li>
</ul>
</div>
</div>
<div class="content">
<div class="page-tracker"></div>
<h1>Source code for mxnet.io</h1><div class="highlight"><pre>
<span></span><span class="c1"># Licensed to the Apache Software Foundation (ASF) under one</span>
<span class="c1"># or more contributor license agreements. See the NOTICE file</span>
<span class="c1"># distributed with this work for additional information</span>
<span class="c1"># regarding copyright ownership. The ASF licenses this file</span>
<span class="c1"># to you under the Apache License, Version 2.0 (the</span>
<span class="c1"># "License"); you may not use this file except in compliance</span>
<span class="c1"># with the License. You may obtain a copy of the License at</span>
<span class="c1">#</span>
<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
<span class="c1">#</span>
<span class="c1"># Unless required by applicable law or agreed to in writing,</span>
<span class="c1"># software distributed under the License is distributed on an</span>
<span class="c1"># "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY</span>
<span class="c1"># KIND, either express or implied. See the License for the</span>
<span class="c1"># specific language governing permissions and limitations</span>
<span class="c1"># under the License.</span>
<span class="sd">"""Data iterators for common data formats."""</span>
<span class="kn">from</span> <span class="nn">__future__</span> <span class="k">import</span> <span class="n">absolute_import</span>
<span class="kn">from</span> <span class="nn">collections</span> <span class="k">import</span> <span class="n">OrderedDict</span><span class="p">,</span> <span class="n">namedtuple</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">ctypes</span>
<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">threading</span>
<span class="k">try</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">h5py</span>
<span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
<span class="n">h5py</span> <span class="o">=</span> <span class="kc">None</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">.base</span> <span class="k">import</span> <span class="n">_LIB</span>
<span class="kn">from</span> <span class="nn">.base</span> <span class="k">import</span> <span class="n">c_str_array</span><span class="p">,</span> <span class="n">mx_uint</span><span class="p">,</span> <span class="n">py_str</span>
<span class="kn">from</span> <span class="nn">.base</span> <span class="k">import</span> <span class="n">DataIterHandle</span><span class="p">,</span> <span class="n">NDArrayHandle</span>
<span class="kn">from</span> <span class="nn">.base</span> <span class="k">import</span> <span class="n">mx_real_t</span>
<span class="kn">from</span> <span class="nn">.base</span> <span class="k">import</span> <span class="n">check_call</span><span class="p">,</span> <span class="n">build_param_doc</span> <span class="k">as</span> <span class="n">_build_param_doc</span>
<span class="kn">from</span> <span class="nn">.ndarray</span> <span class="k">import</span> <span class="n">NDArray</span>
<span class="kn">from</span> <span class="nn">.ndarray.sparse</span> <span class="k">import</span> <span class="n">CSRNDArray</span>
<span class="kn">from</span> <span class="nn">.ndarray.sparse</span> <span class="k">import</span> <span class="n">array</span> <span class="k">as</span> <span class="n">sparse_array</span>
<span class="kn">from</span> <span class="nn">.ndarray</span> <span class="k">import</span> <span class="n">_ndarray_cls</span>
<span class="kn">from</span> <span class="nn">.ndarray</span> <span class="k">import</span> <span class="n">array</span>
<span class="kn">from</span> <span class="nn">.ndarray</span> <span class="k">import</span> <span class="n">concatenate</span>
<div class="viewcode-block" id="DataDesc"><a class="viewcode-back" href="../../api/python/io/io.html#mxnet.io.DataDesc">[docs]</a><span class="k">class</span> <span class="nc">DataDesc</span><span class="p">(</span><span class="n">namedtuple</span><span class="p">(</span><span class="s1">'DataDesc'</span><span class="p">,</span> <span class="p">[</span><span class="s1">'name'</span><span class="p">,</span> <span class="s1">'shape'</span><span class="p">])):</span>
<span class="sd">"""DataDesc is used to store name, shape, type and layout</span>
<span class="sd"> information of the data or the label.</span>
<span class="sd"> The `layout` describes how the axes in `shape` should be interpreted,</span>
<span class="sd"> for example for image data setting `layout=NCHW` indicates</span>
<span class="sd"> that the first axis is number of examples in the batch(N),</span>
<span class="sd"> C is number of channels, H is the height and W is the width of the image.</span>
<span class="sd"> For sequential data, by default `layout` is set to ``NTC``, where</span>
<span class="sd"> N is number of examples in the batch, T the temporal axis representing time</span>
<span class="sd"> and C is the number of channels.</span>
<span class="sd"> Parameters</span>
<span class="sd"> ----------</span>
<span class="sd"> cls : DataDesc</span>
<span class="sd"> The class.</span>
<span class="sd"> name : str</span>
<span class="sd"> Data name.</span>
<span class="sd"> shape : tuple of int</span>
<span class="sd"> Data shape.</span>
<span class="sd"> dtype : np.dtype, optional</span>
<span class="sd"> Data type.</span>
<span class="sd"> layout : str, optional</span>
<span class="sd"> Data layout.</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">__new__</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">shape</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">mx_real_t</span><span class="p">,</span> <span class="n">layout</span><span class="o">=</span><span class="s1">'NCHW'</span><span class="p">):</span> <span class="c1"># pylint: disable=super-on-old-class</span>
<span class="n">ret</span> <span class="o">=</span> <span class="nb">super</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">DataDesc</span><span class="p">)</span><span class="o">.</span><span class="fm">__new__</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">shape</span><span class="p">)</span>
<span class="n">ret</span><span class="o">.</span><span class="n">dtype</span> <span class="o">=</span> <span class="n">dtype</span>
<span class="n">ret</span><span class="o">.</span><span class="n">layout</span> <span class="o">=</span> <span class="n">layout</span>
<span class="k">return</span> <span class="n">ret</span>
<span class="k">def</span> <span class="nf">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="s2">"DataDesc[</span><span class="si">%s</span><span class="s2">,</span><span class="si">%s</span><span class="s2">,</span><span class="si">%s</span><span class="s2">,</span><span class="si">%s</span><span class="s2">]"</span> <span class="o">%</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">dtype</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">layout</span><span class="p">)</span>
<span class="nd">@staticmethod</span>
<div class="viewcode-block" id="DataDesc.get_batch_axis"><a class="viewcode-back" href="../../api/python/io/io.html#mxnet.io.DataDesc.get_batch_axis">[docs]</a> <span class="k">def</span> <span class="nf">get_batch_axis</span><span class="p">(</span><span class="n">layout</span><span class="p">):</span>
<span class="sd">"""Get the dimension that corresponds to the batch size.</span>
<span class="sd"> When data parallelism is used, the data will be automatically split and</span>
<span class="sd"> concatenated along the batch-size dimension. Axis can be -1, which means</span>
<span class="sd"> the whole array will be copied for each data-parallelism device.</span>
<span class="sd"> Parameters</span>
<span class="sd"> ----------</span>
<span class="sd"> layout : str</span>
<span class="sd"> layout string. For example, "NCHW".</span>
<span class="sd"> Returns</span>
<span class="sd"> -------</span>
<span class="sd"> int</span>
<span class="sd"> An axis indicating the batch_size dimension.</span>
<span class="sd"> """</span>
<span class="k">if</span> <span class="n">layout</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="mi">0</span>
<span class="k">return</span> <span class="n">layout</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s1">'N'</span><span class="p">)</span></div>
<span class="nd">@staticmethod</span>
<div class="viewcode-block" id="DataDesc.get_list"><a class="viewcode-back" href="../../api/python/io/io.html#mxnet.io.DataDesc.get_list">[docs]</a> <span class="k">def</span> <span class="nf">get_list</span><span class="p">(</span><span class="n">shapes</span><span class="p">,</span> <span class="n">types</span><span class="p">):</span>
<span class="sd">"""Get DataDesc list from attribute lists.</span>
<span class="sd"> Parameters</span>
<span class="sd"> ----------</span>
<span class="sd"> shapes : a tuple of (name, shape)</span>
<span class="sd"> types : a tuple of (name, type)</span>
<span class="sd"> """</span>
<span class="k">if</span> <span class="n">types</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">type_dict</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">types</span><span class="p">)</span>
<span class="k">return</span> <span class="p">[</span><span class="n">DataDesc</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">type_dict</span><span class="p">[</span><span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">]])</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">shapes</span><span class="p">]</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="p">[</span><span class="n">DataDesc</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">shapes</span><span class="p">]</span></div></div>
<div class="viewcode-block" id="DataBatch"><a class="viewcode-back" href="../../api/python/io/io.html#mxnet.io.DataBatch">[docs]</a><span class="k">class</span> <span class="nc">DataBatch</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="sd">"""A data batch.</span>
<span class="sd"> MXNet's data iterator returns a batch of data for each `next` call.</span>
<span class="sd"> This data contains `batch_size` number of examples.</span>
<span class="sd"> If the input data consists of images, then shape of these images depend on</span>
<span class="sd"> the `layout` attribute of `DataDesc` object in `provide_data` parameter.</span>
<span class="sd"> If `layout` is set to 'NCHW' then, images should be stored in a 4-D matrix</span>
<span class="sd"> of shape ``(batch_size, num_channel, height, width)``.</span>
<span class="sd"> If `layout` is set to 'NHWC' then, images should be stored in a 4-D matrix</span>
<span class="sd"> of shape ``(batch_size, height, width, num_channel)``.</span>
<span class="sd"> The channels are often in RGB order.</span>
<span class="sd"> Parameters</span>
<span class="sd"> ----------</span>
<span class="sd"> data : list of `NDArray`, each array containing `batch_size` examples.</span>
<span class="sd"> A list of input data.</span>
<span class="sd"> label : list of `NDArray`, each array often containing a 1-dimensional array. optional</span>
<span class="sd"> A list of input labels.</span>
<span class="sd"> pad : int, optional</span>
<span class="sd"> The number of examples padded at the end of a batch. It is used when the</span>
<span class="sd"> total number of examples read is not divisible by the `batch_size`.</span>
<span class="sd"> These extra padded examples are ignored in prediction.</span>
<span class="sd"> index : numpy.array, optional</span>
<span class="sd"> The example indices in this batch.</span>
<span class="sd"> bucket_key : int, optional</span>
<span class="sd"> The bucket key, used for bucketing module.</span>
<span class="sd"> provide_data : list of `DataDesc`, optional</span>
<span class="sd"> A list of `DataDesc` objects. `DataDesc` is used to store</span>
<span class="sd"> name, shape, type and layout information of the data.</span>
<span class="sd"> The *i*-th element describes the name and shape of ``data[i]``.</span>
<span class="sd"> provide_label : list of `DataDesc`, optional</span>
<span class="sd"> A list of `DataDesc` objects. `DataDesc` is used to store</span>
<span class="sd"> name, shape, type and layout information of the label.</span>
<span class="sd"> The *i*-th element describes the name and shape of ``label[i]``.</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">data</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">pad</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">index</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">bucket_key</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">provide_data</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">provide_label</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="k">if</span> <span class="n">data</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">assert</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="p">(</span><span class="nb">list</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">)),</span> <span class="s2">"Data must be list of NDArrays"</span>
<span class="k">if</span> <span class="n">label</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">assert</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">label</span><span class="p">,</span> <span class="p">(</span><span class="nb">list</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">)),</span> <span class="s2">"Label must be list of NDArrays"</span>
<span class="bp">self</span><span class="o">.</span><span class="n">data</span> <span class="o">=</span> <span class="n">data</span>
<span class="bp">self</span><span class="o">.</span><span class="n">label</span> <span class="o">=</span> <span class="n">label</span>
<span class="bp">self</span><span class="o">.</span><span class="n">pad</span> <span class="o">=</span> <span class="n">pad</span>
<span class="bp">self</span><span class="o">.</span><span class="n">index</span> <span class="o">=</span> <span class="n">index</span>
<span class="bp">self</span><span class="o">.</span><span class="n">bucket_key</span> <span class="o">=</span> <span class="n">bucket_key</span>
<span class="bp">self</span><span class="o">.</span><span class="n">provide_data</span> <span class="o">=</span> <span class="n">provide_data</span>
<span class="bp">self</span><span class="o">.</span><span class="n">provide_label</span> <span class="o">=</span> <span class="n">provide_label</span>
<span class="k">def</span> <span class="nf">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">data_shapes</span> <span class="o">=</span> <span class="p">[</span><span class="n">d</span><span class="o">.</span><span class="n">shape</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">]</span>
<span class="n">label_shapes</span> <span class="o">=</span> <span class="p">[</span><span class="n">l</span><span class="o">.</span><span class="n">shape</span> <span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">label</span><span class="p">]</span>
<span class="k">return</span> <span class="s2">"</span><span class="si">{}</span><span class="s2">: data shapes: </span><span class="si">{}</span><span class="s2"> label shapes: </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span>
<span class="n">data_shapes</span><span class="p">,</span>
<span class="n">label_shapes</span><span class="p">)</span></div>
<div class="viewcode-block" id="DataIter"><a class="viewcode-back" href="../../api/python/io/io.html#mxnet.io.DataIter">[docs]</a><span class="k">class</span> <span class="nc">DataIter</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="sd">"""The base class for an MXNet data iterator.</span>
<span class="sd"> All I/O in MXNet is handled by specializations of this class. Data iterators</span>
<span class="sd"> in MXNet are similar to standard-iterators in Python. On each call to `next`</span>
<span class="sd"> they return a `DataBatch` which represents the next batch of data. When</span>
<span class="sd"> there is no more data to return, it raises a `StopIteration` exception.</span>
<span class="sd"> Parameters</span>
<span class="sd"> ----------</span>
<span class="sd"> batch_size : int, optional</span>
<span class="sd"> The batch size, namely the number of items in the batch.</span>
<span class="sd"> See Also</span>
<span class="sd"> --------</span>
<span class="sd"> NDArrayIter : Data-iterator for MXNet NDArray or numpy-ndarray objects.</span>
<span class="sd"> CSVIter : Data-iterator for csv data.</span>
<span class="sd"> LibSVMIter : Data-iterator for libsvm data.</span>
<span class="sd"> ImageIter : Data-iterator for images.</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">batch_size</span><span class="o">=</span><span class="mi">0</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span> <span class="o">=</span> <span class="n">batch_size</span>
<span class="k">def</span> <span class="nf">__iter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span>
<div class="viewcode-block" id="DataIter.reset"><a class="viewcode-back" href="../../api/python/io/io.html#mxnet.io.DataIter.reset">[docs]</a> <span class="k">def</span> <span class="nf">reset</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Reset the iterator to the begin of the data."""</span>
<span class="k">pass</span></div>
<div class="viewcode-block" id="DataIter.next"><a class="viewcode-back" href="../../api/python/io/io.html#mxnet.io.DataIter.next">[docs]</a> <span class="k">def</span> <span class="nf">next</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Get next data batch from iterator.</span>
<span class="sd"> Returns</span>
<span class="sd"> -------</span>
<span class="sd"> DataBatch</span>
<span class="sd"> The data of next batch.</span>
<span class="sd"> Raises</span>
<span class="sd"> ------</span>
<span class="sd"> StopIteration</span>
<span class="sd"> If the end of the data is reached.</span>
<span class="sd"> """</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">iter_next</span><span class="p">():</span>
<span class="k">return</span> <span class="n">DataBatch</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">getdata</span><span class="p">(),</span> <span class="n">label</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">getlabel</span><span class="p">(),</span> \
<span class="n">pad</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">getpad</span><span class="p">(),</span> <span class="n">index</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">getindex</span><span class="p">())</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">StopIteration</span></div>
<span class="k">def</span> <span class="nf">__next__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">next</span><span class="p">()</span>
<div class="viewcode-block" id="DataIter.iter_next"><a class="viewcode-back" href="../../api/python/io/io.html#mxnet.io.DataIter.iter_next">[docs]</a> <span class="k">def</span> <span class="nf">iter_next</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Move to the next batch.</span>
<span class="sd"> Returns</span>
<span class="sd"> -------</span>
<span class="sd"> boolean</span>
<span class="sd"> Whether the move is successful.</span>
<span class="sd"> """</span>
<span class="k">pass</span></div>
<div class="viewcode-block" id="DataIter.getdata"><a class="viewcode-back" href="../../api/python/io/io.html#mxnet.io.DataIter.getdata">[docs]</a> <span class="k">def</span> <span class="nf">getdata</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Get data of current batch.</span>
<span class="sd"> Returns</span>
<span class="sd"> -------</span>
<span class="sd"> list of NDArray</span>
<span class="sd"> The data of the current batch.</span>
<span class="sd"> """</span>
<span class="k">pass</span></div>
<div class="viewcode-block" id="DataIter.getlabel"><a class="viewcode-back" href="../../api/python/io/io.html#mxnet.io.DataIter.getlabel">[docs]</a> <span class="k">def</span> <span class="nf">getlabel</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Get label of the current batch.</span>
<span class="sd"> Returns</span>
<span class="sd"> -------</span>
<span class="sd"> list of NDArray</span>
<span class="sd"> The label of the current batch.</span>
<span class="sd"> """</span>
<span class="k">pass</span></div>
<div class="viewcode-block" id="DataIter.getindex"><a class="viewcode-back" href="../../api/python/io/io.html#mxnet.io.DataIter.getindex">[docs]</a> <span class="k">def</span> <span class="nf">getindex</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Get index of the current batch.</span>
<span class="sd"> Returns</span>
<span class="sd"> -------</span>
<span class="sd"> index : numpy.array</span>
<span class="sd"> The indices of examples in the current batch.</span>
<span class="sd"> """</span>
<span class="k">return</span> <span class="kc">None</span></div>
<div class="viewcode-block" id="DataIter.getpad"><a class="viewcode-back" href="../../api/python/io/io.html#mxnet.io.DataIter.getpad">[docs]</a> <span class="k">def</span> <span class="nf">getpad</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Get the number of padding examples in the current batch.</span>
<span class="sd"> Returns</span>
<span class="sd"> -------</span>
<span class="sd"> int</span>
<span class="sd"> Number of padding examples in the current batch.</span>
<span class="sd"> """</span>
<span class="k">pass</span></div></div>
<div class="viewcode-block" id="ResizeIter"><a class="viewcode-back" href="../../api/python/io/io.html#mxnet.io.ResizeIter">[docs]</a><span class="k">class</span> <span class="nc">ResizeIter</span><span class="p">(</span><span class="n">DataIter</span><span class="p">):</span>
<span class="sd">"""Resize a data iterator to a given number of batches.</span>
<span class="sd"> Parameters</span>
<span class="sd"> ----------</span>
<span class="sd"> data_iter : DataIter</span>
<span class="sd"> The data iterator to be resized.</span>
<span class="sd"> size : int</span>
<span class="sd"> The number of batches per epoch to resize to.</span>
<span class="sd"> reset_internal : bool</span>
<span class="sd"> Whether to reset internal iterator on ResizeIter.reset.</span>
<span class="sd"> Examples</span>
<span class="sd"> --------</span>
<span class="sd"> >>> nd_iter = mx.io.NDArrayIter(mx.nd.ones((100,10)), batch_size=25)</span>
<span class="sd"> >>> resize_iter = mx.io.ResizeIter(nd_iter, 2)</span>
<span class="sd"> >>> for batch in resize_iter:</span>
<span class="sd"> ... print(batch.data)</span>
<span class="sd"> [<NDArray 25x10 @cpu(0)>]</span>
<span class="sd"> [<NDArray 25x10 @cpu(0)>]</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">data_iter</span><span class="p">,</span> <span class="n">size</span><span class="p">,</span> <span class="n">reset_internal</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
<span class="nb">super</span><span class="p">(</span><span class="n">ResizeIter</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">data_iter</span> <span class="o">=</span> <span class="n">data_iter</span>
<span class="bp">self</span><span class="o">.</span><span class="n">size</span> <span class="o">=</span> <span class="n">size</span>
<span class="bp">self</span><span class="o">.</span><span class="n">reset_internal</span> <span class="o">=</span> <span class="n">reset_internal</span>
<span class="bp">self</span><span class="o">.</span><span class="n">cur</span> <span class="o">=</span> <span class="mi">0</span>
<span class="bp">self</span><span class="o">.</span><span class="n">current_batch</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">provide_data</span> <span class="o">=</span> <span class="n">data_iter</span><span class="o">.</span><span class="n">provide_data</span>
<span class="bp">self</span><span class="o">.</span><span class="n">provide_label</span> <span class="o">=</span> <span class="n">data_iter</span><span class="o">.</span><span class="n">provide_label</span>
<span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span> <span class="o">=</span> <span class="n">data_iter</span><span class="o">.</span><span class="n">batch_size</span>
<span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">data_iter</span><span class="p">,</span> <span class="s1">'default_bucket_key'</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">default_bucket_key</span> <span class="o">=</span> <span class="n">data_iter</span><span class="o">.</span><span class="n">default_bucket_key</span>
<span class="k">def</span> <span class="nf">reset</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">cur</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">reset_internal</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">data_iter</span><span class="o">.</span><span class="n">reset</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">iter_next</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">cur</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">size</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="k">try</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">current_batch</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">data_iter</span><span class="o">.</span><span class="n">next</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">StopIteration</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">data_iter</span><span class="o">.</span><span class="n">reset</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">current_batch</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">data_iter</span><span class="o">.</span><span class="n">next</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">cur</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="kc">True</span>
<span class="k">def</span> <span class="nf">getdata</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_batch</span><span class="o">.</span><span class="n">data</span>
<span class="k">def</span> <span class="nf">getlabel</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_batch</span><span class="o">.</span><span class="n">label</span>
<span class="k">def</span> <span class="nf">getindex</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_batch</span><span class="o">.</span><span class="n">index</span>
<span class="k">def</span> <span class="nf">getpad</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_batch</span><span class="o">.</span><span class="n">pad</span></div>
<div class="viewcode-block" id="PrefetchingIter"><a class="viewcode-back" href="../../api/python/io/io.html#mxnet.io.PrefetchingIter">[docs]</a><span class="k">class</span> <span class="nc">PrefetchingIter</span><span class="p">(</span><span class="n">DataIter</span><span class="p">):</span>
<span class="sd">"""Performs pre-fetch for other data iterators.</span>
<span class="sd"> This iterator will create another thread to perform ``iter_next`` and then</span>
<span class="sd"> store the data in memory. It potentially accelerates the data read, at the</span>
<span class="sd"> cost of more memory usage.</span>
<span class="sd"> Parameters</span>
<span class="sd"> ----------</span>
<span class="sd"> iters : DataIter or list of DataIter</span>
<span class="sd"> The data iterators to be pre-fetched.</span>
<span class="sd"> rename_data : None or list of dict</span>
<span class="sd"> The *i*-th element is a renaming map for the *i*-th iter, in the form of</span>
<span class="sd"> {'original_name' : 'new_name'}. Should have one entry for each entry</span>
<span class="sd"> in iter[i].provide_data.</span>
<span class="sd"> rename_label : None or list of dict</span>
<span class="sd"> Similar to ``rename_data``.</span>
<span class="sd"> Examples</span>
<span class="sd"> --------</span>
<span class="sd"> >>> iter1 = mx.io.NDArrayIter({'data':mx.nd.ones((100,10))}, batch_size=25)</span>
<span class="sd"> >>> iter2 = mx.io.NDArrayIter({'data':mx.nd.ones((100,10))}, batch_size=25)</span>
<span class="sd"> >>> piter = mx.io.PrefetchingIter([iter1, iter2],</span>
<span class="sd"> ... rename_data=[{'data': 'data_1'}, {'data': 'data_2'}])</span>
<span class="sd"> >>> print(piter.provide_data)</span>
<span class="sd"> [DataDesc[data_1,(25, 10L),<type 'numpy.float32'>,NCHW],</span>
<span class="sd"> DataDesc[data_2,(25, 10L),<type 'numpy.float32'>,NCHW]]</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">iters</span><span class="p">,</span> <span class="n">rename_data</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">rename_label</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="nb">super</span><span class="p">(</span><span class="n">PrefetchingIter</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">iters</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
<span class="n">iters</span> <span class="o">=</span> <span class="p">[</span><span class="n">iters</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">n_iter</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">iters</span><span class="p">)</span>
<span class="k">assert</span> <span class="bp">self</span><span class="o">.</span><span class="n">n_iter</span> <span class="o">></span> <span class="mi">0</span>
<span class="bp">self</span><span class="o">.</span><span class="n">iters</span> <span class="o">=</span> <span class="n">iters</span>
<span class="bp">self</span><span class="o">.</span><span class="n">rename_data</span> <span class="o">=</span> <span class="n">rename_data</span>
<span class="bp">self</span><span class="o">.</span><span class="n">rename_label</span> <span class="o">=</span> <span class="n">rename_label</span>
<span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">provide_data</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">1</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">data_ready</span> <span class="o">=</span> <span class="p">[</span><span class="n">threading</span><span class="o">.</span><span class="n">Event</span><span class="p">()</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">n_iter</span><span class="p">)]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">data_taken</span> <span class="o">=</span> <span class="p">[</span><span class="n">threading</span><span class="o">.</span><span class="n">Event</span><span class="p">()</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">n_iter</span><span class="p">)]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">data_taken</span><span class="p">:</span>
<span class="n">i</span><span class="o">.</span><span class="n">set</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">started</span> <span class="o">=</span> <span class="kc">True</span>
<span class="bp">self</span><span class="o">.</span><span class="n">current_batch</span> <span class="o">=</span> <span class="p">[</span><span class="kc">None</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">n_iter</span><span class="p">)]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">next_batch</span> <span class="o">=</span> <span class="p">[</span><span class="kc">None</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">n_iter</span><span class="p">)]</span>
<span class="k">def</span> <span class="nf">prefetch_func</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">i</span><span class="p">):</span>
<span class="sd">"""Thread entry"""</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">data_taken</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">started</span><span class="p">:</span>
<span class="k">break</span>
<span class="k">try</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">next_batch</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">iters</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">next</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">StopIteration</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">next_batch</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">data_taken</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">data_ready</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">set</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">prefetch_threads</span> <span class="o">=</span> <span class="p">[</span><span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">prefetch_func</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">[</span><span class="bp">self</span><span class="p">,</span> <span class="n">i</span><span class="p">])</span> \
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">n_iter</span><span class="p">)]</span>
<span class="k">for</span> <span class="n">thread</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">prefetch_threads</span><span class="p">:</span>
<span class="n">thread</span><span class="o">.</span><span class="n">setDaemon</span><span class="p">(</span><span class="kc">True</span><span class="p">)</span>
<span class="n">thread</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">__del__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">started</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">data_taken</span><span class="p">:</span>
<span class="n">i</span><span class="o">.</span><span class="n">set</span><span class="p">()</span>
<span class="k">for</span> <span class="n">thread</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">prefetch_threads</span><span class="p">:</span>
<span class="n">thread</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">provide_data</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">rename_data</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">sum</span><span class="p">([</span><span class="n">i</span><span class="o">.</span><span class="n">provide_data</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">iters</span><span class="p">],</span> <span class="p">[])</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">sum</span><span class="p">([[</span>
<span class="n">DataDesc</span><span class="p">(</span><span class="n">r</span><span class="p">[</span><span class="n">x</span><span class="o">.</span><span class="n">name</span><span class="p">],</span> <span class="n">x</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">x</span><span class="o">.</span><span class="n">dtype</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">DataDesc</span><span class="p">)</span> <span class="k">else</span> <span class="n">DataDesc</span><span class="p">(</span><span class="o">*</span><span class="n">x</span><span class="p">)</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">i</span><span class="o">.</span><span class="n">provide_data</span>
<span class="p">]</span> <span class="k">for</span> <span class="n">r</span><span class="p">,</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">rename_data</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">iters</span><span class="p">)],</span> <span class="p">[])</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">provide_label</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">rename_label</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">sum</span><span class="p">([</span><span class="n">i</span><span class="o">.</span><span class="n">provide_label</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">iters</span><span class="p">],</span> <span class="p">[])</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">sum</span><span class="p">([[</span>
<span class="n">DataDesc</span><span class="p">(</span><span class="n">r</span><span class="p">[</span><span class="n">x</span><span class="o">.</span><span class="n">name</span><span class="p">],</span> <span class="n">x</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">x</span><span class="o">.</span><span class="n">dtype</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">DataDesc</span><span class="p">)</span> <span class="k">else</span> <span class="n">DataDesc</span><span class="p">(</span><span class="o">*</span><span class="n">x</span><span class="p">)</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">i</span><span class="o">.</span><span class="n">provide_label</span>
<span class="p">]</span> <span class="k">for</span> <span class="n">r</span><span class="p">,</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">rename_label</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">iters</span><span class="p">)],</span> <span class="p">[])</span>
<span class="k">def</span> <span class="nf">reset</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">data_ready</span><span class="p">:</span>
<span class="n">i</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">iters</span><span class="p">:</span>
<span class="n">i</span><span class="o">.</span><span class="n">reset</span><span class="p">()</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">data_ready</span><span class="p">:</span>
<span class="n">i</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">data_taken</span><span class="p">:</span>
<span class="n">i</span><span class="o">.</span><span class="n">set</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">iter_next</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">data_ready</span><span class="p">:</span>
<span class="n">i</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">next_batch</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">next_batch</span><span class="p">:</span>
<span class="k">assert</span> <span class="n">i</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">,</span> <span class="s2">"Number of entry mismatches between iterators"</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">next_batch</span><span class="p">:</span>
<span class="k">assert</span> <span class="n">batch</span><span class="o">.</span><span class="n">pad</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">next_batch</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">pad</span><span class="p">,</span> \
<span class="s2">"Number of entry mismatches between iterators"</span>
<span class="bp">self</span><span class="o">.</span><span class="n">current_batch</span> <span class="o">=</span> <span class="n">DataBatch</span><span class="p">(</span><span class="nb">sum</span><span class="p">([</span><span class="n">batch</span><span class="o">.</span><span class="n">data</span> <span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">next_batch</span><span class="p">],</span> <span class="p">[]),</span>
<span class="nb">sum</span><span class="p">([</span><span class="n">batch</span><span class="o">.</span><span class="n">label</span> <span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">next_batch</span><span class="p">],</span> <span class="p">[]),</span>
<span class="bp">self</span><span class="o">.</span><span class="n">next_batch</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">pad</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">next_batch</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">index</span><span class="p">,</span>
<span class="n">provide_data</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">provide_data</span><span class="p">,</span>
<span class="n">provide_label</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">provide_label</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">data_ready</span><span class="p">:</span>
<span class="n">i</span><span class="o">.</span><span class="n">clear</span><span class="p">()</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">data_taken</span><span class="p">:</span>
<span class="n">i</span><span class="o">.</span><span class="n">set</span><span class="p">()</span>
<span class="k">return</span> <span class="kc">True</span>
<span class="k">def</span> <span class="nf">next</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">iter_next</span><span class="p">():</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_batch</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">StopIteration</span>
<span class="k">def</span> <span class="nf">getdata</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_batch</span><span class="o">.</span><span class="n">data</span>
<span class="k">def</span> <span class="nf">getlabel</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_batch</span><span class="o">.</span><span class="n">label</span>
<span class="k">def</span> <span class="nf">getindex</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_batch</span><span class="o">.</span><span class="n">index</span>
<span class="k">def</span> <span class="nf">getpad</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_batch</span><span class="o">.</span><span class="n">pad</span></div>
<span class="k">def</span> <span class="nf">_init_data</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">allow_empty</span><span class="p">,</span> <span class="n">default_name</span><span class="p">):</span>
<span class="sd">"""Convert data into canonical form."""</span>
<span class="k">assert</span> <span class="p">(</span><span class="n">data</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">)</span> <span class="ow">or</span> <span class="n">allow_empty</span>
<span class="k">if</span> <span class="n">data</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">data</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">ndarray</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">,</span> <span class="n">h5py</span><span class="o">.</span><span class="n">Dataset</span><span class="p">)</span>
<span class="k">if</span> <span class="n">h5py</span> <span class="k">else</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">ndarray</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">)):</span>
<span class="n">data</span> <span class="o">=</span> <span class="p">[</span><span class="n">data</span><span class="p">]</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">allow_empty</span><span class="p">:</span>
<span class="k">assert</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">OrderedDict</span><span class="p">([(</span><span class="n">default_name</span><span class="p">,</span> <span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">])])</span> <span class="c1"># pylint: disable=redefined-variable-type</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">OrderedDict</span><span class="p">(</span> <span class="c1"># pylint: disable=redefined-variable-type</span>
<span class="p">[(</span><span class="s1">'_</span><span class="si">%d</span><span class="s1">_</span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">default_name</span><span class="p">),</span> <span class="n">d</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">d</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">data</span><span class="p">)])</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">TypeError</span><span class="p">(</span><span class="s2">"Input must be NDArray, numpy.ndarray, h5py.Dataset "</span> <span class="o">+</span> \
<span class="s2">"a list of them or dict with them as values"</span><span class="p">)</span>
<span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">data</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="p">(</span><span class="n">NDArray</span><span class="p">,</span> <span class="n">h5py</span><span class="o">.</span><span class="n">Dataset</span><span class="p">)</span> <span class="k">if</span> <span class="n">h5py</span> <span class="k">else</span> <span class="n">NDArray</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">data</span><span class="p">[</span><span class="n">k</span><span class="p">]</span> <span class="o">=</span> <span class="n">array</span><span class="p">(</span><span class="n">v</span><span class="p">)</span>
<span class="k">except</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">TypeError</span><span class="p">((</span><span class="s2">"Invalid type '</span><span class="si">%s</span><span class="s2">' for </span><span class="si">%s</span><span class="s2">, "</span> <span class="o">%</span> <span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="n">v</span><span class="p">),</span> <span class="n">k</span><span class="p">))</span> <span class="o">+</span> \
<span class="s2">"should be NDArray, numpy.ndarray or h5py.Dataset"</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">list</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">items</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">_has_instance</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">dtype</span><span class="p">):</span>
<span class="sd">"""Return True if ``data`` has instance of ``dtype``.</span>
<span class="sd"> This function is called after _init_data.</span>
<span class="sd"> ``data`` is a list of (str, NDArray)"""</span>
<span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">data</span><span class="p">:</span>
<span class="n">_</span><span class="p">,</span> <span class="n">arr</span> <span class="o">=</span> <span class="n">item</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">arr</span><span class="p">,</span> <span class="n">dtype</span><span class="p">):</span>
<span class="k">return</span> <span class="kc">True</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="k">def</span> <span class="nf">_shuffle</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">idx</span><span class="p">):</span>
<span class="sd">"""Shuffle the data."""</span>
<span class="n">shuffle_data</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">data</span><span class="p">:</span>
<span class="k">if</span> <span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">h5py</span><span class="o">.</span><span class="n">Dataset</span><span class="p">)</span> <span class="k">if</span> <span class="n">h5py</span> <span class="k">else</span> <span class="kc">False</span><span class="p">):</span>
<span class="n">shuffle_data</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">))</span>
<span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">CSRNDArray</span><span class="p">):</span>
<span class="n">shuffle_data</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">k</span><span class="p">,</span> <span class="n">sparse_array</span><span class="p">(</span><span class="n">v</span><span class="o">.</span><span class="n">asscipy</span><span class="p">()[</span><span class="n">idx</span><span class="p">],</span> <span class="n">v</span><span class="o">.</span><span class="n">context</span><span class="p">)))</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">shuffle_data</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">k</span><span class="p">,</span> <span class="n">array</span><span class="p">(</span><span class="n">v</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()[</span><span class="n">idx</span><span class="p">],</span> <span class="n">v</span><span class="o">.</span><span class="n">context</span><span class="p">)))</span>
<span class="k">return</span> <span class="n">shuffle_data</span>
<div class="viewcode-block" id="NDArrayIter"><a class="viewcode-back" href="../../api/python/io/io.html#mxnet.io.NDArrayIter">[docs]</a><span class="k">class</span> <span class="nc">NDArrayIter</span><span class="p">(</span><span class="n">DataIter</span><span class="p">):</span>
<span class="sd">"""Returns an iterator for ``mx.nd.NDArray``, ``numpy.ndarray``, ``h5py.Dataset``</span>
<span class="sd"> ``mx.nd.sparse.CSRNDArray`` or ``scipy.sparse.csr_matrix``.</span>
<span class="sd"> Example usage:</span>
<span class="sd"> ----------</span>
<span class="sd"> >>> data = np.arange(40).reshape((10,2,2))</span>
<span class="sd"> >>> labels = np.ones([10, 1])</span>
<span class="sd"> >>> dataiter = mx.io.NDArrayIter(data, labels, 3, True, last_batch_handle='discard')</span>
<span class="sd"> >>> for batch in dataiter:</span>
<span class="sd"> ... print batch.data[0].asnumpy()</span>
<span class="sd"> ... batch.data[0].shape</span>
<span class="sd"> ...</span>
<span class="sd"> [[[ 36. 37.]</span>
<span class="sd"> [ 38. 39.]]</span>
<span class="sd"> [[ 16. 17.]</span>
<span class="sd"> [ 18. 19.]]</span>
<span class="sd"> [[ 12. 13.]</span>
<span class="sd"> [ 14. 15.]]]</span>
<span class="sd"> (3L, 2L, 2L)</span>
<span class="sd"> [[[ 32. 33.]</span>
<span class="sd"> [ 34. 35.]]</span>
<span class="sd"> [[ 4. 5.]</span>
<span class="sd"> [ 6. 7.]]</span>
<span class="sd"> [[ 24. 25.]</span>
<span class="sd"> [ 26. 27.]]]</span>
<span class="sd"> (3L, 2L, 2L)</span>
<span class="sd"> [[[ 8. 9.]</span>
<span class="sd"> [ 10. 11.]]</span>
<span class="sd"> [[ 20. 21.]</span>
<span class="sd"> [ 22. 23.]]</span>
<span class="sd"> [[ 28. 29.]</span>
<span class="sd"> [ 30. 31.]]]</span>
<span class="sd"> (3L, 2L, 2L)</span>
<span class="sd"> >>> dataiter.provide_data # Returns a list of `DataDesc`</span>
<span class="sd"> [DataDesc[data,(3, 2L, 2L),<type 'numpy.float32'>,NCHW]]</span>
<span class="sd"> >>> dataiter.provide_label # Returns a list of `DataDesc`</span>
<span class="sd"> [DataDesc[softmax_label,(3, 1L),<type 'numpy.float32'>,NCHW]]</span>
<span class="sd"> In the above example, data is shuffled as `shuffle` parameter is set to `True`</span>
<span class="sd"> and remaining examples are discarded as `last_batch_handle` parameter is set to `discard`.</span>
<span class="sd"> Usage of `last_batch_handle` parameter:</span>
<span class="sd"> >>> dataiter = mx.io.NDArrayIter(data, labels, 3, True, last_batch_handle='pad')</span>
<span class="sd"> >>> batchidx = 0</span>
<span class="sd"> >>> for batch in dataiter:</span>
<span class="sd"> ... batchidx += 1</span>
<span class="sd"> ...</span>
<span class="sd"> >>> batchidx # Padding added after the examples read are over. So, 10/3+1 batches are created.</span>
<span class="sd"> 4</span>
<span class="sd"> >>> dataiter = mx.io.NDArrayIter(data, labels, 3, True, last_batch_handle='discard')</span>
<span class="sd"> >>> batchidx = 0</span>
<span class="sd"> >>> for batch in dataiter:</span>
<span class="sd"> ... batchidx += 1</span>
<span class="sd"> ...</span>
<span class="sd"> >>> batchidx # Remaining examples are discarded. So, 10/3 batches are created.</span>
<span class="sd"> 3</span>
<span class="sd"> `NDArrayIter` also supports multiple input and labels.</span>
<span class="sd"> >>> data = {'data1':np.zeros(shape=(10,2,2)), 'data2':np.zeros(shape=(20,2,2))}</span>
<span class="sd"> >>> label = {'label1':np.zeros(shape=(10,1)), 'label2':np.zeros(shape=(20,1))}</span>
<span class="sd"> >>> dataiter = mx.io.NDArrayIter(data, label, 3, True, last_batch_handle='discard')</span>
<span class="sd"> `NDArrayIter` also supports ``mx.nd.sparse.CSRNDArray``</span>
<span class="sd"> with `last_batch_handle` set to `discard`.</span>
<span class="sd"> >>> csr_data = mx.nd.array(np.arange(40).reshape((10,4))).tostype('csr')</span>
<span class="sd"> >>> labels = np.ones([10, 1])</span>
<span class="sd"> >>> dataiter = mx.io.NDArrayIter(csr_data, labels, 3, last_batch_handle='discard')</span>
<span class="sd"> >>> [batch.data[0] for batch in dataiter]</span>
<span class="sd"> [</span>
<span class="sd"> <CSRNDArray 3x4 @cpu(0)>,</span>
<span class="sd"> <CSRNDArray 3x4 @cpu(0)>,</span>
<span class="sd"> <CSRNDArray 3x4 @cpu(0)>]</span>
<span class="sd"> Parameters</span>
<span class="sd"> ----------</span>
<span class="sd"> data: array or list of array or dict of string to array</span>
<span class="sd"> The input data.</span>
<span class="sd"> label: array or list of array or dict of string to array, optional</span>
<span class="sd"> The input label.</span>
<span class="sd"> batch_size: int</span>
<span class="sd"> Batch size of data.</span>
<span class="sd"> shuffle: bool, optional</span>
<span class="sd"> Whether to shuffle the data.</span>
<span class="sd"> Only supported if no h5py.Dataset inputs are used.</span>
<span class="sd"> last_batch_handle : str, optional</span>
<span class="sd"> How to handle the last batch. This parameter can be 'pad', 'discard' or</span>
<span class="sd"> 'roll_over'. 'roll_over' is intended for training and can cause problems</span>
<span class="sd"> if used for prediction.</span>
<span class="sd"> data_name : str, optional</span>
<span class="sd"> The data name.</span>
<span class="sd"> label_name : str, optional</span>
<span class="sd"> The label name.</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">data</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">batch_size</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">shuffle</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="n">last_batch_handle</span><span class="o">=</span><span class="s1">'pad'</span><span class="p">,</span> <span class="n">data_name</span><span class="o">=</span><span class="s1">'data'</span><span class="p">,</span>
<span class="n">label_name</span><span class="o">=</span><span class="s1">'softmax_label'</span><span class="p">):</span>
<span class="nb">super</span><span class="p">(</span><span class="n">NDArrayIter</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">batch_size</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">data</span> <span class="o">=</span> <span class="n">_init_data</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">allow_empty</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">default_name</span><span class="o">=</span><span class="n">data_name</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">label</span> <span class="o">=</span> <span class="n">_init_data</span><span class="p">(</span><span class="n">label</span><span class="p">,</span> <span class="n">allow_empty</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">default_name</span><span class="o">=</span><span class="n">label_name</span><span class="p">)</span>
<span class="k">if</span> <span class="p">((</span><span class="n">_has_instance</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">,</span> <span class="n">CSRNDArray</span><span class="p">)</span> <span class="ow">or</span> <span class="n">_has_instance</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">label</span><span class="p">,</span> <span class="n">CSRNDArray</span><span class="p">))</span> <span class="ow">and</span>
<span class="p">(</span><span class="n">last_batch_handle</span> <span class="o">!=</span> <span class="s1">'discard'</span><span class="p">)):</span>
<span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">(</span><span class="s2">"`NDArrayIter` only supports ``CSRNDArray``"</span> \
<span class="s2">" with `last_batch_handle` set to `discard`."</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">idx</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="c1"># shuffle data</span>
<span class="k">if</span> <span class="n">shuffle</span><span class="p">:</span>
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">shuffle</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">idx</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">data</span> <span class="o">=</span> <span class="n">_shuffle</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">idx</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">label</span> <span class="o">=</span> <span class="n">_shuffle</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">label</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">idx</span><span class="p">)</span>
<span class="c1"># batching</span>
<span class="k">if</span> <span class="n">last_batch_handle</span> <span class="o">==</span> <span class="s1">'discard'</span><span class="p">:</span>
<span class="n">new_n</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">%</span> <span class="n">batch_size</span>
<span class="bp">self</span><span class="o">.</span><span class="n">idx</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">idx</span><span class="p">[:</span><span class="n">new_n</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">data_list</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">]</span> <span class="o">+</span> <span class="p">[</span><span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">label</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">num_source</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">data_list</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">num_data</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">idx</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">assert</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_data</span> <span class="o">>=</span> <span class="n">batch_size</span><span class="p">,</span> \
<span class="s2">"batch_size needs to be smaller than data size."</span>
<span class="bp">self</span><span class="o">.</span><span class="n">cursor</span> <span class="o">=</span> <span class="o">-</span><span class="n">batch_size</span>
<span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span> <span class="o">=</span> <span class="n">batch_size</span>
<span class="bp">self</span><span class="o">.</span><span class="n">last_batch_handle</span> <span class="o">=</span> <span class="n">last_batch_handle</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">provide_data</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""The name and shape of data provided by this iterator."""</span>
<span class="k">return</span> <span class="p">[</span>
<span class="n">DataDesc</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">([</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">]</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="n">v</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">:])),</span> <span class="n">v</span><span class="o">.</span><span class="n">dtype</span><span class="p">)</span>
<span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">data</span>
<span class="p">]</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">provide_label</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""The name and shape of label provided by this iterator."""</span>
<span class="k">return</span> <span class="p">[</span>
<span class="n">DataDesc</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">([</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">]</span> <span class="o">+</span> <span class="nb">list</span><span class="p">(</span><span class="n">v</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">:])),</span> <span class="n">v</span><span class="o">.</span><span class="n">dtype</span><span class="p">)</span>
<span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">label</span>
<span class="p">]</span>
<div class="viewcode-block" id="NDArrayIter.hard_reset"><a class="viewcode-back" href="../../api/python/io/io.html#mxnet.io.NDArrayIter.hard_reset">[docs]</a> <span class="k">def</span> <span class="nf">hard_reset</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Ignore roll over data and set to start."""</span>
<span class="bp">self</span><span class="o">.</span><span class="n">cursor</span> <span class="o">=</span> <span class="o">-</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span></div>
<span class="k">def</span> <span class="nf">reset</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">last_batch_handle</span> <span class="o">==</span> <span class="s1">'roll_over'</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">cursor</span> <span class="o">></span> <span class="bp">self</span><span class="o">.</span><span class="n">num_data</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">cursor</span> <span class="o">=</span> <span class="o">-</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span> <span class="o">+</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">cursor</span><span class="o">%</span><span class="bp">self</span><span class="o">.</span><span class="n">num_data</span><span class="p">)</span><span class="o">%</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span>
<span class="k">else</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">cursor</span> <span class="o">=</span> <span class="o">-</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span>
<span class="k">def</span> <span class="nf">iter_next</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">cursor</span> <span class="o">+=</span> <span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">cursor</span> <span class="o"><</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_data</span>
<span class="k">def</span> <span class="nf">next</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">iter_next</span><span class="p">():</span>
<span class="k">return</span> <span class="n">DataBatch</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">getdata</span><span class="p">(),</span> <span class="n">label</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">getlabel</span><span class="p">(),</span> \
<span class="n">pad</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">getpad</span><span class="p">(),</span> <span class="n">index</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">StopIteration</span>
<span class="k">def</span> <span class="nf">_getdata</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">data_source</span><span class="p">):</span>
<span class="sd">"""Load data from underlying arrays, internal use only."""</span>
<span class="k">assert</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">cursor</span> <span class="o"><</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_data</span><span class="p">),</span> <span class="s2">"DataIter needs reset."</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">cursor</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span> <span class="o"><=</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_data</span><span class="p">:</span>
<span class="k">return</span> <span class="p">[</span>
<span class="c1"># np.ndarray or NDArray case</span>
<span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">][</span><span class="bp">self</span><span class="o">.</span><span class="n">cursor</span><span class="p">:</span><span class="bp">self</span><span class="o">.</span><span class="n">cursor</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">]</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">ndarray</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> <span class="k">else</span>
<span class="c1"># h5py (only supports indices in increasing order)</span>
<span class="n">array</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">][</span><span class="nb">sorted</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">idx</span><span class="p">[</span>
<span class="bp">self</span><span class="o">.</span><span class="n">cursor</span><span class="p">:</span><span class="bp">self</span><span class="o">.</span><span class="n">cursor</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">])][[</span>
<span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">idx</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">cursor</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">cursor</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">])</span><span class="o">.</span><span class="n">index</span><span class="p">(</span><span class="n">i</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">idx</span><span class="p">[</span>
<span class="bp">self</span><span class="o">.</span><span class="n">cursor</span><span class="p">:</span><span class="bp">self</span><span class="o">.</span><span class="n">cursor</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span><span class="p">])</span>
<span class="p">]])</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">data_source</span>
<span class="p">]</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">pad</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_data</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">cursor</span>
<span class="k">return</span> <span class="p">[</span>
<span class="c1"># np.ndarray or NDArray case</span>
<span class="n">concatenate</span><span class="p">([</span><span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">][</span><span class="bp">self</span><span class="o">.</span><span class="n">cursor</span><span class="p">:],</span> <span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">][:</span><span class="n">pad</span><span class="p">]])</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">ndarray</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> <span class="k">else</span>
<span class="c1"># h5py (only supports indices in increasing order)</span>
<span class="n">concatenate</span><span class="p">([</span>
<span class="n">array</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">][</span><span class="nb">sorted</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">idx</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">cursor</span><span class="p">:])][[</span>
<span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">idx</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">cursor</span><span class="p">:])</span><span class="o">.</span><span class="n">index</span><span class="p">(</span><span class="n">i</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">idx</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">cursor</span><span class="p">:])</span>
<span class="p">]]),</span>
<span class="n">array</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">][</span><span class="nb">sorted</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">idx</span><span class="p">[:</span><span class="n">pad</span><span class="p">])][[</span>
<span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">idx</span><span class="p">[:</span><span class="n">pad</span><span class="p">])</span><span class="o">.</span><span class="n">index</span><span class="p">(</span><span class="n">i</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">idx</span><span class="p">[:</span><span class="n">pad</span><span class="p">])</span>
<span class="p">]])</span>
<span class="p">])</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">data_source</span>
<span class="p">]</span>
<span class="k">def</span> <span class="nf">getdata</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_getdata</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">getlabel</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_getdata</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">label</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">getpad</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">last_batch_handle</span> <span class="o">==</span> <span class="s1">'pad'</span> <span class="ow">and</span> \
<span class="bp">self</span><span class="o">.</span><span class="n">cursor</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span> <span class="o">></span> <span class="bp">self</span><span class="o">.</span><span class="n">num_data</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">cursor</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_data</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="mi">0</span></div>
<div class="viewcode-block" id="MXDataIter"><a class="viewcode-back" href="../../api/python/io/io.html#mxnet.io.MXDataIter">[docs]</a><span class="k">class</span> <span class="nc">MXDataIter</span><span class="p">(</span><span class="n">DataIter</span><span class="p">):</span>
<span class="sd">"""A python wrapper a C++ data iterator.</span>
<span class="sd"> This iterator is the Python wrapper to all native C++ data iterators, such</span>
<span class="sd"> as `CSVIter`, `ImageRecordIter`, `MNISTIter`, etc. When initializing</span>
<span class="sd"> `CSVIter` for example, you will get an `MXDataIter` instance to use in your</span>
<span class="sd"> Python code. Calls to `next`, `reset`, etc will be delegated to the</span>
<span class="sd"> underlying C++ data iterators.</span>
<span class="sd"> Usually you don't need to interact with `MXDataIter` directly unless you are</span>
<span class="sd"> implementing your own data iterators in C++. To do that, please refer to</span>
<span class="sd"> examples under the `src/io` folder.</span>
<span class="sd"> Parameters</span>
<span class="sd"> ----------</span>
<span class="sd"> handle : DataIterHandle, required</span>
<span class="sd"> The handle to the underlying C++ Data Iterator.</span>
<span class="sd"> data_name : str, optional</span>
<span class="sd"> Data name. Default to "data".</span>
<span class="sd"> label_name : str, optional</span>
<span class="sd"> Label name. Default to "softmax_label".</span>
<span class="sd"> See Also</span>
<span class="sd"> --------</span>
<span class="sd"> src/io : The underlying C++ data iterator implementation, e.g., `CSVIter`.</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">handle</span><span class="p">,</span> <span class="n">data_name</span><span class="o">=</span><span class="s1">'data'</span><span class="p">,</span> <span class="n">label_name</span><span class="o">=</span><span class="s1">'softmax_label'</span><span class="p">,</span> <span class="o">**</span><span class="n">_</span><span class="p">):</span>
<span class="nb">super</span><span class="p">(</span><span class="n">MXDataIter</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">handle</span> <span class="o">=</span> <span class="n">handle</span>
<span class="c1"># debug option, used to test the speed with io effect eliminated</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_debug_skip_load</span> <span class="o">=</span> <span class="kc">False</span>
<span class="c1"># load the first batch to get shape information</span>
<span class="bp">self</span><span class="o">.</span><span class="n">first_batch</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">first_batch</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">next</span><span class="p">()</span>
<span class="n">data</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">first_batch</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">label</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">first_batch</span><span class="o">.</span><span class="n">label</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="c1"># properties</span>
<span class="bp">self</span><span class="o">.</span><span class="n">provide_data</span> <span class="o">=</span> <span class="p">[</span><span class="n">DataDesc</span><span class="p">(</span><span class="n">data_name</span><span class="p">,</span> <span class="n">data</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">data</span><span class="o">.</span><span class="n">dtype</span><span class="p">)]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">provide_label</span> <span class="o">=</span> <span class="p">[</span><span class="n">DataDesc</span><span class="p">(</span><span class="n">label_name</span><span class="p">,</span> <span class="n">label</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">label</span><span class="o">.</span><span class="n">dtype</span><span class="p">)]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">batch_size</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">__del__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">check_call</span><span class="p">(</span><span class="n">_LIB</span><span class="o">.</span><span class="n">MXDataIterFree</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">handle</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">debug_skip_load</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c1"># Set the iterator to simply return always first batch. This can be used</span>
<span class="c1"># to test the speed of network without taking the loading delay into</span>
<span class="c1"># account.</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_debug_skip_load</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Set debug_skip_load to be true, will simply return first batch'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">reset</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_debug_at_begin</span> <span class="o">=</span> <span class="kc">True</span>
<span class="bp">self</span><span class="o">.</span><span class="n">first_batch</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">check_call</span><span class="p">(</span><span class="n">_LIB</span><span class="o">.</span><span class="n">MXDataIterBeforeFirst</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">handle</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">next</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_debug_skip_load</span> <span class="ow">and</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">_debug_at_begin</span><span class="p">:</span>
<span class="k">return</span> <span class="n">DataBatch</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">getdata</span><span class="p">()],</span> <span class="n">label</span><span class="o">=</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">getlabel</span><span class="p">()],</span> <span class="n">pad</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">getpad</span><span class="p">(),</span>
<span class="n">index</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">getindex</span><span class="p">())</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">first_batch</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">batch</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">first_batch</span>
<span class="bp">self</span><span class="o">.</span><span class="n">first_batch</span> <span class="o">=</span> <span class="kc">None</span>
<span class="k">return</span> <span class="n">batch</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_debug_at_begin</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">next_res</span> <span class="o">=</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">c_int</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="n">check_call</span><span class="p">(</span><span class="n">_LIB</span><span class="o">.</span><span class="n">MXDataIterNext</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">handle</span><span class="p">,</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">byref</span><span class="p">(</span><span class="n">next_res</span><span class="p">)))</span>
<span class="k">if</span> <span class="n">next_res</span><span class="o">.</span><span class="n">value</span><span class="p">:</span>
<span class="k">return</span> <span class="n">DataBatch</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">getdata</span><span class="p">()],</span> <span class="n">label</span><span class="o">=</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">getlabel</span><span class="p">()],</span> <span class="n">pad</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">getpad</span><span class="p">(),</span>
<span class="n">index</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">getindex</span><span class="p">())</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">StopIteration</span>
<span class="k">def</span> <span class="nf">iter_next</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">first_batch</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">True</span>
<span class="n">next_res</span> <span class="o">=</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">c_int</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="n">check_call</span><span class="p">(</span><span class="n">_LIB</span><span class="o">.</span><span class="n">MXDataIterNext</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">handle</span><span class="p">,</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">byref</span><span class="p">(</span><span class="n">next_res</span><span class="p">)))</span>
<span class="k">return</span> <span class="n">next_res</span><span class="o">.</span><span class="n">value</span>
<span class="k">def</span> <span class="nf">getdata</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">hdl</span> <span class="o">=</span> <span class="n">NDArrayHandle</span><span class="p">()</span>
<span class="n">check_call</span><span class="p">(</span><span class="n">_LIB</span><span class="o">.</span><span class="n">MXDataIterGetData</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">handle</span><span class="p">,</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">byref</span><span class="p">(</span><span class="n">hdl</span><span class="p">)))</span>
<span class="k">return</span> <span class="n">_ndarray_cls</span><span class="p">(</span><span class="n">hdl</span><span class="p">,</span> <span class="kc">False</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">getlabel</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">hdl</span> <span class="o">=</span> <span class="n">NDArrayHandle</span><span class="p">()</span>
<span class="n">check_call</span><span class="p">(</span><span class="n">_LIB</span><span class="o">.</span><span class="n">MXDataIterGetLabel</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">handle</span><span class="p">,</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">byref</span><span class="p">(</span><span class="n">hdl</span><span class="p">)))</span>
<span class="k">return</span> <span class="n">_ndarray_cls</span><span class="p">(</span><span class="n">hdl</span><span class="p">,</span> <span class="kc">False</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">getindex</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">index_size</span> <span class="o">=</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">c_uint64</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="n">index_data</span> <span class="o">=</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">POINTER</span><span class="p">(</span><span class="n">ctypes</span><span class="o">.</span><span class="n">c_uint64</span><span class="p">)()</span>
<span class="n">check_call</span><span class="p">(</span><span class="n">_LIB</span><span class="o">.</span><span class="n">MXDataIterGetIndex</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">handle</span><span class="p">,</span>
<span class="n">ctypes</span><span class="o">.</span><span class="n">byref</span><span class="p">(</span><span class="n">index_data</span><span class="p">),</span>
<span class="n">ctypes</span><span class="o">.</span><span class="n">byref</span><span class="p">(</span><span class="n">index_size</span><span class="p">)))</span>
<span class="k">if</span> <span class="n">index_size</span><span class="o">.</span><span class="n">value</span><span class="p">:</span>
<span class="n">address</span> <span class="o">=</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">addressof</span><span class="p">(</span><span class="n">index_data</span><span class="o">.</span><span class="n">contents</span><span class="p">)</span>
<span class="n">dbuffer</span> <span class="o">=</span> <span class="p">(</span><span class="n">ctypes</span><span class="o">.</span><span class="n">c_uint64</span><span class="o">*</span> <span class="n">index_size</span><span class="o">.</span><span class="n">value</span><span class="p">)</span><span class="o">.</span><span class="n">from_address</span><span class="p">(</span><span class="n">address</span><span class="p">)</span>
<span class="n">np_index</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">frombuffer</span><span class="p">(</span><span class="n">dbuffer</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">uint64</span><span class="p">)</span>
<span class="k">return</span> <span class="n">np_index</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">None</span>
<span class="k">def</span> <span class="nf">getpad</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">pad</span> <span class="o">=</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">c_int</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="n">check_call</span><span class="p">(</span><span class="n">_LIB</span><span class="o">.</span><span class="n">MXDataIterGetPadNum</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">handle</span><span class="p">,</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">byref</span><span class="p">(</span><span class="n">pad</span><span class="p">)))</span>
<span class="k">return</span> <span class="n">pad</span><span class="o">.</span><span class="n">value</span></div>
<span class="k">def</span> <span class="nf">_make_io_iterator</span><span class="p">(</span><span class="n">handle</span><span class="p">):</span>
<span class="sd">"""Create an io iterator by handle."""</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">c_char_p</span><span class="p">()</span>
<span class="n">desc</span> <span class="o">=</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">c_char_p</span><span class="p">()</span>
<span class="n">num_args</span> <span class="o">=</span> <span class="n">mx_uint</span><span class="p">()</span>
<span class="n">arg_names</span> <span class="o">=</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">POINTER</span><span class="p">(</span><span class="n">ctypes</span><span class="o">.</span><span class="n">c_char_p</span><span class="p">)()</span>
<span class="n">arg_types</span> <span class="o">=</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">POINTER</span><span class="p">(</span><span class="n">ctypes</span><span class="o">.</span><span class="n">c_char_p</span><span class="p">)()</span>
<span class="n">arg_descs</span> <span class="o">=</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">POINTER</span><span class="p">(</span><span class="n">ctypes</span><span class="o">.</span><span class="n">c_char_p</span><span class="p">)()</span>
<span class="n">check_call</span><span class="p">(</span><span class="n">_LIB</span><span class="o">.</span><span class="n">MXDataIterGetIterInfo</span><span class="p">(</span> \
<span class="n">handle</span><span class="p">,</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">byref</span><span class="p">(</span><span class="n">name</span><span class="p">),</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">byref</span><span class="p">(</span><span class="n">desc</span><span class="p">),</span> \
<span class="n">ctypes</span><span class="o">.</span><span class="n">byref</span><span class="p">(</span><span class="n">num_args</span><span class="p">),</span> \
<span class="n">ctypes</span><span class="o">.</span><span class="n">byref</span><span class="p">(</span><span class="n">arg_names</span><span class="p">),</span> \
<span class="n">ctypes</span><span class="o">.</span><span class="n">byref</span><span class="p">(</span><span class="n">arg_types</span><span class="p">),</span> \
<span class="n">ctypes</span><span class="o">.</span><span class="n">byref</span><span class="p">(</span><span class="n">arg_descs</span><span class="p">)))</span>
<span class="n">iter_name</span> <span class="o">=</span> <span class="n">py_str</span><span class="p">(</span><span class="n">name</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
<span class="n">narg</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">num_args</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
<span class="n">param_str</span> <span class="o">=</span> <span class="n">_build_param_doc</span><span class="p">(</span>
<span class="p">[</span><span class="n">py_str</span><span class="p">(</span><span class="n">arg_names</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">narg</span><span class="p">)],</span>
<span class="p">[</span><span class="n">py_str</span><span class="p">(</span><span class="n">arg_types</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">narg</span><span class="p">)],</span>
<span class="p">[</span><span class="n">py_str</span><span class="p">(</span><span class="n">arg_descs</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">narg</span><span class="p">)])</span>
<span class="n">doc_str</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'</span><span class="si">%s</span><span class="se">\n\n</span><span class="s1">'</span> <span class="o">+</span>
<span class="s1">'</span><span class="si">%s</span><span class="se">\n</span><span class="s1">'</span> <span class="o">+</span>
<span class="s1">'Returns</span><span class="se">\n</span><span class="s1">'</span> <span class="o">+</span>
<span class="s1">'-------</span><span class="se">\n</span><span class="s1">'</span> <span class="o">+</span>
<span class="s1">'MXDataIter</span><span class="se">\n</span><span class="s1">'</span><span class="o">+</span>
<span class="s1">' The result iterator.'</span><span class="p">)</span>
<span class="n">doc_str</span> <span class="o">=</span> <span class="n">doc_str</span> <span class="o">%</span> <span class="p">(</span><span class="n">desc</span><span class="o">.</span><span class="n">value</span><span class="p">,</span> <span class="n">param_str</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">creator</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="sd">"""Create an iterator.</span>
<span class="sd"> The parameters listed below can be passed in as keyword arguments.</span>
<span class="sd"> Parameters</span>
<span class="sd"> ----------</span>
<span class="sd"> name : string, required.</span>
<span class="sd"> Name of the resulting data iterator.</span>
<span class="sd"> Returns</span>
<span class="sd"> -------</span>
<span class="sd"> dataiter: Dataiter</span>
<span class="sd"> The resulting data iterator.</span>
<span class="sd"> """</span>
<span class="n">param_keys</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">param_vals</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">val</span> <span class="ow">in</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="n">param_keys</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">k</span><span class="p">)</span>
<span class="n">param_vals</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">val</span><span class="p">))</span>
<span class="c1"># create atomic symbol</span>
<span class="n">param_keys</span> <span class="o">=</span> <span class="n">c_str_array</span><span class="p">(</span><span class="n">param_keys</span><span class="p">)</span>
<span class="n">param_vals</span> <span class="o">=</span> <span class="n">c_str_array</span><span class="p">(</span><span class="n">param_vals</span><span class="p">)</span>
<span class="n">iter_handle</span> <span class="o">=</span> <span class="n">DataIterHandle</span><span class="p">()</span>
<span class="n">check_call</span><span class="p">(</span><span class="n">_LIB</span><span class="o">.</span><span class="n">MXDataIterCreateIter</span><span class="p">(</span>
<span class="n">handle</span><span class="p">,</span>
<span class="n">mx_uint</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">param_keys</span><span class="p">)),</span>
<span class="n">param_keys</span><span class="p">,</span> <span class="n">param_vals</span><span class="p">,</span>
<span class="n">ctypes</span><span class="o">.</span><span class="n">byref</span><span class="p">(</span><span class="n">iter_handle</span><span class="p">)))</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">args</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">TypeError</span><span class="p">(</span><span class="s1">'</span><span class="si">%s</span><span class="s1"> can only accept keyword arguments'</span> <span class="o">%</span> <span class="n">iter_name</span><span class="p">)</span>
<span class="k">return</span> <span class="n">MXDataIter</span><span class="p">(</span><span class="n">iter_handle</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="n">creator</span><span class="o">.</span><span class="vm">__name__</span> <span class="o">=</span> <span class="n">iter_name</span>
<span class="n">creator</span><span class="o">.</span><span class="vm">__doc__</span> <span class="o">=</span> <span class="n">doc_str</span>
<span class="k">return</span> <span class="n">creator</span>
<span class="k">def</span> <span class="nf">_init_io_module</span><span class="p">():</span>
<span class="sd">"""List and add all the data iterators to current module."""</span>
<span class="n">plist</span> <span class="o">=</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">POINTER</span><span class="p">(</span><span class="n">ctypes</span><span class="o">.</span><span class="n">c_void_p</span><span class="p">)()</span>
<span class="n">size</span> <span class="o">=</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">c_uint</span><span class="p">()</span>
<span class="n">check_call</span><span class="p">(</span><span class="n">_LIB</span><span class="o">.</span><span class="n">MXListDataIters</span><span class="p">(</span><span class="n">ctypes</span><span class="o">.</span><span class="n">byref</span><span class="p">(</span><span class="n">size</span><span class="p">),</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">byref</span><span class="p">(</span><span class="n">plist</span><span class="p">)))</span>
<span class="n">module_obj</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">modules</span><span class="p">[</span><span class="vm">__name__</span><span class="p">]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">size</span><span class="o">.</span><span class="n">value</span><span class="p">):</span>
<span class="n">hdl</span> <span class="o">=</span> <span class="n">ctypes</span><span class="o">.</span><span class="n">c_void_p</span><span class="p">(</span><span class="n">plist</span><span class="p">[</span><span class="n">i</span><span class="p">])</span>
<span class="n">dataiter</span> <span class="o">=</span> <span class="n">_make_io_iterator</span><span class="p">(</span><span class="n">hdl</span><span class="p">)</span>
<span class="nb">setattr</span><span class="p">(</span><span class="n">module_obj</span><span class="p">,</span> <span class="n">dataiter</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span> <span class="n">dataiter</span><span class="p">)</span>
<span class="n">_init_io_module</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div aria-label="main navigation" class="sphinxsidebar rightsidebar" role="navigation">
<div class="sphinxsidebarwrapper">
</div>
</div>
</div><div class="footer">
<div class="section-disclaimer">
<div class="container">
<div>
<img height="60" src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/apache_incubator_logo.png"/>
<p>
Apache MXNet is an effort undergoing incubation at The Apache Software Foundation (ASF), <strong>sponsored by the <i>Apache Incubator</i></strong>. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
</p>
<p>
"Copyright © 2017-2018, The Apache Software Foundation
Apache MXNet, MXNet, Apache, the Apache feather, and the Apache MXNet project logo are either registered trademarks or trademarks of the Apache Software Foundation."
</p>
</div>
</div>
</div>
</div> <!-- pagename != index -->
</div>
<script crossorigin="anonymous" integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS" src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"></script>
<script src="../../_static/js/sidebar.js" type="text/javascript"></script>
<script src="../../_static/js/search.js" type="text/javascript"></script>
<script src="../../_static/js/navbar.js" type="text/javascript"></script>
<script src="../../_static/js/clipboard.min.js" type="text/javascript"></script>
<script src="../../_static/js/copycode.js" type="text/javascript"></script>
<script src="../../_static/js/page.js" type="text/javascript"></script>
<script src="../../_static/js/docversion.js" type="text/javascript"></script>
<script type="text/javascript">
$('body').ready(function () {
$('body').css('visibility', 'visible');
});
</script>
</body>
</html>