| <!DOCTYPE html> |
| |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"/> |
| <meta content="IE=edge" http-equiv="X-UA-Compatible"/> |
| <meta content="width=device-width, initial-scale=1" name="viewport"/> |
| <meta content="mxnet.gluon.data.vision.datasets" property="og:title"> |
| <meta content="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/og-logo.png" property="og:image"> |
| <meta content="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/og-logo.png" property="og:image:secure_url"> |
| <meta content="mxnet.gluon.data.vision.datasets" property="og:description"/> |
| <title>mxnet.gluon.data.vision.datasets — mxnet documentation</title> |
| <link crossorigin="anonymous" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css" integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7" rel="stylesheet"/> |
| <link href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css" rel="stylesheet"/> |
| <link href="../../../../../_static/basic.css" rel="stylesheet" type="text/css"> |
| <link href="../../../../../_static/pygments.css" rel="stylesheet" type="text/css"> |
| <link href="../../../../../_static/mxnet.css" rel="stylesheet" type="text/css"/> |
| <script type="text/javascript"> |
| var DOCUMENTATION_OPTIONS = { |
| URL_ROOT: '../../../../../', |
| VERSION: '', |
| COLLAPSE_INDEX: false, |
| FILE_SUFFIX: '.html', |
| HAS_SOURCE: true, |
| SOURCELINK_SUFFIX: '.txt' |
| }; |
| </script> |
| <script src="https://code.jquery.com/jquery-1.11.1.min.js" type="text/javascript"></script> |
| <script src="../../../../../_static/underscore.js" type="text/javascript"></script> |
| <script src="../../../../../_static/searchtools_custom.js" type="text/javascript"></script> |
| <script src="../../../../../_static/doctools.js" type="text/javascript"></script> |
| <script src="../../../../../_static/selectlang.js" type="text/javascript"></script> |
| <script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script> |
| <script type="text/javascript"> jQuery(function() { Search.loadIndex("/searchindex.js"); Search.init();}); </script> |
| <script> |
| (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ |
| (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new |
| Date();a=s.createElement(o), |
| m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) |
| })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); |
| |
| ga('create', 'UA-96378503-1', 'auto'); |
| ga('send', 'pageview'); |
| |
| </script> |
| <!-- --> |
| <!-- <script type="text/javascript" src="../../../../../_static/jquery.js"></script> --> |
| <!-- --> |
| <!-- <script type="text/javascript" src="../../../../../_static/underscore.js"></script> --> |
| <!-- --> |
| <!-- <script type="text/javascript" src="../../../../../_static/doctools.js"></script> --> |
| <!-- --> |
| <!-- <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script> --> |
| <!-- --> |
| <link href="../../../../../genindex.html" rel="index" title="Index"> |
| <link href="../../../../../search.html" rel="search" title="Search"/> |
| <link href="../../../../index.html" rel="up" title="Module code"/> |
| <link href="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet-icon.png" rel="icon" type="image/png"/> |
| </link></link></link></meta></meta></meta></head> |
| <body background="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet-background-compressed.jpeg" role="document"> |
| <div class="content-block"><div class="navbar navbar-fixed-top"> |
| <div class="container" id="navContainer"> |
| <div class="innder" id="header-inner"> |
| <h1 id="logo-wrap"> |
| <a href="../../../../../" id="logo"><img src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet_logo.png"/></a> |
| </h1> |
| <nav class="nav-bar" id="main-nav"> |
| <a class="main-nav-link" href="../../../../../install/index.html">Install</a> |
| <span id="dropdown-menu-position-anchor"> |
| <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Gluon <span class="caret"></span></a> |
| <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu"> |
| <li><a class="main-nav-link" href="../../../../../gluon/index.html">About</a></li> |
| <li><a class="main-nav-link" href="http://gluon.mxnet.io">Tutorials</a></li> |
| </ul> |
| </span> |
| <span id="dropdown-menu-position-anchor"> |
| <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">API <span class="caret"></span></a> |
| <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu"> |
| <li><a class="main-nav-link" href="../../../../../api/python/index.html">Python</a></li> |
| <li><a class="main-nav-link" href="../../../../../api/c++/index.html">C++</a></li> |
| <li><a class="main-nav-link" href="../../../../../api/clojure/index.html">Clojure</a></li> |
| <li><a class="main-nav-link" href="../../../../../api/julia/index.html">Julia</a></li> |
| <li><a class="main-nav-link" href="../../../../../api/perl/index.html">Perl</a></li> |
| <li><a class="main-nav-link" href="../../../../../api/r/index.html">R</a></li> |
| <li><a class="main-nav-link" href="../../../../../api/scala/index.html">Scala</a></li> |
| </ul> |
| </span> |
| <span id="dropdown-menu-position-anchor-docs"> |
| <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Docs <span class="caret"></span></a> |
| <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu-docs"> |
| <li><a class="main-nav-link" href="../../../../../faq/index.html">FAQ</a></li> |
| <li><a class="main-nav-link" href="../../../../../tutorials/index.html">Tutorials</a> |
| <li><a class="main-nav-link" href="https://github.com/apache/incubator-mxnet/tree/0.11.0/example">Examples</a></li> |
| <li><a class="main-nav-link" href="../../../../../architecture/index.html">Architecture</a></li> |
| <li><a class="main-nav-link" href="../../../../../api/python/gluon/model_zoo.html">Model Zoo</a></li> |
| <li><a class="main-nav-link" href="../../../../../api/python/contrib/onnx.html">ONNX</a></li> |
| </li></ul> |
| </span> |
| <span id="dropdown-menu-position-anchor-community"> |
| <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Community <span class="caret"></span></a> |
| <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu-community"> |
| <li><a class="main-nav-link" href="http://discuss.mxnet.io">Forum</a></li> |
| <li><a class="main-nav-link" href="https://github.com/apache/incubator-mxnet">Github</a></li> |
| <li><a class="main-nav-link" href="../../../../../community/contribute.html">Contribute</a></li> |
| <li><a class="main-nav-link" href="../../../../../community/ecosystem.html">Ecosystem</a></li> |
| <li><a class="main-nav-link" href="../../../../../community/powered_by.html">Powered By</a></li> |
| </ul> |
| </span> |
| <span id="dropdown-menu-position-anchor-version" style="position: relative"><a href="#" class="main-nav-link dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="true">Versions(0.11.0)<span class="caret"></span></a><ul id="package-dropdown-menu" class="dropdown-menu"><li><a class="main-nav-link" href=http://mxnet.incubator.apache.org/>master</a></li><li><a class="main-nav-link" href=http://mxnet.incubator.apache.org/versions/1.2.1/index.html>1.2.1</a></li><li><a class="main-nav-link" href=http://mxnet.incubator.apache.org/versions/1.1.0/index.html>1.1.0</a></li><li><a class="main-nav-link" href=http://mxnet.incubator.apache.org/versions/1.0.0/index.html>1.0.0</a></li><li><a class="main-nav-link" href=http://mxnet.incubator.apache.org/versions/0.12.1/index.html>0.12.1</a></li><li><a class="main-nav-link" href=http://mxnet.incubator.apache.org/versions/0.11.0/index.html>0.11.0</a></li></ul></span></nav> |
| <script> function getRootPath(){ return "../../../../../" } </script> |
| <div class="burgerIcon dropdown"> |
| <a class="dropdown-toggle" data-toggle="dropdown" href="#" role="button">☰</a> |
| <ul class="dropdown-menu" id="burgerMenu"> |
| <li><a href="../../../../../install/index.html">Install</a></li> |
| <li><a class="main-nav-link" href="../../../../../tutorials/index.html">Tutorials</a></li> |
| <li class="dropdown-submenu dropdown"> |
| <a aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" role="button" tabindex="-1">Community</a> |
| <ul class="dropdown-menu"> |
| <li><a href="http://discuss.mxnet.io" tabindex="-1">Forum</a></li> |
| <li><a href="https://github.com/apache/incubator-mxnet" tabindex="-1">Github</a></li> |
| <li><a href="../../../../../community/contribute.html" tabindex="-1">Contribute</a></li> |
| <li><a href="../../../../../community/ecosystem.html" tabindex="-1">Ecosystem</a></li> |
| <li><a href="../../../../../community/powered_by.html" tabindex="-1">Powered By</a></li> |
| </ul> |
| </li> |
| <li class="dropdown-submenu"> |
| <a aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" role="button" tabindex="-1">API</a> |
| <ul class="dropdown-menu"> |
| <li><a href="../../../../../api/python/index.html" tabindex="-1">Python</a> |
| </li> |
| <li><a href="../../../../../api/c++/index.html" tabindex="-1">C++</a> |
| </li> |
| <li><a href="../../../../../api/clojure/index.html" tabindex="-1">Clojure</a> |
| </li> |
| <li><a href="../../../../../api/julia/index.html" tabindex="-1">Julia</a> |
| </li> |
| <li><a href="../../../../../api/perl/index.html" tabindex="-1">Perl</a> |
| </li> |
| <li><a href="../../../../../api/r/index.html" tabindex="-1">R</a> |
| </li> |
| <li><a href="../../../../../api/scala/index.html" tabindex="-1">Scala</a> |
| </li> |
| </ul> |
| </li> |
| <li class="dropdown-submenu"> |
| <a aria-expanded="true" aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" tabindex="-1">Docs</a> |
| <ul class="dropdown-menu"> |
| <li><a href="../../../../../tutorials/index.html" tabindex="-1">Tutorials</a></li> |
| <li><a href="../../../../../faq/index.html" tabindex="-1">FAQ</a></li> |
| <li><a href="../../../../../architecture/index.html" tabindex="-1">Architecture</a></li> |
| <li><a href="https://github.com/apache/incubator-mxnet/tree/0.11.0/example" tabindex="-1">Examples</a></li> |
| <li><a href="../../../../../api/python/gluon/model_zoo.html" tabindex="-1">Gluon Model Zoo</a></li> |
| </ul> |
| </li> |
| <li><a class="main-nav-link" href="https://github.com/dmlc/mxnet">Github</a></li> |
| <li id="dropdown-menu-position-anchor-version-mobile" class="dropdown-submenu" style="position: relative"><a href="#" tabindex="-1">Versions(0.11.0)</a><ul class="dropdown-menu"><li><a tabindex="-1" href=http://mxnet.incubator.apache.org/>master</a></li><li><a tabindex="-1" href=http://mxnet.incubator.apache.org/versions/1.2.1/index.html>1.2.1</a></li><li><a tabindex="-1" href=http://mxnet.incubator.apache.org/versions/1.1.0/index.html>1.1.0</a></li><li><a tabindex="-1" href=http://mxnet.incubator.apache.org/versions/1.0.0/index.html>1.0.0</a></li><li><a tabindex="-1" href=http://mxnet.incubator.apache.org/versions/0.12.1/index.html>0.12.1</a></li><li><a tabindex="-1" href=http://mxnet.incubator.apache.org/versions/0.11.0/index.html>0.11.0</a></li></ul></li></ul> |
| </div> |
| <div class="plusIcon dropdown"> |
| <a class="dropdown-toggle" data-toggle="dropdown" href="#" role="button"><span aria-hidden="true" class="glyphicon glyphicon-plus"></span></a> |
| <ul class="dropdown-menu dropdown-menu-right" id="plusMenu"></ul> |
| </div> |
| <div id="search-input-wrap"> |
| <form action="../../../../../search.html" autocomplete="off" class="" method="get" role="search"> |
| <div class="form-group inner-addon left-addon"> |
| <i class="glyphicon glyphicon-search"></i> |
| <input class="form-control" name="q" placeholder="Search" type="text"/> |
| </div> |
| <input name="check_keywords" type="hidden" value="yes"> |
| <input name="area" type="hidden" value="default"/> |
| </input></form> |
| <div id="search-preview"></div> |
| </div> |
| <div id="searchIcon"> |
| <span aria-hidden="true" class="glyphicon glyphicon-search"></span> |
| </div> |
| <!-- <div id="lang-select-wrap"> --> |
| <!-- <label id="lang-select-label"> --> |
| <!-- <\!-- <i class="fa fa-globe"></i> -\-> --> |
| <!-- <span></span> --> |
| <!-- </label> --> |
| <!-- <select id="lang-select"> --> |
| <!-- <option value="en">Eng</option> --> |
| <!-- <option value="zh">中文</option> --> |
| <!-- </select> --> |
| <!-- </div> --> |
| <!-- <a id="mobile-nav-toggle"> |
| <span class="mobile-nav-toggle-bar"></span> |
| <span class="mobile-nav-toggle-bar"></span> |
| <span class="mobile-nav-toggle-bar"></span> |
| </a> --> |
| </div> |
| </div> |
| </div> |
| <script type="text/javascript"> |
| $('body').css('background', 'white'); |
| </script> |
| <div class="container"> |
| <div class="row"> |
| <div aria-label="main navigation" class="sphinxsidebar leftsidebar" role="navigation"> |
| <div class="sphinxsidebarwrapper"> |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../../../../../api/python/index.html">Python Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../../../../api/r/index.html">R Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../../../../api/julia/index.html">Julia Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../../../../api/c++/index.html">C++ Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../../../../api/scala/index.html">Scala Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../../../../api/perl/index.html">Perl Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../../../../faq/index.html">HowTo Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../../../../architecture/index.html">System Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../../../../tutorials/index.html">Tutorials</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../../../../community/contribute.html">Community</a></li> |
| </ul> |
| </div> |
| </div> |
| <div class="content"> |
| <div class="page-tracker"></div> |
| <h1>Source code for mxnet.gluon.data.vision.datasets</h1><div class="highlight"><pre> |
| <span></span><span class="c1"># Licensed to the Apache Software Foundation (ASF) under one</span> |
| <span class="c1"># or more contributor license agreements. See the NOTICE file</span> |
| <span class="c1"># distributed with this work for additional information</span> |
| <span class="c1"># regarding copyright ownership. The ASF licenses this file</span> |
| <span class="c1"># to you under the Apache License, Version 2.0 (the</span> |
| <span class="c1"># "License"); you may not use this file except in compliance</span> |
| <span class="c1"># with the License. You may obtain a copy of the License at</span> |
| <span class="c1">#</span> |
| <span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span> |
| <span class="c1">#</span> |
| <span class="c1"># Unless required by applicable law or agreed to in writing,</span> |
| <span class="c1"># software distributed under the License is distributed on an</span> |
| <span class="c1"># "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY</span> |
| <span class="c1"># KIND, either express or implied. See the License for the</span> |
| <span class="c1"># specific language governing permissions and limitations</span> |
| <span class="c1"># under the License.</span> |
| |
| <span class="c1"># coding: utf-8</span> |
| <span class="c1"># pylint: disable=</span> |
| <span class="sd">"""Dataset container."""</span> |
| <span class="n">__all__</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'MNIST'</span><span class="p">,</span> <span class="s1">'FashionMNIST'</span><span class="p">,</span> <span class="s1">'CIFAR10'</span><span class="p">,</span> <span class="s1">'CIFAR100'</span><span class="p">,</span> |
| <span class="s1">'ImageRecordDataset'</span><span class="p">,</span> <span class="s1">'ImageFolderDataset'</span><span class="p">]</span> |
| |
| <span class="kn">import</span> <span class="nn">os</span> |
| <span class="kn">import</span> <span class="nn">gzip</span> |
| <span class="kn">import</span> <span class="nn">tarfile</span> |
| <span class="kn">import</span> <span class="nn">struct</span> |
| <span class="kn">import</span> <span class="nn">warnings</span> |
| <span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span> |
| |
| <span class="kn">from</span> <span class="nn">..</span> <span class="k">import</span> <span class="n">dataset</span> |
| <span class="kn">from</span> <span class="nn">...utils</span> <span class="k">import</span> <span class="n">download</span><span class="p">,</span> <span class="n">check_sha1</span><span class="p">,</span> <span class="n">_get_repo_file_url</span> |
| <span class="kn">from</span> <span class="nn">....</span> <span class="k">import</span> <span class="n">nd</span><span class="p">,</span> <span class="n">image</span><span class="p">,</span> <span class="n">recordio</span><span class="p">,</span> <span class="n">base</span> |
| |
| |
| <div class="viewcode-block" id="MNIST"><a class="viewcode-back" href="../../../../../api/python/gluon/data.html#mxnet.gluon.data.vision.datasets.MNIST">[docs]</a><span class="k">class</span> <span class="nc">MNIST</span><span class="p">(</span><span class="n">dataset</span><span class="o">.</span><span class="n">_DownloadedDataset</span><span class="p">):</span> |
| <span class="sd">"""MNIST handwritten digits dataset from http://yann.lecun.com/exdb/mnist</span> |
| |
| <span class="sd"> Each sample is an image (in 3D NDArray) with shape (28, 28, 1).</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> root : str, default $MXNET_HOME/datasets/mnist</span> |
| <span class="sd"> Path to temp folder for storing data.</span> |
| <span class="sd"> train : bool, default True</span> |
| <span class="sd"> Whether to load the training or testing set.</span> |
| <span class="sd"> transform : function, default None</span> |
| <span class="sd"> A user defined callback that transforms each sample. For example:</span> |
| <span class="sd"> ::</span> |
| |
| <span class="sd"> transform=lambda data, label: (data.astype(np.float32)/255, label)</span> |
| |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">root</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">base</span><span class="o">.</span><span class="n">data_dir</span><span class="p">(),</span> <span class="s1">'datasets'</span><span class="p">,</span> <span class="s1">'mnist'</span><span class="p">),</span> |
| <span class="n">train</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_train</span> <span class="o">=</span> <span class="n">train</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_train_data</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'train-images-idx3-ubyte.gz'</span><span class="p">,</span> |
| <span class="s1">'6c95f4b05d2bf285e1bfb0e7960c31bd3b3f8a7d'</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_train_label</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'train-labels-idx1-ubyte.gz'</span><span class="p">,</span> |
| <span class="s1">'2a80914081dc54586dbdf242f9805a6b8d2a15fc'</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_test_data</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'t10k-images-idx3-ubyte.gz'</span><span class="p">,</span> |
| <span class="s1">'c3a25af1f52dad7f726cce8cacb138654b760d48'</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_test_label</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'t10k-labels-idx1-ubyte.gz'</span><span class="p">,</span> |
| <span class="s1">'763e7fa3757d93b0cdec073cef058b2004252c17'</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_namespace</span> <span class="o">=</span> <span class="s1">'mnist'</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">MNIST</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="n">transform</span><span class="p">)</span> |
| |
| <span class="k">def</span> <span class="nf">_get_data</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_train</span><span class="p">:</span> |
| <span class="n">data</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_train_data</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_train_label</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="n">data</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_test_data</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_test_label</span> |
| |
| <span class="n">namespace</span> <span class="o">=</span> <span class="s1">'gluon/dataset/'</span><span class="o">+</span><span class="bp">self</span><span class="o">.</span><span class="n">_namespace</span> |
| <span class="n">data_file</span> <span class="o">=</span> <span class="n">download</span><span class="p">(</span><span class="n">_get_repo_file_url</span><span class="p">(</span><span class="n">namespace</span><span class="p">,</span> <span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]),</span> |
| <span class="n">path</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_root</span><span class="p">,</span> |
| <span class="n">sha1_hash</span><span class="o">=</span><span class="n">data</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> |
| <span class="n">label_file</span> <span class="o">=</span> <span class="n">download</span><span class="p">(</span><span class="n">_get_repo_file_url</span><span class="p">(</span><span class="n">namespace</span><span class="p">,</span> <span class="n">label</span><span class="p">[</span><span class="mi">0</span><span class="p">]),</span> |
| <span class="n">path</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_root</span><span class="p">,</span> |
| <span class="n">sha1_hash</span><span class="o">=</span><span class="n">label</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> |
| |
| <span class="k">with</span> <span class="n">gzip</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">label_file</span><span class="p">,</span> <span class="s1">'rb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">fin</span><span class="p">:</span> |
| <span class="n">struct</span><span class="o">.</span><span class="n">unpack</span><span class="p">(</span><span class="s2">">II"</span><span class="p">,</span> <span class="n">fin</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="mi">8</span><span class="p">))</span> |
| <span class="n">label</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">frombuffer</span><span class="p">(</span><span class="n">fin</span><span class="o">.</span><span class="n">read</span><span class="p">(),</span> <span class="n">dtype</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">uint8</span><span class="p">)</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">int32</span><span class="p">)</span> |
| |
| <span class="k">with</span> <span class="n">gzip</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">data_file</span><span class="p">,</span> <span class="s1">'rb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">fin</span><span class="p">:</span> |
| <span class="n">struct</span><span class="o">.</span><span class="n">unpack</span><span class="p">(</span><span class="s2">">IIII"</span><span class="p">,</span> <span class="n">fin</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="mi">16</span><span class="p">))</span> |
| <span class="n">data</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">frombuffer</span><span class="p">(</span><span class="n">fin</span><span class="o">.</span><span class="n">read</span><span class="p">(),</span> <span class="n">dtype</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">uint8</span><span class="p">)</span> |
| <span class="n">data</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">label</span><span class="p">),</span> <span class="mi">28</span><span class="p">,</span> <span class="mi">28</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> |
| |
| <span class="bp">self</span><span class="o">.</span><span class="n">_data</span> <span class="o">=</span> <span class="n">nd</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">data</span><span class="o">.</span><span class="n">dtype</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_label</span> <span class="o">=</span> <span class="n">label</span></div> |
| |
| |
| <div class="viewcode-block" id="FashionMNIST"><a class="viewcode-back" href="../../../../../api/python/gluon/data.html#mxnet.gluon.data.vision.datasets.FashionMNIST">[docs]</a><span class="k">class</span> <span class="nc">FashionMNIST</span><span class="p">(</span><span class="n">MNIST</span><span class="p">):</span> |
| <span class="sd">"""A dataset of Zalando's article images consisting of fashion products,</span> |
| <span class="sd"> a drop-in replacement of the original MNIST dataset from</span> |
| <span class="sd"> https://github.com/zalandoresearch/fashion-mnist</span> |
| |
| <span class="sd"> Each sample is an image (in 3D NDArray) with shape (28, 28, 1).</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> root : str, default $MXNET_HOME/datasets/fashion-mnist'</span> |
| <span class="sd"> Path to temp folder for storing data.</span> |
| <span class="sd"> train : bool, default True</span> |
| <span class="sd"> Whether to load the training or testing set.</span> |
| <span class="sd"> transform : function, default None</span> |
| <span class="sd"> A user defined callback that transforms each sample. For example:</span> |
| <span class="sd"> ::</span> |
| |
| <span class="sd"> transform=lambda data, label: (data.astype(np.float32)/255, label)</span> |
| |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">root</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">base</span><span class="o">.</span><span class="n">data_dir</span><span class="p">(),</span> <span class="s1">'datasets'</span><span class="p">,</span> <span class="s1">'fashion-mnist'</span><span class="p">),</span> |
| <span class="n">train</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_train</span> <span class="o">=</span> <span class="n">train</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_train_data</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'train-images-idx3-ubyte.gz'</span><span class="p">,</span> |
| <span class="s1">'0cf37b0d40ed5169c6b3aba31069a9770ac9043d'</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_train_label</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'train-labels-idx1-ubyte.gz'</span><span class="p">,</span> |
| <span class="s1">'236021d52f1e40852b06a4c3008d8de8aef1e40b'</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_test_data</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'t10k-images-idx3-ubyte.gz'</span><span class="p">,</span> |
| <span class="s1">'626ed6a7c06dd17c0eec72fa3be1740f146a2863'</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_test_label</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'t10k-labels-idx1-ubyte.gz'</span><span class="p">,</span> |
| <span class="s1">'17f9ab60e7257a1620f4ad76bbbaf857c3920701'</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_namespace</span> <span class="o">=</span> <span class="s1">'fashion-mnist'</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">MNIST</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="n">transform</span><span class="p">)</span> <span class="c1"># pylint: disable=bad-super-call</span></div> |
| |
| |
| <div class="viewcode-block" id="CIFAR10"><a class="viewcode-back" href="../../../../../api/python/gluon/data.html#mxnet.gluon.data.vision.datasets.CIFAR10">[docs]</a><span class="k">class</span> <span class="nc">CIFAR10</span><span class="p">(</span><span class="n">dataset</span><span class="o">.</span><span class="n">_DownloadedDataset</span><span class="p">):</span> |
| <span class="sd">"""CIFAR10 image classification dataset from https://www.cs.toronto.edu/~kriz/cifar.html</span> |
| |
| <span class="sd"> Each sample is an image (in 3D NDArray) with shape (32, 32, 1).</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> root : str, default $MXNET_HOME/datasets/cifar10</span> |
| <span class="sd"> Path to temp folder for storing data.</span> |
| <span class="sd"> train : bool, default True</span> |
| <span class="sd"> Whether to load the training or testing set.</span> |
| <span class="sd"> transform : function, default None</span> |
| <span class="sd"> A user defined callback that transforms each sample. For example:</span> |
| <span class="sd"> ::</span> |
| |
| <span class="sd"> transform=lambda data, label: (data.astype(np.float32)/255, label)</span> |
| |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">root</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">base</span><span class="o">.</span><span class="n">data_dir</span><span class="p">(),</span> <span class="s1">'datasets'</span><span class="p">,</span> <span class="s1">'cifar10'</span><span class="p">),</span> |
| <span class="n">train</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_train</span> <span class="o">=</span> <span class="n">train</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_archive_file</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'cifar-10-binary.tar.gz'</span><span class="p">,</span> <span class="s1">'fab780a1e191a7eda0f345501ccd62d20f7ed891'</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_train_data</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">'data_batch_1.bin'</span><span class="p">,</span> <span class="s1">'aadd24acce27caa71bf4b10992e9e7b2d74c2540'</span><span class="p">),</span> |
| <span class="p">(</span><span class="s1">'data_batch_2.bin'</span><span class="p">,</span> <span class="s1">'c0ba65cce70568cd57b4e03e9ac8d2a5367c1795'</span><span class="p">),</span> |
| <span class="p">(</span><span class="s1">'data_batch_3.bin'</span><span class="p">,</span> <span class="s1">'1dd00a74ab1d17a6e7d73e185b69dbf31242f295'</span><span class="p">),</span> |
| <span class="p">(</span><span class="s1">'data_batch_4.bin'</span><span class="p">,</span> <span class="s1">'aab85764eb3584312d3c7f65fd2fd016e36a258e'</span><span class="p">),</span> |
| <span class="p">(</span><span class="s1">'data_batch_5.bin'</span><span class="p">,</span> <span class="s1">'26e2849e66a845b7f1e4614ae70f4889ae604628'</span><span class="p">)]</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_test_data</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">'test_batch.bin'</span><span class="p">,</span> <span class="s1">'67eb016db431130d61cd03c7ad570b013799c88c'</span><span class="p">)]</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_namespace</span> <span class="o">=</span> <span class="s1">'cifar10'</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">CIFAR10</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="n">transform</span><span class="p">)</span> |
| |
| <span class="k">def</span> <span class="nf">_read_batch</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span> |
| <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">'rb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">fin</span><span class="p">:</span> |
| <span class="n">data</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">frombuffer</span><span class="p">(</span><span class="n">fin</span><span class="o">.</span><span class="n">read</span><span class="p">(),</span> <span class="n">dtype</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">uint8</span><span class="p">)</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3072</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span> |
| |
| <span class="k">return</span> <span class="n">data</span><span class="p">[:,</span> <span class="mi">1</span><span class="p">:]</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">32</span><span class="p">)</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span> \ |
| <span class="n">data</span><span class="p">[:,</span> <span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">int32</span><span class="p">)</span> |
| |
| <span class="k">def</span> <span class="nf">_get_data</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| <span class="k">if</span> <span class="nb">any</span><span class="p">(</span><span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">path</span><span class="p">)</span> <span class="ow">or</span> <span class="ow">not</span> <span class="n">check_sha1</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">sha1</span><span class="p">)</span> |
| <span class="k">for</span> <span class="n">path</span><span class="p">,</span> <span class="n">sha1</span> <span class="ow">in</span> <span class="p">((</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_root</span><span class="p">,</span> <span class="n">name</span><span class="p">),</span> <span class="n">sha1</span><span class="p">)</span> |
| <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">sha1</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_train_data</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">_test_data</span><span class="p">)):</span> |
| <span class="n">namespace</span> <span class="o">=</span> <span class="s1">'gluon/dataset/'</span><span class="o">+</span><span class="bp">self</span><span class="o">.</span><span class="n">_namespace</span> |
| <span class="n">filename</span> <span class="o">=</span> <span class="n">download</span><span class="p">(</span><span class="n">_get_repo_file_url</span><span class="p">(</span><span class="n">namespace</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_archive_file</span><span class="p">[</span><span class="mi">0</span><span class="p">]),</span> |
| <span class="n">path</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_root</span><span class="p">,</span> |
| <span class="n">sha1_hash</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_archive_file</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span> |
| |
| <span class="k">with</span> <span class="n">tarfile</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span> <span class="k">as</span> <span class="n">tar</span><span class="p">:</span> |
| <span class="n">tar</span><span class="o">.</span><span class="n">extractall</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_root</span><span class="p">)</span> |
| |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_train</span><span class="p">:</span> |
| <span class="n">data_files</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_train_data</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="n">data_files</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_test_data</span> |
| <span class="n">data</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="nb">zip</span><span class="p">(</span><span class="o">*</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_read_batch</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_root</span><span class="p">,</span> <span class="n">name</span><span class="p">))</span> |
| <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">_</span> <span class="ow">in</span> <span class="n">data_files</span><span class="p">))</span> |
| <span class="n">data</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">concatenate</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> |
| <span class="n">label</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">concatenate</span><span class="p">(</span><span class="n">label</span><span class="p">)</span> |
| |
| <span class="bp">self</span><span class="o">.</span><span class="n">_data</span> <span class="o">=</span> <span class="n">nd</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">data</span><span class="o">.</span><span class="n">dtype</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_label</span> <span class="o">=</span> <span class="n">label</span></div> |
| |
| |
| <div class="viewcode-block" id="CIFAR100"><a class="viewcode-back" href="../../../../../api/python/gluon/data.html#mxnet.gluon.data.vision.datasets.CIFAR100">[docs]</a><span class="k">class</span> <span class="nc">CIFAR100</span><span class="p">(</span><span class="n">CIFAR10</span><span class="p">):</span> |
| <span class="sd">"""CIFAR100 image classification dataset from https://www.cs.toronto.edu/~kriz/cifar.html</span> |
| |
| <span class="sd"> Each sample is an image (in 3D NDArray) with shape (32, 32, 1).</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> root : str, default $MXNET_HOME/datasets/cifar100</span> |
| <span class="sd"> Path to temp folder for storing data.</span> |
| <span class="sd"> fine_label : bool, default False</span> |
| <span class="sd"> Whether to load the fine-grained (100 classes) or coarse-grained (20 super-classes) labels.</span> |
| <span class="sd"> train : bool, default True</span> |
| <span class="sd"> Whether to load the training or testing set.</span> |
| <span class="sd"> transform : function, default None</span> |
| <span class="sd"> A user defined callback that transforms each sample. For example:</span> |
| <span class="sd"> ::</span> |
| |
| <span class="sd"> transform=lambda data, label: (data.astype(np.float32)/255, label)</span> |
| |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">root</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">base</span><span class="o">.</span><span class="n">data_dir</span><span class="p">(),</span> <span class="s1">'datasets'</span><span class="p">,</span> <span class="s1">'cifar100'</span><span class="p">),</span> |
| <span class="n">fine_label</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">train</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_train</span> <span class="o">=</span> <span class="n">train</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_archive_file</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'cifar-100-binary.tar.gz'</span><span class="p">,</span> <span class="s1">'a0bb982c76b83111308126cc779a992fa506b90b'</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_train_data</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">'train.bin'</span><span class="p">,</span> <span class="s1">'e207cd2e05b73b1393c74c7f5e7bea451d63e08e'</span><span class="p">)]</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_test_data</span> <span class="o">=</span> <span class="p">[(</span><span class="s1">'test.bin'</span><span class="p">,</span> <span class="s1">'8fb6623e830365ff53cf14adec797474f5478006'</span><span class="p">)]</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_fine_label</span> <span class="o">=</span> <span class="n">fine_label</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_namespace</span> <span class="o">=</span> <span class="s1">'cifar100'</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">CIFAR10</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="n">transform</span><span class="p">)</span> <span class="c1"># pylint: disable=bad-super-call</span> |
| |
| <span class="k">def</span> <span class="nf">_read_batch</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span> |
| <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">'rb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">fin</span><span class="p">:</span> |
| <span class="n">data</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">frombuffer</span><span class="p">(</span><span class="n">fin</span><span class="o">.</span><span class="n">read</span><span class="p">(),</span> <span class="n">dtype</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">uint8</span><span class="p">)</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3072</span><span class="o">+</span><span class="mi">2</span><span class="p">)</span> |
| |
| <span class="k">return</span> <span class="n">data</span><span class="p">[:,</span> <span class="mi">2</span><span class="p">:]</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">32</span><span class="p">,</span> <span class="mi">32</span><span class="p">)</span><span class="o">.</span><span class="n">transpose</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span> \ |
| <span class="n">data</span><span class="p">[:,</span> <span class="mi">0</span><span class="o">+</span><span class="bp">self</span><span class="o">.</span><span class="n">_fine_label</span><span class="p">]</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">int32</span><span class="p">)</span></div> |
| |
| |
| <div class="viewcode-block" id="ImageRecordDataset"><a class="viewcode-back" href="../../../../../api/python/gluon/data.html#mxnet.gluon.data.vision.datasets.ImageRecordDataset">[docs]</a><span class="k">class</span> <span class="nc">ImageRecordDataset</span><span class="p">(</span><span class="n">dataset</span><span class="o">.</span><span class="n">RecordFileDataset</span><span class="p">):</span> |
| <span class="sd">"""A dataset wrapping over a RecordIO file containing images.</span> |
| |
| <span class="sd"> Each sample is an image and its corresponding label.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> filename : str</span> |
| <span class="sd"> Path to rec file.</span> |
| <span class="sd"> flag : {0, 1}, default 1</span> |
| <span class="sd"> If 0, always convert images to greyscale.</span> |
| |
| <span class="sd"> If 1, always convert images to colored (RGB).</span> |
| <span class="sd"> transform : function, default None</span> |
| <span class="sd"> A user defined callback that transforms each sample. For example:</span> |
| <span class="sd"> ::</span> |
| |
| <span class="sd"> transform=lambda data, label: (data.astype(np.float32)/255, label)</span> |
| |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">flag</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">ImageRecordDataset</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_flag</span> <span class="o">=</span> <span class="n">flag</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_transform</span> <span class="o">=</span> <span class="n">transform</span> |
| |
| <span class="k">def</span> <span class="nf">__getitem__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">idx</span><span class="p">):</span> |
| <span class="n">record</span> <span class="o">=</span> <span class="nb">super</span><span class="p">(</span><span class="n">ImageRecordDataset</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__getitem__</span><span class="p">(</span><span class="n">idx</span><span class="p">)</span> |
| <span class="n">header</span><span class="p">,</span> <span class="n">img</span> <span class="o">=</span> <span class="n">recordio</span><span class="o">.</span><span class="n">unpack</span><span class="p">(</span><span class="n">record</span><span class="p">)</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_transform</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_transform</span><span class="p">(</span><span class="n">image</span><span class="o">.</span><span class="n">imdecode</span><span class="p">(</span><span class="n">img</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_flag</span><span class="p">),</span> <span class="n">header</span><span class="o">.</span><span class="n">label</span><span class="p">)</span> |
| <span class="k">return</span> <span class="n">image</span><span class="o">.</span><span class="n">imdecode</span><span class="p">(</span><span class="n">img</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_flag</span><span class="p">),</span> <span class="n">header</span><span class="o">.</span><span class="n">label</span></div> |
| |
| |
| <div class="viewcode-block" id="ImageFolderDataset"><a class="viewcode-back" href="../../../../../api/python/gluon/data.html#mxnet.gluon.data.vision.datasets.ImageFolderDataset">[docs]</a><span class="k">class</span> <span class="nc">ImageFolderDataset</span><span class="p">(</span><span class="n">dataset</span><span class="o">.</span><span class="n">Dataset</span><span class="p">):</span> |
| <span class="sd">"""A dataset for loading image files stored in a folder structure like::</span> |
| |
| <span class="sd"> root/car/0001.jpg</span> |
| <span class="sd"> root/car/xxxa.jpg</span> |
| <span class="sd"> root/car/yyyb.jpg</span> |
| <span class="sd"> root/bus/123.jpg</span> |
| <span class="sd"> root/bus/023.jpg</span> |
| <span class="sd"> root/bus/wwww.jpg</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> root : str</span> |
| <span class="sd"> Path to root directory.</span> |
| <span class="sd"> flag : {0, 1}, default 1</span> |
| <span class="sd"> If 0, always convert loaded images to greyscale (1 channel).</span> |
| <span class="sd"> If 1, always convert loaded images to colored (3 channels).</span> |
| <span class="sd"> transform : callable, default None</span> |
| <span class="sd"> A function that takes data and label and transforms them:</span> |
| <span class="sd"> ::</span> |
| |
| <span class="sd"> transform = lambda data, label: (data.astype(np.float32)/255, label)</span> |
| |
| <span class="sd"> Attributes</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> synsets : list</span> |
| <span class="sd"> List of class names. `synsets[i]` is the name for the integer label `i`</span> |
| <span class="sd"> items : list of tuples</span> |
| <span class="sd"> List of all images in (filename, label) pairs.</span> |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">root</span><span class="p">,</span> <span class="n">flag</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">transform</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_root</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">expanduser</span><span class="p">(</span><span class="n">root</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_flag</span> <span class="o">=</span> <span class="n">flag</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_transform</span> <span class="o">=</span> <span class="n">transform</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_exts</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'.jpg'</span><span class="p">,</span> <span class="s1">'.jpeg'</span><span class="p">,</span> <span class="s1">'.png'</span><span class="p">]</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_list_images</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_root</span><span class="p">)</span> |
| |
| <span class="k">def</span> <span class="nf">_list_images</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">root</span><span class="p">):</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">synsets</span> <span class="o">=</span> <span class="p">[]</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">items</span> <span class="o">=</span> <span class="p">[]</span> |
| |
| <span class="k">for</span> <span class="n">folder</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">listdir</span><span class="p">(</span><span class="n">root</span><span class="p">)):</span> |
| <span class="n">path</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="n">folder</span><span class="p">)</span> |
| <span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isdir</span><span class="p">(</span><span class="n">path</span><span class="p">):</span> |
| <span class="n">warnings</span><span class="o">.</span><span class="n">warn</span><span class="p">(</span><span class="s1">'Ignoring </span><span class="si">%s</span><span class="s1">, which is not a directory.'</span><span class="o">%</span><span class="n">path</span><span class="p">,</span> <span class="n">stacklevel</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span> |
| <span class="k">continue</span> |
| <span class="n">label</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">synsets</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">synsets</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">folder</span><span class="p">)</span> |
| <span class="k">for</span> <span class="n">filename</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">listdir</span><span class="p">(</span><span class="n">path</span><span class="p">)):</span> |
| <span class="n">filename</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">filename</span><span class="p">)</span> |
| <span class="n">ext</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">splitext</span><span class="p">(</span><span class="n">filename</span><span class="p">)[</span><span class="mi">1</span><span class="p">]</span> |
| <span class="k">if</span> <span class="n">ext</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_exts</span><span class="p">:</span> |
| <span class="n">warnings</span><span class="o">.</span><span class="n">warn</span><span class="p">(</span><span class="s1">'Ignoring </span><span class="si">%s</span><span class="s1"> of type </span><span class="si">%s</span><span class="s1">. Only support </span><span class="si">%s</span><span class="s1">'</span><span class="o">%</span><span class="p">(</span> |
| <span class="n">filename</span><span class="p">,</span> <span class="n">ext</span><span class="p">,</span> <span class="s1">', '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_exts</span><span class="p">)))</span> |
| <span class="k">continue</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">items</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">filename</span><span class="p">,</span> <span class="n">label</span><span class="p">))</span> |
| |
| <span class="k">def</span> <span class="nf">__getitem__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">idx</span><span class="p">):</span> |
| <span class="n">img</span> <span class="o">=</span> <span class="n">image</span><span class="o">.</span><span class="n">imread</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">items</span><span class="p">[</span><span class="n">idx</span><span class="p">][</span><span class="mi">0</span><span class="p">],</span> <span class="bp">self</span><span class="o">.</span><span class="n">_flag</span><span class="p">)</span> |
| <span class="n">label</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">items</span><span class="p">[</span><span class="n">idx</span><span class="p">][</span><span class="mi">1</span><span class="p">]</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_transform</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_transform</span><span class="p">(</span><span class="n">img</span><span class="p">,</span> <span class="n">label</span><span class="p">)</span> |
| <span class="k">return</span> <span class="n">img</span><span class="p">,</span> <span class="n">label</span> |
| |
| <span class="k">def</span> <span class="nf">__len__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">items</span><span class="p">)</span></div> |
| </pre></div> |
| </div> |
| </div> |
| <div aria-label="main navigation" class="sphinxsidebar rightsidebar" role="navigation"> |
| <div class="sphinxsidebarwrapper"> |
| </div> |
| </div> |
| </div><div class="footer"> |
| <div class="section-disclaimer"> |
| <div class="container"> |
| <div> |
| <img height="60" src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/apache_incubator_logo.png"/> |
| <p> |
| Apache MXNet is an effort undergoing incubation at The Apache Software Foundation (ASF), <strong>sponsored by the <i>Apache Incubator</i></strong>. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. |
| </p> |
| <p> |
| "Copyright © 2017-2018, The Apache Software Foundation |
| Apache MXNet, MXNet, Apache, the Apache feather, and the Apache MXNet project logo are either registered trademarks or trademarks of the Apache Software Foundation." |
| </p> |
| </div> |
| </div> |
| </div> |
| </div> <!-- pagename != index --> |
| </div> |
| <script crossorigin="anonymous" integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS" src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"></script> |
| <script src="../../../../../_static/js/sidebar.js" type="text/javascript"></script> |
| <script src="../../../../../_static/js/search.js" type="text/javascript"></script> |
| <script src="../../../../../_static/js/navbar.js" type="text/javascript"></script> |
| <script src="../../../../../_static/js/clipboard.min.js" type="text/javascript"></script> |
| <script src="../../../../../_static/js/copycode.js" type="text/javascript"></script> |
| <script src="../../../../../_static/js/page.js" type="text/javascript"></script> |
| <script type="text/javascript"> |
| $('body').ready(function () { |
| $('body').css('visibility', 'visible'); |
| }); |
| </script> |
| </body> |
| </html> |
