| <!DOCTYPE html> |
| |
| <html xmlns="http://www.w3.org/1999/xhtml"> |
| <head> |
| <meta charset="utf-8" /> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> |
| <meta http-equiv="x-ua-compatible" content="ie=edge"> |
| <style> |
| .dropdown { |
| position: relative; |
| display: inline-block; |
| } |
| |
| .dropdown-content { |
| display: none; |
| position: absolute; |
| background-color: #f9f9f9; |
| min-width: 160px; |
| box-shadow: 0px 8px 16px 0px rgba(0,0,0,0.2); |
| padding: 12px 16px; |
| z-index: 1; |
| text-align: left; |
| } |
| |
| .dropdown:hover .dropdown-content { |
| display: block; |
| } |
| |
| .dropdown-option:hover { |
| color: #FF4500; |
| } |
| |
| .dropdown-option-active { |
| color: #FF4500; |
| font-weight: lighter; |
| } |
| |
| .dropdown-option { |
| color: #000000; |
| font-weight: lighter; |
| } |
| |
| .dropdown-header { |
| color: #FFFFFF; |
| display: inline-flex; |
| } |
| |
| .dropdown-caret { |
| width: 18px; |
| } |
| |
| .dropdown-caret-path { |
| fill: #FFFFFF; |
| } |
| </style> |
| |
| <title>Sparse NDArrays with Gluon — Apache MXNet documentation</title> |
| |
| <link rel="stylesheet" href="../../../../_static/basic.css" type="text/css" /> |
| <link rel="stylesheet" href="../../../../_static/pygments.css" type="text/css" /> |
| <link rel="stylesheet" type="text/css" href="../../../../_static/mxnet.css" /> |
| <link rel="stylesheet" href="../../../../_static/material-design-lite-1.3.0/material.blue-deep_orange.min.css" type="text/css" /> |
| <link rel="stylesheet" href="../../../../_static/sphinx_materialdesign_theme.css" type="text/css" /> |
| <link rel="stylesheet" href="../../../../_static/fontawesome/all.css" type="text/css" /> |
| <link rel="stylesheet" href="../../../../_static/fonts.css" type="text/css" /> |
| <link rel="stylesheet" href="../../../../_static/feedback.css" type="text/css" /> |
| <script id="documentation_options" data-url_root="../../../../" src="../../../../_static/documentation_options.js"></script> |
| <script src="../../../../_static/jquery.js"></script> |
| <script src="../../../../_static/underscore.js"></script> |
| <script src="../../../../_static/doctools.js"></script> |
| <script src="../../../../_static/language_data.js"></script> |
| <script src="../../../../_static/matomo_analytics.js"></script> |
| <script src="../../../../_static/autodoc.js"></script> |
| <script crossorigin="anonymous" integrity="sha256-Ae2Vz/4ePdIu6ZyI/5ZGsYnb+m0JlOmKPjt6XZ9JJkA=" src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.4/require.min.js"></script> |
| <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/latest.js?config=TeX-AMS-MML_HTMLorMML"></script> |
| <script type="text/x-mathjax-config">MathJax.Hub.Config({"tex2jax": {"inlineMath": [["$", "$"], ["\\(", "\\)"]], "processEscapes": true, "ignoreClass": "document", "processClass": "math|output_area"}})</script> |
| <link rel="shortcut icon" href="../../../../_static/mxnet-icon.png"/> |
| <link rel="index" title="Index" href="../../../../genindex.html" /> |
| <link rel="search" title="Search" href="../../../../search.html" /> |
| <link rel="next" title="ONNX" href="../../onnx/index.html" /> |
| <link rel="prev" title="Train a Linear Regression Model with Sparse Symbols" href="train.html" /> |
| </head> |
| <body><header class="site-header" role="banner"> |
| <div class="wrapper"> |
| <a class="site-title" rel="author" href="/versions/1.9.1/"><img |
| src="../../../../_static/mxnet_logo.png" class="site-header-logo"></a> |
| <nav class="site-nav"> |
| <input type="checkbox" id="nav-trigger" class="nav-trigger"/> |
| <label for="nav-trigger"> |
| <span class="menu-icon"> |
| <svg viewBox="0 0 18 15" width="18px" height="15px"> |
| <path d="M18,1.484c0,0.82-0.665,1.484-1.484,1.484H1.484C0.665,2.969,0,2.304,0,1.484l0,0C0,0.665,0.665,0,1.484,0 h15.032C17.335,0,18,0.665,18,1.484L18,1.484z M18,7.516C18,8.335,17.335,9,16.516,9H1.484C0.665,9,0,8.335,0,7.516l0,0 c0-0.82,0.665-1.484,1.484-1.484h15.032C17.335,6.031,18,6.696,18,7.516L18,7.516z M18,13.516C18,14.335,17.335,15,16.516,15H1.484 C0.665,15,0,14.335,0,13.516l0,0c0-0.82,0.665-1.483,1.484-1.483h15.032C17.335,12.031,18,12.695,18,13.516L18,13.516z"/> |
| </svg> |
| </span> |
| </label> |
| |
| <div class="trigger"> |
| <a class="page-link" href="/versions/1.9.1/get_started">Get Started</a> |
| <a class="page-link" href="/versions/1.9.1/features">Features</a> |
| <a class="page-link" href="/versions/1.9.1/ecosystem">Ecosystem</a> |
| <a class="page-link page-current" href="/versions/1.9.1/api">Docs & Tutorials</a> |
| <a class="page-link" href="/versions/1.9.1/trusted_by">Trusted By</a> |
| <a class="page-link" href="https://github.com/apache/mxnet">GitHub</a> |
| <div class="dropdown" style="min-width:100px"> |
| <span class="dropdown-header">Apache |
| <svg class="dropdown-caret" viewBox="0 0 32 32" class="icon icon-caret-bottom" aria-hidden="true"><path class="dropdown-caret-path" d="M24 11.305l-7.997 11.39L8 11.305z"></path></svg> |
| </span> |
| <div class="dropdown-content" style="min-width:250px"> |
| <a href="https://www.apache.org/foundation/">Apache Software Foundation</a> |
| <a href="https://incubator.apache.org/">Apache Incubator</a> |
| <a href="https://www.apache.org/licenses/">License</a> |
| <a href="/versions/1.9.1/api/faq/security.html">Security</a> |
| <a href="https://privacy.apache.org/policies/privacy-policy-public.html">Privacy</a> |
| <a href="https://www.apache.org/events/current-event">Events</a> |
| <a href="https://www.apache.org/foundation/sponsorship.html">Sponsorship</a> |
| <a href="https://www.apache.org/foundation/thanks.html">Thanks</a> |
| </div> |
| </div> |
| <div class="dropdown"> |
| <span class="dropdown-header">1.9.1 |
| <svg class="dropdown-caret" viewBox="0 0 32 32" class="icon icon-caret-bottom" aria-hidden="true"><path class="dropdown-caret-path" d="M24 11.305l-7.997 11.39L8 11.305z"></path></svg> |
| </span> |
| <div class="dropdown-content"> |
| <a class="dropdown-option" href="/">master</a><br> |
| <a class="dropdown-option-active" href="/versions/1.9.1/">1.9.1</a><br> |
| <a class="dropdown-option" href="/versions/1.8.0/">1.8.0</a><br> |
| <a class="dropdown-option" href="/versions/1.7.0/">1.7.0</a><br> |
| <a class="dropdown-option" href="/versions/1.6.0/">1.6.0</a><br> |
| <a class="dropdown-option" href="/versions/1.5.0/">1.5.0</a><br> |
| <a class="dropdown-option" href="/versions/1.4.1/">1.4.1</a><br> |
| <a class="dropdown-option" href="/versions/1.3.1/">1.3.1</a><br> |
| <a class="dropdown-option" href="/versions/1.2.1/">1.2.1</a><br> |
| <a class="dropdown-option" href="/versions/1.1.0/">1.1.0</a><br> |
| <a class="dropdown-option" href="/versions/1.0.0/">1.0.0</a><br> |
| <a class="dropdown-option" href="/versions/0.12.1/">0.12.1</a><br> |
| <a class="dropdown-option" href="/versions/0.11.0/">0.11.0</a> |
| </div> |
| </div> |
| </div> |
| </nav> |
| </div> |
| </header> |
| <div class="mdl-layout mdl-js-layout mdl-layout--fixed-header mdl-layout--fixed-drawer"><header class="mdl-layout__header mdl-layout__header--waterfall "> |
| <div class="mdl-layout__header-row"> |
| |
| <nav class="mdl-navigation breadcrumb"> |
| <a class="mdl-navigation__link" href="../../../index.html">Python Tutorials</a><i class="material-icons">navigate_next</i> |
| <a class="mdl-navigation__link" href="../../index.html">Packages</a><i class="material-icons">navigate_next</i> |
| <a class="mdl-navigation__link" href="../index.html">NDArray</a><i class="material-icons">navigate_next</i> |
| <a class="mdl-navigation__link" href="index.html">Tutorials</a><i class="material-icons">navigate_next</i> |
| <a class="mdl-navigation__link is-active">Sparse NDArrays with Gluon</a> |
| </nav> |
| <div class="mdl-layout-spacer"></div> |
| <nav class="mdl-navigation"> |
| |
| <form class="form-inline pull-sm-right" action="../../../../search.html" method="get"> |
| <div class="mdl-textfield mdl-js-textfield mdl-textfield--expandable mdl-textfield--floating-label mdl-textfield--align-right"> |
| <label id="quick-search-icon" class="mdl-button mdl-js-button mdl-button--icon" for="waterfall-exp"> |
| <i class="material-icons">search</i> |
| </label> |
| <div class="mdl-textfield__expandable-holder"> |
| <input class="mdl-textfield__input" type="text" name="q" id="waterfall-exp" placeholder="Search" /> |
| <input type="hidden" name="check_keywords" value="yes" /> |
| <input type="hidden" name="area" value="default" /> |
| </div> |
| </div> |
| <div class="mdl-tooltip" data-mdl-for="quick-search-icon"> |
| Quick search |
| </div> |
| </form> |
| |
| <a id="button-show-source" |
| class="mdl-button mdl-js-button mdl-button--icon" |
| href="../../../../_sources/tutorials/packages/ndarray/sparse/train_gluon.ipynb" rel="nofollow"> |
| <i class="material-icons">code</i> |
| </a> |
| <div class="mdl-tooltip" data-mdl-for="button-show-source"> |
| Show Source |
| </div> |
| </nav> |
| </div> |
| <div class="mdl-layout__header-row header-links"> |
| <div class="mdl-layout-spacer"></div> |
| <nav class="mdl-navigation"> |
| </nav> |
| </div> |
| </header><header class="mdl-layout__drawer"> |
| |
| <div class="globaltoc"> |
| <span class="mdl-layout-title toc">Table Of Contents</span> |
| |
| |
| |
| <nav class="mdl-navigation"> |
| <ul class="current"> |
| <li class="toctree-l1 current"><a class="reference internal" href="../../../index.html">Python Tutorials</a><ul class="current"> |
| <li class="toctree-l2"><a class="reference internal" href="../../../getting-started/index.html">Getting Started</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../getting-started/crash-course/index.html">Crash Course</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/1-ndarray.html">Manipulate data with <code class="docutils literal notranslate"><span class="pre">ndarray</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/2-nn.html">Create a neural network</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/3-autograd.html">Automatic differentiation with <code class="docutils literal notranslate"><span class="pre">autograd</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/4-train.html">Train the neural network</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/5-predict.html">Predict with a pre-trained model</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/6-use_gpus.html">Use GPUs</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../getting-started/to-mxnet/index.html">Moving to MXNet from Other Frameworks</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../../getting-started/to-mxnet/pytorch.html">PyTorch vs Apache MXNet</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../getting-started/gluon_from_experiment_to_deployment.html">Gluon: from experiment to deployment</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../getting-started/logistic_regression_explained.html">Logistic regression explained</a></li> |
| <li class="toctree-l3"><a class="reference external" href="https://mxnet.apache.org/api/python/docs/tutorials/packages/gluon/image/mnist.html">MNIST</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l2 current"><a class="reference internal" href="../../index.html">Packages</a><ul class="current"> |
| <li class="toctree-l3"><a class="reference internal" href="../../autograd/index.html">Automatic Differentiation</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../gluon/index.html">Gluon</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../gluon/blocks/index.html">Blocks</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/custom-layer.html">Custom Layers</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/custom_layer_beginners.html">Customer Layers (Beginners)</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/hybridize.html">Hybridize</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/init.html">Initialization</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/naming.html">Parameter and Block Naming</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/nn.html">Layers and Blocks</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/parameters.html">Parameter Management</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/save_load_params.html">Saving and Loading Gluon Models</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/activations/activations.html">Activation Blocks</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l4"><a class="reference internal" href="../../gluon/data/index.html">Data Tutorials</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/data/data_augmentation.html">Image Augmentation</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/data/data_augmentation.html#Spatial-Augmentation">Spatial Augmentation</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/data/data_augmentation.html#Color-Augmentation">Color Augmentation</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/data/data_augmentation.html#Composed-Augmentations">Composed Augmentations</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/data/datasets.html">Gluon <code class="docutils literal notranslate"><span class="pre">Dataset</span></code>s and <code class="docutils literal notranslate"><span class="pre">DataLoader</span></code></a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/data/datasets.html#Using-own-data-with-included-Datasets">Using own data with included <code class="docutils literal notranslate"><span class="pre">Dataset</span></code>s</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/data/datasets.html#Using-own-data-with-custom-Datasets">Using own data with custom <code class="docutils literal notranslate"><span class="pre">Dataset</span></code>s</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/data/datasets.html#Appendix:-Upgrading-from-Module-DataIter-to-Gluon-DataLoader">Appendix: Upgrading from Module <code class="docutils literal notranslate"><span class="pre">DataIter</span></code> to Gluon <code class="docutils literal notranslate"><span class="pre">DataLoader</span></code></a></li> |
| </ul> |
| </li> |
| <li class="toctree-l4"><a class="reference internal" href="../../gluon/image/index.html">Image Tutorials</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/image/image-augmentation.html">Image Augmentation</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/image/info_gan.html">Image similarity search with InfoGAN</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/image/mnist.html">Handwritten Digit Recognition</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/image/pretrained_models.html">Using pre-trained models in MXNet</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l4"><a class="reference internal" href="../../gluon/loss/index.html">Losses</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/loss/custom-loss.html">Custom Loss Blocks</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/loss/kl_divergence.html">Kullback-Leibler (KL) Divergence</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/loss/loss.html">Loss functions</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l4"><a class="reference internal" href="../../gluon/text/index.html">Text Tutorials</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/text/gnmt.html">Google Neural Machine Translation</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/text/transformer.html">Machine Translation with Transformer</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l4"><a class="reference internal" href="../../gluon/training/index.html">Training</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/training/fit_api_tutorial.html">MXNet Gluon Fit API</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/training/trainer.html">Trainer</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/training/learning_rates/index.html">Learning Rates</a><ul> |
| <li class="toctree-l6"><a class="reference internal" href="../../gluon/training/learning_rates/learning_rate_finder.html">Learning Rate Finder</a></li> |
| <li class="toctree-l6"><a class="reference internal" href="../../gluon/training/learning_rates/learning_rate_schedules.html">Learning Rate Schedules</a></li> |
| <li class="toctree-l6"><a class="reference internal" href="../../gluon/training/learning_rates/learning_rate_schedules_advanced.html">Advanced Learning Rate Schedules</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/training/normalization/index.html">Normalization Blocks</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../kvstore/index.html">KVStore</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../kvstore/kvstore.html">Distributed Key-Value Store</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l3 current"><a class="reference internal" href="../index.html">NDArray</a><ul class="current"> |
| <li class="toctree-l4"><a class="reference internal" href="../01-ndarray-intro.html">An Intro: Manipulate Data the MXNet Way with NDArray</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../02-ndarray-operations.html">NDArray Operations</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../03-ndarray-contexts.html">NDArray Contexts</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../gotchas_numpy_in_mxnet.html">Gotchas using NumPy in Apache MXNet</a></li> |
| <li class="toctree-l4 current"><a class="reference internal" href="index.html">Tutorials</a><ul class="current"> |
| <li class="toctree-l5"><a class="reference internal" href="csr.html">CSRNDArray - NDArray in Compressed Sparse Row Storage Format</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="row_sparse.html">RowSparseNDArray - NDArray for Sparse Gradient Updates</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="train.html">Train a Linear Regression Model with Sparse Symbols</a></li> |
| <li class="toctree-l5 current"><a class="current reference internal" href="#">Sparse NDArrays with Gluon</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../onnx/index.html">ONNX</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../onnx/fine_tuning_gluon.html">Fine-tuning an ONNX model</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../onnx/inference_on_onnx_model.html">Running inference on MXNet/Gluon from an ONNX model</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../onnx/super_resolution.html">Importing an ONNX model into MXNet</a></li> |
| <li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/python/docs/tutorials/deploy/export/onnx.html">Export ONNX Models</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../optimizer/index.html">Optimizers</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../viz/index.html">Visualization</a><ul> |
| <li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/faq/visualize_graph">Visualize networks</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../performance/index.html">Performance</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../performance/compression/index.html">Compression</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../../performance/compression/int8.html">Deploy with int-8</a></li> |
| <li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/faq/float16">Float16</a></li> |
| <li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/faq/gradient_compression">Gradient Compression</a></li> |
| <li class="toctree-l4"><a class="reference external" href="https://gluon-cv.mxnet.io/build/examples_deployment/int8_inference.html">GluonCV with Quantized Models</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../performance/backend/index.html">Accelerated Backend Tools</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../../performance/backend/mkldnn/index.html">Intel MKL-DNN</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../../performance/backend/mkldnn/mkldnn_quantization.html">Quantize with MKL-DNN backend</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../../performance/backend/mkldnn/mkldnn_quantization.html#Improving-accuracy-with-Intel®-Neural-Compressor">Improving accuracy with Intel® Neural Compressor</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../../performance/backend/mkldnn/mkldnn_readme.html">Install MXNet with MKL-DNN</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../performance/backend/tensorrt/index.html">TensorRT</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../../performance/backend/tensorrt/tensorrt.html">Optimizing Deep Learning Computation Graphs with TensorRT</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../performance/backend/tvm.html">Use TVM</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../performance/backend/profiler.html">Profiling MXNet Models</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../performance/backend/amp.html">Using AMP: Automatic Mixed Precision</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../deploy/index.html">Deployment</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../deploy/export/index.html">Export</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../../deploy/export/onnx.html">Exporting to ONNX format</a></li> |
| <li class="toctree-l4"><a class="reference external" href="https://gluon-cv.mxnet.io/build/examples_deployment/export_network.html">Export Gluon CV Models</a></li> |
| <li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/python/docs/tutorials/packages/gluon/blocks/save_load_params.html">Save / Load Parameters</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../deploy/inference/index.html">Inference</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../../deploy/inference/cpp.html">Deploy into C++</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../deploy/inference/image_classification_jetson.html">Image Classication using pretrained ResNet-50 model on Jetson module</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../deploy/inference/scala.html">Deploy into a Java or Scala Environment</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../deploy/inference/wine_detector.html">Real-time Object Detection with MXNet On The Raspberry Pi</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../deploy/run-on-aws/index.html">Run on AWS</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../../deploy/run-on-aws/use_ec2.html">Run on an EC2 Instance</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../deploy/run-on-aws/use_sagemaker.html">Run on Amazon SageMaker</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../deploy/run-on-aws/cloud.html">MXNet on the Cloud</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../extend/index.html">Extend</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../extend/custom_layer.html">Custom Layers</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../extend/customop.html">Custom Numpy Operators</a></li> |
| <li class="toctree-l3"><a class="reference external" href="https://mxnet.apache.org/api/faq/new_op">New Operator Creation</a></li> |
| <li class="toctree-l3"><a class="reference external" href="https://mxnet.apache.org/api/faq/add_op_in_backend">New Operator in MXNet Backend</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="toctree-l1"><a class="reference internal" href="../../../../api/index.html">Python API</a><ul> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/ndarray/index.html">mxnet.ndarray</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/ndarray.html">ndarray</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/contrib/index.html">ndarray.contrib</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/image/index.html">ndarray.image</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/linalg/index.html">ndarray.linalg</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/op/index.html">ndarray.op</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/random/index.html">ndarray.random</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/register/index.html">ndarray.register</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/sparse/index.html">ndarray.sparse</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/utils/index.html">ndarray.utils</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/gluon/index.html">mxnet.gluon</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/block.html">gluon.Block</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/hybrid_block.html">gluon.HybridBlock</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/symbol_block.html">gluon.SymbolBlock</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/constant.html">gluon.Constant</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/parameter.html">gluon.Parameter</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/parameter_dict.html">gluon.ParameterDict</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/trainer.html">gluon.Trainer</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/contrib/index.html">gluon.contrib</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/data/index.html">gluon.data</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../../../api/gluon/data/vision/index.html">data.vision</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../../../api/gluon/data/vision/datasets/index.html">vision.datasets</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../../../api/gluon/data/vision/transforms/index.html">vision.transforms</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/loss/index.html">gluon.loss</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/model_zoo/index.html">gluon.model_zoo.vision</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/nn/index.html">gluon.nn</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/rnn/index.html">gluon.rnn</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/utils/index.html">gluon.utils</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/autograd/index.html">mxnet.autograd</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/initializer/index.html">mxnet.initializer</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/optimizer/index.html">mxnet.optimizer</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/lr_scheduler/index.html">mxnet.lr_scheduler</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/metric/index.html">mxnet.metric</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/kvstore/index.html">mxnet.kvstore</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/symbol/index.html">mxnet.symbol</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/symbol.html">symbol</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/contrib/index.html">symbol.contrib</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/image/index.html">symbol.image</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/linalg/index.html">symbol.linalg</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/op/index.html">symbol.op</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/random/index.html">symbol.random</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/register/index.html">symbol.register</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/sparse/index.html">symbol.sparse</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/module/index.html">mxnet.module</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/contrib/index.html">mxnet.contrib</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/autograd/index.html">contrib.autograd</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/io/index.html">contrib.io</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/ndarray/index.html">contrib.ndarray</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/onnx/index.html">contrib.onnx</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/quantization/index.html">contrib.quantization</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/symbol/index.html">contrib.symbol</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/tensorboard/index.html">contrib.tensorboard</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/tensorrt/index.html">contrib.tensorrt</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/text/index.html">contrib.text</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/mxnet/index.html">mxnet</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/attribute/index.html">mxnet.attribute</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/base/index.html">mxnet.base</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/callback/index.html">mxnet.callback</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/context/index.html">mxnet.context</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/engine/index.html">mxnet.engine</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/executor/index.html">mxnet.executor</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/executor_manager/index.html">mxnet.executor_manager</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/image/index.html">mxnet.image</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/io/index.html">mxnet.io</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/kvstore_server/index.html">mxnet.kvstore_server</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/libinfo/index.html">mxnet.libinfo</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/log/index.html">mxnet.log</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/model/index.html">mxnet.model</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/monitor/index.html">mxnet.monitor</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/name/index.html">mxnet.name</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/notebook/index.html">mxnet.notebook</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/operator/index.html">mxnet.operator</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/profiler/index.html">mxnet.profiler</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/random/index.html">mxnet.random</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/recordio/index.html">mxnet.recordio</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/registry/index.html">mxnet.registry</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/rtc/index.html">mxnet.rtc</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/runtime/index.html">mxnet.runtime</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/test_utils/index.html">mxnet.test_utils</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/torch/index.html">mxnet.torch</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/util/index.html">mxnet.util</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/visualization/index.html">mxnet.visualization</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| |
| </nav> |
| |
| </div> |
| |
| </header> |
| <main class="mdl-layout__content" tabIndex="0"> |
| |
| <script type="text/javascript" src="../../../../_static/sphinx_materialdesign_theme.js "></script> |
| <script type="text/javascript" src="../../../../_static/feedback.js"></script> |
| <header class="mdl-layout__drawer"> |
| |
| <div class="globaltoc"> |
| <span class="mdl-layout-title toc">Table Of Contents</span> |
| |
| |
| |
| <nav class="mdl-navigation"> |
| <ul class="current"> |
| <li class="toctree-l1 current"><a class="reference internal" href="../../../index.html">Python Tutorials</a><ul class="current"> |
| <li class="toctree-l2"><a class="reference internal" href="../../../getting-started/index.html">Getting Started</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../getting-started/crash-course/index.html">Crash Course</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/1-ndarray.html">Manipulate data with <code class="docutils literal notranslate"><span class="pre">ndarray</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/2-nn.html">Create a neural network</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/3-autograd.html">Automatic differentiation with <code class="docutils literal notranslate"><span class="pre">autograd</span></code></a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/4-train.html">Train the neural network</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/5-predict.html">Predict with a pre-trained model</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/6-use_gpus.html">Use GPUs</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../getting-started/to-mxnet/index.html">Moving to MXNet from Other Frameworks</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../../getting-started/to-mxnet/pytorch.html">PyTorch vs Apache MXNet</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../getting-started/gluon_from_experiment_to_deployment.html">Gluon: from experiment to deployment</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../getting-started/logistic_regression_explained.html">Logistic regression explained</a></li> |
| <li class="toctree-l3"><a class="reference external" href="https://mxnet.apache.org/api/python/docs/tutorials/packages/gluon/image/mnist.html">MNIST</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l2 current"><a class="reference internal" href="../../index.html">Packages</a><ul class="current"> |
| <li class="toctree-l3"><a class="reference internal" href="../../autograd/index.html">Automatic Differentiation</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../gluon/index.html">Gluon</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../gluon/blocks/index.html">Blocks</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/custom-layer.html">Custom Layers</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/custom_layer_beginners.html">Customer Layers (Beginners)</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/hybridize.html">Hybridize</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/init.html">Initialization</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/naming.html">Parameter and Block Naming</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/nn.html">Layers and Blocks</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/parameters.html">Parameter Management</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/save_load_params.html">Saving and Loading Gluon Models</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/blocks/activations/activations.html">Activation Blocks</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l4"><a class="reference internal" href="../../gluon/data/index.html">Data Tutorials</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/data/data_augmentation.html">Image Augmentation</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/data/data_augmentation.html#Spatial-Augmentation">Spatial Augmentation</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/data/data_augmentation.html#Color-Augmentation">Color Augmentation</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/data/data_augmentation.html#Composed-Augmentations">Composed Augmentations</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/data/datasets.html">Gluon <code class="docutils literal notranslate"><span class="pre">Dataset</span></code>s and <code class="docutils literal notranslate"><span class="pre">DataLoader</span></code></a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/data/datasets.html#Using-own-data-with-included-Datasets">Using own data with included <code class="docutils literal notranslate"><span class="pre">Dataset</span></code>s</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/data/datasets.html#Using-own-data-with-custom-Datasets">Using own data with custom <code class="docutils literal notranslate"><span class="pre">Dataset</span></code>s</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/data/datasets.html#Appendix:-Upgrading-from-Module-DataIter-to-Gluon-DataLoader">Appendix: Upgrading from Module <code class="docutils literal notranslate"><span class="pre">DataIter</span></code> to Gluon <code class="docutils literal notranslate"><span class="pre">DataLoader</span></code></a></li> |
| </ul> |
| </li> |
| <li class="toctree-l4"><a class="reference internal" href="../../gluon/image/index.html">Image Tutorials</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/image/image-augmentation.html">Image Augmentation</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/image/info_gan.html">Image similarity search with InfoGAN</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/image/mnist.html">Handwritten Digit Recognition</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/image/pretrained_models.html">Using pre-trained models in MXNet</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l4"><a class="reference internal" href="../../gluon/loss/index.html">Losses</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/loss/custom-loss.html">Custom Loss Blocks</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/loss/kl_divergence.html">Kullback-Leibler (KL) Divergence</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/loss/loss.html">Loss functions</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l4"><a class="reference internal" href="../../gluon/text/index.html">Text Tutorials</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/text/gnmt.html">Google Neural Machine Translation</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/text/transformer.html">Machine Translation with Transformer</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l4"><a class="reference internal" href="../../gluon/training/index.html">Training</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/training/fit_api_tutorial.html">MXNet Gluon Fit API</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/training/trainer.html">Trainer</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/training/learning_rates/index.html">Learning Rates</a><ul> |
| <li class="toctree-l6"><a class="reference internal" href="../../gluon/training/learning_rates/learning_rate_finder.html">Learning Rate Finder</a></li> |
| <li class="toctree-l6"><a class="reference internal" href="../../gluon/training/learning_rates/learning_rate_schedules.html">Learning Rate Schedules</a></li> |
| <li class="toctree-l6"><a class="reference internal" href="../../gluon/training/learning_rates/learning_rate_schedules_advanced.html">Advanced Learning Rate Schedules</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l5"><a class="reference internal" href="../../gluon/training/normalization/index.html">Normalization Blocks</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../kvstore/index.html">KVStore</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../kvstore/kvstore.html">Distributed Key-Value Store</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l3 current"><a class="reference internal" href="../index.html">NDArray</a><ul class="current"> |
| <li class="toctree-l4"><a class="reference internal" href="../01-ndarray-intro.html">An Intro: Manipulate Data the MXNet Way with NDArray</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../02-ndarray-operations.html">NDArray Operations</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../03-ndarray-contexts.html">NDArray Contexts</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../gotchas_numpy_in_mxnet.html">Gotchas using NumPy in Apache MXNet</a></li> |
| <li class="toctree-l4 current"><a class="reference internal" href="index.html">Tutorials</a><ul class="current"> |
| <li class="toctree-l5"><a class="reference internal" href="csr.html">CSRNDArray - NDArray in Compressed Sparse Row Storage Format</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="row_sparse.html">RowSparseNDArray - NDArray for Sparse Gradient Updates</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="train.html">Train a Linear Regression Model with Sparse Symbols</a></li> |
| <li class="toctree-l5 current"><a class="current reference internal" href="#">Sparse NDArrays with Gluon</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../onnx/index.html">ONNX</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../onnx/fine_tuning_gluon.html">Fine-tuning an ONNX model</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../onnx/inference_on_onnx_model.html">Running inference on MXNet/Gluon from an ONNX model</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../onnx/super_resolution.html">Importing an ONNX model into MXNet</a></li> |
| <li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/python/docs/tutorials/deploy/export/onnx.html">Export ONNX Models</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../optimizer/index.html">Optimizers</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../viz/index.html">Visualization</a><ul> |
| <li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/faq/visualize_graph">Visualize networks</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../performance/index.html">Performance</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../performance/compression/index.html">Compression</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../../performance/compression/int8.html">Deploy with int-8</a></li> |
| <li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/faq/float16">Float16</a></li> |
| <li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/faq/gradient_compression">Gradient Compression</a></li> |
| <li class="toctree-l4"><a class="reference external" href="https://gluon-cv.mxnet.io/build/examples_deployment/int8_inference.html">GluonCV with Quantized Models</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../performance/backend/index.html">Accelerated Backend Tools</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../../performance/backend/mkldnn/index.html">Intel MKL-DNN</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../../performance/backend/mkldnn/mkldnn_quantization.html">Quantize with MKL-DNN backend</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../../performance/backend/mkldnn/mkldnn_quantization.html#Improving-accuracy-with-Intel®-Neural-Compressor">Improving accuracy with Intel® Neural Compressor</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../../performance/backend/mkldnn/mkldnn_readme.html">Install MXNet with MKL-DNN</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../performance/backend/tensorrt/index.html">TensorRT</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../../performance/backend/tensorrt/tensorrt.html">Optimizing Deep Learning Computation Graphs with TensorRT</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../performance/backend/tvm.html">Use TVM</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../performance/backend/profiler.html">Profiling MXNet Models</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../performance/backend/amp.html">Using AMP: Automatic Mixed Precision</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../deploy/index.html">Deployment</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../deploy/export/index.html">Export</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../../deploy/export/onnx.html">Exporting to ONNX format</a></li> |
| <li class="toctree-l4"><a class="reference external" href="https://gluon-cv.mxnet.io/build/examples_deployment/export_network.html">Export Gluon CV Models</a></li> |
| <li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/python/docs/tutorials/packages/gluon/blocks/save_load_params.html">Save / Load Parameters</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../deploy/inference/index.html">Inference</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../../deploy/inference/cpp.html">Deploy into C++</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../deploy/inference/image_classification_jetson.html">Image Classication using pretrained ResNet-50 model on Jetson module</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../deploy/inference/scala.html">Deploy into a Java or Scala Environment</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../deploy/inference/wine_detector.html">Real-time Object Detection with MXNet On The Raspberry Pi</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../deploy/run-on-aws/index.html">Run on AWS</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../../deploy/run-on-aws/use_ec2.html">Run on an EC2 Instance</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../deploy/run-on-aws/use_sagemaker.html">Run on Amazon SageMaker</a></li> |
| <li class="toctree-l4"><a class="reference internal" href="../../../deploy/run-on-aws/cloud.html">MXNet on the Cloud</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../extend/index.html">Extend</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../extend/custom_layer.html">Custom Layers</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../extend/customop.html">Custom Numpy Operators</a></li> |
| <li class="toctree-l3"><a class="reference external" href="https://mxnet.apache.org/api/faq/new_op">New Operator Creation</a></li> |
| <li class="toctree-l3"><a class="reference external" href="https://mxnet.apache.org/api/faq/add_op_in_backend">New Operator in MXNet Backend</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="toctree-l1"><a class="reference internal" href="../../../../api/index.html">Python API</a><ul> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/ndarray/index.html">mxnet.ndarray</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/ndarray.html">ndarray</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/contrib/index.html">ndarray.contrib</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/image/index.html">ndarray.image</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/linalg/index.html">ndarray.linalg</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/op/index.html">ndarray.op</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/random/index.html">ndarray.random</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/register/index.html">ndarray.register</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/sparse/index.html">ndarray.sparse</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/utils/index.html">ndarray.utils</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/gluon/index.html">mxnet.gluon</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/block.html">gluon.Block</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/hybrid_block.html">gluon.HybridBlock</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/symbol_block.html">gluon.SymbolBlock</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/constant.html">gluon.Constant</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/parameter.html">gluon.Parameter</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/parameter_dict.html">gluon.ParameterDict</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/trainer.html">gluon.Trainer</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/contrib/index.html">gluon.contrib</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/data/index.html">gluon.data</a><ul> |
| <li class="toctree-l4"><a class="reference internal" href="../../../../api/gluon/data/vision/index.html">data.vision</a><ul> |
| <li class="toctree-l5"><a class="reference internal" href="../../../../api/gluon/data/vision/datasets/index.html">vision.datasets</a></li> |
| <li class="toctree-l5"><a class="reference internal" href="../../../../api/gluon/data/vision/transforms/index.html">vision.transforms</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/loss/index.html">gluon.loss</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/model_zoo/index.html">gluon.model_zoo.vision</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/nn/index.html">gluon.nn</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/rnn/index.html">gluon.rnn</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/utils/index.html">gluon.utils</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/autograd/index.html">mxnet.autograd</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/initializer/index.html">mxnet.initializer</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/optimizer/index.html">mxnet.optimizer</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/lr_scheduler/index.html">mxnet.lr_scheduler</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/metric/index.html">mxnet.metric</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/kvstore/index.html">mxnet.kvstore</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/symbol/index.html">mxnet.symbol</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/symbol.html">symbol</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/contrib/index.html">symbol.contrib</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/image/index.html">symbol.image</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/linalg/index.html">symbol.linalg</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/op/index.html">symbol.op</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/random/index.html">symbol.random</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/register/index.html">symbol.register</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/sparse/index.html">symbol.sparse</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/module/index.html">mxnet.module</a></li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/contrib/index.html">mxnet.contrib</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/autograd/index.html">contrib.autograd</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/io/index.html">contrib.io</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/ndarray/index.html">contrib.ndarray</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/onnx/index.html">contrib.onnx</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/quantization/index.html">contrib.quantization</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/symbol/index.html">contrib.symbol</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/tensorboard/index.html">contrib.tensorboard</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/tensorrt/index.html">contrib.tensorrt</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/text/index.html">contrib.text</a></li> |
| </ul> |
| </li> |
| <li class="toctree-l2"><a class="reference internal" href="../../../../api/mxnet/index.html">mxnet</a><ul> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/attribute/index.html">mxnet.attribute</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/base/index.html">mxnet.base</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/callback/index.html">mxnet.callback</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/context/index.html">mxnet.context</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/engine/index.html">mxnet.engine</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/executor/index.html">mxnet.executor</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/executor_manager/index.html">mxnet.executor_manager</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/image/index.html">mxnet.image</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/io/index.html">mxnet.io</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/kvstore_server/index.html">mxnet.kvstore_server</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/libinfo/index.html">mxnet.libinfo</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/log/index.html">mxnet.log</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/model/index.html">mxnet.model</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/monitor/index.html">mxnet.monitor</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/name/index.html">mxnet.name</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/notebook/index.html">mxnet.notebook</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/operator/index.html">mxnet.operator</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/profiler/index.html">mxnet.profiler</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/random/index.html">mxnet.random</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/recordio/index.html">mxnet.recordio</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/registry/index.html">mxnet.registry</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/rtc/index.html">mxnet.rtc</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/runtime/index.html">mxnet.runtime</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/test_utils/index.html">mxnet.test_utils</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/torch/index.html">mxnet.torch</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/util/index.html">mxnet.util</a></li> |
| <li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/visualization/index.html">mxnet.visualization</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| |
| </nav> |
| |
| </div> |
| |
| </header> |
| |
| <div class="document"> |
| <div class="page-content" role="main"> |
| |
| <!--- Licensed to the Apache Software Foundation (ASF) under one --><!--- or more contributor license agreements. See the NOTICE file --><!--- distributed with this work for additional information --><!--- regarding copyright ownership. The ASF licenses this file --><!--- to you under the Apache License, Version 2.0 (the --><!--- "License"); you may not use this file except in compliance --><!--- with the License. You may obtain a copy of the License at --><!--- http://www.apache.org/licenses/LICENSE-2.0 --><!--- Unless required by applicable law or agreed to in writing, --><!--- software distributed under the License is distributed on an --><!--- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY --><!--- KIND, either express or implied. See the License for the --><!--- specific language governing permissions and limitations --><!--- under the License. --><div class="section" id="Sparse-NDArrays-with-Gluon"> |
| <h1>Sparse NDArrays with Gluon<a class="headerlink" href="#Sparse-NDArrays-with-Gluon" title="Permalink to this headline">¶</a></h1> |
| <p>When working on machine learning problems, you may encounter situations where the input data is sparse (i.e. the majority of values are zero). One example of this is in recommendation systems. You could have millions of user and product features, but only a few of these features are present for each sample. Without special treatment, the sheer magnitude of the feature space can lead to out-of-memory situations and cause significant slowdowns when training and making predictions.</p> |
| <p>MXNet supports a number of sparse storage types (often called <code class="docutils literal notranslate"><span class="pre">stype</span></code> for short) for these situations. In this tutorial, we’ll start by generating some sparse data, write it to disk in the LibSVM format and then read back using the <a class="reference external" href="/api/python/docs/api/mxnet/io/index.html#mxnet.io.LibSVMIter">LibSVMIter</a> for training. We use the Gluon API to train the model and leverage sparse storage types such as |
| <a class="reference external" href="/api/python/docs/api/ndarray/sparse/index.html#mxnet.ndarray.sparse.CSRNDArray">CSRNDArray</a> and <a class="reference external" href="/api/python/docs/api/ndarray/sparse/index.html#mxnet.ndarray.sparse.RowSparseNDArray">RowSparseNDArray</a> to maximise performance and memory efficiency.</p> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">mxnet</span> <span class="k">as</span> <span class="nn">mx</span> |
| <span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span> |
| <span class="kn">import</span> <span class="nn">time</span> |
| </pre></div> |
| </div> |
| <div class="section" id="Generating-Sparse-Data"> |
| <h2>Generating Sparse Data<a class="headerlink" href="#Generating-Sparse-Data" title="Permalink to this headline">¶</a></h2> |
| <p>You will most likely have a sparse dataset in mind already if you’re reading this tutorial, but let’s create a dummy dataset to use in the examples that follow. Using <code class="docutils literal notranslate"><span class="pre">rand_ndarray</span></code> we will generate 1000 samples, each with 1,000,000 features of which 99.999% of values will be zero (i.e. 10 non-zero features for each sample). We take this as our input data for training and calculate a label based on an arbitrary rule: whether the feature sum is higher than average.</p> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">num_samples</span> <span class="o">=</span> <span class="mi">1000</span> |
| <span class="n">num_features</span> <span class="o">=</span> <span class="mi">1000000</span> |
| <span class="n">data</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">test_utils</span><span class="o">.</span><span class="n">rand_ndarray</span><span class="p">((</span><span class="n">num_samples</span><span class="p">,</span> <span class="n">num_features</span><span class="p">),</span> <span class="n">stype</span><span class="o">=</span><span class="s1">'csr'</span><span class="p">,</span> <span class="n">density</span><span class="o">=</span><span class="mf">0.00001</span><span class="p">)</span> |
| <span class="c1"># generate label: 1 if row sum above average, 0 otherwise.</span> |
| <span class="n">label</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="o">></span> <span class="n">data</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">mean</span><span class="p">()</span> |
| </pre></div> |
| </div> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="n">data</span><span class="p">))</span> |
| <span class="nb">print</span><span class="p">(</span><span class="n">data</span><span class="p">[:</span><span class="mi">10</span><span class="p">]</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">())</span> |
| <span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="si">{:,.0f}</span><span class="s1"> elements'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">product</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">shape</span><span class="p">)))</span> |
| <span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="si">{:,.0f}</span><span class="s1"> non-zero elements'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">size</span><span class="p">))</span> |
| </pre></div> |
| </div> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><class 'mxnet.ndarray.sparse.CSRNDArray'> |
| [[0. 0. 0. ... 0. 0. 0.] |
| [0. 0. 0. ... 0. 0. 0.] |
| [0. 0. 0. ... 0. 0. 0.] |
| ... |
| [0. 0. 0. ... 0. 0. 0.] |
| [0. 0. 0. ... 0. 0. 0.] |
| [0. 0. 0. ... 0. 0. 0.]] |
| 1,000,000,000 elements |
| 10,000 non-zero elements |
| </pre></div> |
| </div> |
| <p>Our storage type is CSR (Compressed Sparse Row) which is the ideal type for sparse data along multiple axes. See <a class="reference external" href="/api/python/docs/tutorials/packages/ndarray/sparse/csr.html">this in-depth tutorial</a> for more information. Just to confirm the generation process ran correctly, we can see that the vast majority of values are indeed zero. One of the first questions to ask would be how much memory is saved by storing this data in a |
| <a class="reference external" href="/api/python/docs/api/ndarray/sparse/index.html#mxnet.ndarray.sparse.CSRNDArray">CSRNDArray</a> versus a standard <a class="reference external" href="/api/python/docs/api/ndarray/ndarray.html#module-mxnet.ndarray">NDArray</a>. Since sparse arrays are constructed from many components (e.g. <code class="docutils literal notranslate"><span class="pre">data</span></code>, <code class="docutils literal notranslate"><span class="pre">indices</span></code> and <code class="docutils literal notranslate"><span class="pre">indptr</span></code>) we define a function called <code class="docutils literal notranslate"><span class="pre">get_nbytes</span></code> to calculate the number of bytes taken in memory to store an array. We compare the same data stored in a standard |
| <a class="reference external" href="/api/python/docs/api/ndarray/ndarray.html#module-mxnet.ndarray">NDArray</a> (with <code class="docutils literal notranslate"><span class="pre">data.tostype('default')</span></code>) to the <a class="reference external" href="/api/python/docs/api/ndarray/sparse/index.html#mxnet.ndarray.sparse.CSRNDArray">CSRNDArray</a>.</p> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">get_nbytes</span><span class="p">(</span><span class="n">array</span><span class="p">):</span> |
| <span class="n">fn</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">a</span><span class="p">:</span> <span class="n">a</span><span class="o">.</span><span class="n">size</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">dtype</span><span class="p">(</span><span class="n">a</span><span class="p">)</span><span class="o">.</span><span class="n">itemsize</span> |
| <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">array</span><span class="p">,</span> <span class="n">mx</span><span class="o">.</span><span class="n">ndarray</span><span class="o">.</span><span class="n">sparse</span><span class="o">.</span><span class="n">CSRNDArray</span><span class="p">):</span> |
| <span class="k">return</span> <span class="n">fn</span><span class="p">(</span><span class="n">array</span><span class="o">.</span><span class="n">data</span><span class="p">)</span> <span class="o">+</span> <span class="n">fn</span><span class="p">(</span><span class="n">array</span><span class="o">.</span><span class="n">indices</span><span class="p">)</span> <span class="o">+</span> <span class="n">fn</span><span class="p">(</span><span class="n">array</span><span class="o">.</span><span class="n">indptr</span><span class="p">)</span> |
| <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">array</span><span class="p">,</span> <span class="n">mx</span><span class="o">.</span><span class="n">ndarray</span><span class="o">.</span><span class="n">sparse</span><span class="o">.</span><span class="n">RowSparseNDArray</span><span class="p">):</span> |
| <span class="k">return</span> <span class="n">fn</span><span class="p">(</span><span class="n">array</span><span class="o">.</span><span class="n">data</span><span class="p">)</span> <span class="o">+</span> <span class="n">fn</span><span class="p">(</span><span class="n">array</span><span class="o">.</span><span class="n">indices</span><span class="p">)</span> |
| <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">array</span><span class="p">,</span> <span class="n">mx</span><span class="o">.</span><span class="n">ndarray</span><span class="o">.</span><span class="n">NDArray</span><span class="p">):</span> |
| <span class="k">return</span> <span class="n">fn</span><span class="p">(</span><span class="n">array</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="ne">TypeError</span><span class="p">(</span><span class="s1">'</span><span class="si">{}</span><span class="s1"> not supported'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="n">array</span><span class="p">)))</span> |
| </pre></div> |
| </div> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="s1">'NDarray:'</span><span class="p">,</span> <span class="n">get_nbytes</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">tostype</span><span class="p">(</span><span class="s1">'default'</span><span class="p">))</span><span class="o">/</span><span class="mi">1000000</span><span class="p">,</span> <span class="s1">'MBs'</span><span class="p">)</span> |
| <span class="nb">print</span><span class="p">(</span><span class="s1">'CSRNDArray'</span><span class="p">,</span> <span class="n">get_nbytes</span><span class="p">(</span><span class="n">data</span><span class="p">)</span><span class="o">/</span><span class="mi">1000000</span><span class="p">,</span> <span class="s1">'MBs'</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">NDarray</span><span class="p">:</span> <span class="mf">4000.0</span> <span class="n">MBs</span> |
| <span class="n">CSRNDArray</span> <span class="mf">0.128008</span> <span class="n">MBs</span> |
| </pre></div> |
| </div> |
| <p>Given the extremely high sparsity of the data, we observe a huge memory saving here! 0.13 MBs versus 4 GBs: ~30,000 times smaller. You can experiment with the amount of sparsity and see how these two storage types compare. When the number of non-zero values increases, this difference will reduce. And when the number of non-zero values exceeds ~1/3 you will find that this sparse storage type take more memory than dense! So use wisely.</p> |
| </div> |
| <div class="section" id="Writing-Sparse-Data"> |
| <h2>Writing Sparse Data<a class="headerlink" href="#Writing-Sparse-Data" title="Permalink to this headline">¶</a></h2> |
| <p>Since there is such a large size difference between dense and sparse storage formats here, we ideally want to store the data on disk in a sparse storage format too. MXNet supports a format called LibSVM and has a data iterator called <a class="reference external" href="/api/python/docs/api/mxnet/io/index.html#mxnet.io.LibSVMIter">LibSVMIter</a> specifically for data formatted this way.</p> |
| <p>A LibSVM file has a row for each sample, and each row starts with the label: in this case <code class="docutils literal notranslate"><span class="pre">0.0</span></code> or <code class="docutils literal notranslate"><span class="pre">1.0</span></code> since we have a classification task. After this we have a variable number of <code class="docutils literal notranslate"><span class="pre">key:value</span></code> pairs separated by spaces, where the key is column/feature index and the value is the value of that feature. When working with your own sparse data in a custom format you should try to convert your data into this format. We define a <code class="docutils literal notranslate"><span class="pre">save_as_libsvm</span></code> function to save the <code class="docutils literal notranslate"><span class="pre">data</span></code> |
| (<a class="reference external" href="/api/python/docs/api/ndarray/sparse/index.html#mxnet.ndarray.sparse.CSRNDArray">CSRNDArray</a>) and <code class="docutils literal notranslate"><span class="pre">label</span></code> (<code class="docutils literal notranslate"><span class="pre">NDArray</span></code>) to disk in LibSVM format.</p> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">save_as_libsvm</span><span class="p">(</span><span class="n">filepath</span><span class="p">,</span> <span class="n">data</span><span class="p">,</span> <span class="n">label</span><span class="p">):</span> |
| <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filepath</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">openfile</span><span class="p">:</span> |
| <span class="k">for</span> <span class="n">row_idx</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]):</span> |
| <span class="n">data_sample</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="n">row_idx</span><span class="p">]</span> |
| <span class="n">label_sample</span> <span class="o">=</span> <span class="n">label</span><span class="p">[</span><span class="n">row_idx</span><span class="p">]</span> |
| <span class="n">col_idxs</span> <span class="o">=</span> <span class="n">data_sample</span><span class="o">.</span><span class="n">indices</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span><span class="o">.</span><span class="n">tolist</span><span class="p">()</span> |
| <span class="n">values</span> <span class="o">=</span> <span class="n">data_sample</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span><span class="o">.</span><span class="n">tolist</span><span class="p">()</span> |
| <span class="n">label_str</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">label_sample</span><span class="o">.</span><span class="n">asscalar</span><span class="p">())</span> |
| <span class="n">value_strs</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'</span><span class="si">{}</span><span class="s1">:</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">idx</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span> <span class="k">for</span> <span class="n">idx</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">col_idxs</span><span class="p">,</span> <span class="n">values</span><span class="p">)]</span> |
| <span class="n">value_str</span> <span class="o">=</span> <span class="s2">" "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">value_strs</span><span class="p">)</span> |
| <span class="n">sample_str</span> <span class="o">=</span> <span class="s1">'</span><span class="si">{}</span><span class="s1"> </span><span class="si">{}</span><span class="se">\n</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">label_str</span><span class="p">,</span> <span class="n">value_str</span><span class="p">)</span> |
| <span class="n">openfile</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">sample_str</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">filepath</span> <span class="o">=</span> <span class="s1">'dataset.libsvm'</span> |
| <span class="n">save_as_libsvm</span><span class="p">(</span><span class="n">filepath</span><span class="p">,</span> <span class="n">data</span><span class="p">,</span> <span class="n">label</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <p>We have now written the <code class="docutils literal notranslate"><span class="pre">data</span></code> and <code class="docutils literal notranslate"><span class="pre">label</span></code> to disk, and can inspect the first 10 lines of the file:</p> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filepath</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span> <span class="k">as</span> <span class="n">openfile</span><span class="p">:</span> |
| <span class="n">lines</span> <span class="o">=</span> <span class="p">[</span><span class="n">openfile</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)]</span> |
| <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">lines</span><span class="p">:</span> |
| <span class="nb">print</span><span class="p">(</span><span class="n">line</span><span class="p">[:</span><span class="mi">80</span><span class="p">]</span> <span class="o">+</span> <span class="s1">'...'</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">line</span><span class="p">)</span> <span class="o">></span> <span class="mi">80</span> <span class="k">else</span> <span class="n">line</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="mf">0.0</span> <span class="mi">35454</span><span class="p">:</span><span class="mf">0.22486156225204468</span> <span class="mi">80954</span><span class="p">:</span><span class="mf">0.39130592346191406</span> <span class="mi">81941</span><span class="p">:</span><span class="mf">0.1988530308008194</span><span class="o">...</span> |
| <span class="mf">1.0</span> <span class="mi">37029</span><span class="p">:</span><span class="mf">0.5980494618415833</span> <span class="mi">52916</span><span class="p">:</span><span class="mf">0.15797750651836395</span> <span class="mi">71623</span><span class="p">:</span><span class="mf">0.32251599431037903</span><span class="o">...</span> |
| <span class="mf">1.0</span> <span class="mi">89962</span><span class="p">:</span><span class="mf">0.47770974040031433</span> <span class="mi">216426</span><span class="p">:</span><span class="mf">0.21326342225074768</span> <span class="mi">271027</span><span class="p">:</span><span class="mf">0.18589609861373</span><span class="o">...</span> |
| <span class="mf">1.0</span> <span class="mi">7071</span><span class="p">:</span><span class="mf">0.9432336688041687</span> <span class="mi">81664</span><span class="p">:</span><span class="mf">0.7788773775100708</span> <span class="mi">117459</span><span class="p">:</span><span class="mf">0.8166475296020508</span> <span class="mf">4.</span><span class="o">..</span> |
| <span class="mf">0.0</span> <span class="mi">380966</span><span class="p">:</span><span class="mf">0.16906292736530304</span> <span class="mi">394363</span><span class="p">:</span><span class="mf">0.7987179756164551</span> <span class="mi">458442</span><span class="p">:</span><span class="mf">0.56873309612274</span><span class="o">...</span> |
| <span class="mf">0.0</span> <span class="mi">89361</span><span class="p">:</span><span class="mf">0.9099966287612915</span> <span class="mi">141813</span><span class="p">:</span><span class="mf">0.5927085280418396</span> <span class="mi">282489</span><span class="p">:</span><span class="mf">0.293381005525589</span> <span class="o">...</span> |
| <span class="mf">0.0</span> <span class="mi">150427</span><span class="p">:</span><span class="mf">0.4747847020626068</span> <span class="mi">169376</span><span class="p">:</span><span class="mf">0.2603490948677063</span> <span class="mi">179377</span><span class="p">:</span><span class="mf">0.237988427281379</span><span class="o">...</span> |
| <span class="mf">0.0</span> <span class="mi">49774</span><span class="p">:</span><span class="mf">0.2822582423686981</span> <span class="mi">91245</span><span class="p">:</span><span class="mf">0.5794865489006042</span> <span class="mi">102970</span><span class="p">:</span><span class="mf">0.7004560232162476</span> <span class="o">...</span> |
| <span class="mf">1.0</span> <span class="mi">97133</span><span class="p">:</span><span class="mf">0.0024336236529052258</span> <span class="mi">109855</span><span class="p">:</span><span class="mf">0.9895315766334534</span> <span class="mi">116765</span><span class="p">:</span><span class="mf">0.2465638816356</span><span class="o">...</span> |
| <span class="mf">0.0</span> <span class="mi">803440</span><span class="p">:</span><span class="mf">0.4020800292491913</span> |
| </pre></div> |
| </div> |
| <p>Some storage overhead is introduced by serializing the data as characters (with spaces and colons). <code class="docutils literal notranslate"><span class="pre">dataset.libsvm</span></code> is 250 KBs but the original <code class="docutils literal notranslate"><span class="pre">data</span></code> and <code class="docutils literal notranslate"><span class="pre">label</span></code> were 132 KBs combined. Compared with the 4GB dense <code class="docutils literal notranslate"><span class="pre">NDArray</span></code> though, this isn’t a huge issue.</p> |
| </div> |
| <div class="section" id="Reading-Sparse-Data"> |
| <h2>Reading Sparse Data<a class="headerlink" href="#Reading-Sparse-Data" title="Permalink to this headline">¶</a></h2> |
| <p>Using <a class="reference external" href="/api/python/docs/api/mxnet/io/index.html#mxnet.io.LibSVMIter">LibSVMIter</a>, we can quickly and easily load data into batches ready for training. Although Gluon <a class="reference external" href="/api/python/docs/api/gluon/data/index.html#mxnet.gluon.data.Dataset">Dataset</a>s can be written to return sparse arrays, Gluon <a class="reference external" href="/api/python/docs/api/gluon/data/index.html#mxnet.gluon.data.DataLoader">DataLoader</a>s currently convert each sample to dense before stacking up to create the batch. As a result, |
| <a class="reference external" href="/api/python/docs/api/mxnet/io/index.html#mxnet.io.LibSVMIter">LibSVMIter</a> is the recommended method of loading sparse data in batches.</p> |
| <p>Similar to using a <a class="reference external" href="/api/python/docs/api/gluon/data/index.html#mxnet.gluon.data.DataLoader">DataLoader</a>, you must specify the required <code class="docutils literal notranslate"><span class="pre">batch_size</span></code>. Since we’re dealing with sparse data and the column shape isn’t explicitly stored in the LibSVM file, we additionally need to provide the shape of the data and label. Our <a class="reference external" href="/api/python/docs/api/mxnet/io/index.html#mxnet.io.LibSVMIter">LibSVMIter</a> returns batches in a slightly different form to a |
| <a class="reference external" href="/api/python/docs/api/gluon/data/index.html#mxnet.gluon.data.DataLoader">DataLoader</a>. We get <code class="docutils literal notranslate"><span class="pre">DataBatch</span></code> objects instead of <code class="docutils literal notranslate"><span class="pre">tuple</span></code>.</p> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">data_iter</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">LibSVMIter</span><span class="p">(</span><span class="n">data_libsvm</span><span class="o">=</span><span class="n">filepath</span><span class="p">,</span> <span class="n">data_shape</span><span class="o">=</span><span class="p">(</span><span class="n">num_features</span><span class="p">,),</span> <span class="n">label_shape</span><span class="o">=</span><span class="p">(</span><span class="mi">1</span><span class="p">,),</span> <span class="n">batch_size</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span> |
| <span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">data_iter</span><span class="p">:</span> |
| <span class="n">data</span> <span class="o">=</span> <span class="n">batch</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> |
| <span class="nb">print</span><span class="p">(</span><span class="s1">'data.stype: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">stype</span><span class="p">))</span> |
| <span class="n">label</span> <span class="o">=</span> <span class="n">batch</span><span class="o">.</span><span class="n">label</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> |
| <span class="nb">print</span><span class="p">(</span><span class="s1">'label.stype: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">label</span><span class="o">.</span><span class="n">stype</span><span class="p">))</span> |
| <span class="k">break</span> |
| </pre></div> |
| </div> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">data</span><span class="o">.</span><span class="n">stype</span><span class="p">:</span> <span class="n">csr</span> |
| <span class="n">label</span><span class="o">.</span><span class="n">stype</span><span class="p">:</span> <span class="n">default</span> |
| </pre></div> |
| </div> |
| <p>We can see that <code class="docutils literal notranslate"><span class="pre">data</span></code> and <code class="docutils literal notranslate"><span class="pre">label</span></code> are in the appropriate storage formats, given their sparse and dense values respectively. We can avoid out-of-memory issues that might have occurred if <code class="docutils literal notranslate"><span class="pre">data</span></code> was in dense storage format. Another benefit of storing the data efficiently is the reduced data transfer time when using GPUs. Although the transfer time for a single batch is small, we transfer <code class="docutils literal notranslate"><span class="pre">data</span></code> and <code class="docutils literal notranslate"><span class="pre">label</span></code> to the GPU every iteration so this time can become significant. We will time the |
| transfer of the sparse <code class="docutils literal notranslate"><span class="pre">data</span></code> to GPU (if available) and compare to the time for its dense counterpart.</p> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">ctx</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">gpu</span><span class="p">()</span> <span class="k">if</span> <span class="n">mx</span><span class="o">.</span><span class="n">test_utils</span><span class="o">.</span><span class="n">list_gpus</span><span class="p">()</span> <span class="k">else</span> <span class="n">mx</span><span class="o">.</span><span class="n">cpu</span><span class="p">()</span> |
| </pre></div> |
| </div> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="o">%%</span><span class="n">timeit</span> |
| <span class="n">data_on_ctx</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">as_in_context</span><span class="p">(</span><span class="n">ctx</span><span class="p">)</span> |
| <span class="n">data_on_ctx</span><span class="o">.</span><span class="n">wait_to_read</span><span class="p">()</span> |
| </pre></div> |
| </div> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="mi">192</span> <span class="n">microseconds</span> <span class="o">+-</span> <span class="mf">51.1</span> <span class="n">microseconds</span> <span class="n">per</span> <span class="n">loop</span> <span class="p">(</span><span class="n">mean</span> <span class="o">+-</span> <span class="n">std</span><span class="o">.</span> <span class="n">dev</span><span class="o">.</span> <span class="n">of</span> <span class="mi">7</span> <span class="n">runs</span><span class="p">,</span> <span class="mi">1</span> <span class="n">loop</span> <span class="n">each</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="s1">'sparse batch: </span><span class="si">{}</span><span class="s1"> MBs'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">get_nbytes</span><span class="p">(</span><span class="n">data</span><span class="p">)</span><span class="o">/</span><span class="mi">1000000</span><span class="p">))</span> |
| <span class="n">data</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">tostype</span><span class="p">(</span><span class="s1">'default'</span><span class="p">)</span> <span class="c1"># avoid timing this sparse to dense conversion</span> |
| <span class="nb">print</span><span class="p">(</span><span class="s1">'dense batch: </span><span class="si">{}</span><span class="s1"> MBs'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">get_nbytes</span><span class="p">(</span><span class="n">data</span><span class="p">)</span><span class="o">/</span><span class="mi">1000000</span><span class="p">))</span> |
| </pre></div> |
| </div> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">sparse</span> <span class="n">batch</span><span class="p">:</span> <span class="mf">0.001348</span> <span class="n">MBs</span> |
| <span class="n">dense</span> <span class="n">batch</span><span class="p">:</span> <span class="mf">40.0</span> <span class="n">MBs</span> |
| </pre></div> |
| </div> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="o">%%</span><span class="n">timeit</span> |
| <span class="n">data_on_ctx</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">as_in_context</span><span class="p">(</span><span class="n">ctx</span><span class="p">)</span> |
| <span class="n">data_on_ctx</span><span class="o">.</span><span class="n">wait_to_read</span><span class="p">()</span> |
| </pre></div> |
| </div> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="mi">4</span> <span class="n">ms</span> <span class="o">+-</span> <span class="mf">36.8</span> <span class="n">microseconds</span> <span class="n">per</span> <span class="n">loop</span> <span class="p">(</span><span class="n">mean</span> <span class="o">+-</span> <span class="n">std</span><span class="o">.</span> <span class="n">dev</span><span class="o">.</span> <span class="n">of</span> <span class="mi">7</span> <span class="n">runs</span><span class="p">,</span> <span class="mi">100</span> <span class="n">loops</span> <span class="n">each</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <p>Although results will change depending on system specifications and degree of sparsity, the sparse array can be transferred from CPU to GPU significantly faster than the dense array. We see a ~25x speed up for sparse vs dense for this specific batch of data.</p> |
| <div class="section" id="Gluon-Models-for-Sparse-Data"> |
| <h3>Gluon Models for Sparse Data<a class="headerlink" href="#Gluon-Models-for-Sparse-Data" title="Permalink to this headline">¶</a></h3> |
| <p>Our next step is to define a network. We have an input of 1,000,000 features and we want to make a binary prediction. We don’t have any spatial or temporal relationships between features, so we’ll use a 3 layer fully-connected network where the last layer has 1 output unit (with sigmoid activation). Since we’re working with sparse data, we’d ideally like to use network operators that can exploit this sparsity for improved performance and memory efficiency.</p> |
| <p>Gluon’s <a class="reference external" href="/api/python/docs/api/gluon/nn/index.html#mxnet.gluon.nn.Dense">nn.Dense</a> block can used with <a class="reference external" href="/api/python/docs/api/ndarray/sparse/index.html#mxnet.ndarray.sparse.CSRNDArray">CSRNDArray</a> input arrays but it doesn’t exploit the sparsity. Under the hood, <a class="reference external" href="/api/python/docs/api/gluon/nn/index.html#mxnet.gluon.nn.Dense">Dense</a> uses the <a class="reference external" href="/api/python/docs/api/ndarray/ndarray.html#mxnet.ndarray.FullyConnected">FullyConnected</a> operator which isn’t optimized for |
| <a class="reference external" href="/api/python/docs/api/ndarray/sparse/index.html#mxnet.ndarray.sparse.CSRNDArray">CSRNDArray</a> arrays. We’ll implement a <code class="docutils literal notranslate"><span class="pre">Block</span></code> that does exploit this sparsity, <em>but first</em>, let’s just remind ourselves of the <a class="reference external" href="/api/python/docs/api/gluon/nn/index.html#mxnet.gluon.nn.Dense">Dense</a> implementation by creating an equivalent <code class="docutils literal notranslate"><span class="pre">Block</span></code> called <code class="docutils literal notranslate"><span class="pre">FullyConnected</span></code>.</p> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">FullyConnected</span><span class="p">(</span><span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">HybridBlock</span><span class="p">):</span> |
| <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">in_units</span><span class="p">,</span> <span class="n">units</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">FullyConnected</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span> |
| <span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">name_scope</span><span class="p">():</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_units</span> <span class="o">=</span> <span class="n">units</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">weight</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">params</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'weight'</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="n">units</span><span class="p">,</span> <span class="n">in_units</span><span class="p">),</span> |
| <span class="n">init</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">allow_deferred_init</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> |
| <span class="n">dtype</span><span class="o">=</span><span class="s1">'float32'</span><span class="p">,</span> <span class="n">stype</span><span class="o">=</span><span class="s1">'default'</span><span class="p">,</span> <span class="n">grad_stype</span><span class="o">=</span><span class="s1">'default'</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">bias</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">params</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'bias'</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="n">units</span><span class="p">),</span> |
| <span class="n">init</span><span class="o">=</span><span class="s1">'zeros'</span><span class="p">,</span> <span class="n">allow_deferred_init</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> |
| <span class="n">dtype</span><span class="o">=</span><span class="s1">'float32'</span><span class="p">,</span> <span class="n">stype</span><span class="o">=</span><span class="s1">'default'</span><span class="p">,</span> <span class="n">grad_stype</span><span class="o">=</span><span class="s1">'default'</span><span class="p">)</span> |
| |
| <span class="k">def</span> <span class="nf">hybrid_forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">F</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">bias</span><span class="p">):</span> |
| <span class="k">return</span> <span class="n">F</span><span class="o">.</span><span class="n">FullyConnected</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">bias</span><span class="p">,</span> <span class="n">num_hidden</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_units</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <p>Our <code class="docutils literal notranslate"><span class="pre">weight</span></code> and <code class="docutils literal notranslate"><span class="pre">bias</span></code> parameters are dense (see <code class="docutils literal notranslate"><span class="pre">stype='default'</span></code>) and so are their gradients (see <code class="docutils literal notranslate"><span class="pre">grad_stype='default'</span></code>). Our <code class="docutils literal notranslate"><span class="pre">weight</span></code> parameter has shape <code class="docutils literal notranslate"><span class="pre">(units,</span> <span class="pre">in_units)</span></code> because the <a class="reference external" href="/api/python/docs/api/ndarray/ndarray.html#mxnet.ndarray.FullyConnected">FullyConnected</a> operator performs the following calculation:</p> |
| <div class="math notranslate nohighlight"> |
| \[Y = XW^T + b\]</div> |
| <p>We could instead have created our parameter with shape <code class="docutils literal notranslate"><span class="pre">(in_units,</span> <span class="pre">units)</span></code> and avoid the transpose of the weight matrix. We’ll see why this is so important later on. And instead of <a class="reference external" href="/api/python/docs/api/ndarray/ndarray.html#mxnet.ndarray.FullyConnected">FullyConnected</a> we could have used <a class="reference external" href="/api/python/docs/api/ndarray/sparse/index.html?#mxnet.ndarray.sparse.dot">mx.sparse.dot</a> to fully exploit the sparsity of the |
| <a class="reference external" href="/api/python/docs/api/ndarray/sparse/index.html#mxnet.ndarray.sparse.CSRNDArray">CSRNDArray</a> input arrays. We’ll now implement an alternative <code class="docutils literal notranslate"><span class="pre">Block</span></code> called <code class="docutils literal notranslate"><span class="pre">FullyConnectedSparse</span></code> using these ideas. We take <code class="docutils literal notranslate"><span class="pre">grad_stype</span></code> of the <code class="docutils literal notranslate"><span class="pre">weight</span></code> as an argument (called <code class="docutils literal notranslate"><span class="pre">weight_grad_stype</span></code>), since we’re going to change this later on.</p> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">FullyConnectedSparse</span><span class="p">(</span><span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">HybridBlock</span><span class="p">):</span> |
| <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">in_units</span><span class="p">,</span> <span class="n">units</span><span class="p">,</span> <span class="n">weight_grad_stype</span><span class="o">=</span><span class="s1">'default'</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">FullyConnectedSparse</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span> |
| <span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">name_scope</span><span class="p">():</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_units</span> <span class="o">=</span> <span class="n">units</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">weight</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">params</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'weight'</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="n">in_units</span><span class="p">,</span> <span class="n">units</span><span class="p">),</span> |
| <span class="n">init</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">allow_deferred_init</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> |
| <span class="n">dtype</span><span class="o">=</span><span class="s1">'float32'</span><span class="p">,</span> <span class="n">stype</span><span class="o">=</span><span class="s1">'default'</span><span class="p">,</span> <span class="n">grad_stype</span><span class="o">=</span><span class="n">weight_grad_stype</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">bias</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">params</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'bias'</span><span class="p">,</span> <span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="n">units</span><span class="p">),</span> |
| <span class="n">init</span><span class="o">=</span><span class="s1">'zeros'</span><span class="p">,</span> <span class="n">allow_deferred_init</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> |
| <span class="n">dtype</span><span class="o">=</span><span class="s1">'float32'</span><span class="p">,</span> <span class="n">stype</span><span class="o">=</span><span class="s1">'default'</span><span class="p">,</span> <span class="n">grad_stype</span><span class="o">=</span><span class="s1">'default'</span><span class="p">)</span> |
| |
| <span class="k">def</span> <span class="nf">hybrid_forward</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">F</span><span class="p">,</span> <span class="n">x</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">bias</span><span class="p">):</span> |
| <span class="k">return</span> <span class="n">F</span><span class="o">.</span><span class="n">sparse</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">weight</span><span class="p">)</span> <span class="o">+</span> <span class="n">bias</span> |
| </pre></div> |
| </div> |
| <p>Once again, we’re using a dense <code class="docutils literal notranslate"><span class="pre">weight</span></code>, so both <code class="docutils literal notranslate"><span class="pre">FullyConnected</span></code> and <code class="docutils literal notranslate"><span class="pre">FullyConnectedSparse</span></code> will return dense array outputs. When constructing a multi-layer network therefore, only the first layer needs to be optimized for sparse inputs. Our first layer is often responsible for reducing the feature dimension dramatically (e.g. 1,000,000 features down to 128 features). We’ll set the number of units in our 3 layers to be 128, 8 and 1.</p> |
| <p>We will use <a class="reference external" href="https://docs.python.org/2/library/timeit.html">timeit</a> to check the performance of these two variants, and analyse some <a class="reference external" href="/api/python/docs/tutorials/performance/backend/profiler.html">MXNet Profiler</a> traces that have been created from these benchmarks. Additionally, we will inspect the memory usage of the weights (and gradients) using the <code class="docutils literal notranslate"><span class="pre">print_memory_allocation</span></code> function defined below:</p> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">print_memory_allocation</span><span class="p">(</span><span class="n">net</span><span class="p">,</span> <span class="n">block_idxs</span><span class="p">):</span> |
| <span class="n">blocks</span> <span class="o">=</span> <span class="p">[</span><span class="n">net</span><span class="p">[</span><span class="n">block_idx</span><span class="p">]</span> <span class="k">for</span> <span class="n">block_idx</span> <span class="ow">in</span> <span class="n">block_idxs</span><span class="p">]</span> |
| <span class="n">weight_nbytes</span> <span class="o">=</span> <span class="p">[</span><span class="n">get_nbytes</span><span class="p">(</span><span class="n">b</span><span class="o">.</span><span class="n">weight</span><span class="o">.</span><span class="n">data</span><span class="p">())</span> <span class="k">for</span> <span class="n">b</span> <span class="ow">in</span> <span class="n">blocks</span><span class="p">]</span> |
| <span class="n">weight_nbytes_pct</span> <span class="o">=</span> <span class="p">[</span><span class="n">b</span><span class="o">/</span><span class="nb">sum</span><span class="p">(</span><span class="n">weight_nbytes</span><span class="p">)</span> <span class="k">for</span> <span class="n">b</span> <span class="ow">in</span> <span class="n">weight_nbytes</span><span class="p">]</span> |
| <span class="n">weight_grad_nbytes</span> <span class="o">=</span> <span class="p">[</span><span class="n">get_nbytes</span><span class="p">(</span><span class="n">b</span><span class="o">.</span><span class="n">weight</span><span class="o">.</span><span class="n">grad</span><span class="p">())</span> <span class="k">for</span> <span class="n">b</span> <span class="ow">in</span> <span class="n">blocks</span><span class="p">]</span> |
| <span class="n">weight_grad_nbytes_pct</span> <span class="o">=</span> <span class="p">[</span><span class="n">b</span><span class="o">/</span><span class="nb">sum</span><span class="p">(</span><span class="n">weight_grad_nbytes</span><span class="p">)</span> <span class="k">for</span> <span class="n">b</span> <span class="ow">in</span> <span class="n">weight_grad_nbytes</span><span class="p">]</span> |
| <span class="nb">print</span><span class="p">(</span><span class="s2">"Memory Allocation for Weight:"</span><span class="p">)</span> |
| <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">block_idxs</span><span class="p">)):</span> |
| <span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="si">{:7.3f}</span><span class="s1"> MBs (</span><span class="si">{:7.3f}</span><span class="s1">%) for </span><span class="si">{:<40}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">weight_nbytes</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">/</span><span class="mi">1000000</span><span class="p">,</span> |
| <span class="n">weight_nbytes_pct</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">*</span><span class="mi">100</span><span class="p">,</span> |
| <span class="n">blocks</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">name</span><span class="p">))</span> |
| <span class="nb">print</span><span class="p">(</span><span class="s2">"Memory Allocation for Weight Gradient:"</span><span class="p">)</span> |
| <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">block_idxs</span><span class="p">)):</span> |
| <span class="nb">print</span><span class="p">(</span><span class="s1">'</span><span class="si">{:7.3f}</span><span class="s1"> MBs (</span><span class="si">{:7.3f}</span><span class="s1">%) for </span><span class="si">{:<40}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">weight_grad_nbytes</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">/</span><span class="mi">1000000</span><span class="p">,</span> |
| <span class="n">weight_grad_nbytes_pct</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">*</span><span class="mi">100</span><span class="p">,</span> |
| <span class="n">blocks</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">name</span><span class="p">))</span> |
| </pre></div> |
| </div> |
| </div> |
| </div> |
| <div class="section" id="Benchmark:-FullyConnected"> |
| <h2>Benchmark: <code class="docutils literal notranslate"><span class="pre">FullyConnected</span></code><a class="headerlink" href="#Benchmark:-FullyConnected" title="Permalink to this headline">¶</a></h2> |
| <p>We’ll create a network using <code class="docutils literal notranslate"><span class="pre">nn.Dense</span></code> and benchmark the training.</p> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">net</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">nn</span><span class="o">.</span><span class="n">Sequential</span><span class="p">()</span> |
| <span class="n">net</span><span class="o">.</span><span class="n">add</span><span class="p">(</span> |
| <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">nn</span><span class="o">.</span><span class="n">Dense</span><span class="p">(</span><span class="n">in_units</span><span class="o">=</span><span class="n">num_features</span><span class="p">,</span> <span class="n">units</span><span class="o">=</span><span class="mi">128</span><span class="p">),</span> |
| <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">nn</span><span class="o">.</span><span class="n">Activation</span><span class="p">(</span><span class="s1">'sigmoid'</span><span class="p">),</span> |
| <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">nn</span><span class="o">.</span><span class="n">Dense</span><span class="p">(</span><span class="n">in_units</span><span class="o">=</span><span class="mi">128</span><span class="p">,</span> <span class="n">units</span><span class="o">=</span><span class="mi">8</span><span class="p">),</span> |
| <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">nn</span><span class="o">.</span><span class="n">Activation</span><span class="p">(</span><span class="s1">'sigmoid'</span><span class="p">),</span> |
| <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">nn</span><span class="o">.</span><span class="n">Dense</span><span class="p">(</span><span class="n">in_units</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">units</span><span class="o">=</span><span class="mi">1</span><span class="p">),</span> |
| <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">nn</span><span class="o">.</span><span class="n">Activation</span><span class="p">(</span><span class="s1">'sigmoid'</span><span class="p">),</span> |
| <span class="p">)</span> |
| <span class="n">net</span><span class="o">.</span><span class="n">initialize</span><span class="p">(</span><span class="n">ctx</span><span class="o">=</span><span class="n">ctx</span><span class="p">)</span> |
| <span class="n">trainer</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">Trainer</span><span class="p">(</span><span class="n">net</span><span class="o">.</span><span class="n">collect_params</span><span class="p">(),</span> <span class="s1">'sgd'</span><span class="p">)</span> |
| <span class="n">loss_fn</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">loss</span><span class="o">.</span><span class="n">SigmoidBinaryCrossEntropyLoss</span><span class="p">()</span> |
| </pre></div> |
| </div> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="o">%%</span><span class="n">timeit</span> |
| <span class="n">data_iter</span><span class="o">.</span><span class="n">reset</span><span class="p">()</span> |
| <span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">data_iter</span><span class="p">:</span> |
| <span class="n">data</span> <span class="o">=</span> <span class="n">batch</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> |
| <span class="n">data</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">as_in_context</span><span class="p">(</span><span class="n">ctx</span><span class="p">)</span> |
| <span class="n">label</span> <span class="o">=</span> <span class="n">batch</span><span class="o">.</span><span class="n">label</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_in_context</span><span class="p">(</span><span class="n">ctx</span><span class="p">)</span> |
| <span class="k">with</span> <span class="n">mx</span><span class="o">.</span><span class="n">autograd</span><span class="o">.</span><span class="n">record</span><span class="p">():</span> |
| <span class="n">pred</span> <span class="o">=</span> <span class="n">net</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> |
| <span class="n">loss</span> <span class="o">=</span> <span class="n">loss_fn</span><span class="p">(</span><span class="n">pred</span><span class="p">,</span> <span class="n">label</span><span class="p">)</span> |
| <span class="n">loss</span><span class="o">.</span><span class="n">backward</span><span class="p">()</span> |
| <span class="n">trainer</span><span class="o">.</span><span class="n">step</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> |
| <span class="n">loss</span><span class="o">.</span><span class="n">wait_to_read</span><span class="p">()</span> |
| </pre></div> |
| </div> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="mi">532</span> <span class="n">ms</span> <span class="o">+-</span> <span class="mf">3.47</span> <span class="n">ms</span> <span class="n">per</span> <span class="n">loop</span> <span class="p">(</span><span class="n">mean</span> <span class="o">+-</span> <span class="n">std</span><span class="o">.</span> <span class="n">dev</span><span class="o">.</span> <span class="n">of</span> <span class="mi">7</span> <span class="n">runs</span><span class="p">,</span> <span class="mi">1</span> <span class="n">loop</span> <span class="n">each</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <img alt="fully connected" src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/ndarray/sparse/fully_connected.png" /> |
| <p>We can see the first <a class="reference external" href="/api/python/docs/api/ndarray/ndarray.html#mxnet.ndarray.FullyConnected">FullyConnected</a> operator takes a significant proportion of time to execute (~25% of the iteration) because there are 1,000,000 input features (to 128). After this, the other <a class="reference external" href="/api/python/docs/api/ndarray/ndarray.html#mxnet.ndarray.FullyConnected">FullyConnected</a> operators are much faster because they have input features of 128 (to 8) and 8 (to 1). On the backward pass, we see the same pattern (but |
| in reverse). And finally, the parameter update step takes a large amount of time on the weight matrix of the first <code class="docutils literal notranslate"><span class="pre">FullyConnected</span></code> <code class="docutils literal notranslate"><span class="pre">Block</span></code>. When checking the memory allocations below, we can see the weight matrix of the first <code class="docutils literal notranslate"><span class="pre">FullyConnected</span></code> <code class="docutils literal notranslate"><span class="pre">Block</span></code> is responsible for 99.999% of the memory compared to other <a class="reference external" href="/api/python/docs/api/ndarray/ndarray.html#mxnet.ndarray.FullyConnected">FullyConnected</a> weight matrices.</p> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">print_memory_allocation</span><span class="p">(</span><span class="n">net</span><span class="p">,</span> <span class="n">block_idxs</span><span class="o">=</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">])</span> |
| </pre></div> |
| </div> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Memory</span> <span class="n">Allocation</span> <span class="k">for</span> <span class="n">Weight</span><span class="p">:</span> |
| <span class="mf">512.000</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">99.999</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">dense0</span> |
| <span class="mf">0.004</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">0.001</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">dense1</span> |
| <span class="mf">0.000</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">0.000</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">dense2</span> |
| <span class="n">Memory</span> <span class="n">Allocation</span> <span class="k">for</span> <span class="n">Weight</span> <span class="n">Gradient</span><span class="p">:</span> |
| <span class="mf">512.000</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">99.999</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">dense0</span> |
| <span class="mf">0.004</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">0.001</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">dense1</span> |
| <span class="mf">0.000</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">0.000</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">dense2</span> |
| </pre></div> |
| </div> |
| </div> |
| <div class="section" id="Benchmark:-FullyConnectedSparse"> |
| <h2>Benchmark: <code class="docutils literal notranslate"><span class="pre">FullyConnectedSparse</span></code><a class="headerlink" href="#Benchmark:-FullyConnectedSparse" title="Permalink to this headline">¶</a></h2> |
| <p>We will now switch the first layer from <code class="docutils literal notranslate"><span class="pre">FullyConnected</span></code> to <code class="docutils literal notranslate"><span class="pre">FullyConnectedSparse</span></code>.</p> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">net</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">nn</span><span class="o">.</span><span class="n">Sequential</span><span class="p">()</span> |
| <span class="n">net</span><span class="o">.</span><span class="n">add</span><span class="p">(</span> |
| <span class="n">FullyConnectedSparse</span><span class="p">(</span><span class="n">in_units</span><span class="o">=</span><span class="n">num_features</span><span class="p">,</span> <span class="n">units</span><span class="o">=</span><span class="mi">128</span><span class="p">),</span> |
| <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">nn</span><span class="o">.</span><span class="n">Activation</span><span class="p">(</span><span class="s1">'sigmoid'</span><span class="p">),</span> |
| <span class="n">FullyConnected</span><span class="p">(</span><span class="n">in_units</span><span class="o">=</span><span class="mi">128</span><span class="p">,</span> <span class="n">units</span><span class="o">=</span><span class="mi">8</span><span class="p">),</span> |
| <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">nn</span><span class="o">.</span><span class="n">Activation</span><span class="p">(</span><span class="s1">'sigmoid'</span><span class="p">),</span> |
| <span class="n">FullyConnected</span><span class="p">(</span><span class="n">in_units</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">units</span><span class="o">=</span><span class="mi">1</span><span class="p">),</span> |
| <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">nn</span><span class="o">.</span><span class="n">Activation</span><span class="p">(</span><span class="s1">'sigmoid'</span><span class="p">),</span> |
| <span class="p">)</span> |
| <span class="n">net</span><span class="o">.</span><span class="n">initialize</span><span class="p">(</span><span class="n">ctx</span><span class="o">=</span><span class="n">ctx</span><span class="p">)</span> |
| <span class="n">trainer</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">Trainer</span><span class="p">(</span><span class="n">net</span><span class="o">.</span><span class="n">collect_params</span><span class="p">(),</span> <span class="s1">'sgd'</span><span class="p">)</span> |
| <span class="n">loss_fn</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">loss</span><span class="o">.</span><span class="n">SigmoidBinaryCrossEntropyLoss</span><span class="p">()</span> |
| </pre></div> |
| </div> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="o">%%</span><span class="n">timeit</span> |
| <span class="n">data_iter</span><span class="o">.</span><span class="n">reset</span><span class="p">()</span> |
| <span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">data_iter</span><span class="p">:</span> |
| <span class="n">data</span> <span class="o">=</span> <span class="n">batch</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> |
| <span class="n">data</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">as_in_context</span><span class="p">(</span><span class="n">ctx</span><span class="p">)</span> |
| <span class="n">label</span> <span class="o">=</span> <span class="n">batch</span><span class="o">.</span><span class="n">label</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_in_context</span><span class="p">(</span><span class="n">ctx</span><span class="p">)</span> |
| <span class="k">with</span> <span class="n">mx</span><span class="o">.</span><span class="n">autograd</span><span class="o">.</span><span class="n">record</span><span class="p">():</span> |
| <span class="n">pred</span> <span class="o">=</span> <span class="n">net</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> |
| <span class="n">loss</span> <span class="o">=</span> <span class="n">loss_fn</span><span class="p">(</span><span class="n">pred</span><span class="p">,</span> <span class="n">label</span><span class="p">)</span> |
| <span class="n">loss</span><span class="o">.</span><span class="n">backward</span><span class="p">()</span> |
| <span class="n">trainer</span><span class="o">.</span><span class="n">step</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> |
| <span class="n">loss</span><span class="o">.</span><span class="n">wait_to_read</span><span class="p">()</span> |
| </pre></div> |
| </div> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="mi">528</span> <span class="n">ms</span> <span class="o">+-</span> <span class="mf">22.7</span> <span class="n">ms</span> <span class="n">per</span> <span class="n">loop</span> <span class="p">(</span><span class="n">mean</span> <span class="o">+-</span> <span class="n">std</span><span class="o">.</span> <span class="n">dev</span><span class="o">.</span> <span class="n">of</span> <span class="mi">7</span> <span class="n">runs</span><span class="p">,</span> <span class="mi">1</span> <span class="n">loop</span> <span class="n">each</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <img alt="fully connected sparse" src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/ndarray/sparse/fully_connected_sparse.png" /> |
| <p>We see the forward pass of <code class="docutils literal notranslate"><span class="pre">dot</span></code> and <code class="docutils literal notranslate"><span class="pre">add</span></code> (equivalent to <a class="reference external" href="/api/python/docs/api/ndarray/ndarray.html#mxnet.ndarray.FullyConnected">FullyConnected</a> operator) is much faster now: 1.54ms vs 0.26ms. And this explains the reduction in overall time for the epoch. We didn’t gain any benefit on the backward pass or parameter updates though.</p> |
| <img alt="fully connected sparse backward" src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/ndarray/sparse/fully_connected_sparse_backward.png" /> |
| <p>Our first weight matrix and its gradients still take up the same amount of memory as before.</p> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">print_memory_allocation</span><span class="p">(</span><span class="n">net</span><span class="p">,</span> <span class="n">block_idxs</span><span class="o">=</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">])</span> |
| </pre></div> |
| </div> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Memory</span> <span class="n">Allocation</span> <span class="k">for</span> <span class="n">Weight</span><span class="p">:</span> |
| <span class="mf">512.000</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">99.999</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">fullyconnectedsparse0</span> |
| <span class="mf">0.004</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">0.001</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">fullyconnected0</span> |
| <span class="mf">0.000</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">0.000</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">fullyconnected1</span> |
| <span class="n">Memory</span> <span class="n">Allocation</span> <span class="k">for</span> <span class="n">Weight</span> <span class="n">Gradient</span><span class="p">:</span> |
| <span class="mf">512.000</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">99.999</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">fullyconnectedsparse0</span> |
| <span class="mf">0.004</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">0.001</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">fullyconnected0</span> |
| <span class="mf">0.000</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">0.000</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">fullyconnected1</span> |
| </pre></div> |
| </div> |
| </div> |
| <div class="section" id="Benchmark:-FullyConnectedSparse-with-grad_stype=row_sparse"> |
| <h2>Benchmark: <code class="docutils literal notranslate"><span class="pre">FullyConnectedSparse</span></code> with <code class="docutils literal notranslate"><span class="pre">grad_stype=row_sparse</span></code><a class="headerlink" href="#Benchmark:-FullyConnectedSparse-with-grad_stype=row_sparse" title="Permalink to this headline">¶</a></h2> |
| <p>One useful outcome of sparsity in our <a class="reference external" href="/api/python/docs/api/ndarray/sparse/index.html#mxnet.ndarray.sparse.CSRNDArray">CSRNDArray</a> input is that our gradients will be row sparse. We can exploit this fact to give us potentially huge memory savings and speed improvements. Creating our <code class="docutils literal notranslate"><span class="pre">weight</span></code> parameter with shape <code class="docutils literal notranslate"><span class="pre">(units,</span> <span class="pre">in_units)</span></code> and not transposing in the forward pass are important pre-requisite for obtaining row sparse gradients. Using |
| <a class="reference external" href="/api/python/docs/api/gluon/nn/index.html#mxnet.gluon.nn.Dense">nn.Dense</a> would have led to column sparse gradients which are not supported in MXNet. We previously had <code class="docutils literal notranslate"><span class="pre">grad_stype</span></code> of the <code class="docutils literal notranslate"><span class="pre">weight</span></code> parameter in the first layer set to <code class="docutils literal notranslate"><span class="pre">'default'</span></code> so we were handling the gradient as a dense array. Switching this to <code class="docutils literal notranslate"><span class="pre">'row_sparse'</span></code> can give us these potential improvements.</p> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">net</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">nn</span><span class="o">.</span><span class="n">Sequential</span><span class="p">()</span> |
| <span class="n">net</span><span class="o">.</span><span class="n">add</span><span class="p">(</span> |
| <span class="n">FullyConnectedSparse</span><span class="p">(</span><span class="n">in_units</span><span class="o">=</span><span class="n">num_features</span><span class="p">,</span> <span class="n">units</span><span class="o">=</span><span class="mi">128</span><span class="p">,</span> <span class="n">weight_grad_stype</span><span class="o">=</span><span class="s1">'row_sparse'</span><span class="p">),</span> |
| <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">nn</span><span class="o">.</span><span class="n">Activation</span><span class="p">(</span><span class="s1">'sigmoid'</span><span class="p">),</span> |
| <span class="n">FullyConnected</span><span class="p">(</span><span class="n">in_units</span><span class="o">=</span><span class="mi">128</span><span class="p">,</span> <span class="n">units</span><span class="o">=</span><span class="mi">8</span><span class="p">),</span> |
| <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">nn</span><span class="o">.</span><span class="n">Activation</span><span class="p">(</span><span class="s1">'sigmoid'</span><span class="p">),</span> |
| <span class="n">FullyConnected</span><span class="p">(</span><span class="n">in_units</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">units</span><span class="o">=</span><span class="mi">1</span><span class="p">),</span> |
| <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">nn</span><span class="o">.</span><span class="n">Activation</span><span class="p">(</span><span class="s1">'sigmoid'</span><span class="p">),</span> |
| <span class="p">)</span> |
| <span class="n">net</span><span class="o">.</span><span class="n">initialize</span><span class="p">(</span><span class="n">ctx</span><span class="o">=</span><span class="n">ctx</span><span class="p">)</span> |
| <span class="n">trainer</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">Trainer</span><span class="p">(</span><span class="n">net</span><span class="o">.</span><span class="n">collect_params</span><span class="p">(),</span> <span class="s1">'sgd'</span><span class="p">)</span> |
| <span class="n">loss_fn</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">loss</span><span class="o">.</span><span class="n">SigmoidBinaryCrossEntropyLoss</span><span class="p">()</span> |
| </pre></div> |
| </div> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="o">%%</span><span class="n">timeit</span> |
| <span class="n">data_iter</span><span class="o">.</span><span class="n">reset</span><span class="p">()</span> |
| <span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">data_iter</span><span class="p">:</span> |
| <span class="n">data</span> <span class="o">=</span> <span class="n">batch</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> |
| <span class="n">data</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">as_in_context</span><span class="p">(</span><span class="n">ctx</span><span class="p">)</span> |
| <span class="n">label</span> <span class="o">=</span> <span class="n">batch</span><span class="o">.</span><span class="n">label</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_in_context</span><span class="p">(</span><span class="n">ctx</span><span class="p">)</span> |
| <span class="k">with</span> <span class="n">mx</span><span class="o">.</span><span class="n">autograd</span><span class="o">.</span><span class="n">record</span><span class="p">():</span> |
| <span class="n">pred</span> <span class="o">=</span> <span class="n">net</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> |
| <span class="n">loss</span> <span class="o">=</span> <span class="n">loss_fn</span><span class="p">(</span><span class="n">pred</span><span class="p">,</span> <span class="n">label</span><span class="p">)</span> |
| <span class="n">loss</span><span class="o">.</span><span class="n">backward</span><span class="p">()</span> |
| <span class="n">trainer</span><span class="o">.</span><span class="n">step</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> |
| <span class="n">loss</span><span class="o">.</span><span class="n">wait_to_read</span><span class="p">()</span> |
| </pre></div> |
| </div> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="mi">334</span> <span class="n">ms</span> <span class="o">+-</span> <span class="mf">16.9</span> <span class="n">ms</span> <span class="n">per</span> <span class="n">loop</span> <span class="p">(</span><span class="n">mean</span> <span class="o">+-</span> <span class="n">std</span><span class="o">.</span> <span class="n">dev</span><span class="o">.</span> <span class="n">of</span> <span class="mi">7</span> <span class="n">runs</span><span class="p">,</span> <span class="mi">1</span> <span class="n">loop</span> <span class="n">each</span><span class="p">)</span> |
| </pre></div> |
| </div> |
| <img alt="fully connected sparse grad backward" src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/ndarray/sparse/fully_connected_sparse_grad_backward.png" /> |
| <p>We can see a huge reduction in the time taken for the backward pass and parameter update step: 3.99ms vs 0.18ms. And this reduces the overall time of the epoch significantly. Our gradient consumes a much smaller amount of memory and means only a subset of parameters need updating as part of the <code class="docutils literal notranslate"><span class="pre">sgd_update</span></code> step. Some optimizers don’t support sparse gradients however, so reference the specific optimizer’s documentation for more details.</p> |
| <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">print_memory_allocation</span><span class="p">(</span><span class="n">net</span><span class="p">,</span> <span class="n">block_idxs</span><span class="o">=</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">])</span> |
| </pre></div> |
| </div> |
| <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Memory</span> <span class="n">Allocation</span> <span class="k">for</span> <span class="n">Weight</span><span class="p">:</span> |
| <span class="mf">512.000</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">99.999</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">fullyconnectedsparse1</span> |
| <span class="mf">0.004</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">0.001</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">fullyconnected2</span> |
| <span class="mf">0.000</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">0.000</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">fullyconnected3</span> |
| <span class="n">Memory</span> <span class="n">Allocation</span> <span class="k">for</span> <span class="n">Weight</span> <span class="n">Gradient</span><span class="p">:</span> |
| <span class="mf">0.059</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">93.490</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">fullyconnectedsparse1</span> |
| <span class="mf">0.004</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">6.460</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">fullyconnected2</span> |
| <span class="mf">0.000</span> <span class="n">MBs</span> <span class="p">(</span> <span class="mf">0.050</span><span class="o">%</span><span class="p">)</span> <span class="k">for</span> <span class="n">fullyconnected3</span> |
| </pre></div> |
| </div> |
| </div> |
| <div class="section" id="Advanced:-Sparse-weight"> |
| <h2>Advanced: Sparse <code class="docutils literal notranslate"><span class="pre">weight</span></code><a class="headerlink" href="#Advanced:-Sparse-weight" title="Permalink to this headline">¶</a></h2> |
| <p>You can optimize this example further by setting the weight’s <code class="docutils literal notranslate"><span class="pre">stype</span></code> to <code class="docutils literal notranslate"><span class="pre">'row_sparse'</span></code>, but whether <code class="docutils literal notranslate"><span class="pre">'row_sparse'</span></code> weights make sense or not will depends on your specific task. See <a class="reference external" href="https://github.com/apache/mxnet/blob/master/python/mxnet/gluon/contrib/nn/basic_layers.py#L118">contrib.SparseEmbedding</a> for an example of this.</p> |
| <div class="section" id="Conclusion"> |
| <h3>Conclusion<a class="headerlink" href="#Conclusion" title="Permalink to this headline">¶</a></h3> |
| <p>As part of this tutorial, we learned how to write sparse data to disk in LibSVM format and load it back in sparse batches with the <a class="reference external" href="/api/python/docs/api/mxnet/io/index.html#mxnet.io.LibSVMIter">LibSVMIter</a>. We learned how to improve the performance of Gluon’s <a class="reference external" href="/api/python/docs/api/gluon/nn/index.html#mxnet.gluon.nn.Dense">nn.Dense</a> on sparse arrays using <code class="docutils literal notranslate"><span class="pre">mx.nd.sparse</span></code>. And lastly, we set <code class="docutils literal notranslate"><span class="pre">grad_stype</span></code> to <code class="docutils literal notranslate"><span class="pre">'row_sparse'</span></code> to reduce the size of the gradient and speed up the parameter |
| update step.</p> |
| </div> |
| <div class="section" id="Recommended-Next-Steps"> |
| <h3>Recommended Next Steps<a class="headerlink" href="#Recommended-Next-Steps" title="Permalink to this headline">¶</a></h3> |
| <ul class="simple"> |
| <li><p>More detail on the <a class="reference external" href="/api/python/docs/api/ndarray/sparse/index.html#mxnet.ndarray.sparse.CSRNDArray">CSRNDArray</a> sparse array format can be found in <a class="reference external" href="/api/python/docs/tutorials/packages/ndarray/sparse/csr.html">this tutorial</a>.</p></li> |
| <li><p>More detail on the <a class="reference external" href="/api/python/docs/api/ndarray/sparse/index.html#mxnet.ndarray.sparse.RowSparseNDArray">RowSparseNDArray</a> sparse array format can be found in <a class="reference external" href="/api/python/docs/tutorials/packages/ndarray/sparse/row_sparse.html">this tutorial</a>.</p></li> |
| <li><p>Users of the Module API can see a symbolic only example in <a class="reference external" href="/api/python/docs/tutorials/packages/ndarray/sparse/train.html">this tutorial</a>.</p></li> |
| </ul> |
| <!-- INSERT SOURCE DOWNLOAD BUTTONS --></div> |
| </div> |
| </div> |
| |
| |
| <hr class="feedback-hr-top" /> |
| <div class="feedback-container"> |
| <div class="feedback-question">Did this page help you?</div> |
| <div class="feedback-answer-container"> |
| <div class="feedback-answer yes-link" data-response="yes">Yes</div> |
| <div class="feedback-answer no-link" data-response="no">No</div> |
| </div> |
| <div class="feedback-thank-you">Thanks for your feedback!</div> |
| </div> |
| <hr class="feedback-hr-bottom" /> |
| </div> |
| <div class="side-doc-outline"> |
| <div class="side-doc-outline--content"> |
| <div class="localtoc"> |
| <p class="caption"> |
| <span class="caption-text">Table Of Contents</span> |
| </p> |
| <ul> |
| <li><a class="reference internal" href="#">Sparse NDArrays with Gluon</a><ul> |
| <li><a class="reference internal" href="#Generating-Sparse-Data">Generating Sparse Data</a></li> |
| <li><a class="reference internal" href="#Writing-Sparse-Data">Writing Sparse Data</a></li> |
| <li><a class="reference internal" href="#Reading-Sparse-Data">Reading Sparse Data</a><ul> |
| <li><a class="reference internal" href="#Gluon-Models-for-Sparse-Data">Gluon Models for Sparse Data</a></li> |
| </ul> |
| </li> |
| <li><a class="reference internal" href="#Benchmark:-FullyConnected">Benchmark: <code class="docutils literal notranslate"><span class="pre">FullyConnected</span></code></a></li> |
| <li><a class="reference internal" href="#Benchmark:-FullyConnectedSparse">Benchmark: <code class="docutils literal notranslate"><span class="pre">FullyConnectedSparse</span></code></a></li> |
| <li><a class="reference internal" href="#Benchmark:-FullyConnectedSparse-with-grad_stype=row_sparse">Benchmark: <code class="docutils literal notranslate"><span class="pre">FullyConnectedSparse</span></code> with <code class="docutils literal notranslate"><span class="pre">grad_stype=row_sparse</span></code></a></li> |
| <li><a class="reference internal" href="#Advanced:-Sparse-weight">Advanced: Sparse <code class="docutils literal notranslate"><span class="pre">weight</span></code></a><ul> |
| <li><a class="reference internal" href="#Conclusion">Conclusion</a></li> |
| <li><a class="reference internal" href="#Recommended-Next-Steps">Recommended Next Steps</a></li> |
| </ul> |
| </li> |
| </ul> |
| </li> |
| </ul> |
| |
| </div> |
| </div> |
| </div> |
| |
| <div class="clearer"></div> |
| </div><div class="pagenation"> |
| <a id="button-prev" href="train.html" class="mdl-button mdl-js-button mdl-js-ripple-effect mdl-button--colored" role="botton" accesskey="P"> |
| <i class="pagenation-arrow-L fas fa-arrow-left fa-lg"></i> |
| <div class="pagenation-text"> |
| <span class="pagenation-direction">Previous</span> |
| <div>Train a Linear Regression Model with Sparse Symbols</div> |
| </div> |
| </a> |
| <a id="button-next" href="../../onnx/index.html" class="mdl-button mdl-js-button mdl-js-ripple-effect mdl-button--colored" role="botton" accesskey="N"> |
| <i class="pagenation-arrow-R fas fa-arrow-right fa-lg"></i> |
| <div class="pagenation-text"> |
| <span class="pagenation-direction">Next</span> |
| <div>ONNX</div> |
| </div> |
| </a> |
| </div> |
| <footer class="site-footer h-card"> |
| <div class="wrapper"> |
| <div class="row"> |
| <div class="col-4"> |
| <h4 class="footer-category-title">Resources</h4> |
| <ul class="contact-list"> |
| <li><a class="u-email" href="mailto:dev@mxnet.apache.org">Dev list</a></li> |
| <li><a class="u-email" href="mailto:user@mxnet.apache.org">User mailing list</a></li> |
| <li><a href="https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+Home">Developer Wiki</a></li> |
| <li><a href="https://issues.apache.org/jira/projects/MXNET/issues">Jira Tracker</a></li> |
| <li><a href="https://github.com/apache/mxnet/labels/Roadmap">Github Roadmap</a></li> |
| <li><a href="https://medium.com/apache-mxnet">Blog</a></li> |
| <li><a href="https://discuss.mxnet.io">Forum</a></li> |
| <li><a href="/community/contribute">Contribute</a></li> |
| |
| </ul> |
| </div> |
| |
| <div class="col-4"><ul class="social-media-list"><li><a href="https://github.com/apache/mxnet"><svg class="svg-icon"><use xlink:href="../../../../_static/minima-social-icons.svg#github"></use></svg> <span class="username">apache/mxnet</span></a></li><li><a href="https://www.twitter.com/apachemxnet"><svg class="svg-icon"><use xlink:href="../../../../_static/minima-social-icons.svg#twitter"></use></svg> <span class="username">apachemxnet</span></a></li><li><a href="https://youtube.com/apachemxnet"><svg class="svg-icon"><use xlink:href="../../../../_static/minima-social-icons.svg#youtube"></use></svg> <span class="username">apachemxnet</span></a></li></ul> |
| </div> |
| |
| <div class="col-4 footer-text"> |
| <p>A flexible and efficient library for deep learning.</p> |
| </div> |
| </div> |
| </div> |
| </footer> |
| |
| <footer class="site-footer2"> |
| <div class="wrapper"> |
| <div class="row"> |
| <div class="col-3"> |
| <img src="../../../../_static/apache_incubator_logo.png" class="footer-logo col-2"> |
| </div> |
| <div class="footer-bottom-warning col-9"> |
| <p>Apache MXNet is an effort undergoing incubation at <a href="http://www.apache.org/">The Apache Software Foundation</a> (ASF), <span style="font-weight:bold">sponsored by the <i>Apache Incubator</i></span>. Incubation is required |
| of all newly accepted projects until a further review indicates that the infrastructure, |
| communications, and decision making process have stabilized in a manner consistent with other |
| successful ASF projects. While incubation status is not necessarily a reflection of the completeness |
| or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. |
| </p><p>"Copyright © 2017-2018, The Apache Software Foundation Apache MXNet, MXNet, Apache, the Apache |
| feather, and the Apache MXNet project logo are either registered trademarks or trademarks of the |
| Apache Software Foundation."</p> |
| </div> |
| </div> |
| </div> |
| </footer> |
| |
| </body> |
| </html> |
