Google Git
Sign in
apache / mxnet-site / d5c5a235eef72f64ccaf224b834a20ef3287469b / . / versions / 1.9.1 / api / python / docs / tutorials / performance / backend / mkldnn / mkldnn_quantization.html
blob: b6bde833a1cd9661dab2ce35934766f8554f6cc3 [file] [log] [blame]
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta charset="utf-8" />
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta http-equiv="x-ua-compatible" content="ie=edge">
<style>
.dropdown {
position: relative;
display: inline-block;
}
.dropdown-content {
display: none;
position: absolute;
background-color: #f9f9f9;
min-width: 160px;
box-shadow: 0px 8px 16px 0px rgba(0,0,0,0.2);
padding: 12px 16px;
z-index: 1;
text-align: left;
}
.dropdown:hover .dropdown-content {
display: block;
}
.dropdown-option:hover {
color: #FF4500;
}
.dropdown-option-active {
color: #FF4500;
font-weight: lighter;
}
.dropdown-option {
color: #000000;
font-weight: lighter;
}
.dropdown-header {
color: #FFFFFF;
display: inline-flex;
}
.dropdown-caret {
width: 18px;
}
.dropdown-caret-path {
fill: #FFFFFF;
}
</style>
<title>Quantize with MKL-DNN backend &#8212; Apache MXNet documentation</title>
<link rel="stylesheet" href="../../../../_static/basic.css" type="text/css" />
<link rel="stylesheet" href="../../../../_static/pygments.css" type="text/css" />
<link rel="stylesheet" type="text/css" href="../../../../_static/mxnet.css" />
<link rel="stylesheet" href="../../../../_static/material-design-lite-1.3.0/material.blue-deep_orange.min.css" type="text/css" />
<link rel="stylesheet" href="../../../../_static/sphinx_materialdesign_theme.css" type="text/css" />
<link rel="stylesheet" href="../../../../_static/fontawesome/all.css" type="text/css" />
<link rel="stylesheet" href="../../../../_static/fonts.css" type="text/css" />
<link rel="stylesheet" href="../../../../_static/feedback.css" type="text/css" />
<script id="documentation_options" data-url_root="../../../../" src="../../../../_static/documentation_options.js"></script>
<script src="../../../../_static/jquery.js"></script>
<script src="../../../../_static/underscore.js"></script>
<script src="../../../../_static/doctools.js"></script>
<script src="../../../../_static/language_data.js"></script>
<script src="../../../../_static/matomo_analytics.js"></script>
<script src="../../../../_static/autodoc.js"></script>
<script crossorigin="anonymous" integrity="sha256-Ae2Vz/4ePdIu6ZyI/5ZGsYnb+m0JlOmKPjt6XZ9JJkA=" src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.4/require.min.js"></script>
<script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script type="text/x-mathjax-config">MathJax.Hub.Config({"tex2jax": {"inlineMath": [["$", "$"], ["\\(", "\\)"]], "processEscapes": true, "ignoreClass": "document", "processClass": "math|output_area"}})</script>
<link rel="shortcut icon" href="../../../../_static/mxnet-icon.png"/>
<link rel="index" title="Index" href="../../../../genindex.html" />
<link rel="search" title="Search" href="../../../../search.html" />
<link rel="next" title="Install MXNet with MKL-DNN" href="mkldnn_readme.html" />
<link rel="prev" title="Intel MKL-DNN" href="index.html" />
</head>
<body><header class="site-header" role="banner">
<div class="wrapper">
<a class="site-title" rel="author" href="/versions/1.9.1/"><img
src="../../../../_static/mxnet_logo.png" class="site-header-logo"></a>
<nav class="site-nav">
<input type="checkbox" id="nav-trigger" class="nav-trigger"/>
<label for="nav-trigger">
<span class="menu-icon">
<svg viewBox="0 0 18 15" width="18px" height="15px">
<path d="M18,1.484c0,0.82-0.665,1.484-1.484,1.484H1.484C0.665,2.969,0,2.304,0,1.484l0,0C0,0.665,0.665,0,1.484,0 h15.032C17.335,0,18,0.665,18,1.484L18,1.484z M18,7.516C18,8.335,17.335,9,16.516,9H1.484C0.665,9,0,8.335,0,7.516l0,0 c0-0.82,0.665-1.484,1.484-1.484h15.032C17.335,6.031,18,6.696,18,7.516L18,7.516z M18,13.516C18,14.335,17.335,15,16.516,15H1.484 C0.665,15,0,14.335,0,13.516l0,0c0-0.82,0.665-1.483,1.484-1.483h15.032C17.335,12.031,18,12.695,18,13.516L18,13.516z"/>
</svg>
</span>
</label>
<div class="trigger">
<a class="page-link" href="/versions/1.9.1/get_started">Get Started</a>
<a class="page-link" href="/versions/1.9.1/features">Features</a>
<a class="page-link" href="/versions/1.9.1/ecosystem">Ecosystem</a>
<a class="page-link page-current" href="/versions/1.9.1/api">Docs & Tutorials</a>
<a class="page-link" href="/versions/1.9.1/trusted_by">Trusted By</a>
<a class="page-link" href="https://github.com/apache/mxnet">GitHub</a>
<div class="dropdown" style="min-width:100px">
<span class="dropdown-header">Apache
<svg class="dropdown-caret" viewBox="0 0 32 32" class="icon icon-caret-bottom" aria-hidden="true"><path class="dropdown-caret-path" d="M24 11.305l-7.997 11.39L8 11.305z"></path></svg>
</span>
<div class="dropdown-content" style="min-width:250px">
<a href="https://www.apache.org/foundation/">Apache Software Foundation</a>
<a href="https://incubator.apache.org/">Apache Incubator</a>
<a href="https://www.apache.org/licenses/">License</a>
<a href="/versions/1.9.1/api/faq/security.html">Security</a>
<a href="https://privacy.apache.org/policies/privacy-policy-public.html">Privacy</a>
<a href="https://www.apache.org/events/current-event">Events</a>
<a href="https://www.apache.org/foundation/sponsorship.html">Sponsorship</a>
<a href="https://www.apache.org/foundation/thanks.html">Thanks</a>
</div>
</div>
<div class="dropdown">
<span class="dropdown-header">1.9.1
<svg class="dropdown-caret" viewBox="0 0 32 32" class="icon icon-caret-bottom" aria-hidden="true"><path class="dropdown-caret-path" d="M24 11.305l-7.997 11.39L8 11.305z"></path></svg>
</span>
<div class="dropdown-content">
<a class="dropdown-option" href="/">master</a><br>
<a class="dropdown-option-active" href="/versions/1.9.1/">1.9.1</a><br>
<a class="dropdown-option" href="/versions/1.8.0/">1.8.0</a><br>
<a class="dropdown-option" href="/versions/1.7.0/">1.7.0</a><br>
<a class="dropdown-option" href="/versions/1.6.0/">1.6.0</a><br>
<a class="dropdown-option" href="/versions/1.5.0/">1.5.0</a><br>
<a class="dropdown-option" href="/versions/1.4.1/">1.4.1</a><br>
<a class="dropdown-option" href="/versions/1.3.1/">1.3.1</a><br>
<a class="dropdown-option" href="/versions/1.2.1/">1.2.1</a><br>
<a class="dropdown-option" href="/versions/1.1.0/">1.1.0</a><br>
<a class="dropdown-option" href="/versions/1.0.0/">1.0.0</a><br>
<a class="dropdown-option" href="/versions/0.12.1/">0.12.1</a><br>
<a class="dropdown-option" href="/versions/0.11.0/">0.11.0</a>
</div>
</div>
</div>
</nav>
</div>
</header>
<div class="mdl-layout mdl-js-layout mdl-layout--fixed-header mdl-layout--fixed-drawer"><header class="mdl-layout__header mdl-layout__header--waterfall ">
<div class="mdl-layout__header-row">
<nav class="mdl-navigation breadcrumb">
<a class="mdl-navigation__link" href="../../../index.html">Python Tutorials</a><i class="material-icons">navigate_next</i>
<a class="mdl-navigation__link" href="../../index.html">Performance</a><i class="material-icons">navigate_next</i>
<a class="mdl-navigation__link" href="../index.html">Accelerated Backend Tools</a><i class="material-icons">navigate_next</i>
<a class="mdl-navigation__link" href="index.html">Intel MKL-DNN</a><i class="material-icons">navigate_next</i>
<a class="mdl-navigation__link is-active">Quantize with MKL-DNN backend</a>
</nav>
<div class="mdl-layout-spacer"></div>
<nav class="mdl-navigation">
<form class="form-inline pull-sm-right" action="../../../../search.html" method="get">
<div class="mdl-textfield mdl-js-textfield mdl-textfield--expandable mdl-textfield--floating-label mdl-textfield--align-right">
<label id="quick-search-icon" class="mdl-button mdl-js-button mdl-button--icon" for="waterfall-exp">
<i class="material-icons">search</i>
</label>
<div class="mdl-textfield__expandable-holder">
<input class="mdl-textfield__input" type="text" name="q" id="waterfall-exp" placeholder="Search" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</div>
</div>
<div class="mdl-tooltip" data-mdl-for="quick-search-icon">
Quick search
</div>
</form>
<a id="button-show-source"
class="mdl-button mdl-js-button mdl-button--icon"
href="../../../../_sources/tutorials/performance/backend/mkldnn/mkldnn_quantization.ipynb" rel="nofollow">
<i class="material-icons">code</i>
</a>
<div class="mdl-tooltip" data-mdl-for="button-show-source">
Show Source
</div>
</nav>
</div>
<div class="mdl-layout__header-row header-links">
<div class="mdl-layout-spacer"></div>
<nav class="mdl-navigation">
</nav>
</div>
</header><header class="mdl-layout__drawer">
<div class="globaltoc">
<span class="mdl-layout-title toc">Table Of Contents</span>
<nav class="mdl-navigation">
<ul class="current">
<li class="toctree-l1 current"><a class="reference internal" href="../../../index.html">Python Tutorials</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../../getting-started/index.html">Getting Started</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../getting-started/crash-course/index.html">Crash Course</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/1-ndarray.html">Manipulate data with <code class="docutils literal notranslate"><span class="pre">ndarray</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/2-nn.html">Create a neural network</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/3-autograd.html">Automatic differentiation with <code class="docutils literal notranslate"><span class="pre">autograd</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/4-train.html">Train the neural network</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/5-predict.html">Predict with a pre-trained model</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/6-use_gpus.html">Use GPUs</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../getting-started/to-mxnet/index.html">Moving to MXNet from Other Frameworks</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../getting-started/to-mxnet/pytorch.html">PyTorch vs Apache MXNet</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../getting-started/gluon_from_experiment_to_deployment.html">Gluon: from experiment to deployment</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../getting-started/logistic_regression_explained.html">Logistic regression explained</a></li>
<li class="toctree-l3"><a class="reference external" href="https://mxnet.apache.org/api/python/docs/tutorials/packages/gluon/image/mnist.html">MNIST</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../../packages/index.html">Packages</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../packages/autograd/index.html">Automatic Differentiation</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../packages/gluon/index.html">Gluon</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/gluon/blocks/index.html">Blocks</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/custom-layer.html">Custom Layers</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/custom_layer_beginners.html">Customer Layers (Beginners)</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/hybridize.html">Hybridize</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/init.html">Initialization</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/naming.html">Parameter and Block Naming</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/nn.html">Layers and Blocks</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/parameters.html">Parameter Management</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/save_load_params.html">Saving and Loading Gluon Models</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/activations/activations.html">Activation Blocks</a></li>
</ul>
</li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/gluon/data/index.html">Data Tutorials</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/data/data_augmentation.html">Image Augmentation</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/data/data_augmentation.html#Spatial-Augmentation">Spatial Augmentation</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/data/data_augmentation.html#Color-Augmentation">Color Augmentation</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/data/data_augmentation.html#Composed-Augmentations">Composed Augmentations</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/data/datasets.html">Gluon <code class="docutils literal notranslate"><span class="pre">Dataset</span></code>s and <code class="docutils literal notranslate"><span class="pre">DataLoader</span></code></a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/data/datasets.html#Using-own-data-with-included-Datasets">Using own data with included <code class="docutils literal notranslate"><span class="pre">Dataset</span></code>s</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/data/datasets.html#Using-own-data-with-custom-Datasets">Using own data with custom <code class="docutils literal notranslate"><span class="pre">Dataset</span></code>s</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/data/datasets.html#Appendix:-Upgrading-from-Module-DataIter-to-Gluon-DataLoader">Appendix: Upgrading from Module <code class="docutils literal notranslate"><span class="pre">DataIter</span></code> to Gluon <code class="docutils literal notranslate"><span class="pre">DataLoader</span></code></a></li>
</ul>
</li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/gluon/image/index.html">Image Tutorials</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/image/image-augmentation.html">Image Augmentation</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/image/info_gan.html">Image similarity search with InfoGAN</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/image/mnist.html">Handwritten Digit Recognition</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/image/pretrained_models.html">Using pre-trained models in MXNet</a></li>
</ul>
</li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/gluon/loss/index.html">Losses</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/loss/custom-loss.html">Custom Loss Blocks</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/loss/kl_divergence.html">Kullback-Leibler (KL) Divergence</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/loss/loss.html">Loss functions</a></li>
</ul>
</li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/gluon/text/index.html">Text Tutorials</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/text/gnmt.html">Google Neural Machine Translation</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/text/transformer.html">Machine Translation with Transformer</a></li>
</ul>
</li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/gluon/training/index.html">Training</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/training/fit_api_tutorial.html">MXNet Gluon Fit API</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/training/trainer.html">Trainer</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/training/learning_rates/index.html">Learning Rates</a><ul>
<li class="toctree-l6"><a class="reference internal" href="../../../packages/gluon/training/learning_rates/learning_rate_finder.html">Learning Rate Finder</a></li>
<li class="toctree-l6"><a class="reference internal" href="../../../packages/gluon/training/learning_rates/learning_rate_schedules.html">Learning Rate Schedules</a></li>
<li class="toctree-l6"><a class="reference internal" href="../../../packages/gluon/training/learning_rates/learning_rate_schedules_advanced.html">Advanced Learning Rate Schedules</a></li>
</ul>
</li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/training/normalization/index.html">Normalization Blocks</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../packages/kvstore/index.html">KVStore</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/kvstore/kvstore.html">Distributed Key-Value Store</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../packages/ndarray/index.html">NDArray</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/ndarray/01-ndarray-intro.html">An Intro: Manipulate Data the MXNet Way with NDArray</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/ndarray/02-ndarray-operations.html">NDArray Operations</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/ndarray/03-ndarray-contexts.html">NDArray Contexts</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/ndarray/gotchas_numpy_in_mxnet.html">Gotchas using NumPy in Apache MXNet</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/ndarray/sparse/index.html">Tutorials</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/ndarray/sparse/csr.html">CSRNDArray - NDArray in Compressed Sparse Row Storage Format</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/ndarray/sparse/row_sparse.html">RowSparseNDArray - NDArray for Sparse Gradient Updates</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/ndarray/sparse/train.html">Train a Linear Regression Model with Sparse Symbols</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/ndarray/sparse/train_gluon.html">Sparse NDArrays with Gluon</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../packages/onnx/index.html">ONNX</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/onnx/fine_tuning_gluon.html">Fine-tuning an ONNX model</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/onnx/inference_on_onnx_model.html">Running inference on MXNet/Gluon from an ONNX model</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/onnx/super_resolution.html">Importing an ONNX model into MXNet</a></li>
<li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/python/docs/tutorials/deploy/export/onnx.html">Export ONNX Models</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../packages/optimizer/index.html">Optimizers</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../packages/viz/index.html">Visualization</a><ul>
<li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/faq/visualize_graph">Visualize networks</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2 current"><a class="reference internal" href="../../index.html">Performance</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../../compression/index.html">Compression</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../compression/int8.html">Deploy with int-8</a></li>
<li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/faq/float16">Float16</a></li>
<li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/faq/gradient_compression">Gradient Compression</a></li>
<li class="toctree-l4"><a class="reference external" href="https://gluon-cv.mxnet.io/build/examples_deployment/int8_inference.html">GluonCV with Quantized Models</a></li>
</ul>
</li>
<li class="toctree-l3 current"><a class="reference internal" href="../index.html">Accelerated Backend Tools</a><ul class="current">
<li class="toctree-l4 current"><a class="reference internal" href="index.html">Intel MKL-DNN</a><ul class="current">
<li class="toctree-l5 current"><a class="current reference internal" href="#">Quantize with MKL-DNN backend</a></li>
<li class="toctree-l5"><a class="reference internal" href="#Improving-accuracy-with-Intel®-Neural-Compressor">Improving accuracy with Intel® Neural Compressor</a></li>
<li class="toctree-l5"><a class="reference internal" href="mkldnn_readme.html">Install MXNet with MKL-DNN</a></li>
</ul>
</li>
<li class="toctree-l4"><a class="reference internal" href="../tensorrt/index.html">TensorRT</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../tensorrt/tensorrt.html">Optimizing Deep Learning Computation Graphs with TensorRT</a></li>
</ul>
</li>
<li class="toctree-l4"><a class="reference internal" href="../tvm.html">Use TVM</a></li>
<li class="toctree-l4"><a class="reference internal" href="../profiler.html">Profiling MXNet Models</a></li>
<li class="toctree-l4"><a class="reference internal" href="../amp.html">Using AMP: Automatic Mixed Precision</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../../deploy/index.html">Deployment</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../deploy/export/index.html">Export</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../deploy/export/onnx.html">Exporting to ONNX format</a></li>
<li class="toctree-l4"><a class="reference external" href="https://gluon-cv.mxnet.io/build/examples_deployment/export_network.html">Export Gluon CV Models</a></li>
<li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/python/docs/tutorials/packages/gluon/blocks/save_load_params.html">Save / Load Parameters</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../deploy/inference/index.html">Inference</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../deploy/inference/cpp.html">Deploy into C++</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../deploy/inference/image_classification_jetson.html">Image Classication using pretrained ResNet-50 model on Jetson module</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../deploy/inference/scala.html">Deploy into a Java or Scala Environment</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../deploy/inference/wine_detector.html">Real-time Object Detection with MXNet On The Raspberry Pi</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../deploy/run-on-aws/index.html">Run on AWS</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../deploy/run-on-aws/use_ec2.html">Run on an EC2 Instance</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../deploy/run-on-aws/use_sagemaker.html">Run on Amazon SageMaker</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../deploy/run-on-aws/cloud.html">MXNet on the Cloud</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../../extend/index.html">Extend</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../extend/custom_layer.html">Custom Layers</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../extend/customop.html">Custom Numpy Operators</a></li>
<li class="toctree-l3"><a class="reference external" href="https://mxnet.apache.org/api/faq/new_op">New Operator Creation</a></li>
<li class="toctree-l3"><a class="reference external" href="https://mxnet.apache.org/api/faq/add_op_in_backend">New Operator in MXNet Backend</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../../api/index.html">Python API</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/ndarray/index.html">mxnet.ndarray</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/ndarray.html">ndarray</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/contrib/index.html">ndarray.contrib</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/image/index.html">ndarray.image</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/linalg/index.html">ndarray.linalg</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/op/index.html">ndarray.op</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/random/index.html">ndarray.random</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/register/index.html">ndarray.register</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/sparse/index.html">ndarray.sparse</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/utils/index.html">ndarray.utils</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/gluon/index.html">mxnet.gluon</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/block.html">gluon.Block</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/hybrid_block.html">gluon.HybridBlock</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/symbol_block.html">gluon.SymbolBlock</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/constant.html">gluon.Constant</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/parameter.html">gluon.Parameter</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/parameter_dict.html">gluon.ParameterDict</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/trainer.html">gluon.Trainer</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/contrib/index.html">gluon.contrib</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/data/index.html">gluon.data</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../../api/gluon/data/vision/index.html">data.vision</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../../../../api/gluon/data/vision/datasets/index.html">vision.datasets</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../../api/gluon/data/vision/transforms/index.html">vision.transforms</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/loss/index.html">gluon.loss</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/model_zoo/index.html">gluon.model_zoo.vision</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/nn/index.html">gluon.nn</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/rnn/index.html">gluon.rnn</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/utils/index.html">gluon.utils</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/autograd/index.html">mxnet.autograd</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/initializer/index.html">mxnet.initializer</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/optimizer/index.html">mxnet.optimizer</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/lr_scheduler/index.html">mxnet.lr_scheduler</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/metric/index.html">mxnet.metric</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/kvstore/index.html">mxnet.kvstore</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/symbol/index.html">mxnet.symbol</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/symbol.html">symbol</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/contrib/index.html">symbol.contrib</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/image/index.html">symbol.image</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/linalg/index.html">symbol.linalg</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/op/index.html">symbol.op</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/random/index.html">symbol.random</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/register/index.html">symbol.register</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/sparse/index.html">symbol.sparse</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/module/index.html">mxnet.module</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/contrib/index.html">mxnet.contrib</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/autograd/index.html">contrib.autograd</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/io/index.html">contrib.io</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/ndarray/index.html">contrib.ndarray</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/onnx/index.html">contrib.onnx</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/quantization/index.html">contrib.quantization</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/symbol/index.html">contrib.symbol</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/tensorboard/index.html">contrib.tensorboard</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/tensorrt/index.html">contrib.tensorrt</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/text/index.html">contrib.text</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/mxnet/index.html">mxnet</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/attribute/index.html">mxnet.attribute</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/base/index.html">mxnet.base</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/callback/index.html">mxnet.callback</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/context/index.html">mxnet.context</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/engine/index.html">mxnet.engine</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/executor/index.html">mxnet.executor</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/executor_manager/index.html">mxnet.executor_manager</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/image/index.html">mxnet.image</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/io/index.html">mxnet.io</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/kvstore_server/index.html">mxnet.kvstore_server</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/libinfo/index.html">mxnet.libinfo</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/log/index.html">mxnet.log</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/model/index.html">mxnet.model</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/monitor/index.html">mxnet.monitor</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/name/index.html">mxnet.name</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/notebook/index.html">mxnet.notebook</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/operator/index.html">mxnet.operator</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/profiler/index.html">mxnet.profiler</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/random/index.html">mxnet.random</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/recordio/index.html">mxnet.recordio</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/registry/index.html">mxnet.registry</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/rtc/index.html">mxnet.rtc</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/runtime/index.html">mxnet.runtime</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/test_utils/index.html">mxnet.test_utils</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/torch/index.html">mxnet.torch</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/util/index.html">mxnet.util</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/visualization/index.html">mxnet.visualization</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</nav>
</div>
</header>
<main class="mdl-layout__content" tabIndex="0">
<script type="text/javascript" src="../../../../_static/sphinx_materialdesign_theme.js "></script>
<script type="text/javascript" src="../../../../_static/feedback.js"></script>
<header class="mdl-layout__drawer">
<div class="globaltoc">
<span class="mdl-layout-title toc">Table Of Contents</span>
<nav class="mdl-navigation">
<ul class="current">
<li class="toctree-l1 current"><a class="reference internal" href="../../../index.html">Python Tutorials</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../../getting-started/index.html">Getting Started</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../getting-started/crash-course/index.html">Crash Course</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/1-ndarray.html">Manipulate data with <code class="docutils literal notranslate"><span class="pre">ndarray</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/2-nn.html">Create a neural network</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/3-autograd.html">Automatic differentiation with <code class="docutils literal notranslate"><span class="pre">autograd</span></code></a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/4-train.html">Train the neural network</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/5-predict.html">Predict with a pre-trained model</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../getting-started/crash-course/6-use_gpus.html">Use GPUs</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../getting-started/to-mxnet/index.html">Moving to MXNet from Other Frameworks</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../getting-started/to-mxnet/pytorch.html">PyTorch vs Apache MXNet</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../getting-started/gluon_from_experiment_to_deployment.html">Gluon: from experiment to deployment</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../getting-started/logistic_regression_explained.html">Logistic regression explained</a></li>
<li class="toctree-l3"><a class="reference external" href="https://mxnet.apache.org/api/python/docs/tutorials/packages/gluon/image/mnist.html">MNIST</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../../packages/index.html">Packages</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../packages/autograd/index.html">Automatic Differentiation</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../packages/gluon/index.html">Gluon</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/gluon/blocks/index.html">Blocks</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/custom-layer.html">Custom Layers</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/custom_layer_beginners.html">Customer Layers (Beginners)</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/hybridize.html">Hybridize</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/init.html">Initialization</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/naming.html">Parameter and Block Naming</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/nn.html">Layers and Blocks</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/parameters.html">Parameter Management</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/save_load_params.html">Saving and Loading Gluon Models</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/blocks/activations/activations.html">Activation Blocks</a></li>
</ul>
</li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/gluon/data/index.html">Data Tutorials</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/data/data_augmentation.html">Image Augmentation</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/data/data_augmentation.html#Spatial-Augmentation">Spatial Augmentation</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/data/data_augmentation.html#Color-Augmentation">Color Augmentation</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/data/data_augmentation.html#Composed-Augmentations">Composed Augmentations</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/data/datasets.html">Gluon <code class="docutils literal notranslate"><span class="pre">Dataset</span></code>s and <code class="docutils literal notranslate"><span class="pre">DataLoader</span></code></a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/data/datasets.html#Using-own-data-with-included-Datasets">Using own data with included <code class="docutils literal notranslate"><span class="pre">Dataset</span></code>s</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/data/datasets.html#Using-own-data-with-custom-Datasets">Using own data with custom <code class="docutils literal notranslate"><span class="pre">Dataset</span></code>s</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/data/datasets.html#Appendix:-Upgrading-from-Module-DataIter-to-Gluon-DataLoader">Appendix: Upgrading from Module <code class="docutils literal notranslate"><span class="pre">DataIter</span></code> to Gluon <code class="docutils literal notranslate"><span class="pre">DataLoader</span></code></a></li>
</ul>
</li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/gluon/image/index.html">Image Tutorials</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/image/image-augmentation.html">Image Augmentation</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/image/info_gan.html">Image similarity search with InfoGAN</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/image/mnist.html">Handwritten Digit Recognition</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/image/pretrained_models.html">Using pre-trained models in MXNet</a></li>
</ul>
</li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/gluon/loss/index.html">Losses</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/loss/custom-loss.html">Custom Loss Blocks</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/loss/kl_divergence.html">Kullback-Leibler (KL) Divergence</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/loss/loss.html">Loss functions</a></li>
</ul>
</li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/gluon/text/index.html">Text Tutorials</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/text/gnmt.html">Google Neural Machine Translation</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/text/transformer.html">Machine Translation with Transformer</a></li>
</ul>
</li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/gluon/training/index.html">Training</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/training/fit_api_tutorial.html">MXNet Gluon Fit API</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/training/trainer.html">Trainer</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/training/learning_rates/index.html">Learning Rates</a><ul>
<li class="toctree-l6"><a class="reference internal" href="../../../packages/gluon/training/learning_rates/learning_rate_finder.html">Learning Rate Finder</a></li>
<li class="toctree-l6"><a class="reference internal" href="../../../packages/gluon/training/learning_rates/learning_rate_schedules.html">Learning Rate Schedules</a></li>
<li class="toctree-l6"><a class="reference internal" href="../../../packages/gluon/training/learning_rates/learning_rate_schedules_advanced.html">Advanced Learning Rate Schedules</a></li>
</ul>
</li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/gluon/training/normalization/index.html">Normalization Blocks</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../packages/kvstore/index.html">KVStore</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/kvstore/kvstore.html">Distributed Key-Value Store</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../packages/ndarray/index.html">NDArray</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/ndarray/01-ndarray-intro.html">An Intro: Manipulate Data the MXNet Way with NDArray</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/ndarray/02-ndarray-operations.html">NDArray Operations</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/ndarray/03-ndarray-contexts.html">NDArray Contexts</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/ndarray/gotchas_numpy_in_mxnet.html">Gotchas using NumPy in Apache MXNet</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/ndarray/sparse/index.html">Tutorials</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/ndarray/sparse/csr.html">CSRNDArray - NDArray in Compressed Sparse Row Storage Format</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/ndarray/sparse/row_sparse.html">RowSparseNDArray - NDArray for Sparse Gradient Updates</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/ndarray/sparse/train.html">Train a Linear Regression Model with Sparse Symbols</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../packages/ndarray/sparse/train_gluon.html">Sparse NDArrays with Gluon</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../packages/onnx/index.html">ONNX</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/onnx/fine_tuning_gluon.html">Fine-tuning an ONNX model</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/onnx/inference_on_onnx_model.html">Running inference on MXNet/Gluon from an ONNX model</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../packages/onnx/super_resolution.html">Importing an ONNX model into MXNet</a></li>
<li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/python/docs/tutorials/deploy/export/onnx.html">Export ONNX Models</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../packages/optimizer/index.html">Optimizers</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../packages/viz/index.html">Visualization</a><ul>
<li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/faq/visualize_graph">Visualize networks</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2 current"><a class="reference internal" href="../../index.html">Performance</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../../compression/index.html">Compression</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../compression/int8.html">Deploy with int-8</a></li>
<li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/faq/float16">Float16</a></li>
<li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/faq/gradient_compression">Gradient Compression</a></li>
<li class="toctree-l4"><a class="reference external" href="https://gluon-cv.mxnet.io/build/examples_deployment/int8_inference.html">GluonCV with Quantized Models</a></li>
</ul>
</li>
<li class="toctree-l3 current"><a class="reference internal" href="../index.html">Accelerated Backend Tools</a><ul class="current">
<li class="toctree-l4 current"><a class="reference internal" href="index.html">Intel MKL-DNN</a><ul class="current">
<li class="toctree-l5 current"><a class="current reference internal" href="#">Quantize with MKL-DNN backend</a></li>
<li class="toctree-l5"><a class="reference internal" href="#Improving-accuracy-with-Intel®-Neural-Compressor">Improving accuracy with Intel® Neural Compressor</a></li>
<li class="toctree-l5"><a class="reference internal" href="mkldnn_readme.html">Install MXNet with MKL-DNN</a></li>
</ul>
</li>
<li class="toctree-l4"><a class="reference internal" href="../tensorrt/index.html">TensorRT</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../tensorrt/tensorrt.html">Optimizing Deep Learning Computation Graphs with TensorRT</a></li>
</ul>
</li>
<li class="toctree-l4"><a class="reference internal" href="../tvm.html">Use TVM</a></li>
<li class="toctree-l4"><a class="reference internal" href="../profiler.html">Profiling MXNet Models</a></li>
<li class="toctree-l4"><a class="reference internal" href="../amp.html">Using AMP: Automatic Mixed Precision</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../../deploy/index.html">Deployment</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../deploy/export/index.html">Export</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../deploy/export/onnx.html">Exporting to ONNX format</a></li>
<li class="toctree-l4"><a class="reference external" href="https://gluon-cv.mxnet.io/build/examples_deployment/export_network.html">Export Gluon CV Models</a></li>
<li class="toctree-l4"><a class="reference external" href="https://mxnet.apache.org/api/python/docs/tutorials/packages/gluon/blocks/save_load_params.html">Save / Load Parameters</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../deploy/inference/index.html">Inference</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../deploy/inference/cpp.html">Deploy into C++</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../deploy/inference/image_classification_jetson.html">Image Classication using pretrained ResNet-50 model on Jetson module</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../deploy/inference/scala.html">Deploy into a Java or Scala Environment</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../deploy/inference/wine_detector.html">Real-time Object Detection with MXNet On The Raspberry Pi</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../deploy/run-on-aws/index.html">Run on AWS</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../deploy/run-on-aws/use_ec2.html">Run on an EC2 Instance</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../deploy/run-on-aws/use_sagemaker.html">Run on Amazon SageMaker</a></li>
<li class="toctree-l4"><a class="reference internal" href="../../../deploy/run-on-aws/cloud.html">MXNet on the Cloud</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../../extend/index.html">Extend</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../extend/custom_layer.html">Custom Layers</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../extend/customop.html">Custom Numpy Operators</a></li>
<li class="toctree-l3"><a class="reference external" href="https://mxnet.apache.org/api/faq/new_op">New Operator Creation</a></li>
<li class="toctree-l3"><a class="reference external" href="https://mxnet.apache.org/api/faq/add_op_in_backend">New Operator in MXNet Backend</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../../api/index.html">Python API</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/ndarray/index.html">mxnet.ndarray</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/ndarray.html">ndarray</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/contrib/index.html">ndarray.contrib</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/image/index.html">ndarray.image</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/linalg/index.html">ndarray.linalg</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/op/index.html">ndarray.op</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/random/index.html">ndarray.random</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/register/index.html">ndarray.register</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/sparse/index.html">ndarray.sparse</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/ndarray/utils/index.html">ndarray.utils</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/gluon/index.html">mxnet.gluon</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/block.html">gluon.Block</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/hybrid_block.html">gluon.HybridBlock</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/symbol_block.html">gluon.SymbolBlock</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/constant.html">gluon.Constant</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/parameter.html">gluon.Parameter</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/parameter_dict.html">gluon.ParameterDict</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/trainer.html">gluon.Trainer</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/contrib/index.html">gluon.contrib</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/data/index.html">gluon.data</a><ul>
<li class="toctree-l4"><a class="reference internal" href="../../../../api/gluon/data/vision/index.html">data.vision</a><ul>
<li class="toctree-l5"><a class="reference internal" href="../../../../api/gluon/data/vision/datasets/index.html">vision.datasets</a></li>
<li class="toctree-l5"><a class="reference internal" href="../../../../api/gluon/data/vision/transforms/index.html">vision.transforms</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/loss/index.html">gluon.loss</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/model_zoo/index.html">gluon.model_zoo.vision</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/nn/index.html">gluon.nn</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/rnn/index.html">gluon.rnn</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/gluon/utils/index.html">gluon.utils</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/autograd/index.html">mxnet.autograd</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/initializer/index.html">mxnet.initializer</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/optimizer/index.html">mxnet.optimizer</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/lr_scheduler/index.html">mxnet.lr_scheduler</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/metric/index.html">mxnet.metric</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/kvstore/index.html">mxnet.kvstore</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/symbol/index.html">mxnet.symbol</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/symbol.html">symbol</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/contrib/index.html">symbol.contrib</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/image/index.html">symbol.image</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/linalg/index.html">symbol.linalg</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/op/index.html">symbol.op</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/random/index.html">symbol.random</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/register/index.html">symbol.register</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/symbol/sparse/index.html">symbol.sparse</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/module/index.html">mxnet.module</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/contrib/index.html">mxnet.contrib</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/autograd/index.html">contrib.autograd</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/io/index.html">contrib.io</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/ndarray/index.html">contrib.ndarray</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/onnx/index.html">contrib.onnx</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/quantization/index.html">contrib.quantization</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/symbol/index.html">contrib.symbol</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/tensorboard/index.html">contrib.tensorboard</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/tensorrt/index.html">contrib.tensorrt</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/contrib/text/index.html">contrib.text</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../../../api/mxnet/index.html">mxnet</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/attribute/index.html">mxnet.attribute</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/base/index.html">mxnet.base</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/callback/index.html">mxnet.callback</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/context/index.html">mxnet.context</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/engine/index.html">mxnet.engine</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/executor/index.html">mxnet.executor</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/executor_manager/index.html">mxnet.executor_manager</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/image/index.html">mxnet.image</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/io/index.html">mxnet.io</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/kvstore_server/index.html">mxnet.kvstore_server</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/libinfo/index.html">mxnet.libinfo</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/log/index.html">mxnet.log</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/model/index.html">mxnet.model</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/monitor/index.html">mxnet.monitor</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/name/index.html">mxnet.name</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/notebook/index.html">mxnet.notebook</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/operator/index.html">mxnet.operator</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/profiler/index.html">mxnet.profiler</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/random/index.html">mxnet.random</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/recordio/index.html">mxnet.recordio</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/registry/index.html">mxnet.registry</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/rtc/index.html">mxnet.rtc</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/runtime/index.html">mxnet.runtime</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/test_utils/index.html">mxnet.test_utils</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/torch/index.html">mxnet.torch</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/util/index.html">mxnet.util</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../../../api/mxnet/visualization/index.html">mxnet.visualization</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</nav>
</div>
</header>
<div class="document">
<div class="page-content" role="main">
<!--- Licensed to the Apache Software Foundation (ASF) under one --><!--- or more contributor license agreements. See the NOTICE file --><!--- distributed with this work for additional information --><!--- regarding copyright ownership. The ASF licenses this file --><!--- to you under the Apache License, Version 2.0 (the --><!--- "License"); you may not use this file except in compliance --><!--- with the License. You may obtain a copy of the License at --><!--- http://www.apache.org/licenses/LICENSE-2.0 --><!--- Unless required by applicable law or agreed to in writing, --><!--- software distributed under the License is distributed on an --><!--- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY --><!--- KIND, either express or implied. See the License for the --><!--- specific language governing permissions and limitations --><!--- under the License. --><div class="section" id="Quantize-with-MKL-DNN-backend">
<h1>Quantize with MKL-DNN backend<a class="headerlink" href="#Quantize-with-MKL-DNN-backend" title="Permalink to this headline"></a></h1>
<p>This document is to introduce how to quantize the customer models from FP32 to INT8 with Apache/MXNet toolkit and APIs under Intel CPU.</p>
<p>If you are not familiar with Apache/MXNet quantization flow, please reference <a class="reference external" href="https://medium.com/apache-mxnet/model-quantization-for-production-level-neural-network-inference-f54462ebba05">quantization blog</a> first, and the performance data is shown in <a class="reference external" href="https://github.com/apache/mxnet/tree/master/cpp-package/example/inference">Apache/MXNet C++ interface</a> and <a class="reference external" href="https://gluon-cv.mxnet.io/build/examples_deployment/int8_inference.html">GluonCV</a>.</p>
<div class="section" id="Installation-and-Prerequisites">
<h2>Installation and Prerequisites<a class="headerlink" href="#Installation-and-Prerequisites" title="Permalink to this headline"></a></h2>
<p>Installing MXNet with MKLDNN backend is an easy and essential process. You can follow <a class="reference external" href="/api/python/docs/tutorials/performance/backend/mkldnn/mkldnn_readme.html">How to build and install MXNet with MKL-DNN backend</a> to build and install MXNet from source. Also, you can install the release or nightly version via PyPi and pip directly by running:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># release version</span>
<span class="n">pip</span> <span class="n">install</span> <span class="n">mxnet</span>
<span class="c1"># latest nightly development version</span>
<span class="n">pip</span> <span class="n">install</span> <span class="o">--</span><span class="n">pre</span> <span class="s2">&quot;mxnet&lt;2&quot;</span> <span class="o">-</span><span class="n">f</span> <span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="n">dist</span><span class="o">.</span><span class="n">mxnet</span><span class="o">.</span><span class="n">io</span><span class="o">/</span><span class="n">python</span>
</pre></div>
</div>
</div>
<div class="section" id="Image-Classification-Demo">
<h2>Image Classification Demo<a class="headerlink" href="#Image-Classification-Demo" title="Permalink to this headline"></a></h2>
<p>A quantization script <a class="reference external" href="https://github.com/apache/mxnet/blob/master/example/quantization/imagenet_gen_qsym_mkldnn.py">imagenet_gen_qsym_mkldnn.py</a> has been designed to launch quantization for image-classification models. This script is integrated with <a class="reference external" href="https://gluon-cv.mxnet.io/model_zoo/classification.html">Gluon-CV modelzoo</a>, so that all pre-trained models can be downloaded from Gluon-CV and then converted for quantization. For details, you can refer <a class="reference external" href="https://github.com/apache/mxnet/blob/master/example/quantization/README.md">Model Quantization with Calibration
Examples</a>.</p>
</div>
<div class="section" id="Integrate-Quantization-Flow-to-Your-Project">
<h2>Integrate Quantization Flow to Your Project<a class="headerlink" href="#Integrate-Quantization-Flow-to-Your-Project" title="Permalink to this headline"></a></h2>
<p>Quantization flow works for both symbolic and Gluon models. If you’re using Gluon, you can first refer <a class="reference external" href="/api/python/docs/tutorials/packages/gluon/blocks/save_load_params.html">Saving and Loading Gluon Models</a> to hybridize your computation graph and export it as a symbol before running quantization.</p>
<p>In general, the quantization flow includes 4 steps. The user can get the acceptable accuracy from step 1 to 3 with minimum effort. Most of thing in this stage is out-of-box and the data scientists and researchers only need to focus on how to represent data and layers in their model. After a quantized model is generated, you may want to deploy it online and the performance will be the next key point. Thus, step 4, calibration, can improve the performance a lot by reducing lots of runtime
calculation.</p>
<img alt="quantization flow" src="https://github.com/dmlc/web-data/raw/master/mxnet/tutorials/mkldnn/quantization/quantization.png" />
<p>Now, we are going to take Gluon ResNet18 as an example to show how each step work.</p>
<div class="section" id="Initialize-Model">
<h3>Initialize Model<a class="headerlink" href="#Initialize-Model" title="Permalink to this headline"></a></h3>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">mxnet</span> <span class="k">as</span> <span class="nn">mx</span>
<span class="kn">from</span> <span class="nn">mxnet.gluon.model_zoo</span> <span class="kn">import</span> <span class="n">vision</span>
<span class="kn">from</span> <span class="nn">mxnet.contrib.quantization</span> <span class="kn">import</span> <span class="o">*</span>
<span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">()</span>
<span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s1">&#39;logger&#39;</span><span class="p">)</span>
<span class="n">logger</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="n">batch_shape</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">224</span><span class="p">,</span> <span class="mi">224</span><span class="p">)</span>
<span class="n">resnet18</span> <span class="o">=</span> <span class="n">vision</span><span class="o">.</span><span class="n">resnet18_v1</span><span class="p">(</span><span class="n">pretrained</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">resnet18</span><span class="o">.</span><span class="n">hybridize</span><span class="p">()</span>
<span class="n">resnet18</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">batch_shape</span><span class="p">))</span>
<span class="n">resnet18</span><span class="o">.</span><span class="n">export</span><span class="p">(</span><span class="s1">&#39;resnet18_v1&#39;</span><span class="p">)</span>
<span class="n">sym</span><span class="p">,</span> <span class="n">arg_params</span><span class="p">,</span> <span class="n">aux_params</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">model</span><span class="o">.</span><span class="n">load_checkpoint</span><span class="p">(</span><span class="s1">&#39;resnet18_v1&#39;</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="c1"># (optional) visualize float32 model</span>
<span class="n">mx</span><span class="o">.</span><span class="n">viz</span><span class="o">.</span><span class="n">plot_network</span><span class="p">(</span><span class="n">sym</span><span class="p">)</span>
</pre></div>
</div>
<p>First, we download resnet18-v1 model from gluon modelzoo and export it as a symbol. You can visualize float32 model. Below is a raw residual block.</p>
<img alt="float32 model" src="https://github.com/dmlc/web-data/raw/master/mxnet/tutorials/mkldnn/quantization/fp32_raw.png" />
<div class="section" id="Model-Fusion">
<h4>Model Fusion<a class="headerlink" href="#Model-Fusion" title="Permalink to this headline"></a></h4>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">sym</span> <span class="o">=</span> <span class="n">sym</span><span class="o">.</span><span class="n">get_backend_symbol</span><span class="p">(</span><span class="s1">&#39;MKLDNN_QUANTIZE&#39;</span><span class="p">)</span>
<span class="c1"># (optional) visualize fused float32 model</span>
<span class="n">mx</span><span class="o">.</span><span class="n">viz</span><span class="o">.</span><span class="n">plot_network</span><span class="p">(</span><span class="n">sym</span><span class="p">)</span>
</pre></div>
</div>
<p>It’s important to add this line to enable graph fusion before quantization to get better performance. Below is a fused residual block. Batchnorm, Activation and elemwise_add are fused into Convolution.</p>
<img alt="float32 fused model" src="https://github.com/dmlc/web-data/raw/master/mxnet/tutorials/mkldnn/quantization/fp32_fusion.png" />
</div>
</div>
<div class="section" id="Quantize-Model">
<h3>Quantize Model<a class="headerlink" href="#Quantize-Model" title="Permalink to this headline"></a></h3>
<p>A python interface <code class="docutils literal notranslate"><span class="pre">quantize_graph</span></code> is provided for the user. Thus, it is very flexible for the data scientist to construct the expected models based on different requirements in a real deployment.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="c1"># quantize configs</span>
<span class="c1"># set exclude layers</span>
<span class="n">excluded_names</span> <span class="o">=</span> <span class="p">[]</span>
<span class="c1"># set calib mode.</span>
<span class="n">calib_mode</span> <span class="o">=</span> <span class="s1">&#39;none&#39;</span>
<span class="c1"># set calib_layer</span>
<span class="n">calib_layer</span> <span class="o">=</span> <span class="kc">None</span>
<span class="c1"># set quantized_dtype</span>
<span class="n">quantized_dtype</span> <span class="o">=</span> <span class="s1">&#39;auto&#39;</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">&#39;Quantizing FP32 model Resnet18-V1&#39;</span><span class="p">)</span>
<span class="n">qsym</span><span class="p">,</span> <span class="n">qarg_params</span><span class="p">,</span> <span class="n">aux_params</span><span class="p">,</span> <span class="n">collector</span> <span class="o">=</span> <span class="n">quantize_graph</span><span class="p">(</span><span class="n">sym</span><span class="o">=</span><span class="n">sym</span><span class="p">,</span> <span class="n">arg_params</span><span class="o">=</span><span class="n">arg_params</span><span class="p">,</span> <span class="n">aux_params</span><span class="o">=</span><span class="n">aux_params</span><span class="p">,</span>
<span class="n">excluded_sym_names</span><span class="o">=</span><span class="n">excluded_names</span><span class="p">,</span>
<span class="n">calib_mode</span><span class="o">=</span><span class="n">calib_mode</span><span class="p">,</span> <span class="n">calib_layer</span><span class="o">=</span><span class="n">calib_layer</span><span class="p">,</span>
<span class="n">quantized_dtype</span><span class="o">=</span><span class="n">quantized_dtype</span><span class="p">,</span> <span class="n">logger</span><span class="o">=</span><span class="n">logger</span><span class="p">)</span>
<span class="c1"># (optional) visualize quantized model</span>
<span class="n">mx</span><span class="o">.</span><span class="n">viz</span><span class="o">.</span><span class="n">plot_network</span><span class="p">(</span><span class="n">qsym</span><span class="p">)</span>
<span class="c1"># save quantized model</span>
<span class="n">mx</span><span class="o">.</span><span class="n">model</span><span class="o">.</span><span class="n">save_checkpoint</span><span class="p">(</span><span class="s1">&#39;quantized-resnet18_v1&#39;</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">qsym</span><span class="p">,</span> <span class="n">qarg_params</span><span class="p">,</span> <span class="n">aux_params</span><span class="p">)</span>
</pre></div>
</div>
<p>By applying <code class="docutils literal notranslate"><span class="pre">quantize_graph</span></code> to the symbolic model, a new quantized model can be generated, named <code class="docutils literal notranslate"><span class="pre">qsym</span></code> along with its parameters. We can see <code class="docutils literal notranslate"><span class="pre">_contrib_requantize</span></code> operators are inserted after <code class="docutils literal notranslate"><span class="pre">Convolution</span></code> to convert the INT32 output to FP32.</p>
<img alt="none calibrated model" src="https://github.com/dmlc/web-data/raw/master/mxnet/tutorials/mkldnn/quantization/none_calib.png" />
<p>Below table gives some descriptions.</p>
<table class="docutils align-default">
<colgroup>
<col style="width: 2%" />
<col style="width: 2%" />
<col style="width: 96%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>param</p></th>
<th class="head"><p>type</p></th>
<th class="head"><p>description</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p>excluded
_sym_names</p></td>
<td><p>list of
strings</p></td>
<td><p>A list of strings representing the names of the symbols that users want to excluding from being quantized.</p></td>
</tr>
<tr class="row-odd"><td><p>calib_mode</p></td>
<td><p>str</p></td>
<td><p>If calib_mode=‘none’, no calibration will be used and the thresholds for requantization after the corresponding layers will be calculated at runtime by calling min and max operators. The quantized models generated in this mode are normally 10-20% slower than those with calibrations during inference.If calib_mode=‘naive’, the min and max values of the layer outputs from a calibration dataset will be directly taken as the thresholds for quantization.If
calib_mode=‘entropy’, the thresholds for quantization will be derived such that the KL divergence between the distributions of FP32 layer outputs and quantized layer outputs is minimized based upon the calibration dataset.</p></td>
</tr>
<tr class="row-even"><td><p>c
alib_layer</p></td>
<td><p>function</p></td>
<td><p>Given a layer’s output name in string, return True or False for deciding whether to calibrate this layer.If yes, the statistics of the layer’s output will be collected; otherwise, no information of the layer’s output will be collected.If not provided, all the layers’ outputs that need requantization will be collected.</p></td>
</tr>
<tr class="row-odd"><td><p>quant
ized_dtype</p></td>
<td><p>str</p></td>
<td><p>The quantized destination type for input data. Currently support ‘int8’, ‘uint8’ and ‘auto’.‘auto’ means automatically select output type according to calibration result.</p></td>
</tr>
</tbody>
</table>
</div>
<div class="section" id="Evaluate-&amp;-Tune">
<h3>Evaluate &amp; Tune<a class="headerlink" href="#Evaluate-&-Tune" title="Permalink to this headline"></a></h3>
<p>Now, you get a pair of quantized symbol and params file for inference. For Gluon inference, only difference is to load model and params by a SymbolBlock as below example:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">quantized_net</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">gluon</span><span class="o">.</span><span class="n">SymbolBlock</span><span class="o">.</span><span class="n">imports</span><span class="p">(</span><span class="s1">&#39;quantized-resnet18_v1-symbol.json&#39;</span><span class="p">,</span> <span class="s1">&#39;data&#39;</span><span class="p">,</span> <span class="s1">&#39;quantized-resnet18_v1-0000.params&#39;</span><span class="p">)</span>
<span class="n">quantized_net</span><span class="o">.</span><span class="n">hybridize</span><span class="p">(</span><span class="n">static_shape</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">static_alloc</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">batch_size</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">nd</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">batch_size</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">224</span><span class="p">,</span><span class="mi">224</span><span class="p">))</span>
<span class="n">quantized_net</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
</pre></div>
</div>
<p>Now, you can get the accuracy from a quantized network. Furthermore, you can try to select different layers or OPs to be quantized by <code class="docutils literal notranslate"><span class="pre">excluded_sym_names</span></code> parameter and figure out an acceptable accuracy.</p>
</div>
<div class="section" id="Calibrate-Model-(optional-for-performance)">
<h3>Calibrate Model (optional for performance)<a class="headerlink" href="#Calibrate-Model-(optional-for-performance)" title="Permalink to this headline"></a></h3>
<p>The quantized model generated in previous steps can be very slow during inference since it will calculate min and max at runtime. We recommend using offline calibration for better performance by setting <code class="docutils literal notranslate"><span class="pre">calib_mode</span></code> to <code class="docutils literal notranslate"><span class="pre">naive</span></code> or <code class="docutils literal notranslate"><span class="pre">entropy</span></code>. And then calling <code class="docutils literal notranslate"><span class="pre">set_monitor_callback</span></code> api to collect layer information with a subset of the validation datasets before int8 inference.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="c1"># quantization configs</span>
<span class="c1"># set exclude layers</span>
<span class="n">excluded_names</span> <span class="o">=</span> <span class="p">[]</span>
<span class="c1"># set calib mode.</span>
<span class="n">calib_mode</span> <span class="o">=</span> <span class="s1">&#39;naive&#39;</span>
<span class="c1"># set calib_layer</span>
<span class="n">calib_layer</span> <span class="o">=</span> <span class="kc">None</span>
<span class="c1"># set quantized_dtype</span>
<span class="n">quantized_dtype</span> <span class="o">=</span> <span class="s1">&#39;auto&#39;</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">&#39;Quantizing FP32 model resnet18-V1&#39;</span><span class="p">)</span>
<span class="n">cqsym</span><span class="p">,</span> <span class="n">cqarg_params</span><span class="p">,</span> <span class="n">aux_params</span><span class="p">,</span> <span class="n">collector</span> <span class="o">=</span> <span class="n">quantize_graph</span><span class="p">(</span><span class="n">sym</span><span class="o">=</span><span class="n">sym</span><span class="p">,</span> <span class="n">arg_params</span><span class="o">=</span><span class="n">arg_params</span><span class="p">,</span> <span class="n">aux_params</span><span class="o">=</span><span class="n">aux_params</span><span class="p">,</span>
<span class="n">excluded_sym_names</span><span class="o">=</span><span class="n">excluded_names</span><span class="p">,</span>
<span class="n">calib_mode</span><span class="o">=</span><span class="n">calib_mode</span><span class="p">,</span> <span class="n">calib_layer</span><span class="o">=</span><span class="n">calib_layer</span><span class="p">,</span>
<span class="n">quantized_dtype</span><span class="o">=</span><span class="n">quantized_dtype</span><span class="p">,</span> <span class="n">logger</span><span class="o">=</span><span class="n">logger</span><span class="p">)</span>
<span class="c1"># download imagenet validation dataset</span>
<span class="n">mx</span><span class="o">.</span><span class="n">test_utils</span><span class="o">.</span><span class="n">download</span><span class="p">(</span><span class="s1">&#39;http://data.mxnet.io/data/val_256_q90.rec&#39;</span><span class="p">,</span> <span class="s1">&#39;dataset.rec&#39;</span><span class="p">)</span>
<span class="c1"># set rgb info for data</span>
<span class="n">mean_std</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;mean_r&#39;</span><span class="p">:</span> <span class="mf">123.68</span><span class="p">,</span> <span class="s1">&#39;mean_g&#39;</span><span class="p">:</span> <span class="mf">116.779</span><span class="p">,</span> <span class="s1">&#39;mean_b&#39;</span><span class="p">:</span> <span class="mf">103.939</span><span class="p">,</span> <span class="s1">&#39;std_r&#39;</span><span class="p">:</span> <span class="mf">58.393</span><span class="p">,</span> <span class="s1">&#39;std_g&#39;</span><span class="p">:</span> <span class="mf">57.12</span><span class="p">,</span> <span class="s1">&#39;std_b&#39;</span><span class="p">:</span> <span class="mf">57.375</span><span class="p">}</span>
<span class="c1"># set batch size</span>
<span class="n">batch_size</span> <span class="o">=</span> <span class="mi">16</span>
<span class="c1"># create DataIter</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ImageRecordIter</span><span class="p">(</span><span class="n">path_imgrec</span><span class="o">=</span><span class="s1">&#39;dataset.rec&#39;</span><span class="p">,</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">batch_size</span><span class="p">,</span> <span class="n">data_shape</span><span class="o">=</span><span class="n">batch_shape</span><span class="p">[</span><span class="mi">1</span><span class="p">:],</span> <span class="n">rand_crop</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">rand_mirror</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="o">**</span><span class="n">mean_std</span><span class="p">)</span>
<span class="c1"># create module</span>
<span class="n">mod</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">mod</span><span class="o">.</span><span class="n">Module</span><span class="p">(</span><span class="n">symbol</span><span class="o">=</span><span class="n">sym</span><span class="p">,</span> <span class="n">label_names</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">context</span><span class="o">=</span><span class="n">mx</span><span class="o">.</span><span class="n">cpu</span><span class="p">())</span>
<span class="n">mod</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="n">for_training</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">data_shapes</span><span class="o">=</span><span class="n">data</span><span class="o">.</span><span class="n">provide_data</span><span class="p">,</span> <span class="n">label_shapes</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
<span class="n">mod</span><span class="o">.</span><span class="n">set_params</span><span class="p">(</span><span class="n">arg_params</span><span class="p">,</span> <span class="n">aux_params</span><span class="p">)</span>
<span class="c1"># calibration configs</span>
<span class="c1"># set num_calib_batches</span>
<span class="n">num_calib_batches</span> <span class="o">=</span> <span class="mi">5</span>
<span class="n">max_num_examples</span> <span class="o">=</span> <span class="n">num_calib_batches</span> <span class="o">*</span> <span class="n">batch_size</span>
<span class="c1"># monitor FP32 Inference</span>
<span class="n">mod</span><span class="o">.</span><span class="n">_exec_group</span><span class="o">.</span><span class="n">execs</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">set_monitor_callback</span><span class="p">(</span><span class="n">collector</span><span class="o">.</span><span class="n">collect</span><span class="p">,</span> <span class="n">monitor_all</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">num_batches</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">num_examples</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">data</span><span class="p">:</span>
<span class="n">mod</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="n">data_batch</span><span class="o">=</span><span class="n">batch</span><span class="p">,</span> <span class="n">is_train</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">num_batches</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">num_examples</span> <span class="o">+=</span> <span class="n">batch_size</span>
<span class="k">if</span> <span class="n">num_examples</span> <span class="o">&gt;=</span> <span class="n">max_num_examples</span><span class="p">:</span>
<span class="k">break</span>
<span class="k">if</span> <span class="n">logger</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">&quot;Collected statistics from </span><span class="si">%d</span><span class="s2"> batches with batch_size=</span><span class="si">%d</span><span class="s2">&quot;</span>
<span class="o">%</span> <span class="p">(</span><span class="n">num_batches</span><span class="p">,</span> <span class="n">batch_size</span><span class="p">))</span>
</pre></div>
</div>
<p>After that, layer information will be filled into the <code class="docutils literal notranslate"><span class="pre">collector</span></code> returned by <code class="docutils literal notranslate"><span class="pre">quantize_graph</span></code> api. Then, you need to write the layer information into int8 model by calling <code class="docutils literal notranslate"><span class="pre">calib_graph</span></code> api.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="c1"># write scaling factor into quantized symbol</span>
<span class="n">cqsym</span><span class="p">,</span> <span class="n">cqarg_params</span><span class="p">,</span> <span class="n">aux_params</span> <span class="o">=</span> <span class="n">calib_graph</span><span class="p">(</span><span class="n">qsym</span><span class="o">=</span><span class="n">cqsym</span><span class="p">,</span> <span class="n">arg_params</span><span class="o">=</span><span class="n">arg_params</span><span class="p">,</span> <span class="n">aux_params</span><span class="o">=</span><span class="n">aux_params</span><span class="p">,</span>
<span class="n">collector</span><span class="o">=</span><span class="n">collector</span><span class="p">,</span> <span class="n">calib_mode</span><span class="o">=</span><span class="n">calib_mode</span><span class="p">,</span>
<span class="n">quantized_dtype</span><span class="o">=</span><span class="n">quantized_dtype</span><span class="p">,</span> <span class="n">logger</span><span class="o">=</span><span class="n">logger</span><span class="p">)</span>
<span class="c1"># (optional) visualize quantized model</span>
<span class="n">mx</span><span class="o">.</span><span class="n">viz</span><span class="o">.</span><span class="n">plot_network</span><span class="p">(</span><span class="n">cqsym</span><span class="p">)</span>
</pre></div>
</div>
<p>Below is a quantized residual block with naive calibration. We can see <code class="docutils literal notranslate"><span class="pre">min_calib_range</span></code> and <code class="docutils literal notranslate"><span class="pre">max_calib_range</span></code> are written into <code class="docutils literal notranslate"><span class="pre">_contrib_requantize</span></code> operators.</p>
<img alt="naive calibrated model" src="https://github.com/dmlc/web-data/raw/master/mxnet/tutorials/mkldnn/quantization/naive_calib.png" />
<p>When you get a quantized model with calibration, keeping sure to call fusion api again since this can fuse some <code class="docutils literal notranslate"><span class="pre">requantize</span></code> or <code class="docutils literal notranslate"><span class="pre">dequantize</span></code> operators for further performance improvement.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="c1"># perform post-quantization fusion</span>
<span class="n">cqsym</span> <span class="o">=</span> <span class="n">cqsym</span><span class="o">.</span><span class="n">get_backend_symbol</span><span class="p">(</span><span class="s1">&#39;MKLDNN_QUANTIZE&#39;</span><span class="p">)</span>
<span class="c1"># (optional) visualize post-quantized model</span>
<span class="n">mx</span><span class="o">.</span><span class="n">viz</span><span class="o">.</span><span class="n">plot_network</span><span class="p">(</span><span class="n">cqsym</span><span class="p">)</span>
<span class="c1"># save quantized model</span>
<span class="n">mx</span><span class="o">.</span><span class="n">model</span><span class="o">.</span><span class="n">save_checkpoint</span><span class="p">(</span><span class="s1">&#39;quantized-resnet18_v1&#39;</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">cqsym</span><span class="p">,</span> <span class="n">cqarg_params</span><span class="p">,</span> <span class="n">aux_params</span><span class="p">)</span>
</pre></div>
</div>
<p>Below is a post-quantized residual block. We can see <code class="docutils literal notranslate"><span class="pre">_contrib_requantize</span></code> operators are fused into <code class="docutils literal notranslate"><span class="pre">Convolution</span></code> operators.</p>
<img alt="post-quantized model" src="https://github.com/dmlc/web-data/raw/master/mxnet/tutorials/mkldnn/quantization/post_quantize.png" />
<p>BTW, You can also modify the <code class="docutils literal notranslate"><span class="pre">min_calib_range</span></code> and <code class="docutils literal notranslate"><span class="pre">max_calib_range</span></code> in the JSON file directly.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="p">{</span>
<span class="s2">&quot;op&quot;</span><span class="p">:</span> <span class="s2">&quot;_sg_mkldnn_conv&quot;</span><span class="p">,</span>
<span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;quantized_sg_mkldnn_conv_bn_act_6&quot;</span><span class="p">,</span>
<span class="s2">&quot;attrs&quot;</span><span class="p">:</span> <span class="p">{</span>
<span class="s2">&quot;max_calib_range&quot;</span><span class="p">:</span> <span class="s2">&quot;3.562147&quot;</span><span class="p">,</span>
<span class="s2">&quot;min_calib_range&quot;</span><span class="p">:</span> <span class="s2">&quot;0.000000&quot;</span><span class="p">,</span>
<span class="s2">&quot;quantized&quot;</span><span class="p">:</span> <span class="s2">&quot;true&quot;</span><span class="p">,</span>
<span class="s2">&quot;with_act&quot;</span><span class="p">:</span> <span class="s2">&quot;true&quot;</span><span class="p">,</span>
<span class="s2">&quot;with_bn&quot;</span><span class="p">:</span> <span class="s2">&quot;true&quot;</span>
<span class="p">},</span>
<span class="o">......</span>
</pre></div>
</div>
</div>
<div class="section" id="Tips-for-Model-Calibration">
<h3>Tips for Model Calibration<a class="headerlink" href="#Tips-for-Model-Calibration" title="Permalink to this headline"></a></h3>
<div class="section" id="Accuracy-Tuning">
<h4>Accuracy Tuning<a class="headerlink" href="#Accuracy-Tuning" title="Permalink to this headline"></a></h4>
<ul class="simple">
<li><p>Try to use <code class="docutils literal notranslate"><span class="pre">entropy</span></code> calib mode;</p></li>
<li><p>Try to exclude some layers which may cause obvious accuracy drop;</p></li>
<li><p>Change calibration dataset by setting different <code class="docutils literal notranslate"><span class="pre">num_calib_batches</span></code> or shuffle your validation dataset;</p></li>
<li><p>Use Intel® Neural Compressor (<a class="reference external" href="#Improving-accuracy-with-Intel-Neural-Compressor">see below</a>)</p></li>
</ul>
</div>
<div class="section" id="Performance-Tuning">
<h4>Performance Tuning<a class="headerlink" href="#Performance-Tuning" title="Permalink to this headline"></a></h4>
<ul class="simple">
<li><p>Keep sure to perform graph fusion before quantization;</p></li>
<li><p>If lots of <code class="docutils literal notranslate"><span class="pre">requantize</span></code> layers exist, keep sure to perform post-quantization fusion after calibration;</p></li>
<li><p>Compare the MXNet profile or <code class="docutils literal notranslate"><span class="pre">MKLDNN_VERBOSE</span></code> of float32 and int8 inference;</p></li>
</ul>
</div>
</div>
</div>
<div class="section" id="Deploy-with-Python/C++">
<h2>Deploy with Python/C++<a class="headerlink" href="#Deploy-with-Python/C++" title="Permalink to this headline"></a></h2>
<p>MXNet also supports deploy quantized models with C++. Refer <a class="reference external" href="https://github.com/apache/mxnet/blob/master/cpp-package/README.md">MXNet C++ Package</a> for more details.</p>
</div>
</div>
<div class="section" id="Improving-accuracy-with-Intel®-Neural-Compressor">
<h1>Improving accuracy with Intel® Neural Compressor<a class="headerlink" href="#Improving-accuracy-with-Intel®-Neural-Compressor" title="Permalink to this headline"></a></h1>
<p>The accuracy of a model can decrease as a result of quantization. When the accuracy drop is significant, we can try to manually find a better quantization configuration (exclude some layers, try different calibration methods, etc.), but for bigger models this might prove to be a difficult and time consuming task. <a class="reference external" href="https://github.com/intel/neural-compressor">Intel® Neural Compressor</a> (INC) tries to automate this process using several tuning heuristics, which aim to find the quantization
configuration that satisfies the specified accuracy requirement.</p>
<p><strong>NOTE:</strong></p>
<p>Most tuning strategies will try different configurations on an evaluation dataset in order to find out how each layer affects the accuracy of the model. This means that for larger models, it may take a long time to find a solution (as the tuning space is usually larger and the evaluation itself takes longer).</p>
<div class="section" id="Installation-and-Prerequisites">
<h2>Installation and Prerequisites<a class="headerlink" href="#Installation-and-Prerequisites" title="Permalink to this headline"></a></h2>
<ul>
<li><p>Install MXNet with MKLDNN enabled as described in the <a class="reference external" href="#Installation-and-Prerequisites">previous section</a>.</p></li>
<li><p>Install Intel® Neural Compressor:</p>
<p>Use one of the commands below to install INC (supported python versions are: 3.6, 3.7, 3.8, 3.9):</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># install stable version from pip</span>
pip<span class="w"> </span>install<span class="w"> </span>neural-compressor
<span class="c1"># install nightly version from pip</span>
pip<span class="w"> </span>install<span class="w"> </span>-i<span class="w"> </span>https://test.pypi.org/simple/<span class="w"> </span>neural-compressor
<span class="c1"># install stable version from conda</span>
conda<span class="w"> </span>install<span class="w"> </span>neural-compressor<span class="w"> </span>-c<span class="w"> </span>conda-forge<span class="w"> </span>-c<span class="w"> </span>intel
</pre></div>
</div>
</li>
</ul>
</div>
<div class="section" id="Configuration-file">
<h2>Configuration file<a class="headerlink" href="#Configuration-file" title="Permalink to this headline"></a></h2>
<p>Quantization tuning process can be customized in the yaml configuration file. Below is a simple example:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="c1"># cnn.yaml</span>
<span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1.0</span>
<span class="nt">model</span><span class="p">:</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">cnn</span>
<span class="w"> </span><span class="nt">framework</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mxnet</span>
<span class="nt">quantization</span><span class="p">:</span>
<span class="w"> </span><span class="nt">calibration</span><span class="p">:</span>
<span class="w"> </span><span class="nt">sampling_size</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">160</span><span class="w"> </span><span class="c1"># number of samples for calibration</span>
<span class="nt">tuning</span><span class="p">:</span>
<span class="w"> </span><span class="nt">strategy</span><span class="p">:</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">basic</span>
<span class="w"> </span><span class="nt">accuracy_criterion</span><span class="p">:</span>
<span class="w"> </span><span class="nt">relative</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">0.01</span>
<span class="w"> </span><span class="nt">exit_policy</span><span class="p">:</span>
<span class="w"> </span><span class="nt">timeout</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">0</span>
<span class="w"> </span><span class="nt">random_seed</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">9527</span>
</pre></div>
</div>
<p>We are using the <code class="docutils literal notranslate"><span class="pre">basic</span></code> strategy, but you could also try out different ones. <a class="reference external" href="https://github.com/intel/neural-compressor/blob/master/docs/tuning_strategies.md">Here</a> you can find a list of strategies available in INC and details of how they work. You can also add your own strategy if the existing ones do not suit your needs.</p>
<p>Since the value of <code class="docutils literal notranslate"><span class="pre">timeout</span></code> is 0, INC will run until it finds a configuration that satisfies the accuracy criterion and then exit. Depending on the strategy this may not be ideal, as sometimes it would be better to further explore the tuning space to find a superior configuration both in terms of accuracy and speed. To achieve this, we can set a specific <code class="docutils literal notranslate"><span class="pre">timeout</span></code> value, which will tell INC how long (in seconds) it should run.</p>
<p>For more information about the configuration file, see the <a class="reference external" href="https://github.com/intel/neural-compressor/blob/master/neural_compressor/template/ptq.yaml">template</a> from the official INC repo. Keep in mind that only the <code class="docutils literal notranslate"><span class="pre">post</span> <span class="pre">training</span> <span class="pre">quantization</span></code> is currently supported for MXNet.</p>
</div>
<div class="section" id="Model-quantization-and-tuning">
<h2>Model quantization and tuning<a class="headerlink" href="#Model-quantization-and-tuning" title="Permalink to this headline"></a></h2>
<p>In general, Intel® Neural Compressor requires 4 elements in order to run: 1. Config file - like the example above 2. Model to be quantized 3. Calibration dataloader 4. Evaluation function - a function that takes a model as an argument and returns the accuracy it achieves on a certain evaluation dataset.</p>
<div class="section" id="Quantizing-ResNet">
<h3>Quantizing ResNet<a class="headerlink" href="#Quantizing-ResNet" title="Permalink to this headline"></a></h3>
<p>The previous sections described how to quantize ResNet using the native MXNet quantization. This example shows how we can achieve the same (with the auto-tuning) using INC.</p>
<ol class="arabic simple">
<li><p>Get the model</p></li>
</ol>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">mxnet</span> <span class="k">as</span> <span class="nn">mx</span>
<span class="kn">from</span> <span class="nn">mxnet.gluon.model_zoo</span> <span class="kn">import</span> <span class="n">vision</span>
<span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">()</span>
<span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s1">&#39;logger&#39;</span><span class="p">)</span>
<span class="n">logger</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="n">batch_shape</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">224</span><span class="p">,</span> <span class="mi">224</span><span class="p">)</span>
<span class="n">resnet18</span> <span class="o">=</span> <span class="n">vision</span><span class="o">.</span><span class="n">resnet18_v1</span><span class="p">(</span><span class="n">pretrained</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</pre></div>
</div>
<ol class="arabic simple" start="2">
<li><p>Prepare the dataset:</p></li>
</ol>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">mx</span><span class="o">.</span><span class="n">test_utils</span><span class="o">.</span><span class="n">download</span><span class="p">(</span><span class="s1">&#39;http://data.mxnet.io/data/val_256_q90.rec&#39;</span><span class="p">,</span> <span class="s1">&#39;data/val_256_q90.rec&#39;</span><span class="p">)</span>
<span class="n">batch_size</span> <span class="o">=</span> <span class="mi">16</span>
<span class="n">mean_std</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;mean_r&#39;</span><span class="p">:</span> <span class="mf">123.68</span><span class="p">,</span> <span class="s1">&#39;mean_g&#39;</span><span class="p">:</span> <span class="mf">116.779</span><span class="p">,</span> <span class="s1">&#39;mean_b&#39;</span><span class="p">:</span> <span class="mf">103.939</span><span class="p">,</span>
<span class="s1">&#39;std_r&#39;</span><span class="p">:</span> <span class="mf">58.393</span><span class="p">,</span> <span class="s1">&#39;std_g&#39;</span><span class="p">:</span> <span class="mf">57.12</span><span class="p">,</span> <span class="s1">&#39;std_b&#39;</span><span class="p">:</span> <span class="mf">57.375</span><span class="p">}</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">io</span><span class="o">.</span><span class="n">ImageRecordIter</span><span class="p">(</span><span class="n">path_imgrec</span><span class="o">=</span><span class="s1">&#39;data/val_256_q90.rec&#39;</span><span class="p">,</span>
<span class="n">batch_size</span><span class="o">=</span><span class="n">batch_size</span><span class="p">,</span>
<span class="n">data_shape</span><span class="o">=</span><span class="n">batch_shape</span><span class="p">[</span><span class="mi">1</span><span class="p">:],</span>
<span class="n">rand_crop</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="n">rand_mirror</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="n">shuffle</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="o">**</span><span class="n">mean_std</span><span class="p">)</span>
<span class="n">data</span><span class="o">.</span><span class="n">batch_size</span> <span class="o">=</span> <span class="n">batch_size</span>
</pre></div>
</div>
<ol class="arabic simple" start="3">
<li><p>Prepare the evaluation function:</p></li>
</ol>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">eval_samples</span> <span class="o">=</span> <span class="n">batch_size</span><span class="o">*</span><span class="mi">10</span>
<span class="k">def</span> <span class="nf">eval_func</span><span class="p">(</span><span class="n">model</span><span class="p">):</span>
<span class="n">data</span><span class="o">.</span><span class="n">reset</span><span class="p">()</span>
<span class="n">metric</span> <span class="o">=</span> <span class="n">mx</span><span class="o">.</span><span class="n">metric</span><span class="o">.</span><span class="n">Accuracy</span><span class="p">()</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">batch</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">data</span><span class="p">):</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">*</span> <span class="n">batch_size</span> <span class="o">&gt;=</span> <span class="n">eval_samples</span><span class="p">:</span>
<span class="k">break</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">batch</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_in_context</span><span class="p">(</span><span class="n">mx</span><span class="o">.</span><span class="n">cpu</span><span class="p">())</span>
<span class="n">label</span> <span class="o">=</span> <span class="n">batch</span><span class="o">.</span><span class="n">label</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_in_context</span><span class="p">(</span><span class="n">mx</span><span class="o">.</span><span class="n">cpu</span><span class="p">())</span>
<span class="n">outputs</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">forward</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">metric</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">label</span><span class="p">,</span> <span class="n">outputs</span><span class="p">)</span>
<span class="k">return</span> <span class="n">metric</span><span class="o">.</span><span class="n">get</span><span class="p">()[</span><span class="mi">1</span><span class="p">]</span>
</pre></div>
</div>
<ol class="arabic simple" start="4">
<li><p>Run Intel® Neural Compressor:</p></li>
</ol>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">neural_compressor.experimental</span> <span class="kn">import</span> <span class="n">Quantization</span>
<span class="n">quantizer</span> <span class="o">=</span> <span class="n">Quantization</span><span class="p">(</span><span class="s2">&quot;./cnn.yaml&quot;</span><span class="p">)</span>
<span class="n">quantizer</span><span class="o">.</span><span class="n">model</span> <span class="o">=</span> <span class="n">resnet18</span>
<span class="n">quantizer</span><span class="o">.</span><span class="n">calib_dataloader</span> <span class="o">=</span> <span class="n">data</span>
<span class="n">quantizer</span><span class="o">.</span><span class="n">eval_func</span> <span class="o">=</span> <span class="n">eval_func</span>
<span class="n">qnet</span> <span class="o">=</span> <span class="n">quantizer</span><span class="o">.</span><span class="n">fit</span><span class="p">()</span><span class="o">.</span><span class="n">model</span>
</pre></div>
</div>
<p>Since this model already achieves good accuracy using native quantization (less than 1% accuracy drop), for the given configuration file, INC will end on the first configuration, quantizing all layers using <code class="docutils literal notranslate"><span class="pre">naive</span></code> calibration mode for each. To see the true potential of INC, we need a model which suffers from a larger accuracy drop after quantization.</p>
</div>
<div class="section" id="Quantizing-BERT">
<h3>Quantizing BERT<a class="headerlink" href="#Quantizing-BERT" title="Permalink to this headline"></a></h3>
<p>This example shows how to use INC to quantize BERT-base for MRPC. In this case, the native MXNet quantization usually introduce a significant accuracy drop (2% - 5% using <code class="docutils literal notranslate"><span class="pre">naive</span></code> calibration mode). To simplify the code, model and task specific boilerplate has been moved to the <code class="docutils literal notranslate"><span class="pre">details.py</span></code> file.</p>
<p>This is the configuration file for this example:</p>
<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1.0</span>
<span class="nt">model</span><span class="p">:</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">bert</span>
<span class="w"> </span><span class="nt">framework</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">mxnet</span>
<span class="nt">quantization</span><span class="p">:</span>
<span class="w"> </span><span class="nt">calibration</span><span class="p">:</span>
<span class="w"> </span><span class="nt">sampling_size</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">320</span><span class="w"> </span><span class="c1"># number of samples for calibration</span>
<span class="nt">tuning</span><span class="p">:</span>
<span class="w"> </span><span class="nt">strategy</span><span class="p">:</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">basic</span>
<span class="w"> </span><span class="nt">accuracy_criterion</span><span class="p">:</span>
<span class="w"> </span><span class="nt">relative</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">0.01</span>
<span class="w"> </span><span class="nt">exit_policy</span><span class="p">:</span>
<span class="w"> </span><span class="nt">timeout</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">0</span>
<span class="w"> </span><span class="nt">max_trials</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">9999</span><span class="w"> </span><span class="c1"># default is 100</span>
<span class="w"> </span><span class="nt">random_seed</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">9527</span>
</pre></div>
</div>
<p>And here is the script:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">pathlib</span> <span class="kn">import</span> <span class="n">Path</span>
<span class="kn">from</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="n">partial</span>
<span class="kn">import</span> <span class="nn">details</span>
<span class="kn">from</span> <span class="nn">neural_compressor.experimental</span> <span class="kn">import</span> <span class="n">Quantization</span><span class="p">,</span> <span class="n">common</span>
<span class="c1"># constants</span>
<span class="n">INC_CONFIG_PATH</span> <span class="o">=</span> <span class="n">Path</span><span class="p">(</span><span class="s1">&#39;./bert.yaml&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">resolve</span><span class="p">()</span>
<span class="n">PARAMS_PATH</span> <span class="o">=</span> <span class="n">Path</span><span class="p">(</span><span class="s1">&#39;./bert_mrpc.params&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">resolve</span><span class="p">()</span>
<span class="n">OUTPUT_DIR_PATH</span> <span class="o">=</span> <span class="n">Path</span><span class="p">(</span><span class="s1">&#39;./output/&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">resolve</span><span class="p">()</span>
<span class="n">OUTPUT_MODEL_PATH</span> <span class="o">=</span> <span class="n">OUTPUT_DIR_PATH</span><span class="o">/</span><span class="s1">&#39;quantized_model&#39;</span>
<span class="n">OUTPUT_DIR_PATH</span><span class="o">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">parents</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">exist_ok</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="c1"># Prepare the dataloaders (calib_dataloader is same as train_dataloader but without shuffling)</span>
<span class="n">train_dataloader</span><span class="p">,</span> <span class="n">dev_dataloader</span><span class="p">,</span> <span class="n">calib_dataloader</span> <span class="o">=</span> <span class="n">details</span><span class="o">.</span><span class="n">preprocess_data</span><span class="p">()</span>
<span class="c1"># Get the model</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">details</span><span class="o">.</span><span class="n">BERTModel</span><span class="p">(</span><span class="n">details</span><span class="o">.</span><span class="n">BACKBONE</span><span class="p">,</span> <span class="n">dropout</span><span class="o">=</span><span class="mf">0.1</span><span class="p">,</span> <span class="n">num_classes</span><span class="o">=</span><span class="n">details</span><span class="o">.</span><span class="n">NUM_CLASSES</span><span class="p">)</span>
<span class="n">model</span><span class="o">.</span><span class="n">hybridize</span><span class="p">(</span><span class="n">static_alloc</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="c1"># finetune or load the parameters of already finetuned model</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">PARAMS_PATH</span><span class="o">.</span><span class="n">exists</span><span class="p">():</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">details</span><span class="o">.</span><span class="n">finetune</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">train_dataloader</span><span class="p">,</span> <span class="n">dev_dataloader</span><span class="p">,</span> <span class="n">OUTPUT_DIR_PATH</span><span class="p">)</span>
<span class="n">model</span><span class="o">.</span><span class="n">save_parameters</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">PARAMS_PATH</span><span class="p">))</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">model</span><span class="o">.</span><span class="n">load_parameters</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">PARAMS_PATH</span><span class="p">),</span> <span class="n">ctx</span><span class="o">=</span><span class="n">details</span><span class="o">.</span><span class="n">CTX</span><span class="p">,</span> <span class="n">cast_dtype</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="c1"># run INC</span>
<span class="n">calib_dataloader</span><span class="o">.</span><span class="n">batch_size</span> <span class="o">=</span> <span class="n">details</span><span class="o">.</span><span class="n">BATCH_SIZE</span>
<span class="n">eval_func</span> <span class="o">=</span> <span class="n">partial</span><span class="p">(</span><span class="n">details</span><span class="o">.</span><span class="n">evaluate</span><span class="p">,</span> <span class="n">dataloader</span><span class="o">=</span><span class="n">dev_dataloader</span><span class="p">)</span>
<span class="n">quantizer</span> <span class="o">=</span> <span class="n">Quantization</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">INC_CONFIG_PATH</span><span class="p">))</span> <span class="c1"># 1. Config file</span>
<span class="n">quantizer</span><span class="o">.</span><span class="n">model</span> <span class="o">=</span> <span class="n">common</span><span class="o">.</span><span class="n">Model</span><span class="p">(</span><span class="n">model</span><span class="p">)</span> <span class="c1"># 2. Model to be quantized</span>
<span class="n">quantizer</span><span class="o">.</span><span class="n">calib_dataloader</span> <span class="o">=</span> <span class="n">calib_dataloader</span> <span class="c1"># 3. Calibration dataloader</span>
<span class="n">quantizer</span><span class="o">.</span><span class="n">eval_func</span> <span class="o">=</span> <span class="n">eval_func</span> <span class="c1"># 4. Evaluation function</span>
<span class="n">quantized_model</span> <span class="o">=</span> <span class="n">quantizer</span><span class="o">.</span><span class="n">fit</span><span class="p">()</span><span class="o">.</span><span class="n">model</span>
<span class="c1"># save the quantized model</span>
<span class="n">quantized_model</span><span class="o">.</span><span class="n">export</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">OUTPUT_MODEL_PATH</span><span class="p">))</span>
</pre></div>
</div>
<p>With the evaluation function hidden in the <code class="docutils literal notranslate"><span class="pre">details.py</span></code> file:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">evaluate</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">dataloader</span><span class="p">):</span>
<span class="n">metric</span> <span class="o">=</span> <span class="n">METRIC</span><span class="p">()</span>
<span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">dataloader</span><span class="p">:</span>
<span class="n">input_ids</span><span class="p">,</span> <span class="n">segment_ids</span><span class="p">,</span> <span class="n">valid_length</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="n">batch</span>
<span class="n">input_ids</span> <span class="o">=</span> <span class="n">input_ids</span><span class="o">.</span><span class="n">as_in_context</span><span class="p">(</span><span class="n">CTX</span><span class="p">)</span>
<span class="n">segment_ids</span> <span class="o">=</span> <span class="n">segment_ids</span><span class="o">.</span><span class="n">as_in_context</span><span class="p">(</span><span class="n">CTX</span><span class="p">)</span>
<span class="n">valid_length</span> <span class="o">=</span> <span class="n">valid_length</span><span class="o">.</span><span class="n">as_in_context</span><span class="p">(</span><span class="n">CTX</span><span class="p">)</span>
<span class="n">label</span> <span class="o">=</span> <span class="n">label</span><span class="o">.</span><span class="n">as_in_context</span><span class="p">(</span><span class="n">CTX</span><span class="p">)</span><span class="o">.</span><span class="n">reshape</span><span class="p">((</span><span class="o">-</span><span class="mi">1</span><span class="p">))</span>
<span class="n">out</span> <span class="o">=</span> <span class="n">model</span><span class="p">(</span><span class="n">input_ids</span><span class="p">,</span> <span class="n">segment_ids</span><span class="p">,</span> <span class="n">valid_length</span><span class="p">)</span>
<span class="n">metric</span><span class="o">.</span><span class="n">update</span><span class="p">([</span><span class="n">label</span><span class="p">],</span> <span class="p">[</span><span class="n">out</span><span class="p">])</span>
<span class="n">metric_name</span><span class="p">,</span> <span class="n">metric_val</span> <span class="o">=</span> <span class="n">metric</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="k">return</span> <span class="n">metric_val</span>
</pre></div>
</div>
<p>For comparision, this is how one could quantize this model using MXNet native quantization (this function is also located in the <code class="docutils literal notranslate"><span class="pre">details.py</span></code> file):</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">native_quantization</span><span class="p">(</span><span class="n">model</span><span class="p">,</span> <span class="n">calib_dataloader</span><span class="p">,</span> <span class="n">dev_dataloader</span><span class="p">):</span>
<span class="n">quantized_model</span> <span class="o">=</span> <span class="n">quantize_net_v2</span><span class="p">(</span><span class="n">model</span><span class="p">,</span>
<span class="n">quantize_mode</span><span class="o">=</span><span class="s1">&#39;smart&#39;</span><span class="p">,</span>
<span class="n">calib_data</span><span class="o">=</span><span class="n">calib_dataloader</span><span class="p">,</span>
<span class="n">calib_mode</span><span class="o">=</span><span class="s1">&#39;naive&#39;</span><span class="p">,</span>
<span class="n">num_calib_examples</span><span class="o">=</span><span class="n">BATCH_SIZE</span><span class="o">*</span><span class="mi">10</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Native quantization results: </span><span class="si">{}</span><span class="s1">&#39;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">evaluate</span><span class="p">(</span><span class="n">quantized_model</span><span class="p">,</span> <span class="n">dev_dataloader</span><span class="p">)))</span>
<span class="k">return</span> <span class="n">quantized_model</span>
</pre></div>
</div>
<p>For complete code, see this example on the <a class="reference external" href="https://github.com/apache/mxnet/tree/v1.x/example/quantization_inc/BERT_MRPC">official GitHub repository</a>.</p>
<div class="section" id="Results:">
<h4>Results:<a class="headerlink" href="#Results:" title="Permalink to this headline"></a></h4>
<p>Environment: - c6i.16xlarge Amazon EC2 instance (Intel(R) Xeon(R) Platinum 8375C CPU &#64; 2.90GHz) - Ubuntu 20.04 LTS - MXNet 1.9 - INC 1.9.1</p>
<p>Results on the validation dataset:</p>
<table class="docutils align-default">
<colgroup>
<col style="width: 26%" />
<col style="width: 9%" />
<col style="width: 7%" />
<col style="width: 24%" />
<col style="width: 26%" />
<col style="width: 8%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>Quantization method</p></th>
<th class="head"><p>Accuracy</p></th>
<th class="head"><p>F1</p></th>
<th class="head"><p>Relative accuracy loss [%]</p></th>
<th class="head"><p>Calibration/tuning time [s]</p></th>
<th class="head"><p>Speedup</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><strong>No quantization (f32)</strong></p></td>
<td><p><strong>0.8529</strong></p></td>
<td><p><strong>0.8956</strong></p></td>
<td><p><strong>0</strong></p></td>
<td><p><strong>0</strong></p></td>
<td><p><strong>1.0</strong></p></td>
</tr>
<tr class="row-odd"><td><p>Native ‘naive’, 10 batches</p></td>
<td><p>0.8259</p></td>
<td><p>0.8775</p></td>
<td><p>3.1657</p></td>
<td><p>31</p></td>
<td><p>1.3811</p></td>
</tr>
<tr class="row-even"><td><p>Native ‘naive’, 20 batches</p></td>
<td><p>0.8210</p></td>
<td><p>0.8731</p></td>
<td><p>3.7402</p></td>
<td><p>58</p></td>
<td><p>1.3866</p></td>
</tr>
<tr class="row-odd"><td><p>Native ‘entropy’, 10 batches</p></td>
<td><p>0.8064</p></td>
<td><p>0.8557</p></td>
<td><p>5.4520</p></td>
<td><p>37</p></td>
<td><p>1.3789</p></td>
</tr>
<tr class="row-even"><td><p>Native ‘entropy’, 20 batches</p></td>
<td><p>0.8137</p></td>
<td><p>0.8624</p></td>
<td><p>4.5961</p></td>
<td><p>67</p></td>
<td><p>1.3460</p></td>
</tr>
<tr class="row-odd"><td><p>INC, ‘basic’</p></td>
<td><p>0.8456</p></td>
<td><p>0.8889</p></td>
<td><p>0.8559</p></td>
<td><p>197</p></td>
<td><p>1.4418</p></td>
</tr>
<tr class="row-even"><td><p>INC, ‘bayesian’</p></td>
<td><p>0.8529</p></td>
<td><p>0.8888</p></td>
<td><p>0</p></td>
<td><p>129</p></td>
<td><p>1.4275</p></td>
</tr>
<tr class="row-odd"><td><p>INC, ‘mse’</p></td>
<td><p>0.8480</p></td>
<td><p>0.8954</p></td>
<td><p>0.5745</p></td>
<td><p>974</p></td>
<td><p>0.9642</p></td>
</tr>
</tbody>
</table>
<p>All INC strategies found configurations meeting the 1% relative accuracy loss criterion. Only the <code class="docutils literal notranslate"><span class="pre">mse</span></code> strategy struggled, taking the longest time and generating configuration that is slower than the f32 model. Although these results may suggest that the <code class="docutils literal notranslate"><span class="pre">mse</span></code> strategy is the worst and the <code class="docutils literal notranslate"><span class="pre">bayesian</span></code> strategy is the best, different strategies may give better results for specific models and tasks. Usually the <code class="docutils literal notranslate"><span class="pre">basic</span></code> strategy is the most stable one.</p>
<p>Here is an example of a configuration generated by INC with the <code class="docutils literal notranslate"><span class="pre">basic</span></code> strategy:</p>
<ul class="simple">
<li><p>Layers quantized using min-max (<code class="docutils literal notranslate"><span class="pre">naive</span></code>) calibration algorithm:
<code class="docutils literal notranslate"><span class="pre">{'bertclassifier0_dropout0_fwd',</span> <span class="pre">'bertencoder0_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer0_dotproductselfattentioncell0_dropout0_fwd',</span> <span class="pre">'bertencoder0_transformer0_dotproductselfattentioncell0_reshape3',</span> <span class="pre">'bertencoder0_transformer0_dotproductselfattentioncell0_reshape7',</span> <span class="pre">'bertencoder0_transformer0_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer0_positionwiseffn0_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer10_dotproductselfattentioncell0_dropout0_fwd',</span> <span class="pre">'bertencoder0_transformer10_dotproductselfattentioncell0_reshape3',</span> <span class="pre">'bertencoder0_transformer10_dotproductselfattentioncell0_reshape7',</span> <span class="pre">'bertencoder0_transformer10_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer10_positionwiseffn0_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer11_dotproductselfattentioncell0_dropout0_fwd',</span> <span class="pre">'bertencoder0_transformer11_dotproductselfattentioncell0_reshape3',</span> <span class="pre">'bertencoder0_transformer11_dotproductselfattentioncell0_reshape7',</span> <span class="pre">'bertencoder0_transformer11_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer1_dotproductselfattentioncell0_dropout0_fwd',</span> <span class="pre">'bertencoder0_transformer1_dotproductselfattentioncell0_reshape3',</span> <span class="pre">'bertencoder0_transformer1_dotproductselfattentioncell0_reshape7',</span> <span class="pre">'bertencoder0_transformer1_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer1_positionwiseffn0_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer2_dotproductselfattentioncell0_dropout0_fwd',</span> <span class="pre">'bertencoder0_transformer2_dotproductselfattentioncell0_reshape3',</span> <span class="pre">'bertencoder0_transformer2_dotproductselfattentioncell0_reshape7',</span> <span class="pre">'bertencoder0_transformer2_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer2_positionwiseffn0_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer3_dotproductselfattentioncell0_dropout0_fwd',</span> <span class="pre">'bertencoder0_transformer3_dotproductselfattentioncell0_reshape3',</span> <span class="pre">'bertencoder0_transformer3_dotproductselfattentioncell0_reshape7',</span> <span class="pre">'bertencoder0_transformer3_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer3_positionwiseffn0_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer4_dotproductselfattentioncell0_dropout0_fwd',</span> <span class="pre">'bertencoder0_transformer4_dotproductselfattentioncell0_reshape3',</span> <span class="pre">'bertencoder0_transformer4_dotproductselfattentioncell0_reshape7',</span> <span class="pre">'bertencoder0_transformer4_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer4_positionwiseffn0_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer5_dotproductselfattentioncell0_dropout0_fwd',</span> <span class="pre">'bertencoder0_transformer5_dotproductselfattentioncell0_reshape3',</span> <span class="pre">'bertencoder0_transformer5_dotproductselfattentioncell0_reshape7',</span> <span class="pre">'bertencoder0_transformer5_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer5_positionwiseffn0_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer6_dotproductselfattentioncell0_dropout0_fwd',</span> <span class="pre">'bertencoder0_transformer6_dotproductselfattentioncell0_reshape3',</span> <span class="pre">'bertencoder0_transformer6_dotproductselfattentioncell0_reshape7',</span> <span class="pre">'bertencoder0_transformer6_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer6_positionwiseffn0_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer7_dotproductselfattentioncell0_dropout0_fwd',</span> <span class="pre">'bertencoder0_transformer7_dotproductselfattentioncell0_reshape3',</span> <span class="pre">'bertencoder0_transformer7_dotproductselfattentioncell0_reshape7',</span> <span class="pre">'bertencoder0_transformer7_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer7_positionwiseffn0_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer8_dotproductselfattentioncell0_dropout0_fwd',</span> <span class="pre">'bertencoder0_transformer8_dotproductselfattentioncell0_reshape3',</span> <span class="pre">'bertencoder0_transformer8_dotproductselfattentioncell0_reshape7',</span> <span class="pre">'bertencoder0_transformer8_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer8_positionwiseffn0_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer9_dotproductselfattentioncell0_dropout0_fwd',</span> <span class="pre">'bertencoder0_transformer9_dotproductselfattentioncell0_reshape3',</span> <span class="pre">'bertencoder0_transformer9_dotproductselfattentioncell0_reshape7',</span> <span class="pre">'bertencoder0_transformer9_layernorm0_layernorm0',</span> <span class="pre">'bertencoder0_transformer9_positionwiseffn0_layernorm0_layernorm0',</span> <span class="pre">'bertmodel0_reshape0',</span> <span class="pre">'sg_mkldnn_fully_connected_0',</span> <span class="pre">'sg_mkldnn_fully_connected_1',</span> <span class="pre">'sg_mkldnn_fully_connected_11',</span> <span class="pre">'sg_mkldnn_fully_connected_12',</span> <span class="pre">'sg_mkldnn_fully_connected_13',</span> <span class="pre">'sg_mkldnn_fully_connected_15',</span> <span class="pre">'sg_mkldnn_fully_connected_16',</span> <span class="pre">'sg_mkldnn_fully_connected_17',</span> <span class="pre">'sg_mkldnn_fully_connected_19',</span> <span class="pre">'sg_mkldnn_fully_connected_20',</span> <span class="pre">'sg_mkldnn_fully_connected_21',</span> <span class="pre">'sg_mkldnn_fully_connected_23',</span> <span class="pre">'sg_mkldnn_fully_connected_24',</span> <span class="pre">'sg_mkldnn_fully_connected_25',</span> <span class="pre">'sg_mkldnn_fully_connected_27',</span> <span class="pre">'sg_mkldnn_fully_connected_28',</span> <span class="pre">'sg_mkldnn_fully_connected_29',</span> <span class="pre">'sg_mkldnn_fully_connected_3',</span> <span class="pre">'sg_mkldnn_fully_connected_31',</span> <span class="pre">'sg_mkldnn_fully_connected_32',</span> <span class="pre">'sg_mkldnn_fully_connected_33',</span> <span class="pre">'sg_mkldnn_fully_connected_35',</span> <span class="pre">'sg_mkldnn_fully_connected_36',</span> <span class="pre">'sg_mkldnn_fully_connected_37',</span> <span class="pre">'sg_mkldnn_fully_connected_39',</span> <span class="pre">'sg_mkldnn_fully_connected_4',</span> <span class="pre">'sg_mkldnn_fully_connected_40',</span> <span class="pre">'sg_mkldnn_fully_connected_41',</span> <span class="pre">'sg_mkldnn_fully_connected_43',</span> <span class="pre">'sg_mkldnn_fully_connected_44',</span> <span class="pre">'sg_mkldnn_fully_connected_45',</span> <span class="pre">'sg_mkldnn_fully_connected_47',</span> <span class="pre">'sg_mkldnn_fully_connected_48',</span> <span class="pre">'sg_mkldnn_fully_connected_49',</span> <span class="pre">'sg_mkldnn_fully_connected_5',</span> <span class="pre">'sg_mkldnn_fully_connected_7',</span> <span class="pre">'sg_mkldnn_fully_connected_8',</span> <span class="pre">'sg_mkldnn_fully_connected_9',</span> <span class="pre">'sg_mkldnn_fully_connected_eltwise_10',</span> <span class="pre">'sg_mkldnn_fully_connected_eltwise_14',</span> <span class="pre">'sg_mkldnn_fully_connected_eltwise_18',</span> <span class="pre">'sg_mkldnn_fully_connected_eltwise_2',</span> <span class="pre">'sg_mkldnn_fully_connected_eltwise_22',</span> <span class="pre">'sg_mkldnn_fully_connected_eltwise_26',</span> <span class="pre">'sg_mkldnn_fully_connected_eltwise_30',</span> <span class="pre">'sg_mkldnn_fully_connected_eltwise_34',</span> <span class="pre">'sg_mkldnn_fully_connected_eltwise_38',</span> <span class="pre">'sg_mkldnn_fully_connected_eltwise_42',</span> <span class="pre">'sg_mkldnn_fully_connected_eltwise_46',</span> <span class="pre">'sg_mkldnn_fully_connected_eltwise_6'}</span></code></p></li>
<li><p>Layers quantized using KL (<code class="docutils literal notranslate"><span class="pre">entropy</span></code>) calibration algorithm:
<code class="docutils literal notranslate"><span class="pre">{'sg_mkldnn_selfatt_qk_0',</span> <span class="pre">'sg_mkldnn_selfatt_qk_10',</span> <span class="pre">'sg_mkldnn_selfatt_qk_12',</span> <span class="pre">'sg_mkldnn_selfatt_qk_14',</span> <span class="pre">'sg_mkldnn_selfatt_qk_16',</span> <span class="pre">'sg_mkldnn_selfatt_qk_18',</span> <span class="pre">'sg_mkldnn_selfatt_qk_2',</span> <span class="pre">'sg_mkldnn_selfatt_qk_20',</span> <span class="pre">'sg_mkldnn_selfatt_qk_22',</span> <span class="pre">'sg_mkldnn_selfatt_qk_4',</span> <span class="pre">'sg_mkldnn_selfatt_qk_6',</span> <span class="pre">'sg_mkldnn_selfatt_qk_8',</span> <span class="pre">'sg_mkldnn_selfatt_valatt_1',</span> <span class="pre">'sg_mkldnn_selfatt_valatt_11',</span> <span class="pre">'sg_mkldnn_selfatt_valatt_13',</span> <span class="pre">'sg_mkldnn_selfatt_valatt_15',</span> <span class="pre">'sg_mkldnn_selfatt_valatt_17',</span> <span class="pre">'sg_mkldnn_selfatt_valatt_19',</span> <span class="pre">'sg_mkldnn_selfatt_valatt_21',</span> <span class="pre">'sg_mkldnn_selfatt_valatt_23',</span> <span class="pre">'sg_mkldnn_selfatt_valatt_3',</span> <span class="pre">'sg_mkldnn_selfatt_valatt_5',</span> <span class="pre">'sg_mkldnn_selfatt_valatt_7',</span> <span class="pre">'sg_mkldnn_selfatt_valatt_9'}</span></code></p></li>
<li><p>Layers excluded from quantization: <code class="docutils literal notranslate"><span class="pre">{'sg_mkldnn_fully_connected_43'}</span></code></p></li>
</ul>
</div>
</div>
</div>
<div class="section" id="Tips">
<h2>Tips<a class="headerlink" href="#Tips" title="Permalink to this headline"></a></h2>
<ul>
<li><p>In order to get a solution that generalizes well, evaluate the model (in eval_func) on a representative dataset.</p></li>
<li><p>With <code class="docutils literal notranslate"><span class="pre">history.snapshot</span></code> file (generated by INC) you can recover any model that was generated during the tuning process:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">neural_compressor.utils.utility</span> <span class="kn">import</span> <span class="n">recover</span>
<span class="n">quantized_model</span> <span class="o">=</span> <span class="n">recover</span><span class="p">(</span><span class="n">f32_model</span><span class="p">,</span> <span class="s1">&#39;nc_workspace/&lt;tuning date&gt;/history.snapshot&#39;</span><span class="p">,</span> <span class="n">configuration_idx</span><span class="p">)</span><span class="o">.</span><span class="n">model</span>
</pre></div>
</div>
</li>
</ul>
<!-- INSERT SOURCE DOWNLOAD BUTTONS --></div>
</div>
<hr class="feedback-hr-top" />
<div class="feedback-container">
<div class="feedback-question">Did this page help you?</div>
<div class="feedback-answer-container">
<div class="feedback-answer yes-link" data-response="yes">Yes</div>
<div class="feedback-answer no-link" data-response="no">No</div>
</div>
<div class="feedback-thank-you">Thanks for your feedback!</div>
</div>
<hr class="feedback-hr-bottom" />
</div>
<div class="side-doc-outline">
<div class="side-doc-outline--content">
<div class="localtoc">
<p class="caption">
<span class="caption-text">Table Of Contents</span>
</p>
<ul>
<li><a class="reference internal" href="#">Quantize with MKL-DNN backend</a><ul>
<li><a class="reference internal" href="#Installation-and-Prerequisites">Installation and Prerequisites</a></li>
<li><a class="reference internal" href="#Image-Classification-Demo">Image Classification Demo</a></li>
<li><a class="reference internal" href="#Integrate-Quantization-Flow-to-Your-Project">Integrate Quantization Flow to Your Project</a><ul>
<li><a class="reference internal" href="#Initialize-Model">Initialize Model</a><ul>
<li><a class="reference internal" href="#Model-Fusion">Model Fusion</a></li>
</ul>
</li>
<li><a class="reference internal" href="#Quantize-Model">Quantize Model</a></li>
<li><a class="reference internal" href="#Evaluate-&amp;-Tune">Evaluate &amp; Tune</a></li>
<li><a class="reference internal" href="#Calibrate-Model-(optional-for-performance)">Calibrate Model (optional for performance)</a></li>
<li><a class="reference internal" href="#Tips-for-Model-Calibration">Tips for Model Calibration</a><ul>
<li><a class="reference internal" href="#Accuracy-Tuning">Accuracy Tuning</a></li>
<li><a class="reference internal" href="#Performance-Tuning">Performance Tuning</a></li>
</ul>
</li>
</ul>
</li>
<li><a class="reference internal" href="#Deploy-with-Python/C++">Deploy with Python/C++</a></li>
</ul>
</li>
<li><a class="reference internal" href="#Improving-accuracy-with-Intel®-Neural-Compressor">Improving accuracy with Intel® Neural Compressor</a><ul>
<li><a class="reference internal" href="#Installation-and-Prerequisites">Installation and Prerequisites</a></li>
<li><a class="reference internal" href="#Configuration-file">Configuration file</a></li>
<li><a class="reference internal" href="#Model-quantization-and-tuning">Model quantization and tuning</a><ul>
<li><a class="reference internal" href="#Quantizing-ResNet">Quantizing ResNet</a></li>
<li><a class="reference internal" href="#Quantizing-BERT">Quantizing BERT</a><ul>
<li><a class="reference internal" href="#Results:">Results:</a></li>
</ul>
</li>
</ul>
</li>
<li><a class="reference internal" href="#Tips">Tips</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div>
<div class="clearer"></div>
</div><div class="pagenation">
<a id="button-prev" href="index.html" class="mdl-button mdl-js-button mdl-js-ripple-effect mdl-button--colored" role="botton" accesskey="P">
<i class="pagenation-arrow-L fas fa-arrow-left fa-lg"></i>
<div class="pagenation-text">
<span class="pagenation-direction">Previous</span>
<div>Intel MKL-DNN</div>
</div>
</a>
<a id="button-next" href="mkldnn_readme.html" class="mdl-button mdl-js-button mdl-js-ripple-effect mdl-button--colored" role="botton" accesskey="N">
<i class="pagenation-arrow-R fas fa-arrow-right fa-lg"></i>
<div class="pagenation-text">
<span class="pagenation-direction">Next</span>
<div>Install MXNet with MKL-DNN</div>
</div>
</a>
</div>
<footer class="site-footer h-card">
<div class="wrapper">
<div class="row">
<div class="col-4">
<h4 class="footer-category-title">Resources</h4>
<ul class="contact-list">
<li><a class="u-email" href="mailto:dev@mxnet.apache.org">Dev list</a></li>
<li><a class="u-email" href="mailto:user@mxnet.apache.org">User mailing list</a></li>
<li><a href="https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+Home">Developer Wiki</a></li>
<li><a href="https://issues.apache.org/jira/projects/MXNET/issues">Jira Tracker</a></li>
<li><a href="https://github.com/apache/mxnet/labels/Roadmap">Github Roadmap</a></li>
<li><a href="https://medium.com/apache-mxnet">Blog</a></li>
<li><a href="https://discuss.mxnet.io">Forum</a></li>
<li><a href="/community/contribute">Contribute</a></li>
</ul>
</div>
<div class="col-4"><ul class="social-media-list"><li><a href="https://github.com/apache/mxnet"><svg class="svg-icon"><use xlink:href="../../../../_static/minima-social-icons.svg#github"></use></svg> <span class="username">apache/mxnet</span></a></li><li><a href="https://www.twitter.com/apachemxnet"><svg class="svg-icon"><use xlink:href="../../../../_static/minima-social-icons.svg#twitter"></use></svg> <span class="username">apachemxnet</span></a></li><li><a href="https://youtube.com/apachemxnet"><svg class="svg-icon"><use xlink:href="../../../../_static/minima-social-icons.svg#youtube"></use></svg> <span class="username">apachemxnet</span></a></li></ul>
</div>
<div class="col-4 footer-text">
<p>A flexible and efficient library for deep learning.</p>
</div>
</div>
</div>
</footer>
<footer class="site-footer2">
<div class="wrapper">
<div class="row">
<div class="col-3">
<img src="../../../../_static/apache_incubator_logo.png" class="footer-logo col-2">
</div>
<div class="footer-bottom-warning col-9">
<p>Apache MXNet is an effort undergoing incubation at <a href="http://www.apache.org/">The Apache Software Foundation</a> (ASF), <span style="font-weight:bold">sponsored by the <i>Apache Incubator</i></span>. Incubation is required
of all newly accepted projects until a further review indicates that the infrastructure,
communications, and decision making process have stabilized in a manner consistent with other
successful ASF projects. While incubation status is not necessarily a reflection of the completeness
or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
</p><p>"Copyright © 2017-2018, The Apache Software Foundation Apache MXNet, MXNet, Apache, the Apache
feather, and the Apache MXNet project logo are either registered trademarks or trademarks of the
Apache Software Foundation."</p>
</div>
</div>
</div>
</footer>
</body>
</html>