api/python/docs/_sources/tutorials/performance/backend/dnnl/dnnl_quantization_inc.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "40e6f0b4",
   "metadata": {},
   "source": [
    "<!--- Licensed to the Apache Software Foundation (ASF) under one -->\n",
    "<!--- or more contributor license agreements.  See the NOTICE file -->\n",
    "<!--- distributed with this work for additional information -->\n",
    "<!--- regarding copyright ownership.  The ASF licenses this file -->\n",
    "<!--- to you under the Apache License, Version 2.0 (the -->\n",
    "<!--- \"License\"); you may not use this file except in compliance -->\n",
    "<!--- with the License.  You may obtain a copy of the License at -->\n",
    "\n",
    "<!---   http://www.apache.org/licenses/LICENSE-2.0 -->\n",
    "\n",
    "<!--- Unless required by applicable law or agreed to in writing, -->\n",
    "<!--- software distributed under the License is distributed on an -->\n",
    "<!--- \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->\n",
    "<!--- KIND, either express or implied.  See the License for the -->\n",
    "<!--- specific language governing permissions and limitations -->\n",
    "<!--- under the License. -->\n",
    "\n",
    "\n",
    "# Improving accuracy with Intel® Neural Compressor\n",
    "\n",
    "The accuracy of a model can decrease as a result of quantization. When the accuracy drop is significant, we can try to manually find a better quantization configuration (exclude some layers, try different calibration methods, etc.), but for bigger models this might prove to be a difficult and time consuming task. [Intel® Neural Compressor](https://github.com/intel/neural-compressor) (INC) tries to automate this process using several tuning heuristics, which aim to find the quantization configuration that satisfies the specified accuracy requirement.\n",
    "\n",
    "**NOTE:**\n",
    "\n",
    "Most tuning strategies will try different configurations on an evaluation dataset in order to find out how each layer affects the accuracy of the model. This means that for larger models, it may take a long time to find a solution (as the tuning space is usually larger and the evaluation itself takes longer).\n",
    "\n",
    "## Installation and Prerequisites\n",
    "\n",
    "- Install MXNet with oneDNN enabled as described in the [Get started](https://mxnet.apache.org/versions/master/get_started?platform=linux&language=python&processor=cpu&environ=pip&). (Until the 2.0 release you can use the nightly build version: `pip install --pre mxnet -f https://dist.mxnet.io/python`)\n",
    "\n",
    "- Install Intel® Neural Compressor:\n",
    "\n",
    "  Use one of the commands below to install INC (supported python versions are: 3.6, 3.7, 3.8, 3.9):\n",
    "\n",
    "  ```bash\n",
    "  # install stable version from pip\n",
    "  pip install neural-compressor\n",
    "\n",
    "  # install nightly version from pip\n",
    "  pip install -i https://test.pypi.org/simple/ neural-compressor\n",
    "\n",
    "  # install stable version from conda\n",
    "  conda install neural-compressor -c conda-forge -c intel\n",
    "  ```\n",
    "  If you get into trouble with dependencies on `cv2` library you can run: `apt-get update && apt-get install -y python3-opencv`\n",
    "\n",
    "## Configuration file\n",
    "\n",
    "Quantization tuning process can be customized in the yaml configuration file. Below is a simple example:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17eb6da9",
   "metadata": {},
   "source": [
    "```yaml\n",
    "# cnn.yaml\n",
    "\n",
    "version: 1.0\n",
    "\n",
    "model:\n",
    "  name: cnn\n",
    "  framework: mxnet\n",
    "\n",
    "quantization:\n",
    "  calibration:\n",
    "    sampling_size: 160 # number of samples for calibration\n",
    "\n",
    "tuning:\n",
    "  strategy:\n",
    "    name: basic\n",
    "  accuracy_criterion:\n",
    "    relative: 0.01\n",
    "  exit_policy:\n",
    "    timeout: 0\n",
    "  random_seed: 9527\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adf5ce4e",
   "metadata": {},
   "source": [
    "We are using the `basic` strategy, but you could also try out different ones. [Here](https://github.com/intel/neural-compressor/blob/master/docs/tuning_strategies.md) you can find a list of strategies available in INC and details of how they work. You can also add your own strategy if the existing ones do not suit your needs.\n",
    "\n",
    "Since the value of `timeout` in the example above is 0, INC will run until it finds a configuration that satisfies the accuracy criterion and then exit. Depending on the strategy this may not be ideal, as sometimes it would be better to further explore the tuning space to find a superior configuration both in terms of accuracy and speed. To achieve this, we can set a specific `timeout` value, which will tell INC how long (in seconds) it should run.\n",
    "\n",
    "For more information about the configuration file, see the [template](https://github.com/intel/neural-compressor/blob/master/neural_compressor/template/ptq.yaml) from the official INC repo. Keep in mind that only the `post training quantization` is currently supported for MXNet.\n",
    "\n",
    "## Model quantization and tuning\n",
    "\n",
    "In general, Intel® Neural Compressor requires 4 elements in order to run:  \n",
    "1. Configuration file - like the example above  \n",
    "2. Model to be quantized  \n",
    "3. Calibration dataloader  \n",
    "4. Evaluation function - a function that takes a model as an argument and returns the accuracy it achieves on a certain evaluation dataset. \n",
    "\n",
    "### Quantizing ResNet\n",
    "\n",
    "The [quantization](https://mxnet.apache.org/versions/master/api/python/docs/tutorials/performance/backend/dnnl/dnnl_quantization.html#Quantization) sections described how to quantize ResNet using the native MXNet quantization. This example shows how we can achieve the similar results (with the auto-tuning) using INC.\n",
    "\n",
    "1. Get the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00390121",
   "metadata": {},
   "source": [
    "```python\n",
    "import logging\n",
    "import mxnet as mx\n",
    "from mxnet.gluon.model_zoo import vision\n",
    "\n",
    "logging.basicConfig()\n",
    "logger = logging.getLogger('logger')\n",
    "logger.setLevel(logging.INFO)\n",
    "\n",
    "batch_shape = (1, 3, 224, 224)\n",
    "resnet18 = vision.resnet18_v1(pretrained=True)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca2ac051",
   "metadata": {},
   "source": [
    "2. Prepare the dataset:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bb5acf1",
   "metadata": {},
   "source": [
    "```python\n",
    "mx.test_utils.download('http://data.mxnet.io/data/val_256_q90.rec', 'data/val_256_q90.rec')\n",
    "\n",
    "batch_size = 16\n",
    "mean_std = {'mean_r': 123.68, 'mean_g': 116.779, 'mean_b': 103.939,\n",
    "            'std_r': 58.393, 'std_g': 57.12, 'std_b': 57.375}\n",
    "\n",
    "data = mx.io.ImageRecordIter(path_imgrec='data/val_256_q90.rec',\n",
    "                             batch_size=batch_size,\n",
    "                             data_shape=batch_shape[1:],\n",
    "                             rand_crop=False,\n",
    "                             rand_mirror=False,\n",
    "                             shuffle=False,\n",
    "                             **mean_std)\n",
    "data.batch_size = batch_size\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6cb2327",
   "metadata": {},
   "source": [
    "3. Prepare the evaluation function:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a060bcfe",
   "metadata": {},
   "source": [
    "```python\n",
    "eval_samples = batch_size*10\n",
    "\n",
    "def eval_func(model):\n",
    "    data.reset()\n",
    "    metric = mx.metric.Accuracy()\n",
    "    for i, batch in enumerate(data):\n",
    "        if i * batch_size >= eval_samples:\n",
    "            break\n",
    "        x = batch.data[0].as_in_context(mx.cpu())\n",
    "        label = batch.label[0].as_in_context(mx.cpu())\n",
    "        outputs = model.forward(x)\n",
    "        metric.update(label, outputs)\n",
    "    return metric.get()[1]\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64a96c67",
   "metadata": {},
   "source": [
    "4. Run Intel® Neural Compressor:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bf93943",
   "metadata": {},
   "source": [
    "```python\n",
    "from neural_compressor.experimental import Quantization\n",
    "quantizer = Quantization(\"./cnn.yaml\")\n",
    "quantizer.model = resnet18\n",
    "quantizer.calib_dataloader = data\n",
    "quantizer.eval_func = eval_func\n",
    "qnet = quantizer.fit().model\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a045f4c2",
   "metadata": {},
   "source": [
    "Since this model already achieves good accuracy using native quantization (less than 1% accuracy drop), for the given configuration file, INC will end on the first configuration, quantizing all layers using `naive` calibration mode for each. To see the true potential of INC, we need a model which suffers from a larger accuracy drop after quantization.\n",
    "\n",
    "### Quantizing ResNet50v2\n",
    "\n",
    "This example shows how to use INC to quantize ResNet50 v2. In this case, the native MXNet quantization introduce a huge accuracy drop (70% using `naive` calibration mode) and INC allows to automatically find better solution.\n",
    "\n",
    "This is the [INC configuration file](https://github.com/apache/incubator-mxnet/blob/master/example/quantization_inc/resnet50v2_mse.yaml) for this example:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3515dba",
   "metadata": {},
   "source": [
    "```yaml\n",
    "version: 1.0\n",
    "\n",
    "model:\n",
    "  name: resnet50_v2\n",
    "  framework: mxnet\n",
    "\n",
    "quantization:\n",
    "  calibration:\n",
    "    sampling_size: 192 # number of samples for calibration\n",
    "\n",
    "tuning:\n",
    "  strategy:\n",
    "    name: mse\n",
    "  accuracy_criterion:\n",
    "    relative: 0.015\n",
    "  exit_policy:\n",
    "    timeout: 0\n",
    "    max_trials: 500\n",
    "  random_seed: 9527\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed82bca0",
   "metadata": {},
   "source": [
    "It could be used with script below \n",
    "([resnet_mse.py](https://github.com/apache/incubator-mxnet/blob/master/example/quantization_inc/resnet_mse.py))\n",
    "to find operator, which caused the most significant accuracy drop and disable it from quantization. \n",
    "You can find description of MSE strategy \n",
    "[here](https://github.com/intel/neural-compressor/blob/master/docs/tuning_strategies.md#user-content-mse)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c276c657",
   "metadata": {},
   "source": [
    "```python\n",
    "import mxnet as mx\n",
    "from mxnet.gluon.model_zoo.vision import resnet50_v2\n",
    "from mxnet.gluon.data.vision import transforms\n",
    "from mxnet.contrib.quantization import quantize_net\n",
    "\n",
    "# Preparing input data\n",
    "rgb_mean = (0.485, 0.456, 0.406)\n",
    "rgb_std = (0.229, 0.224, 0.225)\n",
    "batch_size = 64\n",
    "num_calib_batches = 9\n",
    "# set proper path to ImageNet data set below\n",
    "dataset = mx.gluon.data.vision.ImageRecordDataset('../imagenet/rec/val.rec')\n",
    "# Tuning with INC on whole data set takes a lot of time. Therefore, we take only a part of the data set\n",
    "# as representative part of it:\n",
    "dataset = dataset.take(num_calib_batches * batch_size)\n",
    "transformer = transforms.Compose([transforms.Resize(256),\n",
    "                                  transforms.CenterCrop(224),\n",
    "                                  transforms.ToTensor(),\n",
    "                                  transforms.Normalize(mean=rgb_mean, std=rgb_std)])\n",
    "# Note: as input data is used many times during tuning, it is better to have it prepared earlier.\n",
    "#       Therefore, lazy parameter for transform_first is set to False.\n",
    "val_data = mx.gluon.data.DataLoader(\n",
    "    dataset.transform_first(transformer, lazy=False), batch_size, shuffle=False)\n",
    "val_data.batch_size = batch_size\n",
    "\n",
    "net = resnet50_v2(pretrained=True)\n",
    "\n",
    "def eval_func(model):\n",
    "  metric = mx.gluon.metric.Accuracy()\n",
    "  for x, label in val_data:\n",
    "    output = model(x)\n",
    "    metric.update(label, output)\n",
    "  accuracy = metric.get()[1]\n",
    "  return accuracy\n",
    "\n",
    "\n",
    "from neural_compressor.experimental import Quantization\n",
    "quantizer = Quantization(\"resnet50v2_mse.yaml\")\n",
    "quantizer.model = net\n",
    "quantizer.calib_dataloader = val_data\n",
    "quantizer.eval_func = eval_func\n",
    "qnet_inc = quantizer.fit().model\n",
    "print(\"INC finished\")\n",
    "# You can save optimized model for the later use:\n",
    "qnet_inc.export(\"__quantized_with_inc\")\n",
    "# You can see which configuration was applied by INC and which nodes were excluded from quantization,\n",
    "# to achieve given accuracy loss against floating point calculation.\n",
    "print(quantizer.strategy.best_qmodel.q_config['quant_cfg'])\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5384f00",
   "metadata": {},
   "source": [
    "#### Results:\n",
    "Resnet50 v2 model could be prepared to achieve better performance with various calibration and tuning methods.  \n",
    "It is done by \n",
    "[resnet_tuning.py](https://github.com/apache/incubator-mxnet/blob/master/example/quantization_inc/resnet_tuning.py) \n",
    "script on a small part of data set to reduce time required for tuning (9 batches). \n",
    "Later saved models are validated on a whole data set by \n",
    "[resnet_measurement.py](https://github.com/apache/incubator-mxnet/blob/master/example/quantization_inc/resnet_measurement.py)\n",
    "script.\n",
    "Accuracy results on the whole validation dataset (782 batches) are shown below.\n",
    "\n",
    "| Optimization method  | Top 1 accuracy | Top 5 accuracy | Top 1 relative accuracy loss [%] | Top 5 relative accuracy loss [%] | Cost = one-time optimization on 9 batches [s] | Validation time [s] | Speedup |\n",
    "|----------------------|-------:|-------:|------:|------:|-------:|--------:|------:|\n",
    "| fp32 no optimization | 0.7699 | 0.9340 |  0.00 |  0.00 |   0.00 | 316.50 | 1.0 |\n",
    "| fp32 fused           | 0.7699 | 0.9340 |  0.00 |  0.00 |   0.03 | 147.77 | 2.1 |\n",
    "| int8 full naive      | 0.2207 | 0.3912 | 71.33 | 58.12 |  11.29 |  45.81 | **6.9** |\n",
    "| int8 full entropy    | 0.6933 | 0.8917 |  9.95 |  4.53 |  80.23 |  46.39 | 6.8 |\n",
    "| int8 smart naive     | 0.2210 | 0.3905 | 71.29 | 58.19 |  11.15 |  46.02 | 6.9 |\n",
    "| int8 smart entropy   | 0.6928 | 0.8910 | 10.01 |  4.60 |  79.75 |  45.98 | 6.9 |\n",
    "| int8 INC basic       | 0.7692 | 0.9331 | **0.09** |  0.10 | 266.50 |  48.32 | **6.6** |\n",
    "| int8 INC mse         | 0.7692 | 0.9337 | **0.09** |  0.03 | 106.50 |  49.76 | **6.4** |\n",
    "| int8 INC mycustom    | 0.7699 | 0.9338 | **0.00** |  0.02 | 370.29 |  70.07 | **4.5** |\n",
    "\n",
    "\n",
    "Environment:  \n",
    "- Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz (c6i.16xlarge Amazon EC2 instance)  \n",
    "- Ubuntu 20.04.4 LTS (GNU/Linux Ubuntu 20.04.4 LTS 5.15.0-1017-aws ami-0558cee5b20db1f9c)  \n",
    "- MXNet 2.0.0b20220902 (commit 3a19f0e50d75fedb05eb558a9c835726b57df4cf)  \n",
    "- INC 1.13.1  \n",
    "- scripts above were run as parameter for [run.sh](https://github.com/apache/incubator-mxnet/blob/master/benchmark/python/dnnl/run.sh) \n",
    "script to properly setup parallel computation parameters.  \n",
    "\n",
    "For this model INC `basic`, `mse` and `mycustom` strategies found configurations meeting the 1.5% relative accuracy \n",
    "loss criterion. Only the `bayesian` strategy didn't find solution within 500 attempts limit. \n",
    "Although these results may suggest that the `mse` strategy is the best compromise between time spent\n",
    "to find the optimized model and final model performance efficiency, different strategies may give \n",
    "better results for specific models and tasks. For example for ALBERT model there is no solution \n",
    "given by build-in INC strategies. For such situation you can create your custom strategy, similar \n",
    "to this one: \n",
    "[custom_strategy.py](https://github.com/apache/incubator-mxnet/blob/master/example/quantization_inc/custom_strategy.py). \n",
    "You can notice, that the most important thing done by INC\n",
    "was to find the operator, which had the most significant impact on the loss of accuracy and disable it from quantization if needed. \n",
    "You can see below which operator was excluded by `mse` strategy in last print given by \n",
    "[resnet_mse.py](https://github.com/apache/incubator-mxnet/blob/master/example/quantization_inc/resnet_mse.py) \n",
    "script:  \n",
    "\n",
    "{'excluded_symbols': ['**sg_onednn_conv_bn_act_0**'], 'quantized_dtype': 'auto', 'quantize_mode': 'smart', 'quantize_granularity': 'tensor-wise'}\n",
    "\n",
    "\n",
    "## Tips\n",
    "- In order to get a solution that generalizes well, evaluate the model (in eval_func) on a representative dataset.\n",
    "- With `history.snapshot` file (generated by INC) you can recover any model that was generated during the tuning process:\n",
    "  ```python\n",
    "  from neural_compressor.utils.utility import recover\n",
    "\n",
    "  quantized_model = recover(f32_model, 'nc_workspace/<tuning date>/history.snapshot', configuration_idx).model\n",
    "  ```"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}