{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f1502018",
   "metadata": {},
   "source": [
    "<!--- Licensed to the Apache Software Foundation (ASF) under one -->\n",
    "<!--- or more contributor license agreements.  See the NOTICE file -->\n",
    "<!--- distributed with this work for additional information -->\n",
    "<!--- regarding copyright ownership.  The ASF licenses this file -->\n",
    "<!--- to you under the Apache License, Version 2.0 (the -->\n",
    "<!--- \"License\"); you may not use this file except in compliance -->\n",
    "<!--- with the License.  You may obtain a copy of the License at -->\n",
    "\n",
    "<!---   http://www.apache.org/licenses/LICENSE-2.0 -->\n",
    "\n",
    "<!--- Unless required by applicable law or agreed to in writing, -->\n",
    "<!--- software distributed under the License is distributed on an -->\n",
    "<!--- \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->\n",
    "<!--- KIND, either express or implied.  See the License for the -->\n",
    "<!--- specific language governing permissions and limitations -->\n",
    "<!--- under the License. -->\n",
    "\n",
    "# Step 2: Create a neural network\n",
    "\n",
    "In this step, you learn how to use NP on Apache MXNet to create neural networks\n",
    "in Gluon. In addition to the `np` package that you learned about in the previous\n",
    "step [Step 1: Manipulate data with NP on MXNet](./1-nparray.ipynb), you also need to\n",
    "import the neural network modules from `gluon`. Gluon includes built-in neural\n",
    "network layers in the following two modules:\n",
    "\n",
    "1. `mxnet.gluon.nn`: NN module that maintained by the mxnet team\n",
    "2. `mxnet.gluon.contrib.nn`: Experiemental module that is contributed by the\n",
    "community\n",
    "\n",
    "Use the following commands to import the packages required for this step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b5edb85d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from mxnet import np, npx\n",
    "from mxnet.gluon import nn\n",
    "npx.set_np()  # Change MXNet to the numpy-like mode."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db45d00c",
   "metadata": {},
   "source": [
    "## Create your neural network's first layer\n",
    "\n",
    "In this section, you will create a simple neural network with Gluon. One of the\n",
    "simplest network you can create is a single **Dense** layer or **densely-\n",
    "connected** layer. A dense layer consists of nodes in the input that are\n",
    "connected to every node in the next layer. Use the following code example to\n",
    "start with a dense layer with five output units."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "65f9bd2d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dense(-1 -> 5, linear)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer = nn.Dense(5)\n",
    "layer\n",
    "# output: Dense(-1 -> 5, linear)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1dd2c834",
   "metadata": {},
   "source": [
    "In the example above, the output is `Dense(-1 -> 5, linear)`. The **-1** in the\n",
    "output denotes that the size of the input layer is not specified during\n",
    "initialization.\n",
    "\n",
    "You can also call the **Dense** layer with an `in_units` parameter if you know\n",
    "the shape of your input unit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "76eaacee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dense(3 -> 5, linear)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer = nn.Dense(5,in_units=3)\n",
    "layer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41a603cf",
   "metadata": {},
   "source": [
    "In addition to the `in_units` param, you can also add an activation function to\n",
    "the layer using the `activation` param. The Dense layer implements the operation\n",
    "\n",
    "$$output = \\sigma(W \\cdot X + b)$$\n",
    "\n",
    "Call the Dense layer with an `activation` parameter to use an activation\n",
    "function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "69a4b1e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "layer = nn.Dense(5, in_units=3,activation='relu')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e319ccd",
   "metadata": {},
   "source": [
    "Voila! Congratulations on creating a simple neural network. But for most of your\n",
    "use cases, you will need to create a neural network with more than one dense\n",
    "layer or with multiple types of other layers. In addition to the `Dense` layer,\n",
    "you can find more layers at [mxnet nn layers](../../../api/gluon/nn/index.rst#module-mxnet.gluon.nn)\n",
    "\n",
    "So now that you have created a neural network, you are probably wondering how to\n",
    "pass data into your network?\n",
    "\n",
    "First, you need to initialize the network weights, if you use the default\n",
    "initialization method which draws random values uniformly in the range $[-0.7,\n",
    "0.7]$. You can see this in the following example.\n",
    "\n",
    "**Note**: Initialization is discussed at a little deeper detail in the next\n",
    "notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "021e754d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[03:51:58] /work/mxnet/src/storage/storage.cc:202: Using Pooled (Naive) StorageManager for CPU\n"
     ]
    }
   ],
   "source": [
    "layer.initialize()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e5925d3",
   "metadata": {},
   "source": [
    "Now that you have initialized your network, you can give it data. Passing data\n",
    "through a network is also called a forward pass. You can do a forward pass with\n",
    "random data, shown in the following example. First, you create a `(10,3)` shape\n",
    "random input `x` and feed the data into the layer to compute the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8018b3dd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.00881556, 0.01138476, 0.        , 0.        , 0.01936117],\n",
       "       [0.        , 0.0035577 , 0.06854778, 0.03361227, 0.        ],\n",
       "       [0.05338536, 0.01661206, 0.        , 0.00864646, 0.        ],\n",
       "       [0.        , 0.        , 0.        , 0.03724934, 0.03653988],\n",
       "       [0.        , 0.00657675, 0.00472842, 0.        , 0.04593495],\n",
       "       [0.        , 0.00795121, 0.        , 0.        , 0.0595542 ],\n",
       "       [0.02296758, 0.01650022, 0.        , 0.        , 0.03874438],\n",
       "       [0.02207369, 0.02308735, 0.04558432, 0.01468477, 0.        ],\n",
       "       [0.        , 0.02254252, 0.        , 0.        , 0.03834942],\n",
       "       [0.08092358, 0.04224379, 0.        , 0.        , 0.        ]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.random.uniform(-1,1,(10,3))\n",
    "layer(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "102f8c57",
   "metadata": {},
   "source": [
    "The layer produces a `(10,5)` shape output from your `(10,3)` input.\n",
    "\n",
    "**When you don't specify the `in_unit` parameter, the system  automatically\n",
    "infers it during the first time you feed in data during the first forward step\n",
    "after you create and initialize the weights.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b54c523e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'weight': Parameter (shape=(5, 3), dtype=float32),\n",
       " 'bias': Parameter (shape=(5,), dtype=float32)}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer.params"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63e986db",
   "metadata": {},
   "source": [
    "The `weights` and `bias` can be accessed using the `.data()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "656bda95",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.01607367,  0.05928481, -0.0319057 ],\n",
       "       [-0.05814854,  0.01664302, -0.02215988],\n",
       "       [-0.04094896,  0.03231322,  0.05914024],\n",
       "       [ 0.05500493,  0.03504761,  0.05073748],\n",
       "       [ 0.00943237, -0.06525595, -0.04184696]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer.weight.data()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "011e290e",
   "metadata": {},
   "source": [
    "## Chain layers into a neural network using nn.Sequential\n",
    "\n",
    "Sequential provides a special way of rapidly building networks when when the\n",
    "network architecture follows a common design pattern: the layers look like a\n",
    "stack of pancakes. Many networks follow this pattern: a bunch of layers, one\n",
    "stacked on top of another, where the output of each layer is fed directly to the\n",
    "input to the next layer. To use sequential, simply provide a list of layers\n",
    "(pass in the layers by calling `net.add(<Layer goes here!>`). To do this you can\n",
    "use your previous example of Dense layers and create a 3-layer multi layer\n",
    "perceptron. You can create a sequential block using `nn.Sequential()` method and\n",
    "add layers using `add()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a2ab463c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): Dense(3 -> 5, Activation(relu))\n",
       "  (1): Dense(-1 -> 25, Activation(relu))\n",
       "  (2): Dense(-1 -> 2, linear)\n",
       ")"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net = nn.Sequential()\n",
    "\n",
    "net.add(nn.Dense(5,in_units=3,activation='relu'),\n",
    "        nn.Dense(25, activation='relu'), nn.Dense(2))\n",
    "net"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ee4727c",
   "metadata": {},
   "source": [
    "The layers are ordered exactly the way you defined your neural network with\n",
    "index starting from 0. You can access the layers by indexing the network using\n",
    "`[]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1b66ee0d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dense(-1 -> 25, Activation(relu))"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7f5f323",
   "metadata": {},
   "source": [
    "## Create a custom neural network architecture flexibly\n",
    "\n",
    "`nn.Sequential()` allows you to create your multi-layer neural network with\n",
    "existing layers from `gluon.nn`. It also includes a pre-defined `forward()`\n",
    "function that sequentially executes added layers. But what if the built-in\n",
    "layers are not sufficient for your needs. If you want to create networks like\n",
    "ResNet which has complex but repeatable components, how do you create such a\n",
    "network?\n",
    "\n",
    "In gluon, every neural network layer is defined by using a base class\n",
    "`nn.Block()`. A Block has one main job - define a forward method that takes some\n",
    "input x and generates an output. A Block can just do something simple like apply\n",
    "an activation function. It can combine multiple layers together in a single\n",
    "block or also combine a bunch of other Blocks together in creative ways to\n",
    "create complex networks like Resnet. In this case, you will construct three\n",
    "Dense layers. The `forward()` method can then invoke the layers in turn to\n",
    "generate its output.\n",
    "\n",
    "Create a subclass of `nn.Block` and implement two methods by using the following\n",
    "code.\n",
    "\n",
    "- `__init__` create the layers\n",
    "- `forward` define the forward function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "997c5a6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Net(nn.Block):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "    def forward(self, x):\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b9cc930e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MLP(\n",
       "  (dense1): Dense(-1 -> 5, Activation(relu))\n",
       "  (dense2): Dense(-1 -> 25, Activation(relu))\n",
       "  (dense3): Dense(-1 -> 2, linear)\n",
       ")"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class MLP(nn.Block):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.dense1 = nn.Dense(5,activation='relu')\n",
    "        self.dense2 = nn.Dense(25,activation='relu')\n",
    "        self.dense3 = nn.Dense(2)\n",
    "\n",
    "    def forward(self, x):\n",
    "        layer1 = self.dense1(x)\n",
    "        layer2 = self.dense2(layer1)\n",
    "        layer3 = self.dense3(layer2)\n",
    "        return layer3\n",
    "\n",
    "net = MLP()\n",
    "net"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f275942e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'weight': Parameter (shape=(5, -1), dtype=float32),\n",
       " 'bias': Parameter (shape=(5,), dtype=float32)}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.dense1.params"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab1ec782",
   "metadata": {},
   "source": [
    "Each layer includes parameters that are stored in a `Parameter` class. You can\n",
    "access them using the `params()` method.\n",
    "\n",
    "## Creating custom layers using Parameters (Blocks API)\n",
    "\n",
    "MXNet includes a `Parameter` method to hold your parameters in each layer. You\n",
    "can create custom layers using the `Parameter` class to include computation that\n",
    "may otherwise be not included in the built-in layers. For example, for a dense\n",
    "layer, the weights and biases will be created using the `Parameter` method. But\n",
    "if you want to add additional computation to the dense layer, you can create it\n",
    "using parameter method.\n",
    "\n",
    "Instantiate a parameter, e.g weights with a size `(5,0)` using the `shape`\n",
    "argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "2c3088e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Parameter (shape=(5, -1), dtype=<class 'numpy.float32'>),\n",
       " Parameter (shape=(5, -1), dtype=<class 'numpy.float32'>))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from mxnet.gluon import Parameter\n",
    "\n",
    "weight = Parameter(\"custom_parameter_weight\",shape=(5,-1))\n",
    "bias = Parameter(\"custom_parameter_bias\",shape=(5,-1))\n",
    "\n",
    "weight,bias"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e430e51c",
   "metadata": {},
   "source": [
    "The `Parameter` method includes a `grad_req` argument that specifies how you\n",
    "want to capture gradients for this Parameter. Under the hood, that lets gluon\n",
    "know that it has to call `.attach_grad()` on the underlying array. By default,\n",
    "the gradient is updated everytime the gradient is written to the grad\n",
    "`grad_req='write'`.\n",
    "\n",
    "Now that you know how parameters work, you are ready to create your very own\n",
    "fully-connected custom layer.\n",
    "\n",
    "To create the custom layers using parameters, you can use the same skeleton with\n",
    "`nn.Block` base class. You will create a custom dense layer that takes parameter\n",
    "x and returns computed `w*x + b` without any activation function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "9b0c2ca2",
   "metadata": {},
   "outputs": [],
   "source": [
    "class custom_layer(nn.Block):\n",
    "   def __init__(self, out_units, in_units=0):\n",
    "       super().__init__()\n",
    "       self.weight = Parameter(\"weight\", shape=(in_units,out_units), allow_deferred_init=True)\n",
    "       self.bias = Parameter(\"bias\", shape=(out_units,), allow_deferred_init=True)\n",
    "   def forward(self, x):\n",
    "       return np.dot(x, self.weight.data()) + self.bias.data()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17d3916e",
   "metadata": {},
   "source": [
    "Parameter can be instantiated before the corresponding data is instantiated. For\n",
    "example, when you instantiate a Block but the shapes of each parameter still\n",
    "need to be inferred, the Parameter will wait for the shape to be inferred before\n",
    "allocating memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "76991cb3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.05604633, -0.06238654,  0.02687173],\n",
       "       [-0.02687152, -0.04365591, -0.00518382],\n",
       "       [-0.02849396, -0.09980228, -0.00695815],\n",
       "       [-0.04527343, -0.00275569, -0.01376584]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dense = custom_layer(3,in_units=5)\n",
    "dense.initialize()\n",
    "dense(np.random.uniform(size=(4, 5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fabc9407",
   "metadata": {},
   "source": [
    "Similarly, you can use the following code to implement a famous network called\n",
    "[LeNet](http://yann.lecun.com/exdb/lenet/) through `nn.Block` using the built-in\n",
    "`Dense` layer and using `custom_layer` as the last layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "58855487",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LeNet(nn.Block):\n",
    "   def __init__(self):\n",
    "       super().__init__()\n",
    "       self.conv1 = nn.Conv2D(channels=6, kernel_size=3, activation='relu')\n",
    "       self.pool1 = nn.MaxPool2D(pool_size=2, strides=2)\n",
    "       self.conv2 = nn.Conv2D(channels=16, kernel_size=3, activation='relu')\n",
    "       self.pool2 = nn.MaxPool2D(pool_size=2, strides=2)\n",
    "       self.dense1 = nn.Dense(120, activation=\"relu\")\n",
    "       self.dense2 = nn.Dense(84, activation=\"relu\")\n",
    "       self.dense3 = nn.Dense(10)\n",
    "   def forward(self, x):\n",
    "       x = self.conv1(x)\n",
    "       x = self.pool1(x)\n",
    "       x = self.conv2(x)\n",
    "       x = self.pool2(x)\n",
    "       x = self.dense1(x)\n",
    "       x = self.dense2(x)\n",
    "       x = self.dense3(x)\n",
    "       return x\n",
    "\n",
    "lenet = LeNet()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d69ba80c",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LeNet_custom(nn.Block):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.conv1 = nn.Conv2D(channels=6, kernel_size=3, activation='relu')\n",
    "        self.pool1 = nn.MaxPool2D(pool_size=2, strides=2)\n",
    "        self.conv2 = nn.Conv2D(channels=16, kernel_size=3, activation='relu')\n",
    "        self.pool2 = nn.MaxPool2D(pool_size=2, strides=2)\n",
    "        self.dense1 = nn.Dense(120, activation=\"relu\")\n",
    "        self.dense2 = nn.Dense(84, activation=\"relu\")\n",
    "        self.dense3 = custom_layer(10,84)\n",
    "    def forward(self, x):\n",
    "        x = self.conv1(x)\n",
    "        x = self.pool1(x)\n",
    "        x = self.conv2(x)\n",
    "        x = self.pool2(x)\n",
    "        x = self.dense1(x)\n",
    "        x = self.dense2(x)\n",
    "        x = self.dense3(x)\n",
    "        return x\n",
    "\n",
    "lenet_custom = LeNet_custom()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "32c9aff7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Lenet:\n",
      "[[-0.00081668 -0.00340701  0.00199039 -0.00121501 -0.00063157  0.00081476\n",
      "  -0.0011025  -0.00216652 -0.00015363 -0.00110007]]\n",
      "Custom Lenet:\n",
      "[[-0.02656163  0.04184292 -0.00254037 -0.06494093 -0.00952166 -0.01921579\n",
      "   0.05423243 -0.02774546  0.06823301  0.00313227]]\n"
     ]
    }
   ],
   "source": [
    "image_data = np.random.uniform(-1,1, (1,1,28,28))\n",
    "\n",
    "lenet.initialize()\n",
    "lenet_custom.initialize()\n",
    "\n",
    "print(\"Lenet:\")\n",
    "print(lenet(image_data))\n",
    "\n",
    "print(\"Custom Lenet:\")\n",
    "print(lenet_custom(image_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa9cc4bd",
   "metadata": {},
   "source": [
    "You can use `.data` method to access the weights and bias of a particular layer.\n",
    "For example, the following  accesses the first layer's weight and sixth layer's bias."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "1c293d07",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((6, 1, 3, 3), (120,))"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lenet.conv1.weight.data().shape, lenet.dense1.bias.data().shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c94cea7",
   "metadata": {},
   "source": [
    "## Using predefined (pretrained) architectures\n",
    "\n",
    "Till now, you have seen how to create your own neural network architectures. But\n",
    "what if you want to replicate or baseline your dataset using some of the common\n",
    "models in computer visions or natural language processing (NLP). Gluon includes\n",
    "common architectures that you can directly use. The Gluon Model Zoo provides a\n",
    "collection of off-the-shelf models e.g. RESNET, BERT etc. These architectures\n",
    "are found at:\n",
    "\n",
    "- [Gluon CV model zoo](https://cv.gluon.ai/model_zoo/index.html)\n",
    "\n",
    "- [Gluon NLP model zoo](https://nlp.gluon.ai/model_zoo/index.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "5b7afa70",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading /home/jenkins_slave/.mxnet/models/resnet50_v2-ecdde353.zip00383814-e655-4621-a110-5ffefe3eb69c from https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/models/resnet50_v2-ecdde353.zip...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(1, 1000)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from mxnet.gluon import model_zoo\n",
    "\n",
    "net = model_zoo.vision.resnet50_v2(pretrained=True)\n",
    "net.hybridize()\n",
    "\n",
    "dummy_input = np.ones(shape=(1,3,224,224))\n",
    "output = net(dummy_input)\n",
    "output.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af120df5",
   "metadata": {},
   "source": [
    "## Deciding the paradigm for your network\n",
    "\n",
    "In MXNet, Gluon API (Imperative programming paradigm) provides a user friendly\n",
    "way for quick prototyping, easy debugging and natural control flow for people\n",
    "familiar with python programming.\n",
    "\n",
    "However, at the backend, MXNET can also convert the network using Symbolic or\n",
    "Declarative programming into static graphs with low level optimizations on\n",
    "operators. However, static graphs are less flexible because any logic must be\n",
    "encoded into the graph as special operators like scan, while_loop and cond. It’s\n",
    "also hard to debug.\n",
    "\n",
    "So how can you make use of symbolic programming while getting the flexibility of\n",
    "imperative programming to quickly prototype and debug?\n",
    "\n",
    "Enter **HybridBlock**\n",
    "\n",
    "HybridBlocks can run in a fully imperatively way where you define their\n",
    "computation with real functions acting on real inputs. But they’re also capable\n",
    "of running symbolically, acting on placeholders. Gluon hides most of this under\n",
    "the hood so you will only need to know how it works when you want to write your\n",
    "own layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "abc3493e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "HybridSequential(\n",
       "  (0): Dense(3 -> 5, Activation(relu))\n",
       "  (1): Dense(-1 -> 25, Activation(relu))\n",
       "  (2): Dense(-1 -> 2, linear)\n",
       ")"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net_hybrid_seq = nn.HybridSequential()\n",
    "\n",
    "net_hybrid_seq.add(nn.Dense(5,in_units=3,activation='relu'),\n",
    " nn.Dense(25, activation='relu'), nn.Dense(2) )\n",
    "net_hybrid_seq"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "581c517f",
   "metadata": {},
   "source": [
    "To compile and optimize `HybridSequential`, you can call its `hybridize` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "af36f9fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "net_hybrid_seq.hybridize()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64823d61",
   "metadata": {},
   "source": [
    "## Creating custom layers using Parameters (HybridBlocks API)\n",
    "\n",
    "When you instantiated your custom layer, you specified the input dimension\n",
    "`in_units` that initializes the weights with the shape specified by `in_units`\n",
    "and `out_units`. If you leave the shape of `in_unit` as unknown, you defer the\n",
    "shape to the first forward pass. For the custom layer, you define the\n",
    "`infer_shape()` method and let the shape be inferred at runtime."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "eb9756b1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/work/mxnet/python/mxnet/util.py:755: UserWarning: Parameter 'weight' is already initialized, ignoring. Set force_reinit=True to re-initialize.\n",
      "  return func(*args, **kwargs)\n",
      "/work/mxnet/python/mxnet/util.py:755: UserWarning: Parameter 'bias' is already initialized, ignoring. Set force_reinit=True to re-initialize.\n",
      "  return func(*args, **kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[-0.07053316, -0.07457963,  0.01166525],\n",
       "       [-0.04170407, -0.07482161,  0.00179428],\n",
       "       [-0.07503258,  0.00660181, -0.01401043],\n",
       "       [-0.02333996, -0.06775613,  0.01459978]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class CustomLayer(nn.HybridBlock):\n",
    "    def __init__(self, out_units, in_units=-1):\n",
    "        super().__init__()\n",
    "        self.weight = Parameter(\"weight\", shape=(in_units, out_units), allow_deferred_init=True)\n",
    "        self.bias = Parameter(\"bias\", shape=(out_units,), allow_deferred_init=True)\n",
    "\n",
    "    def forward(self, x):\n",
    "        print(self.weight.shape, self.bias.shape)\n",
    "        return np.dot(x, self.weight.data()) + self.bias.data()\n",
    "\n",
    "    def infer_shape(self, x):\n",
    "        print(self.weight.shape,x.shape)\n",
    "        self.weight.shape = (x.shape[-1],self.weight.shape[1])\n",
    "        dense = CustomLayer(3)\n",
    "\n",
    "dense.initialize()\n",
    "dense(np.random.uniform(size=(4, 5)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "844e4d15",
   "metadata": {},
   "source": [
    "### Performance\n",
    "\n",
    "To get a sense of the speedup from hybridizing, you can compare the performance\n",
    "before and after hybridizing by measuring the time it takes to make 1000 forward\n",
    "passes through the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "8e0d8df9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before hybridizing: 0.6034 sec\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After hybridizing: 0.2799 sec\n"
     ]
    }
   ],
   "source": [
    "from time import time\n",
    "\n",
    "def benchmark(net, x):\n",
    "    y = net(x)\n",
    "    start = time()\n",
    "    for i in range(1,1000):\n",
    "        y = net(x)\n",
    "    return time() - start\n",
    "\n",
    "x_bench = np.random.normal(size=(1,512))\n",
    "\n",
    "net_hybrid_seq = nn.HybridSequential()\n",
    "\n",
    "net_hybrid_seq.add(nn.Dense(256,activation='relu'),\n",
    "                   nn.Dense(128, activation='relu'),\n",
    "                   nn.Dense(2))\n",
    "net_hybrid_seq.initialize()\n",
    "\n",
    "print('Before hybridizing: %.4f sec'%(benchmark(net_hybrid_seq, x_bench)))\n",
    "net_hybrid_seq.hybridize()\n",
    "print('After hybridizing: %.4f sec'%(benchmark(net_hybrid_seq, x_bench)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f1f1349",
   "metadata": {},
   "source": [
    "Peeling back another layer, you also have a `HybridBlock` which is the hybrid\n",
    "version of the `Block` API.\n",
    "\n",
    "Similar to the `Blocks` API, you define a `forward` function for `HybridBlock`\n",
    "that takes an input `x`. MXNet takes care of hybridizing the model at the\n",
    "backend so you don't have to make changes to your code to convert it to a\n",
    "symbolic paradigm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "07c6cce6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before hybridizing: 0.5799 sec\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After hybridizing: 0.2603 sec\n"
     ]
    }
   ],
   "source": [
    "from mxnet.gluon import HybridBlock\n",
    "\n",
    "class MLP_Hybrid(HybridBlock):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.dense1 = nn.Dense(256,activation='relu')\n",
    "        self.dense2 = nn.Dense(128,activation='relu')\n",
    "        self.dense3 = nn.Dense(2)\n",
    "    def forward(self, x):\n",
    "        layer1 = self.dense1(x)\n",
    "        layer2 = self.dense2(layer1)\n",
    "        layer3 = self.dense3(layer2)\n",
    "        return layer3\n",
    "\n",
    "net_hybrid = MLP_Hybrid()\n",
    "net_hybrid.initialize()\n",
    "\n",
    "print('Before hybridizing: %.4f sec'%(benchmark(net_hybrid, x_bench)))\n",
    "net_hybrid.hybridize()\n",
    "print('After hybridizing: %.4f sec'%(benchmark(net_hybrid, x_bench)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12f5f4a9",
   "metadata": {},
   "source": [
    "Given a HybridBlock whose forward computation consists of going through other\n",
    "HybridBlocks, you can compile that section of the network by calling the\n",
    "HybridBlocks `.hybridize()` method.\n",
    "\n",
    "All of MXNet’s predefined layers are HybridBlocks. This means that any network\n",
    "consisting entirely of predefined MXNet layers can be compiled and run at much\n",
    "faster speeds by calling `.hybridize()`.\n",
    "\n",
    "## Saving and Loading your models\n",
    "\n",
    "The Blocks API also includes saving your models during and after training so\n",
    "that you can host the model for inference or avoid training the model again from\n",
    "scratch. Another reason would be to train your model using one language (like\n",
    "Python that has a lot of tools for training) and run inference using a different\n",
    "language.\n",
    "\n",
    "There are two ways to save your model in MXNet.\n",
    "1. Save/load the model weights/parameters only\n",
    "2. Save/load the model weights/parameters and the architectures\n",
    "\n",
    "\n",
    "### 1. Save/load the model weights/parameters only\n",
    "\n",
    "You can use `save_parameters` and `load_parameters` method to save and load the\n",
    "model weights. Take your simplest model `layer` and save your parameters first.\n",
    "The model parameters are the params that you save **after** you train your\n",
    "model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "abcaf4e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "file_name = 'layer.params'\n",
    "layer.save_parameters(file_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70a86a96",
   "metadata": {},
   "source": [
    "And now load this model again. To load the parameters into a model, you will\n",
    "first have to build the model. To do this, you will need to create a simple\n",
    "function to build it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "2bd2f6b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model():\n",
    "    layer = nn.Dense(5, in_units=3,activation='relu')\n",
    "    return layer\n",
    "\n",
    "layer_new = build_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "47024147",
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_new.load_parameters('layer.params')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1647f15",
   "metadata": {},
   "source": [
    "**Note**: The `save_parameters` and `load_parameters` method is used for models\n",
    "that use a `Block` method instead of  `HybridBlock` method to build the model.\n",
    "These models may have complex architectures where the model architectures may\n",
    "change during execution. E.g. if you have a model that uses an if-else\n",
    "conditional statement to choose between two different architectures.\n",
    "\n",
    "### 2. Save/load the model weights/parameters and the architectures\n",
    "\n",
    "For models that use the **HybridBlock**, the model architecture stays static and\n",
    "do no change during execution. Therefore both model parameters **AND**\n",
    "architecture can be saved and loaded using `export`, `imports` methods.\n",
    "\n",
    "Now look at your `MLP_Hybrid` model and export the model using the `export`\n",
    "function. The export function will export the model architecture into a `.json`\n",
    "file and model parameters into a `.params` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "0311a523",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('MLP_hybrid-symbol.json', 'MLP_hybrid-0000.params')"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net_hybrid.export('MLP_hybrid')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "12bea807",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('MLP_hybrid-symbol.json', 'MLP_hybrid-0000.params')"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net_hybrid.export('MLP_hybrid')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "380e7cdc",
   "metadata": {},
   "source": [
    "Similarly, to load this model back, you can use `gluon.nn.SymbolBlock`. To\n",
    "demonstrate that, load the network serialized above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "f2322422",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "with warnings.catch_warnings():\n",
    "    warnings.simplefilter(\"ignore\")\n",
    "    net_loaded = nn.SymbolBlock.imports(\"MLP_hybrid-symbol.json\",\n",
    "                                        ['data'], \"MLP_hybrid-0000.params\",\n",
    "                                        device=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "c7bc1d0f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.13653663, -0.07247495]])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net_loaded(x_bench)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e829ef5a",
   "metadata": {},
   "source": [
    "## Visualizing your models\n",
    "\n",
    "In MXNet, the `Block.Summary()` method allows you to view the block’s shape\n",
    "arguments and view the block’s parameters. When you combine multiple blocks into\n",
    "a model, the `summary()` applied on the model allows you to view each block’s\n",
    "summary, the total parameters, and the order of the blocks within the model. To\n",
    "do this the `Block.summary()` method requires one forward pass of the data,\n",
    "through your network, in order to create the graph necessary for capturing the\n",
    "corresponding shapes and parameters. Additionally, this method should be called\n",
    "before the hybridize method, since the hybridize method converts the graph into\n",
    "a symbolic one, potentially changing the operations for optimal computation.\n",
    "\n",
    "Look at the following examples\n",
    "\n",
    "- layer: our single layer network\n",
    "- Lenet: a non-hybridized LeNet network\n",
    "- net_Hybrid: our MLP Hybrid network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "16668c82",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------------\n",
      "        Layer (type)                                Output Shape         Param #\n",
      "================================================================================\n",
      "               Input                                     (10, 3)               0\n",
      "        Activation-1                                     (10, 5)               0\n",
      "             Dense-2                                     (10, 5)              20\n",
      "================================================================================\n",
      "Parameters in forward computation graph, duplicate included\n",
      "   Total params: 20\n",
      "   Trainable params: 20\n",
      "   Non-trainable params: 0\n",
      "Shared params in forward computation graph: 0\n",
      "Unique parameters in model: 20\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "layer.summary(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "7ece38c2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------------\n",
      "        Layer (type)                                Output Shape         Param #\n",
      "================================================================================\n",
      "               Input                              (1, 1, 28, 28)               0\n",
      "        Activation-1                              (1, 6, 26, 26)               0\n",
      "            Conv2D-2                              (1, 6, 26, 26)              60\n",
      "         MaxPool2D-3                              (1, 6, 13, 13)               0\n",
      "        Activation-4                             (1, 16, 11, 11)               0\n",
      "            Conv2D-5                             (1, 16, 11, 11)             880\n",
      "         MaxPool2D-6                               (1, 16, 5, 5)               0\n",
      "        Activation-7                                    (1, 120)               0\n",
      "             Dense-8                                    (1, 120)           48120\n",
      "        Activation-9                                     (1, 84)               0\n",
      "            Dense-10                                     (1, 84)           10164\n",
      "            Dense-11                                     (1, 10)             850\n",
      "            LeNet-12                                     (1, 10)               0\n",
      "================================================================================\n",
      "Parameters in forward computation graph, duplicate included\n",
      "   Total params: 60074\n",
      "   Trainable params: 60074\n",
      "   Non-trainable params: 0\n",
      "Shared params in forward computation graph: 0\n",
      "Unique parameters in model: 60074\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "lenet.summary(image_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da1e1a8f",
   "metadata": {},
   "source": [
    "You are able to print the summaries of the two networks `layer` and `lenet`\n",
    "easily since you didn't hybridize the two networks. However, the last network\n",
    "`net_Hybrid` was hybridized above and throws an `AssertionError` if you try\n",
    "`net_Hybrid.summary(x_bench)`. To print the summary for `net_Hybrid`, call\n",
    "another instance of the same network and instantiate it for our summary and then\n",
    "hybridize it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "6f429bc7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------------\n",
      "        Layer (type)                                Output Shape         Param #\n",
      "================================================================================\n",
      "               Input                                    (1, 512)               0\n",
      "        Activation-1                                    (1, 256)               0\n",
      "             Dense-2                                    (1, 256)          131328\n",
      "        Activation-3                                    (1, 128)               0\n",
      "             Dense-4                                    (1, 128)           32896\n",
      "             Dense-5                                      (1, 2)             258\n",
      "        MLP_Hybrid-6                                      (1, 2)               0\n",
      "================================================================================\n",
      "Parameters in forward computation graph, duplicate included\n",
      "   Total params: 164482\n",
      "   Trainable params: 164482\n",
      "   Non-trainable params: 0\n",
      "Shared params in forward computation graph: 0\n",
      "Unique parameters in model: 164482\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "net_hybrid_summary = MLP_Hybrid()\n",
    "\n",
    "net_hybrid_summary.initialize()\n",
    "\n",
    "net_hybrid_summary.summary(x_bench)\n",
    "\n",
    "net_hybrid_summary.hybridize()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f76b4ea5",
   "metadata": {},
   "source": [
    "## Next steps:\n",
    "\n",
    "Now that you have created a neural network, learn how to automatically compute\n",
    "the gradients in [Step 3: Automatic differentiation with autograd](./3-autograd.ipynb)."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}