{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--- Licensed to the Apache Software Foundation (ASF) under one -->\n",
    "<!--- or more contributor license agreements.  See the NOTICE file -->\n",
    "<!--- distributed with this work for additional information -->\n",
    "<!--- regarding copyright ownership.  The ASF licenses this file -->\n",
    "<!--- to you under the Apache License, Version 2.0 (the -->\n",
    "<!--- \"License\"); you may not use this file except in compliance -->\n",
    "<!--- with the License.  You may obtain a copy of the License at -->\n",
    "\n",
    "<!---   http://www.apache.org/licenses/LICENSE-2.0 -->\n",
    "\n",
    "<!--- Unless required by applicable law or agreed to in writing, -->\n",
    "<!--- software distributed under the License is distributed on an -->\n",
    "<!--- \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->\n",
    "<!--- KIND, either express or implied.  See the License for the -->\n",
    "<!--- specific language governing permissions and limitations -->\n",
    "<!--- under the License. -->\n",
    "\n",
    "# Activation Blocks\n",
    "\n",
    "Deep neural networks are a way to express a nonlinear function with lots of parameters from input data to outputs. The nonlinearities that allow neural networks to capture complex patterns in data are referred to as activation functions. Over the course of the development of neural networks, several nonlinear activation functions have been introduced to make gradient-based deep learning tractable. \n",
    "\n",
    "If you are looking to answer the question, 'which activation function should I use for my neural network model?', you should probably go with *ReLU*. Unless you're trying to implement something like a gating mechanism, like in LSTMs or GRU cells, then you should opt for sigmoid and/or tanh in those cells. However, if you have a working model architecture and you're trying to improve its performance by swapping out activation functions or treating the activation function as a hyperparameter, then you may want to try hand-designed activations like SELU or a function discovered by reinforcement learning and exhaustive search like Swish. This guide describes these activation functions and others implemented in MXNet in detail.\n",
    "\n",
    "## Visualizing Activations\n",
    "In order to compare the various activation functions and to understand the nuances of their differences we have a snippet of code to plot the activation functions (used in the forward pass) and their gradients (used in the backward pass)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "import numpy as np\n",
    "import mxnet as mx\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "def visualize_activation(activation_fn):\n",
    "    data = np.linspace(-10, 10, 501)\n",
    "    x = mx.nd.array(data)\n",
    "    x.attach_grad()\n",
    "    with mx.autograd.record():\n",
    "        y = activation_fn(x)\n",
    "    y.backward()\n",
    "    \n",
    "    plt.figure()\n",
    "    plt.plot(data, y.asnumpy())\n",
    "    plt.plot(data, x.grad.asnumpy())\n",
    "    activation = activation_fn.name[:-1]\n",
    "    plt.legend([\"{} activation\".format(activation), \"{} gradient\".format(activation)])\n",
    "    \n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sigmoids\n",
    "\n",
    "### Sigmoid\n",
    "\n",
    "The sigmoid activation function, also known as the logistic function or logit function, is perhaps the most widely known activation owing to its [long history](https://web.stanford.edu/class/psych209a/ReadingsByDate/02_06/PDPVolIChapter8.pdf) in neural network training and appearance in logistic regression and kernel methods for classification. \n",
    "\n",
    "The sigmoid activation is a non-linear function that transforms any real valued input to a value between 0 and 1, giving it a natural probabilistic interpretation. The sigmoid takes the form of the function below.\n",
    "\n",
    "$$ \\sigma(x) = \\dfrac{1}{1 + e^x} $$ or alternatively\n",
    "\n",
    "$$ \\sigma(x) = \\dfrac{e^x}{e^x + 1} $$\n",
    "\n",
    "Warning: the term sigmoid is overloaded and can be used to refer to the class of 's' shaped functions or particularly to the logistic function that we've just described. In MxNet the sigmoid activation specifically refers to logistic function sigmoid."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "visualize_activation(mx.gluon.nn.Activation('sigmoid'))\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![sigmoid activation and gradient](images/sigmoid.png)\n",
    "\n",
    "\n",
    "The sigmoid activation has since fallen out of use as the preferred activation function in designing neural networks due to some of its properties, shown in the plot above, like not being zero-centered and inducing vanishing gradients, that leads to poor performance during neural network training. Vanishing gradients here refers to the tendency of the gradient of the sigmoid function to be nearly zero for most input values. \n",
    "\n",
    "### tanh\n",
    "The tanh, or hyperbolic tangent, activation function is also an s shaped curve albeit one whose output values range from -1 to 1. It is defined by the mathematical equation:\n",
    "\n",
    "$$ tanh(x) = \\dfrac{e^x - e^{-x}}{e^x + e^{-x}}$$ \n",
    "\n",
    "tanh addresses the issues of not being zero centered associated with the sigmoid activation function but still retains the vanishing gradient problems due to the gradient being asymptotically zero for values outside a narrow range of inputs. \n",
    "\n",
    "In fact, the tanh can be rewritten as,\n",
    "\n",
    "$$tanh(x) = \\dfrac{e^{2x} - 1}{e^{2x} + 1}$$\n",
    "\n",
    "which shows its direct relation to sigmoid by the following equation:\n",
    "\n",
    "\n",
    "\n",
    "$$ tanh(x) = 2\\sigma(2x) - 1$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "visualize_activation(mx.gluon.nn.Activation('tanh'))\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![tanh activation and gradient](images/tanh.png)\n",
    "\n",
    "\n",
    "The use of tanh as activation functions in place of the logistic function was popularized by the success of the [LeNet architecture](http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf) and the [methods paper](http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf) by LeCun et al.\n",
    "\n",
    "### SoftSign\n",
    "\n",
    "The SoftSign activation is an alternative to tanh that is also centered at zero but converges asymptotically to -1 and 1 polynomially instead of exponentially. This means that the SoftSign activation does not saturate as quickly as tanh. As such, there are a greater range of input values for which the softsign assigns an output of strictly between -1 and 1.\n",
    "\n",
    "$$ softsign(x) = \\dfrac{x}{abs(x) + 1} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "visualize_activation(mx.gluon.nn.Activation('softsign'))\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![softsign activation and gradient](images/softsign.png)\n",
    "\n",
    "\n",
    "The softsign function is not a commonly used activation with most neural networks and still suffers from the vanishing gradient problem as seen in the graph above.\n",
    "\n",
    "## Rectifiers\n",
    "\n",
    "### ReLU\n",
    "ReLU, or Rectified Linear Unit is the most common activation function in convolutional neural networks and introduces a simple nonlinearity. When the value of the input into ReLU is positive, then it retains the same value. When the value is negative then it becomes zero. In equation form, the ReLU function is given as:\n",
    "\n",
    "$$ ReLU(x) = \\mathtt{max}(0, x) $$\n",
    "\n",
    "ReLU was introduced to neural networks in the [paper by Hahnloser et al](https://papers.nips.cc/paper/1793-permitted-and-forbidden-sets-in-symmetric-threshold-linear-networks.pdf) and gained widespread popularity after it was shown in the [paper](https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf) by Alex Krizhevsky et al to perform much better than sigmoid and tanh. This paper also introduced the AlexNet CNN that won the ILSVRC challenge in 2012.\n",
    "\n",
    "ReLU is the most widely used activation due to its simplicity and performance across multiple datasets and although there have been efforts to introduce activation functions, many of them described in this tutorial, that improve on ReLU, they have not gained as much widespread adoption."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "visualize_activation(mx.gluon.nn.Activation('relu'))\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![relu activation and gradient](images/relu.png)\n",
    "\n",
    "\n",
    "As shown above, the ReLU activation mitigates the vanishing gradient problem associated with the sigmoid family of activations, by having a larger (infinite) range of values where its gradient is non-zero. However, one drawback of ReLU as an activation function is a phenomenon referred to as the 'Dying ReLU', where gradient-based parameter updates can happen in such a way that the gradient flowing through a ReLU unit is always zero and the connection is never activated. This can largely be addressed by ensuring that the tuning the learning rate to ensure that it's not set too large when training ReLU networks.\n",
    "\n",
    "### SoftReLU\n",
    "\n",
    "SoftReLU also known as SmoothReLU or SoftPlus is a nonlinear activation function that takes the form\n",
    "\n",
    "$$ SoftReLU(x) = log(1 + e^x)$$\n",
    "\n",
    "The SoftReLU can be seen as a smooth version of the ReLU by observing that its derivative is the sigmoid, seen below, which is a smooth version of the gradient of the ReLU shown above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "visualize_activation(mx.gluon.nn.Activation('softrelu'))\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![softrelu activation and gradient](images/softrelu.png)\n",
    "\n",
    "\n",
    "### Leaky ReLU\n",
    "\n",
    "Leaky ReLUs are a variant of ReLU that multiply the input by a small positive parameter $\\alpha$ when the value is negative. Unlike the ReLU which sets the activation and gradient for negative values to zero, the LeakyReLU allows a small gradient. The equation for the LeakyReLU is:\n",
    "\n",
    "$$ LeakyReLU(\\alpha, x) = \\begin{cases}\n",
    "    x,& \\text{if } x\\geq 0\\\\\n",
    "    \\alpha x,              & \\text{otherwise}\n",
    "\\end{cases}$$ \n",
    "\n",
    "where $\\alpha > 0$ is small positive number. In MXNet, by default the $\\alpha$ parameter is set to 0.01.\n",
    "\n",
    "Here is a visualization for the LeakyReLU with $\\alpha = 0.05$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "visualize_activation(mx.gluon.nn.LeakyReLU(0.05))\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![leakyrelu activation and gradient](images/leakyrelu.png)\n",
    "\n",
    "\n",
    "As shown in the graph, the LeakyReLU's gradient is non-zero everywhere, in an attempt to address the ReLU's gradient being zero for all negative values.\n",
    "\n",
    "### PReLU\n",
    "The PReLU activation function, or Parametric Leaky ReLU introduced by [He et al](https://arxiv.org/pdf/1502.01852.pdf), is a version of LeakyReLU that learns the parameter $\\alpha$ during training. An initialization parameter is passed into the PreLU activation layer and this is treated as a learnable parameter that is updated via gradient descent during training. This is in contrast to LeakyReLU where $\\alpha$ is a hyperparameter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "prelu = mx.gluon.nn.PReLU(mx.init.Normal(0.05))\n",
    "prelu.initialize()\n",
    "visualize_activation(prelu)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![prelu activation and gradient](images/prelu.png)\n",
    "\n",
    "\n",
    "The activation function and activation gradient of PReLU have the same shape as LeakyRELU.\n",
    "\n",
    "### ELU\n",
    "\n",
    "The ELU or exponential linear unit introduced by [Clevert et al](https://arxiv.org/abs/1511.07289) also addresses the vanishing gradient problem like ReLU and its variants but unlike the ReLU family, ELU allows negative values which may allow them to push mean unit activations closer to zero like batch normalization. \n",
    "\n",
    "The ELU function has the form\n",
    "\n",
    "$$ ELU(\\alpha, x) = \\begin{cases}\n",
    "    x,& \\text{if } x\\geq 0\\\\\n",
    "    \\alpha (e^x - 1),              & \\text{otherwise}\n",
    "\\end{cases}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "visualize_activation(mx.gluon.nn.ELU())\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![elu activation and gradient](images/elu.png)\n",
    "\n",
    "\n",
    "### SELU\n",
    "SELU stands for Scaled Exponential Linear Unit and was introduced by [Klambuer et al](https://arxiv.org/abs/1706.02515) and is a modification of the ELU that improves the normalization of its outputs towards a zero mean and unit variance.\n",
    "\n",
    "The SELU function has the form\n",
    "\n",
    "$$ SELU(\\alpha, x) = \\lambda \\cdot\\begin{cases}\n",
    "    x,& \\text{if } x\\geq 0\\\\\n",
    "    \\alpha (e^x - 1),              & \\text{otherwise}\n",
    "\\end{cases}$$\n",
    "\n",
    "In SELU, unlike ELU, the parameters $\\alpha$ and $\\lambda$ are fixed parameters calculated from the data. For standard scaled inputs, these values are $$\\alpha=1.6732, \\lambda=1.0507$$ as calculated in the paper."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "visualize_activation(mx.gluon.nn.SELU())\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![selu activation and gradient](images/selu.png)\n",
    "\n",
    "\n",
    "### Swish\n",
    "Swish is an activation function that attempts to address the shortcomings of ReLU by combining ideas from ReLU and sigmoid. Swish was discovered by searching the space of activation functions using a combination of exhaustive and reinforcement learning-based search and was introduced in the paper by [Ramchandran et al](https://arxiv.org/pdf/1710.05941.pdf).\n",
    "\n",
    "The swish function is given as \n",
    "\n",
    "$$ swish(x) = x\\cdot\\sigma(\\beta x)$$\n",
    "\n",
    "where $\\sigma$ is the sigmoid activation function $\\sigma(x) = \\frac{1}{1 + e^{-x}}$ described above and $\\beta$ is a hyperparameter set to 1 by default in MXNet."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "visualize_activation(mx.gluon.nn.Swish())\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![swish activation and gradient](images/swish.png)\n",
    "\n",
    "\n",
    "\n",
    "## Summary\n",
    "\n",
    "* Activation functions introduce non-linearities to deep neural network that allow the models to capture complex interactions between features of the data.\n",
    "* ReLU is the activation function that is commonly used in many neural network architectures because of its simplicity and performance.\n",
    "* Sigmoids like the logistic (sigmoid) function and tanh where the first kinds of activation functions used in neural networks. They have since fallen out of use because of their tendency to saturate and have vanishing gradients.\n",
    "* Rectifiers like ReLU do not saturate like the Sigmoids and so address the vanishing gradient problem making them the de facto activation functions. ReLU however is still plagued by the dying ReLU problem.\n",
    "* LeakyReLU and PReLU are two similar approaches to improve ReLU and address the dying ReLU by introducing a parameter $\\alpha$ (learned in PReLU) that leaks to the gradient of negative inputs\n",
    "* MXNet also implements custom state-of-the-art activations like ELU, SELU and Swish.\n",
    "\n",
    "\n",
    "\n",
    "## Next Steps\n",
    "\n",
    "Activations are just one component of neural network architectures. Here are a few MXNet resources to learn more about activation functions and how they they combine with other components of neural nets.\n",
    "* Learn how to create a Neural Network with these activation layers and other neural network layers in the [gluon crash course](/api/python/docs/tutorials/getting-started/crash-course/index.html).\n",
    "* Check out the guide to MXNet [gluon layers and blocks](/api/python/docs/tutorials/packages/gluon/blocks/nn.html) to learn about the other neural network layers in implemented in MXNet and how to create custom neural networks with these layers.\n",
    "* Also check out the [guide to normalization layers](/api/python/docs/tutorials/packages/gluon/training/normalization/index.html) to learn about neural network layers that normalize their inputs.\n",
    "* Finally take a look at the [Custom Layer guide](/api/python/docs/tutorials/extend/custom_layer.html) to learn how to implement your own custom activation layer."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}