{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--- Licensed to the Apache Software Foundation (ASF) under one -->\n",
    "<!--- or more contributor license agreements.  See the NOTICE file -->\n",
    "<!--- distributed with this work for additional information -->\n",
    "<!--- regarding copyright ownership.  The ASF licenses this file -->\n",
    "<!--- to you under the Apache License, Version 2.0 (the -->\n",
    "<!--- \"License\"); you may not use this file except in compliance -->\n",
    "<!--- with the License.  You may obtain a copy of the License at -->\n",
    "\n",
    "<!---   http://www.apache.org/licenses/LICENSE-2.0 -->\n",
    "\n",
    "<!--- Unless required by applicable law or agreed to in writing, -->\n",
    "<!--- software distributed under the License is distributed on an -->\n",
    "<!--- \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->\n",
    "<!--- KIND, either express or implied.  See the License for the -->\n",
    "<!--- specific language governing permissions and limitations -->\n",
    "<!--- under the License. -->\n",
    "\n",
    "\n",
    "# Logistic regression explained\n",
    "\n",
    "Logistic Regression is one of the first models newcomers to Deep Learning are implementing. The focus of this tutorial is to show how to do logistic regression using Gluon API.\n",
    "\n",
    "Before anything else, let's import required packages for this tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "import numpy as np\n",
    "import mxnet as mx\n",
    "from mxnet import nd, autograd, gluon\n",
    "from mxnet.gluon import nn, Trainer\n",
    "from mxnet.gluon.data import DataLoader, ArrayDataset\n",
    "\n",
    "mx.random.seed(12345)  # Added for reproducibility\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial we will use fake dataset, which contains 10 features drawn from a normal distribution with mean equals to 0 and standard deviation equals to 1, and a class label, which can be either 0 or 1. The size of the dataset is an arbitrary value. The function below helps us to generate a dataset. Class label `y` is generated via a non-random logic, so the network would have a pattern to look for. Boundary of 3 is selected to make sure that number of positive examples smaller than negative, but not too small"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "def get_random_data(size, ctx):\n",
    "    x = nd.normal(0, 1, shape=(size, 10), ctx=ctx)\n",
    "    y = x.sum(axis=1) > 3\n",
    "    return x, y\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, let's define a set of hyperparameters, that we are going to use later. Since our model is simple and dataset is small, we are going to use CPU for calculations. Feel free to change it to GPU for a more advanced scenario."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "ctx = mx.cpu()\n",
    "train_data_size = 1000\n",
    "val_data_size = 100\n",
    "batch_size = 10\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with data\n",
    "\n",
    "To work with data, Apache MXNet provides [Dataset](https://mxnet.apache.org/api/python/gluon/data.html#mxnet.gluon.data.Dataset) and [DataLoader](https://mxnet.apache.org/api/python/gluon/data.html#mxnet.gluon.data.DataLoader) classes. The former is used to provide an indexed access to the data, the latter is used to shuffle and batchify the data. To learn more about working with data in Gluon, please refer to [Gluon Datasets and Dataloaders](https://mxnet.apache.org/tutorials/gluon/datasets.html) tutorial.\n",
    "\n",
    "Below we define training and validation datasets, which we are going to use in the tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "train_x, train_ground_truth_class = get_random_data(train_data_size, ctx)\n",
    "train_dataset = ArrayDataset(train_x, train_ground_truth_class)\n",
    "train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n",
    "\n",
    "val_x, val_ground_truth_class = get_random_data(val_data_size, ctx)\n",
    "val_dataset = ArrayDataset(val_x, val_ground_truth_class)\n",
    "val_dataloader = DataLoader(val_dataset, batch_size=batch_size, shuffle=True)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining and training the model\n",
    "\n",
    "The only requirement for the logistic regression is that the last layer of the network must be a single neuron. Apache MXNet allows us to do so by using [Dense](https://mxnet.apache.org/api/python/gluon/nn.html#mxnet.gluon.nn.Dense) layer and specifying the number of units to 1. The rest of the network can be arbitrarily complex.\n",
    "\n",
    "Below, we define a model which has an input layer of 10 neurons, a couple of inner layers of 10 neurons each, and output layer of 1 neuron. We stack the layers using [HybridSequential](https://mxnet.apache.org/api/python/gluon/gluon.html#mxnet.gluon.nn.HybridSequential) block and initialize parameters of the network using [Xavier](https://mxnet.apache.org/api/python/optimization/optimization.html#mxnet.initializer.Xavier) initialization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "net = nn.HybridSequential()\n",
    "\n",
    "with net.name_scope():\n",
    "    net.add(nn.Dense(units=10, activation='relu'))  # input layer\n",
    "    net.add(nn.Dense(units=10, activation='relu'))   # inner layer 1\n",
    "    net.add(nn.Dense(units=10, activation='relu'))   # inner layer 2\n",
    "    net.add(nn.Dense(units=1))   # output layer: notice, it must have only 1 neuron\n",
    "\n",
    "net.initialize(mx.init.Xavier())\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After defining the model, we need to define a few more things: our loss, our trainer and our metric.\n",
    "\n",
    "Loss function is used to calculate how the output of the network differs from the ground truth. Because classes  of the logistic regression are either 0 or 1, we are using [SigmoidBinaryCrossEntropyLoss](https://mxnet.apache.org/api/python/gluon/loss.html#mxnet.gluon.loss.SigmoidBinaryCrossEntropyLoss). Notice that we do not specify `from_sigmoid` attribute in the code, which means that the output of the neuron doesn't need to go through sigmoid, but at inference we'd have to pass it through sigmoid. You can learn more about cross entropy on [wikipedia](https://en.wikipedia.org/wiki/Cross_entropy).\n",
    "\n",
    "Trainer object allows to specify the method of training to be used. For our tutorial we use [Stochastic Gradient Descent (SGD)](https://mxnet.apache.org/api/python/optimization/optimization.html#mxnet.optimizer.SGD). For more information on SGD refer to [the following tutorial](https://gluon.mxnet.io/chapter06_optimization/gd-sgd-scratch.html). We also need to parametrize it with learning rate value, which defines the weight updates, and weight decay, which is used for regularization.\n",
    "\n",
    "Metric helps us to estimate how good our model is in terms of a problem we are trying to solve. Where loss function has more importance for the training process, a metric is usually the thing we are trying to improve and reach maximum value. We also can use more than one metric, to measure various aspects of our model. In our example, we are using [Accuracy](https://mxnet.apache.org/api/python/metric/metric.html?highlight=metric.acc#mxnet.metric.Accuracy) and [F1 score](http://mxnet.apache.org/api/python/metric/metric.html?highlight=metric.f1#mxnet.metric.F1) as measurements of success of our model.\n",
    "\n",
    "Below we define these objects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "loss = gluon.loss.SigmoidBinaryCrossEntropyLoss()\n",
    "trainer = Trainer(params=net.collect_params(), optimizer='sgd',\n",
    "                  optimizer_params={'learning_rate': 0.1})\n",
    "accuracy = mx.metric.Accuracy()\n",
    "f1 = mx.metric.F1()\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to define the training function in which we iterate over all batches of training data, execute the forward pass on each batch and calculate training loss. On line 19, we sum losses of every batch per epoch into a single variable, because we calculate loss per single batch, but want to display it per epoch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "def train_model():\n",
    "    cumulative_train_loss = 0\n",
    "\n",
    "    for i, (data, label) in enumerate(train_dataloader):\n",
    "        with autograd.record():\n",
    "            # Do forward pass on a batch of training data\n",
    "            output = net(data)\n",
    "\n",
    "            # Calculate loss for the training data batch\n",
    "            loss_result = loss(output, label)\n",
    "\n",
    "        # Calculate gradients\n",
    "        loss_result.backward()\n",
    "\n",
    "        # Update parameters of the network\n",
    "        trainer.step(batch_size)\n",
    "\n",
    "        # sum losses of every batch\n",
    "        cumulative_train_loss += nd.sum(loss_result).asscalar()\n",
    "\n",
    "    return cumulative_train_loss\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validating the model\n",
    "\n",
    "Our validation function is very similar to the training one. The main difference is that we want to calculate accuracy of the model. We use [Accuracy metric](https://mxnet.apache.org/api/python/metric/metric.html?highlight=metric.acc#mxnet.metric.Accuracy) to do so. \n",
    "\n",
    "`Accuracy` metric requires 2 arguments: 1) a vector of ground-truth classes and 2) A vector or matrix of predictions. When predictions are of the same shape as the vector of ground-truth classes, `Accuracy` class assumes that prediction vector contains predicted classes. So, it converts the vector to `Int32` and compare each item of ground-truth classes to prediction vector.\n",
    "\n",
    "Because of the behaviour above, you will get an unexpected result if you just apply [Sigmoid](https://mxnet.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.sigmoid) function to the network result and pass it to `Accuracy` metric. As mentioned before, we need to apply `Sigmoid` function to the output of the neuron to get a probability of belonging to the class 1. But `Sigmoid` function produces output in range [0; 1], and all numbers in that range are going to be casted to 0, even if it is as high as 0.99. To avoid this we write a custom bit of code on line 12, that:\n",
    "\n",
    "1. Calculates sigmoid using `Sigmoid` function\n",
    "\n",
    "2. Subtracts a threshold from the original sigmoid output. Usually, the threshold is equal to 0.5, but it can be higher, if you want to increase certainty of an item to belong to class 1.\n",
    "\n",
    "3. Uses [mx.nd.ceil](https://mxnet.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.ceil) function, which converts all negative values to 0 and all positive values to 1\n",
    "\n",
    "After these transformations we can pass the result to `Accuracy.update()` method and expect it to behave in a proper way.\n",
    "\n",
    "For `F1` metric to work, instead of one number per class, we must pass probabilities of belonging to both classes. Because of that, on lines 21-22 we:\n",
    "\n",
    "1. Reshape predictions to a single vector\n",
    "\n",
    "2. We stack together two vectors: probabilities of belonging to class 0 (1 - `prediction`) and probabilities of belonging to class 1.\n",
    "\n",
    "Then we pass this stacked matrix to `F1` score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "def validate_model(threshold):\n",
    "    cumulative_val_loss = 0\n",
    "\n",
    "    for i, (val_data, val_ground_truth_class) in enumerate(val_dataloader):\n",
    "        # Do forward pass on a batch of validation data\n",
    "        output = net(val_data)\n",
    "\n",
    "        # Similar to cumulative training loss, calculate cumulative validation loss\n",
    "        cumulative_val_loss += nd.sum(loss(output, val_ground_truth_class)).asscalar()\n",
    "\n",
    "        # getting prediction as a sigmoid\n",
    "        prediction = net(val_data).sigmoid()\n",
    "\n",
    "        # Converting neuron outputs to classes\n",
    "        predicted_classes = mx.nd.ceil(prediction - threshold)\n",
    "\n",
    "        # Update validation accuracy\n",
    "        accuracy.update(val_ground_truth_class, predicted_classes.reshape(-1))\n",
    "\n",
    "        # calculate probabilities of belonging to different classes. F1 metric works only with this notation\n",
    "        prediction = prediction.reshape(-1)\n",
    "        probabilities = mx.nd.stack(1 - prediction, prediction, axis=1)\n",
    "\n",
    "        f1.update(val_ground_truth_class, probabilities)\n",
    "\n",
    "    return cumulative_val_loss\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting it all together\n",
    "\n",
    "By using the defined above functions, we can finally write our main training loop."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "epochs = 10\n",
    "threshold = 0.5\n",
    "\n",
    "for e in range(epochs):\n",
    "    avg_train_loss = train_model() / train_data_size\n",
    "    avg_val_loss = validate_model(threshold) / val_data_size\n",
    "\n",
    "    print(\"Epoch: %s, Training loss: %.2f, Validation loss: %.2f, Validation accuracy: %.2f, F1 score: %.2f\" %\n",
    "          (e, avg_train_loss, avg_val_loss, accuracy.get()[1], f1.get()[1]))\n",
    "\n",
    "    # we reset accuracy, so the new epoch's accuracy would be calculated from the blank state\n",
    "    accuracy.reset()\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Output:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "Epoch: 0, Training loss: 0.43, Validation loss: 0.36, Validation accuracy: 0.85, F1 score: 0.00 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 1, Training loss: 0.22, Validation loss: 0.14, Validation accuracy: 0.96, F1 score: 0.35 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 2, Training loss: 0.09, Validation loss: 0.11, Validation accuracy: 0.97, F1 score: 0.48 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 3, Training loss: 0.07, Validation loss: 0.09, Validation accuracy: 0.96, F1 score: 0.53 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 4, Training loss: 0.06, Validation loss: 0.09, Validation accuracy: 0.97, F1 score: 0.58 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 5, Training loss: 0.04, Validation loss: 0.12, Validation accuracy: 0.97, F1 score: 0.59 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 6, Training loss: 0.05, Validation loss: 0.09, Validation accuracy: 0.99, F1 score: 0.62 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 7, Training loss: 0.05, Validation loss: 0.10, Validation accuracy: 0.97, F1 score: 0.62 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 8, Training loss: 0.05, Validation loss: 0.12, Validation accuracy: 0.95, F1 score: 0.63 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 9, Training loss: 0.04, Validation loss: 0.09, Validation accuracy: 0.98, F1 score: 0.65 <!--notebook-skip-line-->\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our case we hit the accuracy of 0.98 and F1 score of 0.65.\n",
    "\n",
    "## Tip 1: Use only one neuron in the output layer\n",
    "\n",
    "Despite that there are 2 classes, there should be only one output neuron, because `SigmoidBinaryCrossEntropyLoss` accepts only one feature as an input.\n",
    "\n",
    "## Tip 2: Encode classes as 0 and 1\n",
    "\n",
    "For `SigmoidBinaryCrossEntropyLoss` to work it is required that classes were encoded as 0 and 1. In some datasets the class encoding might be different, like -1 and 1 or 1 and 2. If this is how your dataset looks like, then you need to re-encode the data before using `SigmoidBinaryCrossEntropyLoss`.\n",
    "\n",
    "## Tip 3: Use SigmoidBinaryCrossEntropyLoss instead of LogisticRegressionOutput\n",
    "\n",
    "NDArray API has two options to calculate logistic regression loss: [SigmoidBinaryCrossEntropyLoss](https://mxnet.apache.org/api/python/gluon/loss.html#mxnet.gluon.loss.SigmoidBinaryCrossEntropyLoss) and [LogisticRegressionOutput](https://mxnet.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.LogisticRegressionOutput). `LogisticRegressionOutput` is designed to be an output layer when using the Module API, and is not supposed to be used when using Gluon API.\n",
    "\n",
    "## Conclusion\n",
    "\n",
    "In this tutorial I explained some potential pitfalls to be aware of. When doing logistic regression using Gluon API remember to:\n",
    "1. Use only one neuron in the output layer\n",
    "1. Encode class labels as 0 or 1\n",
    "1. Use `SigmoidBinaryCrossEntropyLoss`\n",
    "1. Convert probabilities to classes before calculating Accuracy"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}