{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mxnet as mx\n",
    "import numpy as np\n",
    "import os\n",
    "import logging\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.cm as cm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a Variational Autoencoder in MXNet\n",
    "\n",
    "#### Xiaoyu Lu,  July 5th, 2017\n",
    "\n",
    "This tutorial guides you through the process of building a variational encoder in MXNet. In this notebook we'll focus on an example using the MNIST handwritten digit recognition dataset. Refer to [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114/) for more details on the model description.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "To complete this tutorial, we need following python packages:\n",
    "\n",
    "- numpy, matplotlib "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Loading the Data\n",
    "\n",
    "We first load the MNIST dataset, which contains 60000 training and 10000 test examples. The following code imports required modules and loads the data. These images are stored in a 4-D matrix with shape (`batch_size, num_channels, width, height`). For the MNIST dataset, there is only one color channel, and both width and height are 28, so we reshape each image as a 28x28 array. See below for a visualization:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60000 784\n"
     ]
    }
   ],
   "source": [
    "mnist = mx.test_utils.get_mnist()\n",
    "image = np.reshape(mnist['train_data'],(60000,28*28))\n",
    "label = image\n",
    "image_test = np.reshape(mnist['test_data'],(10000,28*28))\n",
    "label_test = image_test\n",
    "[N,features] = np.shape(image)          #number of examples and features\n",
    "print(N,features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAACWCAYAAAA7UIUvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFI5JREFUeJzt3X+wVfO/x/H3u9+JfikVnb5FaYTEnLmqe5mLDDKJwfGVaYzfVJRBP+5FgzHjd6YoMkyFcbvUJDToxnW7GDql6YpSFwnpVKRfqDM+94+2O+f9Waf94+xfa+3P8zHTnF6rtfd6n/a73afde62lzjkBAAAAQtSs3AUAAAAA5cJiGAAAAMFiMQwAAIBgsRgGAABAsFgMAwAAIFgshgEAABAsFsMAAAAIFothAAAABCuvxbCqnq+q61V1o6pOLlRRAAAAQCloU+9Ap6rNReQrETlXRL4XkRUicqVz7otDPaZLly6ud+/eTToe4uXbb7+V7du3azGemz6pLCtXrtzunOtajOemVyoH7ynIFu8pyEYu7ykt8jjOP4jIRufc1yIiqvpvIjJSRA65GO7du7fU1tbmcUjERXV1ddGemz6pLKq6qVjPTa9UDt5TkC3eU5CNXN5T8hmTOEZENjfI36e2Gap6o6rWqmrttm3b8jgcKhl9gmzRK8gGfYJs0Sso+gl0zrnZzrlq51x1165F+V8NVAD6BNmiV5AN+gTZoleQz2L4BxGpapB7prYBAAAAiZDPYniFiPRT1T6q2kpE/i4iiwtTFgAAAFB8TT6BzjlXr6rjROQdEWkuIi8459YWrDIAAACgyPK5moQ455aIyJIC1QIAAACUFHegAwAAQLBYDAMAACBYLIYBAAAQLBbDAAAACBaLYQAAAASLxTAAAACCxWIYAAAAwWIxDAAAgGCxGAYAAECw8roDHYDiue2220yeMWNGZJ8OHTqYvG7dOpO7d+9e+MIAAKggfDIMAACAYLEYBgAAQLBYDAMAACBYzAwXwdatW00eNGiQyUOHDjV5wYIFRa8J8bdq1SqTn376aZObNYv+23X37t0m19XVmczMMJBcf/zxh8nz5883+Zprrkn7+OXLl5s8ePBgk/ft2xd5TJs2bUw+cOCAyf77UOvWrdPWACQBnwwDAAAgWCyGAQAAECwWwwAAAAgWM8NFMHv2bJP9GeJFixaVshzEVH19vcnz5s3L+TmOP/54k3v16pVXTQDKY+/evZFtU6dONfnJJ580WVXTPuftt99u8mmnnWbyc889F3nMZZddZvL7779vcu/evU32z4Hxz1O48cYbTe7UqdOhCwbKhE+GAQAAECwWwwAAAAgWi2EAAAAEi5nhEnDOlbsExNAnn3xi8owZM3J+jsmTJ5vcsWPHvGoCUB6XXnppZNvSpUtNzjQj7Fu5cqXJtbW1GZ/vtddeS/ucO3bsyOk53333XZOXLFkSeU6uVZzehg0bTO7fv39knxEjRpg8a9Ysk48++ujCF+bZuXOnyX6v+OdTZTJu3LjItqqqqtwLywKfDAMAACBYLIYBAAAQLBbDAAAACBYzw0Wwdu1ak/0ZqksuuaSU5SAm/vzzT5OnTJmS0+NXrFgR2XbyySfnVRMyu+WWW0z+4osvTL7qqqtMHjhwoMmDBw8uTmFItLlz55r8wQcflKmS4vr6669N/uOPPyL7MDOc3vz5801ubNb7jTfeMNk/J8WfGT7qqKNMHjVqVNoaPvvss8g2v2d//PFHk7dt22ayf/5Uphn48847L7KNmWEAAACgwFgMAwAAIFgshgEAABAsZoaLwJ/F8TV2P3hUPn/m6sMPP0y7/xFHHGFyY9eJbNmyZf6FwfCvs/rss8+m3X/58uUm+3NwLVpE32b9uWL/epp9+vRJ+xz79u0zediwYWlrRPn512CdOnWqyQcOHChlOQXjnwMzcuRIky+//HKT27RpU/Sakqa+vt7k1atXm/zEE0/k/Jz+vK6fff41rXOd7y2GIUOGlOxYfDIMAACAYLEYBgAAQLAyLoZV9QVVrVPVzxts66yqS1V1Q+prp+KWCQAAABReNjPDc0TkKRGZ12DbZBFZ5px7SFUnp/KkwpeXDPv37zd548aNZaoEcfbNN9/ktP8pp5xicvfu3QtZDg7Bn+dt27atyb/99lvax/uzdo3Ngq5cudLka665xuRmzeznFP68nn+Mww47zOQuXbpEjvnRRx+ZTD+V1po1a0z2/97wX1OR6LXJ/b7IpF27dia3b9/e5MauLeu/7/j8GeBWrVrlVBOidu3aZfLpp5+edv+amprItuHDh5vsn6OyaNEikzdt2pRLiSXhfw+NnW9RLBn/ZDnn/ktEfvY2jxSRv64YPldELi5wXQAAAEDRNXVmuJtzbkvq5z+JSLcC1QMAAACUTN4n0LmD/7cT/f+dFFW9UVVrVbU206U9EC76BNmiV5AN+gTZolfQ1MXwVlXtISKS+lp3qB2dc7Odc9XOuequXbs28XCodPQJskWvIBv0CbJFr6Cp08mLReRqEXko9fX1glWUQNu3bzf57bffNrmxEyMQnrfeeiun/R955JEiVYJ0jj/+eJP9P98zZ8402T/5xffMM89EtmX69Mk/cSqTPXv2pM0i0e9r4cKFJnPjjtKqq7OfITV2UwP/hDn/RMkrrrjC5NGjR5t87LHHmlxVVZVznSi8vXv3mnz33Xfn9Pjrr78+su2cc84x2e+FBx980OQtW7ZIvt59912T77zzTpMznWzsn5zsv1eW8qZS2Vxa7RUR+VhE+qvq96p6nRxcBJ+rqhtEZFgqAwAAAImS8ZNh59yVh/ilcw6xHQAAAEgE7kAHAACAYJXuisYV7OOPPza5sdkvhOfXX381edmyZTk9vrq6upDloIn8ubY77rgjp8dPmhS9H1GmmeBXXnnF5Ew38vEvqL9hw4bIPv4ccaZ5PhSWPyfaFL169TJ5xowZJvu9inj6/fffTZ4/f77J/nlGN998s8n+fHA2/N7w58kzWbduXWTbmDFjcnqODh06mPzhhx+afMwxx+T0fIXEJ8MAAAAIFothAAAABIvFMAAAAILFzHABnHrqqSb714Js1aqVyW3atCl6TSg//9qRP/zwQ9r9u3WzdzVn9rwy+O8H2bjhhhty2v/ee+81ee7cuZF9xo0bZ/LkyZNNHjFiRE7HRG78vyeOPPJIk3fs2JHxOdavX2/yrFmzTB45cqTJxx13XC4lokTat29v8oIFC0wePny4yY8//njRa/K9+eabJtfU1ET2yfR3lD+n7M8IDxgwoInVFR6fDAMAACBYLIYBAAAQLBbDAAAACBYzwwXw2Wefmbxv3z6T+/TpY3JTZggRf/X19Sb7830+f0b4pZdeMrlZM/6tiuy0a9fO5FGjRkX28WeGJ0yYUNSaYHXv3t3kfv36mZzNzLBv4sSJJk+dOtXkRx991GT/erUoj5YtW5o8ZMgQk9977z2TS3H96Hfeecfkiy66yORszmHxr8E+fvx4k8t5HeFM+NsWAAAAwWIxDAAAgGCxGAYAAECwmBkuAP/aef59xS+//PJSloMymT59usnLli1Lu/+JJ55o8tlnn13wmhCmZ555JuM+/gwrSuu1114zubq6OrKPf23yTOcR+OerjB071uS77rrL5BUrVkSeo2/fvia3aMEyodhat25t8uDBg4t+zO+++87kiy++OO3+Rx11VGTbLbfcYrI/w56keyrwyTAAAACCxWIYAAAAwWIxDAAAgGAxDFQA/uydfz2+QYMGlbIclEk2c5oNnXDCCUWqBKHZuXOnydOmTYvs48/vVVVVFbUmpOf/vTFv3rzIPqNHjzZ569ateR3Tnyk+6aSTIvv417g988wz8zom4sG/jrA/77t//36Tjz76aJM//fTTyHP6+yQZnwwDAAAgWCyGAQAAECwWwwAAAAgWM8MF4M/n+dcZXr16tckjRowoek2IP/oAheLPq2/bti2yj99vnMsQL41dZ9y/hr1/DeBiuOKKK0xesGCByUOHDi16Dcjfrl27TPZnwb/99tu0j9+7d6/J/nkJIswMAwAAABWBxTAAAACCxWIYAAAAwWIxDAAAgGBxAl0B+DfZ8DMqz6ZNmyLbfvzxx7SPad68ucmnn356QWtCOH7++WeTp0+fbnLPnj0jj3n55ZeLWhMKz78xyubNm01euHChyffff7/JO3bsyPmY/smXY8eONfmjjz4yuW3btjkfA8W3Zs0akx977DGTM61TZs2aZfKAAQMKU1hM8ckwAAAAgsViGAAAAMFiMQwAAIBgMTNcAP6Fp3/66SeTL7roolKWgxLwX2MRkd9++y3tY8aMGWNy+/btC1oTKtf+/ftNfvjhh032+/HCCy+MPMfhhx9e+MJQVP55Bj169DC5Y8eOJvs3WiiE1q1bp60J8VBbW2tyrjd1qqmpMfmCCy7Iu6Yk4ZNhAAAABCvjYlhVq1T1fVX9QlXXqur41PbOqrpUVTekvnYqfrkAAABA4WTzyXC9iNzhnBsgIoNFZKyqDhCRySKyzDnXT0SWpTIAAACQGBlnhp1zW0RkS+rnu1X1SxE5RkRGisg/p3abKyL/KSKTilJlzK1atcpk//p9nTrxoXmlefbZZ3N+zFNPPWXyTTfdZPIJJ5yQV02oXFu2bDH5kUceMdm/1usDDzxQ9JqQ3u7du02eOHGiye3atTO5W7dukedwzpn86quvmuz/3ZPJn3/+aXKzZpk/DzvssMNyOgaK7/XXX49su/baa03OND/euXNnk6dNm2Zyhw4dmlhdMuU0M6yqvUXkVBH5RES6pRbKIiI/iUj0TzIAAAAQY1kvhlX1cBFZICITnHPmnxzu4D9f3SEed6Oq1qpqrX9nG+Av9AmyRa8gG/QJskWvIKvFsKq2lIML4Zedc3/d/3GrqvZI/XoPEalr7LHOudnOuWrnXHXXrl0LUTMqEH2CbNEryAZ9gmzRK8g4M6wHB2CfF5EvnXNPNPilxSJytYg8lPoaHWIJRN++fU32r/nZq1evUpaDmPLn//w+YWYYh7Jz506T/fMSHnvsMZMHDRpU9JqQ3ubNm01+7rnncn4O/z3Df939nIk/I9zY4/33IX8+vVWrVjkdE/nbtGmTyffcc09kn19++cXkTL0xbNgwk0M/tymbm278o4iMFpH/UdXVqW3/IgcXwf+uqteJyCYRqTnE4wEAAIBYyuZqEv8tIof6J8Y5hS0HAAAAKB3uQAcAAIBgZTMmgQz8a4D6szr79u0zmes2Jl+PHj1yfow/a3fWWWcVqhxUmPr6epOnTp1qcsuWLU0eOnRo0WtCbvzruF533XUmP//886Usp1Ht27ePbFuyZInJVVVVpSoHKXV19noEp512msn+fLBIdL7cd8YZZ5j84osvmtyiRdjLQT4ZBgAAQLBYDAMAACBYLIYBAAAQrLCHRApkz549JvvzpMwIV55bb701sm3mzJkm+/d2nz59elFrQuVYu3atyYsXLzb5wgsvNJnrCsdP9+7dTX7wwQdNfvPNN03eunVr0WsaP368yaNGjYrsw4xw6flriP79+5u8a5e56W9W15e+7777TJ4wYYLJoc8I+/hkGAAAAMFiMQwAAIBgsRgGAABAsBgaKQD/PuH+rBgqT2OvcWPXfgSa4oEHHjDZn+/z508Rfx07djT5q6++MnnMmDGRx7z00ks5HaO6utrkadOmmTxkyJCcng/FsXfvXpMvu+wyk/0Z4Wz4M8KTJk0y2b/OPSw+GQYAAECwWAwDAAAgWCyGAQAAECwWwwAAAAgWJ9AVwP3331/uEgBUsDPOOMPkgQMHlqkSNJV/EqSf586dG3lMY9uQfO3atTO5rq4up8fX1NREtk2ZMsVkbqqRGz4ZBgAAQLBYDAMAACBYLIYBAAAQLIZKAAAAymTx4sUmjxs3zuS2bduaPGfOnMhzMCOcHz4ZBgAAQLBYDAMAACBYLIYBAAAQLIZMACBm/JnBhQsXlqkSAMXWs2dPkxctWlSmSsLFJ8MAAAAIFothAAAABIvFMAAAAIKlzrnSHUx1m4hsEpEuIrK9ZAduGmpM72/Oua7FeOKE9YlIMuqkV8qPGtMrRZ+I8DoUSqX3Cq9B4ZSrzqz7pKSL4f8/qGqtc6665AfOATWWX1K+vyTUmYQa85GE748a4yEJ3yM1ll8Svr8k1CiSjDoZkwAAAECwWAwDAAAgWOVaDM8u03FzQY3ll5TvLwl1JqHGfCTh+6PGeEjC90iN5ZeE7y8JNYokoM6yzAwDAAAAccCYBAAAAIJV0sWwqp6vqutVdaOqTi7lsdNR1RdUtU5VP2+wrbOqLlXVDamvncpcY5Wqvq+qX6jqWlUdH8c6CyWOvUKfxE8c+0SEXokjeqXJ9QXVJyLx7JW490mqnsT2SskWw6raXESeFpELRGSAiFypqgNKdfwM5ojI+d62ySKyzDnXT0SWpXI51YvIHc65ASIyWETGpn7/4lZn3mLcK3OEPomNGPeJCL0SK/RKXoLpE5FY98ociXefiCS5V5xzJfkhIkNE5J0GeYqITCnV8bOor7eIfN4grxeRHqmf9xCR9eWu0av3dRE5N+51Vlqv0Cfx+RHnPqFX4vWDXqFPKqFXktQnSeuVUo5JHCMimxvk71Pb4qqbc25L6uc/iUi3chbTkKr2FpFTReQTiXGdeUhSr8T2958+iZ3Yvgb0SuzE8jUIoE9EktUrsX0NktYrnECXBXfwnzOxuOyGqh4uIgtEZIJzblfDX4tTnSGK0+8/fRJvcXoN6JV4i8trQJ/EW5xegyT2SikXwz+ISFWD3DO1La62qmoPEZHU17oy1yOq2lIONtjLzrmFqc2xq7MAktQrsfv9p09iK3avAb0SW7F6DQLqE5Fk9UrsXoOk9kopF8MrRKSfqvZR1VYi8ncRWVzC4+dqsYhcnfr51XJw9qVsVFVF5HkR+dI590SDX4pVnQWSpF6J1e8/fRLbPhGJ2WtAr9Ar2QisT0SS1Suxeg0S3SslHqYeLiJficj/isi/lntgukFdr4jIFhE5IAfng64TkSPl4FmPG0TkP0Skc5lr/Cc5+F8La0RkderH8LjVWcm9Qp/E70cc+4ReiecPeoU+SXKvxL1Pkt4r3IEOAAAAweIEOgAAAASLxTAAAACCxWIYAAAAwWIxDAAAgGCxGAYAAECwWAwDAAAgWCyGAQAAECwWwwAAAAjW/wEgPmufEARJLAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x216 with 5 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "nsamples = 5\n",
    "idx = np.random.choice(len(mnist['train_data']), nsamples)\n",
    "_, axarr = plt.subplots(1, nsamples, sharex='col', sharey='row',figsize=(12,3))\n",
    "\n",
    "for i,j in enumerate(idx):\n",
    "    axarr[i].imshow(np.reshape(image[j,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can optionally save the parameters in the directory variable 'model_prefix'. We first create data iterators for MXNet, with each batch of data containing 100 images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_prefix = None\n",
    "\n",
    "batch_size = 100\n",
    "latent_dim = 5\n",
    "nd_iter = mx.io.NDArrayIter(data={'data':image},label={'loss_label':label},\n",
    "                            batch_size = batch_size)\n",
    "nd_iter_test = mx.io.NDArrayIter(data={'data':image_test},label={'loss_label':label_test},\n",
    "                            batch_size = batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.  Building the Network Architecture\n",
    "\n",
    "### 2.1 Gaussian MLP as encoder\n",
    "Next we constuct the neural network, as in the [paper](https://arxiv.org/abs/1312.6114/), we use *Multilayer Perceptron (MLP)* for both the encoder and decoder. For encoder, a Gaussian MLP is used as follows:\n",
    "\n",
    "\\begin{align}\n",
    "\\log q_{\\phi}(z|x) &= \\log \\mathcal{N}(z:\\mu,\\sigma^2I) \\\\\n",
    "\\textit{ where } \\mu &= W_2h+b_2, \\log \\sigma^2 = W_3h+b_3\\\\\n",
    "h &= \\tanh(W_1x+b_1)\n",
    "\\end{align}\n",
    "\n",
    "where $\\{W_1,W_2,W_3,b_1,b_2,b_3\\}$ are the weights and biases of the MLP.\n",
    "Note below that `encoder_mu`(`mu`) and `encoder_logvar`(`logvar`) are symbols. So, we can use `get_internals()` to get the values of them, after which we can sample the latent variable $z$.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "## define data and loss labels as symbols \n",
    "data = mx.sym.var('data')\n",
    "loss_label = mx.sym.var('loss_label')\n",
    "\n",
    "## define fully connected and activation layers for the encoder, where we used tanh activation function.\n",
    "encoder_h  = mx.sym.FullyConnected(data=data, name=\"encoder_h\",num_hidden=400)\n",
    "act_h = mx.sym.Activation(data=encoder_h, act_type=\"tanh\",name=\"activation_h\")\n",
    "\n",
    "## define mu and log variance which are the fully connected layers of the previous activation layer\n",
    "mu  = mx.sym.FullyConnected(data=act_h, name=\"mu\",num_hidden = latent_dim)\n",
    "logvar  = mx.sym.FullyConnected(data=act_h, name=\"logvar\",num_hidden = latent_dim)\n",
    "\n",
    "## sample the latent variables z according to Normal(mu,var)\n",
    "z = mu + mx.symbol.broadcast_mul(mx.symbol.exp(0.5 * logvar), \n",
    "                                 mx.symbol.random_normal(loc=0, scale=1, shape=(batch_size, latent_dim)),\n",
    "                                 name=\"z\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Bernoulli MLP as decoder\n",
    "\n",
    "In this case let $p_\\theta(x|z)$ be a multivariate Bernoulli whose probabilities are computed from $z$ with a feed forward neural network with a single hidden layer:\n",
    "\n",
    "\\begin{align}\n",
    "\\log p(x|z) &= \\sum_{i=1}^D x_i\\log y_i + (1-x_i)\\log (1-y_i) \\\\\n",
    "\\textit{ where }  y &= f_\\sigma(W_5\\tanh (W_4z+b_4)+b_5)\n",
    "\\end{align}\n",
    "\n",
    "where $f_\\sigma(\\dot)$ is the elementwise sigmoid activation function, $\\{W_4,W_5,b_4,b_5\\}$ are the weights and biases of the decoder MLP. A Bernouilli likelihood is suitable for this type of data but you can easily extend it to other likelihood types by parsing into the argument `likelihood` in the `VAE` class, see section 4 for details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define fully connected and tanh activation layers for the decoder\n",
    "decoder_z = mx.sym.FullyConnected(data=z, name=\"decoder_z\",num_hidden=400)\n",
    "act_z = mx.sym.Activation(data=decoder_z, act_type=\"tanh\",name=\"activation_z\")\n",
    "\n",
    "# define the output layer with sigmoid activation function, where the dimension is equal to the input dimension\n",
    "decoder_x = mx.sym.FullyConnected(data=act_z, name=\"decoder_x\",num_hidden=features)\n",
    "y = mx.sym.Activation(data=decoder_x, act_type=\"sigmoid\",name='activation_x')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Joint Loss Function for the Encoder and the Decoder\n",
    "\n",
    "The variational lower bound also called evidence lower bound (ELBO) can be estimated as:\n",
    "\n",
    "\\begin{align}\n",
    "\\mathcal{L}(\\theta,\\phi;x_{(i)}) \\approx \\frac{1}{2}\\left(1+\\log ((\\sigma_j^{(i)})^2)-(\\mu_j^{(i)})^2-(\\sigma_j^{(i)})^2\\right) + \\log p_\\theta(x^{(i)}|z^{(i)})\n",
    "\\end{align}\n",
    "\n",
    "where the first term is the KL divergence of the approximate posterior from the prior, and the second term is an expected negative reconstruction error. We would like to maximize this lower bound, so we can define the loss to be $-\\mathcal{L}$(minus ELBO) for MXNet to minimize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the objective loss function that needs to be minimized\n",
    "KL = 0.5*mx.symbol.sum(1+logvar-pow( mu,2)-mx.symbol.exp(logvar),axis=1)\n",
    "loss = -mx.symbol.sum(mx.symbol.broadcast_mul(loss_label,mx.symbol.log(y)) \n",
    "                      + mx.symbol.broadcast_mul(1-loss_label,mx.symbol.log(1-y)),axis=1)-KL\n",
    "output = mx.symbol.MakeLoss(sum(loss),name='loss')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Training the model\n",
    "\n",
    "Now, we can define the model and train it. First we will initilize the weights and the biases to be Gaussian(0,0.01), and then use stochastic gradient descent for optimization. To warm start the training, one may also initilize with pre-trainined parameters `arg_params` using `init=mx.initializer.Load(arg_params)`. \n",
    "\n",
    "To save intermediate results, we can optionally use `epoch_end_callback = mx.callback.do_checkpoint(model_prefix, 1)` which saves the parameters to the path given by model_prefix, and with period every $1$ epoch. To assess the performance, we output $-\\mathcal{L}$(minus ELBO) after each epoch, with the command `eval_metric = 'Loss'` which is defined above. We will also plot the training loss for mini batches by accessing the log and saving it to a list, and then parsing it to the argument `batch_end_callback`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set up the log\n",
    "nd_iter.reset()\n",
    "logging.getLogger().setLevel(logging.DEBUG)  \n",
    "\n",
    "# define function to trave back training loss\n",
    "def log_to_list(period, lst):\n",
    "    def _callback(param):\n",
    "        \"\"\"The checkpoint function.\"\"\"\n",
    "        if param.nbatch % period == 0:\n",
    "            name, value = param.eval_metric.get()\n",
    "            lst.append(value)\n",
    "    return _callback\n",
    "\n",
    "# define the model\n",
    "model = mx.mod.Module(\n",
    "    symbol = output ,\n",
    "    data_names=['data'],\n",
    "    label_names = ['loss_label'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Epoch[0] Train-loss=373.547317\n",
      "INFO:root:Epoch[0] Time cost=5.020\n",
      "INFO:root:Epoch[1] Train-loss=212.232684\n",
      "INFO:root:Epoch[1] Time cost=4.651\n",
      "INFO:root:Epoch[2] Train-loss=207.448528\n",
      "INFO:root:Epoch[2] Time cost=4.665\n",
      "INFO:root:Epoch[3] Train-loss=205.369479\n",
      "INFO:root:Epoch[3] Time cost=4.758\n",
      "INFO:root:Epoch[4] Train-loss=203.651983\n",
      "INFO:root:Epoch[4] Time cost=4.672\n",
      "INFO:root:Epoch[5] Train-loss=202.061007\n",
      "INFO:root:Epoch[5] Time cost=5.087\n",
      "INFO:root:Epoch[6] Train-loss=199.348143\n",
      "INFO:root:Epoch[6] Time cost=5.056\n",
      "INFO:root:Epoch[7] Train-loss=196.266242\n",
      "INFO:root:Epoch[7] Time cost=4.813\n",
      "INFO:root:Epoch[8] Train-loss=194.694945\n",
      "INFO:root:Epoch[8] Time cost=4.776\n",
      "INFO:root:Epoch[9] Train-loss=193.699284\n",
      "INFO:root:Epoch[9] Time cost=4.756\n",
      "INFO:root:Epoch[10] Train-loss=193.036517\n",
      "INFO:root:Epoch[10] Time cost=4.757\n",
      "INFO:root:Epoch[11] Train-loss=192.555736\n",
      "INFO:root:Epoch[11] Time cost=4.678\n",
      "INFO:root:Epoch[12] Train-loss=192.020813\n",
      "INFO:root:Epoch[12] Time cost=4.630\n",
      "INFO:root:Epoch[13] Train-loss=191.648876\n",
      "INFO:root:Epoch[13] Time cost=5.158\n",
      "INFO:root:Epoch[14] Train-loss=191.057798\n",
      "INFO:root:Epoch[14] Time cost=4.781\n",
      "INFO:root:Epoch[15] Train-loss=190.315835\n",
      "INFO:root:Epoch[15] Time cost=5.117\n",
      "INFO:root:Epoch[16] Train-loss=189.311271\n",
      "INFO:root:Epoch[16] Time cost=4.707\n",
      "INFO:root:Epoch[17] Train-loss=187.285967\n",
      "INFO:root:Epoch[17] Time cost=4.745\n",
      "INFO:root:Epoch[18] Train-loss=185.271324\n",
      "INFO:root:Epoch[18] Time cost=4.692\n",
      "INFO:root:Epoch[19] Train-loss=183.510888\n",
      "INFO:root:Epoch[19] Time cost=4.762\n",
      "INFO:root:Epoch[20] Train-loss=181.756008\n",
      "INFO:root:Epoch[20] Time cost=4.838\n",
      "INFO:root:Epoch[21] Train-loss=180.546818\n",
      "INFO:root:Epoch[21] Time cost=4.764\n",
      "INFO:root:Epoch[22] Train-loss=179.479776\n",
      "INFO:root:Epoch[22] Time cost=4.791\n",
      "INFO:root:Epoch[23] Train-loss=178.352077\n",
      "INFO:root:Epoch[23] Time cost=4.981\n",
      "INFO:root:Epoch[24] Train-loss=177.385084\n",
      "INFO:root:Epoch[24] Time cost=5.292\n",
      "INFO:root:Epoch[25] Train-loss=175.920123\n",
      "INFO:root:Epoch[25] Time cost=5.097\n",
      "INFO:root:Epoch[26] Train-loss=174.377171\n",
      "INFO:root:Epoch[26] Time cost=4.907\n",
      "INFO:root:Epoch[27] Train-loss=172.590589\n",
      "INFO:root:Epoch[27] Time cost=4.484\n",
      "INFO:root:Epoch[28] Train-loss=170.933683\n",
      "INFO:root:Epoch[28] Time cost=4.348\n",
      "INFO:root:Epoch[29] Train-loss=169.866807\n",
      "INFO:root:Epoch[29] Time cost=4.647\n",
      "INFO:root:Epoch[30] Train-loss=169.182084\n",
      "INFO:root:Epoch[30] Time cost=5.034\n",
      "INFO:root:Epoch[31] Train-loss=168.121719\n",
      "INFO:root:Epoch[31] Time cost=5.615\n",
      "INFO:root:Epoch[32] Train-loss=167.389992\n",
      "INFO:root:Epoch[32] Time cost=4.733\n",
      "INFO:root:Epoch[33] Train-loss=166.189067\n",
      "INFO:root:Epoch[33] Time cost=5.041\n",
      "INFO:root:Epoch[34] Train-loss=163.783392\n",
      "INFO:root:Epoch[34] Time cost=5.168\n",
      "INFO:root:Epoch[35] Train-loss=162.167959\n",
      "INFO:root:Epoch[35] Time cost=5.019\n",
      "INFO:root:Epoch[36] Train-loss=161.192039\n",
      "INFO:root:Epoch[36] Time cost=5.064\n",
      "INFO:root:Epoch[37] Train-loss=160.307114\n",
      "INFO:root:Epoch[37] Time cost=5.180\n",
      "INFO:root:Epoch[38] Train-loss=159.591957\n",
      "INFO:root:Epoch[38] Time cost=5.440\n",
      "INFO:root:Epoch[39] Train-loss=159.109593\n",
      "INFO:root:Epoch[39] Time cost=5.119\n",
      "INFO:root:Epoch[40] Train-loss=158.463844\n",
      "INFO:root:Epoch[40] Time cost=5.299\n",
      "INFO:root:Epoch[41] Train-loss=158.037287\n",
      "INFO:root:Epoch[41] Time cost=4.856\n",
      "INFO:root:Epoch[42] Train-loss=157.598576\n",
      "INFO:root:Epoch[42] Time cost=5.227\n",
      "INFO:root:Epoch[43] Train-loss=157.097344\n",
      "INFO:root:Epoch[43] Time cost=5.237\n",
      "INFO:root:Epoch[44] Train-loss=156.594472\n",
      "INFO:root:Epoch[44] Time cost=4.783\n",
      "INFO:root:Epoch[45] Train-loss=156.177069\n",
      "INFO:root:Epoch[45] Time cost=4.834\n",
      "INFO:root:Epoch[46] Train-loss=155.825302\n",
      "INFO:root:Epoch[46] Time cost=4.902\n",
      "INFO:root:Epoch[47] Train-loss=155.318117\n",
      "INFO:root:Epoch[47] Time cost=4.966\n",
      "INFO:root:Epoch[48] Train-loss=154.890766\n",
      "INFO:root:Epoch[48] Time cost=5.012\n",
      "INFO:root:Epoch[49] Train-loss=154.504158\n",
      "INFO:root:Epoch[49] Time cost=4.844\n",
      "INFO:root:Epoch[50] Train-loss=154.035214\n",
      "INFO:root:Epoch[50] Time cost=4.736\n",
      "INFO:root:Epoch[51] Train-loss=153.692903\n",
      "INFO:root:Epoch[51] Time cost=5.057\n",
      "INFO:root:Epoch[52] Train-loss=153.257554\n",
      "INFO:root:Epoch[52] Time cost=5.044\n",
      "INFO:root:Epoch[53] Train-loss=152.849715\n",
      "INFO:root:Epoch[53] Time cost=4.783\n",
      "INFO:root:Epoch[54] Train-loss=152.483047\n",
      "INFO:root:Epoch[54] Time cost=4.842\n",
      "INFO:root:Epoch[55] Train-loss=152.091617\n",
      "INFO:root:Epoch[55] Time cost=5.044\n",
      "INFO:root:Epoch[56] Train-loss=151.715490\n",
      "INFO:root:Epoch[56] Time cost=5.029\n",
      "INFO:root:Epoch[57] Train-loss=151.362293\n",
      "INFO:root:Epoch[57] Time cost=4.873\n",
      "INFO:root:Epoch[58] Train-loss=151.003241\n",
      "INFO:root:Epoch[58] Time cost=4.729\n",
      "INFO:root:Epoch[59] Train-loss=150.619678\n",
      "INFO:root:Epoch[59] Time cost=5.068\n",
      "INFO:root:Epoch[60] Train-loss=150.296043\n",
      "INFO:root:Epoch[60] Time cost=4.458\n",
      "INFO:root:Epoch[61] Train-loss=149.964152\n",
      "INFO:root:Epoch[61] Time cost=4.828\n",
      "INFO:root:Epoch[62] Train-loss=149.694102\n",
      "INFO:root:Epoch[62] Time cost=5.012\n",
      "INFO:root:Epoch[63] Train-loss=149.290113\n",
      "INFO:root:Epoch[63] Time cost=5.193\n",
      "INFO:root:Epoch[64] Train-loss=148.934186\n",
      "INFO:root:Epoch[64] Time cost=4.999\n",
      "INFO:root:Epoch[65] Train-loss=148.657502\n",
      "INFO:root:Epoch[65] Time cost=4.810\n",
      "INFO:root:Epoch[66] Train-loss=148.331948\n",
      "INFO:root:Epoch[66] Time cost=5.201\n",
      "INFO:root:Epoch[67] Train-loss=148.018539\n",
      "INFO:root:Epoch[67] Time cost=4.833\n",
      "INFO:root:Epoch[68] Train-loss=147.746825\n",
      "INFO:root:Epoch[68] Time cost=5.187\n",
      "INFO:root:Epoch[69] Train-loss=147.406399\n",
      "INFO:root:Epoch[69] Time cost=5.355\n",
      "INFO:root:Epoch[70] Train-loss=147.181831\n",
      "INFO:root:Epoch[70] Time cost=4.989\n",
      "INFO:root:Epoch[71] Train-loss=146.860770\n",
      "INFO:root:Epoch[71] Time cost=4.934\n",
      "INFO:root:Epoch[72] Train-loss=146.604369\n",
      "INFO:root:Epoch[72] Time cost=5.283\n",
      "INFO:root:Epoch[73] Train-loss=146.351628\n",
      "INFO:root:Epoch[73] Time cost=5.062\n",
      "INFO:root:Epoch[74] Train-loss=146.102506\n",
      "INFO:root:Epoch[74] Time cost=4.540\n",
      "INFO:root:Epoch[75] Train-loss=145.828805\n",
      "INFO:root:Epoch[75] Time cost=4.875\n",
      "INFO:root:Epoch[76] Train-loss=145.571626\n",
      "INFO:root:Epoch[76] Time cost=4.856\n",
      "INFO:root:Epoch[77] Train-loss=145.365383\n",
      "INFO:root:Epoch[77] Time cost=5.003\n",
      "INFO:root:Epoch[78] Train-loss=145.101047\n",
      "INFO:root:Epoch[78] Time cost=4.718\n",
      "INFO:root:Epoch[79] Train-loss=144.810765\n",
      "INFO:root:Epoch[79] Time cost=5.127\n",
      "INFO:root:Epoch[80] Train-loss=144.619876\n",
      "INFO:root:Epoch[80] Time cost=4.737\n",
      "INFO:root:Epoch[81] Train-loss=144.399066\n",
      "INFO:root:Epoch[81] Time cost=4.742\n",
      "INFO:root:Epoch[82] Train-loss=144.220090\n",
      "INFO:root:Epoch[82] Time cost=4.810\n",
      "INFO:root:Epoch[83] Train-loss=143.904279\n",
      "INFO:root:Epoch[83] Time cost=5.176\n",
      "INFO:root:Epoch[84] Train-loss=143.734935\n",
      "INFO:root:Epoch[84] Time cost=4.921\n",
      "INFO:root:Epoch[85] Train-loss=143.499403\n",
      "INFO:root:Epoch[85] Time cost=4.692\n",
      "INFO:root:Epoch[86] Train-loss=143.304287\n",
      "INFO:root:Epoch[86] Time cost=4.778\n",
      "INFO:root:Epoch[87] Train-loss=143.096145\n",
      "INFO:root:Epoch[87] Time cost=4.962\n",
      "INFO:root:Epoch[88] Train-loss=142.877920\n",
      "INFO:root:Epoch[88] Time cost=4.815\n",
      "INFO:root:Epoch[89] Train-loss=142.677429\n",
      "INFO:root:Epoch[89] Time cost=5.127\n",
      "INFO:root:Epoch[90] Train-loss=142.499622\n",
      "INFO:root:Epoch[90] Time cost=5.463\n",
      "INFO:root:Epoch[91] Train-loss=142.300291\n",
      "INFO:root:Epoch[91] Time cost=4.639\n",
      "INFO:root:Epoch[92] Train-loss=142.111362\n",
      "INFO:root:Epoch[92] Time cost=5.064\n",
      "INFO:root:Epoch[93] Train-loss=141.912848\n",
      "INFO:root:Epoch[93] Time cost=4.894\n",
      "INFO:root:Epoch[94] Train-loss=141.723130\n",
      "INFO:root:Epoch[94] Time cost=4.635\n",
      "INFO:root:Epoch[95] Train-loss=141.516580\n",
      "INFO:root:Epoch[95] Time cost=5.063\n",
      "INFO:root:Epoch[96] Train-loss=141.362380\n",
      "INFO:root:Epoch[96] Time cost=4.785\n",
      "INFO:root:Epoch[97] Train-loss=141.178878\n",
      "INFO:root:Epoch[97] Time cost=4.699\n",
      "INFO:root:Epoch[98] Train-loss=141.004168\n",
      "INFO:root:Epoch[98] Time cost=4.959\n",
      "INFO:root:Epoch[99] Train-loss=140.865592\n",
      "INFO:root:Epoch[99] Time cost=5.155\n"
     ]
    }
   ],
   "source": [
    "# training the model, save training loss as a list.\n",
    "training_loss=list()\n",
    "\n",
    "# initilize the parameters for training using Normal.\n",
    "init = mx.init.Normal(0.01)\n",
    "model.fit(nd_iter,  # train data\n",
    "          initializer=init,\n",
    "          # if eval_data is supplied, test loss will also be reported\n",
    "          # eval_data = nd_iter_test,\n",
    "          optimizer='sgd',  # use SGD to train\n",
    "          optimizer_params={'learning_rate':1e-3,'wd':1e-2},  \n",
    "          # save parameters for each epoch if model_prefix is supplied\n",
    "          epoch_end_callback = None if model_prefix==None else mx.callback.do_checkpoint(model_prefix, 1),\n",
    "          batch_end_callback = log_to_list(N/batch_size,training_loss), \n",
    "          num_epoch=100,\n",
    "          eval_metric = 'Loss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "DEBUG:matplotlib.font_manager:findfont: Matching :family=sans-serif:style=normal:variant=normal:weight=normal:stretch=normal:size=12.0 to DejaVu Sans ('/usr/local/lib/python3.5/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf') with score of 0.050000\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEWCAYAAABIVsEJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmYXFWd//H3t6t676S7kw5kX4AAsgbMBMQNFQUZZ9AZxsFlwJVxfjjO4vxQ0BkUxVHHGZfRB0WG38AMi7tEh0cEFRjHYQmrbEIgW4ck3Ul3J7133brf3x/3VlPp1K0Kla6uTufzep56UnXurapz63bO957lnmPujoiIyEtVU+0MiIjIwUkBREREyqIAIiIiZVEAERGRsiiAiIhIWRRARESkLAogMqnM7Jtm9veTve9MYGaNZvYTM9ttZt+rwvdfbmbXHui+ZnammXVObu4S87HRzM6aiu+Sly5d7QzI9GFmG4EPuPud5X6Gu3+oEvvOEOcDhwNz3T2Y6i93989VYt+XwsyWAxuA2mr8BjK5VAOR/WZmh9QFRwWOdxnwTDkF56H228vBQQFEADCz/wCWAj8xswEzu9TMlpuZm9n7zWwz8Mt43++Z2fa4KeYeMzs+73P+3cw+Gz8/08w6zeyjZtZlZtvM7L1l7js3bv7ZY2YPmNlnzezXRY7nVWb2GzPrM7MtZvaeOP0uM/tA3n7vyf+c+HgvMbNngWfN7Goz+9KEz77VzP42fr7QzH5gZt1mtsHMPpKQn08D/wD8afz7vt/Maszsk2a2KT7mG8ysNd6/4G8/4TNzv9mleb/ZW83sXDN7xsx6zOzyvP0/ZWb/OeHzLzKzzWa208w+UWjfIr/x5fH7NprZu/LSf9/MHo7P1RYz+1Te2+6J/+2Lf4dXxO/5oJk9ZWb9ZvakmZ2a955VZvZY/Pf2HTNryPuut5jZI/F5/o2ZnZS37WNmtjX+zN+Z2RuKHY+Uwd310AN3B9gInJX3ejngwA1AM9AYp78PmAXUA18BHsl7z78Dn42fnwkEwJVALXAuMAS0l7HvLfGjCTgO2AL8OuE4lgH9wDviz5oLrIq33UXUTJfb9z35nxMf7x3AHKAReE38XRZvbweGgYVEF2APEgWGOuAI4Hng7IR8fQr4z7zX7wPWx+9rAX4I/Eex337C5+V+s3+Ij/ODQDdwU3x+jo/zumLi9+d9/rfj4zwZGAVeViivCd/7L/HfwGuBQeCYvO0nxr/PScAO4K0Tvjed93l/AmwFfg8w4ChgWd7f5P3x7z0HeAr4ULztFKALOA1IARfF+9cDx8TnbWHe9x5Z7f9jM+2hGojsj0+5+6C7DwO4+3Xu3u/uo0QFzcm5K+cCMsCV7p5x99uAAaL/3Pu9r5mlgD8GrnD3IXd/Eri+SH7fCdzp7jfHn7XL3R95Ccf7j+7eEx/vfxMVeK+Ot50P/K+7v0BU4M1z9yvdfczdnycqkC/Yz+95F/Av7v68uw8AlwEXTGiu2uu3LyADXOXuGaIA2wF8NT4/TwBPEgWHJJ9292F3fxR4tMS+E/29u4+6+93AfwFvB3D3u9z9t+4euvtjwM1EQSbJB4AvuvsDHlnv7pvytn/N3V9w9x7gJ8CqOP1i4Fvufp+7Z939eqIgeDqQJQokx5lZrbtvdPfnXsKxyX5QAJH9sSX3xMxSZvZ5M3vOzPYQXfFBVHAVssv3bvMfIrrafin7ziMa8LElb1v+84mWAAdSWIx/trs7UcH8jjjpncCN8fNlwMK4+aTPzPqAy4k6yvfHQiC/oNxEdJz57y92nBD9Ztn4eS7I7MjbPkzy7w2wPe95sXMzUa+7D+a93kR0PJjZaWb2q7hZbzfwIZL/PqD0+UrK4zLgoxN+/yVEtY71wF8TXeB0mdktZrZwP49N9pMCiORLmpo5P/2dwHnAWUArUdMARE0PldJN1GSyOC9tSZH9twBHJmwbJGoGy5lfYJ+Jv8PNwPlmtoyoueQHed+zwd3b8h6z3P3cInnL9wJRIZizlOg48wPAdJ0uu93MmvNeLyU6Hoia0NYCS9y9FfgmL/59FDqeYuermC1Eta/837/J3W8GcPeb3P1VRL+xA18o4zukCAUQybeDqD2+mFlEzQS7iAriigz3zBdfYf8Q+JSZNZnZscCFRd5yI3CWmb3dzNJxB3yu2eMR4I/izzkKeP9+fP/DwE7gWuB2d++LN90P9MedtY1x7ewEM/u9/Ty0m4G/MbMVZtZC9Ft+xw+e4a2fNrM6M3s18BYgd2/LLKDH3UfMbA3RRUdONxCy99/ZtcDfmdnLLXJUHKxL+TbwobjGY2bWHHfgzzKzY8zs9WZWD4wQ1cTCAzxemUABRPL9I/DJuDng7xL2uYGouWIrUfv6vVOUtw8T1Xi2A/9BVPiOFtrR3TcTdcJ/FOghChq5tv0vA2NEwfJ6XmyOKuUmolrXTXnfkyUqOFcR3duQCzJJ/UETXRcfyz3x+0eAv9zP91bbdqCXqNZxI1HH9tPxtv8DXGlm/UQd/N/Nvcndh4CrgP+J/85Od/fvxWk3EQ1++DFRh3lR7r6OaODA1+O8rCcaFAFR/8fnic7JduAwoj4mmUS5kSUiBxUz+wIw390vqnZeRA5VqoHIQcHMjjWzk+KmijVETU8/qna+RA5lurtVDhaziJqtFhI1P/0zcGtVcyRyiFMTloiIlEVNWCIiUpYZ3YTV0dHhy5cvr3Y2REQOKg8++OBOd59Xar8ZHUCWL1/OunXrqp0NEZGDipltKr2XmrBERKRMCiAiIlIWBRARESmLAoiIiJRFAURERMqiACIiImVRABERkbLM6PtAREQOBkNjATv7x+gbHqOjpZ7DZzeQqonW4BocDdi2e4TB0YCxbMhoJmTPSIaewTH6hsYIQqcuXUNdqoa6dA2pGiNdY8ybVc/rj93fxTHLowAiIhIbC0I6e4foHwnIZEPGsiFhCKE7oTtmRo1ByozhTJau/lG69ozSP5IhlTJSZoQOA6MZBkYCRjIhqVRUoNeYMZLJMpzJMjSWZc9wht3xY2gsu1c+alPG/NYG9gwH7B7OlHUspyxtUwAREUni7uwaHGPTriG69oxgFhXWZjAahAyPRQX2SCbL8FiWoUyWgZGAPSOZ8SCRDZ0gdLbtHmZr7zBhGfPLNtamXgwyGC0NaVrq0zTU1hCETpCNtjXWpmisS9FYm2LJnCZOaKyltbGWuS11dLTU09ZYS/fAKFt6hnmhb5jWxloWtDWwsLWRWQ1p6tMp6tI1zG5M095UR1tTLemaGjLZkNEgJMg7nlwNppIUQESkYsaCkL6hMXqGxkiZ0VyfprkuTf9ohm27R3ihb5gg6zTXp2iuTzM0lmXzriE29wyxbfcw3QNj7OyPrvBDj2oC7kRX9DVGJhvuc/VeTLrGmNWQZlZDLbMa0lGTj0WftWpJO29btYhlc5tpb44K5nTKqE3VEJXFUYEcupONm40On93AvJZ66tLV7U5O1aRoqE1N+fcqgIjIPsLQ6R0ao3coM15oB2HISCa6qh8aC+jPu5Lfndcc0zs4xq7BMXYOjNI/Ut7y7rMb0ixqb6KjpY4jO5qZ3VhLjRmpuJzOhpANQ1I1NSyZ08jSOU0saG2Mt0VX+w210ZV+Q11N9G9titqUxg1NJgUQkRlmJJMlCJ3muhRm0VXz0FjAroExegbHxgv6gdGA0UyWkSBk93CGLT1DdPZGTSe7BsfIvoS2nOa6FLPj5pg5zXWcsKiVuc11zG2uo725jvamOkJ3BkcDBkYDWurTLGhrZGFrA3XpGgZGAwZHszTU1rBsTjOtTbWV+nlkEimAiEwz7k7P4BizGmrHm0bC0NnaN8z6rgH6hscYHI1qAcNjISNB1L7f2TvMs139bO4Zwh1qDJrr0wRZZzhTvJmnNmUsbGtkcXsjZx4zj3mz6pnXUk97cx2puAO4xqChNkVTXZrG2hSzG9PMbqilpSGtK/tDlAKISAXlgkHv0BhjgZPJhgyOBWzri9r/dw6MjnfajmSyPNc9wLM7BugfjZp+OlrqmNNcx9beYQYT2vprU0Z9OsWC1gZOWNjK205ZRFNdiv6RqJkpVWN0tNQzt6WOOU11tDZFNYWokzdFQ20NDekUNVPQ6SoziwKISAm5kT4DIwFB3L6+s3+UzT1RZ2+und8MRjMhuwZH2TkwRnf/KF39I2SyyU1BsxvSpOOr91SNcURHM287NerI7R/JsGPPCDsHxjjjyA6OPnwWRx/ewtyWeprrX6wJTMVoG5FCFEDkkDQ4GjA4Go3TH8pEN2p19kbDOHcPZ8bb6l/oG2ZLz1Di1X9uVA+AA3WpGua21NPRUseKjmYOn93A4bPrmdNcR326htpU1KG7oK2RBa0NVRk5IzJZFEBkRtqxZ4SHNvXy9Pb+8fHxQ5ksG7oHWd89QHf/aMH31aaM1sba8eGmC9saOf2IuSyb20RbUy2pmmjYZ1tTbTzyp2G8BiFyqFEAkRlhcDTg1+t38qunu/j1+p109g4DUbNSbaqGdI1Rn65h2dxmXnv0PI6Y18zshtrxPoD5sxtY3N7EYbPq1Rcgsp8UQOSglMmGPNa5m/99bif/s34XD27qZSwb0lKf5pVHzeU9Zyzn5cvaOX5ha9Vv8hKZqRRAZNpzd17YPcKjW/p4ZEsfD2/u5bHO3YwGIQDHLZjNRWcs43XHHMbq5XMUMESmiAKITBuDowEPbOzh3ud72NI7xMBI1JG9adcQOweiPou6VA3HL5rNu09fxsuXtXP6EXOZ01xX5ZyLHJoUQKQqBkcDfvF0F09t28PmXUNs3DXI77b3E4RObcpY0t7ErIY0zfVpXnN0B6uWtHHS4jZetmAW9WmNXBKZDhRAZMqMBSG/eW4ntz7yAj97fDvDmSzpGmNxeyNL5zbzwdfM44wj5/LyZe001elPU2S60/9SqRh3p7N3mIe39HHnkzv41dNd9I8GzG5I89ZTFvG2UxZx6tI2DYMVOUgpgMikCkPnrme6uOm+LTy8uZddg2MAtDfVcs4J8zn7+Pm8amWHbqATmQEUQGS/BdmQrv5Rtu8ZYdfAGLsGRtkzkqEuVUNTXZqB0YAb79vEc92DzJ/dwOuOPYyTl7Rx8uJWjlswWzUNkRlGAUQS7R7KcNczXfziqS7WbexhR/9oySm+T1g0m69esIpzT1ygGVpFZjgFEBn34KZebvjfjWzbPcKOPdHcUNnQmdtcxxlHdbBsThML4zmcOlrqmdNSR2tjLWNByHAmSxg6i9sbx9egEJGZTQFExn3xZ0/z+NbdHL+wlZMWt3HeyQt57TGHsWpJW/EZX+unLo8iMn0ogBxCdg9n+PkT2/mv327jpMVt/O0bjx7ftnNglAc29vDh1x3F377pmCrmUkQOFgogM1AmG7Jh5yDP7Ohn066h8Rv1Ht7cx1g2pKG2ht+s38V7zlg+fhf3HU/uIHQ454QFVc69iBwsFECmscHRgK19w4xmQjJhyFgQsnNglK49o+wcGCWTDcmGkA1DeocydPWP0NU/ypaeob0WMepoqWfpnEbeffoy/uDkBTTVpTn7K/fww4c6+cCrjwDgZ49vZ9ncJl62YFa1DldEDjIKINPIWBDy40e28qOHtvJc9wBdCWtWQLR6XV2qJl6vGtqb65jXUs+x82dx9vHzOfrwFlYeNosj5jUXvKv71KVt3HT/Zt7/qhXsGQn4zXM7ed8rV6gDXET2mwJIFWzYOcj1v9nII1v6OOqwFo6dPwt3uO5/NrBt9wgrD2vhNUfPY0VHM0vmNNFYmyKdigLGnOY6Dp/dQFtj7QGtW3HBmqVc+v3HeGBjL529UY3lnBPmT+JRishMpwAyhe7f0MM3736OXz7dRW3KWLWkjbuf6eb7D3YCsGb5HD73Rydy5tHzKl4TeMtJC/jMT57klvs3MzAaMH92Aycvbqvod4rIzKIAMgUe6+zjSz9/hnue6aajpY6/esNK3nX6Ug6b1QBEI6B2D2c4cl7LlOWpqS7NH65aOB683rFmqVbiE5GXpCoBxMz+CfgDYAx4Dnivu/fF2y4D3g9kgY+4++1x+jnAV4EUcK27f74aed8f7s4zOwb41e+6+OXTXdy/oYf2plouP/dY/uz05TTW7T0PVEdLPR0tU38zxTvWLOXG+zYDcPbxar4SkZemWjWQO4DL3D0wsy8AlwEfM7PjgAuA44GFwJ1mlrtZ4RvAG4FO4AEzW+vuT1Yh7/vYM5Lh1oe38tDmPp7fOciG7gH2jARAtFre/z37GC58xTJmNdRWOad7O2FRKycuauWFvmHWrJhT7eyIyEGmKgHE3X+e9/Je4Pz4+XnALe4+Cmwws/XAmnjbend/HsDMbon3rVoAcXce2dLHzfdv5iePbmM4k2VhawNHzGvhvFWLOH7hbM485jDmtzZUK4v75V/fcQr9I0HxO81FRAqYDn0g7wO+Ez9fRBRQcjrjNIAtE9JPK/RhZnYxcDHA0qVLJzWjADv2jPD9Bzv5wUOdPN89SFNdireespB3rlnGiYtbJ/37Km15R3O1syAiB6mKBRAzuxMo1LD+CXe/Nd7nE0AA3DhZ3+vu1wDXAKxevbr41LEvUVf/CGd/5R76hjKsWTGHP3/NEZx74oJp1zQlIjIVKhZA3P2sYtvN7D3AW4A3uHuuoN8KLMnbbXGcRpH0KfPZnz7F0GiWn/7lqzhh0cFX2xARmUxVWbAhHlF1KfCH7j6Ut2ktcIGZ1ZvZCmAlcD/wALDSzFaYWR1RR/vaqczzPc90s/bRF/iLM49U8BARoXp9IF8nmgT8jviGuXvd/UPu/oSZfZeoczwALnH3LICZfRi4nWgY73Xu/sRUZXYkk+WTP36cIzqa+Yszj5yqrxURmdaqNQrrqCLbrgKuKpB+G3BbJfOV5F9/+Sybe4a46YOnaS1vEZGY1hwtIQydb//3Bv7g5IWccWRHtbMjIjJtKICUMJaNplE/dr6mORcRyacAUkIQRgPEalO60U5EJJ8CSAlBNgQgXaOfSkQkn0rFEnIr+6kGIiKyNwWQEoIwroGk9FOJiORTqVhCENdA0ppsUERkLwogJWRyfSBqwhIR2YsCSAm5UVjqRBcR2ZtKxRICdaKLiBSkAFLCeCe6aiAiIntRqVhCbhiv+kBERPamAFJC7kbCWg3jFRHZi0rFEl7sRFcNREQknwJICS8O49VPJSKST6ViCRqFJSJSmAJICRqFJSJSmErFEjSZoohIYQogJWgyRRGRwlQqlpDRZIoiIgUpgJTwYie6fioRkXwqFUt4sQlLNRARkXwKICWMd6JrFJaIyF5UKpYQaD0QEZGCFEBKyE1lklInuojIXhRASlAnuohIYSoVSwjCEDPVQEREJlIAKSGTdXWgi4gUoJKxhCAbqgNdRKQABZASgtB1F7qISAEKICVksqE60EVEClDJWEKQdTVhiYgUoABSQiYMtRaIiEgBKhlLCLKutUBERApQACkhCEOtBSIiUoBKxhIyWY3CEhEpRAGkhECjsEREClLJWEIQahSWiEghCiAlZLKhpjIRESmgKiWjmX3GzB4zs0fM7OdmtjBONzP7mpmtj7efmveei8zs2fhx0VTlVfeBiIgUVq1L639y95PcfRXwU+Af4vQ3Ayvjx8XA1QBmNge4AjgNWANcYWbtU5HRTOiaiVdEpICqBBB335P3shnw+Pl5wA0euRdoM7MFwNnAHe7e4+69wB3AOVORV3Wii4gUlq7WF5vZVcCFwG7gdXHyImBL3m6dcVpSesVlNZmiiEhBFbu0NrM7zezxAo/zANz9E+6+BLgR+PAkfu/FZrbOzNZ1d3cf8OdpMkURkcIqVgNx97P2c9cbgduI+ji2Akvyti2O07YCZ05Ivyvhe68BrgFYvXq1F9rnpdAwXhGRwqo1Cmtl3svzgKfj52uBC+PRWKcDu919G3A78CYza487z98Up1VckHVNpigiUkC1+kA+b2bHACGwCfhQnH4bcC6wHhgC3gvg7j1m9hnggXi/K929ZyoyGjVhqQYiIjJRVQKIu/9xQroDlyRsuw64rpL5KkRNWCIihaltpoRMVuuBiIgUopKxBK0HIiJSmAJICVoPRESksJIlo5mdYGY35O6tMLPrzeykqchctbk7maxTqxsJRUT2UTSAxDf9/Yjonov3xY+7gR/kbgicybJhdBuJaiAiIvsqNQrrSuCN7r4xL+0xM/slcGv8mLGC8QCiGoiIyESlLq3TE4IHAHFabSUyNJ1ksiGA1gMRESmgVMkYmNnSiYlmtgwIKpOl6SPIqgYiIpKkVBPWFcCdZvY54ME4bTXwceBjlczYdJAJoxqIZuMVEdlX0QDi7j82sw3AR4G/jJOfBN7u7o9WOnPV9mINRE1YIiITlZzKJA4UF05BXqad8VFYqoGIiOyj1DDeDjO7wsw+YmYtZnZ1vKbHrWZ21FRlslrGO9FVAxER2UepkvEmoJ5ojfL7gQ3A+UTrmF9b2axVn4bxiogkK9WEdbi7X25mBmxy9y/G6U+bWcFZc2eSXA1EkymKiOyrVMmYhfFp1ndO2BZWJEfTSK4TXZMpiojsq1QN5AgzWwtY3nPi1ysqmrNpIMgN41UfiIjIPkoFkPz5rr40YdvE1zNOJlcD0SgsEZF9lLoP5O6kbWb2HaKJFWcs3QciIpLsQErGV0xaLqap8TvR1QciIrIPXVoXMd6JrlFYIiL7KNqEZWanJm3iEJiNN8iqBiIikqRUJ/o/F9n29GRmZDrKhBrGKyKSpFQn+uumKiPTUaAbCUVEEpWaC+vSvOd/MmHb5yqVqelC64GIiCQrdWl9Qd7zyyZsO2eS8zLt5EZhaTJFEZF9lSoZLeF5odczTq4GktKNhCIi+ygVQDzheaHXM05uNl4N4xUR2VepUVgnm9keotpGY/yc+HVDRXM2DWgYr4hIslKjsFJTlZHpSOuBiIgkU9tMEeMrEqoJS0RkHyoZiwiyTo1BjTrRRUT2oQBSRCYMNROviEgClY5FBFnXWiAiIgkUQIoIsqqBiIgkUelYRCZ0TaQoIpJAAaSIIBtqIkURkQQqHYsIsq57QEREEiiAFBE1YeknEhEpRKVjEVETlmogIiKFKIAUkcm6RmGJiCSoauloZh81Mzezjvi1mdnXzGy9mT2Wvya7mV1kZs/Gj4umIn9BGGoUlohIglKz8VaMmS0B3gRszkt+M7AyfpwGXA2cZmZzgCuA1UTTyD9oZmvdvbeSeQyyrrVAREQSVLMG8mXgUvZeV+Q84AaP3Au0mdkC4GzgDnfviYPGHUzBioiZbKiJFEVEElSldDSz84Ct7v7ohE2LgC15rzvjtKT0Qp99sZmtM7N13d3dB5TPbKhhvCIiSSrWhGVmdwLzC2z6BHA5UfPVpHP3a4BrAFavXn1AqyZmQqdJnegiIgVVLIC4+1mF0s3sRGAF8KiZASwGHjKzNcBWYEne7ovjtK3AmRPS75r0TE8QZENNpigikmDKL6/d/bfufpi7L3f35UTNUae6+3ZgLXBhPBrrdGC3u28DbgfeZGbtZtZOVHu5vdJ51Z3oIiLJqjYKK8FtwLnAemAIeC+Au/eY2WeAB+L9rnT3nkpnRuuBiIgkq3oAiWshuecOXJKw33XAdVOULUDrgYiIFKPL6yK0HoiISDKVjkVoPRARkWQKIEVoPRARkWQqHYvQKCwRkWQKIEVkwlDrgYiIJFDpWESQda0HIiKSQAEkgbsThFoPREQkiUrHBEEYTaOl+0BERApTAEkQZKMAohqIiEhhKh0TZMIQQH0gIiIJFEASZMdrIAogIiKFKIAkGK+BqAlLRKQglY4Jcn0g6kQXESlMASSBOtFFRIpT6Zgg14SlyRRFRApTAEkwXgPRZIoiIgWpdEyQyeY60VUDEREpRAEkwfid6AogIiIFKYAkCHI1EDVhiYgUpNIxQUY3EoqIFKUAkiAYH4Wln0hEpBCVjgleHIWlGoiISCEKIAlyo7BUAxERKUylY4LcKCz1gYiIFKYAkiCjUVgiIkWpdEygPhARkeIUQBJk1YQlIlKUAkiCjIbxiogUpdIxgZqwRESKUwBJ8OJkivqJREQKUemYQJMpiogUpwCSQJMpiogUp9IxQW4yRdVAREQKUwBJEIQhqRrDTAFERKQQBZAEQdY1AktEpAgFkASZrOseEBGRIlRCJgjCUHehi4gUoQCSIJN1jcASESlCJWSCIBtqBJaISBFVCSBm9ikz22pmj8SPc/O2XWZm683sd2Z2dl76OXHaejP7eKXzGISuJiwRkSLSVfzuL7v7l/ITzOw44ALgeGAhcKeZHR1v/gbwRqATeMDM1rr7k5XKXCYbUqsmLBGRRNUMIIWcB9zi7qPABjNbD6yJt6139+cBzOyWeN+KBZAg66Q0jFdEJFE1L7E/bGaPmdl1ZtYepy0CtuTt0xmnJaXvw8wuNrN1Zrauu7u77MxFo7BUAxERSVKxEtLM7jSzxws8zgOuBo4EVgHbgH+erO9192vcfbW7r543b17ZnxOErk50EZEiKtaE5e5n7c9+ZvZt4Kfxy63AkrzNi+M0iqRXhO5EFxEprlqjsBbkvXwb8Hj8fC1wgZnVm9kKYCVwP/AAsNLMVphZHVFH+9pK5jGTVROWiEgx1epE/6KZrQIc2Aj8OYC7P2Fm3yXqHA+AS9w9C2BmHwZuB1LAde7+RCUzGIROQ60CiIhIkqoEEHf/syLbrgKuKpB+G3BbJfOVL8iGpOun2yA1EZHpQ5fYCaLJFNUHIiKSRAEkQRCGmgtLRKQIlZAJgqymMhERKUYBJEEmDLUeiIhIESohE+g+EBGR4hRAEmSyrvtARESKUAmZIAi1HoiISDEKIAkCrUgoIlKUSsgEGa1IKCJSlAJIAq1IKCJSnAJIAe5ONnRSasISEUmkErKAIHQAajWMV0QkkQJIAUE2CiAaxisikkwlZAGZMARQJ7qISBEKIAWM10DUhCUikkgBpIBUjfH7Jy5gxbyWamdFRGTa0opJBbQ21vKNd51a7WyIiExrqoGIiEhZFEBERKQsCiAiIlIWBRARESmLAoiIiJRFAURERMqiACIiImVRABERkbKYu1c7DxVjZt3ApgP4iA5g5yRl52BxKB4zHJrHfSgeMxyax/0mnpXeAAAFlUlEQVRSj3mZu88rtdOMDiAHyszWufvqaudjKh2KxwyH5nEfiscMh+ZxV+qY1YQlIiJlUQAREZGyKIAUd021M1AFh+Ixw6F53IfiMcOhedwVOWb1gYiISFlUAxERkbIogIiISFkUQAows3PM7Hdmtt7MPl7t/FSKmS0xs1+Z2ZNm9oSZ/VWcPsfM7jCzZ+N/26ud18lmZikze9jMfhq/XmFm98Xn/DtmVlftPE42M2szs++b2dNm9pSZvWKmn2sz+5v4b/txM7vZzBpm4rk2s+vMrMvMHs9LK3huLfK1+PgfM7OyV89TAJnAzFLAN4A3A8cB7zCz46qbq4oJgI+6+3HA6cAl8bF+HPiFu68EfhG/nmn+Cngq7/UXgC+7+1FAL/D+quSqsr4K/MzdjwVOJjr+GXuuzWwR8BFgtbufAKSAC5iZ5/rfgXMmpCWd2zcDK+PHxcDV5X6pAsi+1gDr3f15dx8DbgHOq3KeKsLdt7n7Q/HzfqICZRHR8V4f73Y98Nbq5LAyzGwx8PvAtfFrA14PfD/eZSYecyvwGuDfANx9zN37mOHnmmjZ7kYzSwNNwDZm4Ll293uAngnJSef2POAGj9wLtJnZgnK+VwFkX4uALXmvO+O0Gc3MlgOnAPcBh7v7tnjTduDwKmWrUr4CXAqE8eu5QJ+7B/HrmXjOVwDdwP+Lm+6uNbNmZvC5dvetwJeAzUSBYzfwIDP/XOckndtJK+MUQAQzawF+APy1u+/J3+bROO8ZM9bbzN4CdLn7g9XOyxRLA6cCV7v7KcAgE5qrZuC5bie62l4BLASa2beZ55BQqXOrALKvrcCSvNeL47QZycxqiYLHje7+wzh5R65KG//bVa38VcArgT80s41EzZOvJ+obaIubOWBmnvNOoNPd74tff58ooMzkc30WsMHdu909A/yQ6PzP9HOdk3RuJ62MUwDZ1wPAynikRh1Rp9vaKuepIuK2/38DnnL3f8nbtBa4KH5+EXDrVOetUtz9Mndf7O7Lic7tL939XcCvgPPj3WbUMQO4+3Zgi5kdEye9AXiSGXyuiZquTjezpvhvPXfMM/pc50k6t2uBC+PRWKcDu/Oaul4S3YlegJmdS9ROngKuc/erqpylijCzVwH/DfyWF/sDLifqB/kusJRoOvy3u/vEDrqDnpmdCfydu7/FzI4gqpHMAR4G3u3uo9XM32Qzs1VEAwfqgOeB9xJdRM7Yc21mnwb+lGjE4cPAB4ja+2fUuTazm4EziaZt3wFcAfyYAuc2DqZfJ2rOGwLe6+7ryvpeBRARESmHmrBERKQsCiAiIlIWBRARESmLAoiIiJRFAURERMqiACIyTZnZmbnZgkWmIwUQEREpiwKIyAEys3eb2f1m9oiZfStea2TAzL4cr0XxCzObF++7yszujddh+FHeGg1HmdmdZvaomT1kZkfGH9+St4bHjfFNYCLTggKIyAEws5cR3en8SndfBWSBdxFN3LfO3Y8H7ia6MxjgBuBj7n4S0QwAufQbgW+4+8nAGUSzx0I0Q/JfE61NcwTRXE4i00K69C4iUsQbgJcDD8SVg0aiSetC4DvxPv8J/DBek6PN3e+O068Hvmdms4BF7v4jAHcfAYg/735374xfPwIsB35d+cMSKU0BROTAGHC9u1+2V6LZ30/Yr9w5g/LnaMqi/7MyjagJS+TA/AI438wOg/F1qJcR/d/Kzfj6TuDX7r4b6DWzV8fpfwbcHa8G2Wlmb40/o97Mmqb0KETKoKsZkQPg7k+a2SeBn5tZDZABLiFasGlNvK2LqJ8Eomm1vxkHiNyMuBAFk2+Z2ZXxZ/zJFB6GSFk0G69IBZjZgLu3VDsfIpWkJiwRESmLaiAiIlIW1UBERKQsCiAiIlIWBRARESmLAoiIiJRFAURERMry/wGCuVFXyLXDyQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ELBO = [-training_loss[i] for i in range(len(training_loss))]\n",
    "plt.plot(ELBO)\n",
    "plt.ylabel('ELBO');plt.xlabel('epoch');plt.title(\"training curve for mini batches\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the ELBO is monotonically increasing over epoch, and we reproduced the results given in the paper [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114/). Now we can extract/load the parameters and then feed the network forward to calculate $y$ which is the reconstructed image, and we can also calculate the ELBO for the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "arg_params = model.get_params()[0]\n",
    "nd_iter_test.reset()\n",
    "test_batch = nd_iter_test.next()\n",
    "\n",
    "# if saved the parameters, can load them using `load_checkpoint` method at e.g. 100th epoch\n",
    "# sym, arg_params, aux_params = mx.model.load_checkpoint(model_prefix, 100)\n",
    "# assert sym.tojson() == output.tojson()\n",
    "\n",
    "e = y.bind(mx.cpu(), {'data': test_batch.data[0],\n",
    "                     'encoder_h_weight': arg_params['encoder_h_weight'],\n",
    "                     'encoder_h_bias': arg_params['encoder_h_bias'],\n",
    "                     'mu_weight': arg_params['mu_weight'],\n",
    "                     'mu_bias': arg_params['mu_bias'],\n",
    "                     'logvar_weight':arg_params['logvar_weight'],\n",
    "                     'logvar_bias':arg_params['logvar_bias'],\n",
    "                     'decoder_z_weight':arg_params['decoder_z_weight'],\n",
    "                     'decoder_z_bias':arg_params['decoder_z_bias'],\n",
    "                     'decoder_x_weight':arg_params['decoder_x_weight'],\n",
    "                     'decoder_x_bias':arg_params['decoder_x_bias'],                \n",
    "                     'loss_label':label})\n",
    "\n",
    "x_fit = e.forward()\n",
    "x_construction = x_fit[0].asnumpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADACAYAAADhh27FAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmUXWWZ7/HfAxnIPJC6SQgJCRAwYQzUihG4t72SjggyZNnQAkKiMngXAi7TKtpqR1qQq40IF5cNNpNCqyBCEIIKNAi0QFPYQYaQhEwmUCEVMlUChCS894+zo4e8z0l21Rn32d/PWrVy6tnTu8956q03u/azXwshCAAAAMijPerdAAAAAKBeGAwDAAAgtxgMAwAAILcYDAMAACC3GAwDAAAgtxgMAwAAILcYDDcAM/s3M/tavdsBlMPMxppZMLMeJZa/ZGYfrnGzABf9LrKOPrdyjOcMS2a2qejbvpK2SNqefH9hCOGO2rcKeWZmyySdF0J4uN5tScvMxkpaKqlnCGFbfVuDRke/i0ZCn5tv7v8m8iaE0H/H6zQ/EGbWg8RD1pHHqCf6XeQNOdy4uE0iBTP7tpn9wsx+Zmadkj5lZreb2eyidaYmHfqO7/c1s3vMrMPMlprZRbvY/1/2tWM/ZvbVZNvXzexkM/u4mS0ys7Vm9uWibT9kZk+b2Xozazez68ysZ9Hyj5nZQjPbYGb/z8z+08xmFi0/z8xeMbN1ZvagmY2u0NuGKklyYV7ymf/BzA4vWnaZmS02s04ze9nMphctm5l8/teY2ZuSZiexJ83sX5IcWGpmHyvaZpCZ3ZTk1mvJz8KeybI9k+3WmNkSSSftpt3LzGxq8nq2md2V5H6nmb1gZgcleb/azFaY2bSibT9tZvOTdZeY2YU77fvLSRtfT3I6mNmBybLeSTv/bGZvmNm/mlmf8j4FVBv9LhoFfW7z97kMhtObLunfJQ2S9ItdrWhme0i6X9KzkkZJ+ltJXzKz41Mea18VPpt9JP2zpJskfVLSJEkflnS5mY1J1t0m6VJJwyQdK+kESRcm7fgfku6U9KVk+VJJk4va+Ylk2amSWiQ9k5wjGpSZTZJ0swqf8d6SbpB0n5n1TlZZLOl/qpCn35J0u5mNLNrFByUtkTRc0hVFsQUq5Mh3Jd1kZpYsu1WFHDtQhfybJum8ZNn5kj6exFsl/V0XT+dkST+VNETSf0v6rQp5P0rS5cm57bA6OdZASZ+WdI2ZHZW8JydI+qKkqUk7P7zTca6SdJCkI5PloyR9s4ttRX3Q76Ku6HNz0ueGEPgq+pK0TNLUnWLflvQfO8VulzS76PupkpYlr4+VtGSn9b8h6ccljvmXfSX72SRpz+T7IZKCpKOL1n9e0sdL7OsfJN2VvP6MpCeKlpmkdkkzk+8fkjSjaHkPFe7bG1XvzyHvX14eJvEfSfrnnWILJP1Nif3Mk3Rq8nqmpD/vtHympFeLvu+b5NsIFTrvLZL6FC0/U9Kjyev/kPS5omXTkm177O6cJM2W9FDRspN3yvsByb4Gl9jXvZIuTV7fLOk7RcsOTLY9MMn5zZIOKFr+IUlL6/0Z87XrfKff5aveOZjE6XND8/e53DOc3oourLufpDFmtr4otqekx1JuvyaEsKOQ5O3k3zeKlr8tqb8kmdkHJF0t6WgVfqh6qHClQSpc4fhLu0MIwcxW7tTOH5rZtUWx91S4QvJayraitvaTNMPMLi6K9VLhs5aZnavC/9jHJsv6q3D1YQcvj1fteBFCeCu5QNFf0lBJPSW1//WihfYo2sf78kvS8i6ey8457eV9f0nrkz8j/pMKVxv2UCHXXyhqR1vRvorb1JKs+1zROZgKP49ofPS7qDf63Bz0uQyG09v5sRubVfjAdxhR9HqFpEUhhAlVb1XhzxpPS/r7EMImM/sHFf60IRWuRhTfA2Qq/LmiuJ3fCCHs8s+PaCgrJF0RQrhi5wVmtp+kH0s6XtJTIYTtZjZPhY5oh648PmaFClcphgW/6KNdUvG9jmOcdcqW/DnybknnSpoTQthqZvfqr+fVrsJAYofiNq1RoZM/JITAQCN76HdRb/S5OehzuWe4++ZJOsnMhiT3B11StOwpSe+a2Swz2yu56f0wMzu6Cu0YIGmDpM1mNkHJfWuJ+yUdZYVCkB4q3OPWUrT8XyX9Y7KdzGywmXX1HiRUT88kf3Z89VCh4/2cmX3QCvqZ2UlmNkBSPxU63g6pUAAh6dDuHjyE0C7pd5KuNrOBZraHmR1gZn+TrHKnpEusULQ0RNJlZZzrrvSS1FuF89qWXLGYVrT8TkmfNrMJZtZXhT+N7ziH91R4z65J7uWUmY0ys49Wqa2oLvpdVBN9bkHu+lwGw913q6T5KvyZ4jeSfr5jQfI/uhNVKJpYpsL/lG5Q4Ub0SpslaYakzuQYf7naEEJ4Q9LfS/q+pDclHaDCTfNbkuV3JcvuMrONkv4kqaETNmfmqvA/7B1fs0MIbSoUUVwvaZ2kV1W4B00hhJdV+NPtUyr8OewwSf9ZZhvOVaFjfDk53i8l7SgO+bEKBRjPS/qjpF+VeSxXCKFThUHPnUkbzpJ0X9HyByVdJ+lRFd6Pp5NFW5J/v7IjnuT5w5IOrkZbUXW3in4X1UOfq3z2uUy6kSNWeDzL65L+LoTwRL3bA1RDcsXtRUm9S/ypEagZ+l00u2boc7ky3OTM7ITkz3C9VfhTxlZJ/1XnZgEVZWbTrfBsyyGS/q+kX2e1U0b20e+i2TVbn8tguPkdp8IzDjtU+FPc9BDCll1vAmTOhSo8F3OxClP6/p/6Ngc5R7+LZtdUfS63SQAAACC3uDIMAACA3CrrOcNWmJLvWhUepvxvIYSrdrX+sGHDwtixY8s5JKDnnntuTQihZfdrVg65i3ItW7ZMa9assd2vWTnkLSqBPhdZlTZ3uz0YTipkf6jC/O8rJT1rZvcljxpxjR07Vm1tbaUWA6mYWVdn3SkbuYtytba21vyY5C0qgT4XWZU2d8u5TWKyCvNrLwkhvKvC8x5PLWN/AAAAQE2VMxgepffPR71S759yUpJkZheYWZuZtXV0dJRxOKC2yF1kEXmLrCJ3US9VL6ALIdwYQmgNIbS2tNT0liOgLOQusoi8RVaRu6iXcgbDr0kaXfT9vkkMAAAAyIRyBsPPShpvZuPMrJekT6po7moAAACg0XX7aRIhhG1m9nlJv1Xh0Wo3hxBeqljLAAAAgCor6znDIYS5kuZWqC0AAABATTEDHQAAAHKrrCvDAAAA+KsQQr2b8D5mNZ34MpO4MgwAAIDcYjAMAACA3GIwDAAAgNxiMAwAAIDcYjAMAACA3OJpEgAAAN1QjSdH8PSH2uPKMAAAAHKLwTAAAAByi8EwAAAAcovBMAAAAHKLAjoAAIBdqOUUy2mPRaFd5XBlGAAAALnFYBgAAAC5xWAYAAAAucVgGAAAALlVVgGdmS2T1Clpu6RtIYTWSjQKQP7UskCl0ihkAbIpbb9Taj0vvn379lSxUrz+pEePeLjmrefFSrX9vffeS7XuHnvE1029WFf6wUbrMyvxNIn/HUJYU4H9AAAAADXFbRIAAADIrXIHw0HS78zsOTO7wFvBzC4wszYza+vo6CjzcEDtkLvIIvIWWUXuol7KHQwfF0I4StLHJF1kZv9r5xVCCDeGEFpDCK0tLS1lHg6oHXIXWUTeIqvIXdRLWfcMhxBeS/5dbWb3SJos6fFKNAxoduUUjHWlmMMrkvBUo6DBa081iizqXbhR7nki5r2nb775ZhSbM2eOu/0ll1wSxd56661Ux+7Xr18Uu+6669x1zz333CjmFTuh/rycShsrVQC3devWKLZhw4ZUsVL9eO/evVPFevXqFcW8fqfU7wAv3rNnzyjWt2/fVMf2iupKtanRdPvKsJn1M7MBO15LmibpxUo1DAAAAKi2cv77OlzSPcmIv4ekfw8h/KYirQIAAABqoNuD4RDCEklHVLAtAAAAQE3xaDUAAADkVm7u8n/66aej2LXXXuuuO2rUqCjWp0+fKDZjxowoNnToUHefpeLIB69QwYt5RRpbtmyJYp2dne5xNm3alGr7rhSIpG3TO++8E8XefffdKLbnnnu6x/EKMrxijr322iuKjRgxIooNGzbMPY63z7QzOaE6lixZEsW+/e1vR7HbbrutrOOUKvDZ2dtvvx3Fzj//fHfdhQsXRrErr7yy28dGZaQtjEvbN3v9myStWLEiij377LNRbOXKlVHM6/MkaeDAgVFs7733jmLeuMIr3vT6Ycnv271+c/To0VFs8ODBUSzLOZ7dlgMAAABlYjAMAACA3GIwDAAAgNxiMAwAAIDcYjAMAACA3MrN0yS8Jz8sWrSorH1eccUVUWzQoEHuulOmTCnrWLUwduzYKPbVr37VXXfMmDFVbk02lZr2ctu2bVHMm8Zz3bp1UeyVV16JYk8++aR7nD//+c9RbOPGjamOU2qq2s2bN6eKlTPVreRXUHtPfvB+xiZPnhzFZs6c6R7Hy12mz62NUtPPfulLX4pi9957b8WP7z3JxIuVqr73fO9734tixxxzTBQ75ZRTUu8T6XVlavq0T5Pwnpbj9a2SdOedd0axF154IYp5fZn3O1fyn9Tg9ZvePr0noZRqu/d7YMiQIVHMe+qF155S/WgWnszDlWEAAADkFoNhAAAA5BaDYQAAAOQWg2EAAADkVm6qRrxijHnz5rnrHnLIIVHspZdeimLPPPNMFJszZ467z9/+9rdRbNy4cVFs6dKl7vZpeTewjxw5Mop5U0h6St3g/5WvfKVL7cqLUsUc3rSXXsGZV+jwu9/9Loo9+uij7nHa29tTHSftVNCleNt75967d+8o5k2nLElr166NYhs2bIhiXju9IsHjjjvOPY433ToFdJXn5cMNN9zgrpu2WK5Pnz5R7NBDD3XXnTVrVhT7yEc+EsW8aW4vu+yyKOYVypVyxx13RLGPfvSjUcz7+UD1eP2WV9zsFZbNnTvX3ef9998fxbwitkmTJkWxAw880N2nl9NeUZ1X2Ob1owsWLHCP4z1EwOufW1paotg+++yTatus4MowAAAAcovBMAAAAHKLwTAAAAByi8EwAAAAcmu3VSNmdrOkj0taHUI4NIkNlfQLSWMlLZN0RgghvuO8gUyYMCFVrJTDDz88ip155plR7KqrrnK3X7ZsWRTzCuiWLFmSuk0e74Z6r4DOO3ZHR0cU+8AHPlBWe1CaV8yxxx7x/0+9Ipthw4a5+/SKlrziMK8YwyuSkPwCI2/WLm+WIW+2OG+GI8kvaH3ggQeimFcgkrZIsFqyMMNSrXnv/0UXXZR6ey/HvvnNb0axL3/5y11rWAqzZ8+OYt5MY5K0fPnyKPbLX/4yinkz0J199tldbxxSSTvbnDfboPf7+vHHH3eP4xU9jx49OopNnDgxinkzZ0p+/963b98o5vU7XSmEXr16dRTzZt8bMWJEFPMKAgcMGOAex/u91mh9Zporw7dKOmGn2GWSHgkhjJf0SPI9AAAAkCm7HQyHEB6XtPOlmFMl3Za8vk3SaRVuFwAAAFB13b1neHgIYccDTVdJGl5qRTO7wMzazKzN+zM80KjIXWQReYusIndRL2UX0IXCjTn+TAOF5TeGEFpDCK2l7kkEGhG5iywib5FV5C7qpbvTLr1hZiNDCO1mNlJSfBd2TpWagSVtIVpXivrS8mbKW7NmTRT74Ac/GMWmTZtW8fbkkVdA4BVEeLOjHX/88anWk6TNmzdHMa8Yw8vHUr98vHam5RXvebMzSdLWrVuj2G9+85so5r2Xw4fHf5zyij6k9MV/qK/LL788ilWjWM7j9eN33323u25ra2uqfV555ZVR7NRTT41i/fv3T7U/7Fran2mv3/EK2UvNDtuzZ88otv/++0exD33oQ1HM67ckPwe8fittsZxXFCdJ69evj2LerJ/eeKGzszOKlfod4v0e8Aoc69kPd/fK8H2SZiSvZ0jy5yAGAAAAGthuB8Nm9jNJT0k62MxWmtlnJV0l6W/NbJGkqcn3AAAAQKbs9jaJEEL8MN2C+G+3AAAAQIYwAx0AAAByi8EwAAAAcqu7T5NAg/KeJjB9+vQo5k1L+YMf/CCK9enTpzINywnvSQeSXwnsTZ3tTX3sPc1hzJgx7nG86uKBAwdGMW/aTK/iV0pf4Zt2+tNSlc0rV66MYl4+e+/HUUcdFcW8acgl/zx5mkTlvfjii6nX9abovvjiiyvZnC7xcvSss84qa5+vvPJKFPvud78bxbynaKB6vD7Km/J927Zt7vZeP+7lsxfr3bu3u89Sv0fStGnjxo1RzHtqhCStW7cuinlT23uxd955J4qVerqF97vBi3lq1TdzZRgAAAC5xWAYAAAAucVgGAAAALnFYBgAAAC5RQFdk7n11luj2KpVq6KYV6i13377VaNJuVLqZv+0BRFeoZ0XK1XY6BUleAVj5U5JnLb4wSuoKDWt6R/+8Ico9u6770axww8/PIqdeOKJUWzw4MHucSiWq42XX3459br1LGr0iotmzpwZxRYuXFjxY3v99ezZs9110/YhedOVPPHW9fpCr6jOW0/y+7hNmzZFMe/3sFfcLPn9u9cmb5rk+fPnR7H29nb3OB0dHVHM69u9Yjmvby5VZOi9R+X+fFe6f+CnCwAAALnFYBgAAAC5xWAYAAAAucVgGAAAALlFAV1GLV682I1/8YtfTLX9U089FcVGjBhRVptQmnezf9qCGG+9UsUcaY9dDV6Bx4YNG6LYnDlz3O2XLFkSxbwCE69Y7uCDD45iPXv2dI+D2mhtbU29rlfI85Of/CSKfe5zn0u9T68QyCs4+vrXvx7Ffv3rX6c+TjnOOOOMKEaBZ/V4763XTwwbNiyKlSp282are+mll6LYgw8+GMVeffVVd5977bVXFNu6dWsU6+zsjGLez9Ly5cvd43jFo9579Pbbb0cxb3bQUrOLejPteb/XvBgz0AEAAABVxmAYAAAAucVgGAAAALnFYBgAAAC5tdsCOjO7WdLHJa0OIRyaxGZLOl/Sjju1vxZCmFutRiJWqsDDu8n+9NNPj2L7779/xduErimnMKBUAV3ameEqva3kzz70wgsvRLHHHnvM3d6bpWjy5MlR7OSTT45igwYNimKlChQpUKqNcvuYW265JYqNHz8+ih1wwAHu9tdff30Uu+aaa8pqUzm8HD3nnHOiGPlZPV6f0KtXryh26KGHpopJfgGdV6jp9XsLFixw9zlgwIAo5hWh9e3bN4p551NqZjhvZjkv/7y+2Suq88Yfkl9cXe7vm0pLc2X4VkknOPFrQghHJl8MhAEAAJA5ux0MhxAelxT/1wcAAADIuHLuGf68mf3JzG42syGlVjKzC8yszczavOffAY2K3EUWkbfIKnIX9dLdwfCPJB0g6UhJ7ZKuLrViCOHGEEJrCKG1paWlm4cDao/cRRaRt8gqchf10q0Z6EIIb+x4bWY/lnR/xVqEiHdT+j333OOu691k/53vfCeKdWUGM5TPK0rwCgjSFs+UKj5Ie5y025ba3iuIWLVqVRS7/fbbo9iKFSvc44wcOTKKfeITn4hi48aNi2LeLFIUItWXV6x09dX+dZNZs2ZFsba2tig2bdq08hvWTUcccYQbf/7551NtP3Xq1NT7RPnSzvrZo0c8DBo7dmwUO+2009zjeL9zX3nllSjmFat5hWml2uT952D06NGp2vPmm2+6x0k745s3I16fPn2iWJbHFd26Mmxmxb+1pkt6sTLNAQAAAGonzaPVfibpw5KGmdlKSf8k6cNmdqSkIGmZpAur2EYAAACgKnY7GA4hnOmEb6pCWwAAAICaYgY6AAAA5BaDYQAAAORWt54mgdq66ab4rpQnnnjCXfess86KYky93JjKedpBV578UM56pdZ96623otgdd9wRxR5++OHUxzn++OOjmFeB701BmrYqGrXjvf+XXHKJu6431e3Pf/7zVDFvWthSxx8+fHgU86b87spTUA455BA3vrNLL7001XqoHi8nvCcgDBw4MIpNmTLF3aeXU4sWLYpi3hMdSvVRw4YNi2LeU3S89To7O6PY0qVL3eP069cvinnvx5Ah8VQS3nvkPdWn1D4brX/myjAAAAByi8EwAAAAcovBMAAAAHKLwTAAAAByiwK6BjNv3rwodvHFF0exwYMHu9tffvnlFW8Tmk9Xpm3etm1bFHvuueeimFdAt379+ih29NFHu8c555xzopg3BWkWijHg8wodJb9Q0ot94xvfiGKvvfaau09vSluvWC6tMWPGuPEDDzwwim3YsCGKHXbYYd0+NqrHy0mvEGzQoEHu9hMnToxi3nTOW7ZsiWLetPaSP/1x2sJhbz2vAE7yC+i86Zy9ftgroPO2LdXORit65sowAAAAcovBMAAAAHKLwTAAAAByi8EwAAAAcosCujryZk4688wzo9j27duj2Nlnn+3uk9nm8i1tAYJXLFeqmGPVqlVR7Prrr49i3ixHXiHI9OnT3eMcdNBBUcwrgvLOkQK6fNhvv/1SxaqhVIGp1z97RUxewREak9efeH2RlL4Ar1T/mvb43nHS7tMr3it1HC9PvVifPn2iWLkz0NWzb+fKMAAAAHKLwTAAAAByi8EwAAAAcovBMAAAAHJrtwV0ZjZa0k8kDZcUJN0YQrjWzIZK+oWksZKWSTojhLCuek3NNu9G95NOOimKLViwIIpNmDAhin3rW9+qTMOQS14xkFfQKUl33XVXFPv9738fxbZu3RrFvNnmTjnlFPc43uxF5RQEUlSHSlq8eLEb9wpHb7jhhmo3BzVWqj9JO5NaVwro0vL6vc2bN0exdev8oZlX8Ja2WC5tcXOpeKP1z2muDG+TNCuEMFHSFEkXmdlESZdJeiSEMF7SI8n3AAAAQGbsdjAcQmgPIfwxed0pab6kUZJOlXRbstptkk6rViMBAACAaujSPcNmNlbSJEnPSBoeQmhPFq1S4TYKb5sLzKzNzNo6OjrKaCpQW+Qusoi8RVaRu6iX1INhM+sv6W5JXwghbCxeFgo3rrhPJA8h3BhCaA0htLa0tJTVWKCWyF1kEXmLrCJ3US+pZqAzs54qDITvCCH8Kgm/YWYjQwjtZjZS0upqNbIZrF27Noo99thjqbb96U9/GsWGDh1abpOQE16RhVfsNn/+fHf7uXPnRrFNmzZFscGDB0exT33qU1Fsn332cY+TdpYioB66UhT3wAMPRLHzzjuvks1Bg0jbR3mFdl3hFeB5sXfeeSeKlWqjV7Ts9cOetIWDu4p3d71q2O2nY4XW3SRpfgjh+0WL7pM0I3k9Q9KcyjcPAAAAqJ40V4aPlXSOpBfMbF4S+5qkqyTdaWaflbRc0hnVaSIAAABQHbsdDIcQnpRU6tr18ZVtDgAAAFA7zEAHAACA3GIwDAAAgNxK9TQJpLdhwwY3PmXKlFTb33777VFs0qRJZbUJ+eE9OWL79u1RzMvTxx9/3N3nihUrolj//v2j2LRp06KYN/Vyr1693ONUupLYey9K4akVqKQnn3wyiq1fvz6KeU9gQfaV25+UM7289zSIUk+I8J5G4U2z7D05oiv9aznnUytcGQYAAEBuMRgGAABAbjEYBgAAQG4xGAYAAEBuUUBXYbfccosbX7JkSartjzvuuCjWaDeao/5KFS94BRFbtmyJYsuXL49iCxcudPfpTdl52GGHRbHTTz89ig0ZMiSKlTstaTWKMbJQ4IHsWLt2bRTzfr4mT55ci+agAZTbn3h9u9dvbd26NYqV6nP33nvvKDZw4MAo5hXgecfuSlFdo+HKMAAAAHKLwTAAAAByi8EwAAAAcovBMAAAAHKLAroyLFq0KIrNnj279g1BJqUtNuhKoYJXPOHNfPX8889HsXXr1rn79IrgjjjiiCh28MEHR7GuzFzkFZhQ2AYA6fXs2TOKjRkzxl1348aNUaylpSWKjRs3Lop5hdWlCvWy0GdzZRgAAAC5xWAYAAAAucVgGAAAALnFYBgAAAC5tdsCOjMbLeknkoZLCpJuDCFca2azJZ0vqSNZ9WshhLnVamgjeuKJJ6KYd0N6KRMmTIhiffr0KatNyI60BWNebNu2be4+N2/eHMVWrFgRxRYvXhzFNm3a5O6zb9++Ucybucgrnkg7a1K9ZaHAA/U1ffp0N37PPfdEsbfeeiuK7bvvvhVvE/LD66N69IiHcMOHD49iU6dOdfd51FFHRbH+/ftHsdGjR0exAQMGRDFvpjrJb3uj9blpniaxTdKsEMIfzWyApOfM7KFk2TUhhH+pXvMAAACA6tntYDiE0C6pPXndaWbzJY2qdsMAAACAauvSPcNmNlbSJEnPJKHPm9mfzOxmM4sfRlrY5gIzazOzto6ODm8VoCGRu8gi8hZZRe6iXlIPhs2sv6S7JX0hhLBR0o8kHSDpSBWuHF/tbRdCuDGE0BpCaPUe5gw0KnIXWUTeIqvIXdRLqsGwmfVUYSB8RwjhV5IUQngjhLA9hPCepB9Lmly9ZgIAAACVl+ZpEibpJknzQwjfL4qPTO4nlqTpkl6sThObwzHHHBPFHnrooSjG0ySwM6/qttS0l95UnN50ygcddFDq4/fr1y+KHXvssVFs8ODBUaxUOz1pp14ud4rmRqtiRjaUqsh//fXXa9wS5JHXb3lPb/Ce/rP//vu7+/SmWfaeUOEdx4uV6luz0OemeZrEsZLOkfSCmc1LYl+TdKaZHanC49aWSbqwKi0EAAAAqiTN0ySelOQN63P1TGEAAAA0H2agAwAAQG4xGAYAAEBupblnGCV85jOfSRUD0ko7bWWpggRvikyvWG78+PFRzJs6WUo/fXK5bS93ewBoVmkL6EpNiVzpYzcbrgwDAAAgtxgMAwAAILcYDAMAACC3GAwDAAAgtyxtcUxFDmbWIWl58u0wSWtqdvDq43xqZ78QQk0nri/K3UZ+X7qD86mdeuat1NjvTXdwPrVD7lYW51M7qXK3poPh9x3YrC2E0FqXg1cB55MPzfa+cD750WzvDeeTH8323nA+jYfbJAAAAJBbDIYBAACQW/UcDN9Yx2NXA+eTD832vnA++dFs7w3nkx/N9t5wPg2mbvcMAwAAAPXGbRIAAADILQbDAAAAyK2aD4Zu3yMbAAACn0lEQVTN7AQzW2Bmr5rZZbU+frnM7GYzW21mLxbFhprZQ2a2KPl3SD3b2BVmNtrMHjWzl83sJTO7NIln9pyqhdxtLORuOlnPW6m5cpe8TS/rudtMeSs1d+7WdDBsZntK+qGkj0maKOlMM5tYyzZUwK2STtgpdpmkR0II4yU9knyfFdskzQohTJQ0RdJFyWeS5XOqOHK3IZG7u9EkeSs1V+6Styk0Se7equbJW6mJc7fWV4YnS3o1hLAkhPCupJ9LOrXGbShLCOFxSWt3Cp8q6bbk9W2STqtpo8oQQmgPIfwxed0pab6kUcrwOVUJudtgyN1UMp+3UnPlLnmbWuZzt5nyVmru3K31YHiUpBVF369MYlk3PITQnrxeJWl4PRvTXWY2VtIkSc+oSc6pgsjdBkbultSseSs1wedM3u5Ss+ZuU3zOzZa7FNBVWCg8qy5zz6szs/6S7pb0hRDCxuJlWT0ndE1WP2dyF1n8nMlbZPVzbsbcrfVg+DVJo4u+3zeJZd0bZjZSkpJ/V9e5PV1iZj1VSOw7Qgi/SsKZPqcqIHcbELm7W82at1KGP2fyNpVmzd1Mf87Nmru1Hgw/K2m8mY0zs16SPinpvhq3oRrukzQjeT1D0pw6tqVLzMwk3SRpfgjh+0WLMntOVULuNhhyN5VmzVspo58zeZtas+ZuZj/nps7dEEJNvySdKGmhpMWS/rHWx69A+38mqV3SVhXuYfqspL1VqKBcJOlhSUPr3c4unM9xKvxJ40+S5iVfJ2b5nKr4XpG7DfRF7qZ+nzKdt8k5NE3ukrddeq8ynbvNlLfJ+TRt7jIdMwAAAHKLAjoAAADkFoNhAAAA5BaDYQAAAOQWg2EAAADkFoNhAAAA5BaDYQAAAOQWg2EAAADk1v8HLgAFn4S3bUIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x216 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# learning images on the test set\n",
    "f, ((ax1, ax2, ax3, ax4)) = plt.subplots(1,4,  sharex='col', sharey='row',figsize=(12,3))\n",
    "ax1.imshow(np.reshape(image_test[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax1.set_title('True image')\n",
    "ax2.imshow(np.reshape(x_construction[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax2.set_title('Learned image')\n",
    "ax3.imshow(np.reshape(image_test[99,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax3.set_title('True image')\n",
    "ax4.imshow(np.reshape(x_construction[99,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax4.set_title('Learned image')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('loss', 140.17346005859375)]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate the ELBO which is minus the loss for test set\n",
    "metric = mx.metric.Loss()\n",
    "model.score(nd_iter_test, metric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. All together: MXNet-based class VAE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from VAE import VAE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can directly call the class `VAE` to do the training:\n",
    "\n",
    "```VAE(n_latent=5,num_hidden_ecoder=400,num_hidden_decoder=400,x_train=None,x_valid=None,\n",
    "batch_size=100,learning_rate=0.001,weight_decay=0.01,num_epoch=100,optimizer='sgd',model_prefix=None,\n",
    "initializer = mx.init.Normal(0.01),likelihood=Bernoulli)```\n",
    "\n",
    "The outputs are the learned model and training loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Epoch[0] Train-loss=383.478870\n",
      "INFO:root:Epoch[0] Time cost=5.075\n",
      "INFO:root:Epoch[1] Train-loss=211.923867\n",
      "INFO:root:Epoch[1] Time cost=4.741\n",
      "INFO:root:Epoch[2] Train-loss=206.789445\n",
      "INFO:root:Epoch[2] Time cost=4.601\n",
      "INFO:root:Epoch[3] Train-loss=204.428186\n",
      "INFO:root:Epoch[3] Time cost=4.865\n",
      "INFO:root:Epoch[4] Train-loss=202.417322\n",
      "INFO:root:Epoch[4] Time cost=4.606\n",
      "INFO:root:Epoch[5] Train-loss=200.635136\n",
      "INFO:root:Epoch[5] Time cost=4.711\n",
      "INFO:root:Epoch[6] Train-loss=199.009614\n",
      "INFO:root:Epoch[6] Time cost=5.159\n",
      "INFO:root:Epoch[7] Train-loss=197.565788\n",
      "INFO:root:Epoch[7] Time cost=4.588\n",
      "INFO:root:Epoch[8] Train-loss=196.524507\n",
      "INFO:root:Epoch[8] Time cost=4.905\n",
      "INFO:root:Epoch[9] Train-loss=195.725745\n",
      "INFO:root:Epoch[9] Time cost=4.426\n",
      "INFO:root:Epoch[10] Train-loss=194.902025\n",
      "INFO:root:Epoch[10] Time cost=4.685\n",
      "INFO:root:Epoch[11] Train-loss=194.026873\n",
      "INFO:root:Epoch[11] Time cost=4.622\n",
      "INFO:root:Epoch[12] Train-loss=193.350646\n",
      "INFO:root:Epoch[12] Time cost=4.712\n",
      "INFO:root:Epoch[13] Train-loss=192.737502\n",
      "INFO:root:Epoch[13] Time cost=4.618\n",
      "INFO:root:Epoch[14] Train-loss=192.338165\n",
      "INFO:root:Epoch[14] Time cost=4.763\n",
      "INFO:root:Epoch[15] Train-loss=191.888625\n",
      "INFO:root:Epoch[15] Time cost=5.168\n",
      "INFO:root:Epoch[16] Train-loss=191.170650\n",
      "INFO:root:Epoch[16] Time cost=4.809\n",
      "INFO:root:Epoch[17] Train-loss=190.307264\n",
      "INFO:root:Epoch[17] Time cost=4.622\n",
      "INFO:root:Epoch[18] Train-loss=188.988063\n",
      "INFO:root:Epoch[18] Time cost=4.543\n",
      "INFO:root:Epoch[19] Train-loss=187.616311\n",
      "INFO:root:Epoch[19] Time cost=5.154\n",
      "INFO:root:Epoch[20] Train-loss=186.352783\n",
      "INFO:root:Epoch[20] Time cost=4.661\n",
      "INFO:root:Epoch[21] Train-loss=185.428020\n",
      "INFO:root:Epoch[21] Time cost=5.193\n",
      "INFO:root:Epoch[22] Train-loss=184.543097\n",
      "INFO:root:Epoch[22] Time cost=4.519\n",
      "INFO:root:Epoch[23] Train-loss=184.029907\n",
      "INFO:root:Epoch[23] Time cost=4.732\n",
      "INFO:root:Epoch[24] Train-loss=183.643270\n",
      "INFO:root:Epoch[24] Time cost=5.011\n",
      "INFO:root:Epoch[25] Train-loss=183.246912\n",
      "INFO:root:Epoch[25] Time cost=4.706\n",
      "INFO:root:Epoch[26] Train-loss=183.065233\n",
      "INFO:root:Epoch[26] Time cost=4.673\n",
      "INFO:root:Epoch[27] Train-loss=182.680542\n",
      "INFO:root:Epoch[27] Time cost=4.628\n",
      "INFO:root:Epoch[28] Train-loss=182.428677\n",
      "INFO:root:Epoch[28] Time cost=4.772\n",
      "INFO:root:Epoch[29] Train-loss=182.219946\n",
      "INFO:root:Epoch[29] Time cost=4.571\n",
      "INFO:root:Epoch[30] Train-loss=182.070927\n",
      "INFO:root:Epoch[30] Time cost=4.603\n",
      "INFO:root:Epoch[31] Train-loss=181.837968\n",
      "INFO:root:Epoch[31] Time cost=4.559\n",
      "INFO:root:Epoch[32] Train-loss=181.624303\n",
      "INFO:root:Epoch[32] Time cost=5.069\n",
      "INFO:root:Epoch[33] Train-loss=181.534547\n",
      "INFO:root:Epoch[33] Time cost=4.654\n",
      "INFO:root:Epoch[34] Train-loss=181.239556\n",
      "INFO:root:Epoch[34] Time cost=4.776\n",
      "INFO:root:Epoch[35] Train-loss=181.098188\n",
      "INFO:root:Epoch[35] Time cost=4.571\n",
      "INFO:root:Epoch[36] Train-loss=180.820560\n",
      "INFO:root:Epoch[36] Time cost=4.815\n",
      "INFO:root:Epoch[37] Train-loss=180.828095\n",
      "INFO:root:Epoch[37] Time cost=4.455\n",
      "INFO:root:Epoch[38] Train-loss=180.495569\n",
      "INFO:root:Epoch[38] Time cost=5.096\n",
      "INFO:root:Epoch[39] Train-loss=180.389106\n",
      "INFO:root:Epoch[39] Time cost=4.797\n",
      "INFO:root:Epoch[40] Train-loss=180.200965\n",
      "INFO:root:Epoch[40] Time cost=5.054\n",
      "INFO:root:Epoch[41] Train-loss=179.851014\n",
      "INFO:root:Epoch[41] Time cost=4.642\n",
      "INFO:root:Epoch[42] Train-loss=179.719933\n",
      "INFO:root:Epoch[42] Time cost=4.603\n",
      "INFO:root:Epoch[43] Train-loss=179.431740\n",
      "INFO:root:Epoch[43] Time cost=4.341\n",
      "INFO:root:Epoch[44] Train-loss=179.235384\n",
      "INFO:root:Epoch[44] Time cost=4.638\n",
      "INFO:root:Epoch[45] Train-loss=179.108771\n",
      "INFO:root:Epoch[45] Time cost=4.754\n",
      "INFO:root:Epoch[46] Train-loss=178.714163\n",
      "INFO:root:Epoch[46] Time cost=4.457\n",
      "INFO:root:Epoch[47] Train-loss=178.508338\n",
      "INFO:root:Epoch[47] Time cost=4.960\n",
      "INFO:root:Epoch[48] Train-loss=178.288002\n",
      "INFO:root:Epoch[48] Time cost=4.562\n",
      "INFO:root:Epoch[49] Train-loss=178.083288\n",
      "INFO:root:Epoch[49] Time cost=4.619\n",
      "INFO:root:Epoch[50] Train-loss=177.791330\n",
      "INFO:root:Epoch[50] Time cost=4.580\n",
      "INFO:root:Epoch[51] Train-loss=177.570741\n",
      "INFO:root:Epoch[51] Time cost=4.704\n",
      "INFO:root:Epoch[52] Train-loss=177.287114\n",
      "INFO:root:Epoch[52] Time cost=5.172\n",
      "INFO:root:Epoch[53] Train-loss=177.122645\n",
      "INFO:root:Epoch[53] Time cost=4.678\n",
      "INFO:root:Epoch[54] Train-loss=176.816022\n",
      "INFO:root:Epoch[54] Time cost=4.819\n",
      "INFO:root:Epoch[55] Train-loss=176.670484\n",
      "INFO:root:Epoch[55] Time cost=4.568\n",
      "INFO:root:Epoch[56] Train-loss=176.459671\n",
      "INFO:root:Epoch[56] Time cost=4.450\n",
      "INFO:root:Epoch[57] Train-loss=176.174175\n",
      "INFO:root:Epoch[57] Time cost=4.579\n",
      "INFO:root:Epoch[58] Train-loss=175.935856\n",
      "INFO:root:Epoch[58] Time cost=4.552\n",
      "INFO:root:Epoch[59] Train-loss=175.739928\n",
      "INFO:root:Epoch[59] Time cost=4.385\n",
      "INFO:root:Epoch[60] Train-loss=175.579695\n",
      "INFO:root:Epoch[60] Time cost=4.496\n",
      "INFO:root:Epoch[61] Train-loss=175.403871\n",
      "INFO:root:Epoch[61] Time cost=5.088\n",
      "INFO:root:Epoch[62] Train-loss=175.157114\n",
      "INFO:root:Epoch[62] Time cost=4.628\n",
      "INFO:root:Epoch[63] Train-loss=174.953950\n",
      "INFO:root:Epoch[63] Time cost=4.826\n",
      "INFO:root:Epoch[64] Train-loss=174.743393\n",
      "INFO:root:Epoch[64] Time cost=4.832\n",
      "INFO:root:Epoch[65] Train-loss=174.554056\n",
      "INFO:root:Epoch[65] Time cost=4.375\n",
      "INFO:root:Epoch[66] Train-loss=174.366719\n",
      "INFO:root:Epoch[66] Time cost=4.583\n",
      "INFO:root:Epoch[67] Train-loss=174.160622\n",
      "INFO:root:Epoch[67] Time cost=4.586\n",
      "INFO:root:Epoch[68] Train-loss=173.981699\n",
      "INFO:root:Epoch[68] Time cost=5.149\n",
      "INFO:root:Epoch[69] Train-loss=173.751617\n",
      "INFO:root:Epoch[69] Time cost=4.495\n",
      "INFO:root:Epoch[70] Train-loss=173.548732\n",
      "INFO:root:Epoch[70] Time cost=4.588\n",
      "INFO:root:Epoch[71] Train-loss=173.380950\n",
      "INFO:root:Epoch[71] Time cost=5.042\n",
      "INFO:root:Epoch[72] Train-loss=173.158519\n",
      "INFO:root:Epoch[72] Time cost=4.817\n",
      "INFO:root:Epoch[73] Train-loss=172.970726\n",
      "INFO:root:Epoch[73] Time cost=4.791\n",
      "INFO:root:Epoch[74] Train-loss=172.782357\n",
      "INFO:root:Epoch[74] Time cost=4.377\n",
      "INFO:root:Epoch[75] Train-loss=172.581992\n",
      "INFO:root:Epoch[75] Time cost=4.518\n",
      "INFO:root:Epoch[76] Train-loss=172.385020\n",
      "INFO:root:Epoch[76] Time cost=4.863\n",
      "INFO:root:Epoch[77] Train-loss=172.198309\n",
      "INFO:root:Epoch[77] Time cost=5.104\n",
      "INFO:root:Epoch[78] Train-loss=172.022333\n",
      "INFO:root:Epoch[78] Time cost=4.571\n",
      "INFO:root:Epoch[79] Train-loss=171.816585\n",
      "INFO:root:Epoch[79] Time cost=4.557\n",
      "INFO:root:Epoch[80] Train-loss=171.643714\n",
      "INFO:root:Epoch[80] Time cost=4.567\n",
      "INFO:root:Epoch[81] Train-loss=171.460581\n",
      "INFO:root:Epoch[81] Time cost=4.735\n",
      "INFO:root:Epoch[82] Train-loss=171.284854\n",
      "INFO:root:Epoch[82] Time cost=5.012\n",
      "INFO:root:Epoch[83] Train-loss=171.113129\n",
      "INFO:root:Epoch[83] Time cost=4.877\n",
      "INFO:root:Epoch[84] Train-loss=170.947790\n",
      "INFO:root:Epoch[84] Time cost=4.487\n",
      "INFO:root:Epoch[85] Train-loss=170.766223\n",
      "INFO:root:Epoch[85] Time cost=4.723\n",
      "INFO:root:Epoch[86] Train-loss=170.602559\n",
      "INFO:root:Epoch[86] Time cost=4.803\n",
      "INFO:root:Epoch[87] Train-loss=170.448713\n",
      "INFO:root:Epoch[87] Time cost=4.636\n",
      "INFO:root:Epoch[88] Train-loss=170.273053\n",
      "INFO:root:Epoch[88] Time cost=4.562\n",
      "INFO:root:Epoch[89] Train-loss=170.099485\n",
      "INFO:root:Epoch[89] Time cost=4.567\n",
      "INFO:root:Epoch[90] Train-loss=169.934289\n",
      "INFO:root:Epoch[90] Time cost=4.905\n",
      "INFO:root:Epoch[91] Train-loss=169.768920\n",
      "INFO:root:Epoch[91] Time cost=4.636\n",
      "INFO:root:Epoch[92] Train-loss=169.620803\n",
      "INFO:root:Epoch[92] Time cost=4.429\n",
      "INFO:root:Epoch[93] Train-loss=169.448189\n",
      "INFO:root:Epoch[93] Time cost=4.985\n",
      "INFO:root:Epoch[94] Train-loss=169.295794\n",
      "INFO:root:Epoch[94] Time cost=4.649\n",
      "INFO:root:Epoch[95] Train-loss=169.143627\n",
      "INFO:root:Epoch[95] Time cost=4.602\n",
      "INFO:root:Epoch[96] Train-loss=168.989410\n",
      "INFO:root:Epoch[96] Time cost=4.904\n",
      "INFO:root:Epoch[97] Train-loss=168.841089\n",
      "INFO:root:Epoch[97] Time cost=4.602\n",
      "INFO:root:Epoch[98] Train-loss=168.694906\n",
      "INFO:root:Epoch[98] Time cost=4.589\n",
      "INFO:root:Epoch[99] Train-loss=168.527604\n",
      "INFO:root:Epoch[99] Time cost=4.560\n",
      "INFO:root:Epoch[100] Train-loss=168.385596\n",
      "INFO:root:Epoch[100] Time cost=4.835\n",
      "INFO:root:Epoch[101] Train-loss=168.246526\n",
      "INFO:root:Epoch[101] Time cost=4.558\n",
      "INFO:root:Epoch[102] Train-loss=168.093663\n",
      "INFO:root:Epoch[102] Time cost=4.609\n",
      "INFO:root:Epoch[103] Train-loss=167.938807\n",
      "INFO:root:Epoch[103] Time cost=4.599\n",
      "INFO:root:Epoch[104] Train-loss=167.814916\n",
      "INFO:root:Epoch[104] Time cost=4.394\n",
      "INFO:root:Epoch[105] Train-loss=167.676473\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Epoch[105] Time cost=4.724\n",
      "INFO:root:Epoch[106] Train-loss=167.560241\n",
      "INFO:root:Epoch[106] Time cost=4.316\n",
      "INFO:root:Epoch[107] Train-loss=167.424132\n",
      "INFO:root:Epoch[107] Time cost=4.646\n",
      "INFO:root:Epoch[108] Train-loss=167.284482\n",
      "INFO:root:Epoch[108] Time cost=4.472\n",
      "INFO:root:Epoch[109] Train-loss=167.184511\n",
      "INFO:root:Epoch[109] Time cost=4.768\n",
      "INFO:root:Epoch[110] Train-loss=167.037793\n",
      "INFO:root:Epoch[110] Time cost=4.717\n",
      "INFO:root:Epoch[111] Train-loss=166.916652\n",
      "INFO:root:Epoch[111] Time cost=4.803\n",
      "INFO:root:Epoch[112] Train-loss=166.796803\n",
      "INFO:root:Epoch[112] Time cost=4.617\n",
      "INFO:root:Epoch[113] Train-loss=166.655028\n",
      "INFO:root:Epoch[113] Time cost=4.420\n",
      "INFO:root:Epoch[114] Train-loss=166.561129\n",
      "INFO:root:Epoch[114] Time cost=4.333\n",
      "INFO:root:Epoch[115] Train-loss=166.434593\n",
      "INFO:root:Epoch[115] Time cost=4.526\n",
      "INFO:root:Epoch[116] Train-loss=166.322805\n",
      "INFO:root:Epoch[116] Time cost=4.310\n",
      "INFO:root:Epoch[117] Train-loss=166.195452\n",
      "INFO:root:Epoch[117] Time cost=4.458\n",
      "INFO:root:Epoch[118] Train-loss=166.073792\n",
      "INFO:root:Epoch[118] Time cost=4.333\n",
      "INFO:root:Epoch[119] Train-loss=165.967437\n",
      "INFO:root:Epoch[119] Time cost=4.459\n",
      "INFO:root:Epoch[120] Train-loss=165.876094\n",
      "INFO:root:Epoch[120] Time cost=5.070\n",
      "INFO:root:Epoch[121] Train-loss=165.748064\n",
      "INFO:root:Epoch[121] Time cost=4.782\n",
      "INFO:root:Epoch[122] Train-loss=165.656283\n",
      "INFO:root:Epoch[122] Time cost=4.640\n",
      "INFO:root:Epoch[123] Train-loss=165.540462\n",
      "INFO:root:Epoch[123] Time cost=4.522\n",
      "INFO:root:Epoch[124] Train-loss=165.448734\n",
      "INFO:root:Epoch[124] Time cost=4.858\n",
      "INFO:root:Epoch[125] Train-loss=165.347751\n",
      "INFO:root:Epoch[125] Time cost=4.842\n",
      "INFO:root:Epoch[126] Train-loss=165.230048\n",
      "INFO:root:Epoch[126] Time cost=4.495\n",
      "INFO:root:Epoch[127] Train-loss=165.147932\n",
      "INFO:root:Epoch[127] Time cost=4.766\n",
      "INFO:root:Epoch[128] Train-loss=165.036021\n",
      "INFO:root:Epoch[128] Time cost=4.526\n",
      "INFO:root:Epoch[129] Train-loss=164.977613\n",
      "INFO:root:Epoch[129] Time cost=5.091\n",
      "INFO:root:Epoch[130] Train-loss=164.881467\n",
      "INFO:root:Epoch[130] Time cost=5.223\n",
      "INFO:root:Epoch[131] Train-loss=164.785627\n",
      "INFO:root:Epoch[131] Time cost=4.165\n",
      "INFO:root:Epoch[132] Train-loss=164.707629\n",
      "INFO:root:Epoch[132] Time cost=4.527\n",
      "INFO:root:Epoch[133] Train-loss=164.598039\n",
      "INFO:root:Epoch[133] Time cost=4.167\n",
      "INFO:root:Epoch[134] Train-loss=164.502932\n",
      "INFO:root:Epoch[134] Time cost=4.354\n",
      "INFO:root:Epoch[135] Train-loss=164.422286\n",
      "INFO:root:Epoch[135] Time cost=4.387\n",
      "INFO:root:Epoch[136] Train-loss=164.344749\n",
      "INFO:root:Epoch[136] Time cost=4.662\n",
      "INFO:root:Epoch[137] Train-loss=164.264898\n",
      "INFO:root:Epoch[137] Time cost=4.671\n",
      "INFO:root:Epoch[138] Train-loss=164.178707\n",
      "INFO:root:Epoch[138] Time cost=4.776\n",
      "INFO:root:Epoch[139] Train-loss=164.109071\n",
      "INFO:root:Epoch[139] Time cost=4.787\n",
      "INFO:root:Epoch[140] Train-loss=163.993291\n",
      "INFO:root:Epoch[140] Time cost=4.726\n",
      "INFO:root:Epoch[141] Train-loss=163.956234\n",
      "INFO:root:Epoch[141] Time cost=4.337\n",
      "INFO:root:Epoch[142] Train-loss=163.845638\n",
      "INFO:root:Epoch[142] Time cost=4.787\n",
      "INFO:root:Epoch[143] Train-loss=163.790882\n",
      "INFO:root:Epoch[143] Time cost=5.563\n",
      "INFO:root:Epoch[144] Train-loss=163.723495\n",
      "INFO:root:Epoch[144] Time cost=4.529\n",
      "INFO:root:Epoch[145] Train-loss=163.634262\n",
      "INFO:root:Epoch[145] Time cost=5.028\n",
      "INFO:root:Epoch[146] Train-loss=163.552854\n",
      "INFO:root:Epoch[146] Time cost=4.933\n",
      "INFO:root:Epoch[147] Train-loss=163.501429\n",
      "INFO:root:Epoch[147] Time cost=4.912\n",
      "INFO:root:Epoch[148] Train-loss=163.444245\n",
      "INFO:root:Epoch[148] Time cost=5.034\n",
      "INFO:root:Epoch[149] Train-loss=163.348476\n",
      "INFO:root:Epoch[149] Time cost=4.600\n",
      "INFO:root:Epoch[150] Train-loss=163.256955\n",
      "INFO:root:Epoch[150] Time cost=4.704\n",
      "INFO:root:Epoch[151] Train-loss=163.216139\n",
      "INFO:root:Epoch[151] Time cost=4.670\n",
      "INFO:root:Epoch[152] Train-loss=163.144691\n",
      "INFO:root:Epoch[152] Time cost=4.678\n",
      "INFO:root:Epoch[153] Train-loss=163.050236\n",
      "INFO:root:Epoch[153] Time cost=4.595\n",
      "INFO:root:Epoch[154] Train-loss=162.991225\n",
      "INFO:root:Epoch[154] Time cost=5.307\n",
      "INFO:root:Epoch[155] Train-loss=162.907200\n",
      "INFO:root:Epoch[155] Time cost=4.684\n",
      "INFO:root:Epoch[156] Train-loss=162.838075\n",
      "INFO:root:Epoch[156] Time cost=4.686\n",
      "INFO:root:Epoch[157] Train-loss=162.759286\n",
      "INFO:root:Epoch[157] Time cost=4.750\n",
      "INFO:root:Epoch[158] Train-loss=162.725998\n",
      "INFO:root:Epoch[158] Time cost=4.637\n",
      "INFO:root:Epoch[159] Train-loss=162.635852\n",
      "INFO:root:Epoch[159] Time cost=4.498\n",
      "INFO:root:Epoch[160] Train-loss=162.563777\n",
      "INFO:root:Epoch[160] Time cost=5.048\n",
      "INFO:root:Epoch[161] Train-loss=162.527387\n",
      "INFO:root:Epoch[161] Time cost=5.040\n",
      "INFO:root:Epoch[162] Train-loss=162.395881\n",
      "INFO:root:Epoch[162] Time cost=4.764\n",
      "INFO:root:Epoch[163] Train-loss=162.353654\n",
      "INFO:root:Epoch[163] Time cost=4.561\n",
      "INFO:root:Epoch[164] Train-loss=162.285584\n",
      "INFO:root:Epoch[164] Time cost=5.051\n",
      "INFO:root:Epoch[165] Train-loss=162.204332\n",
      "INFO:root:Epoch[165] Time cost=4.455\n",
      "INFO:root:Epoch[166] Train-loss=162.147100\n",
      "INFO:root:Epoch[166] Time cost=5.021\n",
      "INFO:root:Epoch[167] Train-loss=162.051296\n",
      "INFO:root:Epoch[167] Time cost=4.551\n",
      "INFO:root:Epoch[168] Train-loss=161.978708\n",
      "INFO:root:Epoch[168] Time cost=4.744\n",
      "INFO:root:Epoch[169] Train-loss=161.927990\n",
      "INFO:root:Epoch[169] Time cost=4.821\n",
      "INFO:root:Epoch[170] Train-loss=161.883088\n",
      "INFO:root:Epoch[170] Time cost=4.365\n",
      "INFO:root:Epoch[171] Train-loss=161.785367\n",
      "INFO:root:Epoch[171] Time cost=4.448\n",
      "INFO:root:Epoch[172] Train-loss=161.716386\n",
      "INFO:root:Epoch[172] Time cost=4.622\n",
      "INFO:root:Epoch[173] Train-loss=161.656391\n",
      "INFO:root:Epoch[173] Time cost=4.500\n",
      "INFO:root:Epoch[174] Train-loss=161.598127\n",
      "INFO:root:Epoch[174] Time cost=4.677\n",
      "INFO:root:Epoch[175] Train-loss=161.518613\n",
      "INFO:root:Epoch[175] Time cost=4.958\n",
      "INFO:root:Epoch[176] Train-loss=161.418783\n",
      "INFO:root:Epoch[176] Time cost=4.607\n",
      "INFO:root:Epoch[177] Train-loss=161.407767\n",
      "INFO:root:Epoch[177] Time cost=4.427\n",
      "INFO:root:Epoch[178] Train-loss=161.319552\n",
      "INFO:root:Epoch[178] Time cost=4.930\n",
      "INFO:root:Epoch[179] Train-loss=161.234087\n",
      "INFO:root:Epoch[179] Time cost=4.240\n",
      "INFO:root:Epoch[180] Train-loss=161.187404\n",
      "INFO:root:Epoch[180] Time cost=4.484\n",
      "INFO:root:Epoch[181] Train-loss=161.123118\n",
      "INFO:root:Epoch[181] Time cost=4.937\n",
      "INFO:root:Epoch[182] Train-loss=160.999420\n",
      "INFO:root:Epoch[182] Time cost=4.489\n",
      "INFO:root:Epoch[183] Train-loss=160.955369\n",
      "INFO:root:Epoch[183] Time cost=4.894\n",
      "INFO:root:Epoch[184] Train-loss=160.908542\n",
      "INFO:root:Epoch[184] Time cost=4.269\n",
      "INFO:root:Epoch[185] Train-loss=160.846908\n",
      "INFO:root:Epoch[185] Time cost=4.998\n",
      "INFO:root:Epoch[186] Train-loss=160.765964\n",
      "INFO:root:Epoch[186] Time cost=4.467\n",
      "INFO:root:Epoch[187] Train-loss=160.687773\n",
      "INFO:root:Epoch[187] Time cost=4.609\n",
      "INFO:root:Epoch[188] Train-loss=160.652674\n",
      "INFO:root:Epoch[188] Time cost=5.327\n",
      "INFO:root:Epoch[189] Train-loss=160.551175\n",
      "INFO:root:Epoch[189] Time cost=4.267\n",
      "INFO:root:Epoch[190] Train-loss=160.477424\n",
      "INFO:root:Epoch[190] Time cost=4.798\n",
      "INFO:root:Epoch[191] Train-loss=160.501221\n",
      "INFO:root:Epoch[191] Time cost=4.695\n",
      "INFO:root:Epoch[192] Train-loss=160.370335\n",
      "INFO:root:Epoch[192] Time cost=4.640\n",
      "INFO:root:Epoch[193] Train-loss=160.279749\n",
      "INFO:root:Epoch[193] Time cost=4.653\n",
      "INFO:root:Epoch[194] Train-loss=160.242415\n",
      "INFO:root:Epoch[194] Time cost=5.044\n",
      "INFO:root:Epoch[195] Train-loss=160.197063\n",
      "INFO:root:Epoch[195] Time cost=4.684\n",
      "INFO:root:Epoch[196] Train-loss=160.132983\n",
      "INFO:root:Epoch[196] Time cost=4.460\n",
      "INFO:root:Epoch[197] Train-loss=160.083149\n",
      "INFO:root:Epoch[197] Time cost=4.713\n",
      "INFO:root:Epoch[198] Train-loss=160.025012\n",
      "INFO:root:Epoch[198] Time cost=4.779\n",
      "INFO:root:Epoch[199] Train-loss=159.945513\n",
      "INFO:root:Epoch[199] Time cost=4.659\n"
     ]
    }
   ],
   "source": [
    "# can initilize weights and biases with the learned parameters as follows: \n",
    "# init = mx.initializer.Load(params)\n",
    "\n",
    "# call the VAE, output model contains the learned model and training loss\n",
    "out = VAE(n_latent=2, x_train=image, x_valid=None, num_epoch=200) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# encode test images to obtain mu and logvar which are used for sampling\n",
    "[mu,logvar] = VAE.encoder(out,image_test)\n",
    "# sample in the latent space\n",
    "z = VAE.sampler(mu,logvar)\n",
    "# decode from the latent space to obtain reconstructed images\n",
    "x_construction = VAE.decoder(out,z)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADACAYAAADhh27FAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xm0XFWZ9/HfQxJCyETm4ZLkQgiahOYNcknC8NpR0jSDLMAWFbUNk9AuURHpBmmlozRL+qVB8JWlQAeChrZBZQgsWhtBRpkueYMEAoQkF5NwM0IGQshA9vtHnWh591Pk3NSt4dT5fta6K3WfOsM+Vfvu2jl1nvNYCEEAAABAHu1V6wYAAAAAtcJkGAAAALnFZBgAAAC5xWQYAAAAucVkGAAAALnFZBgAAAC5xWS4DpjZf5jZZbVuB1AOM2s2s2Bm3Us8/5KZTatyswAX4y6yjjG36xj3GZbM7J2iX/eVtFXS+8nv54cQbq9+q5BnZtYm6dwQwm9r3Za0zKxZ0lJJPUIIO2rbGtQ7xl3UE8bcfHP/N5E3IYQ+ux6n+YMws+50PGQd/Ri1xLiLvKEP1y8uk0jBzP7VzO4ws5+b2SZJXzCzOWY2s2iZ6cmAvuv3/c3sbjNbY2ZLzewrH7D9P21r13bM7FvJum+a2clm9gkzW2Rmb5nZPxWte6SZPW1m682s3cx+aGY9ip4/wcxeM7MNZvZ/zexJMzuz6PlzzewVM3vbzP7bzEZ10cuGCkn6wvzkPf+9mR1a9NylZrbYzDaZ2ctmdlrRc2cm7/8PzGydpJlJ7Akz+/ekDyw1sxOK1ulvZrOSvrUi+VvoljzXLVlvrZktkXTSbtrdZmbTk8czzewXSd/fZGYvmtnBSb9fbWbLzOy4onXPMrOFybJLzOz8Dtv+p6SNbyZ9OpjZQclzPZN2/tHMVpnZT8ysV3nvAiqNcRf1gjG38cdcJsPpnSbpPyX1l3THBy1oZntJul/Sc5KaJP2NpH80s2NT7mt/Fd6bkZKukDRL0mclHSZpmqTvmdnoZNkdkr4uabCkoyUdL+n8pB1DJd0p6R+T55dKmlzUzr9LnjtF0hBJzyTHiDplZodJukWF93iQpBslzTWznskiiyX9bxX66XclzTGzEUWbmCJpiaRhkq4sir2qQh/5P5JmmZklz81WoY8dpEL/O07SuclzX5L0iSTeIulTnTyckyX9TNIASf9P0m9U6PdNkr6XHNsuq5N99ZN0lqQfmNlHktfkeEkXSZqetHNah/1cJelgSZOS55skXd7JtqI2GHdRU4y5ORlzQwj8FP1IapM0vUPsXyU93CE2R9LMot+nS2pLHh8taUmH5b8j6eYS+/zTtpLtvCOpW/L7AElB0uFFy78g6RMltnWxpF8kj8+W9HjRcyapXdKZye8PSppR9Hx3Fa7ba6r1+5D3H68fJvEfS7qiQ+xVSX9dYjvzJZ2SPD5T0h87PH+mpNeLft836W/DVRi8t0rqVfT8GZJ+lzx+WNI/FD13XLJu990dk6SZkh4seu7kDv2+b7Kt/Ups6x5JX08e3yLp+0XPHZSse1DS5zdLGlv0/JGSltb6Pebng/s74y4/te6DSZwxNzT+mMs1w+kt68SyYySNNrP1RbFukh5Juf7aEMKuRJItyb+rip7fIqmPJJnZhyVdI+lwFf6ouqtwpkEqnOH4U7tDCMHMlndo5w1mdn1RbKcKZ0hWpGwrqmuMpBlm9tWi2N4qvNcysy+q8D/25uS5PiqcfdjF68crdz0IIbybnKDoI2mgpB6S2v980kJ7FW3jL/qXpDc6eSwd+7TX7/tIWp98jfgvKpxt2EuFvv5iUTtai7ZV3KYhybLPFx2DqfD3iPrHuItaY8zNwZjLZDi9jrfd2KzCG77L8KLHyyQtCiGMr3irCl9rPC3pMyGEd8zsYhW+2pAKZyOKrwEyFb6uKG7nd0IIH/j1I+rKMklXhhCu7PiEmY2RdLOkYyU9FUJ438zmqzAQ7dKZ28csU+EsxeDgJ320Syq+1nG0s0zZkq8jfyXpi5LuDSFsN7N79OfjaldhIrFLcZvWqjDITwwhMNHIHsZd1Bpjbg7GXK4Z3nPzJZ1kZgOS64O+VvTcU5K2mdk3zWyf5KL3vzKzwyvQjr6SNkjabGbjlVy3lrhf0keskAjSXYVr3IYUPf8TSf+crCcz28/MOnsNEiqnR9J/dv10V2Hg/Qczm2IFvc3sJDPrK6m3CgPvGqmQACHpkD3deQihXdL/SLrGzPqZ2V5mNtbM/jpZ5E5JX7NC0tIASZeWcawfZG9JPVU4rh3JGYvjip6/U9JZZjbezPZV4avxXcewU4XX7AfJtZwysyYz+9sKtRWVxbiLSmLMLcjdmMtkeM/NlrRQha8pfi3pv3Y9kfyP7kQVkibaVPif0o0qXIje1b4paYakTck+/nS2IYSwStJnJF0raZ2ksSpcNL81ef4XyXO/MLONkv4gqa47bM48oML/sHf9zAwhtKqQRPEjSW9Lel2Fa9AUQnhZha9un1Lh67C/kvRkmW34ogoD48vJ/n4paVdyyM0qJGC8IGmepLvK3JcrhLBJhUnPnUkbPidpbtHz/y3ph5J+p8Lr8XTy1Nbk30t2xZN+/ltJH6pEW1Fxs8W4i8phzFU+x1yKbuSIFW7P8qakT4UQHq91e4BKSM64LZDUs8RXjUDVMO6i0TXCmMuZ4QZnZscnX8P1VOGrjO2Snq1xs4AuZWanWeHelgMk/Zuk+7I6KCP7GHfR6BptzGUy3PiOUeEeh2tU+CrutBDC1g9eBcic81W4L+ZiFUr6frm2zUHOMe6i0TXUmMtlEgAAAMgtzgwDAAAgt8q6z7AVSvJdr8LNlP8jhHDVBy0/ePDg0NzcXM4uAT3//PNrQwhDdr9k16HvolxtbW1au3at7X7JrkO/RVdgzEVWpe27ezwZTjJkb1Ch/vtySc+Z2dzkViOu5uZmtba2lnoaSMXMOlt1p2z0XZSrpaWl6vuk36IrMOYiq9L23XIuk5isQn3tJSGEbSrc7/GUMrYHAAAAVFU5k+Em/WU96uX6y5KTkiQzO8/MWs2sdc2aNWXsDqgu+i6yiH6LrKLvolYqnkAXQrgphNASQmgZMqSqlxwBZaHvIovot8gq+i5qpZzJ8ApJo4p+3z+JAQAAAJlQzmT4OUnjzOwAM9tb0mdVVLsaAAAAqHd7fDeJEMIOM7tA0m9UuLXaLSGEl7qsZQAAAECFlXWf4RDCA5Ie6KK2AAAAAFVFBToAAADkVllnhgEAAFB9IYQu36ZZVYtk1g3ODAMAACC3mAwDAAAgt5gMAwAAILeYDAMAACC3mAwDAAAgt7ibBAAAQJ1Ie5cIbzkvVq07RGT5ThScGQYAAEBuMRkGAABAbjEZBgAAQG4xGQYAAEBukUAHIFMqUYLUk+VkEADVkXY82rlzZxR7//333WV37NiRallvm9641b27P9Xba6/4fGi3bt1SbTNtrJRy1+9qnBkGAABAbjEZBgAAQG4xGQYAAEBuMRkGAABAbpWVQGdmbZI2SXpf0o4QQktXNApA/pRTdamc5aT0iRsk1WXbhg0botjNN98cxebMmRPFXnjhBXebvXv3jmILFy6MYqNGjUrTRNSBtJXdJD+JzUuAe/fdd6NYe3u7u82XX345inl9av369VGsR48eUWzw4MHufpqbm6PYmDFjUq3ft2/fKNarVy93P14CX9rkPW85qevH4q64m8THQghru2A7AAAAQFVxmQQAAAByq9zJcJD0P2b2vJmd5y1gZueZWauZta5Zs6bM3QHVQ99FFtFvkVX0XdRKuZPhY0IIH5F0gqSvmNlHOy4QQrgphNASQmgZMmRImbsDqoe+iyyi3yKr6LuolbKuGQ4hrEj+XW1md0uaLOmxrmgYkEflVlerRHJZ2mQSL5Ek7XKd2Y+XOOElWZRKsPCWTRsrhcS62inVn7yEt/POi7/AnDdvXqr9lOoPW7ZsiWLLli2LYiTQ1ae0406panHbt2+PYhs3boxiixYtimIPPPCAu81HHnkkii1dujTVvvv06RPF+vfv7+5n7NixUezggw+OYhMnToxiH/rQh6LYyJEj3f3069cvinmJfp5SY2tXj7l7fGbYzHqbWd9djyUdJ2lBVzUMAAAAqLRyzgwPk3R3MjvvLuk/Qwi/7pJWAQAAAFWwx5PhEMISSf+rC9sCAAAAVBW3VgMAAEBudUXRjUx4+umno9j111/vLtvU1BTFvMoqM2bMiGIDBw50t1kqjnwoJwnNq2a0bds2dz9bt26NYu+9916q2ObNm1Nv00sa8ioseTFve6W26SWIeIkTw4cPj2Jecojk/33vu+++UcxL8CBRrra8222deOKJ7rJpE+PSKpVA5/3Ntra2RrGjjjoqinl/h9ddd527n1tvvTWKvfTSS1HMq/aFgnISjEslanpj1Lp166LY888/H8UWLPDTrFavXh3FvDmEVy3OS6Ar9Xnh9Wmvqt2bb74ZxYYOHRrFBg0a5O6n1GuXRrlJ5WlxZhgAAAC5xWQYAAAAucVkGAAAALnFZBgAAAC5xWQYAAAAuZWbtFPvzg9eecTOuPLKK6NYqbKHU6dOLWtf1dDc3BzFvvWtb7nLjh49usKtyaZSma9py3t6Wb8bNmyIYl5WveRnIb/++utR7I033ohiK1eudLe5atWqKOZlHHsZwz179oxi++yzj7sf7zXy9uM56KCDotjRRx/tLjt9+vQo5pXKTVsuFJWxadOmKHbEEUdEMa/0cSn77bdfFDvnnHOi2LRp06KYV35Wkm644YYo1tLSEsUWL14cxc4666wo9uSTT7r78Tz22GNR7OMf/3jq9VE+bxz3+u4777wTxXr37u1u0+vnH/7wh6NYqT7ZkTfeS/5dL7w75nh3KPHGx1J3MunWrVsU8+5kUcu79XBmGAAAALnFZBgAAAC5xWQYAAAAucVkGAAAALmVmwS6e+65J4rNnz/fXXbixIlRzCt7+cwzz0Sxe++9193mb37zmyh2wAEHRLGlS5e666flXcA+YsSIKJY26cRLqpOkSy65pFPtyru05T290p5vv/12FHvllVfc/XjlPRcuXBjFXnvttSjmJd9JfvlkLwmuX79+UWzYsGFRrG/fvu5+vLLTS5YsiWJeUp33uh144IHufkqVJu2I0svV4/19XHHFFVHMG7e85BxJOvPMM6PYhRdeGMUmTJiQooWlXX311VHMG+8vuOCCKPbCCy+k3s/pp58exbxEP5Tm/U17fc9brtR4kDYR2tuPV2JZ8hN6jzzyyCg2ZMiQKOYl6nljeKm497nk3RjAG++9svaSPy+pt6Q6zgwDAAAgt5gMAwAAILeYDAMAACC3mAwDAAAgt3abQGdmt0j6hKTVIYRDkthASXdIapbUJunTIYQ4y6eOjB8/PlWslEMPPTSKnXHGGVHsqquuctdva2uLYl4CnZcw1Bl77713FPMS6Lx9e1XNvKo3qBwvWcBLKvCSzSTp3XffjWJegseAAQOi2ODBg91teklwI0eOjGJesuXQoUOjWKkKdF6lPK8qnlcRz0ugK7UfL8nDS+boTBINyvPtb387il1zzTVRzHufvGQ1SfrYxz5WfsOKeIlFknTrrbdGMS9Rb8uWLVHM+5u78cYb3f2cfPLJUcwbG9A5lfib9vqKl0DXp08fd33vM9tLYvOqwHmfDRs3bnT3431e9OrVK4oNHz48ig0aNCjVupKfQJc2Wa6eEuhmSzq+Q+xSSQ+FEMZJeij5HQAAAMiU3U6GQwiPSXqrQ/gUSbclj2+TdGoXtwsAAACouD39jmVYCKE9ebxSUvw9asLMzjOzVjNr9b6GB+oVfRdZRL9FVtF3UStlX3AUChfCxBfD/Pn5m0IILSGEFu8G0UC9ou8ii+i3yCr6LmplTyvQrTKzESGEdjMbIckvXZVDpZJ20iaidSapLy2vUt7atWuj2JQpU6LYcccd1+XtyaO0iXFeokHv3r2jmJdgIUlNTU2ptuklkXnrStLBBx+calmvIpGX8ORVSCoV9xI/vOQUL5lj3Lhx7n68RBTvNUL1/PKXv0y1nFdVrqsT5SS/6uPZZ5/tLjt37txU2/Qmd88991wU86qPofZKJXKlraTmxbykOknatGlTFHvrrY5Xq/oJcIsWLYpib7zxhrsfbyz1EqG9PulVz+vZs6e7H298TZsYV08JdJ65kmYkj2dI8msQAwAAAHVst5NhM/u5pKckfcjMlpvZOZKukvQ3ZrZI0vTkdwAAACBTdvvdYAghvpluwbFd3BYAAACgqrhjNwAAAHKLyTAAAAByixTqBrN58+Yodtppp0UxL4v0uuuui2Klyiuic7yMWC8L2cu69d4D744IkjRmzJgoNnr06CjmZQd7pZMlv3Szd9cU7xi9UtBeprQkzZs3L4otX748inklxw855JAoNmHCBHc/3t050pZjRm194Qtf6PJteiW/r7jiiiiW9q4RknTCCSdEsR/+8IdRjDtHZEdn7ibhjeOdubPOggULoph3hxNvLFy2bFkUK1WOeezYsVHMG0u9uwd5n0ul7sqThdL2nBkGAABAbjEZBgAAQG4xGQYAAEBuMRkGAABAbpFA12Bmz54dxbwEEa98rZd8hc4plRRQquxmR2lLNPft29dd30vI6dOnTxTz3n8vsazU/r3j9I5x27ZtUWz+/Pnufu65554o5iWYeCXLp06dGsX2339/dz9e0gmy4Y477ohiRx55pLtsjx49opiX0HnJJZdEsTlz5qRuk5dIdPnll0exAw88MPU2UX86k0Dnlbv3xlcv4V2SlixZEsW8MsveWOaNmV4pcMkvqTxy5Mgo5n2GlFNiuR5xZhgAAAC5xWQYAAAAucVkGAAAALnFZBgAAAC5RQJdRi1evNiNX3TRRanWf+qpp6LY8OHDy2oTOidtsoGXoOElNEhSz549o5iX4JO2gpzkJx15y+7YsSOKrVixIordfvvt7n7a2tqimJcgMm3atCjmJVF5SSySn6SI2jrggAOi2Ouvvx7FfvKTn0SxF1980d3mT3/60yh29dVXR7G0yXKlKjQ+++yzUYzKcvnhjc9egrOXxFYqmddLglu/fn0US/sZUqqSrDcWep8h3nJZTpbz8KkAAACA3GIyDAAAgNxiMgwAAIDcYjIMAACA3NptAp2Z3SLpE5JWhxAOSWIzJX1J0ppksctCCA9UqpGI3XfffW58+/btUez000+PYlRDqq60Fds8XqWfUslh3jbTJj/s3LnT3WbaZTdt2hTFHn744Sj26KOPuvvxtulVm/vUpz4VxUaMGBHFvApkUuMlfjSCO++8M4oNGDAg1bpPPvmkGx87duwet+fcc8+NYldddZW7bNp2IttKjRve+OolOHtjlFcJVPIrw3m8MdNLZN6yZYu7/rp161It632ueLEsj61pzgzPlnS8E/9BCGFS8sNEGAAAAJmz28lwCOExSW9VoS0AAABAVZVzzfAFZvYHM7vFzEp+T2Rm55lZq5m1rlmzptRiQN2h7yKL6LfIKvouamVPJ8M/ljRW0iRJ7ZKuKbVgCOGmEEJLCKHFu+k0UK/ou8gi+i2yir6LWtmjCnQhhFW7HpvZzZLu77IWIeIlxd19993usl71mO9///tRzKuag9rzkjG896pU5aK0SXmeziQ/bNu2LYotXbo0is2dOzeKedWVJL8CopfIdPjhh0cxr8JSZyrNZTnxoxF4Fbu8KpmXXHJJFHviiSfcbZZKCE3j1FNPjWIkyuVbqbHVGzu8Cp/9+/ePYqWqGo4bNy6KeZ8DXnXQP/7xj1HMq14nSStXroxi3hn5ziQoZ9UenRk2s+JX5jRJC7qmOQAAAED1pLm12s8lTZM02MyWS/oXSdPMbJKkIKlN0vkVbCMAAABQEbudDIcQznDCsyrQFgAAAKCqqEAHAACA3GIyDAAAgNzao7tJoLpmzYqvSnn88cfdZT/3uc9FMUov1ycvC9mLeeWYS90poZwM+lK8jGUv49i7c8S8efOiWKlS0p/85Cej2Gc+85ko5pU65e4o2eb1+8mTJ0ex2bNnR7GJEye62yxVgjaNyy+/PIq1tLS4y3ILsGxLW2q4lHLG8f3228/dpnc3iZEjR0Yxb2x+7rnnotjChQvd/bz33ntRbMOGDVGsEp8r9YYzwwAAAMgtJsMAAADILSbDAAAAyC0mwwAAAMgtEujqzPz586PYV7/61ShW6sL7733ve13eJlSPlxjnJXOUKh9cTiKZl4wh+aU8vVK59913X6r9TJkyxY1/+ctfjmJeCVyS5fLLKx9bKlHu5JNPjmJeCdm77rorinmJn4cccoi7H6/8bc+ePd1l0XjSJuB546uXDCxJTU1NUWzUqFFRbMeOHVFs7dq1UWz16tXufrzPm+3bt0exziQUZhVnhgEAAJBbTIYBAACQW0yGAQAAkFtMhgEAAJBbJNDVkJf4ccYZZ0Qx78L7z3/+8+42qTbXeNJWOOoMLyGiVCLSq6++GsXuvffeKOYlN40ZMyaKnX766e5+vL5Lslx+vfPOO1Hs7LPPjmK9e/d21/cS4zzf+MY3otiPfvSjKOYlJkl+da9Jkyal2jfqU2eq0qVNlvOS3QYOHOhu06tq6CUTb968OYrtvffeqWKlpD32Rkuq48wwAAAAcovJMAAAAHKLyTAAAAByi8kwAAAAcmu3CXRmNkrSTyUNkxQk3RRCuN7MBkq6Q1KzpDZJnw4hvF25pmbbzp07o9hJJ50UxbxkpfHjx0ex7373u13TMOSSlyzX1tbmLnv//fdHsaeffjqKeQkeH/3oR6PYtGnT3P10JskDjc9LoHvllVeiWKkEOq+6lufaa6+NYocffngUO+uss9z1L7rooijmVWMs1U5UTznJYd5nuOQnxm3YsCGKvfvuu1GsVKXCffbZJ4p5SXlvvfVWFGtvb0/VHslP4POqNJabsJ0FaUaLHZK+GUKYIGmqpK+Y2QRJl0p6KIQwTtJDye8AAABAZux2MhxCaA8hzEseb5K0UFKTpFMk3ZYsdpukUyvVSAAAAKASOnXNsJk1SzpM0jOShoUQdp2PX6nCZRTeOueZWauZta5Zs6aMpgLVRd9FFtFvkVX0XdRK6smwmfWR9CtJF4YQNhY/FwoX2Lh3YA4h3BRCaAkhtHg3kgbqFX0XWUS/RVbRd1ErqSrQmVkPFSbCt4cQdpX1WWVmI0II7WY2QtLqSjWyEXgXuj/yyCOp1v3Zz34WxUpVrgE62rp1axTzqsX9/ve/d9f34l6y22GHHRbFTjnllChW6kMubcIT8sFLOBs9enQUW7Vqlbu+d2bR63telcNhw9wvOl0vv/xyqm0iO7xkue3bt7vLrl+/PoqtWLEiVaxU0rC3fy8Bz0tkfuqpp6KYN/+QpKFDh0axfv36RbE8jM27PUIrpBHOkrQwhFCcdjtX0ozk8QxJcX1WAAAAoI6lOTN8tKS/l/Simc1PYpdJukrSnWZ2jqQ3JH26Mk0EAAAAKmO3k+EQwhOSSt1k7tiubQ4AAABQPY1/IQgAAABQApNhAAAA5Faqu0kgvVJlD6dOnZpq/Tlz5kQxL0sf8HilQd9+O66S7mXAP/vss+42N27cGMUOPPDAKDZ58uQo5pUS98p9Svko+Yn0+vbtG8UuvvjiKPa1r33NXX/kyJFRzMuKP/roo6PYY489lqaJkvw7VHjldJFtpe4msW7duij22muvRbEFCxakWleSNm/eHMWWL18exZYuXZpq3TFjxrj7aWpqimLenVTKLdGchbGdM8MAAADILSbDAAAAyC0mwwAAAMgtJsMAAADILRLoutitt97qxpcsWZJq/WOOOSaKZeHic1SXV65T8pMnli1bFsXmz58fxdrb291tDhgwIIp5CXRTpkyJYl4SVGdKe9L3UWzGjBlRbN68ee6ys2fPjmLe382jjz5aVpt+/etfl7U+sqHUWOSVVPYSzrZu3RrFvERmSVq8eHEU8xKhvbHUK1k+bdo0dz/Tp0+PYl4CXffu8VTR23eWx2vODAMAACC3mAwDAAAgt5gMAwAAILeYDAMAACC3SKArw6JFi6LYzJkzq98QNDQv6cdLxpCk9evXR7GVK1dGMa/y0b777utu06uwNXHixCjmVTPyEi86I4QQxbKcpIHy9OnTJ4rdeOON7rLNzc1RbNasWVHMSzD1+vLDDz/s7mfEiBFuHPUn7djhJYd5iXKSNHTo0Ch2xBFHRLHBgwdHMa9KouQn0HljvrfNQw89NIp5yc2Sn2znfQ5069YtijXaOMyZYQAAAOQWk2EAAADkFpNhAAAA5BaTYQAAAOTWbrNbzGyUpJ9KGiYpSLophHC9mc2U9CVJa5JFLwshPFCphtajxx9/PIpt3Lgx9frjx4+PYr169SqrTcg2L2HMS6DbsWOHu74X9xI/vMSNnj17utscOHBgFJs0aVIU86rNeUkW3jGWindm/TTrovGUStL8zne+kyqGfPPGCS9hzItJ/rjZv3//KHbQQQdFsWOPPdbdpjfmpx33vOS/Um3PQ2JcWmlSvXdI+mYIYZ6Z9ZX0vJk9mDz3gxDCv1eueQAAAEDl7HYyHEJol9SePN5kZgslxfedAQAAADKmU9cMm1mzpMMkPZOELjCzP5jZLWY2oMQ655lZq5m1rlmzxlsEqEv0XWQR/RZZRd9FraSeDJtZH0m/knRhCGGjpB9LGitpkgpnjq/x1gsh3BRCaAkhtHg37wfqFX0XWUS/RVbRd1ErqSbDZtZDhYnw7SGEuyQphLAqhPB+CGGnpJslTa5cMwEAAICul+ZuEiZplqSFIYRri+IjkuuJJek0SQsq08TGcNRRR0WxBx98MIpxNwl05GX39ujRw1120KBBqZb1SoiWuhOKly3trb/PPvu461dDXjOgAXS9zownjD2NIc3dJI6W9PeSXjSz+UnsMklnmNkkFW631ibp/Iq0EAAAAKiQNHeTeEKS91+fXN1TGAAAAI2HCnQAAADILSbDAAAAyK001wyjhLPPPjtVDEirWqVB999//yiWttxnZ3jHU27CCQkrAICuxJlhAAAA5BaTYQAAAOQWk2EAAADkFpNhAAAA5JZVImmm5M7M1kh6I/l1sKS1Vdt55XGxTVv2AAADPUlEQVQ81TMmhFDVwvVFfbeeX5c9wfFUTy37rVTfr82e4Hiqh77btTie6knVd6s6Gf6LHZu1hhBaarLzCuB48qHRXheOJz8a7bXhePKj0V4bjqf+cJkEAAAAcovJMAAAAHKrlpPhm2q470rgePKh0V4Xjic/Gu214Xjyo9FeG46nztTsmmEAAACg1rhMAgAAALnFZBgAAAC5VfXJsJkdb2avmtnrZnZptfdfLjO7xcxWm9mCothAM3vQzBYl/w6oZRs7w8xGmdnvzOxlM3vJzL6exDN7TJVC360v9N10st5vpcbqu/Tb9LLedxup30qN3XerOhk2s26SbpB0gqQJks4wswnVbEMXmC3p+A6xSyU9FEIYJ+mh5Pes2CHpmyGECZKmSvpK8p5k+Zi6HH23LtF3d6NB+q3UWH2XfptCg/Td2Wqcfis1cN+t9pnhyZJeDyEsCSFsk/Rfkk6pchvKEkJ4TNJbHcKnSLoteXybpFOr2qgyhBDaQwjzksebJC2U1KQMH1OF0HfrDH03lcz3W6mx+i79NrXM991G6rdSY/fdak+GmyQtK/p9eRLLumEhhPbk8UpJw2rZmD1lZs2SDpP0jBrkmLoQfbeO0XdLatR+KzXA+0y//UCN2ncb4n1utL5LAl0XC4V71WXufnVm1kfSryRdGELYWPxcVo8JnZPV95m+iyy+z/RbZPV9bsS+W+3J8ApJo4p+3z+JZd0qMxshScm/q2vcnk4xsx4qdOzbQwh3JeFMH1MF0HfrEH13txq130oZfp/pt6k0at/N9PvcqH232pPh5ySNM7MDzGxvSZ+VNLfKbaiEuZJmJI9nSLq3hm3pFDMzSbMkLQwhXFv0VGaPqULou3WGvptKo/ZbKaPvM/02tUbtu5l9nxu674YQqvoj6URJr0laLOmfq73/Lmj/zyW1S9quwjVM50gapEIG5SJJv5U0sNbt7MTxHKPCVxp/kDQ/+Tkxy8dUwdeKvltHP/Td1K9TpvttcgwN03fpt516rTLddxup3ybH07B9l3LMAAAAyC0S6AAAAJBbTIYBAACQW0yGAQAAkFtMhgEAAJBbTIYBAACQW0yGAQAAkFtMhgEAAJBb/x851DqC1HpbSAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x216 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "f, ((ax1, ax2, ax3, ax4)) = plt.subplots(1,4,  sharex='col', sharey='row',figsize=(12,3))\n",
    "ax1.imshow(np.reshape(image_test[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax1.set_title('True image')\n",
    "ax2.imshow(np.reshape(x_construction[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax2.set_title('Learned image')\n",
    "ax3.imshow(np.reshape(image_test[146,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax3.set_title('True image')\n",
    "ax4.imshow(np.reshape(x_construction[146,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "ax4.set_title('Learned image')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "DEBUG:matplotlib.font_manager:findfont: Matching :family=sans-serif:style=normal:variant=normal:weight=normal:stretch=normal:size=15.0 to DejaVu Sans ('/usr/local/lib/python3.5/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf') with score of 0.050000\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEICAYAAABLdt/UAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXuQJFd15r9T1VXqqepBgmx5tUbT2RIWbLQxL7URhLyWlsK2GMkrO0wIyy0Y1oTbKtiNMQFhHh1reTeiYQkcYFgZRIM1zKpqWWQEy2MF2EiyMCC09IDQogeyzHSPhBCaaUkITQ2a0fTZP7KylZWdj5uZNx+VeX4RJ2a6KjPrZlXmd0+ee+65xMwQBEEQykMt7wYIgiAIehFhFwRBKBki7IIgCCVDhF0QBKFkiLALgiCUDBF2QRCEkiHCLuQKEa0R0avzbocglAkRdmFsICImol/RdKwLiehBHccShKIhwi4IglAyRNiFwkBELyei24jocSL6CRFdTUTN4XtfH272fSJ6koheN3z9EiK6Y7jPt4joRY7jrRHR24noTiL6GRF9mogmiagN4MsAfnl4rCeJ6Jc92rObiO4mop8T0Y+J6O3D1y8kogeJ6N1EdGT4OQuO/S4mou8R0RNE9AAR/aXruL8xbOvjw/ffOHz9FCL6KyI6REQ/JaJriGiH1i9ZqAQi7EKROAngrQCmAbwSQAfAmwGAmX9zuM2LmXmKmT9NRC8FcC2APwVgAPgYgC8Q0SmOY14G4CIAZwF4EYA3MvNRAK8B8NDwWFPM/JBHe/4WwJ8y804ALwRws+O9M4btfC6APQBWiOgFw/eOAngDgNMAXAygS0S/BwBEZMLqVP47gNMBvATAHcP9/huA5w9f+5Xhsf9C8bsThC1E2IXCwMwHmPnbzPw0M6/BEuoLAnZZBPAxZr6dmU8y834ATwF4hWObDzPzQ8z8KIAvwhJNVU4AmCOiZzHzY8z8Xdf7/5mZn2LmWwH8H1idCJj5H5n5/zHzJjPfCeBTjvP4IwBfY+ZPMfMJZt5g5juIiIbn81ZmfpSZfw7gPQD+MEJ7BQGACLtQIIjo+UT0JSJ6mIiegCVs0wG7mADeNgxpPE5EjwPYBcAZVnnY8f8BgKkITfoDALsBrBPRrUT0Ssd7jw09f5t1+3OJ6DwiuoWIDhPRzwBc6TiPXQD+xeOzTgfQAnDAcS5fGb4uCJEQYReKxEcB3AvgHGZ+FoB3A6CA7R8AsMzMpzmsxcyfUvis0LKmzPwdZr4UwC8B+N8Arne8/exhrN5mBoAdzvmfAL4AYBcznwrgGsd5PADgeR4fdwTAMQC/6jiXU5k5SkckCABE2IVisRPAEwCeJKJ/A6Drev+nAM52/P1xAFcOPWQiovZw4HKnwmf9FIBBRKd6vUlETSJaIKJTmfnEsF2brs3+y3C7fwvgEgB/5ziPR5n5F0T0cljhF5s+gFcT0WVENEFEBhG9hJk3h+fzQSL6pWEbnktEv6NwLoIwggi7UCTeDksEfw5L5D7tev8vAewfhiouY+ZVAH8C4GoAjwG4H8AbVT6Ime+FFfv+0fB427JiALwewNowLHQlgAXHew8PP/MhWGJ95fCYgDXg+1+J6OewBj+3PH1mPgQrvPM2AI/CGjh98fDtdwzP4dvDz/waAHtAVhCUIVloQxCiQUQXAugx85l5t0UQvBCPXRAEoWSIsAuCIJQMCcUIgiCUDPHYBUEQSsaEjoMQ0WkAPgFr2jUD+GNmvs1v++npaZ6dndXx0YIgCJXhwIEDR5g5dNKaFmEH8CEAX2Hm1w6LNrWCNp6dncXq6qqmjxYEQagGRLSusl1iYR9O8PhNDPOHmfk4gONJjysIgiDEQ0eM/SwAhwHsG5Yq/YRrqjUAgIgWiWiViFYPHz6s4WMFQRAEL3QI+wSAlwH4KDO/FFbJ0ne6N2LmFWaeZ+b500+XukaCIAhpoUPYHwTwIDPfPvz7M7CEXhAEQciBxMLOzA8DeMCxyEAHwN1JjysIgiDEQ1ce+38C0CeiO2EtZPAeTccVhBH6/T5mZ2dRq9UwOzuLfr+fd5MEoXBoSXdk5jsAzOs4liD40e/3sbi4iMFgAABYX1/H4uIiAGBhYSFoV0GoFDLzVBgblpaWtkTdZjAYYGlpKacWCUIxEWEXCo0z9LK+7j0349ChQxm3ShCKjQi7UFjs0Mv6+jqCitXNzMxsbS/x92Ijv1FGMHPmdu6557IghGGaJsOqPeRrrVaLe70e93o9brVanu8J+uj1emyaJhMRm6YZ6fuV3yg5AFZZQWNF2IXCQkS+gu4WFr9OwDTNfE+iRCQVZvmNkqMq7LnUY5+fn2cpAiaEMTs76xlXN00Ta2trI6/VajXPcA0RYXPTvQa1EIcov4cX8hslh4gOMHNoBqLE2IXCsry8jFZrtFBoq9XC8vLytm3tOLvq60J0/AapVQev5TfKDhF2obAsLCxgZWUFpmmCiGCaJlZWVjxz1qN0AoJF1IHMpMIsv1GGqMRrdJvE2IU0SDKwVzX84uXdbtf3O/Tap9FosGEYyt+5/EbJgMTYBUHwwy9eTkQjcfBWqzXylNTv97F3715sbGx4Hte9vaAXibELguCLX1zc7eh5zew9duyY73EHgwH27t0rueo5I8IuCBUkyoDl+vr6ljh7lXVws7GxsTWpzK7nI+KeLSLsgpAiecy0dH7m9PQ0pqent33+7t27QUQj+7n/dmKLs19ZhyAGgwH27Nkj4p4lKoF43SaDp0Ke9Ho9NgxjawDQMIxUBvHSnGnpNwjp9Zlum5yc9JzwNTc3F7ifPUgatE2QOc9dBlHjAZl5KpSdOOLQ6/W40WhsE51ms6ldXNKaaRnUYaiUYfCzer0ee19Vs38nKS0QDxF2odR4CXSj0QgVhyDhs0XH7iwMw/BN5VPpVPy8WyLadi5ROqigDiNtYU5q9jmm0eFVARF2oRCk9cjtDKU4zTCMwP3CQglBYYyoBcf8BMwwjJHOw91BhYUskoRDwkQ3bWG3z8Xv84VgRNiF3NH9yK0aaggiaH+VUETQNm6P029CT7PZVBJAv++v3W5rF9xGo8ETExOpinpYuEg89nBE2IXc0XkDqwwKuoXdKSK2IPt5+jrMK8TiHqRV/fygkEUa7Y7SrrifEzTAKzF2NUTYhdzR+citKnJ2KCZKR2Bb0sFDZxjI6/OjiGJQyEK3dTqdzMIwdskC5/ctWTHqiLALuaPTY1cRHmdmS1xvN4nAOYU9ibdte69TU1OZCHueJp56NFSFXSYoCamhs5pf2EzJer2Oa6+9dqtGSdx1UJl5a6KOYRgwDANEhHq9HrrvxsbG1kSgKBN5ms0m2u321t87duwAABw9ejRi68cPWYw8JVTUX7eJx14ddGXFBIVWnF5ft9tVCqmEeeb2U0XS3HAV63Q6kcNGZTLJhlEHEooRyoaXaDs7i263q/z474z1BgluFrHnWq2Wu7jmaZINo46qsEsoRhgL+v0+9u/fj5MnT269Zod17PDLysqK0rH27NmD888/P3S7m266yfJ+UqbMy8IF1Z8BZKGN1FBRf90mHnt1iRuaURmI9XrfywzDqHToI2tz/t7OhTyCZvYK3kBCMULRSJK/rJI6mVV6oNh28xvX8AuzSC57PDIXdgB1AN8D8KWwbUXYq0mS9EeVfU855ZTcBa6qNjc3t61jDRJqmX0aD1Vh1xlj3wvgHo3HE0pGklXuw1In+/0+nnrqqeSNFGJxzz33jIxHEBH27NkDANvq0QfVdY+bpiq4UFH/MANwJoCbALwK4rELPiT10oLi8+NQ2bBqVqvVPCtwBtXKEY89GGTssf81gD8H4Du8T0SLRLRKRKuHDx/W9LHCOJF0wtLCwgLW1tawubmJtbW1kQWT46zsI6TL5uYmTpw4MfLaiRMncPz4cc/tJUNGH4mFnYguAfAIMx8I2o6ZV5h5npnnTz/99KQfK4whCwsLWFlZgWmaICKYpqltRXuVmaFCsdF1LQgAsSMuFusARO8F8HoATwOYBPAsAJ9l5iv89pmfn+fV1dVEnysITsLypYV0ISIk0RLTNLG2tqavQSWFiA4w83zYdok9dmZ+FzOfycyzAP4QwM1Boi4Iuun3+yLsOaMq6o1GA81mc+Q1CcHoR2aeCmPP0tJSIm9R8Ea1+Jl7HyfNZnOrkJppmti3bx+uvfbaVMJxwjMkDsXEQUIxgk5qtZoIu2ZM08Ty8jL27t2LjY2NyPseOnQIMzMzIyUfhOSohmImsmiMIKTJzMyMZMVopNVqYffu3VhcXMRgMIi0r8TKi4GEYgQAVpzaPZFkXPBKoxTiYU8suvHGGz1FPWgsQ2LlxUGEXUC/38fi4iLW19fBzFhfX8fi4uJYibu9OIWQDGbGjTfe6DsD1C/kFRQrH2enYWxRmcWk22TmabHwm7VZr9cLX3kvztqmYuEWZdHvoIUypNiXXiDVHQVVVKoi6rwZda2qxCylBNKyZrOpvABIUBkA1TISOq+JMiPCLnjidQOpiqOOOh66PTgp1ZudTUxMbKv9EvbbqZRb9rsmnLXbRewtRNgFZh4VcsMwthVgsm8g1XBGWh523E5DPPZszTCMbWIbpzib8/f22yZKGeCqIMIuKMef7ZtRNa6ahoftF6cNe0RXPceqryuqy9y/U9gTmMoTWpSnrqpXfxRhrwhu4XM+vvqtauNlUbz2JDdYFI9dNWzT6/WUzjXK9yHm/7s7rzmVlZPCOucoT11BA7VVACLs5UdnRkicWLWuNvs9AYR1AmFhJqfV63Vut9u5C+M4mVcoRNUBiCLAXteE3/UoHrsIe+nJO74cJRzjFmGVRYyDOhuvsJGItz7zG7xMMtAe5Ll7PXlKmuR2IMJeLrxuirwzQgzDUG57nKwH1UE1d5tUQzNi/qIcp7N1/rZe4TKvJ6put8vMzN1ud+s3q9fr3O12JQXSA4iwlwc/YYwyiSQt82uv84b0a2dY1kOUR3R3m/Lu9MbRGo1GqHiGeex2x+om6FrtdDqer9uiLzwDRNjLg9/NNDk56XuzvATgpwE+nLIYtNvtkbBK1EFYt7kf4d2dhMoxgkIGdrpe3iJaRHM/gXl5zGHjOkQ0IshRwjduq9frWdxeYwVE2PND9yNkVO+zXqvxNwB+mCh1YddtYYNuKk8ptVrNM9Zuf4/2b5P3uRbNVCYNOfPWg45l/05Jv2cJv4wCEfZ8SKM2RlSP5wqAD05M8Ad27Bg7YQ/LeihC+GlczTCMkVh20HevkpaaRecoA6ajQIQ9H6LkaavS6/W40WiwCVg/mYddMPycKYB/DPClAF+F9EMxOs1+jA962hFPO7q5s5BUMk6Cjhd2rafRfsECisIuZXs141fu1O91VYgIPwHwCpf9HYBjAB4YbvcXAO4B8PmQYxURZsb+/ft9ywf3+33UanLJRsEwDBw7dgwbGxtb3+n+/fuxZ88eGIaxtZ277LHf9+xcKm95eTnRtTQxobbOz8bGhpT6jYqK+us28dhH8Ur1UjnmxQCfBPgNw7+fD/BRgF84/PsqFNNj9wun+IUIbI8z73aPmwVlThmGERhDDzquTa/X46mpqUzOpeoTk2wgoZh8iBpj73a7nhdyp9MJFLNzAH4M4Ksdr30Z4L9x/H0ViifstVrN95zF9FncuQ6maQaGWJyzfr2yY9KaIGYP7Lo7E3cWTtmBCHt+RMmKiTORZgrguwD+BsATw9cuAvgJgJ8H8KlDey/AR4b/b6YsJFfBO/b/O67tOp2O9tisePPeIsisPw5uX8t+37kdw8/6nKempnzvM3eBO79c+3EAIuzFIEzkwy7YywE+CCvkcnD49w0APwTwGY7t9sJbWG1bSvnGugrWE8R5LnuWazt7EE/nZwfl81fRnGELnTOU7UHMsFBN2PtpmZf37tfJNJvNsRR3iLDnT9Djqi30QeVkLwf4SYwK9FMAnwD4/OHFaW/7XFiZMU7bB/Djw//PpHxTXQX1sI942OnZxMREZOdBxZzhxLBwS56zoolopGRwUKc2jnF7iLDnj8pjcFAo5iC8ve8nsN0z3umx/1UIF1vVUJDp0xaG1XGofJbz5svjpi+7OSdgxS3f7Gd2+CKKN57X72wLdljnMo4lgCHCnj+qF3a73d66+YiIJyYmGLDCL6xoF3gc9yoEiy2z/6Oq25rY3plcD/AA4LOHn3Vi+HnHAf4uwL+fw01dNbPDe7Kg9zNme+0q3924ARH2/FEduHKO+Dtv0IPwFvGDmm6ATqcTWxDcqZYLAL8V4AsB/l2AvzRsq4h7upakFktZTWW1LImxi7DHRtUb9itM5RVjf3L4ep43jleqpZd9C+DvFeBGL6u1221mjjYbVzx7S9THdaFsZCXsAHYBuAXA3QDuArA3bJ+yCrtqudoo5pUVk+dN4ZVq6WdvH7a7VoCbuYxme52qHrvUqrdsYmKCG43GyGvjkg+PDIX9XwN42fD/OwHcB2AuaJ+iC3vQOqLu3t15Y1VhUNAr1dLP3gardLAIe3rmHJsJ29YW9rzbXHQrsgePvEIxsMqU/FbQNkUWdpWBKOfU6zI/2rqfFq6HlW55vuL+3wL4QAHOoypWBcciKytqVUkoCjtZ2+qBiGYBfB3AC5n5Cdd7iwAWAWBmZubc9fV1bZ+rk9nZWai0zTAMTE1NKW07jlwO4OMA2o7XGMA/AniXa9u7AXwRwA0A7h3u8ycALgLwe8P3hGwgIui8p6uMaZpYW1vLuxkjENEBZp4P3U7XRUBEUwBuBbDMzJ8N2nZ+fp5XV1e1fK5uarWa3BgADgKYVdz2QgCvB/DvYMXlNgF8F8B7AHwlhbYJQhYQETY3N/Nuxgiqwq5WNzP8wxqwHLZ+mKgXnZmZmdJ64VGY8Xl9E0Dd4/VbU2yLkC31eh0nT57MuxmpMTk5iV/84heh2z3nOc/JoDXpkLi4NVkFmf8WwD3M/IHkTcqX5eVltFqtXNtQhHrpftXjk1WVF4pOq9XC/v37R+qulw0VUR93dKxacD6sJ/FXEdEdQ9ut4bip0O/3MTs7i1qthtnZ2W0F/BcWFrCysgLTNEFEME0T7Xbb52jp4BcKIqLM2vJuAEddrx0dvi6Ul8FggKWlJSwuLubdlNx59NFH825CfFRGWHVbXlkxKgv0+qU0qsxmy8I6nU5mn1W0HHqxbMyekl+Uaz4vK2LJAcjM01GCJmYErSZjU4SLPI2St2JibgtbbKMKNu7pjpUQ9rj55s7VYvK+0AApdyuWvtneehUdCGd1zCKKOrMI+whxvQ/7Ii/KJKQq3mxi2ZlzWn2Ue8a5LsA4W7fbjbT6WR5AhP0ZggQxaFEAeSQVK7NNTk5uiZi9YLizjEbe7cvaarXatk6qaDVkIML+DH7iXK/XfUsD2DG2cfOSbS9DwjZiYWYLutd7rVZra12AqptzVaa8gQj7MwQJt3Mbr0cwv06hCIOpfhdhULvFxFRt3JyaNK1erxei1C+qKOxe4uysvmhnxUT5Ufw6hbB1H/Mye8FhuSnFxNK1PDJnUDVhVx3kjPJj+HUKRY4/Rq3RLSYmFt+yznVH1YQ9ipCp/BhB4ZtxEE3DMAobLhIbbytLFoyfnQvwPoDvhTU5b1/AtlkviA1FYddRUqAQHDqkXsUkaFu75MAVV1yBwWAw8t5gMMCePXvGokjYxsZG4SrTCeNNvV5Ht9vFU089hV6vl3dzUuN8AL8B4DsAHg7ZdmbGr1xeCN/8JnDeecDkJHDWWcCHPxzvOH6oqL9uK4rH7rVSkko4R+LXYlW1RqMRmlgw7kaO/38H/h67agLGNv75n5nbbebXvY75ppuY3/te5nqd+eMfD9U5VC0Uo5qa2Gq1uNvtJk4HFHEXq6rZA/RFmZEd1UzAkj4Pu8C1rZ+w12q1LeH2Sy/2Hc9bXGQ+5xzmEyeeea3bZT7zTObNzUCdQ9WEfXjSoRekqleudIGU1GMREwuzcRlr8rImwOe57HqABwCf7drWT9idnZtbT0z4dxx8yy3Mu3Yxv+Mdo+J1663W+3feGaZx1RP2sIlIQdtENTucU6QMmbIPaomJpWEXwxokfYPHe0GhGD89cXYcrwCYb7uN+bWvZZ6cZP7BDyzZ3bdvVLweecR6/frrAzUOVRs8BYDdu73LwC8uLmJhYQEAtAx8tlotLC8vAwA+8pGPJD6eLt70pjfl3QRBGCvOAdAD8FEA/yPivv1+3zMR4ziA24f2E9MEjhwBbrgBuOYa4LTTrI3sf22e/Wzr38cei9gKH1TUX7dl7bE7B0vjxMYNwwgcFCnKI2lR2iE2XqZjzCmKERF3Op3cr9cpgO8C+BsAT/hsE+Sx+5UBt63VavHn3/9+5lNPZX7zmy2xePBByzP/3OdGxevECev1j30sUONQxVBMkGgTUegP4bdfWBGgoFrvYmJFNjtMmbXI2gOLWd437oVjbgf4IYDPCNgnSNjDbArg+xoNfuScc5iPH7fE4sknLdn95CdHRURzKKZUwp7WxakyW1WKbomNq+ksSx3lPsha1J8ELMkb2ibAfxmyXxRh9+s4zt6xY1Q/du1ifuc7RwXk61+32iWDp9tJu3Z6UG6qpD+KiWErZJl3O9x2EKOibttDGM2O2QnwNMB/MLT7Ab7Z8bff8cM6jpHZ7ouLzC94AfPTTz/z2lveYgm+pDt645wkkMYF4ue95x0vFBMrihXxXjgJb2F32wVD83vf7/gHfbZ3dhx8223MP/vZMxOULr+c+eabmd/3PuaJCZmgpEpQ+qM9EDo1NRXrwrUZx3xeSYsUS9M6nU5hVh2z7SC8hfegpuOrdhx8yy2WcPzTPzH/+q8zn3IKs2kyf+hDSpoGEXa1OuxxPHu78H5RS/eKiemyRqMRecENe0A277Y7zStU8uTwdR3HP+g6tm0HHdvoACLsFmH1G+J424ZhcKPRyP1ijdPuvNsgNj5Wq9W40+nEutaZOff2u809uKlL1O1jh3UcOmq3Q4RdjagDrkFrpBbZ5ubmCjmoJVYc8yrzHPeJdtzCkzosrOOwyxAkASLs6ji9eq9Ffd0evwikWNksjsPiNz4la6X6W1Igwp4eVfRGxMprdkw8isNie5/dbncrH71er8dKRqiSJV0zFSLs3ijXTA45hsw0FSuLxVkA3S+sIE+z6hZnzVRkKewALgLwQwD3A3hn2PZ5CbtKlowquuLs4xivFyuX2VleUceb/BaNF1O3qGumIithB1AH8C8AzgbQBPB9AHNB++Ql7GFFwsLQPfnJMAyJ2YsVwuyVkeIIdLPZHMsssSJY1DVTkaGwvxLAVx1/vwvAu4L2yUvY/QRU5cvVXa7A+aRQpJruYtU123NnlnEk3eY356XIHvtrAXzC8ffrAVztsd0igFUAqzMzM5FORhdJPHadF7rtqTuRkIxYUUyuRf3Wbre1hIFRNGF32rjF2KPMogvrAPwGncKeCBqNRiFKATSbTe50Orm3Qyxdk/CgftORuAEJxXgT9uW634+yRqpTtL3EL6wTCcqnd8Y/7fe9JpRkYXLTV9ckGyy+6QAZCvsEgB8BOAvPDJ7+atA+Rc1j9/Kao4hYs9n0zS7wW7AjSS8usXkxsfEwHbNOmTMUduuzsBvAfbCyY5bCti+qsOuIowd5NO5YftL0SxngEhMrvtVqNS11YpgzFvaoVlRhTzvEYGcd2B66XyegMpirEvefm5vL/aIOu+AlTU6szOaVKJEEiLBHx88D1iX4qkLmTL/0CtWopF7aC3DnfWGrXPgSsxcrm8Wd+BgGRNij4xcasQuBAdkMHDpF2f15KsWa7Fj/uAjmuLRTTCzMkmS8qAAR9ngEDWZm4QEnTWt0PvqNg8cuJlYmSxuIsOsnzLOM63naS/UlTWFUGZwVExNLx6LOIo0DFIW9BkGZmZkZ3/darRauvPJKmKYJIoJpmjAMI/SYrVYL+/fvx3XXXYdjx45hc3MzdJ92uw0i2nac5eXlrb/7/T6WlpYwGAxQr9cBAKZpotPpbP0tCIIe3Pdf7qiov24bN4/dWRjJyyv3G/n28pibzea2iUfM6mETv1DNKaecsvX/dru9bRt7rEA8eDExfZZ2TN0NJBSjh6BJSyo/qOoEJJUwjixkoM8WAf57gB8G+HGAvwHwbxWgXWLjY1mEXtxAhF0PSUv9Jv0csXRsHeCPAXwpwK8G+JOw1qr83QK0Taz4Zmee2eioA6MCRNj1kKTUbxRkoDNbMzxe+ybANxegbWLFtlqtNlIeROcCPmFAhF0PWXnszNYFIiVT9ZgJWJe3h13gs8/VAN9bgLYX3ewxJXeN8SotYu0U7iw1AiLsekirN/Z7dFMNyeRV2XFcrAnweS67HuABwGf77HMA4C8UoO1FNr9BePdEviqYLdxZPdUzi7D7EicWFmUflW2DZrjmfbGW1S6GFUN/g8/7/wGWN39hAdpaZAt6orRnTBPR2Dkecdobtgi4eOwpoFpjxcv79tvXS7DdNdLdtWDcx+/1elLTOmM7B+DHYIVavN5/GcBPAvzBArS1zGZ7tvb1X4T7wA4rBWlDmHBLjD0jYff7ov28DWfP6rWvX+64V7541AtALJ51Af4OwI8CfBTgO4evubebAvguWOmMEx7vnwXwTwD+PMC1ApxX2cyeVW0OF61JEn6MY41GYyREZHcmXk/TQc5bmHBLVkwGwh71QnHGwtK4yMIe2cSi27uHdgnArwJ4GVaoZWP470GALwf4BoAfAvgMj2OcDvB9AH8b4B0FOKcymn3tB4lj2ES+uJ69zpK5WQl3GKiysEet2eL02NOoNBg2yCKW3C4H+ARgXdJDe2r42vke27dhefz3ATxdgPaPu/ld2/bKQSrhDD/h7PV6nuWu7ZRDdzaZ7hroRQJlE/YoPabfRWQYRugjlW6vWiUtSszjhseoSDvtAo/tD/ps+wS2Z8fsBPirsIT/co/38z73Mlm73WZm9cwRv/Etd8jTPUGoKqBMwh51cCLssS+og4gSYw8z9zqnEmNXN9V0xTos7/sk/DsCt10Q8n7e5142C3JobI/ebw6H6thYVUCZhD1OOlGSmJif1+B1gTUajcAUMHcbJaUxnnmlK/4rhIv4wQK0Xczf7MHNOA5PGnniRQdlEvYsJwCE4ddhqLZRwjG5FRJlAAAR4UlEQVTRzS9dsQ7wubA88M8AvIlRUX8SVqgl7/aLBVvcnHfx2Mdc2LOcAOCFivev2sY8bhx3RcharcadTmfknLrd7kjpXxXLovxBWLqi066H5dU7s2LyEisxfaYyNlYVUCZhz3ICgMpn2+GXqJOfirQO6cTEhGdd+F6vp7R/rVZLtISfqgWlK7rtYlie+vMK8P2K6THVsbGqgDIJO3N+eaQqoRP3xQd4T4QochgmSvZOs9lMpYO6HJanbXvc18PKXPFKV/Sy9wB8DJKTXhYrc9piXFA2Yc8LVQELml1qi2beN0qSc0jbLocVE2eHbcIqo+uVrvh/AX4LrFrqrwH4AwAfB/i9BfgexZKbnS0jjAIRdj2oetkqBYGKUBsjzGyCamSkYQcxKupBdgHAKwD/EFY5gcMAfwvghQJ8f2L6TNgORNj1oOppF2V2qdcMPVUjom2Pvlmdj18e+skCCIxY9lav17fdhxJjF2H3Jc4FEpb9oRqfTmu9UvcgaBJP253Fk5XXfhDewn6wACIjlo8579kk6w6XCWQh7ADeD+BeAHcC+ByA01T2y0vY42bXRLmwgjx8ItIeu/ZK+UwSI7e9drvzc6+Sk5Z5xdglD7265vTYw5yLKqU+IiNh/20AE8P/vw/A+1T2y0vYk+TDq9Rjt18PughVPGrTNJVyxMPKKsQRZ6+c4azMnRUjol5ts+8nlXBgVSYrIetQDIDfB9BX2TYvYdc9g9XvCcBvJp3TCwm6SP2O7TT7M9yLFng9mkYJp0xOTuZ+Q4uJ2RZUK8ZpVSkvgByE/YsArlDZdhw99ijH8/OQnQXB/C5WZ5qX07uPMojp9OTHIc1STCzIVJ4ixWOPKOwAvgbgBx52qWObJVgxdgo4ziKAVQCrMzMzsU8saXEvnTNYg54Aut3uiDftFHW7Le4MlkajEalcgcoNMQ4plmJiQeYc97H/dr4vMfYUPHYAbwRwG4CW6j5xPXYdwqwzbSrpE4BqW/JOoRQTy8L8nBD3/VTl1EdkNHh6EYC7AZweZb+4wq47lJKUtGvYZD1JSEwsb3M7Me41DaoOMhL2+wE8AOCOoV2jsl9cYS9S+V6btLyHLKb1y5OAWNYWdM35vVelUEsYUBR2srbNlvn5eV5dXY283+zsLNbX17e9bpom1tbWNLSsOPidKwDUajXs2LEDR48eTfQZzWYTjUYj8XEEQQUiQq1Ww8mTJyPvW8Z7PA5EdICZ58O2q2XRGF0sLy+j1WqNvNZqtbC8vJxTi9Lj0KFDvu9tbm5CR4d8/PhxTE9PwzTNxMcShDCYOZaoA8H3g7CdsRL2hYUFrKyswDRNEBFM08TKygoWFhbybpp2ZmZmAt8fDAbKx2o2m77vHTp0CLt371Y+liDkQdj9ILhQidfotnEqApYXXiuzxzE77h93+TExsSKY1Ga3gGKMfaw89iqxsLCAnTt3JjpGq9XC7t27sbS0hM3NTU0tE4Ts2djYwBVXXIHp6Wn0+/28m1N4RNgLQr/fx+zsLGq1GmZnZ9Hv97GxsRH5OHa8vF6vYzAY4JprrvEdhBWEIkJEvu9tbGxgcXFxm7h73T+VRsWt120SihnFLx8+ajqiYRiZLDAtJpa3Oeeu5LkmctagjOmOZWV6ejqWd+6k2WyCmXHixAlNrRKE4kJEW+HFKqVBlzLdsYzEDbk0Gg0YhrGVHbRz504RdWHsqdfrSts5s2T8Qo1VTpEUYc+ZpaUl3/cMw9iWt2+/vm/fPhw5cgTXXXcdACh3Do1GA+12O15jBSFlTp48GZiea3PkyBFMT08HxuMrnSKpEq/RbeMcY9ddQgABccRerxf4eaplB9yrPWVRfybJ2qti1TXDMLZdO7VaLfKykl7r95YByJqn+kljkMavol2tVgvdV0WgvfJ/064RY3cgUjK4mhZ3ScWgRTXiXEtlBCLs+kmjumSSCzNIoIOeJtL02JvN5sgiH3ktsxckHlmt41plsx0e1e1rtZrWjK6yLrwBEXb9pFFdMkln4bdvvV4PDBWpriMZZu70Snt2q2ma3O12t9pXFM/dfnrJu8OJGlYYV8sr9basqY7MIuypkIbHniS8oyJQfsfqdrvKN4qXh9tsNtkwDCYiz7ioVzt03OhEFLmjqNfr276DXq+Xi/BURdSzNvc4UlmBCLt+0poIkXS5P3tf1RVonPuGiZthGJ41a+LUnlHpAILMPo+oHnfQE5WquOuo2yOWjlWpjgxE2NMhqghnuYxXklCRX6el28OMK5B2Bxonq8cOTdmhI+dvoRKScoaVxNRtcnIyk88pc+jFDUTY8yfrqc5prMGq8waMW2HSbku329Wa0dNoNEKP1263mVnWnS26lXWw1A1E2PMn6zVa0+hIdN14Ojx1VXG1t9MxaGsYBjOzxMbHwKrgtUOEPX/yWKNVd+gnSgzaGeawwxf230kGKk3TjBwKsT87qVjYE13yFi2xcKtCSAYi7PmTtseeRfy+1+ttG/BsNBrbhNsr68T5ftKbNo5I6/DY43QqYslscnIydjpq2UMyEGHPnzRj7FnG7+MMGKtmvxiGoSScUUM5umLinU4nd6GrmnW73dhPSWk+DRcBiLAXg7S86qzj91FQDbs44+c6a8tkPdCpa8KXnzWbzUjzDopojUYj8nUR1OH7DcQX4fpPE4iwl5s84vdeRM2k8erkdNaVUQ2b6FoDtt1upx6qabfbY72AirP9qh2gfX34zaHodruVWVzDCUTYy00RPHa/cFDQDWsLuFPYdQmj3amFHU9SF/VaGpO37N/SPYnOORmp2+1uXU/1ep273W5m135eQIS93BRhOTA/AVX1hu32Rk1j9DP7xh7HLJZxLXNsC63usYgwB6UI138eQIS9/ITVak87YyZIaFVDK6qpkK1WayQTp91ub3UgtVqN2+32SKplEbzyiYkJ5e9gHDsjpwDrDEcRUaj3XYQn1jyACHt1ycqbCbqZ7bx2lRs5zMO3i3g5OytnZUlVEW80Gtpi6zpFTOX7TNOSfifM0Wfm2tlQROQ5+Svsei3KGFPWQFHYZWm8ErK0tITBYDDy2mAw8F2Gr9/vY3Z2FrVaDbOzs+j3+0qfs7y87Ll0HwAcP34cAAKXLgOAWq22tSixH/v37wcALC4uYn19HcyMjY2NreUAres9nBMnToR+Vhj1eh1EpLw2ZxjO5dt2794d+n2lQavVQqPRSHwMVYgIl1122dbfx44d27bNYDDAnj17fK9Fv2XvKr0cnhMV9Q8zAG+D1WNOq2wvHnu6RPFmonr37hBP2ml4uhdgSGLO70VHqKfRaBR6URJVi3INEBF3Oh3lMQW/a1Fi7CmHYgDsAvBVAOsQYS8EUeKPUbbNqgJkUc0pGjrCJs7VpsZ1dmutVouUqtrtdiN31H5x8ywrpxYFZCjsnwHwYgBrEGEvBFG8mSjevZ/4JPFe2+32WGSE2MXAgr7jOGaPH+ga7C3KalV+FqcDK3vcPArIQtgBXArgQ8P/ryFA2AEsAlgFsDozM5PBV1BtVLyZoIlBXl5SGpkm9qzKooRb/MxrMQddqzDpWl1qHCzONVT2TJcoQJewA/gagB942KUAbgdwKisIu9PEY8+fII/Tz7v387aSeon2jRslVuteb9XP5ubmtImS+3vRGRc3DGNsY+xRf+sonVgV4uZRQNoeO4BfA/AILEFfA/A0gEMAzgjbV4Q9f4Ieif2WGvML8XhN745idt6yijfnvtGz9nSd3qPuuHgVVmqy01ZVZqtWJW4eBWSdxw7x2AuDShgmTESDshH86r3EFSW7jnvUG1138TAVc8Z7dYemnEXRyui9O8cpwq4XCb94AxH2aqI6cKoiwnFuLlVvzNk2Fa87SkZPFtbpdFL5fPs8k3SUft9z3mMZkraYHMjM02qimr6o4hWGZSN4ee9Ra7GrZIREzejJyubm5rR71u7vPMmx7EW8nb+NfX2kMQO33W4H/tZRriPBGxH2ihJ1clLcx2E/T0vFI3TH8MM8U7+6ITo82qSdg87BWa/vPO7AtH1eTlFPowqjiokHrg8R9ooSpzhSnMfhOKLqV1o17OkhaILKOOTAq5p73dgohcGctXncnVWj0cjt6cbO0xf0IMJeUeLGLKM+DscViqBBWb99/EJCUbzQoPYWoTBYrVbb1kmpPgE5xVNnXF7H9yKTi/Qiwl5hsohZJhEQPw886tNGlDaE5ci7OwidYm8/qcQJq6hkDDnFU1eb3QuWB7U9KDddslv0IsIupEpY+CRICII88ChPG1EW6AiaJWrn0Ts7Q93ZIyrfWVDbgs7VKZ46Swo4jxv220h2SzaIsAupExYDjlKuwHlM1aeNKB67fSw/gXS3SWdMul6vb52bs8NQeSpwpj96dTZu8dTZGbk74LDfRrJb0keEXciEqCGZqBOfgojiAdsipSpiqudlL2Yd1hHEnXTkHmwOW+dTZ4xdwijFQ4RdyAQVwbLzqe36Ll6zVuM+xrs7hLBYb5Q8f5WMG2eH4PfZ9mpBScVV5Xvy2sYr28a9GpXXwK143MVDhF3IjLB8eDvG7SdKUQdNg7x7nbFgleqNzjYGCXvc0I7K0nlenVKcpx8JoxQfEXYhc4KEJ+i9KJOqvAYSvbxW3bFglQ4h6Dz8zj/Mm3fO2KzqOp/CM4iwC5kTJH5xRM/LE1Ud/AxrZxzvNGy/oPNQeZLwysl3Lp8X9clGKB8i7EIu+IlfEtGzCQv3qLYvrbQ8FfEO6hjCxgckpVAQYRcKRVgIRcWLVs3lDiJtrzdJrFol1CKx8Gojwi4UBi9P0554EwU/UbYHZ1UocpxaQi1CGKrCXoMgpMzS0hIGg8HIa8yMG2+8MdJxlpeX0Wq1Rl4jIlx55ZVYWFhQOsbMzEyk17PE6/xarRaWl5dzapEwtqiov24Tj71a6PSSk4YiksbB0ybvzxeKDSQUIxSFooUY/MRTBieFoqMq7GRtmy3z8/O8urqa+ecK+dDv97G4uDgSjmm1WlhZWVEOoWTB7Ows1tfXt71umibW1tayb5AguCCiA8w8H7adxNiF1FlYWMDKygpM0wQRwTTNwok6ABw6dCjS64JQVMRjF4Qh4rELRUc8dkGIiGSlCGVBhF0QhoxLyEgQwpBQjCAIwpggoRhBEISKIsIuCIJQMkTYBUEQSoYIuyAIQskQYRcEQSgZuWTFENFhANtngpSLaQBH8m5Ejsj5y/nL+evHZObTwzbKRdirABGtqqQllRU5fzl/Of/8zl9CMYIgCCVDhF0QBKFkiLCnx0reDcgZOf9qI+efIxJjFwRBKBnisQuCIJQMEXZBEISSIcKeMkT0NiJiIprOuy1ZQkTvJ6J7iehOIvocEZ2Wd5uygIguIqIfEtH9RPTOvNuTJUS0i4huIaK7ieguItqbd5vygIjqRPQ9IvpSXm0QYU8RItoF4LcBVHFttX8A8EJmfhGA+wC8K+f2pA4R1QH8DYDXAJgDcDkRzeXbqkx5GsDbmHkOwCsAvKVi52+zF8A9eTZAhD1dPgjgz2GteF8pmPnvmfnp4Z/fBnBmnu3JiJcDuJ+Zf8TMxwH8LwCX5tymzGDmnzDzd4f//zkscXtuvq3KFiI6E8DFAD6RZztE2FOCiC4F8GNm/n7ebSkAfwzgy3k3IgOeC+ABx98PomLCZkNEswBeCuD2fFuSOX8Ny5nbzLMRE3l++LhDRF8DcIbHW0sA3g0rDFNags6fmT8/3GYJ1iN6P8u2CflBRFMAbgDwZ8z8RN7tyQoiugTAI8x8gIguzLMtIuwJYOZXe71ORL8G4CwA3yciwApDfJeIXs7MD2fYxFTxO38bInojgEsAdLgaEyZ+DGCX4+8zh69VBiJqwBL1PjN/Nu/2ZMz5AP49Ee0GMAngWUTUY+Yrsm6ITFDKACJaAzDPzJWpdkdEFwH4AIALmPlw3u3JAiKagDVQ3IEl6N8B8EfMfFeuDcsIsryY/QAeZeY/y7s9eTL02N/OzJfk8fkSYxfS4moAOwH8AxHdQUTX5N2gtBkOFv9HAF+FNXB4fVVEfcj5AF4P4FXD3/yOofcqZIx47IIgCCVDPHZBEISSIcIuCIJQMkTYBUEQSoYIuyAIQskQYRcEQSgZIuyCIAglQ4RdEAShZPx/7EEwhkbXQPkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAACPCAYAAAAfidZ8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXmsXdWV5r8FMYHEmHjCGGP8jHHwQAzGDjghjkgYhEMqgNLEga4qSk11ulsVpao7ippUq1v1T3VXSqpqtVrd6oqqUpCESlIRICBBIsSYyQNgYwYP2MZ4wMYDZjSEEAi7/3jXO99e9e72uffd6Zzz/SSLdd8+7979znf2uZu7vruWhRAghBBCCCFEHTmu3xMQQgghhBCiX2gzLIQQQgghaos2w0IIIYQQorZoMyyEEEIIIWqLNsNCCCGEEKK2aDMshBBCCCFqizbDQgghhBCitmgzXBAb5jtm9krj33fMzPo9L9E6ZvY5M1tpZm+Y2a5+z0e0j5l9y8w2mtkRM9tpZt/q95xEe5jZfzSzF8zsTTN7ycz+p5l9qN/zEu1hZieY2RYz29vvuYj2MLO/MLP3zOwt+ndWv+fVDbQZLs7XAFwD4DwACwD8HoB/19cZiXZ5G8D3AGjjVH4MwB8CGA/gSgBfN7Ov9ndKok3uBnBBCGEcgHMxfK/9Rn+nJEbBtwC83O9JiFHzkxDCWPr3Qr8n1A20GXaY2XL3f0HvmtmDAG4E8DchhL0hhH0A/gbAH/VzriJPMy1DCI+HEH4AoJKLuopktPzrEMKTIYT3QwhbAdwF4OJ+z1c0J6PljhDC60cPA/ABgLP7OFWRIfNeCTObCeD3AfyPvk5SFCKnZV3QZtgRQoj/FwTgdAxvmH4EYD6Ap+nQpxs/EwNKRktRMopo2bAtLQWwqQ9TFAXJaWlmN5jZmwAOY/iT4b/r30xFjmOsyf8N4M8BvNOv+YniHEPL3zOzV81sk5n9h/7NsrtoM9wEMzsOwD8BeDCE8HcAxgJ4gw55A8BY+YYHnxG0FCXlGFr+BYbvaf/Y63mJ1hlJyxDCPzVsEh8H8P8AHOzjFEUBvI5mdi2A40MId/Z5aqJFRliT/wxgLoDJAP4tgP9mZtf3cYpdQ5vh5vwlgJPxO8/aWwDG0fg4AG+FEEKvJyZaxmspysuIWprZ1zHsHb4qhPBuPyYmWqbpugwhbMfwJ/z/t9eTEi0TdTSzjwL4a+heW1aSNRlC2BxCeCmE8NsQwmoA/wvAv+rnBLuFvqk7Ao0v4FwP4JMhhPcaP96E4bTd443H50Hp2IGniZaihDTT0sz+DYCbAXw2hKBvrpeAguvyQwBm9W5WolW8jmY2H8AQgEcaSdMTAJxiZgcALAkh7OrXXEWegmsyYNjPXzn0ybDDzBZi2O90TQiBvwn7fQD/ycymmdnpAL4J4JY+TFEUpJmWZnacmZ0IYMzwQzvRzE7o1zzFsclo+a8B/HcAl1f1W85VI6PlH5vZqY14HoBvA1jRn1mKY9FEx40ApgM4v/HvjzFsdTkfwIv9mKc4Npk1ebWZjW+Ulr0Qw58Y39WveXYTfTL8L7kaw2WaHiU78CMAvgDgLADPNn7299CXOwadZlp+B8BKOu4dAA8BuKSXkxMt0UzLOQAmAniCfv7DEMK/7/kMRVGaaXkAwF+a2VgMl+T6KYD/2pcZiiKMqGMIYdnRB2b2KoAPQggH+jA/UZxma/J1DJch/TCAvQC+E0K4tS8z7DImy6sQQgghhKgrskkIIYQQQojaos2wEEIIIYSoLdoMCyGEEEKI2jKqzbCZXWlmW83seTO7uVOTEkIIIYQQohe0/QU6MzsewDYAl2P4W4ZPALg+hLC5c9MTQgghhBCie4ymtNqFAJ4/WtvTzH6M4fIcTTfDkyZNCkNDQ6N4SdEuu3btwuHDhztWLFta9o9Oaikd+8v69esPhxAmd+K5pGV/kZbVQVpWg1beK0ezGZ6GtIj2XgAX5X5haGgI69atG8VLinZZvHhxR59PWvaPTmopHfuLme3u1HNJy/4iLauDtKwGrbxXdv0LdGb2NTNbZ2brXn755WP/ghhYpGU1kI7VQVpWB2lZHaRl+RjNZngfhtsuHuWMxs8SQgjfDSEsDiEsnjy5I1kH0SekZTWQjtVBWlYHaVkdpGX5GM1m+AkAs81sppmdAOCrAO7uzLSEEEIIIYToPm17hkMI75vZ1wHcB+B4AN8LIWzq2MyEEEIIIYToMqP5Ah1CCPcCuLdDcxFCCCGEEKKnjGozLIQQnSBX79yP8WOOzdIKOscdd1zTMdFZiurnj/vggw9G/B3Wzj+Wlp2l3V4D7T5/M809Xmd+nLsGdH38jk5r67XLre2cDkXXcy+1VDtmIYQQQghRW7QZFkIIIYQQtUU2CSFEX8il2PixT80VTa0PSvqtihS1rgCpXrmxdvWSlsUomjLvRGq9nfXrydkkBjHN3i86rVdu7Le//W0yxo9zNomcha1Z7Om2lvpkWAghhBBC1BZthoUQQgghRG3RZlgIIYQQQtQWeYZR3O/2m9/8JsaHDh1Kjtu7d2+MP/zhDydjs2bNivHYsWOTseOPPz7GKh0zeor6UN97770Yv/baa8lxe/bsifH777+fjM2YMSPGEydOTMbGjBkTY3nahimqR27d5TyGufOc88G141uso47NNGpFy6Lll3LnupXSe0Wog5Y5HTrxnP752D/67rvvxvj1119Pjnv11Vdj/NZbbyVj/H74kY98JBmbNGlSjE855ZQY+/dbfg5PWbXtxPXfyv2Q3/fefvvtGPt9z4EDB2Ls30dZl/Hjxydjp59+eoz5ffTEE09MjvvQh5pvUTutpT4ZFkIIIYQQtUWbYSGEEEIIUVtkk2gBTq1v2bIlGbvvvvti7FMCy5cvj7FP/eRSOs3IpTvKmgZql1ZSf81sEjt27EiOu/POO2PM1hgA+NKXvhTjCy+8MBnjlE5RjaqgZdGyaLmyPLkxb1Xh88zWFL+WciWBeF6+nA8/D8e50m2tlHUbZIraVXLlslrpUtXs/LaS0uVji2pU1TJ8ne44lrse/Lp84403YvzCCy/EeO3atclxzz333Ii/41+PrRAAMG/evBh/9rOfjfHs2bOT4z760Y/GuEzaebptjcjdY1955ZUYb9q0KcYPPfRQchy/d3o7DK+xU089NRlbsGBBjJcuXRrjc845JzmObaW5+3sndNYnw0IIIYQQorZoMyyEEEIIIWqLNsNCCCGEEKK21NIz3K7P9Ne//nWMn3zyyeQ49kF534sYDJp5hrdu3Zoct23bthh731rR9rFVJ9dulf1nHHv/NfsF2aMGAC+//HKMubQPAIwbN27E2Pvx2U/steIx/3v+cTOKelwH/TrJeaubacnrB0hLZPlyWfzYexNPOumkGPN5z5VYyrV29WW2TjjhBBShaCnEMpErY5c7Lre2uWTaSy+9lIyxn3TNmjUx3rx5c3Icr+133nknGePX89cA36e5xNd1112XHMfeYn89lJVOeMH92ub7sddyxYoVMX744Ydj7Pc9fA/39wSe88knn5yM7d+/P8Z8DXi9eC/VzverWkGfDAshhBBCiNqizbAQQgghhKgtlbJJdLqsjH9OTtdyKh0ADh48GGMuGwKkKYJcSZ9OpOeKpsbKRLu2llz6jy0v3ibBXXW46xEATJkyJca+O06VtWylxBKnyzgFxp2nAGD79u0x5vI9QKoBWxqAVIPTTjstxj6tymk1rxXr6sv+cGo91wGprOQ6h/lUJ6fFjxw5EmNOdQOplrt3707G3nzzzRh72wLrwLp6exL/nk+XciktX9ayWaeyKtwXgfbtOUU7yf3qV79KxjZu3Bjj22+/PRl74oknYsxpcJ+ez9laOHXv7TZ8jfFzslUKSLub8TVVB4p2BQTSsmh33HFHMvbAAw/EeOfOnSM+H5DanHx3XX5f8K/9/PPPx5jX6LRp05Lj+LG/VtSBTgghhBBCiA6hzbAQQgghhKgtpc8B9rLjDqd+nnnmmeQ4tlBwmgZI0wfeJiFGphO65rTklL3vJsjpOZ9m45S6T9c2S9v4n5e58sBRiqZZeV3s3bs3Oe7ZZ5+NMadAgdSecNZZZyVjs2bNivGECRNi7KtVcIWKXKcr/01nTrvmrC9l0Qoorpe3SbA1gr9xzilxIK0Y8NprryVjfP8788wzkzE+95xy9Vryc/oxtkn46gecWs11K6wiRTuY+dQ33/8ee+yxZOz73/9+0zG2VLDmvF6B1BrjK7ewrcpbEfft2xdj7nC3evXq5LiLL754xNcCqm978tc/rxXfafXWW2+N8b333puMcbUO1ogrdQDpeuZ7MZBq6V97165dI475yiMXXXRRjL0FqtNaamcmhBBCCCFqizbDQgghhBCitmgzLIQQQgghakvpDTTs2+tGaTX2U23YsCHG3v/I3qT58+cnY1wWqNu+wzL5GHsNa8mlXXxpNT6HH//4x5OxnP+7ndJqueMGWcuiZZvYN+ZLbvEa8l63mTNnxvjTn/50MnbGGWfEmL2f7G/1r82lD4HUSzd16tSm889RFT8x43Vgzzd32Xz66aeT49jP6b19Z599dowvuOCCZIw9+Ozp9WW1uBQil90D0nXty+tNnDgxxr70UxVo5fsIPMZlr3yHxwcffDDG7BEGUp8wawKknlH2euZ8pr4s2uuvvx5jvy5XrlwZY76XeJ/p2rVrY7x48eJkrCqe4WY+Ye+n37NnT4x/+MMfJmNcTo09wkBa7ozvv4sWLUqOmzt3boz9uufvaaxfvz4Z4/s2+4f9vorv2957zs/RifvtMT8ZNrPvmdkhM9tIP5tgZveb2fbGf8fnnkMIIYQQQohBpIhN4hYAV7qf3QxgRQhhNoAVjcdCCCGEEEKUimPmDEIID5vZkPvx1QAuacS3AngQwH/u4LzaouhH5a3YKbjU0OOPPx5jn8Y755xzYuxLQdWhjE+nKZqGbwXWctWqVTH23bQ4HeO7CeYsL0Upawq9GblygWxd8Ckw7kzmSyCde+65MZ4xY0Yyxim8XIqQux5xWTAgTZdyGt8/J1M13UbC/+2sH5ez8lYFLos2Z86cZGzJkiUxnj59ejLGZdF4XftrivXyKV0uk8jPBwBDQ0MxbqZrVfH3SbZGcBk0thUAwG233RZjXz4tV3Zy2bJlI8b+/ZA18jrz83st2YrDNgm2VgCp7c2/T/N1Wqb1nCsTyu9rfl3+5Cc/ibHvGMj3Y29xuPTSS2N85ZW/+yzUvx+yNcZbUPjc+5KNL774Yoz5vsLla/0c/XP4jnSjpd0v0E0JIRyd9QEA9ep5KIQQQgghKsGoq0mE4f9lafpRnZl9zczWmdk6/wmcKBfSshpIx+ogLauDtKwO0rJ8tLsZPmhmUwGg8d9DzQ4MIXw3hLA4hLB48uTJbb6cGASkZTWQjtVBWlYHaVkdpGX5aLfOyN0AbgTwV43/3tWxGfUZ781hXyO3IPUtLNknx+V8gNQXJZ/p6ClaTs+PsbdwzZo1MWZvKZB63KZNm5aMsZa50mo5qqBlrqwY+9m4ja73g3HpM9/Kk8sq+ZJY7MFnjdkHCaQle3bu3JmMcYtR36q5bt7S3PXIvj/+hMvf/1g/X1qSS+F5nXkNsafVw9eR939z+SUu1Qak1xjr2o0ynIOGv475PrdxYywOhR/84AfJcewT5vMHAKeffnqMly9fnoxdc801MWavtvd25u7fvC75OYDUo8z3AD9Hvh68Z7gqG1Nef+yZvueee5LjfvrTn8aYfbpA2gr9iiuuSMa+8pWvxPj888+PsS+Fxz5hryXr7PdErDP/Lb6VO++//D2n0xQprfYjAGsAnGNme83sJgxvgi83s+0ALms8FkIIIYQQolQUqSZxfZOhS5v8XAghhBBCiFJQjXYsHcR/1M8pOS7zwSW2AOAzn/lMjH0XpKI2iSqkzwcJn1bZsWNHjLds2RJjb3dYuHBhjDmVBBTvelNnLbkEzuHDh2Psv0jCtgZ/nrmskj+XnE5niwN3FQRSjbdt25aMcQrWd9Li66Yq6fSi1iKfWue0Jevlj/vYxz4WY06lA6nNxZeZ5LlwaTxvXeFOWl5nThP7jpGcQs/pyo9z6d4y4e9/zbqRsWUMSHX2tpMbbrhhxBhIdffvj83InWu+poA0Rc/XkbfXcEc9X46rrPjzxJYXLvnKXeWA9D3P70u4fNr116efeXLnPj7vuTKa/p7AFgq/hnitc5x7/m6vw1FXkxBCCCGEEKKsaDMshBBCCCFqizbDQgghhBCittTSM+y9J+zH8b6XdevWxZi9SL5F7KJFi2Ls2xLmylCJ7uG9ZI888kiM2ZPoPY7s//bet06UySsjrbTHZj8b+4RfeeWV5Dhea74EEvtAvY+MfaDsiVu9enVy3DPPPBNjLrMGpCW+fEk29rCx79J7MMu6rnNz9X8j68D3P1+OkH/Pt8X2jxl+Hm6zzH5vIPUJc/tWIP17eI7+MXvDfbmv3Lrm63vQdeY1xaUkgbQd7y9/+csY+3bG3O78qquuSsa4nBqXzAOAMWPGxLjoefL3DtbBv4/ytcL3du9DZ28sz6nM+PcyLhXJ5dP4ngek54b3KEBaPu2iiy5Kxvg7HDkfb25t8Jhfl7zW+d7vvzvC16LXudPok2EhhBBCCFFbtBkWQgghhBC1pZY2iRw+/bdq1aoY88f+S5YsSY7jTmXtdiYTncWn/1hLTutyKTUAmDVrVoxzlpc6k7MWcUqPY58u57Jdzz33XDLGKTFfEoh15a5KW7duTY7j1Lov1cWdjXJdj7hTkl/XVSyZ6G0Szbo++fPJOnCpJyA9n3498bnn5/AdAzn9y+X6gFQjn45la86RI0difNJJJyXH8fWWSwsPOrzenn766WTs3nvvjTGXDPVWggULFsT42muvTcbYGtFtO6C3L3Fqna9Lb3lh25vXuayWF15DAHD//ffHeOXKlTH21z/bOb/4xS8mYxdffHGMfZfPduyAuRKY3trE92bec40fPz45bvr06TGWTUIIIYQQQoguoc2wEEIIIYSoLbW0SfhvsHI64uDBg8kYp/w4ncQVB4A0VTfo6Zcqw+kz36lq8+bNMWYtP/WpTyXHcapGWrYOnzNOv/kU2P79+2O8YcOGZIxT5j4dy6lATqX6tCqn6fya5/S8t0nwt/D52828xkd6zrLCenmLwMSJE2PMWvq/ndeWrwyycePGGHstuTII/x5bGoDUluFtOVwlwl8DbKngVLPvbuatOGXB68BrgytGAKmNiM8ZawykFkDf0a8TFSNyY5wy3759ezK2e/fuGHPKfMqUKclxbHvjTpaDjj8X3D2P/3YAeOCBB2LMexZfjYGtENxxDkjXQCesQX5dsp1t06ZNyRhXGeK1N2/evOQ4trx0276kT4aFEEIIIURt0WZYCCGEEELUFm2GhRBCCCFEbamlZ9jDPlNfjob9g+x59B1b2vFSidHjfVZcvmvFihXJGPsQJ02aFGPv/5aWo4NLHbHn0PvB2C/K/mEg9Xr68kj8/OwX9CWWWEfvuWN8OUV+nPPTVuXa4L/Dd1zkkpGsn/cFsz9/7969yRh7sr2HkzXi2PtA2f+9b9++ZCzX/Y6vMcaXaSpzOTWGS8n5bmTeh32U2bNnJ4/5OxS5kltFyX1Hh73LQFpi8ec//3kyxvcI9jmzLxYAli5dGmN/PZcJPjePPfZYMsba8vk888wzk+MuvPDCGPtOq7wG/L2sWZk87wvmtefLLT766KMxXrt2bTLG99izzz47xqwdkL5Pq7SaEEIIIYQQXUKbYSGEEEIIUVtqY5PgVE0utc7dXIA0PccpX+7EA1QnZVoGct3POE3oUzPMueeeG+OZM2cmY6ylv1ak8zA5+wCnVvncLlu2LDmOU2C7du1KxjiN5kuajRs3LsannHJKjP21wL/nS25xWSH//BMmTBjxON+pq6ypdX8N89/hS4zxfY5TmHzegbRTlO/8yM/vSz/xuefn9OltLs3kSybmOtyxfny9eetNOx23BgF/zbO9yNtV+Fj+++fMmZMcxxYVf40XLSfIx3GJMCC1zaxZsyYZu/POO2O8fv36ZIx1Wbx4cYy//OUvJ8dx1zWfWh9kbf25ZduBL03GlhdeQ/y+BgBz586NsV/bufe5Zjp7GxLbSLncGwD8+Mc/jrG/v/O6vOyyy2LsO/vyddpt7cp5NxdCCCGEEKIDaDMshBBCCCFqizbDQgghhBCitlTWM9xKC0j2MHFpFyD1oC1YsCDGZWrzWEaK6seebgDYsmVLjPfs2ZOMsWdq0aJFMfY+RtEa3pfH5/nUU0+NsT/P7E3k9stAWrqLPcJA6pHj9Xno0KHkOH5O9gH7eXH5MD8v/lty/sNcaaJBJ1dajc8b+/d86bPzzjsvxtxqFUjXq9eSy+GxluwDBoADBw7E2Hsf2ZPKvmAg/Z4H/y05/3eZtMy18PXw9cta+vcyfg5fdpBfz58XHuPf8x7ve+65J8a/+MUvkjEugeh1Zj/p8uXLY8z3cv97ZfL151pr+5bx/HfxdzR8a21eU/57E7n7F5dM43l47+99990X45/97GfJGOvu7/1XXHFFjK+77roYn3baaU3nL8+wEEIIIYQQXUKbYSGEEEIIUVsqa5PIwSkAANi5c2eMfScsTifNnz8/xr6ED6NyXO3RTtken8Z7/PHHY+y7T3E5l/PPPz/GOS3FyBTtzMapWZ+C53QmawOkJXx8+Sgm18WO072+Ox2n032ZRE4bd7vr0SDQTC8g1ZatBb4cHZ9Pn47NlUJs1o3Md7jzncoYtl74Lltsh+H5567ZQb9f5+6T/H7FfzuQ2sb4vL/00kvJcc8++2yMfSktTnf7MS6px2XRHn744eS4J598MsY+/c9rz3d5vfHGG2PMpdV8Cj7XWW3QyJV85cf+/sX3Tj7OlzXctm1b09dme4VfX2xL4m53/P4KABs2bGj62tyx9+qrr07GbrrpphhzBzr/d/bS5nLMVzKz6Wa20sw2m9kmM/vTxs8nmNn9Zra98d/xx3ouIYQQQgghBoki2+73AXwzhDAPwBIAf2Jm8wDcDGBFCGE2gBWNx0IIIYQQQpSGY26GQwj7QwhPNuIjALYAmAbgagC3Ng67FcA13ZqkEEIIIYQQ3aAls6SZDQFYCOAxAFNCCEdNegcATGnyawNBzmfKHiY/xqVKuNVhrqzMoPuUyob3UrHnm9svA6n/2+tw5plnxnhoaKjpcaI1cucv5y1m2F8G5Ms7sSeVvW7cvhRIr5NcaTXvV27HJ1yVayhXViznLWbfvfeGsw6+FCJry7p63zEfx15HIC3z5v3f7K2sikaM/5v4uuZSoACwffv2GPNaWbVqVXIcn3u+TwKp79r7urmkHntVfcnDXCm8z33uczFmXymQfmeHvdFlarncCnydex24dTl/V2L16tXJcayRb2/NeO82t/LmEpX+Hsvnmr2/AHDNNb/7fPSGG25Ixnidsk+4FS9/p3Uu7E42s7EAbgfwZyGEpAhkGN6tjOjqN7Ovmdk6M1vn60+KciEtq4F0rA7SsjpIy+ogLctHoc2wmY3B8Eb4thDCHY0fHzSzqY3xqQAOjfS7IYTvhhAWhxAWT548uRNzFn1CWlYD6VgdpGV1kJbVQVqWj2PaJGz4s+h/ALAlhPC3NHQ3gBsB/FXjv3d1ZYbHoGg5Lk7V+RIg3PXGl9nij/59Skf0Bq8xp1p9WSDuXOXLP7GWnJbPpVty11dV0nGdptl58eeSU2J+3eXSY5xm5RSpL8uT05g7HfluZLk5N5tjHeC/N6elPy885tOgXOKLrwGvJdtcuKucH/N2m9ycmx1XJl39XPnvv+yyy5IxTnevWbMmxocPH06Oe+ihh2LMpbOAf3lPZbjUGp9r3xVt9uzZMf785z+fjC1btizGU6dOTcbYfpOzMpW1M6SfG9u3uPseAOzYsSPGK1eujLF/P+QSaV47Xm/evsT7JT7vvlvnJz/5yRhfe+21Tcf8uuTXzt07ekkRz/DFAP4AwLNm9lTjZ3+O4U3wP5vZTQB2A/hKd6YohBBCCCFEdzjmZjiE8CiAZtv1Szs7HSGEEEIIIXpHZVtv+U5H/LH/q6++moy9/fbbMfbfLJ8xY0aMuTuOUuujJ3eecl2rOB3HaSD/e96rNWvWrBizlr3sclM3iq6FVr4Rzt+y5vSeT+HxN6L983GVAU+zOWvtNqdoajqnA997fSe5s846K8ZshQLS9K+32/B9P9ftq0zkrB9cZcF3cOMqEXzOuOMckJ7f3DXP91AAmD59eozZkrZo0aLkOJ5XrvpHK9apZscNOrm/ibU877zzkjGuEsE2L18xgi0wbC8D0vc97uIIpBYVfu1LLrkkOY619R0P2V7RbsfHXmqpXYAQQgghhKgt2gwLIYQQQojaos2wEEIIIYSoLaXzDBf1mebGfNke7kzmPSoXXHBBjLmET87jWKZyLoNEM09fTldfLoa9TtyZCkg7Mp188skxlpadpdN+zKIeVO9L43XO/jsgX6aJHw9K2Z9BoF1di66nXBc79jT6Mk2sl/eZNnutnIexTPh587nwJc2uuuqqGM+dOzfGTz31VHIcd/Hk72cA6f3W31/nzJkTY/6ujdcrt/Zy662sGhXF/318LfuyrpdffnmMuVQdd9MF0hJsvmMg+7N9GbuFCxfGmP36/jtVOS07/T7abf31ybAQQgghhKgt2gwLIYQQQojaUjqbRFH8R+pcfmRoaCgZ+8Y3vhFjX8aLU61FS4WI9mh2Dv255pT30qVLkzHueuN/j1N80rL/FO0OloPXtS/1xKlatsUAaRkoX4KP1zyn/uqWtgU6Y3kp+hx8Pn1XQO4YeOTIkWSMNfKluthewWnnqmiX+zu8ZYTXAFvG5s+fnxzH5ej8+yFrWbTTYCvp805TJp1zc/Xrga/rT3ziEzGeN29echyXnvTrkB/798Bm+rVbIq1dVFpNCCGEEEKIHqDNsBBCCCGEqC3aDAshhBBCiNpSKc9wzl/CvhfvYfKl1tp5ftFZcq16+bEvwcSteos+v+gsRc9tUf+hp5nf218LvrQUw9eJL7vG/ry6l1Zr52/23sScN5zPL5933x6Wy1/6MlN8//ZZ5+vrAAAESElEQVRlvJq1aq7q9wSK/h25MnPenyr6Qzttiv17pb8niubok2EhhBBCCFFbtBkWQgghhBC1pXQ2iVzqoBNlgKqSLisDOtf1pmgaMGeh4NS3tztxKancc7STjhTN8ecsV8Kpmc6+syR3y/K0U8ZLugohGH0yLIQQQgghaos2w0IIIYQQorZoMyyEEEIIIWpL6TzDOeQDE6J61LENcpVoxxvuvb/tPr8QQhRBnwwLIYQQQojaos2wEEIIIYSoLdaJcmSFX8zsZQC7AUwCcLhnL9ycOs1jRghhcqeerKHl26jP+StKqbTUmmxKr+YhLbuPtBw9dZtHp7UclPdKoF5aFtaxp5vh+KJm60IIi3v+wppHRxmUeQ/KPIDBmksrDMq8NY/RMyhz1zxGz6DMXfMYHYM070GZy6DM4yiySQghhBBCiNqizbAQQgghhKgt/doMf7dPr+vRPEbHoMx7UOYBDNZcWmFQ5q15jJ5BmbvmMXoGZe6ax+gYpHkPylwGZR4A+uQZFkIIIYQQYhCQTUIIIYQQQtSWnm6GzexKM9tqZs+b2c09fu3vmdkhM9tIP5tgZveb2fbGf8d3eQ7TzWylmW02s01m9qf9mEcn6JeWg6Bj4zWl5ehfV1p2kLrfXxuvKS1H/9p917IqOgK6v5ZFy55ths3seAD/B8AyAPMAXG9m83r1+gBuAXCl+9nNAFaEEGYDWNF43E3eB/DNEMI8AEsA/EnjHPR6HqOiz1regv7rCEjLTnALpGVH0P01Ii1Hzy3ov5al1xHou5a3oP86AmXRMoTQk38APgXgPnr8bQDf7tXrN15zCMBGerwVwNRGPBXA1h7P5y4Al/d7HmXTctB0lJbSst//+q2jtJSW0nHwtBw0HQdZy17aJKYBeJEe7238rJ9MCSHsb8QHAEzp1Qub2RCAhQAe6+c82mTQtOzr+ZOWHUVatseg6QhIy3aRlkSJdQQGT0utySboC3QNwvD/nvSktIaZjQVwO4A/CyG82a95VJFenz9p2T2kZXWQltVB75XVQGsypZeb4X0AptPjMxo/6ycHzWwqADT+e6jbL2hmYzB8QdwWQrijX/MYJYOmZV/On7TsCtKyPQZNR0Batou0RCV0BAZPS63JJvRyM/wEgNlmNtPMTgDwVQB39/D1R+JuADc24hsx7GXpGmZmAP4BwJYQwt/2ax4dYNC07Pn5k5ZdQ1q2x6DpCEjLdqm9lhXRERg8LbUmm9Fj4/QXAGwDsAPAf+nxa/8IwH4A72HYt3MTgIkY/hbjdgC/BDChy3P4DIZTAc8AeKrx7wu9nkeZtRwEHaWltBzEf3W/v0rL6mhZFR37qeUg6FgmLdWBTgghhBBC1BZ9gU4IIYQQQtQWbYaFEEIIIURt0WZYCCGEEELUFm2GhRBCCCFEbdFmWAghhBBC1BZthoUQQgghRG3RZlgIIYQQQtQWbYaFEEIIIURt+f+UtoefRFpU0wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x180 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "z1 = z[:,0]\n",
    "z2 = z[:,1]\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "ax.plot(z1,z2,'ko')\n",
    "plt.title(\"latent space\")\n",
    "\n",
    "#np.where((z1>3) & (z2<2) & (z2>0))\n",
    "#select the points from the latent space\n",
    "a_vec = [2,5,7,789,25,9993]\n",
    "for i in range(len(a_vec)):\n",
    "    ax.plot(z1[a_vec[i]],z2[a_vec[i]],'ro')  \n",
    "    ax.annotate('z%d' %i, xy=(z1[a_vec[i]],z2[a_vec[i]]), \n",
    "                xytext=(z1[a_vec[i]],z2[a_vec[i]]),color = 'r',fontsize=15)\n",
    "\n",
    "\n",
    "f, ((ax0, ax1, ax2, ax3, ax4,ax5)) = plt.subplots(1,6,  sharex='col', sharey='row',figsize=(12,2.5))\n",
    "for i in range(len(a_vec)):\n",
    "    eval('ax%d' %(i)).imshow(np.reshape(x_construction[a_vec[i],:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n",
    "    eval('ax%d' %(i)).set_title('z%d'%i)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above is a plot of points in the 2D latent space and their corresponding decoded images, it can be seen that points that are close in the latent space get mapped to the same digit from the decoder, and we can see how it evolves from left to right."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}