scripts/nn/examples/Example - Neural Collaborative Filtering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Collaborative Filtering (NCF)\n",
    "\n",
    "This examples trains a neural network on the MovieLens data set using the concept of [Neural Collaborative Filtering (NCF)](https://dl.acm.org/doi/abs/10.1145/3038912.3052569) that is aimed at approaching recommendation problems using deep neural networks as opposed to common matrix factorization approaches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup and Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download Data - MovieLens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MovieLens data set is provided by the Unniversity of Minnesota and the GroupLens Research Group:\n",
    "\n",
    "> This dataset (ml-latest-small) describes 5-star rating and free-text tagging activity from [MovieLens](http://movielens.org/), a movie recommendation service. It contains 100836 ratings and 3683 tag applications across 9742 movies. These data were created by 610 users between March 29, 1996 and September 24, 2018. This dataset was generated on September 26, 2018.<br/>\n",
    "Users were selected at random for inclusion. All selected users had rated at least 20 movies. No demographic information is included. Each user is represented by an id, and no other information is provided.<br/>\n",
    "The data are contained in the files links.csv, movies.csv, ratings.csv and tags.csv. More details about the contents and use of all these files follows.<br/>\n",
    "This is a development dataset. As such, it may change over time and is not an appropriate dataset for shared research results. See available benchmark datasets if that is your intent.<br/>\n",
    "This and other GroupLens data sets are publicly available for download at http://grouplens.org/datasets/."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  ml-latest-small.zip\n",
      "   creating: ml-latest-small/\n",
      "  inflating: ml-latest-small/links.csv  \n",
      "  inflating: ml-latest-small/tags.csv  \n",
      "  inflating: ml-latest-small/ratings.csv  \n",
      "  inflating: ml-latest-small/README.txt  \n",
      "  inflating: ml-latest-small/movies.csv  \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "  5  955k    5 50411    0     0  68679      0  0:00:14 --:--:--  0:00:14 68586\r",
      "100  955k  100  955k    0     0   640k      0  0:00:01  0:00:01 --:--:--  640k\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "DATASET=ml-latest-small\n",
    "\n",
    "mkdir -p data/$DATASET/\n",
    "cd data/$DATASET\n",
    "curl -O http://files.grouplens.org/datasets/movielens/$DATASET.zip\n",
    "unzip $DATASET.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_loc = \"data/ml-latest-small/ml-latest-small/\"\n",
    "negative_split = 1.5  # how many negatives for one positive\n",
    "\n",
    "# load interactions from MovieLens\n",
    "raw_ratings = pd.read_csv(data_loc + \"ratings.csv\")\n",
    "positives = pd.DataFrame(raw_ratings, columns=['userId', 'movieId'])\n",
    "\n",
    "# sample negatives\n",
    "negatives = pd.DataFrame(columns=[\"userId\", \"movieId\"])\n",
    "\n",
    "while len(negatives) < len(positives) * negative_split:\n",
    "    user = positives[\"userId\"].sample().values[0]\n",
    "    movie = positives[\"movieId\"].sample().values[0]\n",
    "    if len(positives.loc[(positives[\"userId\"] == user) & (positives[\"movieId\"] == movie)]) == 0:\n",
    "        negatives = negatives.append({\"userId\": user, \"movieId\": movie}, ignore_index=True)\n",
    "\n",
    "# write out final data\n",
    "targets = np.hstack([np.ones(len(positives)), np.zeros(len(negatives))])\n",
    "all_ratings = np.vstack([positives, negatives])\n",
    "\n",
    "user_item_targets = np.hstack([all_ratings, targets[:, np.newaxis]])\n",
    "\n",
    "np.random.shuffle(user_item_targets)\n",
    "\n",
    "split = train_test_split(user_item_targets, train_size=0.8)\n",
    "\n",
    "np.savetxt(data_loc + \"sampled-train.csv\", split[0], delimiter=\",\")\n",
    "np.savetxt(data_loc + \"sampled-test.csv\", split[1], delimiter=\",\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SystemDS NCF implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### with synthetic dummy data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using user supplied systemds jar file target/SystemDS.jar\n",
      "###############################################################################\n",
      "#  SYSTEMDS_ROOT= .\n",
      "#  SYSTEMDS_JAR_FILE= target/SystemDS.jar\n",
      "#  CONFIG_FILE= --config ./target/testTemp/org/apache/sysds/api/mlcontext/MLContext/SystemDS-config.xml\n",
      "#  LOG4JPROP= -Dlog4j.configuration=file:conf/log4j-silent.properties\n",
      "#  CLASSPATH= target/SystemDS.jar:./lib/*:./target/lib/*\n",
      "#  HADOOP_HOME= /Users/patrick/Uni Offline/Architectures of Machine Learning Systems (AMLS)/systemml/target/lib/hadoop\n",
      "#\n",
      "#  Running script scripts/nn/examples/ncf-dummy-data.dml locally with opts: \n",
      "###############################################################################\n",
      "Executing command:     java       -Xmx4g      -Xms4g      -Xmn400m   -cp target/SystemDS.jar:./lib/*:./target/lib/*   -Dlog4j.configuration=file:conf/log4j-silent.properties   org.apache.sysds.api.DMLScript   -f scripts/nn/examples/ncf-dummy-data.dml   -exec singlenode   --config ./target/testTemp/org/apache/sysds/api/mlcontext/MLContext/SystemDS-config.xml   \n",
      "\n",
      "NCF training starting with 1000 training samples, 100 validation samples, 50 items and 60 users...\n",
      "Epoch: 1, Iter: 1, Train Loss: 0.6953457411615849, Train Accuracy: 0.5, Val Loss: 0.6995101788248107, Val Accuracy: 0.47\n",
      "Epoch: 2, Iter: 1, Train Loss: 0.6667911468574823, Train Accuracy: 0.6875, Val Loss: 0.6992050630414124, Val Accuracy: 0.47\n",
      "Epoch: 3, Iter: 1, Train Loss: 0.6570450250431727, Train Accuracy: 0.6875, Val Loss: 0.7014387912966833, Val Accuracy: 0.47\n",
      "Epoch: 4, Iter: 1, Train Loss: 0.6521926651745862, Train Accuracy: 0.6875, Val Loss: 0.7053126102214489, Val Accuracy: 0.43999999999999995\n",
      "Epoch: 5, Iter: 1, Train Loss: 0.6431405119563119, Train Accuracy: 0.6875, Val Loss: 0.7115121778198469, Val Accuracy: 0.43999999999999995\n",
      "Epoch: 6, Iter: 1, Train Loss: 0.6353498336109219, Train Accuracy: 0.6875, Val Loss: 0.7193490066131873, Val Accuracy: 0.44999999999999996\n",
      "Epoch: 7, Iter: 1, Train Loss: 0.6308046978859394, Train Accuracy: 0.6875, Val Loss: 0.7306240107462888, Val Accuracy: 0.48\n",
      "Epoch: 8, Iter: 1, Train Loss: 0.6260145322748087, Train Accuracy: 0.75, Val Loss: 0.7435853055111923, Val Accuracy: 0.49\n",
      "Epoch: 9, Iter: 1, Train Loss: 0.6163475345953953, Train Accuracy: 0.6875, Val Loss: 0.757023909929672, Val Accuracy: 0.5\n",
      "Epoch: 10, Iter: 1, Train Loss: 0.6029424406867099, Train Accuracy: 0.6875, Val Loss: 0.7749021987872134, Val Accuracy: 0.51\n",
      "Epoch: 11, Iter: 1, Train Loss: 0.5791958103856243, Train Accuracy: 0.8125, Val Loss: 0.7921418272873325, Val Accuracy: 0.51\n",
      "Epoch: 12, Iter: 1, Train Loss: 0.5543597535155846, Train Accuracy: 0.8125, Val Loss: 0.8131440342665028, Val Accuracy: 0.5\n",
      "Epoch: 13, Iter: 1, Train Loss: 0.5342062981571314, Train Accuracy: 0.8125, Val Loss: 0.8340415360672659, Val Accuracy: 0.45999999999999996\n",
      "Epoch: 14, Iter: 1, Train Loss: 0.5156903349054259, Train Accuracy: 0.875, Val Loss: 0.8534000391024407, Val Accuracy: 0.47\n",
      "Epoch: 15, Iter: 1, Train Loss: 0.5042912981017884, Train Accuracy: 0.8125, Val Loss: 0.873901869293276, Val Accuracy: 0.44999999999999996\n",
      "Epoch: 16, Iter: 1, Train Loss: 0.48722704019844537, Train Accuracy: 0.8125, Val Loss: 0.898510539121238, Val Accuracy: 0.47\n",
      "Epoch: 17, Iter: 1, Train Loss: 0.47048381704431463, Train Accuracy: 0.875, Val Loss: 0.9284775525937294, Val Accuracy: 0.48\n",
      "Epoch: 18, Iter: 1, Train Loss: 0.45151030675588855, Train Accuracy: 0.875, Val Loss: 0.9574504971357228, Val Accuracy: 0.47\n",
      "Epoch: 19, Iter: 1, Train Loss: 0.43940495503523824, Train Accuracy: 0.875, Val Loss: 0.9937811553464448, Val Accuracy: 0.45999999999999996\n",
      "Epoch: 20, Iter: 1, Train Loss: 0.42553379542786246, Train Accuracy: 0.875, Val Loss: 1.0231502880025147, Val Accuracy: 0.43999999999999995\n",
      "Epoch: 21, Iter: 1, Train Loss: 0.4163223594480222, Train Accuracy: 0.875, Val Loss: 1.0595479122098816, Val Accuracy: 0.45999999999999996\n",
      "Epoch: 22, Iter: 1, Train Loss: 0.4050461773338017, Train Accuracy: 0.875, Val Loss: 1.0944624240337406, Val Accuracy: 0.48\n",
      "Epoch: 23, Iter: 1, Train Loss: 0.3957080838041942, Train Accuracy: 0.875, Val Loss: 1.1315613394576827, Val Accuracy: 0.47\n",
      "Epoch: 24, Iter: 1, Train Loss: 0.39252816032717697, Train Accuracy: 0.8125, Val Loss: 1.1608315131205158, Val Accuracy: 0.47\n",
      "Epoch: 25, Iter: 1, Train Loss: 0.38656611677400526, Train Accuracy: 0.8125, Val Loss: 1.2010764396137235, Val Accuracy: 0.45999999999999996\n",
      "Epoch: 26, Iter: 1, Train Loss: 0.3910140006546419, Train Accuracy: 0.8125, Val Loss: 1.2394434665872176, Val Accuracy: 0.44999999999999996\n",
      "Epoch: 27, Iter: 1, Train Loss: 0.39012809759646405, Train Accuracy: 0.8125, Val Loss: 1.267704284952889, Val Accuracy: 0.43999999999999995\n",
      "Epoch: 28, Iter: 1, Train Loss: 0.3986668930898999, Train Accuracy: 0.8125, Val Loss: 1.3134788291583197, Val Accuracy: 0.44999999999999996\n",
      "Epoch: 29, Iter: 1, Train Loss: 0.39096586484137014, Train Accuracy: 0.8125, Val Loss: 1.3457368548231847, Val Accuracy: 0.44999999999999996\n",
      "Epoch: 30, Iter: 1, Train Loss: 0.3913665786483714, Train Accuracy: 0.8125, Val Loss: 1.395200160764677, Val Accuracy: 0.44999999999999996\n",
      "Epoch: 31, Iter: 1, Train Loss: 0.39306020872450564, Train Accuracy: 0.8125, Val Loss: 1.4547617764166234, Val Accuracy: 0.44999999999999996\n",
      "Epoch: 32, Iter: 1, Train Loss: 0.3961123079325197, Train Accuracy: 0.8125, Val Loss: 1.4988918781732432, Val Accuracy: 0.45999999999999996\n",
      "Epoch: 33, Iter: 1, Train Loss: 0.39167597788728836, Train Accuracy: 0.875, Val Loss: 1.5580225154760752, Val Accuracy: 0.44999999999999996\n",
      "Epoch: 34, Iter: 1, Train Loss: 0.3936826951721131, Train Accuracy: 0.875, Val Loss: 1.592168642509798, Val Accuracy: 0.43999999999999995\n",
      "Epoch: 35, Iter: 1, Train Loss: 0.39446093556125095, Train Accuracy: 0.8125, Val Loss: 1.6504423270813886, Val Accuracy: 0.43000000000000005\n",
      "Epoch: 36, Iter: 1, Train Loss: 0.3917767876760818, Train Accuracy: 0.8125, Val Loss: 1.6894229810333048, Val Accuracy: 0.43000000000000005\n",
      "Epoch: 37, Iter: 1, Train Loss: 0.3936299068718723, Train Accuracy: 0.8125, Val Loss: 1.7342536990495687, Val Accuracy: 0.43000000000000005\n",
      "Epoch: 38, Iter: 1, Train Loss: 0.4086856463043926, Train Accuracy: 0.8125, Val Loss: 1.7709575584324264, Val Accuracy: 0.44999999999999996\n",
      "Epoch: 39, Iter: 1, Train Loss: 0.3946728895715752, Train Accuracy: 0.8125, Val Loss: 1.8323990419424212, Val Accuracy: 0.43000000000000005\n",
      "Epoch: 40, Iter: 1, Train Loss: 0.4092882424416999, Train Accuracy: 0.8125, Val Loss: 1.8647938002160964, Val Accuracy: 0.44999999999999996\n",
      "Epoch: 41, Iter: 1, Train Loss: 0.4050641439255627, Train Accuracy: 0.8125, Val Loss: 1.891264442380163, Val Accuracy: 0.44999999999999996\n",
      "Epoch: 42, Iter: 1, Train Loss: 0.4170644006779869, Train Accuracy: 0.8125, Val Loss: 1.9423174900115594, Val Accuracy: 0.44999999999999996\n",
      "Epoch: 43, Iter: 1, Train Loss: 0.3923480753991977, Train Accuracy: 0.8125, Val Loss: 1.9731695043639572, Val Accuracy: 0.44999999999999996\n",
      "Epoch: 44, Iter: 1, Train Loss: 0.40490676281916327, Train Accuracy: 0.8125, Val Loss: 2.010804834458905, Val Accuracy: 0.44999999999999996\n",
      "Epoch: 45, Iter: 1, Train Loss: 0.40181821707001014, Train Accuracy: 0.8125, Val Loss: 2.051962004205519, Val Accuracy: 0.44999999999999996\n",
      "Epoch: 46, Iter: 1, Train Loss: 0.40355348381441153, Train Accuracy: 0.8125, Val Loss: 2.0891022279849456, Val Accuracy: 0.44999999999999996\n",
      "Epoch: 47, Iter: 1, Train Loss: 0.38715605504077866, Train Accuracy: 0.8125, Val Loss: 2.117280026954698, Val Accuracy: 0.44999999999999996\n",
      "Epoch: 48, Iter: 1, Train Loss: 0.39836973023268446, Train Accuracy: 0.8125, Val Loss: 2.141835697116999, Val Accuracy: 0.43999999999999995\n",
      "Epoch: 49, Iter: 1, Train Loss: 0.3901144594871556, Train Accuracy: 0.8125, Val Loss: 2.176511579483428, Val Accuracy: 0.43999999999999995\n",
      "Epoch: 50, Iter: 1, Train Loss: 0.3917649057215277, Train Accuracy: 0.8125, Val Loss: 2.2288326304130806, Val Accuracy: 0.43999999999999995\n",
      "NCF training completed after 50 epochs\n",
      "SystemDS Statistics:\n",
      "Total execution time:\t\t9.206 sec.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "20/05/28 15:04:03 INFO api.DMLScript: BEGIN DML run 05/28/2020 15:04:03\n",
      "20/05/28 15:04:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "20/05/28 15:04:13 INFO api.DMLScript: END DML run 05/28/2020 15:04:13\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cd ../../..\n",
    "bin/systemds target/SystemDS.jar scripts/nn/examples/ncf-dummy-data.dml > scripts/nn/examples/run_log.txt && cat scripts/nn/examples/run_log.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### with real data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "cd ../../..\n",
    "bin/systemds target/SystemDS.jar scripts/nn/examples/ncf-real-data.dml > scripts/nn/examples/run_log.txt && cat scripts/nn/examples/run_log.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ### Plot training results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nOzdd3iUVdrA4d+ZyaT3RkkhlNARkEgRFAGVpuKqiCgqrIgd0NXVteEq9l39cEUUBBFEXZXFVYqIIrKA9N4DoSQBQkjv0873x4whQkICTJhk8tzXNdfMvPV5h/DMmXPOe47SWiOEEMJzGdwdgBBCiNoliV4IITycJHohhPBwkuiFEMLDSaIXQggP5+XuACoTGRmpExIS3B2GEELUG5s2bTqltY6qbF2dTPQJCQls3LjR3WEIIUS9oZQ6UtU6qboRQggPJ4leCCE8nCR6IYTwcNXW0Sul4oA5QCNAA9O11lPO2OYu4GlAAQXAQ1rrbc51h53LbIBVa510IYFaLBbS0tIoLS29kN2Fk6+vL7GxsZhMJneHIoS4RGrSGGsF/qK13qyUCgI2KaWWaa13V9jmENBXa52jlBoMTAd6VFjfT2t96mICTUtLIygoiISEBJRSF3OoBktrTVZWFmlpaTRv3tzd4QghLpFqq2601se11pudrwuAPUDMGdus0VrnON+uBWJdHWhpaSkRERGS5C+CUoqIiAj5VSREA3NedfRKqQSgK7DuHJvdByyp8F4DPyqlNimlxp3j2OOUUhuVUhszMzOr2uZ8whWVkM9QiIanxoleKRUIzAcmaq3zq9imH45E/3SFxX201pcDg4FHlFJXV7av1nq61jpJa50UFVVpn38hhPBY2zK3MXvn7Fo5do0SvVLKhCPJz9Na/6eKbS4DPgaGaa2zfl+utU53Pp8EFgDdLzZoIYTwJCvTVjJ26Vi+3v81xZZilx+/2kSvHL/1ZwJ7tNbvVLFNPPAf4G6t9f4KywOcDbgopQKA64Gdrgj8UsvNzeWDDz447/2GDBlCbm7uee83evRovvnmm/PeTwhRvyxIXsD45eNpEdqCOYPn4G/yd/k5alKi7w3cDfRXSm11PoYopR5USj3o3OZFIAL4wLn+9/ELGgGrlFLbgPXAIq31D66+iEuhqkRvtVrPud/ixYsJDQ2trbCEEPWU1pqPd3zMi2tepHvj7swaOIsIv4haOVe13Su11qtw9I8/1zZjgbGVLE8BOl9wdFX4+/e72H2s0maCC9a+aTCTbuxQ5fpnnnmGgwcP0qVLF0wmE76+voSFhbF3717279/PzTffTGpqKqWlpUyYMIFx4xztzr+P21NYWMjgwYPp06cPa9asISYmhv/+97/4+flVG9vPP//Mk08+idVq5YorrmDatGn4+PjwzDPP8N133+Hl5cX111/PP/7xD77++mv+/ve/YzQaCQkJYeXKlS77jIQQrmHXdt7a8Bbz9sxjcPPBvNr7VUzG2ru3pU4OalYXvfHGG+zcuZOtW7eyYsUKhg4dys6dO8v7o8+aNYvw8HBKSkq44ooruPXWW4mI+OO3c3JyMl988QUzZszg9ttvZ/78+YwaNeqc5y0tLWX06NH8/PPPtG7dmnvuuYdp06Zx9913s2DBAvbu3YtSqrx66OWXX2bp0qXExMRcUJWREKJ2mW1mnlv1HD8c/oFR7Ubx1BVPYVC1O0hBvUz05yp5Xyrdu3f/w01H7733HgsWLAAgNTWV5OTksxJ98+bN6dKlCwDdunXj8OHD1Z5n3759NG/enNatWwNw7733MnXqVB599FF8fX257777uOGGG7jhhhsA6N27N6NHj+b222/nlltuccWlCiFcZH/Ofl757RW2Zm7l8W6PM6bDmEvS5VnGurlAAQEB5a9XrFjBTz/9xG+//ca2bdvo2rVrpTcl+fj4lL82Go3V1u+fi5eXF+vXr+e2225j4cKFDBo0CIAPP/yQyZMnk5qaSrdu3cjKyqrmSEKI2lZgLuDN9W9y+/e3cyj/EG9e9SZ/7vjnS3ZfS70s0btDUFAQBQUFla7Ly8sjLCwMf39/9u7dy9q1a1123jZt2nD48GEOHDhAq1atmDt3Ln379qWwsJDi4mKGDBlC7969adGiBQAHDx6kR48e9OjRgyVLlpCamnrWLwshxKWhtWZhykLe2fQOWSVZ3Nr6ViZ0nUCo76XtoCGJvoYiIiLo3bs3HTt2xM/Pj0aNGpWvGzRoEB9++CHt2rWjTZs29OzZ02Xn9fX15ZNPPmH48OHljbEPPvgg2dnZDBs2jNLSUrTWvPOOo+frU089RXJyMlprBgwYQOfOLm8LF0LUwL7sfby27jU2n9xMx4iO/Kv/v+gY2dEtsSittVtOfC5JSUn6zBmm9uzZQ7t27dwUkWeRz1KI2mHXdlalr+LzPZ+z+thqQnxCmHj5RG5JvKXWG1yVUpuqGh1YSvRCCHGR8s35LEhewJd7vyStMI0ovyge7vIwI9uMvOTVNJWRRO9mjzzyCKtXr/7DsgkTJjBmzBg3RSSEqKmj+UeZvWs2C1MWUmIt4fLoy5lw+QQGNBuAyVB35nyQRO9mU6dOdXcIQojzlJKbwvQd01lyaAkmg4mhLYZyR5s7aBdRN6tEJdELIUQN7cvex0fbP+KnIz/h6+XLPe3v4d4O9xLpF+nu0M5JEr0QQlTjYO5B/m/z/7EidQWBpkDGdhrL3e3vJsw3zN2h1YgkeiGEqILZZmbmjplM3zEdPy8/Hu7yMHe1u4tg72B3h3ZeJNELIUQltp7cyktrXuJg3kGGNB/CX6/4a62NLlnbZAiEGrrU49ELIdyjyFLE6+te554l91BkLWLqgKm8efWb9TbJgyT6GpPx6IXwfGvS13Dzf2/mi71fMLLtSL4d9i1Xx1Y6+2m9Uj+rbpY8Ayd2uPaYjTvB4DeqXH2px6OfMWMG06dPx2w2l49x4+/vT0ZGBg8++CApKSkATJs2jSuvvJI5c+bwj3/8A6UUl112GXPnznXt5yOEByuzlfHupneZt2ceLUIcMz11ie7i7rBcRkr0NfTGG2/QsmVLtm7dyttvv83mzZuZMmUK+/c7Zk6cNWsWmzZtYuPGjbz33nuVjhqZnJzMI488wq5duwgNDWX+/PlVnu+WW25hw4YNbNu2jXbt2jFz5kwAxo8fT9++fdm2bRubN2+mQ4cO7Nq1i8mTJ7N8+XK2bdvGlClTaudDEMID7c/Zzx0L72Dennnc1e4u/n3Dvz0qyUN9LdGfo+R9qdT2ePQ7d+7k+eefJzc3l8LCQgYOHAjA8uXLmTNnDkD5LFJz5sxh+PDhREY6+vKGh4e77DqF8FR2befzPZ/z7qZ3CfIO4oMBH3BV7FXuDqtW1GRy8Dil1C9Kqd1KqV1KqQmVbKOUUu8ppQ4opbYrpS6vsO5epVSy83Gvqy/AXWp7PPrRo0fz/vvvs2PHDiZNmlTp8YQQFyazOJOHf3qYNze8Sa+mvZh/03yPTfJQs6obK/AXrXV7oCfwiFKq/RnbDAYSnY9xwDQApVQ4MAnoAXQHJiml6scdBme41OPRFxQU0KRJEywWC/PmzStfPmDAAKZNmwaAzWYjLy+P/v378/XXX5dXF2VnZ1/0+YXwRFprvj/4PX/67k9sytjE8z2e51/9/1Wve9TURLWJXmt9XGu92fm6ANgDxJyx2TBgjnZYC4QqpZoAA4FlWutsrXUOsAwY5NIruEQqjkf/1FNP/WHdoEGDsFqttGvXjmeeecYl49G/8sor9OjRg969e9O2bdvy5VOmTOGXX36hU6dOdOvWjd27d9OhQweee+45+vbtS+fOnXniiScu+vxCeJoTRSd4+OeHeXbVsyQEJ/DvG/7NiLYjLtksT+50XuPRK6USgJVAR611foXlC4E3tNarnO9/Bp4GrgF8tdaTnctfAEq01v+o5NjjcPwaID4+vtuRI0f+sF7GUHcd+SxFQ2LXdr7e9zXvbn4Xu7Yzvut4RrYdidFgdHdoLuWS8eiVUoHAfGBixSTvKlrr6cB0cEw84urjCyEaniP5R5i0ZhKbMjbRo0kPXur1ErFBse4O65KrUaJXSplwJPl5Wuv/VLJJOhBX4X2sc1k6jlJ9xeUrLiRQTyXj0QvhermluczYMYMv936Jj9GHl698mZtb3dwgqmkqU22iV45PZiawR2v9ThWbfQc8qpT6EkfDa57W+rhSainwWoUG2OuBv7kgbo8h49EL4TrFlmLm7p7L7F2zKbYWc2OLGxl/+Xii/aPdHZpb1aRE3xu4G9ihlNrqXPYsEA+gtf4QWAwMAQ4AxcAY57pspdQrwAbnfi9rraVLiBDCpSw2C98kf8NH2z4iqzSL/nH9eazrY7QKa+Xu0OqEahO9s4H1nL93tKNF95Eq1s0CZl1QdEIIUY0dmTv468q/klaYRrdG3fi/fv/ncXe2Xqz6eWesEEIAK1JX8NSvTxHhF8G0a6fRu2nvBlsPfy6S6IUQ9dJX+77i1XWv0i68He8PeL/OT+fnTjKoWS0JDAysct3hw4fp2LHjJYxGCM+htWbK5im8svYV+sT0YdbAWZLkqyEleiFEvWGxWXhxzYssTFnIrYm38nzP5/EySBqrTr38hN5c/yZ7s/e69Jhtw9vydPenq1z/zDPPEBcXxyOPONqcX3rpJby8vPjll1/IycnBYrEwefJkhg0bdl7nLS0t5aGHHmLjxo14eXnxzjvv0K9fP3bt2sWYMWMwm83Y7Xbmz59P06ZNuf3220lLS8Nms/HCCy8wYsSIi7puIeqL3NJcnlz5JOuOr+Oxro9xf6f7pT6+huploneHESNGMHHixPJE/9VXX7F06VLGjx9PcHAwp06domfPntx0003n9cc3depUlFLs2LGDvXv3cv3117N//34+/PBDJkyYwF133YXZbMZms7F48WKaNm3KokWLAMdgakJ4umOFx5i7ey7zk+djsVl4tc+r3NTyJneHVa/Uy0R/rpJ3benatSsnT57k2LFjZGZmEhYWRuPGjXn88cdZuXIlBoOB9PR0MjIyaNy4cY2Pu2rVKh577DEA2rZtS7Nmzdi/fz+9evXi1VdfJS0tjVtuuYXExEQ6derEX/7yF55++mluuOEGrrrKc4dVFWJP1h4+2fUJPx7+EYVicPPBjOk4hsSwRHeHVu/Uy0TvLsOHD+ebb77hxIkTjBgxgnnz5pGZmcmmTZswmUwkJCS4bNz4O++8kx49erBo0SKGDBnCRx99RP/+/dm8eTOLFy/m+eefZ8CAAbz44osuOZ8QdcWGExuYvn06a4+vJcAUwKh2oxjVfhSNA2pegBJ/JIn+PIwYMYL777+fU6dO8euvv/LVV18RHR2NyWTil19+4cwRN2viqquuYt68efTv35/9+/dz9OhR2rRpQ0pKCi1atGD8+PEcPXqU7du307ZtW8LDwxk1ahShoaF8/PHHtXCVQrhHSl4K7258lxVpK4j2i+bxbo8zvPVwgryD3B1avSeJ/jx06NCBgoICYmJiaNKkCXfddRc33ngjnTp1Iikp6Q/jxtfUww8/zEMPPUSnTp3w8vJi9uzZ+Pj48NVXXzF37lxMJhONGzfm2WefZcOGDTz11FMYDAZMJlP5BCRC1Gc5pTl8sPUDvt7/Nb5evky8fCKj2o/Cx+hT/c6iRs5rPPpLJSkpSW/cuPEPy2QMddeRz1LUBWW2Mj7f8zkzts+g2FrMba1v46HOD3n8bE+1xSXj0QshhKucKjnFfUvvIyUvhatjr+Yv3f5Ci9AW7g7LY0mir0U7duzg7rvv/sMyHx8f1q1b56aIhHC/vLI8Hlj2AMeLjjN1wFSujr3a3SF5PEn0tahTp05s3bq1+g2FaCCKLcU8/PPDHMo7xPsD3ufKple6O6QGQRK9EOKSKLOVMX75eHad2sU/r/mnJPlLSBK9EKLWWewWnlzxJOtOrOO1Pq8xIH6Au0NqUGT0SiFErbJrO8+vep4VaSt4rsdz3NjyRneH1ODUZM7YWcANwEmt9Vlj6yqlngLuqnC8dkCUcxrBw0ABYAOsVXX9EUJ4JqvdymvrXmPxocVMuHwCd7S9w90hNUg1KdHPBgZVtVJr/bbWuovWuguOib9/PWNe2H7O9Q0qyZ9rPHohGoI9WXu4a/FdfL3/a+7reB9jO411d0gNVk3mjF2plEqo4fFGAl9cTEBCiPqt2FLMtG3TmLt7LqE+obzd920GNhvo7rAaNJc1xiql/HGU/B+tsFgDPyqlNPCR1nq6K8514rXXKNvj2vHofdq1pfGzz1a53pXj0RcWFjJs2LBK95szZw7/+Mc/UEpx2WWXMXfuXDIyMnjwwQdJSUkBYNq0aVx5pfRYEHXPmvQ1vLz2ZdIL07k18VYe7/Y4IT4h7g6rwXNlr5sbgdVnVNv00VqnK6WigWVKqb1a65WV7ayUGgeMA4iPj3dhWK7hyvHofX19WbBgwVn77d69m8mTJ7NmzRoiIyPJznZ8lOPHj6dv374sWLAAm81GYWFhrV+vEOcjryyPN9a/wcKUhSQEJ/DJwE9IatygamvrNFcm+js4o9pGa53ufD6plFoAdAcqTfTO0v50cIx1c64TnavkXVtcOR691ppnn332rP2WL1/O8OHDiYx0zH8ZHh4OwPLly5kzZw4ARqORkBApIYm6Y3fWbp5Y8QQZxRk82PlBxnYaKwOS1TEuSfRKqRCgLzCqwrIAwKC1LnC+vh542RXncxdXjUdfm+PYC3Ep/Sf5P7y69lXC/cL5dNCnXBZ1mbtDEpWotteNUuoL4DegjVIqTSl1n1LqQaXUgxU2+xPwo9a6qMKyRsAqpdQ2YD2wSGv9gyuDv9RGjBjBl19+yTfffMPw4cPJy8u7oPHoq9qvf//+fP3112RlZQGUV90MGDCgfEhim80mUwgKtyu1lvLC6heYtGYS3Rp146sbvpIkX4fVpNfNyBpsMxtHN8yKy1KAzhcaWF3kqvHoq9qvQ4cOPPfcc/Tt2xej0UjXrl2ZPXs2U6ZMYdy4ccycOROj0ci0adPo1atXbV6qEFVKLUjliRVPsDd7Lw9c9gAPdX4Io8Ho7rDEOch49A2QfJbiQv105CdeXOOYvvKNq96QkSfrEBmPXghxUfLK8srvcG0f0Z5/9v0nsUGx7g5L1JAk+lok49ELT/Br6q+89NtL5Jbm8nCXhxnbaSwmg8ndYYnzUK8Svda62j7qdUldHI++LlbVibop35zPm+vf5LuD35EYlsgHAz6gXYRU+dVH9SbR+/r6kpWVRURERL1K9nWJ1pqsrCx8fX3dHYqo49YdX8ezq54lqySL+zvdz0OdH8JklFJ8fVVvEn1sbCxpaWlkZma6O5R6zdfXl9hYqVsVVfs19VcmrphIfFA8U/pNoWPkWYPWinqm3iR6k8lE8+bN3R2GEB5tZdpKHl/xOG3C2jD9+ukEewe7OyThAjLxiBACgP+l/Y+Jv0wkMSyRj677SJK8B5FEL4RgVfoqJv4ykVahrZh+3XQZcdLDSKIXooFbnb6aCcsn0DK0JTOunyFJ3gNJoheiAVuTvobxy8fTIrSFJHkPVm8aY4UQrpNvzueDrR/w5d4vaRXaihnXSZL3ZJLohWhA7NrOguQFTNk8hTxzHsNbD+exro9JkvdwkuiFaCC2Z27n9XWvszNrJ12ju/K37n+TO10bCEn0Qni4EmsJr697nQUHFhDlF8XrV73O0OZD5Q7zBkQSvRAerNRaymPLH2PDiQ2M6TCGBzo/QIApwN1hiUtMEr0QHqrMVsbEXyay/vh6JveZzE0tb3J3SMJNJNEL4YHMNjNPrHiC1cdW8/KVL0uSb+BqMmfsLKXUSaXUzirWX6OUylNKbXU+XqywbpBSap9S6oBS6hlXBi6EqJzFbuHJX59kZdpKXuj5An9K/JO7QxJuVpMbpmYDg6rZ5n9a6y7Ox8sASikjMBUYDLQHRiql2l9MsEKIc7ParTy98ml+Sf2Fv3X/G7e3ud3dIYk6oNpEr7VeCWRfwLG7Awe01ilaazPwJTDsAo4jhKgBm93Gs/97lmVHlvFU0lPc2e5Od4ck6ghXDYHQSym1TSm1RCnVwbksBkitsE2ac1mllFLjlFIblVIbZcx5Ic5Pdmk2D/30EEsOL2Hi5RO5p8M97g5J1CGuaIzdDDTTWhcqpYYA3wKJ53sQrfV0YDpAUlKSzHcnRA1tObmFJ399ktzSXF7q9RK3tr7V3SGJOuaiS/Ra63ytdaHz9WLApJSKBNKBuAqbxjqXCSFcQGvNp7s+ZcwPY/Ax+vDZkM8kyYtKXXSJXinVGMjQWmulVHccXx5ZQC6QqJRqjiPB3wFIpaEQLpBvzueFVS+wPHU5A+IH8ErvVwjyDnJ3WKKOqjbRK6W+AK4BIpVSacAkwASgtf4QuA14SCllBUqAO7TWGrAqpR4FlgJGYJbWeletXIUQDci+7H1M/GUiJ4pO8FTSU9zd/m4ZzkCck3Lk5LolKSlJb9y40d1hCFHn7Mvex30/3oeP0Yd/9v0nXaK7uDskUUcopTZprZMqWyd3xgpRTxzMPci4ZePwNfoye9BsYoNi3R2SqCdkhikh6oHDeYcZ++NYDMrAzIEzJcmL8yIleiHquNSCVO778T7s2s6sgbNoFtzM3SGJekYSvRB12LHCY4xdOpYyWxkzr59Jy9CW7g5J1EOS6IWoozKKMrhv6X0UmAv4eODHtAlv4+6QRD0ldfRC1EGH8g4xZukYcspy+PC6D2kfIeMBigsniV6IOmbDiQ2MWjyKIksRH133EZdFXebukEQ9J4leiDrkvwf+y7hl44j0i2TekHl0jurs7pCEB5A6eiHqALu28/6W95mxYwY9mvTgnWveIdg72N1hCQ8hiV4INyu1lvLC6hf44fAP3Jp4K8/1fA6TweTusIQHkUQvhBvtztrNK7+9ws6snTzR7QlGdxgt49YIl5NEL4QbnCg6wXub3+P7lO8J8wnj3Wve5dpm17o7LOGhJNELcQkVWYqYuWMmc3bPQWvNmI5juL/T/TLEsKhVkuiFuATs2s785PlM3TKVrNIsBjcfzITLJxATWOXsmkK4jCR6IWpZvjmfZ//3LL+m/crl0Zfzr/7/olNUJ3eHJRoQSfRC1KLknGQm/jKRY4XH+Fv3vzGy7UhpbBWXnCR6IWrJD4d/4MXVLxJgCmDWoFl0je7q7pBEAyWJXggXs9qtTNk8hdm7ZtMlqgv/vOafRPtHuzss0YDVZM7YWcANwEmtdcdK1t8FPA0ooAB4SGu9zbnusHOZDbBWNc2VEJ7iRNEJnl/1POtOrGNEmxE8fcXTmIxy85Nwr5qU6GcD7wNzqlh/COirtc5RSg0GpgM9Kqzvp7U+dVFRClHHpRemM3PHTL498C0KxSu9X+HmVje7OywhgBokeq31SqVUwjnWr6nwdi0gc5yJBuNo/lFm7JjBwoMLUUpxc6ubua/TfdJtUtQprq6jvw9YUuG9Bn5USmngI6319Kp2VEqNA8YBxMfHuzgsIVzrSP4Rpm2bxpJDSzAZTIxoO4LRHUbTOKCxu0MT4iwuS/RKqX44En2fCov7aK3TlVLRwDKl1F6t9crK9nd+CUwHSEpK0q6KSwhXstltzN09l/e3vo9BGbin/T3c2+FeIv0i3R2aEFVySaJXSl0GfAwM1lpn/b5ca53ufD6plFoAdAcqTfRC1HUpuSm8sOYFtmdup39cf57v+TxR/lHuDkuIal10oldKxQP/Ae7WWu+vsDwAMGitC5yvrwdevtjzCXGpWe1WPt31KR9s/QB/kz9vXf0WgxIGyY1Pot6oSffKL4BrgEilVBowCTABaK0/BF4EIoAPnH/4v3ejbAQscC7zAj7XWv9QC9cgRK05kHOAF1a/wM6snVwbfy3P9XxOqmlEvVOTXjcjq1k/FhhbyfIUQOZBE/WSXduZu3suUzZPIdAUyNt932Zgs4FSihf1ktwZK8QZjhce5/nVz7P+xHr6xfVjUq9JRPhFuDssIS6YJHohnLTWLExZyGvrXsOu7bx85cvc3OpmKcWLek8SvRBAbmkur6x9hR+P/EjX6K682udV4oLi3B2WEC4hiV40aGW2Mubvn8/07dPJM+cx4fIJjOkwBqPB6O7QhHAZSfSiQTLbzCxIXsCMHTPIKM7g8ujLeab7M7SLaOfu0IRwOUn0okGx2Cx8e/BbZmyfwfGi43SJ6sLkPpPp0biH1MULjyWJXjQYy48u560Nb5FemM5lkZcxqdckrmx6pSR44fEk0QuPd6rkFK+te41lR5aRGJbI1AFTuSrmKknwosGQRC88ltaabw98y9sb36bMWsb4ruMZ3XE0JoNMBCIaFkn0wiMdzT/K33/7O+tPrKdbo25M6jWJ5iHN3R2WEG4hiV54FLPNzJzdc/hw24d4G7yZ1GsStyTegkEZ3B2aEG4jiV54jDXpa3h9/esczj/Mdc2u45nuz8ik3EIgiV54gBNFJ3hrw1ssO7KM+KB4pl07jT4xfarfUYgGQhK9qLcsNgtz98zlw20fYtd2Hu3yKGM6jsHb6O3u0ISoUyTRi3ppc8ZmXv7tZQ7mHaRfXD+e7v60TMgtRBUk0Yt6Ja8sj3c3vcv85Pk0CWjC+/3fp29cX3eHJUSdJole1Ataa344/ANvrn+TnLIc7m1/Lw93eRh/k7+7QxOizpNEL+q8tII0Jq+bzOr01bSPaM+0a6fJ4GNCnIcadS5WSs1SSp1USu2sYr1SSr2nlDqglNqulLq8wrp7lVLJzse9rgpceL7M4kxeX/c6w74dxuaMzTx9xdN8PuRzSfJCnKealuhnA+8Dc6pYPxhIdD56ANOAHkqpcByTiScBGtiklPpOa51zMUELz3aq5BSzds7iq31fYbVbGdZqGA91fojGAY3dHZoQ9VKNEr3WeqVSKuEcmwwD5mitNbBWKRWqlGoCXAMs01pnAyillgGDgC8uJmjhmbJLs5m9czZf7P0Cs93MDS1u4MHLHiQuWGZ6EuJiuKqOPgZIrfA+zbmsquVnUUqNA8YBxMfHuygsUR/Y7Da+2PsF/9ryL0qsJQxtMZQHLnuAhJAEd4cmhEeoM42xWuvpwHSApKQk7eZwxCWyO2s3f//t7+zO2k3vmN78NemvtAht4RmtOEYAACAASURBVO6whPAorkr06UDF39exzmXpOKpvKi5f4aJzinqs2FLM1K1T+WzPZ4T5hPF237cZ2GygjBEvRC1wVaL/DnhUKfUljsbYPK31caXUUuA1pVSYc7vrgb+56Jyinvo19VdeXfcqx4uOM7z1cCZ2m0iwd7C7wxLCY9Uo0SulvsBRMo9USqXh6EljAtBafwgsBoYAB4BiYIxzXbZS6hVgg/NQL//eMCsans0Zm/lg2wesO76OliEtmTN4Dl2ju7o7LCE8nnJ0lKlbkpKS9MaNG90dhnCRigk+wjeCP3f8MyPbjsRklJmehHAVpdQmrXVSZevqTGOs8DxnJvinkp5ieJvh+Hn5uTs0IRoUSfTC5bZnbuf9Le/z2/HfJMELUQdIohcusy97H+9veZ8VaSsI8wnjyaQnub3N7ZLghXAzSfTioqXkpfDB1g9YengpQaYgHuv6GKPajZKRJYWoIyTRiwu2I3MH8/bOY8mhJfgYfbi/0/3c2+FeQnxC3B2aEKICSfTivJhtZpYeXsrnez5nZ9ZOAkwBjGo3ij93/DMRfhHuDk8IUQlJ9OI0rSHnMIQlwBl3qGYUZfDV/q/4Zv83ZJdmkxCcwN+6/42bWt5EoHfg2ceylMKxzY5j1iV+odCog7ujuPQydkFJ7tnLlQFiuoGXzLPrySTRi9NWvQM/vww9H4aBr1FoKeKnoz+xKGUR60+sR2vN1bFXc2fbO+nZtCcGVcV0BmUF8OlNjkRfF137d+gz0d1RXDqrp8CyF6teH38l3P0fMEmjuaeSRC8cNsyEn1/GHNaM/23/hEUFO/i19Dhmu5nYwFjGdhrLza1uJi6omiGDLaXw5Z1wfBvc8H8QXscGKNv0Cfw0CfzCoFsDmAdn8xxHkm9/MyT9+ez1mXthydPw9WgY8RnITWweSRK9wLL936xd/hxLW3RiuclOgVkTXpDCbVGXM6TnU1wWeVnNBhuzWWH+fXBoJfxpOnQeUfvBn6/4Xo5fHAsnOqpx2g9zd0S1Z/d38P0EaHUt3DKj8uqZFn3BYIRFf4H/PgI3fwiGGk08J+oRSfQNlMVuYd3xdSzd/gnLj68lv3EUgV5W+sX2Z0jC9fRcOQ2vTQsh4WaI6lz9AbV2JJW9C2HwW3UzyYMj2d0+F+beDPPHgk8wtOzn7qhc7+Avji/d2Cvg9jnnroO/YiyU5MDyyY5fOoPeOKuNRtRvkugbELu2s/XkVr5P+Z5lR5aRV5ZHoN1Of+3D9VdNpleza/E2OhPC7T3gs1thwQPgGwKJ11V9YK3hx+dh62fQ9xno8cCluaAL5e0Pd/4bPhkKX94F934HsZUOEVI/pW1yXFdEouM6vQOq3+eqJ6E4B9ZOBb9wuObp2o9TXDIyqFkDcCjvEAtTFrIoZRHphen4efnRP6obg3Yt5UpjMN5//hECo8/esTQPZt8Ap5Lhnm8hvmflJ/jfPx2NuN3HOUrz9aU0WHACZg2C0lwYswSiPWDS8ZN74ZNBji/nPy+FoPOYZ9dud1TfbPscBr8NPcbVXpzC5c41qJkkek+w53tI2/CHRVm2Mn4oSeP7kqPssuRiAHr6RHODXxwDfJviv2O+o2vdn3+AsGZVH7sw05E4CjMdjZdnJvHibNgyFzrdDn/6qP7V7+YchpkDHdd12e3ujubibf8atN3x7xre/Pz3t1nhq3tg3yJH461PkOtjFFXzDoS+f72gXSXRe7LCk/BuR9A2So1e/OLrw/cBvqzx9camFG3NFm4oKmVIcRlRdvvp/YIaw8h/Q3Tb6s+RexTm3Q45hypf32YI3DK9/vbYyNgNX9wBhRnujuTiBUbDyC8v7l4BS6mjF07KLy4LS9RQQDQ8vuOCdpVE78EsP73Exo3TWHzFHSzLWEeRpYhG/o0Y2mIoN7a4kVZhrdwdohDiEpDx6D1MVkkWq9JX8evRn1mTupyiJtEEZKzjumbXcWOLG0lqnFT1zUxCiAZHEn09oLXmQO4BVqSuYEXaCnZk7kCjiTb6M7iwkL59nqdHp7tkOGAhRKVqOmfsIGAKYAQ+1lq/ccb6d4HfOyP7A9Fa61DnOhvwe6XTUa31Ta4I3NNZ7Va2nNzCL6m/sCJ1BakFqQB0jOjIQ10e4pomvWn72R2osA7QdayboxVC1GXVJnqllBGYClwHpAEblFLfaa13/76N1vrxCts/BlSc8blEa93FdSF7Jq01RwuOsjljMxszNrIybSW5ZbmYDCZ6NOnB6A6juSbuGqL9nd0gt34O+elw4xT3Bi6EqPNqUqLvDhzQWqcAKKW+BIYBu6vYfiQwyTXheS7Ltn+zZ+fnbGk/iC25+9hycgvZpdkAhPiEcFXMVfSL60fvmN4EmM644cVuh9XvQXQHx+3tQghxDjVJ9DFAaoX3aUCPyjZUSjUDmgPLKyz2VUptBKzAG1rrb6vYdxwwDiA+Pr4GYdUvp0pOse3kNrZmbmXbkRXszj9EmUHB9g+IC4yhT0wfukZ3pWt0V5qHND93Y2ryj5C5xzGeTH25OUkI4Tauboy9A/hGa22rsKyZ1jpdKdUCWK6U2qG1Pnjmjlrr6cB0cHSvdHFcl1RuaS67s3ezN3sve7L2sOPUDtIL0wEwKS/alxYzwiuEzglDufx//yIyrjkMexG8fGp2gtVTICQOOt5Si1chhPAUNUn06UDFsWljncsqcwfwSMUFWut053OKUmoFjvr7sxJ9faO15lTJKVILUkkrTCO1IJX92fvZk72H40XHy7eLCYyhfUR7RrYdSWflR/tvn8A7JAZGLQH/cAhpDd8+6BiA6rbZYKzmnyR1PRxd4xh4qr7eoCSEuKRqkug3AIlKqeY4EvwdwJ1nbqSUaguEAb9VWBYGFGuty5RSkUBv4C1XBF6btNYUW4vJKM7gZPFJMoqcz8UZZBRlkFaYRlpBGqW20vJ9FIpmwc3oEt2FkeEjaRfRjnbh7U7Pn5q53zGUgF8Y3L3AkeQBuox0jBy49G+OoXNv+te5q2NWTwHfUOh6dy1+AkIIT1JtotdaW5VSjwJLcXSvnKW13qWUehnYqLX+zrnpHcCX+o+32rYDPlJK2QEDjjr6qhpxL9qcnV/i7UV5/bZBGVAoDMqAxW6hzFZGqbWUEmsJpbZSSq2lFFmKyDfnOx5l+eWvrXbrWccP8QmhkX8j4oLi6NW0F3FBccQFxREbGEtMYAymqkrYuakw90+OsWXu+RaCm/5xfa+HoSQbVr7t+CK4/pXKj5O5H/YugqufAp9Kpu8TQohK1KiOXmu9GFh8xrIXz3j/UiX7rQE6XUR8NVZmtfH2xrdAWard1oABPy8ffI0++Bl9CTYFEuIdRJPAOIJNAeXvo30jiPaNoLFfJFG+4fgaq6hDtwP5xypfZymBf4+CsnwYvRAiWla+Xb/nHCX7Ne+Bb7BjkLAz/e+fjnr87jKqoBCi5jzmzlit4d7YGXy29jB5JRZ6tgzjvj7N6RgTjF7wIIZDK/HTdvzsGi/gkvZV8fJ1VNc0OccEHko5hob9fQKI5ZMr3+6KsRAYVTtxCiE8kscNalZYZuWztUeYsTKFrCIzA1t48+HxEah2N0Li9S6OtIaadqn5aII2C+xbDGWFZ68zGKHNYMdY40IIUUGDGtQs0MeLB/u25J5ezZi39ihHV3yC0jYePtwbZWlHYqNAWjcKonWjQJpFBGAy1rHBv4wmz57HVAhxyXlcov+dv7cX91/dAlv6IYoPRWGJ7sy+9DwW7zzO7z9iTEZFXLg/kYE+RAR4ExHoTXiA43VUkA/NIwNoHhmAr8no3osRQoiL4LGJHgBLKcaU5fhfdjszbnTczFtitnHgZCH7MwrYf7KAo1nFZBWZ2Z9RQHaKmdwSCxVrs5SC2DA/WkYFlj8SIvyJj/CnSYgfRoPcmSqEqNs8O9EfWgmWImg7tHyRn7eRTrEhdIqtvJ7barOTU2whI7+UQ6eKOJhZyMHMIg6eLGRtShalltOzNJmMitgwf+LC/YkP96NxsC9hAd6E+3s7ngO8CfP3JszfhFddqyISQjQYnp3o9y1yzMHY/Ooa7+JlNBAV5ENUkA8dY/74ZWC3a47llXA0q5gj2cUczS7maJbjeevRHPJLz+57/7tQfxPhAd5EOL8AIgJ9aBTkS6voQBIbBZIQEYC3l3wZCCFcz3MTvd0O+5ZAqwE1H0OmGgaDowQfG+bPlZWsL7PayC22kF1kJqfITHaxmewiM1mFjufsIjNZRWUcOlXEpiM5ZBWZy6uJvAyKhMgAWjsbi7vGh9E1PpRgXxnmQAhxcTw30R/b7Jjsuc3Q6rd1ER8vI42CjTQK9q3R9qUWGwczC0nOcLQZJJ8sZPexfJbsPIHWjvaBNo2CuCIhnKSEMJISwokJlVmkhBDnx3MT/d5FoIyQeJ27I6mSr8lIh6YhdGj6xyqiwjIrW4/msvFINhsP5/CfzWnMXXsEgM5xoYzqEc+NnZtKbyAhRI143A1T5ab2gIAox7AD9ZzVZmfviQLWHDzFvzekcjCziBA/E7d1i+WuHvG0iJJxb4Ro6BrUDVMAZB2EzL3QbbS7I3EJL6OBjjEhdIwJ4f6rWrA2JZvP1h3h0zWHmbnqEL1bRXB3zwSubRctvXuEEGfxzES/zzn+Wpsh7o2jFiil6NUygl4tIzhZUMpXG1L5Yn0qD362iZhQP+7u1Yw7rogj1N/b3aEKIeoIz6y6mTXYMVrkQ6tdF1QdZrXZ+WnPSWavOcTalGx8TQb+1DWGe69MoG3jYHeHJ4S4BBpW1U1RFqSuhauedHckl4yX0cCgjo0Z1LExe47n8+maw/xnczpfrE+le0I4N3VpypBOTQgPkFK+EA2R55Xot8yD/z4M41ZA066uDKteySky8+WGVL7Z5Gi89TIorkqM5KYuTbmufWMCfTzvO16IhuxcJXrPS/Rf3gXHtsDju849JV8DobVm9/F8vtt2jIXbjpOeW4KvyUD/ttEM7tiEfm2jJekL4QEaTtWNpQQOLocud0qSd1JKlffVf3pgWzYfzeG7bcdYvOM4i3ecwNto4KrESAZ2bMy17RpJ9Y4QHqhGiV4pNQiYgmPO2I+11m+csX408DaOycMB3tdaf+xcdy/wvHP5ZK31py6Iu3IpK8BS7JG9bQDsRUXYzeZK1xmDglBe5/7nNBgUSQnhJCWEM+nGDmw+msMPO0/ww84T/Lz3JEaD4oqEMJKahdMxJpiOMSHEhPqh5EtTiHqt2qobpZQR2A9cB6QBG4CRFSf5dib6JK31o2fsGw5sBJIADWwCummtc851zguuuvnvo7D7v/DUQfCqvyVTe2kp5pQUSvfvpyw52fk4gPX48Sr3USYT3i1a4NO6NT6JifgktsInsTWmmKbVJmqtNbuO5Zcn/P0ZBdjsjr+LUH8THZuG0CEmmCtbRtKjebjckStEHXSxVTfdgQNa6xTnwb4EhgG7z7mXw0BgmdY627nvMmAQ8EVNAj8vdhvs/wFaXVtvkry2WjEfPUrZfmcydyZ289GjjkHZcCbwli3xvyIJn5atMAQEVHIgjTXjBKXJyRRv2kj+99+Xr/Ju1owmr7+O/+VVN0wrpcpvyHpyYBtKLTb2nihgR3oeu9Lz2JGex6xVh/jo1xT8TEZ6t4pkQLto+rWJpnFIzcb1EUK4T00SfQyQWuF9GtCjku1uVUpdjaP0/7jWOrWKfWMqO4lSahwwDiA+Pr4GYZ3BZoE+j0Ojjue/by3TWmM9dozS5OTTST05GfPBg2iLxbGRwYB3fDw+iYkEDxniKJm3TsQ7Pr7aKpkz2QoKKEs+QNm+vWR9PJMjo0YRMXYsUY8+gvKu/kvQ12SkS1woXeJCy5eVWmz8djCL5XtPsnzvSX7akwFA+ybB9G8bTb+20XSJC5WJWISog2pSdXMbMEhrPdb5/m6gR8VqGqVUBFCotS5TSj0AjNBa91dKPQn4aq0nO7d7ASjRWv/jXOd0yVg3dUDZwYOcmjqVwl9XYi8qKl/u1aSJo2qlVSI+rRMdVS0tW2LwdX3p2FZYSMYbb5D3zXx82rWj6Ztv4Nu69UUdU2vN/oxCZ9LPYNORHPzLiulgPkV/nwI6mbOIzkrHUFKMT8uWzuqkmlclXSi72UzRqlXkL1xE2aFD+DRvfvrzTUzEFBuLMsgQEcIzXWzVTToQV+F9LKcbXQHQWmdVePsx8FaFfa85Y98VNTjnBTky6u4qGyur4hUVReT9Y/Hr0sVlcZhTUzn1/lTyvv8eg68vwTfeiG+7do6k06oVxuBLd7eqMTCQppMnE9S/P8dfeJHDt95G1OOPEz76XlAK68nMCu0AyVjS0/Hv1o3gG4bi06JFpcdUSpEYaqJJ6QFu3LOEoi1bsJ88Wb6+yMuXrcGNMQUFEr9mHb4LTw8sZ/D3dyTeCgnYJzERr8jIC7o+bbVStG4d+YsWU7BsGfaCAoyhofh26EDJtm3kL158Om4/P7wTElCmSsb4V+AdE3s6rtatMcXEyBeD8Ag1KdF74aiOGYAjcW8A7tRa76qwTROt9XHn6z8BT2utezobYzcBlzs33YyjMTb7XOe80BJ96iOPos8z0Zfu2oUtO5vAa64hasJ4fNu1O+/z/s5y4gSnpn1I7vz5KKORsLvuImLsfXiFh1/wMV3Jmp3N8RdfpPCnnzE1i8eem4ctL698vTEqElNUNKV79oDW+LRtS/DQIQQPHoJ3bAzaYqFozRryFi2i8KefsRcXY4yMJLD3leWNwKaWrdhr82P5vkx+2pPBrmP5+FtKGOBbyLW+RbQvy8T76CHKkpOx5ZxukzeGh+PTujWBV19N8JDBmBo3rvI6tN1OyZYt5C9aTP7SpdiysjAEBBB07bUE3zCUgJ49y5O5rbAI88EDp6vLDh9B2+1nH9RmxXzkKJb002UY5eeHT6tWZ30xeUVF1ZmeSLa8PMoOHChv47FmnsI7IaE8Xu8WLTDUoLpO1H8XfcOUUmoI8H84ulfO0lq/qpR6Gdiotf5OKfU6cBNgBbKBh7TWe537/hl41nmoV7XWn1R3vktZdWMvKiL7s3lkzZyJPT+foEGDiBr/WJWl2YqsOTnO/2DJlO7aRf6iRWitCRt+GxEPPIipUfQluILzo7Um79v/kv/9d5ji4k+Xqlsn4hUWBoDl5EkKfviB/EWLKdm2DQDfjh2xpKZiy8vDEBJC8PXXETxkCP7du6OMVffCOZJVxJKdJ1iy4zjb0hxfKm0bB9GtWRjdgjUdyjKJyEzHciCZkh07Kdu7F5Ry/KoYOoSggQPxCg9Ha03p7t2O5L5kCdbjx1E+PgRecw3BQ4cQ2LcvBp+Ln0ns9y+G0v37He0pBxw9nmynTpVvYwwJOZ38f+/l1KoVxpA/ziug7XYs6emnG9kPH0HbbBcdI4AtN9eR2DMyypcZAgPxiorCnJYGv7f9GI2OxF/eEysR39atMcXFnfPfzRW02Uzh6tUUrV6Db7t2BF13bbW/ZstSDpG/eDHmI0dqNTZXUgYD3gnNTv8SdFMVYcO6M/YC2fLzyZ49m+zZn2IvLSXo+uvxioo6aztttWA5coTS5GRsmaf/8xtCQggaMIDIhx/GO7bS9uZ6yZyWRv6ixRQuX44pNpbgoUMJ7NO7Ro26Z0rLKeaHnSdYvvck29PyKCxzzLEb4G2kY0wIneNC6WEqInH3WizLfsB84CAYjQT06I7l2HHMhw+DlxeBffo4knu//hgDK+mFVAus2dmOBu7fu7zu30/ZgQPYCwvLt/Fq1MhR4o8IpyzlEGUHDqBLSk6vb9z4gj63yhgCA/CtUPXl07q14/hKoc1mzEeOUJac/IcOAJbUVH6fu1L5+FRoPzn9i8WrUaOL+rWibTaK168nf/Fi8n9chj0vD2UyoS0WlMlEwFVXOb7A+/XD4O8PgCU9nbzFi8lfvISyPXtAKUwxMVBPqs202Yz1xIny98rPz/HZtmqFISjovI5lCPAneuLEC4pDEv15sGZnkzXjY/K+++50j5gKlFKY4uL+8B/MJzERr+i683O+PrDbNSmnitiWmsv2tFy2puWx51g+ZpujWiUxKoDr/Iu48shmorb+hm/jRo5qpOuuwxgaWs3RLw2tNdYTJ04nf2ditWVl492iOb7l9zQk4t2y1SX7UqqKvbiYsoMpf+jK66juySzfxhAcXF7694qMgvP4k7ZlZZH/4zJsp05h8Pcn6LprCR7qqEor3bfv9K+xjAyUnx+BV1+NNSODkq1bAfDr3Nn5K25Qnfw1fC72oiLKDh4s/4VflryfsgMHsZeWntdxvMLDafnDkguKQRK9qBdKLTZ2pOex/lA26w9ls+lITnmpPzzAm1ZRgbSMDqRlVACtogNpFR1I0xA/DNKl86JYc3IwHzjgKP1X+AVgz88/r+MoHx8C+/Z1/Orre3Wlvci03U7Jpk3kLVpEwU8/4xUeTvDQoQQPHYJ3bOwFxa+1prDMSpBvJY3sDYgkelEv2eyaPcfz2XA4m30nCjiYWciBk4XkFJ/+pRXo40X7psF0bBpCx5hgOjQNoWVUgMy0dZG01uXVPDWm1Fm/am12zaFThTQN9cPfu/pOflprDmYWklVoJtTfmzB/E6H+3nh7nf73PJlfyra0PLan5bItLY8dabnkFFvo3SqC0Vc2p3/b6AZ5P4ckeuFRsgrLOJhZxIGThew9kc/O9Dx2H8+n1OKo9vHxMtC2cRCtooPKS/6J0YHEhfu7LQFYbHayCs2EB/wxaXkarTUHThay+sAp1hzMYm1KFvmlVrwMik6xIXRvHk6P5uF0axZOiJ8JrR1VeL85t12bks2pwrKzjuvvbSTM3xur3U5GvmO90aBIjA6kc2woEYHeLNiSzvG8UuLC/bi7ZzNGJMUT4n+6lJ9ZUMbW1Fy2puawNTWXErONQF8TgT5GAn28CPDxIsjHCy+jAbPVTpnV5nx2PLyNBq5sFcFViVE1Gvwvv9SCn8mI6RIVOiTRC49ns2tSMgvZeSyPnen57Dmez4GThZwsOJ00vL0MNAv3J9TfRKCPl/M/uRdBvo7/4JFBPjQK9qFRsC+Ngn0J9/f+Q7VQidlGdrGZnCIzOcVmSsw2vIwKo8GAl0FhNCi8DIoyq53DWUUcyizi0CnH42h2MVa7xqCgSYgf8eH+jkeEP7FhfviajGitsTsL0natsWuN1aYps9oxW23OZ0fSKbHYKCqzUlBmpajMSmGplcIyK2abnVA/E2H+3oQFeBMe4E2ov4kQPxM2u6bUYqPUYi9/LrPasNo0VrvjfDb76YfRqAjwNuLv7UWAj+PZ39v4h0T4+zHKLHZOFZax7lA2mc7PPC7cjytbRHJ5s1AOZxWz/lA229Nysdg0SkGbRkFkF5nL/40aBfvQq4VjmsyYUH9yS8zkFFvIK3Y85xSbQUOHmBA6xzpGZPXzPt1zyGqz8+PuDGavOcz6Q9n4mYzccFkTSq12thzNIS3H0TDuZVC0bRJEqJ83BWVWCkstFJXZKCyzllcVGhT4eBnx9jLg42XAx2Qgv8RKXokFpeCy2FD6to7imjZRdI4NJbOgjF3Ov71dx/LYdSyf9NwSjAZFbJgfCREBJET4kxAZQEJEAIG+XpjL/z1t5V8mJqPiT10vrApLEr1osPJKLOVVPgdOFnL4VBEFzqRYWGZ1vraU/xqoyGRURAf5Ytea7CIzZdZK+t+fg6/JQEJEAM0jHY8moX5kFpSRml3MUecjs+Ds0mt1DMoxTIXjy8rLURr1drz2NhrIK7GQ7fwyqipuL4PC12TEx8uAyWjAaFAYDOBlMGBQjtKy1a4pLrNRZLZSbLaVD3R3Jm8vA75eBoJ8TXRrFkbvVhFc2TKSuHD/s7YtMdvYkppT3gYT6u9dntwTIvxd1qFh9zHHTGvfbk0nIsCbrvFhjmE94kPpeMYXREV2u8amNV6GyquhdqTn8eu+TFbsP8nW1Fy0Bm+jobwTgVLQPDKADk1DaNckiOIyG4ezihyPU8XlXyRViQjwZtML113QNUuiF6IaZqudzMIyMvJLycgrJSO/lBP5ZZzML8VoUIQFeDtKyf6m8td+JiM2rbHZ7VhtjlKw1a7xMioSIgJoHOxbbUNxsdlKek4JZpsdg1IoBQalcOymMBkVPl6OhPx76fJ82x9KzDbySix4GR3J3fcCjqG1xmyzU1xmw2KzO2IyGfA2Gup0Y7jdrmstvpwiM/87cIqtR3OJD/ejQ0wI7ZoEVzmRj9aarCIzh08VUWKx4W004GMy4m00/OGXQ3TQhQ2FIoleCCE83LkSvee2CgkhhAAk0QshhMeTRC+EEB5OEr0QQng4SfRCCOHhJNELIYSHk0QvhBAeThK9EEJ4uDp5w5RSKhOoboqZSOBUNdt4IrnuhkWuu2G5mOtuprU+e7Yk6miirwml1Maq7gLzZHLdDYtcd8NSW9ctVTdCCOHhJNELIYSHq8+Jfrq7A3ATue6GRa67YamV6663dfRCCCFqpj6X6IUQQtSAJHohhPBw9S7RK6UGKaX2KaUOKKWecXc8tUkpNUspdVIptbPCsnCl1DKlVLLzOcydMbqaUipOKfWLUmq3UmqXUmqCc7mnX7evUmq9Umqb87r/7lzeXCm1zvn3/m+lVPWzUtdDSimjUmqLUmqh831Due7DSqkdSqmtSqmNzmUu/1uvV4leKWUEpgKDgfbASKVUe/dGVatmA4POWPYM8LPWOhH42fnek1iBv2it2wM9gUec/8aeft1lQH+tdWegCzBIKdUTeBN4V2vdCsgB7nNjjLVpArCnwvuGct0A/bTWXSr0n3f533q9SvRAd+CA1jpFa20GvgSGuTmmWqO1Xglkn7F4GPCp8/WnwM2XNKhaprX+//buGDSKIArj+P8jRhAVgkGD5JQgCFaijSCmCAEtNFiJCArprC1E0EYQ0or2aqeBoEZTGjCFlUhUUNBGEPSIuSqojaB+FjPBQ0QI3Lns3PvBsTOzW8yD2bfDzN3eku3nYkgeQQAAAgJJREFUufyFdPMPU37ctv01V/vzx8A4cDe3Fxc3gKQGcAy4keuiB+L+h46P9bol+mHgQ1v9Y27rJUO2l3L5EzBUZWe6SdIIsB94Sg/EnZcvXgItYB54B6zY/p4vKXW8XwMuAD9zfZDeiBvSw/yRpEVJZ3Nbx8f63/+uPNSCbUsq8vuxkjYB94Bztj+nSV5Saty2fwD7JA0As8CeirvUdZImgJbtRUljVfenAqO2m5K2AfOS3raf7NRYr9uMvgnsaKs3clsvWZa0HSAfWxX3p+Mk9ZOS/G3b93Nz8XGvsr0CLAAHgQFJqxOyEsf7IeC4pPekpdhx4Drlxw2A7WY+tkgP9wN0YazXLdE/A3bnHfn1wClgruI+/W9zwGQuTwIPK+xLx+X12ZvAG9tX206VHvfWPJNH0gbgMGl/YgE4kS8rLm7bF203bI+Q7ufHtk9TeNwAkjZK2rxaBo4Ar+nCWK/dL2MlHSWt6fUBt2xPVdylrpE0DYyRXl26DFwGHgAzwE7Sq5xP2v5zw7a2JI0CT4BX/F6zvURapy857r2kjbc+0gRsxvYVSbtIM90twAvgjO1v1fW0e/LSzXnbE70Qd45xNlfXAXdsT0kapMNjvXaJPoQQwtrUbekmhBDCGkWiDyGEwkWiDyGEwkWiDyGEwkWiDyGEwkWiDyGEwkWiDyGEwv0Cq/VFyOWgdRgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "log_name = \"run_log\"\n",
    "txt_name = log_name + \".txt\"\n",
    "csv_name = log_name + \".csv\"\n",
    "\n",
    "# convert to CSV\n",
    "with open(txt_name, \"r\") as txt_file:\n",
    "    data = txt_file.readlines()\n",
    "    csv_lines = list(map(lambda x: x.replace(\"Epoch: \", \"\")\n",
    "                         .replace(\", Iter: \", \",\")\n",
    "                         .replace(\", Train Loss: \", \",\")\n",
    "                         .replace(\", Train Accuracy: \", \",\")\n",
    "                         .replace(\", Val Loss: \", \",\")\n",
    "                         .replace(\", Val Accuracy: \", \",\"),\n",
    "                            filter(lambda x: \"Epoch: \" in x, data)))\n",
    "    with open(csv_name, \"w\") as csv_file:\n",
    "        csv_file.write(\"epoch,iter,train_loss,train_acc,val_loss,val_acc\\n\")\n",
    "        for item in csv_lines:\n",
    "            csv_file.write(\"%s\" % item)\n",
    "\n",
    "# plot\n",
    "log = pd.read_csv(csv_name)\n",
    "plot_log = log[log[\"iter\"] == 1]\n",
    "\n",
    "for val in [\"train_loss\", \"train_acc\", \"val_loss\", \"val_acc\"]:\n",
    "    plt.plot(plot_log[\"epoch\"], plot_log[val], label=val)\n",
    "\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}