| { |
| "cells": [ |
| { |
| "cell_type": "markdown", |
| "id": "b75fdb6a", |
| "metadata": {}, |
| "source": [ |
| "<!--- Licensed to the Apache Software Foundation (ASF) under one -->\n", |
| "<!--- or more contributor license agreements. See the NOTICE file -->\n", |
| "<!--- distributed with this work for additional information -->\n", |
| "<!--- regarding copyright ownership. The ASF licenses this file -->\n", |
| "<!--- to you under the Apache License, Version 2.0 (the -->\n", |
| "<!--- \"License\"); you may not use this file except in compliance -->\n", |
| "<!--- with the License. You may obtain a copy of the License at -->\n", |
| "\n", |
| "<!--- http://www.apache.org/licenses/LICENSE-2.0 -->\n", |
| "\n", |
| "<!--- Unless required by applicable law or agreed to in writing, -->\n", |
| "<!--- software distributed under the License is distributed on an -->\n", |
| "<!--- \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->\n", |
| "<!--- KIND, either express or implied. See the License for the -->\n", |
| "<!--- specific language governing permissions and limitations -->\n", |
| "<!--- under the License. -->\n", |
| "\n", |
| "\n", |
| "# Advanced Learning Rate Schedules\n", |
| "\n", |
| "Given the importance of learning rate and the learning rate schedule for training neural networks, there have been a number of research papers published recently on the subject. Although many practitioners are using simple learning rate schedules such as stepwise decay, research has shown that there are other strategies that work better in most situations. We implement a number of different schedule shapes in this tutorial and introduce cyclical schedules.\n", |
| "\n", |
| "See the \"Learning Rate Schedules\" tutorial for a more basic overview of learning rates, and an example of how to use them while training your own models." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 1, |
| "id": "0bd0e94d", |
| "metadata": {}, |
| "outputs": [], |
| "source": [ |
| "%matplotlib inline\n", |
| "import copy\n", |
| "import math\n", |
| "import mxnet as mx\n", |
| "import numpy as np\n", |
| "import matplotlib.pyplot as plt" |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 2, |
| "id": "98e04541", |
| "metadata": {}, |
| "outputs": [], |
| "source": [ |
| "def plot_schedule(schedule_fn, iterations=1500):\n", |
| " # Iteration count starting at 1\n", |
| " iterations = [i+1 for i in range(iterations)]\n", |
| " lrs = [schedule_fn(i) for i in iterations]\n", |
| " plt.scatter(iterations, lrs)\n", |
| " plt.xlabel(\"Iteration\")\n", |
| " plt.ylabel(\"Learning Rate\")\n", |
| " plt.show()" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "d27a0989", |
| "metadata": {}, |
| "source": [ |
| "## Custom Schedule Shapes\n", |
| "\n", |
| "### (Slanted) Triangular\n", |
| "\n", |
| "While trying to push the boundaries of batch size for faster training, [Priya Goyal et al. (2017)](https://arxiv.org/abs/1706.02677) found that having a smooth linear warm up in the learning rate at the start of training improved the stability of the optimizer and lead to better solutions. It was found that a smooth increases gave improved performance over stepwise increases.\n", |
| "\n", |
| "We look at \"warm-up\" in more detail later in the tutorial, but this could be viewed as a specific case of the **\"triangular\"** schedule that was proposed by [Leslie N. Smith (2015)](https://arxiv.org/abs/1506.01186). Quite simply, the schedule linearly increases then decreases between a lower and upper bound. Originally it was suggested this schedule be used as part of a cyclical schedule but more recently researchers have been using a single cycle.\n", |
| "\n", |
| "One adjustment proposed by [Jeremy Howard, Sebastian Ruder (2018)](https://arxiv.org/abs/1801.06146) was to change the ratio between the increasing and decreasing stages, instead of the 50:50 split. Changing the increasing fraction (`inc_fraction!=0.5`) leads to a **\"slanted triangular\"** schedule. Using `inc_fraction<0.5` tends to give better results." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 3, |
| "id": "9375849b", |
| "metadata": {}, |
| "outputs": [], |
| "source": [ |
| "class TriangularSchedule():\n", |
| " def __init__(self, min_lr, max_lr, cycle_length, inc_fraction=0.5):\n", |
| " \"\"\"\n", |
| " min_lr: lower bound for learning rate (float)\n", |
| " max_lr: upper bound for learning rate (float)\n", |
| " cycle_length: iterations between start and finish (int)\n", |
| " inc_fraction: fraction of iterations spent in increasing stage (float)\n", |
| " \"\"\"\n", |
| " self.min_lr = min_lr\n", |
| " self.max_lr = max_lr\n", |
| " self.cycle_length = cycle_length\n", |
| " self.inc_fraction = inc_fraction\n", |
| "\n", |
| " def __call__(self, iteration):\n", |
| " if iteration <= self.cycle_length*self.inc_fraction:\n", |
| " unit_cycle = iteration * 1 / (self.cycle_length * self.inc_fraction)\n", |
| " elif iteration <= self.cycle_length:\n", |
| " unit_cycle = (self.cycle_length - iteration) * 1 / (self.cycle_length * (1 - self.inc_fraction))\n", |
| " else:\n", |
| " unit_cycle = 0\n", |
| " adjusted_cycle = (unit_cycle * (self.max_lr - self.min_lr)) + self.min_lr\n", |
| " return adjusted_cycle" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "0971a205", |
| "metadata": {}, |
| "source": [ |
| "We look an example of a slanted triangular schedule that increases from a learning rate of 1 to 2, and back to 1 over 1000 iterations. Since we set `inc_fraction=0.2`, 200 iterations are used for the increasing stage, and 800 for the decreasing stage. After this, the schedule stays at the lower bound indefinitely." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 4, |
| "id": "eab6c0c6", |
| "metadata": {}, |
| "outputs": [ |
| { |
| "data": { |
| "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABLi0lEQVR4nO3de1xUdf4/8NcMyE0FRBPkJpiWFQp4J6ktJV0I7LZZ6WpettZbapm1mpfWNLPNttS0+i5GNzR1zVajzPCKaaiAhqZdHAQVRETuCsqc3x/+nBwFOZ/hnLmceT0fj3k8dObzYd5HkXl7zvt93jpJkiQQERERaYTe1gEQERERKYnJDREREWkKkxsiIiLSFCY3REREpClMboiIiEhTmNwQERGRpjC5ISIiIk1xtXUA1mY0GnH69Gm0bt0aOp3O1uEQERGRDJIkobKyEoGBgdDrb35uxumSm9OnTyMkJMTWYRAREZEFCgoKEBwcfNM1TpfctG7dGsCVPxxvb28bR0NERERyVFRUICQkxPQ5fjNOl9xcvRTl7e3N5IaIiMjByCkpYUExERERaQqTGyIiItIUJjdERESkKUxuiIiISFOY3BAREZGmMLkhIiIiTWFyQ0RERJrC5IaIiIg0hckNERERaYrT3aGYlFF32YhP9+ThRGkNOvp5YURMGNxcmSsTEZHt2fTTaOHChejduzdat26N9u3b4+GHH8axY8ea3Ld27Vp07doVHh4e6NatG9LS0qwQLV21MO0Ibp/1DV77+md8sucEXvv6Z9w26xss+PqwrUMjIiKybXKzY8cOTJw4EXv37sWWLVtw6dIlDBo0CNXV1Y3u+eGHH/DUU09h7NixyM7OxsMPP4yHH34Yubm5VozceS1MO4IPdhogNfDa/+3Kw98+zrR6TERERNfSSZLU0OeUTZw9exbt27fHjh07cO+99za45oknnkB1dTU2bdpkeq5fv36IiorC+++/3+R7VFRUwMfHB+Xl5RycKajushG3z/qmwcTmWmNjwzE78U6rxERERM5B5PPbrookysvLAQB+fn6NrtmzZw/i4uLMnhs8eDD27NnT4Pra2lpUVFSYPcgyH//Q8Bmb6yVnGJB2qFD1eIiIiBpiN8mN0WjE1KlT0b9/f0RERDS6rqioCP7+/mbP+fv7o6ioqMH1CxcuhI+Pj+kREhKiaNzOZNPB07LXTkrNQr3Rbk4KEhGRE7Gb5GbixInIzc3F6tWrFf26M2bMQHl5uelRUFCg6Nd3FvVGCUeKKmWvNwJ4fMUP6gVERETUCLtoBZ80aRI2bdqEnTt3Ijg4+KZrAwICcObMGbPnzpw5g4CAgAbXu7u7w93dXbFYnVWmoRSX6sXOxGQVlGHjwdNIigxUKSoiIqIb2fTMjSRJmDRpEr788kts3boV4eHhTe6JiYlBenq62XNbtmxBTEyMWmESgKKKixbte25VNi9PERGRVdk0uZk4cSI+++wzpKamonXr1igqKkJRUREuXLhgWjNy5EjMmDHD9PspU6bg22+/xeLFi3H06FG8+uqr2L9/PyZNmmSLQ3AapVW1Fu+Nef17BSMhIiK6OZsmNytWrEB5eTnuu+8+dOjQwfT44osvTGvy8/NRWPhH583dd9+N1NRUfPjhh4iMjMS6deuwYcOGmxYhU/PllzZ+76GmFFfVYd5G3uCPiIisw67uc2MNvM+NuHqjhKh5m1F5sb5ZX+eX+fEc0UBERBZx2PvckH3KNJQ2O7EBgN7zv1MgGiIioptjckNNsrSY+HrlF+sxJoXjGYiISF1MbqhJzSkmvt7Wo2exUeBmgERERKKY3FCTmlNM3BC2hxMRkZqY3NBN1RslrM8+pfjXHfjWNsW/JhEREcDkhpqgVDHx9fJKL2Bsyj7Fvy4RERGTG7opkWJiHw+xb6f0o8WsvyEiIsUxuaGbkltM7O3hin2zBgt//cmsvyEiIoUxuaGbkltM/Eh0ENxc9VgyNFLo60vg9HAiIlIWkxtqlEgxcaifFwBgSI9ghLX1EHqfq9PDiYiIlMDkhholUkzs19LN9Ov0aQOE34vt4UREpBQmN9QokWLiAB9P069d9DosHxYt/H5sDyciIiUwuaFGiRQT9wn3M3suoXsgEiL8hd4vr/QCp4cTEVGzMbmhRvl6uTW9CFeKiV30uhueXzqsJ1xufPqmVu7OQ9qhQrFNRERE12ByQ40qra6TtS64jVeDz7vodVj6lPjlqUmpWay/ISIiizG5oUYdOFEqa935msaToITugRgb21HofY1gezgREVmOyQ01qN4oYdevJbLWNnBFyszsxAhEBXsLvT/bw4mIyFJMbqhBmYZSVNfJawOP6dSuyTX/nRArHAPbw4mIyBJMbqhBctvAvdxc0O/Wtk2uc9HrsOzJKOE4Yl7/XngPERE5NyY31KCSSnlt4PERAQ12SjUkMSoIA7s2fZbnWsVVdWwPJyIiIUxuqEFyi4n9vcVGLSSP6gv/1i2E9qzcnYe6y0ahPURE5LyY3NANlCwmbsiul+OE9/Se/534GxERkVNickM3ULqY+Hpurnrh9vDyi/UYk5Ip/F5EROR8mNzQDZQuJm7I7MQI3NWhpdCerUfPsj2ciIiaxOSGbiB3plSCQDFxQ76ech+8PVyE9rA9nIiImsLkhm6QX1ota13MreKXpK63f9Yg4T2cHk5ERDfD5IbM1BslrM8+JWtt2U3GLsjl5qrH6P6hQnvySi9gbMq+Zr83ERFpE5MbMpNpKEXlRXnFxH4t5U0Nb8rcpG64pZVYe3j60WLW3xARUYOY3JAZucXEABDg46nY++6d+YDwnsmsvyEiogYwuSEzcouJvT1c0SfcT7H3ddHrsGRopNAeCZweTkREN2JyQ2Z8veRdanokOqhZnVINGdIjGGFtxe54zOnhRER0PSY3ZKa0Wl6RcHAbL1XeP33aAOE9bA8nIqJrMbkhM3JnSp1XoFOqIS56HZYPixbex/ZwIiK6iskNmag9U0quhO6BSIjwF9qTV3qB08OJiAgAkxu6htozpUQsHdYTLoIJ1MrdeUg7VKhOQERE5DCY3JCJNWZKyeWi12HpU+KXpyalZrH+hojIyTG5IZOSSnlt4PHNnCklV0L3QOHp4UawPZyIyNkxuSETucXE/t5i7drNMTsxAlHB3kJ72B5OROTcmNwQAPspJm7IfyfECu9hezgRkfNickMA7KuY+Houeh2WPRklvK/fgi3KB0NERHaPyQ0BsK9i4oYkRgVhYFexpOps9SWMSclUKSIiIrJXTG4IgP0VEzckeVRf+LcWmx6+9ehZ1t8QETkZJjcEwD6LiRuy6+U44T2svyEici5Mbsiui4mv5+aqF24PB4DHlmeoEA0REdkjJjdk18XEDZmdGIG7OrQU2pNzsgKvbTqiUkRERGRPmNyQ3RcTN+TrKfehXUux+pvkDAPHMxAROQEmN+QQxcQN+fGVB4T3cDwDEZH2MbkhnK+pk7XO1sXE13PR67BkaKTQHo5nICLSPiY3BJ3MkzFy11nTkB7BCGsrlnRxPAMRkbYxuSEUlV2Qtc7XU6zGxVrSpw0Q3sP2cCIi7WJy4+TqjRLScotkrW3Xyl3laCxj6XiGuMXbFY+FiIhsj8mNk9t7/BwuXDLKWhvg46lyNJZLjApCjxCx6eGGczX4KueUShEREZGtMLlxcnt+PydrXSt3V/QJ91M5muZZOz5W+Bt6yuocXp4iItIYJjdOToK8D/bYLm3tqg28IS56HZYNixbex+nhRETawuTGycktJu4Z2kblSJSR0D1QeDwDp4cTEWkLkxsnpoVi4obMToxAVLBY/Q2nhxMRaYdNk5udO3ciKSkJgYGB0Ol02LBhQ5N7Pv/8c0RGRsLLywsdOnTAmDFjcO6cvLoRMqeVYuKG/HdCrPAetocTEWmDTZOb6upqREZG4r333pO1fvfu3Rg5ciTGjh2Lw4cPY+3atcjMzMQzzzyjcqTapKVi4uuxPZyIyHnZNLmJj4/H/Pnz8cgjj8hav2fPHoSFhWHy5MkIDw9HbGws/v73vyMzk/USltBSMXFDEqOCMOB2sUGfhnM1nB5OROTgHKrmJiYmBgUFBUhLS4MkSThz5gzWrVuHhISERvfU1taioqLC7EFXaK2YuCErR/eDj4eL0J7kDAPqLsu7XEdERPbHoZKb/v374/PPP8cTTzwBNzc3BAQEwMfH56aXtRYuXAgfHx/TIyQkxIoR2y+tFhM3ZN+sQcJ7es//ToVIiIjIGhwquTly5AimTJmCOXPm4MCBA/j222+Rl5eHcePGNbpnxowZKC8vNz0KCgqsGLH90nIx8fXcXPXC7eHlF+vZHk5E5KBcbR2AiIULF6J///6YPn06AKB79+5o2bIl7rnnHsyfPx8dOnS4YY+7uzvc3R37zIMatFxM3JDZiRFI//kM8s5dlL3nant4UmSgipEREZHSHOrMTU1NDfR685BdXK7UU0gSW3hFaL2YuCHp0wZA9EjYHk5E5HhsmtxUVVUhJycHOTk5AACDwYCcnBzk5+cDuHJJaeTIkab1SUlJWL9+PVasWIHjx49j9+7dmDx5Mvr06YPAQP7vWoSvZwtZ6xy5mPh6LnodllrQHv7Y8gzlgyEiItXYNLnZv38/oqOjER19ZR7QCy+8gOjoaMyZMwcAUFhYaEp0AGDUqFF4++23sWzZMkRERODxxx/H7bffjvXr19skfkfm11LepTq56xyFJe3hOScr2B5ORORAdJKTXc+pqKiAj48PysvL4e0tdot+LXlxTQ7WZZ1qct3sB+/A2Hs6WSEi6+r92nc4W31JaM/yYT2Q0P3Gui4iIlKfyOe3Q9XckDLqjRK+/qlQ1lq/lm4qR2Mbe195QHjPpNQs1t8QETkAJjdOyJnawBvjotdhydBIoT1GAI+v+EGdgIiISDFMbpyQs7WBN2ZIj2CEtfUQ2pNVUMbp4UREdo7JjRNyxjbwxqRPGyC8h+3hRET2jcmNE3KGmVJycXo4EZH2MLlxMs40U0quxKgg9AgR65wznKvBVzlNd5sREZH1MblxMiwmbtja8bHC/ximrM7h5SkiIjvE5MbJsJi4YS56HZYNixbe12/BFhWiISKi5mBy42RYTNy4hO6BwtPDz1Zf4vRwIiI7w+TGybCY+OZmJ0YgKlis/ubq9HAiIrIPTG6cCIuJ5fnvhFjhPWwPJyKyH0xunAiLieVhezgRkWNjcuNEWEwsnyXTww3najg9nIjIDjC5cSIsJhazcnQ/+Hi4CO1JzjCg7rK8s2NERKQOJjdOxNezhax1zlpM3JB9swYJ7+k9/zsVIiEiIrmY3DiRk+drZK3za+m8xcTXc3PVC7eHl1+sZ3s4EZENMblxEvVGCeuz5Y0LKKupUzkaxzI7MUJ4ejjbw4mIbIfJjZPINJSi8mK9rLV+Ld1UjsbxpE8bANEqJLaHExHZBpMbJ1FUcVH2WmduA2+Mi16HpRa0hz+2PEP5YIiI6KaY3DiJ0qpaWeu8PdgG3hhL2sNzTlawPZyIyMqY3DiJ/NJqWeseiQ5iG/hNrBzdD7e0lNd1dlVyhgFphwpVioiIiK7H5MYJiBQTh/p5qRyN49v7ygPCeyalZrH+hojISpjcOAEWEyvLRa/DkqGRQnuMAB5f8YM6ARERkRkmN06AxcTKG9IjWLg9PKugjO3hRERWwOTGCbCYWB3p0wYI72F7OBGR+pjcOAEWE6vD0unhMa9/r3wwRERkwuRG41hMrK7EqCD0CPEW2lNcVYd5Gw+rFBERETG50TgWE6tv7fhY4X9IK3fncXo4EZFKmNxoHIuJ1eei12HZsGjhfb1e26xCNERExORG41hMbB0J3QOFp4dX1BqRuHSXShERETkvJjca5+sl71ITi4mbb3ZiBO6/TWw8Q+4pjmcgIlIakxuNK62uk7UuuA2LiZXw0Zh+8PFwEdqTnGFg/Q0RkYKY3GjcgROlstadr5GXBFHT9s0aJLzn3kXpKkRCROScmNxoWL1Rwq5fS2St5RUp5bi56jG6f6jQnqLKOoxN2adSREREzoXJjYZlGkpRXSevDTymUzuVo3Euc5O64ZZWYtPD048WczwDEZECmNxomNw2cC83F/S7VawQlpq2d6b49HCOZyAiaj4mNxpWUimvDTw+IoCdUiqwdDzDY8szlA+GiMiJMLnRMLnFxP7eYtOtST5LxjPknGR7OBFRczC50SgWE9sPS8YzJGcYkHaoUJV4iIi0jsmNRrGY2H5YOp5h8qos1t8QEVmAyY1GsZjYviR0DxRuD78sAc+lZqkUERGRdjG50Si5M6USWExsNXOTuiHMT6y+KS23iJeniIgEMbnRqPzSalnrYm7lJSlrSn9xgPCeiam8PEVEJILJjQbVGyWszz4la20Zxy5YlSXt4RKAuMXb1QiHiEiTmNxoUKahFJUX5RUT+7WUNzWclGNJe7jhXA2+ypGXsBIROTsmNxokt5gYAAJ8PFWMhBqzdnwsRCudpqzO4eUpIiIZmNxokNxiYm8PV/QJ91M5GmqIi16HpRbcvTjm9e+VD4aISGOaldxcvCj/DAFZj6+XvEtNj0QHsVPKhhKjgjCwq1hBd3FVHeZtPKxSRERE2iCc3BiNRrz22msICgpCq1atcPz4cQDA7NmzkZycrHiAJK60Wl6RcHAbL5UjoaYkj+qLsDZi7eErd+eh7rJRpYiIiByfcHIzf/58pKSk4M0334Sb2x9nCCIiIvCf//xH0eDIMnJnSp1np5RdSJ8u3h7e67XNKkRCRKQNwsnNJ598gg8//BDDhw+Hi4uL6fnIyEgcPXpU0eBIHGdKOR5L2sMrao1IXLpLnYCIiByccHJz6tQpdO7c+YbnjUYjLl26pEhQZDnOlHJMiVFBGHC72BiM3FOcHk5E1BDh5ObOO+/Erl03/o9x3bp1iI4WHw5IyuJMKce1cnQ/+Hi4NL3wGskZBtbfEBFdx1V0w5w5c/D000/j1KlTMBqNWL9+PY4dO4ZPPvkEmzZtUiNGElBSKa8NPJ4zpezSvlmDcNusb4T23LsoHXtfeUCliIiIHI/wmZuHHnoIGzduxPfff4+WLVtizpw5+Pnnn7Fx40Y88AB/wNqa3GJif2+xDh2yDjdXvfD08KLKOoxN2adSREREjkf4zA0A3HPPPdiyZYvSsVAzsZhYG+YmdcOmg4U4WyW/hi39aDE2HjyNpMhAFSMjInIMwmduOnXqhHPnzt3wfFlZGTp16qRIUGQZFhNrx96Z4mdBn1uVzfEMRESwILnJy8tDff2NH6C1tbU4dUpssN/OnTuRlJSEwMBA6HQ6bNiwock9tbW1eOWVV9CxY0e4u7sjLCwMK1euFHpfrWIxsXZY0h4OAI8tz1A+GCIiByP7stT//vc/0683b94MHx8f0+/r6+uRnp6OsLAwoTevrq5GZGQkxowZg0cffVTWnqFDh+LMmTNITk5G586dUVhYCKOR3SIAi4m1JjEqCCt3H0dWQYXsPTknr7SHz068U8XIiIjsm+zk5uGHHwYA6HQ6PP3002avtWjRAmFhYVi8eLHQm8fHxyM+Pl72+m+//RY7duzA8ePH4ed3ZeBjUwlVbW0tamv/+NCvqJD/QeFoWEysPWvHx6LLzDSIpO/JGQb0DG2DhO4dVIuLiMieyb4sZTQaYTQaERoaiuLiYtPvjUYjamtrcezYMSQmJqoZK/73v/+hV69eePPNNxEUFITbbrsNL774Ii5cuNDonoULF8LHx8f0CAkJUTVGW2ExsTa56HVYNkz8/lGTV2Wx/oaInJZwzY3BYEC7drYpRj1+/DgyMjKQm5uLL7/8Eu+88w7WrVuHCRMmNLpnxowZKC8vNz0KCgqsGLH1sJhYuxK6Bwq3h1+WgOdSs1SKiIjIvlnUCl5dXY0dO3YgPz8fdXXmwxcnT56sSGANMRqN0Ol0+Pzzz001P2+//Tb+8pe/YPny5fD09Lxhj7u7O9zd3VWLyV6wmFjb5iZ1w7afi5FXKu/vGQDScouQdqiQl6eIyOkIJzfZ2dlISEhATU0Nqqur4efnh5KSEnh5eaF9+/aqJjcdOnRAUFCQWTHzHXfcAUmScPLkSXTp0kW197Z3LCbWvvQXB+DWmWlCeyamZuG3iAT+nRORUxG+LPX8888jKSkJ58+fh6enJ/bu3YsTJ06gZ8+eeOutt9SI0aR///44ffo0qqqqTM/98ssv0Ov1CA4OVvW97d35mrqmF4HFxI7MkvZwCUDc4u1qhENEZLeEk5ucnBxMmzYNer0eLi4uqK2tRUhICN58803MnDlT6GtVVVUhJycHOTk5AK7U8+Tk5CA/Px/AlXqZkSNHmtYPGzYMbdu2xejRo3HkyBHs3LkT06dPx5gxYxq8JOVMdDL/Yy53HdmnxKgg9AjxFtpjOFeDr3LE7kFFROTIhJObFi1aQK+/sq19+/amRMTHx0e4WHf//v2Ijo42TRN/4YUXEB0djTlz5gAACgsLTV8fAFq1aoUtW7agrKwMvXr1wvDhw5GUlIQlS5aIHobmFJU13jF2LV/PFipHQmpbOz4WojnqlNU57J4iIqchXHMTHR2Nffv2oUuXLvjTn/6EOXPmoKSkBJ9++ikiIiKEvtZ9990HSWr8B25KSsoNz3Xt2pVzra5Tb5SQllska227VtovrtY6F70OS5+MwqTVOUL7Yl7/HpmzONyWiLRP+MzN66+/jg4drnRfLFiwAG3atMH48eNx9uxZfPDBB4oHSE3be/wcLlySd5u3AB/nvnynFYlRQRjYVaylv7iqDvM2HlYpIiIi+6GTbnbqRIMqKirg4+OD8vJyeHuL1S7Yq7c2H8Oybb81ua6VuysOzh3EzhkNuW9ROvLOy28PB4Bf5sfDzVX4/zVERDYl8vmt2E+4rKws1e9QTA2TIC8/je3SlomNxqRPHyC8p9drm1WIhIjIfgglN5s3b8aLL76ImTNn4vjx4wCAo0eP4uGHH0bv3r05wNJG5BYT9wxto3IkZG2WtIdX1BqRuHSXOgEREdkB2clNcnIy4uPjkZKSgkWLFqFfv3747LPPEBMTg4CAAOTm5iItTewGY9R8LCamxKggDLhd7K7TuaeuTA8nItIi2cnNu+++i0WLFqGkpARr1qxBSUkJli9fjp9++gnvv/8+7rjjDjXjpEawmJgAYOXofvDxcBHak5xhQN1lnm0lIu2Rndz8/vvvePzxxwEAjz76KFxdXfGvf/3L6e8MbGt7fj8na10rd1f0CfdTORqypX2zBgnvuXdRugqREBHZluzk5sKFC/Dy8gIA6HQ6uLu7m1rCyXZYTExXubnqhaeHF1XWYcxHmSpFRERkG0I38fvPf/6DVq1aAQAuX76MlJQUtGtnfq8NNQdn0o1YTEzXmpvUDZsOFuJs1SXZe7YeO4vXNh3B7MQ7VYyMiMh6ZN/nJiwsDLomBhPpdDpTF5W90tJ9buqNEiLmfiur5ubfQyPxSA9eQnQG9UZJeHo4ACwf1gMJ3Xk2lojsk8jnt+wzN3l5ec2NixTGYmJqyNX2cNHxDJNXZWFwRAIvXxKRw+NtSh0Yi4mpMZZMD78sAc+lZqkUERGR9TC5cWAsJqabWTs+VvgfeFpuEdIOFaoSDxGRtTC5cWC+ni1krWMxsXNy0euwbFi08L5JqVmoNzrVyDki0hgmNw7Mr6W8Ow7LXUfak9A9EGNjOwrtMQJ4fMUP6gRERGQFTG4c2J7fS2StK6upUzkSsmezEyMQLVh/k1VQho0HT6sUERGRuoSTm4qKigYflZWVqKvjh6i11BslfP2TvNoIv5ZuKkdD9m7d+FiIVl09tyqbl6eIyCEJJze+vr5o06bNDQ9fX194enqiY8eOmDt3LieEq4xt4CTCRa/DUsHp4QAQ8/r3ygdDRKQy4eQmJSUFgYGBmDlzJjZs2IANGzZg5syZCAoKwooVK/Dss89iyZIleOONN9SIl/4/toGTqMSoIAzs2q7phdcorqrDvI2HVYqIiEgdQuMXAODjjz/G4sWLMXToUNNzSUlJ6NatGz744AOkp6cjNDQUCxYswMyZMxUNlv7ANnCyRPKovui74DucqZQ/nmHl7jz8I/4OuLmyRI+IHIPwT6sffvgB0dE3tpdGR0djz549AIDY2Fjk5+c3PzpqFGdKkaV2vRwnvKf3/O9UiISISB3CyU1ISAiSk5NveD45ORkhISEAgHPnzqFNG36oqqXeKCEtt0jW2nat2AZO5txc9cLt4eUX6zEmhdPDicgxCF+Weuutt/D444/jm2++Qe/evQEA+/fvx9GjR7Fu3ToAwL59+/DEE08oGymZsJiYmmt2YgT2/l6Cw4XVsvdsPXoWGw+eRlJkoIqRERE1n+yp4NcyGAz44IMP8MsvvwAAbr/9dvz9739HWFiY0vEpTgtTwd/afAzLtv3W5LpW7q44OHcQa26oUd1f/RYVF+uF9vz+OodrEpH1qTIV/Frh4eHshrIhFhOTUvbPGoTbZn0jtGfgW9uw/aUBKkVERNR8FiU3ZWVlyMzMRHFx8Q33sxk5cqQigVHjWExMSnFz1WN0/1B8tFt+A0Be6QWMTdmH5FG9VYyMiMhywsnNxo0bMXz4cFRVVcHb2xs63R9nBnQ6HZMblbGYmJQ2N6kbNh0sxNkq+e3h6UeLWX9DRHZLuFtq2rRpGDNmDKqqqlBWVobz58+bHqWlpWrESNdgMTGpYe/MB4T3TOZ4BiKyU8LJzalTpzB58mR4eXmpEQ81gXcmJjW46HVYMjRSaI8E4C/Ld6sTEBFRMwgnN4MHD8b+/fvViIVkYDExqWVIj2CEtfUQ2pN9shyvbTqiUkRERJYRrrl58MEHMX36dBw5cgTdunVDixYtzF4fMmSIYsHRjXw9WzS9CCwmJsukTxuAW2emCe1JzjCgZ2gbJHTvoFJURERihJObZ555BgAwb968G17T6XSorxe7ZwaJOXm+RtY6v5YsJiZxLnodlg+LxoTUbKF9k1dlYXAE739DRPZB+LKU0Whs9MHERl31Rgnrs0/JWltWU6dyNKRVCd0DkRDhL7TnsgQ8l5qlUkRERGI45teBZBpKUSnzbrJ+Ld1Ujoa0bOmwnnARPAmTlluEtEOF6gRERCRA1mWpJUuW4Nlnn4WHhweWLFly07WTJ09WJDC6UVHFRdlr2QZOzeGi12HpU+KXpyalZuFXXp4iIhuTNVsqPDwc+/fvR9u2bREeHt74F9PpcPz4cUUDVJojz5ZK3nUcr339c5PrvD1ckT2HM6Wo+V7blIvkjBNCe3qE+GL9xP4qRUREzkrx2VIGg6HBX5N15ZfKm+D8SHQQExtSxOzECGSdOI/sggrZe7IKynj3YiKyKdbcOAiRYuJQP95gkZSzbnwsRFPl53j3YiKyIeFW8Pr6eqSkpCA9Pb3BwZlbt25VLDj6A4uJyVZc9DosfTIKk1bnCO2Lef17ZM4SH+tARNRcwmdupkyZgilTpqC+vh4RERGIjIw0e5A6WExMtpQYFYSBXdsJ7SmuqsO8jYdVioiIqHHCZ25Wr16NNWvWICEhQY14qBGlVbWy1nl7cKYUqSN5VF/0XfAdzlTKnx6+cnce/hF/B9xceQWciKxH+CeOm5sbOnfurEYsdBMsJiZ7sOvlOOE9ved/p0IkRESNE05upk2bhnfffRcyOshJISwmJnvh5qrH2NiOQnvKL9ZjTEqmShEREd1I+LJURkYGtm3bhm+++QZ33XXXDYMz169fr1hwdAWLicmezE6MwN7fS3C4UN7ZRADYevQs28OJyGqEkxtfX1888sgjasRCjWAxMdmbr6fch+6vfosKmUk3cKU9PKFbB142JSLVCSU3ly9fxv33349BgwYhICBArZjoOiwmJnu0f9Yg3DbrG6E9A9/ahu0vDVApIiKiK4RqblxdXTFu3DjU1sr7sCVl+HrJu9TEYmKyJjdXPUb3DxXak1d6AWNT9qkUERHRFcIFxX369EF2ttgwPWqe0uo6WeuC27CYmKxrblI33NKqRdMLr5F+tBgbD55WKSIiIgtqbiZMmIBp06bh5MmT6NmzJ1q2bGn2evfu3RULjq44cKJU1rrzNfKSICIl7Z35AG6dmSa0ZzLrb4hIRcLJzZNPPgkAmDx5suk5nU4HSZKg0+lQXy+/wJCaVm+UsOvXEllr+TlBtuCi12HJ0EhMXnNQ9h4JwF+W78aXk2LVC4yInJZwcsOp4NaVaShFdZ28hDGmk9jt8YmUMqRHMN5OP4a8c/I7+7JPluO1TUcwO/FOFSMjImcknNx07Ch2Ay9qHrlt4F5uLuh3a1uVoyFqXPq0AcKXp5IzDOgZ2gYJ3TuoFBUROSPh5OaqI0eOID8/H3V15nUeQ4YMaXZQ9IeSSnmdafERAaxfIJty0euwfFg0JqSKNRxMXpWFwREJ/P4lIsUIJzfHjx/HI488gp9++slUawNcqbsBwJobhcktJvb39lA5EqKmJXQPRMKh00jLPSN7z2UJeC41C8v/2lPFyIjImQi3gk+ZMgXh4eEoLi6Gl5cXDh8+jJ07d6JXr17Yvn27CiE6LxYTkyNaOqwnXAS/H9Nyi5B2qFCdgIjI6QgnN3v27MG8efPQrl076PV66PV6xMbGYuHChWYdVNR8LCYmR+Si12HpU9HC+yalZqHeyIG8RNR8wslNfX09WrduDQBo164dTp++cjOujh074tixY0Jfa+fOnUhKSkJgYCB0Oh02bNgge+/u3bvh6uqKqKgoofd0JCwmJkeV0D1QeHq4EcDjK35QJyAicirCyU1ERAQOHrxyP4u+ffvizTffxO7duzFv3jx06tRJ6GtVV1cjMjIS7733ntC+srIyjBw5EgMHDhTa52jkzpRKYDEx2aHZiRGIDvEW2pNVUMa7FxNRswkXFM+aNQvV1dUAgHnz5iExMRH33HMP2rZtiy+++ELoa8XHxyM+Pl40BIwbNw7Dhg2Di4uL0NkeR5NfWi1rXcytvCRF9mnd+Fh0npkGkYtNnB5ORM0lfOZm8ODBePTRRwEAnTt3xtGjR1FSUoLi4mIMGKD+tN+PPvoIx48fx9y5c2Wtr62tRUVFhdnDEdQbJazPPiVrbRnHLpCdctHrsPTJKOF9/RZsUT4YInIawsnNVb/99hs2b96MCxcuwM/PT8mYGvXrr7/iH//4Bz777DO4uso76bRw4UL4+PiYHiEhISpHqYxMQykqL8orJvZrKW9qOJEtJEYFYWBXsbOLZ6svYUxKpkoREZHWCSc3586dw8CBA3HbbbchISEBhYVX2jfHjh2LadOmKR7gVfX19Rg2bBj++c9/4rbbbpO9b8aMGSgvLzc9CgoKVItRSXKLiQEgwMdTxUiImi95VF/4txabHr716FnW3xCRRYSTm+effx4tWrRAfn4+vLy8TM8/8cQT+PbbbxUN7lqVlZXYv38/Jk2aBFdXV7i6umLevHk4ePAgXF1dsXXr1gb3ubu7w9vb2+zhCOQWE3t7uKJPuHXOnBE1x66X44T3PLcqm+3hRCRMuKD4u+++w+bNmxEcHGz2fJcuXXDixAnFAruet7c3fvrpJ7Pnli9fjq1bt2LdunUIDw9X7b1twddL3qWmR6KDWHhJDsHNVY+xsR2RnCH2c+Kx5RnYMOkelaIiIi0STm6qq6vNzthcVVpaCnd3d6GvVVVVhd9++830e4PBgJycHPj5+SE0NBQzZszAqVOn8Mknn0Cv1yMiIsJsf/v27eHh4XHD81pQWi2vSDi4zY1/F0T2anZiBPb+XoLDhfI6AQEg52QFp4cTkRDhy1L33HMPPvnkE9PvdTodjEYj3nzzTdx///1CX2v//v2Ijo5GdPSVu5m+8MILiI6Oxpw5cwAAhYWFyM/PFw1RE+TOlDrPTilyMF9PuQ/tWorV3yRnGDiegYhk00lXJ1/KlJubi4EDB6JHjx7YunUrhgwZgsOHD6O0tBS7d+/GrbfeqlasiqioqICPjw/Ky8vttv6m3iih+6ubZY1emHT/rXhxcFcrREWknHqjhFtnpgnt0QP49XVODydyViKf3xbdofiXX35BbGwsHnroIVRXV+PRRx9Fdna23Sc2joIzpUjrXPQ6LBkaKbSH4xmISC7hmhsA8PHxwSuvvGL23MmTJ/Hss8/iww8/VCQwZ8aZUuQMhvQIxtvpx5B3Tv5tD66OZ0iKDFQxMiJydBbfxO96586dQ3JyslJfzqmVVMprA4/nTClycOnTxO9qzvZwImqKYskNKUduMbG/t4fKkRCpy0WvwzILxjMMfGub8sEQkWYwubEz9UYJu34tkbWWJ21ICxKjgtBDcHp4XukFzNt4WKWIiMjRMbmxMywmJme0dnys8A+jlbvz2B5ORA2SXVB8dRJ4Y8rKypobC4HFxOScXPQ6LBsWjQmp2UL7JqVm4dcItocTkTnZ/1m6drJ2Q4+OHTti5MiRasbqFFhMTM4qoXsgxsZ2FNrD9nAiaojsMzcfffSRmnHQ/8diYnJmsxMjcCDvPHJOVsjew/ZwIroea27sCIuJiYD/TogV3sP2cCK6FpMbO8JiYiLL28PjFm9XPBYickxMbuwIi4mJrkiMCsLArmIJvOFcDV7bdESliIjIkTC5sSMsJib6Q/KovvBvLT49vO6yUaWIiMhRMLmxI+dr6mStYzExOYtdL8cJ7+k9/zsVIiEiR8Lkxo7oZJ6MkbuOyNG5ueqF28PLL9ZjTEqmShERkSNgcmNHisouyFrn6yl2qp7Ikc1OjEBYW7GzlVuPnsXGg6dVioiI7B2TGztRb5SQllska227Vu4qR0NkX9KnDYDoCUu2hxM5LyY3dmLv8XO4cEleIWSAj6fK0RDZFxe9DkstaA9/bHmG8sEQkd1jcmMn9vx+Tta6Vu6u6BPup3I0RPYnMSoIA24XuwVCzskKtocTOSEmN3ZCgrzT57Fd2rINnJzWytH9cEtL8fZwTg8nci5MbuyE3GLinqFtVI6EyL7tfeUB4T2TUrNYf0PkRJjc2AEWExPJ56LXYcnQSKE9nB5O5FyY3NgBFhMTiRnSI1i4Pfzq9HAi0j4mN3aAxcRE4tKnDRDew/ZwIufA5MYOsJiYSJyl08MHvrVN+WCIyK4wubEDLCYmskxiVBB6hHgL7ckrvYB5Gw+rFBER2QMmNzbGYmKi5lk7Plb4B9nK3XlsDyfSMCY3NsZiYqLmcdHrsGxYtPA+tocTaReTGxtjMTFR8yV0DxSeHs72cCLtYnJjYywmJlLG7MQIRAWL1d+wPZxIm5jc2BiLiYmU898JscJ72B5OpD1Mbmyo3ijh+5+LZa1lMTFR0yxtD49bvF3xWIjIdpjc2FCmoRTlFy/LWstiYiJ5EqOCMLBrO6E9hnM1nB5OpCFMbmyoqOKirHW+ni1YTEwkIHlUX/i3Fp8eXndZXuciEdk3Jjc2VFJZK2vdwDvas5iYSNCul+OE9/Se/50KkRCRtTG5saHzNXWy1vl7iw0IJCLAzVUv3B5efrEeY1IyVYqIiKyFyY0N6WSejJG7jojMzU6MEJ4evvXoWbaHEzk4Jjc2JLcN3NdTrHaAiP6QPm0ARP9/wPZwIsfG5MZGOFOKyDpc9DostaA9/LHlGcoHQ0RWweTGRjhTish6EqOCMOD2tkJ7ck5WsD2cyEExubERzpQisq6Vo/vhlpbi7eGcHk7keJjc2AhnShFZ395XHhDew+nhRI6HyY2NcKYUkfW56HVYMjRSaA+nhxM5HiY3NsBiYiLbGdIjWLg9nNPDiRwLkxsbYDExkW2lTxsgvIft4USOg8mNDbCYmMi2LJ0e3m/BFuWDISLFMbmxARYTE9leYlQQeoR4C+05W32J4xmIHACTGxtgMTGRfVg7Plb4hyDHMxDZPyY3VsZiYiL74aLXYdmwaOF9rL8hsm9MbqyMxcRE9iWhe6Dw9HAAiFu8XflgiEgRTG6sjMXERPZndmIE7r9NbDyD4VwNxzMQ2SkmN1bGYmIi+/TRmH7w8XAR2pOcYUDdZXlnYonIepjcWJmvp7zZNiwmJrK+fbMGCe+5d1G6CpEQUXMwubGyk+drZK3za8liYiJrc3PVC9ffFFXWYWzKPpUiIiJLMLmxonqjhPXZp2StLaupUzkaImrI7MQI4fEM6UeL2R5OZEeY3FhRpqEUlRfrZa31a+mmcjRE1Jj0aQMgWvHG9nAi+2HT5Gbnzp1ISkpCYGAgdDodNmzYcNP169evxwMPPIBbbrkF3t7eiImJwebNm60TrAKKKi7KXss2cCLbcdHrsNSC8QyPLc9QPhgiEmbT5Ka6uhqRkZF47733ZK3fuXMnHnjgAaSlpeHAgQO4//77kZSUhOzsbJUjVUZpVa2sdd4ebAMnsjVLxjPknKxgeziRHXC15ZvHx8cjPj5e9vp33nnH7Pevv/46vvrqK2zcuBHR0eJ3GbW2/NJqWeseiQ5iGziRHVg7PhZdZqZBpNk7OcOAnqFtkNC9g2pxEdHNOXTNjdFoRGVlJfz8Gj/LUVtbi4qKCrOHLYgUE4f6eakcDRHJYel4hsmrslh/Q2RDDp3cvPXWW6iqqsLQoUMbXbNw4UL4+PiYHiEhIVaM8A8sJiZyTAndAzG6f6jQnssS8FxqlkoREVFTHDa5SU1NxT//+U+sWbMG7du3b3TdjBkzUF5ebnoUFBRYMco/sJiYyHHNTeqGMD+x9vC03CKkHSpUKSIiuhmHTG5Wr16Nv/3tb1izZg3i4uJuutbd3R3e3t5mD1tgMTGRY0t/cYDwnompvDxFZAsOl9ysWrUKo0ePxqpVq/Dggw/aOhzZWExM5Nhc9DosE2wPl8Dp4US2YNPkpqqqCjk5OcjJyQEAGAwG5OTkID8/H8CVS0ojR440rU9NTcXIkSOxePFi9O3bF0VFRSgqKkJ5ebktwpeNxcRE2mBJe7jhXA2+ypH375+IlGHT5Gb//v2Ijo42tXG/8MILiI6Oxpw5cwAAhYWFpkQHAD788ENcvnwZEydORIcOHUyPKVOm2CR+uVhMTKQda8fHCt+9eMrqHF6eIrIim97n5r777oMkNf4PPiUlxez327dvVzcglbCYmEg7rt69eNLqHKF9/RZswb7Z4lPHiUicw9XcOCIWExNpS2JUEAZ2bSe052z1JYxJyVQpIiK6FpMbK/D1knepicXERI4jeVRfhLURaw/fevQsp4cTWQGTGysora6TtS64DYuJiRxJ+nTx9nBODydSH5MbKzhwolTWuvM18pIgIrIPlrSHA2wPJ1IbkxuV1Rsl7Pq1RNZaXpEicjyJUUEYcHtboT2GczWcHk6kIiY3Kss0lKK6Tl4beEwnsQJFIrIPK0f3g4+Hi9Ce5AwD6i6LzBsnIrmY3KhMbhu4l5sL+t0q9r8/IrIf+2aJt3nfuyhdhUiIiMmNyuS2gSdEBLBTisiBubnqMTa2o9Ceoso6jE3Zp1JERM6LyY3K5M6UirmVl6SIHN3sxAiEtRVrD08/Wsz2cCKFMblRkchMqTJ2ShFpQvq0AcLjGdgeTqQsJjcq4kwpIudzdTyDqMeWZygfDJGTYnKjIs6UInJOlkwPzzlZwfZwIoUwuVERZ0oROa+142OFf8AmZxiQdqhQlXiInAmTGxXJLSbmTCki7XHR67BsWLTwvsmrslh/Q9RMTG5UIlJMHOrHmVJEWpTQPRCj+4cK7bksAc+lZqkUEZFzYHKjEhYTExEAzE3qhjA/sfbwtNwiXp4iagYmNyphMTERXZX+ovj08ImpvDxFZCkmNyphMTERXWXJ9HAJnB5OZCkmNyrx9ZJ3qYnFxETOwZL2cMO5GnyVI692j4j+wORGJaXV8u44HNyGxcREzmLt+FjhuxdPWZ3Dy1NEgpjcqOTAiVJZ685z7AKR07D07sX9FmxRPhgiDWNyo4J6o4Rdv5bIWssrUkTOJTEqCAO7ig3KPVt9CWNSMlWKiEh7mNyoINNQiuo6eW3gMZ04DZzI2SSP6ouwNmLt4VuPnuX0cCKZmNyoQG4buJebC/rd2lblaIjIHqVPF28P5/RwInmY3KigpFJeG3h8RAA7pYiclCXt4QDbw4nkYHKjArnFxP7eYqeliUhbEqOCMOB2sbO3hnM1nB5O1AQmNwpjMTERiVg5uh98PFyE9iRnGFB32ahSRESOj8mNwlhMTESi9s0aJLzn3kXpKkRCpA1MbhTGYmIiEuXmqsfY2I5Ce4oq6zDmI7aHEzWEyY3CWExMRJaYnRiBsLaC7eHHzrL+hqgBTG4UxmJiIrJU+rQBwuMZkjMMSDtUqEo8RI6KyY2CWExMRM1h6XiGyauyeP8bomswuVEQi4mJqLksmR5+WQKeS81SKSIix8PkRkEsJiYiJawdHyv8wzktt4iXp4j+PyY3CmIxMREpwUWvw7Jh0cL7JqXy8hQRwORGUedr6mStYzExETUloXugcHu4EcDjK35QJyAiB8LkRkE6mSdj5K4jIuc2OzEC0YL1N1kFZZweTk6PyY2CisouyFrn69lC5UiISCvWjY8Vbg/n9HBydkxuFFJvlJCWWyRrbbtW7ipHQ0RaYWl7eMzr3ysfDJGDYHKjkL3Hz+HCJXmD7AJ8PFWOhoi0JDEqCAO7it0+oriqDvM2HlYpIiL7xuRGIXt+PydrXSt3V/QJ91M5GiLSmuRRfeHfWuyS9srdeZweTk6JyY1CJMi7vh3bpS3bwInIIrtejhPe0+u1zSpEQmTfmNwoRG4xcc/QNipHQkRaZcn08IpaIxKX7lIpIiL7xORGASwmJiJrmZ0Ygbs6tBTak3uqgtPDyakwuVEAi4mJyJq+nnIfvD1chPYkZxhYf0NOg8mNAlhMTETWtn/WIOE99y5KVyESIvvD5EYBLCYmImtzc9VjdP9QoT1FlXUYm7JPpYiI7AeTGwWwmJiIbGFuUjfc0kqsPTz9aDHHM5DmMblppnqjhO9/Lpa1lsXERKS0vTMfEN4zmeMZSOOY3DRTpqEU5Rcvy1rLYmIiUpqLXodlguMZJAB/Wb5blXiI7AGTm2Yqqrgoa52vZwsWExORKhKjgtBDcHp49slytoeTZjG5aaaSylpZ6wbe0Z7FxESkmrXjY4V/oCdnGJB2qFCVeIhsiclNM52vqZO1zt/bQ+VIiMiZueh1WDYsWnjf5FVZrL8hzWFy00w6mSdj5K4jIrJUQvdAJET4C+25LAHPpWapFBGRbTC5aSa5beC+nmLtmkREllg6rCdcBP8zlZZbxMtTpCmutg7AkXGmFBHZGxe9DkufisaE1GyhfRNSs6BLBa7mRRKa/jUA6HXA1atacvao/WvGZNuYdDrA080FHXw88FiPYIyJ7QQ3V+ufR7HpmZudO3ciKSkJgYGB0Ol02LBhQ5N7tm/fjh49esDd3R2dO3dGSkqK6nE2hjOliMgeJXQPFJ4eDlz5kDL+/4ecXxtx5bKWyB61f82YbBtTvQRU1dbj1+JqvPHtMdw26xssTLN+V55Nk5vq6mpERkbivffek7XeYDDgwQcfxP3334+cnBxMnToVf/vb37B582aVI20YZ0oRkb2anRiBaMH2cCI1fLDTYPUEx6aXpeLj4xEfHy97/fvvv4/w8HAsXrwYAHDHHXcgIyMD//73vzF48GC1wrwJeR0G93CmFBHZwLrxseg8M03mTyoi9fzfLgOmDepqtUtUDlVQvGfPHsTFxZk9N3jwYOzZs6fRPbW1taioqDB7KCWmUztZ6/7aN0yx9yQikstFr8NSwbsXE6nBKAGf7smz2vs5VHJTVFQEf3/zNkd/f39UVFTgwoWGu5YWLlwIHx8f0yMkJESxePrd2ha+XjfvgvL1aoF+t7ZV7D2JiEQkRgVhYFd5/xEjUtOJ0hqrvZdDJTeWmDFjBsrLy02PgoICxb62i16HNx7tdtM1bzzajZekiMimkkf1hX9r3o6CbKujn5fV3suhkpuAgACcOXPG7LkzZ87A29sbnp4NdyO5u7vD29vb7KGkP0d0wPt/7YEAb/NW7wBvd7z/1x74c0QHRd+PiMgSu16Oa3oRkUr0OmBETJjV3s+h7nMTExODtLQ0s+e2bNmCmJgYG0V0xZ8jOuCBOwOQaShFceVFtG/tgT7hfjxjQ0R2w81Vj7/fG44PdhpsHQo5oWfuCbfq/W5smtxUVVXht99+M/3eYDAgJycHfn5+CA0NxYwZM3Dq1Cl88sknAIBx48Zh2bJleOmllzBmzBhs3boVa9aswddff22rQzBx0esQw9oaIrJjMxLuBAAmOGRVf7833PS9Zy06SZJs1iW4fft23H///Tc8//TTTyMlJQWjRo1CXl4etm/fbrbn+eefx5EjRxAcHIzZs2dj1KhRst+zoqICPj4+KC8vV/wSFRGRI6i7bERyxu9Yt78Ap85fQF39left5S639njnXcZk+zsUi3x+2zS5sQUmN0RERI5H5PPboQqKiYiIiJrC5IaIiIg0hckNERERaQqTGyIiItIUJjdERESkKUxuiIiISFOY3BAREZGmMLkhIiIiTWFyQ0RERJriUIMzlXD1hswVFRU2joSIiIjkuvq5LWewgtMlN5WVlQCAkJAQG0dCREREoiorK+Hj43PTNU43W8poNOL06dNo3bo1dDpd0xsEVFRUICQkBAUFBU4xt4rHq208Xm1ztuMFnO+YtXa8kiShsrISgYGB0OtvXlXjdGdu9Ho9goODVX0Pb29vTXwjycXj1TYer7Y52/ECznfMWjreps7YXMWCYiIiItIUJjdERESkKUxuFOTu7o65c+fC3d3d1qFYBY9X23i82uZsxws43zE72/Fey+kKiomIiEjbeOaGiIiINIXJDREREWkKkxsiIiLSFCY3REREpClMbhTy3nvvISwsDB4eHujbty8yMzNtHZJFFi5ciN69e6N169Zo3749Hn74YRw7dsxszcWLFzFx4kS0bdsWrVq1wmOPPYYzZ86YrcnPz8eDDz4ILy8vtG/fHtOnT8fly5eteSgWeeONN6DT6TB16lTTc1o73lOnTuGvf/0r2rZtC09PT3Tr1g379+83vS5JEubMmYMOHTrA09MTcXFx+PXXX82+RmlpKYYPHw5vb2/4+vpi7NixqKqqsvahNKm+vh6zZ89GeHg4PD09ceutt+K1114zm03jyMe7c+dOJCUlITAwEDqdDhs2bDB7XaljO3ToEO655x54eHggJCQEb775ptqH1qibHfOlS5fw8ssvo1u3bmjZsiUCAwMxcuRInD592uxrONIxN/V3fK1x48ZBp9PhnXfeMXvekY5XMRI12+rVqyU3Nzdp5cqV0uHDh6VnnnlG8vX1lc6cOWPr0IQNHjxY+uijj6Tc3FwpJydHSkhIkEJDQ6WqqirTmnHjxkkhISFSenq6tH//fqlfv37S3XffbXr98uXLUkREhBQXFydlZ2dLaWlpUrt27aQZM2bY4pBky8zMlMLCwqTu3btLU6ZMMT2vpeMtLS2VOnbsKI0aNUr68ccfpePHj0ubN2+WfvvtN9OaN954Q/Lx8ZE2bNggHTx4UBoyZIgUHh4uXbhwwbTmz3/+sxQZGSnt3btX2rVrl9S5c2fpqaeessUh3dSCBQuktm3bSps2bZIMBoO0du1aqVWrVtK7775rWuPIx5uWlia98sor0vr16yUA0pdffmn2uhLHVl5eLvn7+0vDhw+XcnNzpVWrVkmenp7SBx98YK3DNHOzYy4rK5Pi4uKkL774Qjp69Ki0Z88eqU+fPlLPnj3NvoYjHXNTf8dXrV+/XoqMjJQCAwOlf//732avOdLxKoXJjQL69OkjTZw40fT7+vp6KTAwUFq4cKENo1JGcXGxBEDasWOHJElXfni0aNFCWrt2rWnNzz//LAGQ9uzZI0nSlX+Mer1eKioqMq1ZsWKF5O3tLdXW1lr3AGSqrKyUunTpIm3ZskX605/+ZEputHa8L7/8shQbG9vo60ajUQoICJD+9a9/mZ4rKyuT3N3dpVWrVkmSJElHjhyRAEj79u0zrfnmm28knU4nnTp1Sr3gLfDggw9KY8aMMXvu0UcflYYPHy5JkraO9/oPPqWObfny5VKbNm3Mvpdffvll6fbbb1f5iJp2sw/7qzIzMyUA0okTJyRJcuxjbux4T548KQUFBUm5ublSx44dzZIbRz7e5uBlqWaqq6vDgQMHEBcXZ3pOr9cjLi4Oe/bssWFkyigvLwcA+Pn5AQAOHDiAS5cumR1v165dERoaajrePXv2oFu3bvD39zetGTx4MCoqKnD48GErRi/fxIkT8eCDD5odF6C94/3f//6HXr164fHHH0f79u0RHR2N//u//zO9bjAYUFRUZHa8Pj4+6Nu3r9nx+vr6olevXqY1cXFx0Ov1+PHHH613MDLcfffdSE9Pxy+//AIAOHjwIDIyMhAfHw9Ae8d7LaWObc+ePbj33nvh5uZmWjN48GAcO3YM58+ft9LRWK68vBw6nQ6+vr4AtHfMRqMRI0aMwPTp03HXXXfd8LrWjlcuJjfNVFJSgvr6erMPNgDw9/dHUVGRjaJShtFoxNSpU9G/f39EREQAAIqKiuDm5mb6QXHVtcdbVFTU4J/H1dfszerVq5GVlYWFCxfe8JrWjvf48eNYsWIFunTpgs2bN2P8+PGYPHkyPv74YwB/xHuz7+eioiK0b9/e7HVXV1f4+fnZ3fH+4x//wJNPPomuXbuiRYsWiI6OxtSpUzF8+HAA2jveayl1bI70/X29ixcv4uWXX8ZTTz1lGhyptWNetGgRXF1dMXny5AZf19rxyuV0U8FJvokTJyI3NxcZGRm2DkU1BQUFmDJlCrZs2QIPDw9bh6M6o9GIXr164fXXXwcAREdHIzc3F++//z6efvppG0envDVr1uDzzz9Hamoq7rrrLuTk5GDq1KkIDAzU5PHSHy5duoShQ4dCkiSsWLHC1uGo4sCBA3j33XeRlZUFnU5n63DsCs/cNFO7du3g4uJyQ/fMmTNnEBAQYKOomm/SpEnYtGkTtm3bhuDgYNPzAQEBqKurQ1lZmdn6a483ICCgwT+Pq6/ZkwMHDqC4uBg9evSAq6srXF1dsWPHDixZsgSurq7w9/fX1PF26NABd955p9lzd9xxB/Lz8wH8Ee/Nvp8DAgJQXFxs9vrly5dRWlpqd8c7ffp009mbbt26YcSIEXj++edNZ+m0drzXUurYHOn7+6qric2JEyewZcsW01kbQFvHvGvXLhQXFyM0NNT08+vEiROYNm0awsLCAGjreEUwuWkmNzc39OzZE+np6abnjEYj0tPTERMTY8PILCNJEiZNmoQvv/wSW7duRXh4uNnrPXv2RIsWLcyO99ixY8jPzzcdb0xMDH766Sezf1BXf8Bc/8FqawMHDsRPP/2EnJwc06NXr14YPny46ddaOt7+/fvf0Nr/yy+/oGPHjgCA8PBwBAQEmB1vRUUFfvzxR7PjLSsrw4EDB0xrtm7dCqPRiL59+1rhKOSrqamBXm/+Y87FxQVGoxGA9o73WkodW0xMDHbu3IlLly6Z1mzZsgW333472rRpY6Wjke9qYvPrr7/i+++/R9u2bc1e19IxjxgxAocOHTL7+RUYGIjp06dj8+bNALR1vEJsXdGsBatXr5bc3d2llJQU6ciRI9Kzzz4r+fr6mnXPOIrx48dLPj4+0vbt26XCwkLTo6amxrRm3LhxUmhoqLR161Zp//79UkxMjBQTE2N6/Wpr9KBBg6ScnBzp22+/lW655Ra7bI1uyLXdUpKkrePNzMyUXF1dpQULFki//vqr9Pnnn0teXl7SZ599ZlrzxhtvSL6+vtJXX30lHTp0SHrooYcabB+Ojo6WfvzxRykjI0Pq0qWLXbRGX+/pp5+WgoKCTK3g69evl9q1aye99NJLpjWOfLyVlZVSdna2lJ2dLQGQ3n77bSk7O9vUGaTEsZWVlUn+/v7SiBEjpNzcXGn16tWSl5eXzdqEb3bMdXV10pAhQ6Tg4GApJyfH7GfYtZ1AjnTMTf0dX+/6bilJcqzjVQqTG4UsXbpUCg0Nldzc3KQ+ffpIe/futXVIFgHQ4OOjjz4yrblw4YI0YcIEqU2bNpKXl5f0yCOPSIWFhWZfJy8vT4qPj5c8PT2ldu3aSdOmTZMuXbpk5aOxzPXJjdaOd+PGjVJERITk7u4ude3aVfrwww/NXjcajdLs2bMlf39/yd3dXRo4cKB07NgxszXnzp2TnnrqKalVq1aSt7e3NHr0aKmystKahyFLRUWFNGXKFCk0NFTy8PCQOnXqJL3yyitmH3SOfLzbtm1r8N/r008/LUmScsd28OBBKTY2VnJ3d5eCgoKkN954w1qHeIObHbPBYGj0Z9i2bdtMX8ORjrmpv+PrNZTcONLxKkUnSdfcqpOIiIjIwbHmhoiIiDSFyQ0RERFpCpMbIiIi0hQmN0RERKQpTG6IiIhIU5jcEBERkaYwuSEiIiJNYXJDREREmsLkhoicTlhYGN555x1bh0FEKmFyQ0SqGjVqFB5++GEAwH333YepU6da7b1TUlLg6+t7w/P79u3Ds88+a7U4iMi6XG0dABGRqLq6Ori5uVm8/5ZbblEwGiKyNzxzQ0RWMWrUKOzYsQPvvvsudDoddDod8vLyAAC5ubmIj49Hq1at4O/vjxEjRqCkpMS097777sOkSZMwdepUtGvXDoMHDwYAvP322+jWrRtatmyJkJAQTJgwAVVVVQCA7du3Y/To0SgvLze936uvvgrgxstS+fn5eOihh9CqVSt4e3tj6NChOHPmjOn1V199FVFRUfj0008RFhYGHx8fPPnkk6isrFT3D42ILMLkhois4t1330VMTAyeeeYZFBYWorCwECEhISgrK8OAAQMQHR2N/fv349tvv8WZM2cwdOhQs/0ff/wx3NzcsHv3brz//vsAAL1ejyVLluDw4cP4+OOPsXXrVrz00ksAgLvvvhvvvPMOvL29Te/34osv3hCX0WjEQw89hNLSUuzYsQNbtmzB8ePH8cQTT5it+/3337FhwwZs2rQJmzZtwo4dO/DGG2+o9KdFRM3By1JEZBU+Pj5wc3ODl5cXAgICTM8vW7YM0dHReP31103PrVy5EiEhIfjll19w2223AQC6dOmCN9980+xrXlu/ExYWhvnz52PcuHFYvnw53Nzc4OPjA51OZ/Z+10tPT8dPP/0Eg8GAkJAQAMAnn3yCu+66C/v27UPv3r0BXEmCUlJS0Lp1awDAiBEjkJ6ejgULFjTvD4aIFMczN0RkUwcPHsS2bdvQqlUr06Nr164Arpwtuapnz5437P3+++8xcOBABAUFoXXr1hgxYgTOnTuHmpoa2e//888/IyQkxJTYAMCdd94JX19f/Pzzz6bnwsLCTIkNAHTo0AHFxcVCx0pE1sEzN0RkU1VVVUhKSsKiRYtueK1Dhw6mX7ds2dLstby8PCQmJmL8+PFYsGAB/Pz8kJGRgbFjx6Kurg5eXl6KxtmiRQuz3+t0OhiNRkXfg4iUweSGiKzGzc0N9fX1Zs/16NED//3vfxEWFgZXV/k/kg4cOACj0YjFixdDr79yEnrNmjVNvt/17rjjDhQUFKCgoMB09ubIkSMoKyvDnXfeKTseIrIfvCxFRFYTFhaGH3/8EXl5eSgpKYHRaMTEiRNRWlqKp556Cvv27cPvv/+OzZs3Y/To0TdNTDp37oxLly5h6dKlOH78OD799FNTofG171dVVYX09HSUlJQ0eLkqLi4O3bp1w/Dhw5GVlYXMzEyMHDkSf/rTn9CrVy/F/wyISH1MbojIal588UW4uLjgzjvvxC233IL8/HwEBgZi9+7dqK+vx6BBg9CtWzdMnToVvr6+pjMyDYmMjMTbb7+NRYsWISIiAp9//jkWLlxotubuu+/GuHHj8MQTT+CWW265oSAZuHJ56auvvkKbNm1w7733Ii4uDp06dcIXX3yh+PETkXXoJEmSbB0EERERkVJ45oaIiIg0hckNERERaQqTGyIiItIUJjdERESkKUxuiIiISFOY3BAREZGmMLkhIiIiTWFyQ0RERJrC5IaIiIg0hckNERERaQqTGyIiItKU/wfn/VEfoYwR4gAAAABJRU5ErkJggg==\n", |
| "text/plain": [ |
| "<Figure size 640x480 with 1 Axes>" |
| ] |
| }, |
| "metadata": {}, |
| "output_type": "display_data" |
| } |
| ], |
| "source": [ |
| "schedule = TriangularSchedule(min_lr=1, max_lr=2, cycle_length=1000, inc_fraction=0.2)\n", |
| "plot_schedule(schedule)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "2a5d2102", |
| "metadata": {}, |
| "source": [ |
| " <!--notebook-skip-line-->\n", |
| "\n", |
| "\n", |
| "### Cosine\n", |
| "\n", |
| "Continuing with the idea that smooth decay profiles give improved performance over stepwise decay, [Ilya Loshchilov, Frank Hutter (2016)](https://arxiv.org/abs/1608.03983) used **\"cosine annealing\"** schedules to good effect. As with triangular schedules, the original idea was that this should be used as part of a cyclical schedule, but we begin by implementing the cosine annealing component before the full Stochastic Gradient Descent with Warm Restarts (SGDR) method later in the tutorial." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 5, |
| "id": "ee781d46", |
| "metadata": {}, |
| "outputs": [], |
| "source": [ |
| "class CosineAnnealingSchedule():\n", |
| " def __init__(self, min_lr, max_lr, cycle_length):\n", |
| " \"\"\"\n", |
| " min_lr: lower bound for learning rate (float)\n", |
| " max_lr: upper bound for learning rate (float)\n", |
| " cycle_length: iterations between start and finish (int)\n", |
| " \"\"\"\n", |
| " self.min_lr = min_lr\n", |
| " self.max_lr = max_lr\n", |
| " self.cycle_length = cycle_length\n", |
| "\n", |
| " def __call__(self, iteration):\n", |
| " if iteration <= self.cycle_length:\n", |
| " unit_cycle = (1 + math.cos(iteration * math.pi / self.cycle_length)) / 2\n", |
| " adjusted_cycle = (unit_cycle * (self.max_lr - self.min_lr)) + self.min_lr\n", |
| " return adjusted_cycle\n", |
| " else:\n", |
| " return self.min_lr" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "c54b05b1", |
| "metadata": {}, |
| "source": [ |
| "We look at an example of a cosine annealing schedule that smoothing decreases from a learning rate of 2 to 1 across 1000 iterations. After this, the schedule stays at the lower bound indefinietly." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 6, |
| "id": "78829eb0", |
| "metadata": {}, |
| "outputs": [ |
| { |
| "data": { |
| "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABAxklEQVR4nO3deXhTZfrG8TtpaVlb9qWlUGQVilDABcEFARkLVXRGQRmQZVQUpCg6Dii4IsqM/gQUlxkWdwQHGIECYtkEEWQpUpBNClRoQZa2tGBLm/P7gyFDBUpOmtM0yfdzXbkuSN43eV6E5vac55zXZhiGIQAAAD9h93YBAAAAnkS4AQAAfoVwAwAA/ArhBgAA+BXCDQAA8CuEGwAA4FcINwAAwK8Ee7uA0uZwOHT48GFVqVJFNpvN2+UAAAAXGIahU6dOKSIiQnZ78cdmAi7cHD58WFFRUd4uAwAAuCEtLU3169cvdkzAhZsqVapIOveHExYW5uVqAACAK7KzsxUVFeX8Hi9OwIWb86eiwsLCCDcAAPgYV1pKaCgGAAB+hXADAAD8CuEGAAD4FcINAADwK4QbAADgVwg3AADArxBuAACAXyHcAAAAv0K4AQAAfiXg7lBslfwCh/757V59sm6/jp06q0Ljf68ZkoKDbKoYEqT2Dapp8v3tVLk8f/QAAFjBq0duJkyYoGuvvVZVqlRR7dq11bt3b+3ateuK8+bMmaMWLVqofPnyat26tRITE0uh2subkLhDzZ5brL8v3aP07LM6a0gO/e9hSDpbaCjrTIGW7/pVMS8sVfTfFqnJmEWKfelr3f/BOq3adVSFDqP4DwIAAFfk1XCzatUqDRs2TN9//72WLVums2fP6vbbb1dubu5l53z33Xe6//77NWTIEG3ZskW9e/dW7969lZKSUoqV/8+ExB16f3WqW3MLHNLJ02e1bt8JPTjjBzUek6i2Ly7V6H//qDP5hR6uFACAwGAzDKPMHC749ddfVbt2ba1atUo333zzJcf06dNHubm5WrhwofO5G264QW3bttV77713xc/Izs5WeHi4srKySrxxZn6BQ82eW1yi9yhOSJBNf2xXX+PiW6lCSJBlnwMAQFln5vu7TDUUZ2VlSZKqV69+2THr1q1Tt27dijzXo0cPrVu37pLj8/LylJ2dXeThKR+v2++x97qU/EJDn/+QpqvHLVHLsYl6b+Ve5Rc4LP1MAAB8XZkJNw6HQyNHjlSnTp0UExNz2XEZGRmqU6dOkefq1KmjjIyMS46fMGGCwsPDnY+oqCiP1XzgxGmPvdeVnD5r6LUlu9TsucXq8X8rOW0FAMBllJlwM2zYMKWkpGjWrFkefd/Ro0crKyvL+UhLS/PYezesXtFj72XGriO5unrcErV/+Wvl/FbglRoAACirykS4GT58uBYuXKgVK1aofv36xY6tW7eujhw5UuS5I0eOqG7dupccHxoaqrCwsCIPT+nfMdpj7+WO47lnFfPCUsW+uIQjOQAA/JdXw41hGBo+fLjmzZun5cuXq1GjRlec07FjRyUlJRV5btmyZerYsaNVZV5WSLBdj9x85ZqtdvJMoa4et0Q3vrqMkAMACHheDTfDhg3TJ598os8++0xVqlRRRkaGMjIydObMGeeYAQMGaPTo0c7fJyQkaMmSJXrjjTe0c+dOvfDCC9q4caOGDx/ujSVodFzLMhFwJOlwdv65kDPhG0IOACBgefVScJvNdsnnZ8yYoYEDB0qSbr31VkVHR2vmzJnO1+fMmaPnnntO+/fvV9OmTTVx4kTFxcW59JmevBT8QsXdodhb1zfFRoXpy0c7K8h+6T9nAAB8hZnv7zJ1n5vSYFW4uZIz+YV6YcE2JW3P0MnThSrN4yqT72ujO9sV38sEAEBZRrgphrfCzaWcDzyLf0xXdp61x3eiq5dX0lO3cRQHAOCTCDfFKEvh5kKFDkNrdv2qd1ft0Yb9mZadyuIoDgDAFxFuilFWw83v5fxWoL7vr1VKeo7H35ujOAAAX0O4KYavhJvzCh2GVv90VI9/sVk5+Z49nsNRHACAryDcFMPXws2FzuQX6rY3Vig9K89j79mqXiUtSrjVY+8HAIAVfHbjTBSvQkiQ1o3upp9e+oOqVyznkffcnp6r1uMWq9ARUBkXAODHCDc+qEJIkDaPu10pL/RQ5ZCgEr/fqXyHGo9J1MLkQx6oDgAA7yLc+LDK5YOV8tIfNKlvW4+83/BZyRo843uPvBcAAN5CuPEDd7WN1M+vxik2KrzE77V813F1evVrD1QFAIB3EG78RJDdpnnDOmvK/bElfq9D2WfV/NlF9OEAAHwS4cbPxLeJ8MhRnLxCqfGYRCX+eNhDlQEAUDoIN37Ik0dxHvtsi8Yv2u6BqgAAKB2EGz/mqaM4//x2v15ckOKhqgAAsBbhxs956ijOjLUHNHjGeg9VBQCAdQg3AeL8UZwqoe7fF2f5rmPqNWmVB6sCAMDzCDcBJMhu07YX/6CYSPe3nUhJz1FPAg4AoAwj3ASghY/fVKIb/21Pz1HcWys9Vg8AAJ5EuAlQ52/85+5Zqh0Zuer82jeeLQoAAA8g3ASwILtNu8b3VERYiFvzf8nMU6cJyzxcFQAAJUO4gb4b012RVUPdmnsoK183EnAAAGUI4QaSpLV/66aoauXdmns4K1+xLy7xcEUAALiHcAOnb5/pqlub1XBr7skzhYp9aamHKwIAwDzCDYqYOfgG3da8lltzT54uUCeajAEAXka4wUWmD7pOXVu4F3AOZebpDi4TBwB4EeEGlzRt4HUa1Cnarbk/cZk4AMCLCDe4rOfjW2lI52i35v6SmUfAAQB4BeEGxRrbq5UeuqmRW3N/ycxTHFs1AABKGeEGV/Rsz5aa+kA7t+buSM9Rr8mrPVwRAACXR7iBS+Kuqafdr9zh1tyUw6c0ZOYPHq4IAIBLI9zAZSHBdk19INatuUk7j2rB1sMerggAgIsRbmBK3DUReuimaLfmPv75FhU6DM8WBADA7xBuYNqzPd2/iqrNC2zTAACwFuEGbhnby72Ak5Pv4C7GAABLEW7gNncDziEuEQcAWIhwgxIZ26uVBnVqaHrejvQcxU/51oKKAACBjnCDEns+Pka3Na9pet62Q9l6eeEOCyoCAAQywg08Yvqg69WqbiXT86atSVV+gcOCigAAgYpwA49ZNPJW1awUbHreNS8stqAaAECgItzAo9Y/e7vpOb8VSJ1fT7KgGgBAICLcwKOC7Da37mL8y8nfNGjGegsqAgAEGsINPM7duxiv2HWMBmMAQIkRbmCJZ3u6d4n4tDWpSvwx3YKKAACBgnADyzwfH6MuzWqYnvfYZ5vZgwoA4DbCDSw1Y/ANqh8eYnreDeOXWVANACAQEG5guTWju6tCOZupOb/mnqXBGADgFsINSsXW5/9geg4NxgAAdxBuUCpCgu0a0pkGYwCA9Qg3KDVje8UoJqKy6XnDaTAGAJhAuEGpWjjiFjWsVt7UHIekP01da01BAAC/Q7hBqVv+9G2m52z5JYv+GwCASwg3KHXubtFA/w0AwBWEG3hF3DURbjUY038DALgSwg28ZmyvGMVGhZma45B077vfWVMQAMAvEG7gVV8+2tn0X8LNaZlasPWwJfUAAHwf4QZeFWS36W03+m8e/3wLp6cAAJdEuIHXudt/0/UfKyyoBgDg67wablavXq34+HhFRETIZrNp/vz5V5zz6aefqk2bNqpYsaLq1aunwYMH6/jx49YXC0u503+z/8QZvbRgu0UVAQB8lVfDTW5urtq0aaN33nnHpfFr167VgAEDNGTIEG3fvl1z5szRhg0b9NBDD1lcKUqDO/0309fu5/JwAEARXg03d9xxh1555RXdfffdLo1ft26doqOjNWLECDVq1EidO3fWI488og0bNlhcKUqDu/03XB4OALiQT/XcdOzYUWlpaUpMTJRhGDpy5Ii+/PJLxcXFXXZOXl6esrOzizxQdrnTf8Pl4QCAC/lUuOnUqZM+/fRT9enTRyEhIapbt67Cw8OLPa01YcIEhYeHOx9RUVGlWDHcMbZXjNrWN9d/w+XhAIDzfCrc7NixQwkJCRo3bpw2bdqkJUuWaP/+/Ro6dOhl54wePVpZWVnOR1paWilWDHf9+7HOpudweTgAQPKxcDNhwgR16tRJTz/9tK655hr16NFDU6dO1fTp05Wefumm0tDQUIWFhRV5oOwLstv0dt+2pud1e2Olx2sBAPgWnwo3p0+flt1etOSgoCBJkmHwf+z+plfbSHVtUdPUnNTjp9k9HAACnFfDTU5OjpKTk5WcnCxJSk1NVXJysg4ePCjp3CmlAQMGOMfHx8dr7ty5evfdd7Vv3z6tXbtWI0aM0HXXXaeIiAhvLAEWmzbwetWpUs7cnDWpyi9wWFQRAKCs82q42bhxo2JjYxUbe+7y3yeffFKxsbEaN26cJCk9Pd0ZdCRp4MCBevPNN/X2228rJiZG9957r5o3b665c+d6pX6Ujm+f6WZ6TtykVRZUAgDwBTYjwM7nZGdnKzw8XFlZWfTf+JCXF6Zo2poDpuYM6dxIY3u1tKgiAEBpMvP97VM9NwhcY3vFKLpGeVNzpq1J5e7FABCACDfwGUmjbpPN5JzHP+fuxQAQaAg38BlBdpummLw8vNCQHv9sszUFAQDKJMINfEqvtpFqZ3L38MSUDK6eAoAAQriBz5nzaGfTp6eufeVrS2oBAJQ9hBv4HHdOT2X9VqjBM9k9HgACAeEGPsmduxcv3/krm2sCQAAg3MBnuXP34ie+YHNNAPB3hBv4NLN3Ly5wSJOW7baoGgBAWUC4gU8LCbZrUKcGpuZMXrGXozcA4McIN/B5z8e3Vq3K5k5PdXz1G4uqAQB4G+EGfuH7Md1NjT+ak6+XFmy3qBoAgDcRbuAXguw2jejS2NSc6Wv3c3M/APBDhBv4jYTuzWU3eXe/npNXW1MMAMBrCDfwG0F2myb3aWtqzp6judz7BgD8DOEGfqVX20jd1ryGqTkjPufeNwDgTwg38DvTB92g8PJBLo83JN377nfWFQQAKFWEG/ilH5673dT4zWmZnJ4CAD9BuIFfCgm2Ky6mjqk5nJ4CAP9AuIHfmvJAe5m5eIrTUwDgHwg38FtBdpum9G1rag6npwDA9xFu4Nd6tY1U01oVTc1h53AA8G2EG/i9RQm3mBrPzuEA4NsIN/B7IcF2Denc0NQcdg4HAN9FuEFAGNsrRtE1ypuaQ3MxAPgmwg0CRtKo20yNp7kYAHwT4QYBw52dw2kuBgDfQ7hBQDG7czjNxQDgewg3CCju7BxOczEA+BbCDQJOr7aRahcVZmoOzcUA4DsINwhIcx7tbGo8zcUA4DsINwhINBcDgP8i3CBg0VwMAP6JcIOARXMxAPgnwg0CGs3FAOB/CDcIeDQXA4B/Idwg4NFcDAD+hXADiOZiAPAnhBtANBcDgD8h3AD/RXMxAPgHwg1wAZqLAcD3EW6AC9BcDAC+j3AD/A7NxQDg2wg3wO/QXAwAvo1wA1wCzcUA4LsIN8Bl0FwMAL6JcANcBs3FAOCbShRufvvtN0/VAZRJNBcDgO8xHW4cDodefvllRUZGqnLlytq3b58kaezYsZo2bZrHCwS8ieZiAPA9psPNK6+8opkzZ2rixIkKCQlxPh8TE6N//etfHi0OKAtoLgYA32I63Hz00Uf64IMP1K9fPwUFBTmfb9OmjXbu3OnR4oCyguZiAPAdpsPNoUOH1KRJk4uedzgcOnv2rEeKAsoamosBwHeYDjctW7bUt99+e9HzX375pWJjYz1SFFAW0VwMAL4h2OyEcePG6cEHH9ShQ4fkcDg0d+5c7dq1Sx999JEWLlxoRY1AmXC+uXj4rGSX50xesVcJ3ZspyEwqAgCUiOkjN3fddZcWLFigb775RpUqVdK4ceP0008/acGCBerevbsVNQJlBs3FAFD22QzDCKimgOzsbIWHhysrK0thYea+pABJKnQYajwm0dScKffHKr5NhEUVAYD/M/P9bfrIzVVXXaXjx49f9HxmZqauuuoqs28H+ByaiwGgbDMdbvbv36/CwsKLns/Ly9OhQ4dMvdfq1asVHx+viIgI2Ww2zZ8//4pz8vLy9Oyzz6phw4YKDQ1VdHS0pk+fbupzgZKiuRgAyi6XG4q/+uor56+XLl2q8PBw5+8LCwuVlJSk6OhoUx+em5urNm3aaPDgwbrnnntcmnPffffpyJEjmjZtmpo0aaL09HQ5HA5TnwuUFM3FAFB2udxzY7efO8hjs9n0+ynlypVTdHS03njjDfXq1cu9Qmw2zZs3T717977smCVLlqhv377at2+fqlev7tL75uXlKS8vz/n77OxsRUVF0XMDj7jnnW+1OS3b5fHtoqpq7rBOFlYEAP7Jkp4bh8Mhh8OhBg0a6OjRo87fOxwO5eXladeuXW4HG1d99dVX6tChgyZOnKjIyEg1a9ZMTz31lM6cOXPZORMmTFB4eLjzERUVZWmNCCzcuRgAyh7TPTepqamqWbOmFbVc0b59+7RmzRqlpKRo3rx5euutt/Tll1/qscceu+yc0aNHKysry/lIS0srxYrh79xpLh41O5nmYgCwkOmb+EnnemVWrVqlgwcPKj8/v8hrI0aM8Ehhl+JwOGSz2fTpp586e37efPNN/elPf9LUqVNVoUKFi+aEhoYqNDTUspqAhO7N9fbKn+VqXskvNDQlaY9Gdm9mbWEAEKBMh5stW7YoLi5Op0+fVm5urqpXr65jx46pYsWKql27tqXhpl69eoqMjCzSzHz11VfLMAz98ssvatq0qWWfDVyOO83FU5bv0eNdm9JcDAAWMH1a6oknnlB8fLxOnjypChUq6Pvvv9eBAwfUvn17/eMf/7CiRqdOnTrp8OHDysnJcT63e/du2e121a9f39LPBopj9s7FhQaXhgOAVUyHm+TkZI0aNUp2u11BQUHKy8tTVFSUJk6cqDFjxph6r5ycHCUnJys5OVnSuX6e5ORkHTx4UNK5fpkBAwY4xz/wwAOqUaOGBg0apB07dmj16tV6+umnNXjw4EuekgJK05xHO8vMcZjJK/bSewMAFjAdbsqVK+e8LLx27drOIBIeHm66WXfjxo2KjY117ib+5JNPKjY2VuPGjZMkpaenO99fkipXrqxly5YpMzNTHTp0UL9+/RQfH6/JkyebXQbgcUF2mxK6NjE1h32nAMDzTO8tdfvtt2vgwIF64IEH9NBDD+nHH3/UiBEj9PHHH+vkyZNav369VbV6BHtLwUqFDkPNnk1UoYl/Vew7BQBXZuneUq+++qrq1asnSRo/fryqVaumRx99VL/++qvef/999yoG/ESQ3aZht7LvFAB4E7uCAx5W6DDU9NlEly8Nl6QRXZroyR7NrSsKAHycpUduLmfz5s2W36EY8AXnLw03g+ZiAPAcU+Fm6dKleuqppzRmzBjt27dPkrRz50717t1b1157LRtYAv9l9tJwieZiAPAUl8PNtGnTdMcdd2jmzJl6/fXXdcMNN+iTTz5Rx44dVbduXaWkpCgxMdHKWgGfwr5TAOAdLoebSZMm6fXXX9exY8c0e/ZsHTt2TFOnTtW2bdv03nvv6eqrr7ayTsDnuLPv1F+/3MrpKQAoIZfDzc8//6x7771XknTPPfcoODhYf//737kzMFCMhO7NZWaHhTNnHfr+5+PWFQQAAcDlcHPmzBlVrFhRkmSz2RQaGuq8JBzApbnTXDxm/o/WFAMAAcLUxpn/+te/VLlyZUlSQUGBZs6cqZo1axYZY+XGmYAv6tU2UtPX7tPmtGyXxh84fkYLth7mxn4A4CaX73MTHR0tm6344+s2m815FVVZxX1u4A2FDkONx7jecG+3SXvGx7FrOAD8l5nvb5eP3Ozfv7+kdQEBK8hu0x/bRejfm127GsphSI9/tllT/9ze4soAwP947CZ+AIo34Z42psYnpmQov4B7RwGAWYQboJSEBNsVF1PH1Jyek1dbVA0A+C/CDVCKpjzQXma6aPYczeXGfgBgEuEGKEVBdpum9G1rag67hgOAOYQboJT1ahupprUqujy+wCFNWrbbwooAwL+YDjfZ2dmXfJw6dUr5+flW1Aj4nUUJt5ga//ZKdg0HAFeZDjdVq1ZVtWrVLnpUrVpVFSpUUMOGDfX888+zQzhQDLPNxQ6DozcA4CrT4WbmzJmKiIjQmDFjNH/+fM2fP19jxoxRZGSk3n33XT388MOaPHmyXnvtNSvqBfyG2ebiySs4egMArnD5DsXnde3aVY888ojuu+++Is/Pnj1b77//vpKSkvTxxx9r/Pjx2rlzp0eL9QTuUIyyZGHyIQ2flezy+HZRVTV3WCfrCgKAMsrM97fpIzffffedYmNjL3o+NjZW69atkyR17txZBw8eNPvWQMDp1TZSdcNCXR6/OS2TS8MB4ApMh5uoqChNmzbtouenTZumqKgoSdLx48dVrVq1klcHBIC//8ncnYu5NBwAimdqV3BJ+sc//qF7771Xixcv1rXXXitJ2rhxo3bu3Kkvv/xSkvTDDz+oT58+nq0U8FM3NqmpcnbprIs9+OcvDX+yR3NrCwMAH2W650aSUlNT9f7772v37nNXbzRv3lyPPPKIoqOjPV2fx9Fzg7LIbO8Nu4YDCDRmvr/dCje+jHCDsuqed77V5rRsl8eP6NKEozcAAobl4SYzM1MbNmzQ0aNHL7qfzYABA8y+Xaki3KCsKnQYajwm0eXxNkl7X+XoDYDAYOb723TPzYIFC9SvXz/l5OQoLCxMNtv/frDabLYyH26AsirIbtOILo01ecXPLo03JD3+2WZN/XN7awsDAB9j+mqpUaNGafDgwcrJyVFmZqZOnjzpfJw4ccKKGoGAkdC9ucwciElMyVB+AXcDB4ALmQ43hw4d0ogRI1Sxousb/wFwTZDdpsl92pqa03PyamuKAQAfZTrc9OjRQxs3brSiFgD6767htSu5PH7P0Vxu7AcAFzDdc9OzZ089/fTT2rFjh1q3bq1y5coVef3OO+/0WHFAoFo04mY1e26xy+MTZm1RXOt6NBcDgNy4Wspuv/zBHpvNpsLCwhIXZSWuloKv6PPed1q//6TL4+Ni6tJcDMBvWbq3lMPhuOyjrAcbwJd8/JcbTI2nuRgAzjEdbgCUjpBgu+Ji6pia03/a9xZVAwC+w6Wem8mTJ+vhhx9W+fLlNXny5GLHjhgxwiOFAZCmPNBei8ckytVzx+tTTyq/wKGQYP6/BUDgcqnnplGjRtq4caNq1KihRo0aXf7NbDbt27fPowV6Gj038DVm951qVruyvn7yFusKAgAv8PgdilNTUy/5awDW69U2UpOSdmvPr6ddGr/7aI4WbD2s+DYRFlcGAGUTx64BH7AowdyRmCe+2KJCR0DtiQsATqbvc1NYWKiZM2cqKSnpkhtnLl++3GPFATjnfHNxYsoRl8YXOKRJy3azaziAgGT6yE1CQoISEhJUWFiomJgYtWnTpsgDgDWmPNBeZm7RN3nFXo7eAAhIpo/czJo1S7Nnz1ZcXJwV9QC4jCC7TY+b2DVcku599zvNHdbJwqoAoOwxfeQmJCRETZo0saIWAFeQ0L25qaM3m9My2XcKQMAxHW5GjRqlSZMmyeSuDQA84PzRGzNoLgYQaEyfllqzZo1WrFihxYsXq1WrVhdtnDl37lyPFQfgYgndm+vtlT/L1bxCczGAQGP6yE3VqlV1991365ZbblHNmjUVHh5e5AHAWkF2myb3aWtqztRVNBcDCBymjtwUFBSoS5cuuv3221W3bl2ragJwBb3aRmr62n3anJbt0vgChzQlaY9Gdm9mcWUA4H2mjtwEBwdr6NChysvLs6oeAC6a82hnU+MnJ+3h6A2AgGD6tNR1112nLVu2WFELABOC7DaNMNFc7JD0+GebrSsIAMoI0w3Fjz32mEaNGqVffvlF7du3V6VKlYq8fs0113isOADFM9tcnJiSwa7hAPyeS7uCX8huv/iHos1mk2EYstlsKiws9FhxVmBXcPibt5bt0ltJe10e37R2JS178lbrCgIAC3h8V/ALsSs4ULY83rWZ3lmxV2cdVx4rSXuO5rJrOAC/ZjrcNGzY0Io6ALgpyG7T/93XVsNnJbs8J2HWFsW1rqcgu5n7HQOAbzAdbs7bsWOHDh48qPz8/CLP33nnnSUuCoA5vdpGalLSbu359bRL4x3GuebiqX9ub3FlAFD6TIebffv26e6779a2bducvTbSub4bSWW+5wbwV4sSblGz5xa7PJ7mYgD+yvRPtYSEBDVq1EhHjx5VxYoVtX37dq1evVodOnTQypUrLSgRgCtCgu2Ki6ljak7PyastqgYAvMd0uFm3bp1eeukl1axZU3a7XXa7XZ07d9aECRM0YsQIK2oE4KIpD7Q3tWv4+eZiAPAnpsNNYWGhqlSpIkmqWbOmDh8+94OxYcOG2rVrl6n3Wr16teLj4xURESGbzab58+e7PHft2rUKDg5W27ZtTX0m4M+C7DZN6dvW1Bx2DQfgb0yHm5iYGG3dulWSdP3112vixIlau3atXnrpJV111VWm3is3N1dt2rTRO++8Y2peZmamBgwYoK5du5qaBwSCXm0j1bRWRZfHn981HAD8hemb+C1dulS5ubm65557tHfvXvXq1Uu7d+9WjRo19MUXX+i2225zrxCbTfPmzVPv3r2vOLZv375q2rSpgoKCNH/+fCUnJ7v8OdzED4Egv8BhqrnYJmnvq3FcGg6gzDLz/W36yE2PHj10zz33SJKaNGminTt36tixYzp69KjbwcaMGTNmaN++fXr++eddGp+Xl6fs7OwiD8DfmW0uNsS+UwD8h9vXgO7du1dLly7VmTNnVL16dU/WdFl79uzR3/72N33yyScKDnbtKvYJEyYoPDzc+YiKirK4SqBsMNtcfP7ScADwdabDzfHjx9W1a1c1a9ZMcXFxSk9PlyQNGTJEo0aN8niB5xUWFuqBBx7Qiy++qGbNmrk8b/To0crKynI+0tLSLKsRKEvcaS7m0nAA/sB0uHniiSdUrlw5HTx4UBUr/q9psU+fPlqyZIlHi7vQqVOntHHjRg0fPlzBwcEKDg7WSy+9pK1btyo4OFjLly+/5LzQ0FCFhYUVeQCBolfbSDWtXcnl8VwaDsAfmL5D8ddff62lS5eqfv36RZ5v2rSpDhw44LHCfi8sLEzbtm0r8tzUqVO1fPlyffnll2rUqJFlnw34skUjbjbVXMy+UwB8nelwk5ubW+SIzXknTpxQaGioqffKycnR3r17nb9PTU1VcnKyqlevrgYNGmj06NE6dOiQPvroI9ntdsXExBSZX7t2bZUvX/6i5wH8T0iwXddHV9P6/SddGs++UwB8nenTUjfddJM++ugj5+9tNpscDocmTpyoLl26mHqvjRs3KjY2VrGxsZKkJ598UrGxsRo3bpwkKT09XQcPHjRbIoDf+fgvN5gaT3MxAF9m+j43KSkp6tq1q9q1a6fly5frzjvv1Pbt23XixAmtXbtWjRs3tqpWj+A+NwhUj32yUYkpR1we37R2JS178lbrCgIAEyy9z01MTIx2796tzp0766677nLe0G/Lli1lPtgAgYx9pwAECtNHbi7nl19+0UsvvaQPPvjAE29nGY7cIJAtTD6k4bOSXR5vt0l7xnPnYgDeZ+mRm8s5fvy4pk2b5qm3A2ABs/tOnW8uBgBf4rFwA8A3LEq4xdR4mosB+BrCDRBgzO47JXHnYgC+hXADBCCaiwH4M5dv4nd+J/DLyczMLGktAErJ+X2nzDQXc+diAL7C5SM3F+6sfalHw4YNNWDAACtrBeBBNBcD8FceuxTcV3ApOPA/+QUOU/tOSdLuV+5QSDBntAGULq9cCg7A99BcDMAfEW6AAEdzMQB/Q7gBAtz55mIzEmZtUaEjoM5oA/AhhBsANBcD8CuEGwCSuHMxAP9BuAEgieZiAP6DcAPAieZiAP6AcAPAieZiAP6AcAOgCJqLAfg6wg2Ai9BcDMCXEW4AXITmYgC+jHAD4JJoLgbgqwg3AC6J5mIAvopwA+CyaC4G4IsINwCKRXMxAF9DuAFQLHeai+MmrbKoGgC4MsINgCsy21y899fTennhDsvqAYDiEG4AXJE7zcXT1qQq8cd0awoCgGIQbgC4xGxzscTVUwC8g3ADwGVmm4vPOgxNWrbbomoA4NIINwBc5k5z8eQVezl6A6BUEW4AmGK2uViS7n33O0tqAYBLIdwAMMWd5uLNaZlszQCg1BBuAJjWq22k2kWFmZrzxBc0FwMoHYQbAG6Z82hnU6enChyiuRhAqSDcAHCLO6enaC4GUBoINwDc5s7pqY6vfmNRNQBwDuEGQInMebSzqfFHc/L10oLtFlUDAIQbACUUZLdpRJfGpuZMX7ufncMBWIZwA6DEEro3l93kzW96Tl5tTTEAAh7hBkCJBdltmtynrak5e47mcu8bAJYg3ADwiF5tI3Vb8xqm5jz+Ofe+AeB5hBsAHjN90A0KLx9kak63N1ZaUwyAgEW4AeBRPzx3u6nxqcdP6z/JhyyqBkAgItwA8Ch3dg5PmJXM6SkAHkO4AeBx7uwczukpAJ5CuAHgce5szZB6/LReXrjDmoIABBTCDQBL9GobqViTWzNMW5PKzf0AlBjhBoBlvjS5NYMkxU1aZUElAAIJ4QaAZYLsNr1t8vTU3l85PQWgZAg3ACzlzs7h09akKvHHdIsqAuDvCDcALDfn0c6mr556/PPNXB4OwC2EGwCWc+fqqUJDevyzzdYUBMCvEW4AlAp3Tk8lpmRw9RQA0wg3AEqNO6enrn3la0tqAeC/CDcASo07p6eyfivU4JkbrCkIgF8i3AAoVb3aRqpri5qm5izf+asWbD1sUUUA/I1Xw83q1asVHx+viIgI2Ww2zZ8/v9jxc+fOVffu3VWrVi2FhYWpY8eOWrp0aekUC8Bjpg28XnWqlDM15/HPt3D1FACXeDXc5Obmqk2bNnrnnXdcGr969Wp1795diYmJ2rRpk7p06aL4+Hht2bLF4koBeNq3z3QzPafrP1ZYUAkAf2MzDKNM/K+QzWbTvHnz1Lt3b1PzWrVqpT59+mjcuHEujc/OzlZ4eLiysrIUFmbuyg0AnvXigm2asfagqTldW9TWtIHXWlQRgLLKzPe3T/fcOBwOnTp1StWrV7/smLy8PGVnZxd5ACgbno9vrVqVzZ2eStp5lP4bAMXy6XDzj3/8Qzk5ObrvvvsuO2bChAkKDw93PqKiokqxQgBX8v2Y7qbnjKD/BkAxfDbcfPbZZ3rxxRc1e/Zs1a5d+7LjRo8eraysLOcjLS2tFKsEcCVBdpsm39fG1BxD0p+mrrWmIAA+zyfDzaxZs/SXv/xFs2fPVrduxTclhoaGKiwsrMgDQNlyZ7v6iq5R3tScLb9ksXs4gEvyuXDz+eefa9CgQfr888/Vs2dPb5cDwEOSRt1meg67hwO4FK+Gm5ycHCUnJys5OVmSlJqaquTkZB08eO7qidGjR2vAgAHO8Z999pkGDBigN954Q9dff70yMjKUkZGhrKwsb5QPwIOC7DZNfSDW9Lzhn7F7OICivBpuNm7cqNjYWMXGnvuB9uSTTyo2NtZ5WXd6eroz6EjSBx98oIKCAg0bNkz16tVzPhISErxSPwDPirsmQnExdUzNcYj+GwBFlZn73JQW7nMDlG2FDkPNnk1UocmfTEM6N9LYXi2tKQqA1wXMfW4A+J8gu01T7jd/eor+GwDnEW4AlDlx10RoSOeGpufRfwNAItwAKKPG9opRbJS5U8f03wCQCDcAyrAvH+1s+ocU978BQLgBUGYF2W16243Lw+m/AQIb4QZAmeZu/81j9N8AAYtwA6DMc6f/RpJu+/tyC6oBUNYRbgD4BHf6bw6c/E3xU761pB4AZRfhBoBPcLf/ZtuhbBqMgQBDuAHgM9ztv5m2JlX5BQ4LKgJQFhFuAPiUsb1i1KVZDdPzWj+/2IJqAJRFhBsAPmfG4BtUq1I5U3PyCqV2L39tUUUAyhLCDQCf9P2z3U3POZF7Vj0nr7agGgBlCeEGgE8Ksts01Y0G4+2HT+mFr1IsqAhAWUG4AeCz3G0wnvndAY1fxBVUgL8i3ADwae42GP/zW7ZoAPwV4QaAz5sx+AbVDw8xPY8tGgD/RLgB4BfWjO6uyiHmf6RdwyXigN8h3ADwG1tf+IPpOblnDcW+tNSCagB4C+EGgN8Istv0dt+2puedPF2gTq994/mCAHgF4QaAX+nVNlJdW9Q0Pe9QZp7iJq2yoCIApY1wA8DvTBt4vVrVrWR63o70HPXiJn+AzyPcAPBLi0beqkg3rqBKOXxKg2dssKAiAKWFcAPAb60d3V3VKgSZnrd81696nrsYAz6LcAPAr215/g+q5MYl4h9+d0BDZnIEB/BFhBsAfu9HNy4Rl6Sknb8ScAAfRLgB4Pfc3WRTOhdwOEUF+BbCDYCAEHdNhB66KdqtuR9+d0CDZ6z3bEEALEO4ARAwnu3ZSoM6md9FXJKW7zqmXtwHB/AJhBsAAeX5+Bjd1tz8Tf4kKSU9Rz0JOECZR7gBEHCmD7peMfUquzV3e3qO4t5a6dmCAHgU4QZAQFqYcItauRlwdmTkqjN7UQFlFuEGQMBaVIKA80tmnjpNWObhigB4AuEGQEArScA5lJWvjq9+7eGKAJQU4QZAwFuUcItiIqq4NTc9+6yaP7tIhQ7Dw1UBcBfhBgAkLRxxs7o0c+8qqrxCqfGYRC1MPuThqgC4g3ADAP81Y/D1uq15LbfnD5+VrCEzudkf4G2EGwC4wPRB16lrC/cDTtLOY+rJpeKAVxFuAOB3pg28ToM6Rbs9f3tGrtq/tIQ+HMBLCDcAcAnPx7fSkM7Rbs8/frqQPhzASwg3AHAZY3u10kM3NSrRewyflaxB09d5qCIAriDcAEAxnu3ZUlMfaFei91ix+4Raj1vMaSqglBBuAOAK4q6pp59fjVNokPvvcSrfocZjEvXV5l88VxiASyLcAIALguw27RrfU5HhoSV6nxGzt+rWiUkcxQEsRLgBABPWju6mVm7ezfi8/Sd+U+MxiZq/Mc1DVQG4EOEGAExaNOJmdW1Ru8TvM/LLH3XNC4uVX+DwQFUAziPcAIAbpg28VlPujy3x+2T/5lCz5xbrkY83cKoK8BDCDQC4Kb5NhH5+NU5VStJp/F9Lt/+qxmMS9cTnWziSA5QQ4QYASiDIbtO2F/+gLiXYk+pC87YeVrPnFmvoJz9wJAdwE+EGADxgxqDrPHKa6rwlKUfVeEyi/p74EyEHMMlmGEZA/avJzs5WeHi4srKyFBYW5u1yAPiZQoeh68cv07Hcsx59386Na+ifD16rCiElPwUG+CIz39+EGwCwwOCZG7R8568ef9+K5ewa0bWpBne+SiHBHHxH4CDcFINwA6C0LNh6WCM+3yKrfshWrRCsobc0JuggIBBuikG4AVCaCh2Ghn+6SYu3H7H0c8rZpeialfTHdvUJO/BLhJtiEG4AeEN+gUM3TUzSkez8UvvMYLtUpXw5tahbRUNvaazOTWspyG4rtc8HPIlwUwzCDQBv+k/yISXMSvZqDbb/PiTJKMGvJcluk85fzFWS9/LUr6nJuzXZbFKFkCDVCy/v8aOIZr6/vXrccvXq1YqPj1dERIRsNpvmz59/xTkrV65Uu3btFBoaqiZNmmjmzJmW1wkAnnJX20j9/Gqc/tCyjtdqMCQ5/vsoya8dkgoMz7wXNflHTYWGlJNXqD1Hc/Xakl1q9txiTUjcodLm1XCTm5urNm3a6J133nFpfGpqqnr27KkuXbooOTlZI0eO1F/+8hctXbrU4koBwHOC7Da9N6CDdr9yh66LrurtcgBLvb86tdQDTpk5LWWz2TRv3jz17t37smOeeeYZLVq0SCkpKc7n+vbtq8zMTC1ZssSlz+G0FICyJr/AoT//a5027M/0dimAJew2aefLd5ToFJXPnJYya926derWrVuR53r06KF169Zddk5eXp6ys7OLPACgLAkJtmv20E7a/codupYjOfBDDkP6eN3+Uvs8nwo3GRkZqlOn6HnqOnXqKDs7W2fOnLnknAkTJig8PNz5iIqKKo1SAcC0kGC75vw35PRuW8/b5QAedeDE6VL7LJ8KN+4YPXq0srKynI+0tDRvlwQAxQoJtuutvu3086tx+vDBa9WsVkVvlwSUWMPqpff3OLjUPskD6tatqyNHit4I68iRIwoLC1OFChUuOSc0NFShoaGlUR4AeFSQ3aZbrq6tW66urUKHodU/HdVzX23Toaw8b5cGmGK3Sf07Rpfa5/lUuOnYsaMSExOLPLds2TJ17NjRSxUBQOkIstvUpVUdrW1Vxxl0Jn79k3YfyVWht4sDruChmxqV6l2zvRpucnJytHfvXufvU1NTlZycrOrVq6tBgwYaPXq0Dh06pI8++kiSNHToUL399tv661//qsGDB2v58uWaPXu2Fi1a5K0lAECpOx90urQ614OYX+DQtDU/68uNaTp08ozyC8/dcwQoCx65uZFGx7Us1c/06qXgK1euVJcuXS56/sEHH9TMmTM1cOBA7d+/XytXriwy54knntCOHTtUv359jR07VgMHDnT5M7kUHECgOJNfqBcWbNPyHUeUebpAhR6+G60UOHfepSbfukNxmbnPTWkh3AAA4Hv89j43AAAAV0K4AQAAfoVwAwAA/ArhBgAA+BXCDQAA8CuEGwAA4FcINwAAwK8QbgAAgF8h3AAAAL/iUxtnesL5GzJnZ2d7uRIAAOCq89/brmysEHDh5tSpU5KkqKgoL1cCAADMOnXqlMLDw4sdE3B7SzkcDh0+fFhVqlSRzWa78gQTsrOzFRUVpbS0tIDYt4r1+jfW698Cbb1S4K3Z39ZrGIZOnTqliIgI2e3Fd9UE3JEbu92u+vXrW/oZYWFhfvEXyVWs17+xXv8WaOuVAm/N/rTeKx2xOY+GYgAA4FcINwAAwK8QbjwoNDRUzz//vEJDQ71dSqlgvf6N9fq3QFuvFHhrDrT1XijgGooBAIB/48gNAADwK4QbAADgVwg3AADArxBuAACAXyHceMg777yj6OholS9fXtdff702bNjg7ZLcMmHCBF177bWqUqWKateurd69e2vXrl1Fxvz2228aNmyYatSoocqVK+uPf/yjjhw5UmTMwYMH1bNnT1WsWFG1a9fW008/rYKCgtJciltee+012Ww2jRw50vmcv6330KFD+vOf/6waNWqoQoUKat26tTZu3Oh83TAMjRs3TvXq1VOFChXUrVs37dmzp8h7nDhxQv369VNYWJiqVq2qIUOGKCcnp7SXckWFhYUaO3asGjVqpAoVKqhx48Z6+eWXi+xN48vrXb16teLj4xURESGbzab58+cXed1Ta/vxxx910003qXz58oqKitLEiROtXtplFbfms2fP6plnnlHr1q1VqVIlRUREaMCAATp8+HCR9/ClNV/pv/GFhg4dKpvNprfeeqvI8760Xo8xUGKzZs0yQkJCjOnTpxvbt283HnroIaNq1arGkSNHvF2aaT169DBmzJhhpKSkGMnJyUZcXJzRoEEDIycnxzlm6NChRlRUlJGUlGRs3LjRuOGGG4wbb7zR+XpBQYERExNjdOvWzdiyZYuRmJho1KxZ0xg9erQ3luSyDRs2GNHR0cY111xjJCQkOJ/3p/WeOHHCaNiwoTFw4EBj/fr1xr59+4ylS5cae/fudY557bXXjPDwcGP+/PnG1q1bjTvvvNNo1KiRcebMGeeYP/zhD0abNm2M77//3vj222+NJk2aGPfff783llSs8ePHGzVq1DAWLlxopKamGnPmzDEqV65sTJo0yTnGl9ebmJhoPPvss8bcuXMNSca8efOKvO6JtWVlZRl16tQx+vXrZ6SkpBiff/65UaFCBeP9998vrWUWUdyaMzMzjW7duhlffPGFsXPnTmPdunXGddddZ7Rv377Ie/jSmq/03/i8uXPnGm3atDEiIiKM//u//yvymi+t11MINx5w3XXXGcOGDXP+vrCw0IiIiDAmTJjgxao84+jRo4YkY9WqVYZhnPvhUa5cOWPOnDnOMT/99JMhyVi3bp1hGOf+MdrtdiMjI8M55t133zXCwsKMvLy80l2Ai06dOmU0bdrUWLZsmXHLLbc4w42/rfeZZ54xOnfufNnXHQ6HUbduXePvf/+787nMzEwjNDTU+Pzzzw3DMIwdO3YYkowffvjBOWbx4sWGzWYzDh06ZF3xbujZs6cxePDgIs/dc889Rr9+/QzD8K/1/v6Lz1Nrmzp1qlGtWrUif5efeeYZo3nz5hav6MqK+7I/b8OGDYYk48CBA4Zh+PaaL7feX375xYiMjDRSUlKMhg0bFgk3vrzekuC0VAnl5+dr06ZN6tatm/M5u92ubt26ad26dV6szDOysrIkSdWrV5ckbdq0SWfPni2y3hYtWqhBgwbO9a5bt06tW7dWnTp1nGN69Oih7Oxsbd++vRSrd92wYcPUs2fPIuuS/G+9X331lTp06KB7771XtWvXVmxsrP75z386X09NTVVGRkaR9YaHh+v6668vst6qVauqQ4cOzjHdunWT3W7X+vXrS28xLrjxxhuVlJSk3bt3S5K2bt2qNWvW6I477pDkf+u9kKfWtm7dOt18880KCQlxjunRo4d27dqlkydPltJq3JeVlSWbzaaqVatK8r81OxwO9e/fX08//bRatWp10ev+tl5XEW5K6NixYyosLCzyxSZJderUUUZGhpeq8gyHw6GRI0eqU6dOiomJkSRlZGQoJCTE+YPivAvXm5GRcck/j/OvlTWzZs3S5s2bNWHChIte87f17tu3T++++66aNm2qpUuX6tFHH9WIESP04YcfSvpfvcX9fc7IyFDt2rWLvB4cHKzq1auXufX+7W9/U9++fdWiRQuVK1dOsbGxGjlypPr16yfJ/9Z7IU+tzZf+fv/eb7/9pmeeeUb333+/c+NIf1vz66+/ruDgYI0YMeKSr/vbel0VcLuCw3XDhg1TSkqK1qxZ4+1SLJOWlqaEhAQtW7ZM5cuX93Y5lnM4HOrQoYNeffVVSVJsbKxSUlL03nvv6cEHH/RydZ43e/Zsffrpp/rss8/UqlUrJScna+TIkYqIiPDL9eJ/zp49q/vuu0+GYejdd9/1djmW2LRpkyZNmqTNmzfLZrN5u5wyhSM3JVSzZk0FBQVddPXMkSNHVLduXS9VVXLDhw/XwoULtWLFCtWvX9/5fN26dZWfn6/MzMwi4y9cb926dS/553H+tbJk06ZNOnr0qNq1a6fg4GAFBwdr1apVmjx5soKDg1WnTh2/Wm+9evXUsmXLIs9dffXVOnjwoKT/1Vvc3+e6devq6NGjRV4vKCjQiRMnytx6n376aefRm9atW6t///564oknnEfp/G29F/LU2nzp7/d554PNgQMHtGzZMudRG8m/1vztt9/q6NGjatCggfPn14EDBzRq1ChFR0dL8q/1mkG4KaGQkBC1b99eSUlJzuccDoeSkpLUsWNHL1bmHsMwNHz4cM2bN0/Lly9Xo0aNirzevn17lStXrsh6d+3apYMHDzrX27FjR23btq3IP6jzP2B+/8XqbV27dtW2bduUnJzsfHTo0EH9+vVz/tqf1tupU6eLLu3fvXu3GjZsKElq1KiR6tatW2S92dnZWr9+fZH1ZmZmatOmTc4xy5cvl8Ph0PXXX18Kq3Dd6dOnZbcX/TEXFBQkh8Mhyf/WeyFPra1jx45avXq1zp496xyzbNkyNW/eXNWqVSul1bjufLDZs2ePvvnmG9WoUaPI6/605v79++vHH38s8vMrIiJCTz/9tJYuXSrJv9Zrirc7mv3BrFmzjNDQUGPmzJnGjh07jIcfftioWrVqkatnfMWjjz5qhIeHGytXrjTS09Odj9OnTzvHDB061GjQoIGxfPlyY+PGjUbHjh2Njh07Ol8/f2n07bffbiQnJxtLliwxatWqVSYvjb6UC6+WMgz/Wu+GDRuM4OBgY/z48caePXuMTz/91KhYsaLxySefOMe89tprRtWqVY3//Oc/xo8//mjcddddl7x8ODY21li/fr2xZs0ao2nTpmXi0ujfe/DBB43IyEjnpeBz5841atasafz1r391jvHl9Z46dcrYsmWLsWXLFkOS8eabbxpbtmxxXhnkibVlZmYaderUMfr372+kpKQYs2bNMipWrOi1y4SLW3N+fr5x5513GvXr1zeSk5OL/Ay78EogX1rzlf4b/97vr5YyDN9ar6cQbjxkypQpRoMGDYyQkBDjuuuuM77//ntvl+QWSZd8zJgxwznmzJkzxmOPPWZUq1bNqFixonH33Xcb6enpRd5n//79xh133GFUqFDBqFmzpjFq1Cjj7Nmzpbwa9/w+3PjbehcsWGDExMQYoaGhRosWLYwPPvigyOsOh8MYO3asUadOHSM0NNTo2rWrsWvXriJjjh8/btx///1G5cqVjbCwMGPQoEHGqVOnSnMZLsnOzjYSEhKMBg0aGOXLlzeuuuoq49lnny3yRefL612xYsUl/70++OCDhmF4bm1bt241OnfubISGhhqRkZHGa6+9VlpLvEhxa05NTb3sz7AVK1Y438OX1nyl/8a/d6lw40vr9RSbYVxwq04AAAAfR88NAADwK4QbAADgVwg3AADArxBuAACAXyHcAAAAv0K4AQAAfoVwAwAA/ArhBgAA+BXCDYCAEx0drbfeesvbZQCwCOEGgKUGDhyo3r17S5JuvfVWjRw5stQ+e+bMmapatepFz//www96+OGHS60OAKUr2NsFAIBZ+fn5CgkJcXt+rVq1PFgNgLKGIzcASsXAgQO1atUqTZo0STabTTabTfv375ckpaSk6I477lDlypVVp04d9e/fX8eOHXPOvfXWWzV8+HCNHDlSNWvWVI8ePSRJb775plq3bq1KlSopKipKjz32mHJyciRJK1eu1KBBg5SVleX8vBdeeEHSxaelDh48qLvuukuVK1dWWFiY7rvvPh05csT5+gsvvKC2bdvq448/VnR0tMLDw9W3b1+dOnXK2j80AG4h3AAoFZMmTVLHjh310EMPKT09Xenp6YqKilJmZqZuu+02xcbGauPGjVqyZImOHDmi++67r8j8Dz/8UCEhIVq7dq3ee+89SZLdbtfkyZO1fft2ffjhh1q+fLn++te/SpJuvPFGvfXWWwoLC3N+3lNPPXVRXQ6HQ3fddZdOnDihVatWadmyZdq3b5/69OlTZNzPP/+s+fPna+HChVq4cKFWrVql1157zaI/LQAlwWkpAKUiPDxcISEhqlixourWret8/u2331ZsbKxeffVV53PTp09XVFSUdu/erWbNmkmSmjZtqokTJxZ5zwv7d6Kjo/XKK69o6NChmjp1qkJCQhQeHi6bzVbk834vKSlJ27ZtU2pqqqKioiRJH330kVq1aqUffvhB1157raRzIWjmzJmqUqWKJKl///5KSkrS+PHjS/YHA8DjOHIDwKu2bt2qFStWqHLlys5HixYtJJ07WnJe+/btL5r7zTffqGvXroqMjFSVKlXUv39/HT9+XKdPn3b583/66SdFRUU5g40ktWzZUlWrVtVPP/3kfC46OtoZbCSpXr16Onr0qKm1AigdHLkB4FU5OTmKj4/X66+/ftFr9erVc/66UqVKRV7bv3+/evXqpUcffVTjx49X9erVtWbNGg0ZMkT5+fmqWLGiR+ssV65ckd/bbDY5HA6PfgYAzyDcACg1ISEhKiwsLPJcu3bt9O9//1vR0dEKDnb9R9KmTZvkcDj0xhtvyG4/dxB69uzZV/y837v66quVlpamtLQ059GbHTt2KDMzUy1btnS5HgBlB6elAJSa6OhorV+/Xvv379exY8fkcDg0bNgwnThxQvfff79++OEH/fzzz1q6dKkGDRpUbDBp0qSJzp49qylTpmjfvn36+OOPnY3GF35eTk6OkpKSdOzYsUuerurWrZtat26tfv36afPmzdqwYYMGDBigW265RR06dPD4nwEA6xFuAJSap556SkFBQWrZsqVq1aqlgwcPKiIiQmvXrlVhYaFuv/12tW7dWiNHjlTVqlWdR2QupU2bNnrzzTf1+uuvKyYmRp9++qkmTJhQZMyNN96ooUOHqk+fPqpVq9ZFDcnSudNL//nPf1StWjXdfPPN6tatm6666ip98cUXHl8/gNJhMwzD8HYRAAAAnsKRGwAA4FcINwAAwK8QbgAAgF8h3AAAAL9CuAEAAH6FcAMAAPwK4QYAAPgVwg0AAPArhBsAAOBXCDcAAMCvEG4AAIBf+X93hKBucPtMUwAAAABJRU5ErkJggg==\n", |
| "text/plain": [ |
| "<Figure size 640x480 with 1 Axes>" |
| ] |
| }, |
| "metadata": {}, |
| "output_type": "display_data" |
| } |
| ], |
| "source": [ |
| "schedule = CosineAnnealingSchedule(min_lr=1, max_lr=2, cycle_length=1000)\n", |
| "plot_schedule(schedule)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "f176f010", |
| "metadata": {}, |
| "source": [ |
| " <!--notebook-skip-line-->\n", |
| "\n", |
| "\n", |
| "## Custom Schedule Modifiers\n", |
| "\n", |
| "We now take a look some adjustments that can be made to existing schedules. We see how to add linear warm-up and its compliment linear cool-down, before using this to implement the \"1-Cycle\" schedule used by [Leslie N. Smith, Nicholay Topin (2017)](https://arxiv.org/abs/1708.07120) for \"super-convergence\". We then look at cyclical schedules and implement the original cyclical schedule from [Leslie N. Smith (2015)](https://arxiv.org/abs/1506.01186) before finishing with a look at [\"SGDR: Stochastic Gradient Descent with Warm Restarts\" by Ilya Loshchilov, Frank Hutter (2016)](https://arxiv.org/abs/1608.03983).\n", |
| "\n", |
| "Unlike the schedules above and those implemented in `mx.lr_scheduler`, these classes are designed to modify existing schedules so they take the argument `schedule` (for initialized schedules) or `schedule_class` when being initialized.\n", |
| "\n", |
| "### Warm-Up\n", |
| "\n", |
| "Using the idea of linear warm-up of the learning rate proposed in [\"Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour\" by Priya Goyal et al. (2017)](https://arxiv.org/abs/1706.02677), we implement a wrapper class that adds warm-up to an existing schedule. Going from `start_lr` to the initial learning rate of the `schedule` over `length` iterations, this adjustment is useful when training with large batch sizes." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 7, |
| "id": "9e58d294", |
| "metadata": {}, |
| "outputs": [], |
| "source": [ |
| "class LinearWarmUp():\n", |
| " def __init__(self, schedule, start_lr, length):\n", |
| " \"\"\"\n", |
| " schedule: a pre-initialized schedule (e.g. TriangularSchedule(min_lr=0.5, max_lr=2, cycle_length=500))\n", |
| " start_lr: learning rate used at start of the warm-up (float)\n", |
| " length: number of iterations used for the warm-up (int)\n", |
| " \"\"\"\n", |
| " self.schedule = schedule\n", |
| " self.start_lr = start_lr\n", |
| " # calling mx.lr_scheduler.LRScheduler effects state, so calling a copy\n", |
| " self.finish_lr = copy.copy(schedule)(0)\n", |
| " self.length = length\n", |
| "\n", |
| " def __call__(self, iteration):\n", |
| " if iteration <= self.length:\n", |
| " return iteration * (self.finish_lr - self.start_lr)/(self.length) + self.start_lr\n", |
| " else:\n", |
| " return self.schedule(iteration - self.length)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "b734efa7", |
| "metadata": {}, |
| "source": [ |
| "As an example, we add a linear warm-up of the learning rate (from 0 to 1 over 250 iterations) to a stepwise decay schedule. We first create the `MultiFactorScheduler` (and set the `base_lr`) and then pass it to `LinearWarmUp` to add the warm-up at the start. We can use `LinearWarmUp` with any other schedule including `CosineAnnealingSchedule`." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 8, |
| "id": "652138c7", |
| "metadata": {}, |
| "outputs": [ |
| { |
| "data": { |
| "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA/CElEQVR4nO3deXxU9b3/8fdkTwyZACEJgWAAUUBWE+BGXOo1Gpei1N5KKQWkVi+KFQQ3XMBfrQZREawLrkV7tWC9atWyXIyiIsFAICqiqCQYRJIQQhYIZGDm/P7gwchIgDPJnNnyej4e83gkZz6T+XwBM2/P+ZxzbIZhGAIAAAgTEYFuAAAAwJcINwAAIKwQbgAAQFgh3AAAgLBCuAEAAGGFcAMAAMIK4QYAAISVqEA34G8ul0s//vijOnToIJvNFuh2AACACYZhqLGxURkZGYqIOPG+mXYXbn788UdlZmYGug0AANAK27dvV/fu3U9Y0+7CTYcOHSQd/sNJSkoKcDcAAMCMhoYGZWZmuj/HT6TdhZsjh6KSkpIINwAAhBgzIyUMFAMAgLBCuAEAAGGFcAMAAMIK4QYAAIQVwg0AAAgrhBsAABBWCDcAACCsEG4AAEBYIdwAAICw0u6uUIzA2+9w6r53vlDhl5Xa0+SUIcmQdOSak1Z8LUkRNsllWPceVvRkkxQdHaHMjvH69Vnd9Ydzeikmiv8nAYATCehvyY8++kijRo1SRkaGbDab3nrrrZO+ZtWqVTrrrLMUGxur0047TYsWLbK8T/jOH19ap36zlmvJuh2qaXLKKcmlwx/mLgu/dkk6ZFj7Hlb05JR04KBL31bv05zlW3T6PctUsHRzG/4GACD8BTTc7Nu3T4MHD9aTTz5pqr68vFyXX365LrjgApWWlmratGn64x//qBUrVljcKXzhiic+1ntfVQe6jZD3zEflBBwAOIGAHpa69NJLdemll5quX7hwoXr27KlHH31UktSvXz+tXr1ajz32mPLz861qEz7w9oYf9PkPDYFuI2w893G5Zlzcl0NUANCCkPrNWFRUpLy8PI9t+fn5KioqOu5rmpub1dDQ4PGAfzldhu5484tAtxFWXIb096JtgW4DAIJSSIWbyspKpaWleWxLS0tTQ0OD9u/f3+JrCgoKZLfb3Y/MzEx/tIqjFJfXav9BV6DbCDvf1zYFugUACEohFW5aY+bMmaqvr3c/tm/fHuiW2p3KhgOBbiEsndopIdAtAEBQCqlTwdPT01VVVeWxraqqSklJSYqPj2/xNbGxsYqNjfVHeziOmsbmQLcQdiJs0vjcrEC3AQBBKaT23OTm5qqwsNBj28qVK5WbmxugjmBGbRPhxteuO7cnw8QAcBwB/e24d+9elZaWqrS0VNLhU71LS0tVUVEh6fAhpQkTJrjrJ0+erLKyMt1+++36+uuv9dRTT+m1117TLbfcEoj2YVLJtj2BbiGs/Pd5PTXzsv6BbgMAglZAD0utX79eF1xwgfv76dOnS5ImTpyoRYsWaefOne6gI0k9e/bUv//9b91yyy1asGCBunfvrueff57TwIOY02WodHud6foIcYXio7/mCsUA4D2bYRhGoJvwp4aGBtntdtXX1yspKSnQ7YS9T76r0bjnPzVV+8q1IzSyT4rFHQEAQpE3n9/8LyAstWZrjam6uKgI/UfvzhZ3AwBoDwg3sNS68lpTdYO62xUZYTt5IQAAJ0G4gWW8mbfJyepobTMAgHaDcAPLrC3bLYfT3EjXyN5dLO4GANBeEG5gGeZtAACBQLiBZZi3AQAEAuEGlmDeBgAQKIQbWIJ5GwBAoBBuYAnmbQAAgUK4gSV27Nlvqo55GwCArxFuYIn9jkOm6rKZtwEA+BjhBj7ndBn6xORhqU4JMRZ3AwBobwg38Lni8lrtbXaZqk1JjLW4GwBAe0O4gc9VNhwwXZtuj7ewEwBAe0S4gc+t/rbaVF1SXJSG9+xkcTcAgPaGcAOfcroMLdtUaao2t3dnzpQCAPgc4QY+VVxeqyaHuXmbPqmJFncDAGiPCDfwKW/mbXJ7pVjYCQCgvSLcwKfMztvER3NlYgCANQg38Blv5m3OO70L8zYAAEsQbuAzzNsAAIIB4QY+w7wNACAYEG7gMzWNzabqmLcBAFiJcAOfqW0yF26YtwEAWIlwA58p2bbHVF18dKTFnQAA2jPCDXzC6TJUur3OVG1Gcpy1zQAA2jXCDXxibdluOZyGqdqRvbtY3A0AoD0j3MAn1mytMVUXF8UwMQDAWoQb+MS68lpTdYO62xkmBgBYinCDNvNm3iYnq6O1zQAA2j3CDdqMeRsAQDAh3KDNmLcBAAQTwg3ajHkbAEAwIdygTZi3AQAEG8IN2oR5GwBAsCHcoE2YtwEABBvCDdqEeRsAQLAh3KDVmLcBAAQjwg1ajXkbAEAwItyg1Yq27jZVx7wNAMCfCDdoNUPm9tr8om8X5m0AAH5DuEGr7axrMlU3NJN5GwCA/xBu0CpOl6HlX1aZqq3ff9DibgAA+AnhBq1SXF6rJofLVC1HpAAA/kS4QatUNhwwXZvbK8XCTgAA8ES4Qaus/rbaVF18NGdKAQD8i3ADrzldhpZtqjRVe97pnCkFAPAvwg285s28TZ/URIu7AQDAE+EGXmPeBgAQzAg38BrzNgCAYEa4gVeYtwEABDvCDbzCvA0AINgRbuAV5m0AAMGOcAOv1DQ2m6pj3gYAECiEG3iltslcuGHeBgAQKIQbeKVk2x5TdfHRkRZ3AgBAywg3MM3pMlS6vc5UbUZynLXNAABwHAEPN08++aSysrIUFxenESNGqLi4+IT18+fP1xlnnKH4+HhlZmbqlltu0YED5odc0Xpry3bL4TRM1Y7s3cXibgAAaFlAw82SJUs0ffp0zZ49Wxs2bNDgwYOVn5+v6uqWLxL36quv6s4779Ts2bP11Vdf6YUXXtCSJUt01113+bnz9mnN1hpTdXFRDBMDAAInoOFm3rx5uu666zRp0iT1799fCxcuVEJCgl588cUW69esWaORI0fqd7/7nbKysnTxxRdr7NixJ9zb09zcrIaGBo8HWmddea2pukHd7QwTAwACJmDhxuFwqKSkRHl5eT81ExGhvLw8FRUVtfias88+WyUlJe4wU1ZWpqVLl+qyyy477vsUFBTIbre7H5mZmb5dSDvhzbxNTlZHa5sBAOAEogL1xjU1NXI6nUpLS/PYnpaWpq+//rrF1/zud79TTU2NzjnnHBmGoUOHDmny5MknPCw1c+ZMTZ8+3f19Q0MDAacVmLcBAISKgA8Ue2PVqlV68MEH9dRTT2nDhg1644039O9//1v333//cV8TGxurpKQkjwe8x7wNACBUBGzPTUpKiiIjI1VVVeWxvaqqSunp6S2+5t5779X48eP1xz/+UZI0cOBA7du3T9dff73uvvtuRUSEVFYLKczbAABCRcDSQExMjLKzs1VYWOje5nK5VFhYqNzc3BZf09TUdEyAiYw8fLE4wzB3yATeY94GABBKArbnRpKmT5+uiRMnKicnR8OHD9f8+fO1b98+TZo0SZI0YcIEdevWTQUFBZKkUaNGad68eRo6dKhGjBih7777Tvfee69GjRrlDjnwPeZtAAChJKDhZsyYMdq1a5dmzZqlyspKDRkyRMuXL3cPGVdUVHjsqbnnnntks9l0zz33aMeOHerSpYtGjRqlBx54IFBLaBeYtwEAhBKb0c6O5zQ0NMhut6u+vp7hYpOmLd6ot0p/PGnd8KyOem3y2X7oCADQ3njz+c0ELk5qv+OQqbps5m0AAEGAcIMTcroMfWLysFSnhBiLuwEA4OQINzih4vJa7W12mapNSYy1uBsAAE6OcIMTqmwwf8f1dHu8hZ0AAGAO4QYntPrblu/Q/nNJcVEa3rOTxd0AAHByhBscl9NlaNmmSlO1ub07c2ViAEBQINzguIrLa9XkMDdv0yc10eJuAAAwh3CD4/Jm3ia3V4qFnQAAYB7hBsdldt4mPporEwMAggfhBi3yZt7mvNO7MG8DAAgahBu0iHkbAECoItygRczbAABCFeEGLappbDZVx7wNACDYEG7Qotomc+GGeRsAQLAh3KBFJdv2mKqLj460uBMAALxDuMExnC5DpdvrTNVmJMdZ2wwAAF4i3OAYa8t2y+E0TNWO7N3F4m4AAPAO4QbHWLO1xlRdXBTDxACA4EO4wTHWldeaqhvU3c4wMQAg6BBu4MGbeZucrI7WNgMAQCsQbuCBeRsAQKgj3MAD8zYAgFBHuIEH5m0AAKGOcAM35m0AAOGAcAM35m0AAOGAcAM35m0AAOGAcAO3HXv2m6pj3gYAEMwIN3Db7zhkqi6beRsAQBAj3EDS4WHiT0weluqUEGNxNwAAtB7hBpKk4vJa7W12mapNSYy1uBsAAFqPcANJUmXDAdO16fZ4CzsBAKBtCDeQJK3+ttpUXVJclIb37GRxNwAAtB7hBnK6DC3bVGmqNrd3Z86UAgAENcINVFxeqyaHuXmbPqmJFncDAEDbEG7g1bxNbq8UCzsBAKDtCDcwPW8TH82ViQEAwY9w0855M29z3uldmLcBAAQ9wk07x7wNACDcEG7aOeZtAADhhnDTztU0NpuqY94GABAqCDftXG2TuXDDvA0AIFQQbtq5km17TNXFR0da3AkAAL5BuGnHnC5DpdvrTNVmJMdZ2wwAAD5CuGnH1pbtlsNpmKod2buLxd0AAOAbhJt2bM3WGlN1cVEMEwMAQgfhph1bV15rqm5QdzvDxACAkNGmcHPggPlrpCC4eDNvk5PV0dpmAADwIa/Djcvl0v33369u3bopMTFRZWVlkqR7771XL7zwgs8bhDWYtwEAhCuvw81f/vIXLVq0SHPnzlVMTIx7+4ABA/T888/7tDlYh3kbAEC48jrcvPzyy3r22Wc1btw4RUb+dO2TwYMH6+uvv/Zpc7AO8zYAgHDldbjZsWOHTjvttGO2u1wuHTx40CdNwVrM2wAAwpnX4aZ///76+OOPj9n++uuva+jQoT5pCtZi3gYAEM6ivH3BrFmzNHHiRO3YsUMul0tvvPGGtmzZopdfflnvvvuuFT3Cx5i3AQCEM6/33Fx55ZV655139N577+mUU07RrFmz9NVXX+mdd97RRRddZEWP8LEde/abqmPeBgAQirzecyNJ5557rlauXOnrXuAn+x2HTNVlM28DAAhBXu+56dWrl3bv3n3M9rq6OvXq1csnTcE6TpehT0weluqUEHPyIgAAgozX4Wbbtm1yOp3HbG9ubtaOHTu8buDJJ59UVlaW4uLiNGLECBUXF5+wvq6uTlOmTFHXrl0VGxur008/XUuXLvX6fdur4vJa7W12mapNSYy1uBsAAHzP9GGpt99+2/31ihUrZLfb3d87nU4VFhYqKyvLqzdfsmSJpk+froULF2rEiBGaP3++8vPztWXLFqWmph5T73A4dNFFFyk1NVWvv/66unXrpu+//17JyclevW97Vtlg/pYZ6fZ4CzsBAMAaNsMwTJ0THBFxeCePzWbTz18SHR2trKwsPfroo/rlL39p+s1HjBihYcOG6YknnpB0+Fo5mZmZ+tOf/qQ777zzmPqFCxfq4Ycf1tdff63o6GjT73O0hoYG2e121dfXKykpqVU/I5TNeG2j/nfDjyetS4qL0sZZFzNQDAAICt58fps+LOVyueRyudSjRw9VV1e7v3e5XGpubtaWLVu8CjYOh0MlJSXKy8v7qZmICOXl5amoqKjF17z99tvKzc3VlClTlJaWpgEDBujBBx9s8TDZEc3NzWpoaPB4tFdOl6FlmypN1eb27kywAQCEJK9nbsrLy5WSktLmN66pqZHT6VRaWprH9rS0NFVWtvwBXFZWptdff11Op1NLly7Vvffeq0cffVR/+ctfjvs+BQUFstvt7kdmZmabew9VxeW1anKYm7fpk5pocTcAAFijVaeC79u3Tx9++KEqKirkcDg8nrv55pt90lhLXC6XUlNT9eyzzyoyMlLZ2dnasWOHHn74Yc2ePbvF18ycOVPTp093f9/Q0NBuA4438za5vdoeYAEACASvw83GjRt12WWXqampSfv27VOnTp1UU1OjhIQEpaammg43KSkpioyMVFVVlcf2qqoqpaent/iarl27Kjo62uOGnf369VNlZaUcDofHXcqPiI2NVWwsZ/1I0upvq03VxUdzZWIAQOjy+rDULbfcolGjRmnPnj2Kj4/X2rVr9f333ys7O1uPPPKI6Z8TExOj7OxsFRYWure5XC4VFhYqNze3xdeMHDlS3333nVyunw6tfPPNN+ratWuLwQY/8Wbe5rzTuzBvAwAIWV6Hm9LSUs2YMUMRERGKjIxUc3OzMjMzNXfuXN11111e/azp06frueee00svvaSvvvpKN9xwg/bt26dJkyZJkiZMmKCZM2e662+44QbV1tZq6tSp+uabb/Tvf/9bDz74oKZMmeLtMtod5m0AAO2F14eloqOj3aeFp6amqqKiQv369ZPdbtf27du9+lljxozRrl27NGvWLFVWVmrIkCFavny5e8i4oqLC/V6SlJmZqRUrVuiWW27RoEGD1K1bN02dOlV33HGHt8tod5i3AQC0F16Hm6FDh2rdunXq06ePzj//fM2aNUs1NTX6+9//rgEDBnjdwE033aSbbrqpxedWrVp1zLbc3FytXbvW6/dp72oam03VMW8DAAh1Xh+WevDBB9W1a1dJ0gMPPKCOHTvqhhtu0K5du/TMM8/4vEH4Rm2TuXDDvA0AINR5vecmJyfH/XVqaqqWL1/u04ZgjZJte0zVxUdHnrwIAIAg5vWem+PZsGGDV1cohv84XYZKt9eZqs1IjrO2GQAALOZVuFmxYoVuvfVW3XXXXSorK5Mkff311xo9erSGDRvmcYo2gsfast1yOE3dQkwje3exuBsAAKxl+rDUCy+8oOuuu06dOnXSnj179Pzzz2vevHn605/+pDFjxmjTpk3q16+flb2ildZsrTFVFxfFMDEAIPSZ3nOzYMECPfTQQ6qpqdFrr72mmpoaPfXUU/riiy+0cOFCgk0QW1dea6puUHc7w8QAgJBnOtxs3bpVv/nNbyRJV111laKiovTwww+re/fuljWHtvNm3iYnq6O1zQAA4Aemw83+/fuVkJAgSbLZbIqNjXWfEo7gxbwNAKC98epU8Oeff16JiYcvzX/o0CEtWrRIKSmeV7O18q7g8B7zNgCA9sZmGIap/63PysqSzXbieQybzeY+iypYNTQ0yG63q76+XklJSYFux3JXL1yjYhPXuBme1VGvTT7bDx0BAOA9bz6/Te+52bZtW1v7gp8xbwMAaI98dhE/BB/mbQAA7RHhJowxbwMAaI8IN2Fsx579puq4vg0AIJwQbsLYfschU3XZzNsAAMII4SZMOV2GPjF5WKpTQozF3QAA4D9eXedGOnwqVkuOXNgvJoYPymBQXF6rvc3mbmSakhhrcTcAAPiP1+EmOTn5hNe76d69u6655hrNnj1bERHsGAqUyoYDpmvT7fEWdgIAgH95HW4WLVqku+++W9dcc42GDx8uSSouLtZLL72ke+65R7t27dIjjzyi2NhY3XXXXT5vGOas/rbaVF1SXJSG9+xkcTcAAPiP1+HmpZde0qOPPqqrr77avW3UqFEaOHCgnnnmGRUWFqpHjx564IEHCDcB4nQZWrap0lRtbu/OnCkFAAgrXh83WrNmjYYOHXrM9qFDh6qoqEiSdM4556iioqLt3aFVistr1eQwN2/TJzXR4m4AAPAvr8NNZmamXnjhhWO2v/DCC8rMzJQk7d69Wx07cnpxoHgzb5PbK+XkRQAAhBCvD0s98sgj+s1vfqNly5Zp2LBhkqT169fr66+/1uuvvy5JWrduncaMGePbTmGa2Xmb+GiuTBzsnC5DH31Vrbn/95XKd+3T0TvkDEk2H38tSRE2yWVY9x70FBz9RUZIiXHRyj8zXbNHnan4mEgB4cL0XcGPVl5ermeeeUbffPONJOmMM87Qf//3fysrK8vX/flcuN8V3OkyNPC+5aYOS+WfmaZnxuf4oSu0xvJNO3XTqxt1yOX1f6KA1y7qn6rnJgwLdBvAcVlyV/Cj9ezZU3PmzGlVc7AW8zbhYfmmnZr8PxsC3QbakZWbq3Xdy+sIOAgLrQo3dXV1Ki4uVnV1tVwuzw/SCRMm+KQxtA7zNqHP6TJ075ufB7oNtEMrN1drv8PJISqEPK/DzTvvvKNx48Zp7969SkpK8rign81mI9wEWE1js6k65m2CV3F5rXbtM3dfMMDXHly6WfePHhjoNoA28fpsqRkzZugPf/iD9u7dq7q6Ou3Zs8f9qK2ttaJHeKG2yVy4Oe/0LlzfJkhVN5rf+wb42rbdTYFuAWgzr8PNjh07dPPNNyshIcGKftBGJdv2mKqLj2a3c7BK7RAX6BbQjmV15nc7Qp/X4SY/P1/r16+3ohe0kdNlqHR7nanajGQ+QIPV8J6d1OWUVo3DAW1212X9A90C0GZe/wa9/PLLddttt2nz5s0aOHCgoqOjPZ6/4oorfNYcvLO2bLccTnOnDY/s3cXibtBakRE23f+rQZwtBb+7qH8qw8QIC15f5+ZEd/q22WxyOp1tbspK4Xydm4dXfK0nP9h60rq4qAh9+edLmLkJclznBv7EdW4Q7Cy9zs3PT/1G8FhXbm6ge1B3O8EmBFwyoKu2/CWdKxTTE1coBrzEgf0w4c28TU4W9/0KFZERNl1wZpouODMt0K0AQMgwFW4ef/xxXX/99YqLi9Pjjz9+wtqbb77ZJ43BO8zbAABwmKlw89hjj2ncuHGKi4vTY489dtw6m81GuAmQNVtrTNXFRXHxPgBAeDMVbsrLy1v8GsGDeRsAAA7z+jo3CD7M2wAA8BOvB4qdTqcWLVqkwsLCFm+c+f777/usOZjDvA0AAD/xOtxMnTpVixYt0uWXX64BAwZ43DgTgcG8DQAAP/E63CxevFivvfaaLrvsMiv6QSvs2LPfVB3zNgCA9sDrmZuYmBiddtppVvSCVtrvOGSqLpt5GwBAO+B1uJkxY4YWLFggL+/aAIs4XYZWfbPLVG2nhBiLuwEAIPC8Piy1evVqffDBB1q2bJnOPPPMY26c+cYbb/isOZzc2rLdaj5kLmimJMZa3A0AAIHndbhJTk7Wr371Kyt6QSsUbd1tujbdHm9hJwAABAevws2hQ4d0wQUX6OKLL1Z6erpVPcEL31U3mqpLjI3U8J6dLO4GAIDA82rmJioqSpMnT1Zzc7NV/cALTpeh1d+Zm7fJPzOdM6UAAO2C1wPFw4cP18aNG63oBV4qLq/V3mbXyQslnXNaisXdAAAQHLyeubnxxhs1Y8YM/fDDD8rOztYpp5zi8fygQYN81hxOrLLhgOla5m0AAO2F1+Hmt7/9rSR53P3bZrPJMAzZbDY5nU7fdYcTWv1ttam6pLgo5m0AAO2G1+GGu4IHB6fL0LJNlaZqc3t3Zt4GANBueB1uTj31VCv6gJeKy2vV5DA3b9MnNdHibgAACB5eh5sjNm/erIqKCjkcDo/tV1xxRZubwsl5M2+T24thYgBA++F1uCkrK9OvfvUrffHFF+5ZG0nuu4Mzc+MfNY3mTsePj+ZO4ACA9sXrU8GnTp2qnj17qrq6WgkJCfryyy/10UcfKScnR6tWrbKgRbSktslcuDnv9C7M2wAA2hWv99wUFRXp/fffV0pKiiIiIhQREaFzzjlHBQUFuvnmm7kGjp+UbNtjqi4+OtLiTgAACC5e77lxOp3q0KGDJCklJUU//vijpMODxlu2bPFtd2iR02WodHudqdqM5DhrmwEAIMh4HW4GDBigzz77TJI0YsQIzZ07V5988on+/Oc/q1evXq1q4sknn1RWVpbi4uI0YsQIFRcXm3rd4sWLZbPZNHr06Fa9b6haW7ZbDqe5O4GP7N3F4m4AAAguXoebe+65Ry7X4VOQ//znP6u8vFznnnuuli5dqscff9zrBpYsWaLp06dr9uzZ2rBhgwYPHqz8/HxVV5/4AnXbtm3TrbfeqnPPPdfr9wx1a7bWmKqLi2KYGADQ/tiMI6c7tUFtba06duzoPmPKGyNGjNCwYcP0xBNPSJJcLpcyMzP1pz/9SXfeeWeLr3E6nTrvvPP0hz/8QR9//LHq6ur01ltvtVjb3NzscaPPhoYGZWZmqr6+XklJSV73GwyuXrhGxSZmboZnddRrk8/2Q0cAAFiroaFBdrvd1Oe313tujvjuu++0YsUK7d+/X506te7S/g6HQyUlJcrLy/upoYgI5eXlqaio6Liv+/Of/6zU1FRde+21J32PgoIC2e129yMzM7NVvQYLb+ZtcrI6WtsMAABByOtws3v3bl144YU6/fTTddlll2nnzp2SpGuvvVYzZszw6mfV1NTI6XQqLS3NY3taWpoqK1u+tcDq1av1wgsv6LnnnjP1HjNnzlR9fb37sX37dq96DDbM2wAAcGJeh5tbbrlF0dHRqqioUEJCgnv7mDFjtHz5cp8293ONjY0aP368nnvuOaWkmLvqbmxsrJKSkjweoYx5GwAATszr69z83//9n1asWKHu3bt7bO/Tp4++//57r35WSkqKIiMjVVVV5bG9qqpK6enpx9Rv3bpV27Zt06hRo9zbjgw3R0VFacuWLerdu7dXPYSadeW1puoGdbdz8T4AQLvk9Z6bffv2eeyxOaK2tlaxsbFe/ayYmBhlZ2ersLDQvc3lcqmwsFC5ubnH1Pft21dffPGFSktL3Y8rrrhCF1xwgUpLS0N+nuZkmLcBAODkvN5zc+655+rll1/W/fffL+nwPaVcLpfmzp2rCy64wOsGpk+frokTJyonJ0fDhw/X/PnztW/fPk2aNEmSNGHCBHXr1k0FBQWKi4vTgAEDPF6fnJwsScdsD0fM2wAAcHJeh5u5c+fqwgsv1Pr16+VwOHT77bfryy+/VG1trT755BOvGxgzZox27dqlWbNmqbKyUkOGDNHy5cvdQ8YVFRWKiGj1SV1hhXkbAABOrlXXuamvr9cTTzyhzz77THv37tVZZ52lKVOmqGvXrlb06FPenCcfbLi+DQCgvfLm89vrPTeSZLfbdffdd3ts++GHH3T99dfr2Wefbc2PxEkwbwMAgDk+O96ze/duvfDCC776cfgZ5m0AADCHYZYQwbwNAADmEG5CxI49+03VcX0bAEB7R7gJEfsdh0zVZTNvAwBo50wPFF911VUnfL6urq6tveA4nC5Dn5g8LNUpIcbibgAACG6mw43dbj/p8xMmTGhzQzhWcXmt9ja7TNWmJHp3lWgAAMKN6XDzt7/9zco+cAKVDQdM16bb4y3sBACA4MfMTQhY/W21qbqkuCgN79nJ4m4AAAhuhJsg53QZWrap0lRtbu/OnCkFAGj3CDdBrri8Vk0Oc/M2fVITLe4GAIDgR7gJct7M2+T2SrGwEwAAQgPhJsiZnbeJj+bKxAAASISboObNvM15p3dh3gYAABFughrzNgAAeI9wE8SYtwEAwHuEmyBW09hsqo55GwAAfkK4CWK1TebCDfM2AAD8hHATxEq27TFVFx8daXEnAACEDsJNkHK6DJVurzNVm5EcZ20zAACEEMJNkFpbtlsOp2GqdmTvLhZ3AwBA6CDcBKk1W2tM1cVFMUwMAMDRCDdBal15ram6Qd3tDBMDAHAUwk0Q8mbeJiero7XNAAAQYgg3QYh5GwAAWo9wE4SYtwEAoPUIN0GIeRsAAFqPcBNkmLcBAKBtCDdBhnkbAADahnATZJi3AQCgbQg3QWbHnv2m6pi3AQCgZYSbILPfcchUXTbzNgAAtIhwE0ScLkOfmDws1SkhxuJuAAAITYSbIFJcXqu9zS5TtSmJsRZ3AwBAaCLcBJHKhgOma9Pt8RZ2AgBA6CLcBJHV31abqkuKi9Lwnp0s7gYAgNBEuAkSTpehZZsqTdXm9u7MmVIAABwH4SZIFJfXqslhbt6mT2qixd0AABC6ogLdAA7zZt4mt1eKhZ0AaA/2O5ya9fbnWvFFpfY2u2RIOrI/2IqvJSnCJrmM1v+sqEibEmIild2jox4fe5YS4/gIQ8v4lxEkzM7bxEdzZWIAbXPdy+u0cvOxv3MMi792/ezOMt6+/qDTUP3+Q3p/yy4NuG+FBnVP0ts3nSvg5zgsFQS8mbc57/QuzNsAaLXjBZtQ9PkPDbriiY8D3QaCEOEmCDBvA8Af9jucYRNsjvj8hwbtPWDuyu5oPwg3QYB5GwD+8ODSzYFuwRK3LNkY6BYQZAg3QaCmsdlUHfM2ANpi2+6mQLdgiQqTNxxG+0G4CQK1TebCDfM2ANoiq3NCoFuwRI+OXLEdngg3QaBk2x5TdfHRkRZ3AiCc3XVZ/0C3YInHxgwNdAsIMoSbAHO6DJVurzNVm5EcZ20zAMJafEykLuqfGug2fGpQ9ySud4NjEG4CbG3ZbjmcxskLJY3s3cXibgCEu+cmDAubgMN1bnA8xN0AW7O1xlRdXBTDxAB847kJw7hCMcIa/zICbF15ram6Qd3tDBMD8Jn4mEg9/F9D9fB/BboTwPc4LBVA3szb5GR1tLYZAADCBOEmgJi3AQDA9wg3AcS8DQAAvke4CSDmbQAA8D3CTYAwbwMAgDUINwHCvA0AANYIinDz5JNPKisrS3FxcRoxYoSKi4uPW/vcc8/p3HPPVceOHdWxY0fl5eWdsD5YMW8DAIA1Ah5ulixZounTp2v27NnasGGDBg8erPz8fFVXV7dYv2rVKo0dO1YffPCBioqKlJmZqYsvvlg7duzwc+dts8PkXWyZtwEAwDsBDzfz5s3Tddddp0mTJql///5auHChEhIS9OKLL7ZY/8orr+jGG2/UkCFD1LdvXz3//PNyuVwqLCz0c+dts99xyFRdNvM2AAB4JaDhxuFwqKSkRHl5ee5tERERysvLU1FRkamf0dTUpIMHD6pTp04tPt/c3KyGhgaPR6A5XYY+MXlYqlNCjMXdAAAQXgIabmpqauR0OpWWluaxPS0tTZWVlaZ+xh133KGMjAyPgHS0goIC2e129yMzM7PNfbdVcXmt9ja7TNWmJMZa3A0AAOEl4Iel2mLOnDlavHix3nzzTcXFxbVYM3PmTNXX17sf27dv93OXx6psOGC6Nt0eb2EnAACEn4DeODMlJUWRkZGqqqry2F5VVaX09PQTvvaRRx7RnDlz9N5772nQoEHHrYuNjVVsbHDt/Vj9bcvD0j+XFBel4T1bPtwGAABaFtA9NzExMcrOzvYYBj4yHJybm3vc182dO1f333+/li9frpycHH+06jNOl6Flm8wdcsvt3ZkzpQAA8FJA99xI0vTp0zVx4kTl5ORo+PDhmj9/vvbt26dJkyZJkiZMmKBu3bqpoKBAkvTQQw9p1qxZevXVV5WVleWezUlMTFRiYmLA1mFWcXmtmhzm5m36pAb/egAACDYBDzdjxozRrl27NGvWLFVWVmrIkCFavny5e8i4oqJCERE/7WB6+umn5XA49F//9V8eP2f27Nm67777/Nl6q3gzb5PbK8XCTgAACE82wzDM3QMgTDQ0NMhut6u+vl5JSUl+f/8Zr23U/2748aR18dER2vT/LuGwFAAA8u7zO6TPlgo13szbnHd6F4INAACtQLjxI+ZtAACwHuHGj5i3AQDAeoQbP6ppbDZVFx/NncABAGgtwo0f1TaZCzfM2wAA0HqEGz8q2bbHVF18dKTFnQAAEL4IN37idBkq3V5nqjYjueX7ZAEAgJMj3PjJ2rLdcjjNXVJoZO8uFncDAED4Itz4yZqtNabq4qIYJgYAoC0IN36yrrzWVN2g7naGiQEAaAPCjR94M2+Tk9XR2mYAAAhzhBs/YN4GAAD/Idz4AfM2AAD4T1SgG2gPmLcBgLbb73Dqvne+UOGXldrT5JQhyZB05Lemt19LUoRNchmte70VX4d6T5ERUmxUhE7tfIpuvbivzj8jMBelJdxYjHkbAGi7615ep5Wbq1t8zmjD166jv2njz/LV16Hc0yGXdMjh0uadjfrDS+sUHWnTX8cO1SUDusqfOCxlMeZtAKBtThRsENwOOg1N/p8NWr5pp1/fl3BjMeZtAKD19jucBJswcN/bm+X8+e4fCxFuLLZjz35TdczbAMCxHly6OdAtwAcqGw6o2OT8qS8Qbiy233HIVF028zYAcIxtu5sC3QJ8pLrxgN/ei3BjIafL0Kpvdpmq7ZQQY3E3ABB6sjonBLoF+EhqB//dFJpwY6G1ZbvVfMjcMcaUxFiLuwGA0HPXZf0D3QJ8ID0pTsN7dvLb+xFuLFS0dbfp2nR7vIWdAEBoio+J1EX9UwPdBtroviv6+3WulHBjoe+qG03VJcZG+jXRAkAoeW7CMAJOiIqOtGnh78/y+3VuuIifRZwuQ6u/Mzdvk39mOmdKAcAJPDdhGFcoDoGeuEJxmCsur9XeZpep2nNOS7G4GwAIffExkXro10OkXwe6EwQ7DktZpLLB/ClvzNsAAOA7hBuLrP7W3BU1k+KimLcBAMCHCDcWcLoMLdtUaao2t3dn5m0AAPAhwo0Fistr1eQwN2/TJzXR4m4AAGhfCDcW8GbeJrcXw8QAAPgS4cYCNY3Npurio7kTOAAAvka4sUBtk7lwc97pgTn/HwCAcEa4sUDJtj2m6uKjIy3uBACA9odw42NOl6HS7XWmajOS/XeHVAAA2gvCjY+tLdsth9PcncBH9u5icTcAALQ/hBsfW7O1xlRdXBTDxAAAWIFw42PrymtN1Q3qbmeYGAAACxBufMibeZucrI7WNgMAQDtFuPEh5m0AAAg8wo0PMW8DAEDgEW58iHkbAAACj3DjI8zbAAAQHAg3PsK8DQAAwYFw4yPM2wAAEBwINz6yY89+U3XM2wAAYC3CjY/sdxwyVZfNvA0AAJYi3PiA02Vo1Te7TNV2SoixuBsAANo3wo0PrC3breZD5oaJUxJjLe4GAID2jXDjA0Vbd5uuTbfHW9gJAAAg3PjAd9WNpuoSYyM1vGcni7sBAKB9I9y0kdNlaPV35uZt8s9M50wpAAAsRrhpo+LyWu1tdpmqPee0FIu7AQAAhJs2qmw4YLqWeRsAAKxHuGmj1d9Wm6pLioti3gYAAD8g3LSB02Vo2aZKU7W5vTszbwMAgB8QbtqguLxWTQ5z8zZ9UhMt7gYAAEiEmzbxZt4mtxfDxAAA+ENQhJsnn3xSWVlZiouL04gRI1RcXHzC+n/+85/q27ev4uLiNHDgQC1dutRPnXqqaWw2VRcfzZ3AAQDwl4CHmyVLlmj69OmaPXu2NmzYoMGDBys/P1/V1S0P6q5Zs0Zjx47Vtddeq40bN2r06NEaPXq0Nm3a5OfOpT1NDlN1553ehXkbAAD8JODhZt68ebruuus0adIk9e/fXwsXLlRCQoJefPHFFusXLFigSy65RLfddpv69eun+++/X2eddZaeeOKJFuubm5vV0NDg8fAVs3mFeRsAAPwnoOHG4XCopKREeXl57m0RERHKy8tTUVFRi68pKiryqJek/Pz849YXFBTIbre7H5mZmT7r3+wcDfM2AAD4T0DDTU1NjZxOp9LS0jy2p6WlqbKy5VOsKysrvaqfOXOm6uvr3Y/t27f7pnlJ/9G7s5ITok9Yk5wQzbwNAAB+FPDDUlaLjY1VUlKSx8NXIiNsmnPVwBPWzLlqIPM2AAD4UUDDTUpKiiIjI1VVVeWxvaqqSunp6S2+Jj093at6q10yoKsW/v4spSfFemxPT4rVwt+fpUsGdA1IXwAAtFdRgXzzmJgYZWdnq7CwUKNHj5YkuVwuFRYW6qabbmrxNbm5uSosLNS0adPc21auXKnc3Fw/dNyySwZ01UX901VcXqvqxgNK7RCn4T07sccGAIAACGi4kaTp06dr4sSJysnJ0fDhwzV//nzt27dPkyZNkiRNmDBB3bp1U0FBgSRp6tSpOv/88/Xoo4/q8ssv1+LFi7V+/Xo9++yzgVyGIiNsymW2BgCAgAt4uBkzZox27dqlWbNmqbKyUkOGDNHy5cvdQ8MVFRWKiPjp6NnZZ5+tV199Vffcc4/uuusu9enTR2+99ZYGDBgQqCUAAIAgYjMMwwh0E/7U0NAgu92u+vp6nw4XAwAA63jz+R32Z0sBAID2hXADAADCCuEGAACEFcINAAAIK4QbAAAQVgg3AAAgrBBuAABAWAn4Rfz87chlfRoaGgLcCQAAMOvI57aZy/O1u3DT2NgoScrMzAxwJwAAwFuNjY2y2+0nrGl3Vyh2uVz68ccf1aFDB9lsvr2xZUNDgzIzM7V9+/Z2cfVj1hveWG94Y73hL9zWbBiGGhsblZGR4XFbppa0uz03ERER6t69u6XvkZSUFBb/kMxiveGN9YY31hv+wmnNJ9tjcwQDxQAAIKwQbgAAQFgh3PhQbGysZs+erdjY2EC34hesN7yx3vDGesNfe1zzEe1uoBgAAIQ39twAAICwQrgBAABhhXADAADCCuEGAACEFcKNjzz55JPKyspSXFycRowYoeLi4kC31CoFBQUaNmyYOnTooNTUVI0ePVpbtmzxqDlw4ICmTJmizp07KzExUb/+9a9VVVXlUVNRUaHLL79cCQkJSk1N1W233aZDhw75cylemzNnjmw2m6ZNm+beFo5r3bFjh37/+9+rc+fOio+P18CBA7V+/Xr384ZhaNasWeratavi4+OVl5enb7/91uNn1NbWaty4cUpKSlJycrKuvfZa7d27199LOSmn06l7771XPXv2VHx8vHr37q3777/f4940obzejz76SKNGjVJGRoZsNpveeustj+d9tbbPP/9c5557ruLi4pSZmam5c+davbQWnWi9Bw8e1B133KGBAwfqlFNOUUZGhiZMmKAff/zR42eE0nqlk/8dH23y5Mmy2WyaP3++x/ZQW7NPGGizxYsXGzExMcaLL75ofPnll8Z1111nJCcnG1VVVYFuzWv5+fnG3/72N2PTpk1GaWmpcdlllxk9evQw9u7d666ZPHmykZmZaRQWFhrr1683/uM//sM4++yz3c8fOnTIGDBggJGXl2ds3LjRWLp0qZGSkmLMnDkzEEsypbi42MjKyjIGDRpkTJ061b093NZaW1trnHrqqcY111xjfPrpp0ZZWZmxYsUK47vvvnPXzJkzx7Db7cZbb71lfPbZZ8YVV1xh9OzZ09i/f7+75pJLLjEGDx5srF271vj444+N0047zRg7dmwglnRCDzzwgNG5c2fj3XffNcrLy41//vOfRmJiorFgwQJ3TSivd+nSpcbdd99tvPHGG4Yk48033/R43hdrq6+vN9LS0oxx48YZmzZtMv7xj38Y8fHxxjPPPOOvZbqdaL11dXVGXl6esWTJEuPrr782ioqKjOHDhxvZ2dkePyOU1msYJ/87PuKNN94wBg8ebGRkZBiPPfaYx3OhtmZfINz4wPDhw40pU6a4v3c6nUZGRoZRUFAQwK58o7q62pBkfPjhh4ZhHP4FEh0dbfzzn/9013z11VeGJKOoqMgwjMP/MUZERBiVlZXumqefftpISkoympub/bsAExobG40+ffoYK1euNM4//3x3uAnHtd5xxx3GOeecc9znXS6XkZ6ebjz88MPubXV1dUZsbKzxj3/8wzAMw9i8ebMhyVi3bp27ZtmyZYbNZjN27NhhXfOtcPnllxt/+MMfPLZdddVVxrhx4wzDCK/1/vyDz1dre+qpp4yOHTt6/Hu+4447jDPOOMPiFZ3YiT7ojyguLjYkGd9//71hGKG9XsM4/pp/+OEHo1u3bsamTZuMU0891SPchPqaW4vDUm3kcDhUUlKivLw897aIiAjl5eWpqKgogJ35Rn19vSSpU6dOkqSSkhIdPHjQY719+/ZVjx493OstKirSwIEDlZaW5q7Jz89XQ0ODvvzySz92b86UKVN0+eWXe6xJCs+1vv3228rJydFvfvMbpaamaujQoXruuefcz5eXl6uystJjzXa7XSNGjPBYc3JysnJyctw1eXl5ioiI0Keffuq/xZhw9tlnq7CwUN98840k6bPPPtPq1at16aWXSgq/9R7NV2srKirSeeedp5iYGHdNfn6+tmzZoj179vhpNa1TX18vm82m5ORkSeG5XpfLpfHjx+u2227TmWeeeczz4bhmMwg3bVRTUyOn0+nx4SZJaWlpqqysDFBXvuFyuTRt2jSNHDlSAwYMkCRVVlYqJibG/cviiKPXW1lZ2eKfx5HngsnixYu1YcMGFRQUHPNcuK1VksrKyvT000+rT58+WrFihW644QbdfPPNeumllyT91POJ/j1XVlYqNTXV4/moqCh16tQp6NZ855136re//a369u2r6OhoDR06VNOmTdO4ceMkhd96j+artYXav/EjDhw4oDvuuENjx4513zQyHNf70EMPKSoqSjfffHOLz4fjms1od3cFh3lTpkzRpk2btHr16kC3Yont27dr6tSpWrlypeLi4gLdjl+4XC7l5OTowQcflCQNHTpUmzZt0sKFCzVx4sQAd+d7r732ml555RW9+uqrOvPMM1VaWqpp06YpIyMjLNeLww4ePKirr75ahmHo6aefDnQ7likpKdGCBQu0YcMG2Wy2QLcTVNhz00YpKSmKjIw85gyaqqoqpaenB6irtrvpppv07rvv6oMPPlD37t3d29PT0+VwOFRXV+dRf/R609PTW/zzOPJcsCgpKVF1dbXOOussRUVFKSoqSh9++KEef/xxRUVFKS0tLWzWekTXrl3Vv39/j239+vVTRUWFpJ96PtG/5/T0dFVXV3s8f+jQIdXW1gbdmm+77Tb33puBAwdq/PjxuuWWW9x76sJtvUfz1dpC7d/4kWDz/fffa+XKle69NlL4rffjjz9WdXW1evTo4f4d9v3332vGjBnKysqSFH5rNotw00YxMTHKzs5WYWGhe5vL5VJhYaFyc3MD2FnrGIahm266SW+++abef/999ezZ0+P57OxsRUdHe6x3y5YtqqiocK83NzdXX3zxhcd/UEd+yfz8gzWQLrzwQn3xxRcqLS11P3JycjRu3Dj31+Gy1iNGjhx5zKn933zzjU499VRJUs+ePZWenu6x5oaGBn366acea66rq1NJSYm75v3335fL5dKIESP8sArzmpqaFBHh+WsuMjJSLpdLUvit92i+Wltubq4++ugjHTx40F2zcuVKnXHGGerYsaOfVmPOkWDz7bff6r333lPnzp09ng+39Y4fP16ff/65x++wjIwM3XbbbVqxYoWk8FuzaYGeaA4HixcvNmJjY41FixYZmzdvNq6//nojOTnZ4wyaUHHDDTcYdrvdWLVqlbFz5073o6mpyV0zefJko0ePHsb7779vrF+/3sjNzTVyc3Pdzx85Pfriiy82SktLjeXLlxtdunQJ2tOjj3b02VKGEX5rLS4uNqKioowHHnjA+Pbbb41XXnnFSEhIMP7nf/7HXTNnzhwjOTnZ+Ne//mV8/vnnxpVXXtni6cNDhw41Pv30U2P16tVGnz59guLU6J+bOHGi0a1bN/ep4G+88YaRkpJi3H777e6aUF5vY2OjsXHjRmPjxo2GJGPevHnGxo0b3WcH+WJtdXV1RlpamjF+/Hhj06ZNxuLFi42EhISAnCZ8ovU6HA7jiiuuMLp3726UlpZ6/P46+iygUFqvYZz87/jnfn62lGGE3pp9gXDjI3/961+NHj16GDExMcbw4cONtWvXBrqlVpHU4uNvf/ubu2b//v3GjTfeaHTs2NFISEgwfvWrXxk7d+70+Dnbtm0zLr30UiM+Pt5ISUkxZsyYYRw8eNDPq/Hez8NNOK71nXfeMQYMGGDExsYaffv2NZ599lmP510ul3HvvfcaaWlpRmxsrHHhhRcaW7Zs8ajZvXu3MXbsWCMxMdFISkoyJk2aZDQ2NvpzGaY0NDQYU6dONXr06GHExcUZvXr1Mu6++26PD7tQXu8HH3zQ4n+vEydONAzDd2v77LPPjHPOOceIjY01unXrZsyZM8dfS/RwovWWl5cf9/fXBx984P4ZobRewzj53/HPtRRuQm3NvmAzjKMu1QkAABDimLkBAABhhXADAADCCuEGAACEFcINAAAIK4QbAAAQVgg3AAAgrBBuAABAWCHcAACAsEK4AdDuZGVlaf78+YFuA4BFCDcALHXNNddo9OjRkqRf/OIXmjZtmt/ee9GiRUpOTj5m+7p163T99df7rQ8A/hUV6AYAwFsOh0MxMTGtfn2XLl182A2AYMOeGwB+cc011+jDDz/UggULZLPZZLPZtG3bNknSpk2bdOmllyoxMVFpaWkaP368ampq3K/9xS9+oZtuuknTpk1TSkqK8vPzJUnz5s3TwIEDdcoppygzM1M33nij9u7dK0latWqVJk2apPr6evf73XfffZKOPSxVUVGhK6+8UomJiUpKStLVV1+tqqoq9/P33XefhgwZor///e/KysqS3W7Xb3/7WzU2Nlr7hwagVQg3APxiwYIFys3N1XXXXaedO3dq586dyszMVF1dnf7zP/9TQ4cO1fr167V8+XJVVVXp6quv9nj9Sy+9pJiYGH3yySdauHChJCkiIkKPP/64vvzyS7300kt6//33dfvtt0uSzj77bM2fP19JSUnu97v11luP6cvlcunKK69UbW2tPvzwQ61cuVJlZWUaM2aMR93WrVv11ltv6d1339W7776rDz/8UHPmzLHoTwtAW3BYCoBf2O12xcTEKCEhQenp6e7tTzzxhIYOHaoHH3zQve3FF19UZmamvvnmG51++umSpD59+mju3LkeP/Po+Z2srCz95S9/0eTJk/XUU08pJiZGdrtdNpvN4/1+rrCwUF988YXKy8uVmZkpSXr55Zd15plnat26dRo2bJikwyFo0aJF6tChgyRp/PjxKiws1AMPPNC2PxgAPseeGwAB9dlnn+mDDz5QYmKi+9G3b19Jh/eWHJGdnX3Ma9977z1deOGF6tatmzp06KDx48dr9+7dampqMv3+X331lTIzM93BRpL69++v5ORkffXVV+5tWVlZ7mAjSV27dlV1dbVXawXgH+y5ARBQe/fu1ahRo/TQQw8d81zXrl3dX59yyikez23btk2//OUvdcMNN+iBBx5Qp06dtHr1al177bVyOBxKSEjwaZ/R0dEe39tsNrlcLp++BwDfINwA8JuYmBg5nU6PbWeddZb+93//V1lZWYqKMv8rqaSkRC6XS48++qgiIg7vhH7ttddO+n4/169fP23fvl3bt293773ZvHmz6urq1L9/f9P9AAgeHJYC4DdZWVn69NNPtW3bNtXU1MjlcmnKlCmqra3V2LFjtW7dOm3dulUrVqzQpEmTThhMTjvtNB08eFB//etfVVZWpr///e/uQeOj32/v3r0qLCxUTU1Ni4er8vLyNHDgQI0bN04bNmxQcXGxJkyYoPPPP185OTk+/zMAYD3CDQC/ufXWWxUZGan+/furS5cuqqioUEZGhj755BM5nU5dfPHFGjhwoKZNm6bk5GT3HpmWDB48WPPmzdNDDz2kAQMG6JVXXlFBQYFHzdlnn63JkydrzJgx6tKlyzEDydLhw0v/+te/1LFjR5133nnKy8tTr169tGTJEp+vH4B/2AzDMALdBAAAgK+w5wYAAIQVwg0AAAgrhBsAABBWCDcAACCsEG4AAEBYIdwAAICwQrgBAABhhXADAADCCuEGAACEFcINAAAIK4QbAAAQVv4/f91caRcciZkAAAAASUVORK5CYII=\n", |
| "text/plain": [ |
| "<Figure size 640x480 with 1 Axes>" |
| ] |
| }, |
| "metadata": {}, |
| "output_type": "display_data" |
| } |
| ], |
| "source": [ |
| "schedule = mx.lr_scheduler.MultiFactorScheduler(step=[250, 750, 900], factor=0.5)\n", |
| "schedule.base_lr = 1\n", |
| "schedule = LinearWarmUp(schedule, start_lr=0, length=250)\n", |
| "plot_schedule(schedule)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "91faede2", |
| "metadata": {}, |
| "source": [ |
| " <!--notebook-skip-line-->\n", |
| "\n", |
| "\n", |
| "### Cool-Down\n", |
| "\n", |
| "Similarly, we could add a linear cool-down period to our schedule and this is used in the \"1-Cycle\" schedule proposed by [Leslie N. Smith, Nicholay Topin (2017)](https://arxiv.org/abs/1708.07120) to train neural networks very quickly in certain circumstances (coined \"super-convergence\"). We reduce the learning rate from its value at `start_idx` of `schedule` to `finish_lr` over a period of `length`, and then maintain `finish_lr` thereafter." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 9, |
| "id": "fe2fddff", |
| "metadata": {}, |
| "outputs": [], |
| "source": [ |
| "class LinearCoolDown():\n", |
| " def __init__(self, schedule, finish_lr, start_idx, length):\n", |
| " \"\"\"\n", |
| " schedule: a pre-initialized schedule (e.g. TriangularSchedule(min_lr=0.5, max_lr=2, cycle_length=500))\n", |
| " finish_lr: learning rate used at end of the cool-down (float)\n", |
| " start_idx: iteration to start the cool-down (int)\n", |
| " length: number of iterations used for the cool-down (int)\n", |
| " \"\"\"\n", |
| " self.schedule = schedule\n", |
| " # calling mx.lr_scheduler.LRScheduler effects state, so calling a copy\n", |
| " self.start_lr = copy.copy(self.schedule)(start_idx)\n", |
| " self.finish_lr = finish_lr\n", |
| " self.start_idx = start_idx\n", |
| " self.finish_idx = start_idx + length\n", |
| " self.length = length\n", |
| "\n", |
| " def __call__(self, iteration):\n", |
| " if iteration <= self.start_idx:\n", |
| " return self.schedule(iteration)\n", |
| " elif iteration <= self.finish_idx:\n", |
| " return (iteration - self.start_idx) * (self.finish_lr - self.start_lr) / (self.length) + self.start_lr\n", |
| " else:\n", |
| " return self.finish_lr" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "5da168bb", |
| "metadata": {}, |
| "source": [ |
| "As an example, we apply learning rate cool-down to a `MultiFactorScheduler`. Starting the cool-down at iteration 1000, we reduce the learning rate linearly from 0.125 to 0.001 over 500 iterations, and hold the learning rate at 0.001 after this." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 10, |
| "id": "85627870", |
| "metadata": {}, |
| "outputs": [ |
| { |
| "data": { |
| "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA1RElEQVR4nO3de3hU1b3/8c9MrgRIAoQkXAaDglwEAQmkQaz1GI1CUWpbKaUEqcUDRblqERXwqBDES8ECokgFrQrqz1qlCAfDRbEpgQDKHZVLEElCwFwIkEBm/f7gZMpICBkyyWR23q/nmedJ1l579neFmPm49tp724wxRgAAABZh93UBAAAA3kS4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlkK4AQAAlhLo6wJqm9Pp1Pfff6/GjRvLZrP5uhwAAFAFxhgVFRWpZcuWstsrn5upd+Hm+++/l8Ph8HUZAADgChw+fFitW7eutE+9CzeNGzeWdP6HEx4e7uNqAABAVRQWFsrhcLg+xytT78JN+amo8PBwwg0AAH6mKktKWFAMAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAshXADAAAspd7dobimnC4t05Mfb9eaXTnKP3VOZeZ8u5FUfi/Fmvhakuw2yVlLx/NGTQF2KSTQrquaNdTDt3fUzR2aK8DOQ0wBAN7h05mbzz77TAMGDFDLli1ls9n04YcfXnafdevW6YYbblBISIjatWunxYsX13idlzPijU3qNHWllm06omPF53TWSE6dfxnV7NdOSedq8XjeqOmcUyoudWrX0SL9fskmdZzyiVbuOFqtfwMAAMr5NNwUFxerW7dumjdvXpX6HzhwQP3799ctt9yibdu2ady4cfrDH/6gVatW1XCllzbijU1avSvXZ8e3grNlRiP/toWAAwDwCp+elrrzzjt15513Vrn/ggUL1LZtW73wwguSpE6dOmnDhg3685//rOTk5Joq85JOl5YRbLzoyY926bbOsZyiAgBUi18tKE5PT1dSUpJbW3JystLT0y+5T0lJiQoLC91e3jJjxS6vvRek7MIzyjhwwtdlAAD8nF+Fm+zsbMXExLi1xcTEqLCwUKdPn65wn9TUVEVERLheDofDa/UcPH7Ka++F83KLzvi6BACAn/OrcHMlJk+erIKCAtfr8OHDXnvvuGZhXnsvnBfdONTXJQAA/JxfhZvY2Fjl5OS4teXk5Cg8PFwNGjSocJ+QkBCFh4e7vbzlsX6dvfZekGLDQ9W7bVNflwEA8HN+FW4SExOVlpbm1rZ69WolJib6pJ4GwQG6rXO0T45tRU/e1ZnFxACAavNpuDl58qS2bdumbdu2STp/qfe2bduUlZUl6fwppZSUFFf/kSNHav/+/frTn/6kPXv2aP78+Xr33Xc1fvx4X5QvSVqY0ouAU01BATYt+N0NuqNLC1+XAgCwAJ9eCr5582bdcsstru8nTJggSRo2bJgWL16so0ePuoKOJLVt21b//Oc/NX78eM2ZM0etW7fWa6+95pPLwC+0MKUXdyj2oCbuUAwAqEk2Y4zxdRG1qbCwUBERESooKPDq+hsAAFBzPPn89qs1NwAAAJdDuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJZCuAEAAJbi83Azb948xcXFKTQ0VAkJCcrIyKi0/+zZs9WhQwc1aNBADodD48eP15kzZ2qpWgAAUNf5NNwsW7ZMEyZM0LRp07RlyxZ169ZNycnJys3NrbD/22+/rUcffVTTpk3T7t27tWjRIi1btkyPPfZYLVcOAADqKpsxxvjq4AkJCerVq5fmzp0rSXI6nXI4HHrooYf06KOPXtT/wQcf1O7du5WWluZqmzhxojZu3KgNGzZUeIySkhKVlJS4vi8sLJTD4VBBQYHCw8O9PCIAAFATCgsLFRERUaXPb5/N3JSWliozM1NJSUn/KcZuV1JSktLT0yvcp0+fPsrMzHSdutq/f79WrFihfv36XfI4qampioiIcL0cDod3BwIAAOqUQF8dOC8vT2VlZYqJiXFrj4mJ0Z49eyrc57e//a3y8vLUt29fGWN07tw5jRw5stLTUpMnT9aECRNc35fP3AAAAGvy+YJiT6xbt04zZszQ/PnztWXLFn3wwQf65z//qaeffvqS+4SEhCg8PNztBQAArMtnMzdRUVEKCAhQTk6OW3tOTo5iY2Mr3GfKlCkaOnSo/vCHP0iSunbtquLiYj3wwAN6/PHHZbf7VVYDAAA1wGdpIDg4WD179nRbHOx0OpWWlqbExMQK9zl16tRFASYgIECS5MN10QAAoA7x2cyNJE2YMEHDhg1TfHy8evfurdmzZ6u4uFjDhw+XJKWkpKhVq1ZKTU2VJA0YMEAvvviievTooYSEBH3zzTeaMmWKBgwY4Ao5AACgfvNpuBk0aJCOHTumqVOnKjs7W927d9fKlStdi4yzsrLcZmqeeOIJ2Ww2PfHEEzpy5IiaN2+uAQMGaPr06b4aAgAAqGN8ep8bX/DkOnkAAFA3+MV9bgAAAGoC4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFgK4QYAAFhKtcLNmTNnvFUHAACAV3gcbpxOp55++mm1atVKjRo10v79+yVJU6ZM0aJFi7xeIAAAgCc8DjfPPPOMFi9erFmzZik4ONjV3qVLF7322mteLQ4AAMBTHoebN954Q6+++qqGDBmigIAAV3u3bt20Z88erxYHAADgKY/DzZEjR9SuXbuL2p1Op86ePeuVogAAAK6Ux+Gmc+fO+vzzzy9qf//999WjRw+vFAUAAHClAj3dYerUqRo2bJiOHDkip9OpDz74QHv37tUbb7yh5cuX10SNAAAAVebxzM3dd9+tjz/+WJ9++qkaNmyoqVOnavfu3fr4449122231USNAAAAVWYzxhhfF1GbCgsLFRERoYKCAoWHh/u6HAAAUAWefH57PHNz9dVX6/jx4xe15+fn6+qrr/b07QAAALzK43Bz8OBBlZWVXdReUlKiI0eOeFzAvHnzFBcXp9DQUCUkJCgjI6PS/vn5+Ro9erRatGihkJAQXXvttVqxYoXHxwUAANZU5QXFH330kevrVatWKSIiwvV9WVmZ0tLSFBcX59HBly1bpgkTJmjBggVKSEjQ7NmzlZycrL179yo6Ovqi/qWlpbrtttsUHR2t999/X61atdKhQ4cUGRnp0XEBAIB1VXnNjd1+fpLHZrPpx7sEBQUpLi5OL7zwgn7+859X+eAJCQnq1auX5s6dK+n8vXIcDoceeughPfrooxf1X7BggZ577jnt2bNHQUFBVTpGSUmJSkpKXN8XFhbK4XCw5gYAAD9SI2tunE6nnE6n2rRpo9zcXNf3TqdTJSUl2rt3r0fBprS0VJmZmUpKSvpPMXa7kpKSlJ6eXuE+H330kRITEzV69GjFxMSoS5cumjFjRoWnycqlpqYqIiLC9XI4HFWuEQAA+B+P19wcOHBAUVFR1T5wXl6eysrKFBMT49YeExOj7OzsCvfZv3+/3n//fZWVlWnFihWaMmWKXnjhBT3zzDOXPM7kyZNVUFDgeh0+fLjatQMAgLrL45v4SVJxcbHWr1+vrKwslZaWum0bM2aMVwqriNPpVHR0tF599VUFBASoZ8+eOnLkiJ577jlNmzatwn1CQkIUEhJSYzUBAIC6xeNws3XrVvXr10+nTp1ScXGxmjZtqry8PIWFhSk6OrrK4SYqKkoBAQHKyclxa8/JyVFsbGyF+7Ro0UJBQUFuD+zs1KmTsrOzVVpa6vaUcgAAUD95fFpq/PjxGjBggH744Qc1aNBA//73v3Xo0CH17NlTzz//fJXfJzg4WD179lRaWpqrzel0Ki0tTYmJiRXuc+ONN+qbb76R0+l0te3bt08tWrQg2AAAAElXEG62bdumiRMnym63KyAgQCUlJXI4HJo1a5Yee+wxj95rwoQJWrhwoZYsWaLdu3dr1KhRKi4u1vDhwyVJKSkpmjx5sqv/qFGjdOLECY0dO1b79u3TP//5T82YMUOjR4/2dBgAAMCiPD4tFRQU5LosPDo6WllZWerUqZMiIiI8Xqw7aNAgHTt2TFOnTlV2dra6d++ulStXuhYZZ2VluY4lSQ6HQ6tWrdL48eN1/fXXq1WrVho7dqwmTZrk6TAAAIBFefxsqdtvv1333Xeffvvb32rEiBH66quvNGbMGL355pv64YcftHHjxpqq1St4thQAAP6nRp8tNWPGDLVo0UKSNH36dDVp0kSjRo3SsWPH9Morr1xZxQAAAF7CU8EBAECdV6MzN5eyZcsWj+5QDAAAUBM8CjerVq3Sww8/rMcee0z79++XJO3Zs0cDBw5Ur1693C7RBgAA8IUqXy21aNEijRgxQk2bNtUPP/yg1157TS+++KIeeughDRo0SDt27FCnTp1qslYAAIDLqvLMzZw5c/Tss88qLy9P7777rvLy8jR//nxt375dCxYsINgAAIA6ocoLihs2bKidO3cqLi5OxhiFhIRo7dq1uvHGG2u6Rq9iQTEAAP6nRhYUnz59WmFhYZIkm82mkJAQ1yXhAAAAdYVHdyh+7bXX1KhRI0nSuXPntHjxYkVFRbn1qcmnggMAAFxOlU9LxcXFyWazVf5mNpvrKqq6itNSAAD4H08+v6s8c3Pw4MHq1gUAAFDjvHYTPwAAgLqAcAMAACyFcAMAACyFcAMAACyFcAMAACzFo/vcSOcvxapI+Y39goODq10UAADAlfI43ERGRlZ6v5vWrVvrvvvu07Rp02S3MzEEAABql8fhZvHixXr88cd13333qXfv3pKkjIwMLVmyRE888YSOHTum559/XiEhIXrssce8XjAAAEBlPA43S5Ys0QsvvKB7773X1TZgwAB17dpVr7zyitLS0tSmTRtNnz6dcAMAAGqdx+eN/vWvf6lHjx4Xtffo0UPp6emSpL59+yorK6v61QEAAHjI45kbh8OhRYsWaebMmW7tixYtksPhkCQdP35cTZo08U6FsJzSc04t2vCt3tuUpcM/nFGZUzKSyldy1cTXkmS3SU5Tc8egJt/WJEnBQXZFNgjSLR2iNXXAdWoQHCAA9Y/H4eb555/Xr3/9a33yySfq1auXJGnz5s3as2eP3n//fUnSpk2bNGjQIO9WCktIXbFLr3x2oMJtpoa/dl74TS0cj5pqv6YzZ53KPluidzYd1jubDuu2ztFamNJLAOqXKj8V/EIHDhzQK6+8on379kmSOnTooP/+7/9WXFyct+vzOp4K7juVBRugphBwAGuokaeCX6ht27YXnZYCKlN6zkmwgU+s3pWr06VlnKIC6pErCjf5+fnKyMhQbm6unE6n27aUlBSvFAZreTP9oK9LQD02Y8UuPT2wq6/LAFBLPA43H3/8sYYMGaKTJ08qPDzc7YZ+NpuNcIMKHTpxytcloB47eJzfP6A+8fhS8IkTJ+r3v/+9Tp48qfz8fP3www+u14kTJ2qiRljAVU3DfF0C6rG4Zvz+AfWJx+HmyJEjGjNmjMLC+GOBqhuaGOfrElCPPdavs69LAFCLPA43ycnJ2rx5c03UAgsLDrTrv3/a1tdloB66rXM0i4mBesbjNTf9+/fXI488ol27dqlr164KCgpy237XXXd5rThYy+T/+79nrppCbeEycKB+8vg+N5U96dtms6msrKzaRdUk7nPje9yhmJq4QzEAT3ny+X1FN/HzZ4QbAAD8jyef3x6vuQEAAKjLqrTm5qWXXtIDDzyg0NBQvfTSS5X2HTNmjFcKAwAAuBJVOi3Vtm1bbd68Wc2aNVPbtpe+4sVms2n//v1eLdDbOC0FAID/8fqzpQ4cOFDh1wAAAHUNa24AAICleHyfm7KyMi1evFhpaWkVPjhzzZo1XisOAADAUx6Hm7Fjx2rx4sXq37+/unTp4vbgTAAAAF/zONwsXbpU7777rvr161cT9QAAAFSLx2tugoOD1a5du5qoBQAAoNo8DjcTJ07UnDlzVM9ubAwAAPyEx6elNmzYoLVr1+qTTz7Rddddd9GDMz/44AOvFQcAAOApj8NNZGSkfvGLX9RELQAAANXmUbg5d+6cbrnlFt1+++2KjY2tqZoAAACumEdrbgIDAzVy5EiVlJTUVD0AAADV4vGC4t69e2vr1q01UQsAAEC1ebzm5o9//KMmTpyo7777Tj179lTDhg3dtl9//fVeKw4AAMBTVXoq+IXs9osne2w2m4wxstlsKisr81pxNYGnggMA4H+8/lTwC/FUcAAAUJd5HG6uuuqqmqgDAADAKzwON+V27dqlrKwslZaWurXfdddd1S4KAADgSnkcbvbv369f/OIX2r59u2utjSTX08Hr+pobAABgbR5fCj527Fi1bdtWubm5CgsL086dO/XZZ58pPj5e69atq4ESAQAAqs7jmZv09HStWbNGUVFRstvtstvt6tu3r1JTUzVmzBjugQMAAHzK45mbsrIyNW7cWJIUFRWl77//XtL5hcZ79+69oiLmzZunuLg4hYaGKiEhQRkZGVXab+nSpbLZbBo4cOAVHRcAAFiPx+GmS5cu+vLLLyVJCQkJmjVrlr744gs99dRTuvrqqz0uYNmyZZowYYKmTZumLVu2qFu3bkpOTlZubm6l+x08eFAPP/ywbrrpJo+PCQAArMvjcPPEE0/I6XRKkp566ikdOHBAN910k1asWKGXXnrJ4wJefPFFjRgxQsOHD1fnzp21YMEChYWF6a9//esl9ykrK9OQIUP0P//zP1cUqAAAgHV5vOYmOTnZ9XW7du20Z88enThxQk2aNHFdMVVVpaWlyszM1OTJk11tdrtdSUlJSk9Pv+R+Tz31lKKjo3X//ffr888/r/QYJSUlbg/6LCws9KhGAADgXzyeuSn3zTffaNWqVTp9+rSaNm16Re+Rl5ensrIyxcTEuLXHxMQoOzu7wn02bNigRYsWaeHChVU6RmpqqiIiIlwvh8NxRbUCAAD/4HG4OX78uG699VZde+216tevn44ePSpJuv/++zVx4kSvF3ihoqIiDR06VAsXLlRUVFSV9pk8ebIKCgpcr8OHD9dojQAAwLc8Djfjx49XUFCQsrKyFBYW5mofNGiQVq5c6dF7RUVFKSAgQDk5OW7tOTk5io2Nvaj/t99+q4MHD2rAgAEKDAxUYGCg3njjDX300UcKDAzUt99+e9E+ISEhCg8Pd3sBAADr8njNzf/+7/9q1apVat26tVt7+/btdejQIY/eKzg4WD179lRaWprrcm6n06m0tDQ9+OCDF/Xv2LGjtm/f7tb2xBNPqKioSHPmzOGUEwAA8DzcFBcXu83YlDtx4oRCQkI8LmDChAkaNmyY4uPj1bt3b82ePVvFxcUaPny4JCklJUWtWrVSamqqQkND1aVLF7f9IyMjJemidgAAUD95HG5uuukmvfHGG3r66aclnX+mlNPp1KxZs3TLLbd4XMCgQYN07NgxTZ06VdnZ2erevbtWrlzpWmSclZUlu/2K1z0DAIB6xmbKn3xZRTt27NCtt96qG264QWvWrNFdd92lnTt36sSJE/riiy90zTXX1FStXlFYWKiIiAgVFBSw/gYAAD/hyef3Fd2heN++ferbt6/uvvtuFRcX65577tHWrVvrfLABAADW5/FpKUmKiIjQ448/7tb23Xff6YEHHtCrr77qlcIAAACuhNcWsxw/flyLFi3y1tsBAABcEVbqAgAASyHcAAAASyHcAAAAS6nyguJ77rmn0u35+fnVrQUAAKDaqhxuIiIiLrs9JSWl2gUBAABUR5XDzeuvv16TdQAAAHgFa24AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClEG4AAIClBPq6AADwhjKn0Ya9x/Ty+q+18/sCFZcaSZKRZPu/Pt76WpLsNslpqv9eAQE2NWsYrFs6RGvqgOvUIDjAw5ED+DGbMcb4uojaVFhYqIiICBUUFCg8PNzX5QDwgpU7jmrM0m0qPef0dSnVdlvnaC1M6eXrMoA6x5PPb05LAfBrK3cc1ci/bbFEsJGk1btyNeKNTb4uA/BrhBsAfqvMaTT1w+2+LsPrVu/K1enSMl+XAfgtwg0Av5Vx4IRyT571dRk1YsaKXb4uAfBbhBsAfiu36IyvS6gxB4+f8nUJgN8i3ADwW9GNQ31dQo2Jaxbm6xIAv0W4AeC3erdtquhGQb4uo0Y81q+zr0sA/BbhBoDfCrDb9NTArr4uw+tu6xzN/W6AaiDcAPBrd3RpoQW/u0HBgdb4c8Z9boDq4w7FAPzeHV1aaPdTsdyhGIAkwg0Aiwiw23Rzp2jd3Cna16UA8DFrzOMCAAD8H8INAACwFMINAACwFMINAACwFMINAACwFMINAACwlDoRbubNm6e4uDiFhoYqISFBGRkZl+y7cOFC3XTTTWrSpImaNGmipKSkSvsDAID6xefhZtmyZZowYYKmTZumLVu2qFu3bkpOTlZubm6F/detW6fBgwdr7dq1Sk9Pl8Ph0O23364jR47UcuUAAKAushljjC8LSEhIUK9evTR37lxJktPplMPh0EMPPaRHH330svuXlZWpSZMmmjt3rlJSUi7bv7CwUBERESooKFB4eHi16wcAADXPk89vn87clJaWKjMzU0lJSa42u92upKQkpaenV+k9Tp06pbNnz6pp06YVbi8pKVFhYaHbCwAAWJdPw01eXp7KysoUExPj1h4TE6Ps7OwqvcekSZPUsmVLt4B0odTUVEVERLheDoej2nUDAIC6y+drbqpj5syZWrp0qf7+978rNDS0wj6TJ09WQUGB63X48OFarhIAANQmnz44MyoqSgEBAcrJyXFrz8nJUWxsbKX7Pv/885o5c6Y+/fRTXX/99ZfsFxISopCQEK/UCwAA6j6fztwEBwerZ8+eSktLc7U5nU6lpaUpMTHxkvvNmjVLTz/9tFauXKn4+PjaKBUAAPgJn87cSNKECRM0bNgwxcfHq3fv3po9e7aKi4s1fPhwSVJKSopatWql1NRUSdKzzz6rqVOn6u2331ZcXJxrbU6jRo3UqFEjn40DAADUDT4PN4MGDdKxY8c0depUZWdnq3v37lq5cqVrkXFWVpbs9v9MML388ssqLS3Vr371K7f3mTZtmp588snaLB0AANRBPr/PTW3jPjcAAPgfv7nPDQAAgLcRbgAAgKUQbgAAgKUQbgAAgKUQbgAAgKUQbgAAgKUQbgAAgKUQbgAAgKUQbgAAgKUQbgAAgKUQbgAAgKUQbgAAgKUQbgAAgKUE+roAAKjPTp45pzFvb1bGgRM6dda42o0k2xV+LUl2m+Q0lfcLCLCpWcNg3dIhWlMHXKcGwQFeGxfgSzZjjLl8N+vw5JHpAFCT7pr7ub76rtDXZbjYbdI1zRvqlze01u/7Xq3gQCb3UXd48vlNuAEAH6hrwaYiwQE29W0XpZcG36BGoUz0w7c8+fzmtxUAatnJM+fqfLCRpNIyozV7j6nLk6skSa0iQzUk4Sr94SZmdVC3MXMDALVsxJJNWr0719dlVEuQXYqPa6qRN1+jvu2bK8Buu/xOQDUwcwMAdVjWD6d9XUK1nXVK6ftPKH3/CUlSRINA9evSgoXJqBMINwBQy9o0aaC92UW+LsOrCk6f0zubDuudTYdZmAyf47QUANSyk2fOudax1AcsTIY3cFoKAOqwRqGBur51uF8sKvaGHy9Mjg0P4d46qFHM3ACAj/jD5eA1zS6pY4vGevj2jrq5AwuTcWnc56YShBsAdYmv7lDs9OoovKdhcIAGdGupaczq4EcIN5Ug3ADAeeXB6vNvjutsHUw7NkntolmYjPMIN5Ug3ADAxUrPObVow7d6f/NhHcg7XSdndkICbLrhqibcW6eeItxUgnADAJd3urRMT368XSu+PKqi0roYdaTIBoG6k3vr1BuEm0oQbgDAM2VOow17j2n+un3KzCrQuTr4qcHCZOsj3FSCcAMA1VN6zqmFn3+jv6Uf1NHCs74up0IsTLYewk0lCDcA4F0sTEZtINxUgnADADWHhcmoKYSbShBuAKD2lC9M/uSroyosqYtRh4XJ/oJwUwnCDQD4RvnC5JfXf63MrPw6eQqLh37WXYSbShBuAKBuKD+F9d6mLB04fkZ18cOIh37WHYSbShBuAKBuOl1apqkffaXl247qdF283lxSq8hQDUm4Sn+4iVmd2ka4qQThBgDqvjKn0We7czXrf3drb05xnVyYHGSX4uOasjC5lhBuKkG4AQD/4w8LkyMaBKofC5NrDOGmEoQbAPBvLEyunwg3lSDcAIC1sDC5fiDcVIJwAwDWxsJkayLcVIJwAwD1BwuTrYNwUwnCDQDUX+ULk9fsytGx4nO+LqdCLEyuGOGmEoQbAEA5HvrpPwg3lSDcAAAqwkM/6zbCTSUINwCAqig/hbXiy6MqKq2LUad+PfSTcFMJwg0AwFPl99aZv26fMrMKVBcvwrJL6tiisR6+vaNu7mC9WR3CTSUINwCA6io959TCz7/R39IP6mjhWV+XU6GGwQEa0K2lpllkVodwUwnCDQDA21iYXPMIN5Ug3AAAahILk2sG4aYShBsAQG3yh4XJLcJDNKxPXJ2e1SHcVIJwAwDwFX9YmFxXn4NFuKkE4QYAUFdcuDA5u/BsnXzoZ115DhbhphKEGwBAXVXXH/oZaJeujfHN5eaEm0oQbgAA/sAfHvpZm5ebE24qQbgBAPij8oXJn3x1VIUldS/q1PTl5oSbShBuAAD+rnxh8svrv1ZmVn6dvLdO8nXNNX9IL6+duiLcVIJwAwCwmvJ767y3KUsHjp+pMwuT7TZp/pAbdEeXFtV+L08+v+vExezz5s1TXFycQkNDlZCQoIyMjEr7v/fee+rYsaNCQ0PVtWtXrVixopYqBQCg7gkOtGvUz9przSO36sDM/tr91B36dXxLNQj07c35nEYa+bctWrnjaK0e1+fhZtmyZZowYYKmTZumLVu2qFu3bkpOTlZubm6F/f/1r39p8ODBuv/++7V161YNHDhQAwcO1I4dO2q5cgAA6qYGwQF67lc9tPuZfvp2Rj+9PjRenWIa+uxD/8mPdqnMWXvzST4/LZWQkKBevXpp7ty5kiSn0ymHw6GHHnpIjz766EX9Bw0apOLiYi1fvtzV9pOf/ETdu3fXggULLupfUlKikpIS1/eFhYVyOByclgIA1EvlC5M/2vp9rV5u/s6InyjxmmZXvL/fnJYqLS1VZmamkpKSXG12u11JSUlKT0+vcJ/09HS3/pKUnJx8yf6pqamKiIhwvRwOh/cGAACAn2kQHKBnf9ldu5/pp33P3KlJd1yra6Ia1HggyC06U8NH+A+fhpu8vDyVlZUpJibGrT0mJkbZ2dkV7pOdne1R/8mTJ6ugoMD1Onz4sHeKBwDAz5Wv1Ul7+L+0///W6gzq1UqNg70fD6Ibh3r9PS+l7jw0ooaEhIQoJCTE12UAAFDnlc/qPPvL7l693Dw2PFS92zb1XqGX4dNwExUVpYCAAOXk5Li15+TkKDY2tsJ9YmNjPeoPAAA8F2C36eZO0bq5U7Sk6l1u/uRdnWv1UQ0+PS0VHBysnj17Ki0tzdXmdDqVlpamxMTECvdJTEx06y9Jq1evvmR/AABQfVdyuXlQgE0Lfued+9x4wuenpSZMmKBhw4YpPj5evXv31uzZs1VcXKzhw4dLklJSUtSqVSulpqZKksaOHaubb75ZL7zwgvr376+lS5dq8+bNevXVV305DAAA6pXyy82f+1UP13Ownv90r3KLShQTHqoJt3Wo9YdrlvN5uBk0aJCOHTumqVOnKjs7W927d9fKlStdi4azsrJkt/9ngqlPnz56++239cQTT+ixxx5T+/bt9eGHH6pLly6+GgIAAPVagN2mW66L0S3XxVy+cy3w+X1uahuPXwAAwP/4zX1uAAAAvI1wAwAALIVwAwAALIVwAwAALIVwAwAALIVwAwAALIVwAwAALIVwAwAALMXndyiubeX3LCwsLPRxJQAAoKrKP7ercu/hehduioqKJEkOh8PHlQAAAE8VFRUpIiKi0j717vELTqdT33//vRo3biybzbsP8yosLJTD4dDhw4frxaMdGK+1MV5rY7zWZ7UxG2NUVFSkli1buj1zsiL1bubGbrerdevWNXqM8PBwS/wiVRXjtTbGa22M1/qsNObLzdiUY0ExAACwFMINAACwFMKNF4WEhGjatGkKCQnxdSm1gvFaG+O1NsZrffVxzOXq3YJiAABgbczcAAAASyHcAAAASyHcAAAASyHcAAAASyHceMm8efMUFxen0NBQJSQkKCMjw9clXZHU1FT16tVLjRs3VnR0tAYOHKi9e/e69Tlz5oxGjx6tZs2aqVGjRvrlL3+pnJwctz5ZWVnq37+/wsLCFB0drUceeUTnzp2rzaF4bObMmbLZbBo3bpyrzYpjPXLkiH73u9+pWbNmatCggbp27arNmze7thtjNHXqVLVo0UINGjRQUlKSvv76a7f3OHHihIYMGaLw8HBFRkbq/vvv18mTJ2t7KJdVVlamKVOmqG3btmrQoIGuueYaPf30027PpvHn8X722WcaMGCAWrZsKZvNpg8//NBtu7fG9tVXX+mmm25SaGioHA6HZs2aVdNDq1Bl4z179qwmTZqkrl27qmHDhmrZsqVSUlL0/fffu72HP41Xuvy/8YVGjhwpm82m2bNnu7X725i9wqDali5daoKDg81f//pXs3PnTjNixAgTGRlpcnJyfF2ax5KTk83rr79uduzYYbZt22b69etn2rRpY06ePOnqM3LkSONwOExaWprZvHmz+clPfmL69Onj2n7u3DnTpUsXk5SUZLZu3WpWrFhhoqKizOTJk30xpCrJyMgwcXFx5vrrrzdjx451tVttrCdOnDBXXXWVue+++8zGjRvN/v37zapVq8w333zj6jNz5kwTERFhPvzwQ/Pll1+au+66y7Rt29acPn3a1eeOO+4w3bp1M//+97/N559/btq1a2cGDx7siyFVavr06aZZs2Zm+fLl5sCBA+a9994zjRo1MnPmzHH18efxrlixwjz++OPmgw8+MJLM3//+d7ft3hhbQUGBiYmJMUOGDDE7duww77zzjmnQoIF55ZVXamuYLpWNNz8/3yQlJZlly5aZPXv2mPT0dNO7d2/Ts2dPt/fwp/Eac/l/43IffPCB6datm2nZsqX585//7LbN38bsDYQbL+jdu7cZPXq06/uysjLTsmVLk5qa6sOqvCM3N9dIMuvXrzfGnP8DEhQUZN577z1Xn927dxtJJj093Rhz/j9Gu91usrOzXX1efvllEx4ebkpKSmp3AFVQVFRk2rdvb1avXm1uvvlmV7ix4lgnTZpk+vbte8ntTqfTxMbGmueee87Vlp+fb0JCQsw777xjjDFm165dRpLZtGmTq88nn3xibDabOXLkSM0VfwX69+9vfv/737u13XPPPWbIkCHGGGuN98cffN4a2/z5802TJk3cfp8nTZpkOnToUMMjqlxlH/TlMjIyjCRz6NAhY4x/j9eYS4/5u+++M61atTI7duwwV111lVu48fcxXylOS1VTaWmpMjMzlZSU5Gqz2+1KSkpSenq6DyvzjoKCAklS06ZNJUmZmZk6e/as23g7duyoNm3auMabnp6url27KiYmxtUnOTlZhYWF2rlzZy1WXzWjR49W//793cYkWXOsH330keLj4/XrX/9a0dHR6tGjhxYuXOjafuDAAWVnZ7uNOSIiQgkJCW5jjoyMVHx8vKtPUlKS7Ha7Nm7cWHuDqYI+ffooLS1N+/btkyR9+eWX2rBhg+68805J1hvvhbw1tvT0dP30pz9VcHCwq09ycrL27t2rH374oZZGc2UKCgpks9kUGRkpyZrjdTqdGjp0qB555BFdd911F2234pirgnBTTXl5eSorK3P7cJOkmJgYZWdn+6gq73A6nRo3bpxuvPFGdenSRZKUnZ2t4OBg1x+LcheONzs7u8KfR/m2umTp0qXasmWLUlNTL9pmtbFK0v79+/Xyyy+rffv2WrVqlUaNGqUxY8ZoyZIlkv5Tc2W/z9nZ2YqOjnbbHhgYqKZNm9a5MT/66KP6zW9+o44dOyooKEg9evTQuHHjNGTIEEnWG++FvDU2f/sdL3fmzBlNmjRJgwcPdj000orjffbZZxUYGKgxY8ZUuN2KY66KevdUcFTd6NGjtWPHDm3YsMHXpdSIw4cPa+zYsVq9erVCQ0N9XU6tcDqdio+P14wZMyRJPXr00I4dO7RgwQINGzbMx9V537vvvqu33npLb7/9tq677jpt27ZN48aNU8uWLS05Xpx39uxZ3XvvvTLG6OWXX/Z1OTUmMzNTc+bM0ZYtW2Sz2XxdTp3CzE01RUVFKSAg4KIraHJychQbG+ujqqrvwQcf1PLly7V27Vq1bt3a1R4bG6vS0lLl5+e79b9wvLGxsRX+PMq31RWZmZnKzc3VDTfcoMDAQAUGBmr9+vV66aWXFBgYqJiYGMuMtVyLFi3UuXNnt7ZOnTopKytL0n9qruz3OTY2Vrm5uW7bz507pxMnTtS5MT/yyCOu2ZuuXbtq6NChGj9+vGumzmrjvZC3xuZvv+PlwebQoUNavXq1a9ZGst54P//8c+Xm5qpNmzauv2GHDh3SxIkTFRcXJ8l6Y64qwk01BQcHq2fPnkpLS3O1OZ1OpaWlKTEx0YeVXRljjB588EH9/e9/15o1a9S2bVu37T179lRQUJDbePfu3ausrCzXeBMTE7V9+3a3/6DK/8j8+IPVl2699VZt375d27Ztc73i4+M1ZMgQ19dWGWu5G2+88aJL+/ft26errrpKktS2bVvFxsa6jbmwsFAbN250G3N+fr4yMzNdfdasWSOn06mEhIRaGEXVnTp1Sna7+5+5gIAAOZ1OSdYb74W8NbbExER99tlnOnv2rKvP6tWr1aFDBzVp0qSWRlM15cHm66+/1qeffqpmzZq5bbfaeIcOHaqvvvrK7W9Yy5Yt9cgjj2jVqlWSrDfmKvP1imYrWLp0qQkJCTGLFy82u3btMg888ICJjIx0u4LGX4waNcpERESYdevWmaNHj7pep06dcvUZOXKkadOmjVmzZo3ZvHmzSUxMNImJia7t5ZdH33777Wbbtm1m5cqVpnnz5nX28ugLXXi1lDHWG2tGRoYJDAw006dPN19//bV56623TFhYmPnb3/7m6jNz5kwTGRlp/vGPf5ivvvrK3H333RVePtyjRw+zceNGs2HDBtO+ffs6cWn0jw0bNsy0atXKdSn4Bx98YKKiosyf/vQnVx9/Hm9RUZHZunWr2bp1q5FkXnzxRbN161bX1UHeGFt+fr6JiYkxQ4cONTt27DBLly41YWFhPrlMuLLxlpaWmrvuusu0bt3abNu2ze3v14VXAfnTeI25/L/xj/34ailj/G/M3kC48ZK//OUvpk2bNiY4ONj07t3b/Pvf//Z1SVdEUoWv119/3dXn9OnT5o9//KNp0qSJCQsLM7/4xS/M0aNH3d7n4MGD5s477zQNGjQwUVFRZuLEiebs2bO1PBrP/TjcWHGsH3/8senSpYsJCQkxHTt2NK+++qrbdqfTaaZMmWJiYmJMSEiIufXWW83evXvd+hw/ftwMHjzYNGrUyISHh5vhw4eboqKi2hxGlRQWFpqxY8eaNm3amNDQUHP11Vebxx9/3O3Dzp/Hu3bt2gr/ex02bJgxxntj+/LLL03fvn1NSEiIadWqlZk5c2ZtDdFNZeM9cODAJf9+rV271vUe/jReYy7/b/xjFYUbfxuzN9iMueBWnQAAAH6ONTcAAMBSCDcAAMBSCDcAAMBSCDcAAMBSCDcAAMBSCDcAAMBSCDcAAMBSCDcAAMBSCDcA6p24uDjNnj3b12UAqCGEGwA16r777tPAgQMlST/72c80bty4Wjv24sWLFRkZeVH7pk2b9MADD9RaHQBqV6CvCwAAT5WWlio4OPiK92/evLkXqwFQ1zBzA6BW3HfffVq/fr3mzJkjm80mm82mgwcPSpJ27NihO++8U40aNVJMTIyGDh2qvLw8174/+9nP9OCDD2rcuHGKiopScnKyJOnFF19U165d1bBhQzkcDv3xj3/UyZMnJUnr1q3T8OHDVVBQ4Drek08+Keni01JZWVm6++671ahRI4WHh+vee+9VTk6Oa/uTTz6p7t27680331RcXJwiIiL0m9/8RkVFRTX7QwNwRQg3AGrFnDlzlJiYqBEjRujo0aM6evSoHA6H8vPz9V//9V/q0aOHNm/erJUrVyonJ0f33nuv2/5LlixRcHCwvvjiCy1YsECSZLfb9dJLL2nnzp1asmSJ1qxZoz/96U+SpD59+mj27NkKDw93He/hhx++qC6n06m7775bJ06c0Pr167V69Wrt379fgwYNcuv37bff6sMPP9Ty5cu1fPlyrV+/XjNnzqyhnxaA6uC0FIBaERERoeDgYIWFhSk2NtbVPnfuXPXo0UMzZsxwtf31r3+Vw+HQvn37dO2110qS2rdvr1mzZrm954Xrd+Li4vTMM89o5MiRmj9/voKDgxURESGbzeZ2vB9LS0vT9u3bdeDAATkcDknSG2+8oeuuu06bNm1Sr169JJ0PQYsXL1bjxo0lSUOHDlVaWpqmT59evR8MAK9j5gaAT3355Zdau3atGjVq5Hp17NhR0vnZknI9e/a8aN9PP/1Ut956q1q1aqXGjRtr6NChOn78uE6dOlXl4+/evVsOh8MVbCSpc+fOioyM1O7du11tcXFxrmAjSS1atFBubq5HYwVQO5i5AeBTJ0+e1IABA/Tss89etK1Fixaurxs2bOi27eDBg/r5z3+uUaNGafr06WratKk2bNig+++/X6WlpQoLC/NqnUFBQW7f22w2OZ1Orx4DgHcQbgDUmuDgYJWVlbm13XDDDfp//+//KS4uToGBVf+TlJmZKafTqRdeeEF2+/lJ6Hffffeyx/uxTp066fDhwzp8+LBr9mbXrl3Kz89X586dq1wPgLqD01IAak1cXJw2btyogwcPKi8vT06nU6NHj9aJEyc0ePBgbdq0Sd9++61WrVql4cOHVxpM2rVrp7Nnz+ovf/mL9u/frzfffNO10PjC4508eVJpaWnKy8ur8HRVUlKSunbtqiFDhmjLli3KyMhQSkqKbr75ZsXHx3v9ZwCg5hFuANSahx9+WAEBAercubOaN2+urKwstWzZUl988YXKysp0++23q2vXrho3bpwiIyNdMzIV6datm1588UU9++yz6tKli9566y2lpqa69enTp49GjhypQYMGqXnz5hctSJbOn176xz/+oSZNmuinP/2pkpKSdPXVV2vZsmVeHz+A2mEzxhhfFwEAAOAtzNwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABLIdwAAABL+f/x42ntwSShmAAAAABJRU5ErkJggg==\n", |
| "text/plain": [ |
| "<Figure size 640x480 with 1 Axes>" |
| ] |
| }, |
| "metadata": {}, |
| "output_type": "display_data" |
| } |
| ], |
| "source": [ |
| "schedule = mx.lr_scheduler.MultiFactorScheduler(step=[250, 750, 900], factor=0.5)\n", |
| "schedule.base_lr = 1\n", |
| "schedule = LinearCoolDown(schedule, finish_lr=0.001, start_idx=1000, length=500)\n", |
| "plot_schedule(schedule)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "da4cef78", |
| "metadata": {}, |
| "source": [ |
| " <!--notebook-skip-line-->\n", |
| "\n", |
| "\n", |
| "#### 1-Cycle: for \"Super-Convergence\"\n", |
| "\n", |
| "So we can implement the \"1-Cycle\" schedule proposed by [Leslie N. Smith, Nicholay Topin (2017)](https://arxiv.org/abs/1708.07120) we use a single and symmetric cycle of the triangular schedule above (i.e. `inc_fraction=0.5`), followed by a cool-down period of `cooldown_length` iterations." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 11, |
| "id": "b7b0dab5", |
| "metadata": {}, |
| "outputs": [], |
| "source": [ |
| "class OneCycleSchedule():\n", |
| " def __init__(self, start_lr, max_lr, cycle_length, cooldown_length=0, finish_lr=None):\n", |
| " \"\"\"\n", |
| " start_lr: lower bound for learning rate in triangular cycle (float)\n", |
| " max_lr: upper bound for learning rate in triangular cycle (float)\n", |
| " cycle_length: iterations between start and finish of triangular cycle: 2x 'stepsize' (int)\n", |
| " cooldown_length: number of iterations used for the cool-down (int)\n", |
| " finish_lr: learning rate used at end of the cool-down (float)\n", |
| " \"\"\"\n", |
| " if (cooldown_length > 0) and (finish_lr is None):\n", |
| " raise ValueError(\"Must specify finish_lr when using cooldown_length > 0.\")\n", |
| " if (cooldown_length == 0) and (finish_lr is not None):\n", |
| " raise ValueError(\"Must specify cooldown_length > 0 when using finish_lr.\")\n", |
| "\n", |
| " finish_lr = finish_lr if (cooldown_length > 0) else start_lr\n", |
| " schedule = TriangularSchedule(min_lr=start_lr, max_lr=max_lr, cycle_length=cycle_length)\n", |
| " self.schedule = LinearCoolDown(schedule, finish_lr=finish_lr, start_idx=cycle_length, length=cooldown_length)\n", |
| "\n", |
| " def __call__(self, iteration):\n", |
| " return self.schedule(iteration)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "a5a9babd", |
| "metadata": {}, |
| "source": [ |
| "As an example, we linearly increase and then decrease the learning rate from 0.1 to 0.5 and back over 500 iterations (i.e. single triangular cycle), before reducing the learning rate further to 0.001 over the next 750 iterations (i.e. cool-down)." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 12, |
| "id": "5ad73499", |
| "metadata": {}, |
| "outputs": [ |
| { |
| "data": { |
| "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABEs0lEQVR4nO3deXRU5eH/8c9MQhIiJAFCEpZA2BQRBCQQglq05itV6tpvRQ4FpBaPW9WifhUXaF0aRKVoRVEqxa0F25/VyhehNIKKBsMuIKCyBYEkBEzCHpi5vz/4zpSBZHJvMuud9+ucnAOTJ5nnipKP93nu83EYhmEIAADAJpzhngAAAEAgEW4AAICtEG4AAICtEG4AAICtEG4AAICtEG4AAICtEG4AAICtxId7AqHmdru1Z88etWzZUg6HI9zTAQAAJhiGoYMHD6p9+/ZyOv3fm4m5cLNnzx5lZ2eHexoAAKARdu3apY4dO/odE3PhpmXLlpJO/cNJSUkJ82wAAIAZNTU1ys7O9v4c9yfmwo1nKSolJYVwAwBAlDGzpYQNxQAAwFYINwAAwFYINwAAwFYINwAAwFYINwAAwFYINwAAwFYINwAAwFYINwAAwFYINwAAwFZi7oRihJ/Lbahk+wFVHDymjJZJGtSlteKclJgCAAIjIu7czJgxQzk5OUpKSlJeXp5KSkrqHTtnzhw5HA6fj6SkpBDOFk2x4Ku9yn1qsUbOWq57567VyFnLdfGUIi3csDfcUwMA2ETYw828efM0YcIETZ48WatXr1bfvn01bNgwVVRU1Ps1KSkp2rt3r/dj586dIZwxGqtwwde68y+r9cOREz6vl9Uc1+1vrybgAAACIuzhZtq0aRo/frzGjRunXr16aebMmUpOTtbs2bPr/RqHw6GsrCzvR2ZmZghnjMZY8NUevfrpdr9jHn5vvVxuI0QzAgDYVVjDTW1trVatWqWCggLva06nUwUFBSouLq736w4dOqTOnTsrOztb1113nTZu3Fjv2OPHj6umpsbnA6Hlcht64O/rGhxXdeSElm/dH4IZAQDsLKzhprKyUi6X66w7L5mZmSorK6vza8477zzNnj1bH3zwgd5++2253W4NGTJE33//fZ3jCwsLlZqa6v3Izs4O+HXAv+Xb9utIrdvU2DeX7wjuZAAAthf2ZSmr8vPzNWbMGPXr109Dhw7Ve++9p7Zt2+rVV1+tc/zEiRNVXV3t/di1a1eIZ4wvtlaaHrtkcwVLUwCAJgnro+Dp6emKi4tTeXm5z+vl5eXKysoy9T2aNWum/v3767vvvqvz84mJiUpMTGzyXNF4JdsPmB5b6zK0fOt+XdwjPYgzAgDYWVjv3CQkJGjAgAEqKiryvuZ2u1VUVKT8/HxT38Plcmn9+vVq165dsKaJJnC5Da3Z+YOlr2FpCgDQFGE/xG/ChAkaO3ascnNzNWjQIE2fPl2HDx/WuHHjJEljxoxRhw4dVFhYKEl64oknNHjwYHXv3l1VVVV69tlntXPnTv3qV78K52WgHsu37ddJi6tMnqUpDvYDADRG2MPNiBEjtG/fPk2aNEllZWXq16+fFi5c6N1kXFpaKqfzPzeYfvjhB40fP15lZWVq1aqVBgwYoC+++EK9evUK1yXADyv7bTxYmgIANIXDMIyY2r1ZU1Oj1NRUVVdXKyUlJdzTsb2bZn6hkh3WlqUk6e7Lu+mBYT2DMCMAQDSy8vM76p6WQvRwuQ2t3VXVqK/lgSkAQGMRbhA0y7ftV62rcSmlrPpYgGcDAIgVhBsEzVvFOxr9tQvW7+W8GwBAoxBuEBQut6GiTeUND6zHsZNuqhgAAI1CuEFQLN+2XyfMNS7Ui/NuAACNQbhBUDTmEfAzUcUAAGgMwg2CwkrlQn08590AAGAF4QYB15jKhfqwNAUAsIpwg4BrTOVCfViaAgBYRbhBwDXlEfAzsTQFALCKcIOAauoj4HX5fOu+gH4/AIC9EW4QUIF4BPxMKwKwORkAEDsINwgoK4+An5d5jqlxG/bUsO8GAGAa4QYBZfYuS0KcQ1f0zDI19ugJd0AeLQcAxAbCDQLGSgv45T0zdHGPdNPfu6z6aCNnBQCINYQbBIyVFvAxg3M0uGsbJcY7TI1f9l3TTzwGAMQGwg0Cxux+m6R4pwZ3a6M4p0OXn5dh6mv+9ytawgEA5hBuEDBm98Vc2DFVcc5Td2y6Z7Q09TW0hAMAzCLcICCsVC60S03y/jq/WxvT70EVAwDADMINAsJK5UKHVs29vx7ctY2amfy3kCoGAIAZhBsEhJXKhYu7tfX+Os7p0BXnZ5r6OqoYAABmEG7QZFYqFxLiHBp8xlLU6Pwc0+9FFQMAoCGEGzSZlcqFy3tmeDcTewzu2kYmnwinigEA0CDCDZrMypLUmME5Z70W53Sof+dWpr5+3ffV7LsBAPhFuEGTNHVJymNQl9amvgf7bgAADSHcoEmauiTlMaSb+SoGHgkHAPhDuEGTWGkBr2tJysPKI+GffrOPpSkAQL0IN2gSKy3g9S1JSdYeCaclHADgD+EGjWa1Bby+JSkPK4+E0xIOAKgP4QaNZrUFvCG0hAMAAoFwg0az2gLeEFrCAQCBQLhBozWmBbwhtIQDAJqKcINGaWwLeEOstIRTxQAAqAvhBo3S2BbwhlDFAABoKsINGqWxLeANoYoBANBUhBtYFqjKhfpQxQAAaArCDSwLVOVCfahiAAA0BeEGljW1BbwhVqoYlmyuYGkKAOCDcANLgr0kJVmrYmBpCgBwJsINLAn2kpSHlSoGlqYAAKcj3MCSQLWAN4SWcABAYxFuYMnuH8wVVjZ2ScqDlnAAQGMRbmDJ0dqTpsY1ZUnKg5ZwAEBjEG5gmsttaOk35ioPck0exOePlZbwykPHm/x+AAB7INzAtOXb9uu4yc6F9BaJTX6/OKdDl51r7nTjlSZ7rgAA9ke4gWlWzrfJSjXfJ+VP84R4U+M47wYA4EG4gSlWzrdp3sxpukKhIWZLNznvBgDgQbiBKVbOt/nRuW2bvJnYgyoGAIBVhBuYEuzKhfpQxQAAsIpwgwaFonKhPlQxAACsItygQaGqXKgPVQwAACsIN2hQqCoX6sPSFADACsINGmS22iDQS1IeLE0BAKwg3MAvl9vQGpMH5PXtmBrwJSkPK0tTn281d4oyAMCeCDfwa/m2/TJ5KLEGBuhsm7oM7tpGJpsYTJd7AgDsiXADv4otLPFc3M1cVUJjxDkd6m+yr2pv9bGgzQMAEPkIN/Dru4qDpsYlxTuDst/mdGZPPf7q+2o2FQNADIuIcDNjxgzl5OQoKSlJeXl5KikpMfV1c+fOlcPh0PXXXx/cCcYol9vQki0VpsYOv7Bd0PbbeJg9rfjYSTebigEghoU93MybN08TJkzQ5MmTtXr1avXt21fDhg1TRYX/H6o7duzQAw88oEsvvTREM409VlrAL+luviahsQZ3baOEOHMBivNuACB2hT3cTJs2TePHj9e4cePUq1cvzZw5U8nJyZo9e3a9X+NyuTRq1Cj97ne/U9euXf1+/+PHj6umpsbnA+aEowXcnzinQ/2y00yN5bwbAIhdYQ03tbW1WrVqlQoKCryvOZ1OFRQUqLi4uN6ve+KJJ5SRkaFbb721wfcoLCxUamqq9yM7Ozsgc7e7cLWAN8TsE1mcdwMAsSus4aayslIul0uZmb4HtGVmZqqsrKzOr1m2bJlef/11zZo1y9R7TJw4UdXV1d6PXbt2NXnesSBcLeANoSUcANCQ+HBPwIqDBw9q9OjRmjVrltLTzf2QS0xMVGJiYpBnZj/hrlyoj6eKwUzw8ixNhSp4AQAiQ1jDTXp6uuLi4lRe7rv8UV5erqysrLPGb926VTt27NA111zjfc3tPvVTLj4+Xlu2bFG3bt2CO+kYEe7Khfp4qhgWbmx4ycyzNHVxj+BvdgYARI6wLkslJCRowIABKioq8r7mdrtVVFSk/Pz8s8b37NlT69ev19q1a70f1157rS6//HKtXbuW/TQBEimVC/WhigEA4E/Yl6UmTJigsWPHKjc3V4MGDdL06dN1+PBhjRs3TpI0ZswYdejQQYWFhUpKSlLv3r19vj4tLU2SznodjRcplQv18VQxmJnjCpN3oAAA9hH2cDNixAjt27dPkyZNUllZmfr166eFCxd6NxmXlpbK6Qz7E+sxxcoj4MGsXKiPp4phxY6G7y6t+7/Titl3AwCxw2EYRkwdBlJTU6PU1FRVV1crJSUl3NOJOC63oZ6PLTC1YTchzqFNT14VluDw7KLNmrFkq6mx79yax74bAIhyVn5+c0sEPqw8An55z4yw3RGx8kg4+24AILYQbuDDSgt4KB8BP5PnkXAzVppYvgIA2AfhBj4iqQXcH88j4WbQEg4AsYVwA69IawFviNlHwmkJB4DYQriBV6S1gDeElnAAQF0IN/CyUrkQihbwhtASDgCoC+EGXmYrF0LZAt4QWsIBAGci3ECStcqF3u1Twr7fxoNHwgEAZyLcQFLkVy7Ux1PFYAZVDAAQGwg3kBT5lQv18VQxmLGOR8IBICYQbiCX21DRpnJTYxPiHGE936YuZvf/sO8GAGID4QZRU7lQHyv7bngkHADsj3ADS0tS4axcqI+VKgYeCQcA+yPcxLhoX5KSrFUxsDQFAPZHuIlx0b4k5WG2ikHikXAAsDvCTYyLlhbwhtASDgDwINzEOEPm9p+EuwW8IbSEAwA8CDcxbm/VEVPjru4T/hbwhtASDgCQCDcxzeU2NP+rvabGZqUmBXk2TWelJZx9NwBgX4SbGLZ8237VusyNjfCbNpJOLU317ZhqaixVDABgX4SbGGblfJv8ruYPygunDq2STY2jigEA7ItwE6PscL5NXTq0am5qHOfdAIB9EW5ilF3OtzkTVQwAAMJNjIr2yoX6UMUAACDcxCC7LklJVDEAAAg3McmuS1IeVqoYWJoCAPsh3MSgL7ZWmh4bTUtSHixNAUBsI9zEoN0/HDU1LtqWpDxYmgKA2Ea4iUG7fzBXudC3Y2rULUl5WFmaKt5m/k4WACDyEW5ijMttaN331abGDuzSOsizCZ5TVQzmxn5bcSi4kwEAhBThJsacqlwwt8fk4m5tgzyb4IlzOnRN3/amxn76zT723QCAjRBuYozZ822S4p1Rud/mdJf0yDA17ugJWsIBwE4INzHEyvk2F0bxfhuPrBTzTea0hAOAfRBuYoiV821yc1oFdzIhMKhLayXGmwtotIQDgH0QbmKIlcqFaN5v4xHndOjCjmmmxtISDgD2QbiJEXauXPBnkMknvjjvBgDsg3ATI+xeuVAfWsIBIPYQbmKE3SsX6kMVAwDEHsJNjCgxuWHWTktSElUMABCLCDcxwOU2tGbnD6bGRnPlQn1oCQeA2EK4iQHLt+3XSZOrLdFcuVAflqYAILYQbmKAlf02dngE/EwsTQFAbCHcxACzB9TZbb/N6WgJB4DYQbixOZfb0NpdVabG2ukR8DNZaQlnVQoAohvhxuastIDb6RHwM8U5Hfrphe1MjS2rPhbk2QAAgolwY3Ox1ALekHZpyabGLVi/l03FABDFmhRujh3j/3AjWay1gDfEIXPXd+ykm03FABDFLIcbt9utJ598Uh06dFCLFi20bds2SdLjjz+u119/PeATROPFWgt4Q/It3JnivBsAiF6Ww81TTz2lOXPmaOrUqUpISPC+3rt3b/3pT38K6OTQNLH+CPiZOO8GAGKD5XDz5ptv6rXXXtOoUaMUF/efx0/69u2rzZs3B3RyaJpYrVyoD+fdAEBssBxudu/ere7du5/1utvt1okTJwIyKTRdrFcu1IcqBgCwP8vhplevXvrss8/Oev3vf/+7+vfvH5BJoelivXKhPixNAYD9xVv9gkmTJmns2LHavXu33G633nvvPW3ZskVvvvmm5s+fH4w5ohHMPgIuxcZ+Gw/P0tTCjQ0/ReZZmrq4R3oIZgYACBTLd26uu+46ffjhh/r3v/+tc845R5MmTdKmTZv04Ycf6r/+67+CMUdYZOUR8FjZb3M6K0tTn2/dF7yJAACCwvKdG0m69NJLtXjx4kDPBQFi5RFwO1cu1Gdw1zaKd8jUsp3ZXi4AQOSwfOema9eu2r//7KdIqqqq1LVr14BMCk1j5RFwO1cu1CfO6VD/zubO9dmwp4Z9NwAQZSyHmx07dsjlcp31+vHjx7V79+5GTWLGjBnKyclRUlKS8vLyVFJSUu/Y9957T7m5uUpLS9M555yjfv366a233mrU+9oVLeANG2RyE/XRE27Tj9QDACKD6WWpf/7zn95fL1q0SKmpqd7fu1wuFRUVKScnx/IE5s2bpwkTJmjmzJnKy8vT9OnTNWzYMG3ZskUZGRlnjW/durUeffRR9ezZUwkJCZo/f77GjRunjIwMDRs2zPL72w0t4OYM6ZauGUu2mhpbVn00yLMBAASSwzAMU/fcnc5TN3kcDofO/JJmzZopJydHzz//vH76059amkBeXp4GDhyol156SdKp83Kys7P161//Wg8//LCp73HRRRdp+PDhevLJJxscW1NTo9TUVFVXVyslJcXSXKPB599VatSfvjQ19p1b82L2SSCX21CvSR/puImNNz+7qIOev6lf8CcFAKiXlZ/fppel3G633G63OnXqpIqKCu/v3W63jh8/ri1btlgONrW1tVq1apUKCgr+MyGnUwUFBSouLm7w6w3DUFFRkbZs2aIf/ehHdY45fvy4ampqfD7szOx+m1hoAfcnzunQ5eedfWewLv/7FS3hABBNLO+52b59u9LTA/N/+5WVlXK5XMrM9D0SPzMzU2VlZfV+XXV1tVq0aKGEhAQNHz5cf/zjH+t9DL2wsFCpqanej+zs7IDMPVKZ3R8SCy3gDeme0dLUOFrCASC6NOpR8MOHD+uTTz5RaWmpamtrfT53zz33BGRi/rRs2VJr167VoUOHVFRUpAkTJqhr16667LLLzho7ceJETZgwwfv7mpoa2wYcK5UL7VKTgjybyJffrY1eWvKdqbFvLt8Rs0t4ABBtLIebNWvW6Oqrr9aRI0d0+PBhtW7dWpWVlUpOTlZGRoalcJOenq64uDiVl/seOFdeXq6srKx6v87pdHr7rfr166dNmzapsLCwznCTmJioxMRE03OKZlYqFzq0ah7cyUQBTxWDmTOBPFUMsX63CwCigeVlqd/85je65ppr9MMPP6h58+Zavny5du7cqQEDBui5556z9L0SEhI0YMAAFRUVeV9zu90qKipSfn6+6e/j2fcT66hcsIaWcACwJ8vhZu3atbr//vvldDoVFxen48ePKzs7W1OnTtUjjzxieQITJkzQrFmz9MYbb2jTpk264447dPjwYY0bN06SNGbMGE2cONE7vrCwUIsXL9a2bdu0adMmPf/883rrrbf0i1/8wvJ72wmVC41DFQMA2I/lZalmzZp5HwvPyMhQaWmpzj//fKWmpmrXrl2WJzBixAjt27dPkyZNUllZmfr166eFCxd6NxmXlpZ63086td/nzjvv1Pfff6/mzZurZ8+eevvttzVixAjL720nVC40DlUMAGA/lsNN//79tWLFCvXo0UNDhw7VpEmTVFlZqbfeeku9e/du1CTuvvtu3X333XV+bunSpT6/f+qpp/TUU0816n3szMqSVCxWLtTHU8WwYkfDG7HXfV/NvhsAiAKWl6V+//vfq127dpKkp59+Wq1atdIdd9yhffv26dVXXw34BNEwlqSaxmwVA/tuACA6WL5zk5ub6/11RkaGFi5cGNAJwTqWpJrGShUDj4QDQOSzfOemPqtXr7Z8QjECgxbwpvE8Em7Gp9/s47RiAIhwlsLNokWL9MADD+iRRx7Rtm3bJEmbN2/W9ddfr4EDB8rtNnn7AAG1+wdzxY4sSdXNyiPhtIQDQOQzHW5ef/11XXXVVZozZ46eeeYZDR48WG+//bby8/OVlZWlDRs2aMGCBcGcK+pxtPakqXEsSdXPyiPhtIQDQGQzHW5eeOEFPfPMM6qsrNS7776ryspKvfzyy1q/fr1mzpyp888/P5jzRD1cbkNLvzF3/kpu51ZBnk30Gty1jRLjzQW/Zd+ZXwYEAISe6XCzdetW/fznP5ck3XjjjYqPj9ezzz6rjh07Bm1yaNjybft13GTnQnqL2KihaAxawgHAPkyHm6NHjyo5OVmS5HA4lJiY6H0kHOFj5XybrFT6pPyhJRwA7MHSo+B/+tOf1KJFC0nSyZMnNWfOHKWn+z4WG4pWcJxi5Xyb5s2cps9ziVVWWsI/37qPR8IBIEKZDjedOnXSrFmzvL/PysrSW2+95TPG4XAQbkLIyvk2Pzq3LZuJG0AVAwDYg+lws2PHjiBOA41B5UJgUcUAAPYQsEP8EFpULgQHVQwAEP0IN1GKyoXgGNLN/D6aN5fvCN5EAACNRriJUlQuBIeVKoYlmyt4JBwAIhDhJkqZrQBgScoaK1UMLE0BQGQi3EQhl9vQmp0Nb3qVpL4dU1mSsshKFQNLUwAQeSydcyNJNTU1db7uOdgvISGhyZOCf8u37Tf1uLIkDeRsG8s8S1Nm9jR5WsIJkAAQOSzfuUlLS1OrVq3O+khLS1Pz5s3VuXNnTZ48mYbwICq2sBRycbe2QZyJPdESDgDRzfKdmzlz5ujRRx/VLbfcokGDBkmSSkpK9MYbb+ixxx7Tvn379NxzzykxMVGPPPJIwCcM6buKg6bGJcU72W/TSKPzc7Rwo7lH7WkJB4DIYjncvPHGG3r++ed10003eV+75ppr1KdPH7366qsqKipSp06d9PTTTxNugsDlNrRkS4WpscMvbMdySSN5WsLNlJJWHjoeghkBAMyyvCz1xRdfqH///me93r9/fxUXF0uSLrnkEpWWljZ9djiLlRbwS7rTfdRYcU6HLjvX3JLeSpObuwEAoWE53GRnZ+v1118/6/XXX39d2dnZkqT9+/erVatWTZ8dzkILeOg0TzB3Y5PzbgAgslhelnruuef085//XB999JEGDhwoSVq5cqU2b96sv//975KkFStWaMSIEYGdKWgBD7EOrcyFQ895N7SEA0BksHzn5tprr9XmzZt11VVX6cCBAzpw4ICuuuoqbd68WT/96U8lSXfccYemTZsW8MnGOlrAQ4sqBgCITpbv3EhSly5dNGXKlEDPBQ2gBTy0rJx341maIlACQPg1KtxUVVWppKREFRUVZ51nM2bMmIBMDL5oAQ89z3k3Zh4JZ2kKACKH5XDz4YcfatSoUTp06JBSUlLkcPzn/1QdDgfhJkhoAQ8PK+fdvLl8B+EGACKA5T03999/v375y1/q0KFDqqqq0g8//OD9OHCAk1qDhRbw8KAlHACij+Vws3v3bt1zzz1KTk4OxnxQD1rAw4OWcACIPpbDzbBhw7Ry5cpgzAX1oAU8vKy0hH++dV/wJgIAMMXynpvhw4frwQcf1Ndff60+ffqoWbNmPp+/9tprAzY5nEILeHgN7tpG8Q6Z+jPY/QM9UwAQbpbDzfjx4yVJTzzxxFmfczgccrlcTZ8VfNACHl5xTof6d26lFTsavnu2t/pYCGYEAPDH8rKU2+2u94NgExy0gIef2dOev/q+mk3FABBmlsMNQosW8Mhg9rTiYyfdbCoGgDAztSz14osv6rbbblNSUpJefPFFv2PvueeegEwMp9ACHhkGd22jhDiHal0N/1lw3g0AhJepcPOHP/xBo0aNUlJSkv7whz/UO87hcBBuAowW8MgQ53SoX3aaSkzsu6GKAQDCy1S42b59e52/RnDRAh5ZBnZpbSrcUMUAAOHFnpsIRgt4ZKElHACig+VHwV0ul+bMmaOioqI6izM//vjjgE0u1lG5EFloCQeA6GA53Nx7772aM2eOhg8frt69e/sUZyKwqFyILLSEA0B0sBxu5s6dq3fffVdXX311MOaD/0PlQmSy0hL++dZ9hBsACAPLe24SEhLUvXv3YMwFp6FyITJ5qhjMWGHyzhsAILAsh5v7779fL7zwggyDU1iDycoj4FQuhI6nisGMdZxWDABhYXlZatmyZVqyZIk++ugjXXDBBWcVZ7733nsBm1yssvIIOPttQm9Ql9ameqbYdwMA4WE53KSlpemGG24Ixlzwf6w8An55zwz224TYkG7pmrFkq6mx7LsBgNCzFG5Onjypyy+/XFdeeaWysrKCNaeYZ6UFnEfAQ8/KI+ErTdzhAQAElqU9N/Hx8br99tt1/PjxYM0HogU80nkeCTeDlnAACD3LG4oHDRqkNWvWBGMuEC3g0WJ0fo6pcbSEA0DoWd5zc+edd+r+++/X999/rwEDBuicc87x+fyFF14YsMnFIlrAowMt4QAQuSyHm5tvvlmSfNq/HQ6HDMOQw+GQy+UK3OxikJXKBVrAw4eWcACIXJbDDa3gwWW2coEW8PCjJRwAIpPlcNO5c+dgzAOyVrnQu30KdwLCjEfCASAyWQ43Hl9//bVKS0tVW1vr8/q1117b5EnFKioXoounisHMnxlVDAAQOpbDzbZt23TDDTdo/fr13r02krzt4Oy5aTwqF6KLp4rBzGnFnioG7rYBQPBZfhT83nvvVZcuXVRRUaHk5GRt3LhRn376qXJzc7V06dIgTDE2ULkQnczue/LsuwEABJ/lcFNcXKwnnnhC6enpcjqdcjqduuSSS1RYWOjzBBWsoXIhOg3pZn4fzZvLdwRvIgAAL8vhxuVyqWXLlpKk9PR07dmzR9KpjcZbtmxp1CRmzJihnJwcJSUlKS8vTyUlJfWOnTVrli699FK1atVKrVq1UkFBgd/x0cLKkhSVC5HDU8VghueRcABAcFkON71799a6deskSXl5eZo6dao+//xzPfHEE+ratavlCcybN08TJkzQ5MmTtXr1avXt21fDhg1TRUXdp/QuXbpUI0eO1JIlS1RcXKzs7GxdeeWV2r17t+X3jhQsSUUvK1UMLE0BQGhYDjePPfaY3O5T6ydPPPGEtm/frksvvVQLFizQiy++aHkC06ZN0/jx4zVu3Dj16tVLM2fOVHJysmbPnl3n+HfeeUd33nmn+vXrp549e+pPf/qT3G63ioqKLL93pGBJKrqZrWKQTj0SDgAILstPSw0bNsz76+7du2vz5s06cOCAWrVq5X1iyqza2lqtWrVKEydO9L7mdDpVUFCg4uJiU9/jyJEjOnHihFq3rntj5/Hjx32KPmtqaizNMRRoAY9utIQDQGSxfOfG47vvvtOiRYt09OjReoNFQyorK+VyuZSZ6XtbPzMzU2VlZaa+x0MPPaT27duroKCgzs8XFhYqNTXV+5Gdnd2ouQaTIXP7MGgBj0y0hANAZLEcbvbv368rrrhC5557rq6++mrt3btXknTrrbfq/vvvD/gE/ZkyZYrmzp2rf/zjH0pKSqpzzMSJE1VdXe392LVrV0jnaMbeqiOmxl3dhxbwSEVLOABEDsvh5je/+Y2aNWum0tJSJScne18fMWKEFi5caOl7paenKy4uTuXlvptpy8vLlZWV5fdrn3vuOU2ZMkX/+te//DaRJyYmKiUlxecjkrjchuZ/tdfU2KzUugMcws/TEm4G+24AILgsh5t//etfeuaZZ9SxY0ef13v06KGdO3da+l4JCQkaMGCAz2Zgz+bg/Pz8er9u6tSpevLJJ7Vw4ULl5uZau4AIs3zbftWaPNSZmzaRK87pUN+OqabGUsUAAMFlOdwcPnzY546Nx4EDB5SYmGh5AhMmTNCsWbP0xhtvaNOmTbrjjjt0+PBhjRs3TpI0ZswYnw3HzzzzjB5//HHNnj1bOTk5KisrU1lZmQ4dOmT5vSOBlfNt8rtSvBjJOrQ6+7+Luqxj3w0ABJXlcHPppZfqzTff9P7e4XDI7XZr6tSpuvzyyy1PYMSIEXruuec0adIk9evXT2vXrtXChQu9m4xLS0u9+3ok6ZVXXlFtba3++7//W+3atfN+PPfcc5bfO9w438ZeOrRqbmoc590AQHA5DE/zpUkbNmzQFVdcoYsuukgff/yxrr32Wm3cuFEHDhzQ559/rm7dugVrrgFRU1Oj1NRUVVdXh33/zeffVWrUn740NXbYBZl6dXR0L8HZHX+eABA8Vn5+N+qE4m+++UaXXHKJrrvuOh0+fFg33nij1qxZE/HBJtJQuWAvVDEAQGSwfIifJKWmpurRRx/1ee3777/Xbbfdptdeey0gE7M7lqTsx3PezcKNDf+5epamLu7BPioACLRGH+J3pv379+v1118P1LezPSoX7MlKFQMt4QAQHAELN7Dmi62VpseyJBU9WJoCgPAj3IRJicmzTliSii60hANA+BFuwsDlNvTV91WmxvbtmMqSVJShJRwAwsv0huIbb7zR7+erqqqaOpeYUbL9gI6fNLccMbBL40pJET5WWsL3VB0L/oQAIMaYDjepqf6Plk9NTdWYMWOaPKFYUFZj/gfaxd3aBnEmCIY4p0M/7pmhRV9XNDj26AmT3RsAANNMh5s///nPwZxHTFn2bcM/9CSpeTMn+22iVG5OG1Ph5tNv9snlNlh6BIAAYs9NiLnchj5ct8fU2B+d25YfelEqvaW5nrWjJ9xsKgaAACPchJiVFvAeGS2COxkETVZKkumxnHcDAIFFuAkxWsBjw6AurZVk8sAbzrsBgMAi3IQQlQuxI87p0GXnmtsMznk3ABBYhJsQonIhtlDFAADhQbgJISoXYgtVDAAQHoSbEKJyIbZQxQAA4UG4CRGX29CanT+YGkvlgn1QxQAAoUe4CZHl2/bLZOMClQs2MrhrG8WbzKkrTN7ZAwD4R7gJESuPgFO5YB9xTof6d25lauy676vZdwMAAUC4CQGX29DSb8wtObDfxn4GmbwTx74bAAgMwk0IlGw/oGMmnwHnEXD7GdLN/GGMxdvMP1EHAKgb4SYErLSA8wi4/Qzu2kYJcebGfltxKLiTAYAYQLgJAVrAY1uc06Fr+rY3NdbTEg4AaDzCTZDRAg5JuqRHhqlxtIQDQNMRboKMFnBItIQDQCgRboLMSuUCLeD2RUs4AIQO4SbIqFyAREs4AIQS4SaIqFzA6WgJB4DQINwEEZULOB0t4QAQGoSbIKJyAaejJRwAQoNwEyQut6GiTeWmxrLfJnbQEg4AwUe4CZLl2/bLZOMClQsxhJZwAAg+wk2QWFmSonIhdtASDgDBR7gJAlrA4Q8t4QAQXISbIKAFHP7QEg4AwUW4CQJawOEPLeEAEFyEmyCoPHjc1DhawGMTLeEAEFyEmyBYucPcPglawGMXLeEAEDyEmwBzuQ19vLnC1NjmzUyuTcB2aAkHgOAh3ASYlfNt2qeZ/wEHe6ElHACCh3ATYFQuwAxawgEgeAg3AUTlAqygigEAgoNwE0BULsAKqhgAIDgINwFE5QKsoIoBAIKDcBMgLEmhMahiAIDAI9wECEtSaAwrVQw8Eg4A5hBuAuSLreY7gFiSgsfgrm1k8olwHgkHAJMINwGy+4ejpsaxJIXTxTkduuL8TFNjWZoCAHMINwGy+4cjpsb17ZjKkhR8WHkknJZwAGgY4SYAXG5D676vNjV2oMkNpIgdtIQDQGARbgJg+bb9qnWZ2wvBqcQ4Ey3hABBYhJsAMHu+TVK8k/02qBMt4QAQOISbJrJyvs2F7LdBPay0hFPFAAD+EW6ayMr5Nrk55k6jRewZ1KW1Ek12MVDFAAD+EW6aiBZwBEKc06ELO6aZGksVAwD4R7hpAioXEEhUMQBAYBBumoDKBQSSlSqGX875UjOXfqfakyb/BQSAGBL2cDNjxgzl5OQoKSlJeXl5KikpqXfsxo0b9bOf/Uw5OTlyOByaPn166CZaByoXEEhWqhiOu6QpC7fo3Mc+0rmPLtAv/1yiQ8dOBneCABAlwhpu5s2bpwkTJmjy5MlavXq1+vbtq2HDhqmioqLO8UeOHFHXrl01ZcoUZWVlhXi2Z6NyAYFkpYrhdLUuQx9v2afev12kLg//r65+4VN9vIkeKgCxK6zhZtq0aRo/frzGjRunXr16aebMmUpOTtbs2bPrHD9w4EA9++yzuvnmm5WYmGjqPY4fP66amhqfj0Bpn2bu8V2WpGCWlSqGuhiSvt57UL98Y4W6PbJAFz3xL728hOUrALElbOGmtrZWq1atUkFBwX8m43SqoKBAxcXFAXufwsJCpaamej+ys7MD9r3NPv3EkhTMGty1jeID+F/lgSMnNHXRqeWr8x5boIf/31c6WusK3BsAQAQKW7iprKyUy+VSZqbvbfjMzEyVlZUF7H0mTpyo6upq78euXbsC9r0Hd2ujtORmfsekJTdjSQqmxTkduq6fuSoGq46fNDR3xS6dP2mhujz8v/qvaUvZlAzAlsK+oTjYEhMTlZKS4vMRKHFOh6bc2MfvmCk39mFJCpYU3tg36O9hSPq24rB3U3Kvxz/SRO7qALCJsIWb9PR0xcXFqbzc95yY8vLyiNgsbNZPerfTzF9cpKwU3z1AWSmJmvmLi/ST3u3CNDNEq4R4p8Zd3Cmk73nkhFt//b+7Ot0f+V+NfK1Yn2xhUzKA6BQfrjdOSEjQgAEDVFRUpOuvv16S5Ha7VVRUpLvvvjtc02qUn/Rup//qlaWS7QdUcfCYMlomaVCX1tyxQaNNvqaPPt68Tzv3m3siL5BOuqXibQdUvO1UzUNWSqJuGZKjX17SVQmB3BAEAEEStnAjSRMmTNDYsWOVm5urQYMGafr06Tp8+LDGjRsnSRozZow6dOigwsJCSac2IX/99dfeX+/evVtr165VixYt1L1797Bdh3RqiSqfvTUIoE8e/LFunbNCRZvrPhohVMpqjmvKwi2asnCLEuMcuqhzK90+tJsu6dGWAA8gIjkMwwjrfeeXXnpJzz77rMrKytSvXz+9+OKLysvLkyRddtllysnJ0Zw5cyRJO3bsUJcuXc76HkOHDtXSpUtNvV9NTY1SU1NVXV0d0P03QLAcrXXptx+u1z/X7NHRk5G1TNQuJVFjuasDIASs/PwOe7gJNcINolntSbdeX7ZVf1+5S9srjyqSnnNKbubUdf06aNI1F6h5Qly4pwPAZgg3fhBuYCeeuzofrNmjYxF0VyfOIQ09t61eHHmRWiSFdfUbgE0Qbvwg3MCuPHd15izbrvJDJ8I9HR8d0pI0Kq+zfnUpy1cAGodw4wfhBrHA5Ta0bMs+vbz0G63YWR1Ry1eJ8Q7d0L+jJrN8BcACwo0fhBvEoqO1Lk3651eav3ZvRG1KdjqkvC6tefoKQIMIN34QbhDrXG5Dn26q0NR/bdLm8sOKpL8AOFMHQH0IN34QbgBfh46d1D1/WanPvtuvExG0fsWZOgBOR7jxg3AD1K/2pFuzPvtOsz7dpqqjkdUzxZk6QGwj3PhBuAHMOX35akv54YjalMyZOkDsIdz4QbgBGidST0qOd0oDc9iUDNgd4cYPwg3QdN4zdT7fofKDteGejo+05vG6qnc77uoANkO48YNwAwSW50ydVz75Vit2VskVQX+jcFIyYB+EGz8IN0BweZavFqzbq4O1kbNTxyHp/HYt9cCVPTX0PJavgGhDuPGDcAOETiSflNw6uZl+dWlXKiGAKEG48YNwA4SP56TkD1bvUQTd1FEzp5TLpmQgohFu/CDcAJEhks/U4aRkIPIQbvwg3ACRhzN1ADSEcOMH4QaIfJ5NyR99tVc1xyMn6nCmDhA+hBs/CDdAdInkok/O1AFCh3DjB+EGiG6eos9Pvt0fUWfqcFcHCC7CjR+EG8A+InX5SqLoEwg0wo0fhBvAniL5pGQ2JQNNR7jxg3ADxAbPXZ0P1uzRsQgq+nRK6slJyYBlhBs/CDdA7PEWfS7brvJDJ8I9HR+clAyYQ7jxg3ADxLbTKyFWlVYrgm7qKDHeoRv6d9Rklq+AsxBu/CDcADhdpJ6U7HRIeV14+grwINz4QbgBUJ9IPlMntXm8ruZMHcQwwo0fhBsAZnGmDhA5CDd+EG4ANIbn6auPvy7XvsMnwz0dH5ypg1hAuPGDcAMgECL1rg5n6sCuCDd+EG4ABJrnrs4/1+zR0Qh6/MrpkLq1PUc/u6gjd3UQ9Qg3fhBuAAST90ydz3eo/GBtuKfjg7s6iGaEGz8INwBC5fRKiFWlVToRQfVXcQ5p6Llt9eLIi9QiKT7c0wEaRLjxg3ADIFwi9aRkh6TzqYRAhCPc+EG4ARAJTj8pecXOakXQTR2dkxCna/q256RkRBTCjR+EGwCR6GitS5P++ZU+WL1HtRGUdDgpGZGCcOMH4QZApPNUQrxdvEN7ayJn+UqSslISdQtn6iAMCDd+EG4ARBvO1AEIN34RbgBEM87UQawi3PhBuAFgF5ypg1hCuPGDcAPAjk4/U2d1aVVEbUrmTB0EAuHGD8INgFgQqXd1HJK6Z7B8BesIN34QbgDEmtPv6qzYWRVRm5IT4hy6pHs6d3XQIMKNH4QbALHOsyn5gzV7dCyCNiVzUjL8Idz4QbgBgP/wLF/9bUWptu8/pkj6gdA6uZl+dWlX/epSlq9AuPGLcAMA9YvUk5IT4x26oX9HKiFiGOHGD8INAJjjOSl51qfbVHXUFe7peLEpOTYRbvwg3ACAdS63oU83VWjqvzZpc/nhiFq+4kyd2EC48YNwAwBN56mE+Oy7/ToRQctX8U5pYA5Fn3ZEuPGDcAMAgRWpZ+pIUruURI2l6NMWCDd+EG4AIHg4UwfBQrjxg3ADAKETqUWfnKkTfQg3fhBuACA8PMtXf1+5S9srjyqCtuoorXm8rurdjk3JEYxw4wfhBgAiQ6SelEzRZ2Qi3PhBuAGAyOPdlLxsu8oPnQj3dHx0SEvSqLzOnJQcZoQbPwg3ABDZPJuSX176jVbsrI6o5StJ8sQbQ6f27gTy15LkdEhuI3jvEcw5xTmlxHinOrc5J+D7mQg3fhBuACC6eCoh5q/dG1GbktGwZnEO/XFkf/2kd7smfy8rP78j4v7ajBkzlJOTo6SkJOXl5amkpMTv+L/97W/q2bOnkpKS1KdPHy1YsCBEMwUAhFrzhDg9+9/9tempq7X191frz6NzdX7mOeL5psh3wmXo9rdXa+GGvSF937CHm3nz5mnChAmaPHmyVq9erb59+2rYsGGqqKioc/wXX3yhkSNH6tZbb9WaNWt0/fXX6/rrr9eGDRtCPHMAQKjFOR26/IJMffSby7R9ynBt+O0w/fjcNooj6US03/7za7ncobvrFvZlqby8PA0cOFAvvfSSJMntdis7O1u//vWv9fDDD581fsSIETp8+LDmz5/vfW3w4MHq16+fZs6c2eD7sSwFAPYUqWfq4JS/jh+s/G5tGv31UbMsVVtbq1WrVqmgoMD7mtPpVEFBgYqLi+v8muLiYp/xkjRs2LB6xx8/flw1NTU+HwAA+2meEKdnftZPm566Wt88dZUe+sm56pbePPxLFJAkVRw8FrL3CusD/JWVlXK5XMrMzPR5PTMzU5s3b67za8rKyuocX1ZWVuf4wsJC/e53vwvMhAEAUSEh3qk7LuuhOy7rISlyz9SJJRktk0L2XrYPtBMnTlR1dbX3Y9euXeGeEgAgxDx3dTafdlcns2VCuKcVM7JSkjSoS+uQvV9Y79ykp6crLi5O5eXlPq+Xl5crKyurzq/JysqyND4xMVGJiYmBmTAAIOqdflcnkos+7eS31/YKaX9XWO/cJCQkaMCAASoqKvK+5na7VVRUpPz8/Dq/Jj8/32e8JC1evLje8QAA1CfO6dDQ8zM09/aLtbVwuDY98RONGNhBLRNsv7AREs3iHJr5i4sCcs6NFWEvzZgwYYLGjh2r3NxcDRo0SNOnT9fhw4c1btw4SdKYMWPUoUMHFRYWSpLuvfdeDR06VM8//7yGDx+uuXPnauXKlXrttdfCeRkAABvwLF8987N+Piclf7W7WsdPRtZpwLF2QrEVYQ83I0aM0L59+zRp0iSVlZWpX79+WrhwoXfTcGlpqZzO/yToIUOG6C9/+Ysee+wxPfLII+rRo4fef/999e7dO1yXAACwIc9dnaHnZ4R7KrAo7OfchBrn3AAAEH2i5pwbAACAQCPcAAAAWyHcAAAAWyHcAAAAWyHcAAAAWyHcAAAAWyHcAAAAWyHcAAAAWwn7CcWh5jmzsKamJswzAQAAZnl+bps5ezjmws3BgwclSdnZ2WGeCQAAsOrgwYNKTU31Oybm6hfcbrf27Nmjli1byuEIbJlXTU2NsrOztWvXrpioduB67Y3rtTeu1/7sds2GYejgwYNq3769T+dkXWLuzo3T6VTHjh2D+h4pKSm2+BfJLK7X3rhee+N67c9O19zQHRsPNhQDAABbIdwAAABbIdwEUGJioiZPnqzExMRwTyUkuF5743rtjeu1v1i8Zo+Y21AMAADsjTs3AADAVgg3AADAVgg3AADAVgg3AADAVgg3ATJjxgzl5OQoKSlJeXl5KikpCfeUGqWwsFADBw5Uy5YtlZGRoeuvv15btmzxGXPs2DHdddddatOmjVq0aKGf/exnKi8v9xlTWlqq4cOHKzk5WRkZGXrwwQd18uTJUF6KZVOmTJHD4dB9993nfc2O17p792794he/UJs2bdS8eXP16dNHK1eu9H7eMAxNmjRJ7dq1U/PmzVVQUKBvv/3W53scOHBAo0aNUkpKitLS0nTrrbfq0KFDob6UBrlcLj3++OPq0qWLmjdvrm7duunJJ5/06aaJ5uv99NNPdc0116h9+/ZyOBx6//33fT4fqGv76quvdOmllyopKUnZ2dmaOnVqsC+tTv6u98SJE3rooYfUp08fnXPOOWrfvr3GjBmjPXv2+HyPaLpeqeE/49Pdfvvtcjgcmj59us/r0XbNAWGgyebOnWskJCQYs2fPNjZu3GiMHz/eSEtLM8rLy8M9NcuGDRtm/PnPfzY2bNhgrF271rj66quNTp06GYcOHfKOuf32243s7GyjqKjIWLlypTF48GBjyJAh3s+fPHnS6N27t1FQUGCsWbPGWLBggZGenm5MnDgxHJdkSklJiZGTk2NceOGFxr333ut93W7XeuDAAaNz587GLbfcYnz55ZfGtm3bjEWLFhnfffedd8yUKVOM1NRU4/333zfWrVtnXHvttUaXLl2Mo0ePesf85Cc/Mfr27WssX77c+Oyzz4zu3bsbI0eODMcl+fX0008bbdq0MebPn29s377d+Nvf/ma0aNHCeOGFF7xjovl6FyxYYDz66KPGe++9Z0gy/vGPf/h8PhDXVl1dbWRmZhqjRo0yNmzYYPz1r381mjdvbrz66quhukwvf9dbVVVlFBQUGPPmzTM2b95sFBcXG4MGDTIGDBjg8z2i6XoNo+E/Y4/33nvP6Nu3r9G+fXvjD3/4g8/nou2aA4FwEwCDBg0y7rrrLu/vXS6X0b59e6OwsDCMswqMiooKQ5LxySefGIZx6i+QZs2aGX/729+8YzZt2mRIMoqLiw3DOPUfo9PpNMrKyrxjXnnlFSMlJcU4fvx4aC/AhIMHDxo9evQwFi9ebAwdOtQbbux4rQ899JBxySWX1Pt5t9ttZGVlGc8++6z3taqqKiMxMdH461//ahiGYXz99deGJGPFihXeMR999JHhcDiM3bt3B2/yjTB8+HDjl7/8pc9rN954ozFq1CjDMOx1vWf+4AvUtb388stGq1atfP59fuihh4zzzjsvyFfkn78f9B4lJSWGJGPnzp2GYUT39RpG/df8/fffGx06dDA2bNhgdO7c2SfcRPs1NxbLUk1UW1urVatWqaCgwPua0+lUQUGBiouLwzizwKiurpYktW7dWpK0atUqnThxwud6e/bsqU6dOnmvt7i4WH369FFmZqZ3zLBhw1RTU6ONGzeGcPbm3HXXXRo+fLjPNUn2vNZ//vOfys3N1c9//nNlZGSof//+mjVrlvfz27dvV1lZmc81p6amKi8vz+ea09LSlJub6x1TUFAgp9OpL7/8MnQXY8KQIUNUVFSkb775RpK0bt06LVu2TFdddZUk+13v6QJ1bcXFxfrRj36khIQE75hhw4Zpy5Yt+uGHH0J0NY1TXV0th8OhtLQ0Sfa8XrfbrdGjR+vBBx/UBRdccNbn7XjNZhBumqiyslIul8vnh5skZWZmqqysLEyzCgy326377rtPF198sXr37i1JKisrU0JCgvcvC4/Tr7esrKzOfx6ez0WSuXPnavXq1SosLDzrc3a7Vknatm2bXnnlFfXo0UOLFi3SHXfcoXvuuUdvvPGGpP/M2d+/z2VlZcrIyPD5fHx8vFq3bh1x1/zwww/r5ptvVs+ePdWsWTP1799f9913n0aNGiXJftd7ukBdW7T9O+5x7NgxPfTQQxo5cqS3NNKO1/vMM88oPj5e99xzT52ft+M1mxFzreAw76677tKGDRu0bNmycE8lKHbt2qV7771XixcvVlJSUrinExJut1u5ubn6/e9/L0nq37+/NmzYoJkzZ2rs2LFhnl3gvfvuu3rnnXf0l7/8RRdccIHWrl2r++67T+3bt7fl9eKUEydO6KabbpJhGHrllVfCPZ2gWbVqlV544QWtXr1aDocj3NOJKNy5aaL09HTFxcWd9QRNeXm5srKywjSrprv77rs1f/58LVmyRB07dvS+npWVpdraWlVVVfmMP/16s7Ky6vzn4flcpFi1apUqKip00UUXKT4+XvHx8frkk0/04osvKj4+XpmZmba5Vo927dqpV69ePq+df/75Ki0tlfSfOfv79zkrK0sVFRU+nz958qQOHDgQcdf84IMPeu/e9OnTR6NHj9ZvfvMb7506u13v6QJ1bdH277gn2OzcuVOLFy/23rWR7He9n332mSoqKtSpUyfv32E7d+7U/fffr5ycHEn2u2azCDdNlJCQoAEDBqioqMj7mtvtVlFRkfLz88M4s8YxDEN33323/vGPf+jjjz9Wly5dfD4/YMAANWvWzOd6t2zZotLSUu/15ufna/369T7/QXn+kjnzB2s4XXHFFVq/fr3Wrl3r/cjNzdWoUaO8v7bLtXpcfPHFZz3a/80336hz586SpC5duigrK8vnmmtqavTll1/6XHNVVZVWrVrlHfPxxx/L7XYrLy8vBFdh3pEjR+R0+v41FxcXJ7fbLcl+13u6QF1bfn6+Pv30U504ccI7ZvHixTrvvPPUqlWrEF2NOZ5g8+233+rf//632rRp4/N5u13v6NGj9dVXX/n8Hda+fXs9+OCDWrRokST7XbNp4d7RbAdz5841EhMTjTlz5hhff/21cdtttxlpaWk+T9BEizvuuMNITU01li5dauzdu9f7ceTIEe+Y22+/3ejUqZPx8ccfGytXrjTy8/ON/Px87+c9j0dfeeWVxtq1a42FCxcabdu2jdjHo093+tNShmG/ay0pKTHi4+ONp59+2vj222+Nd955x0hOTjbefvtt75gpU6YYaWlpxgcffGB89dVXxnXXXVfn48P9+/c3vvzyS2PZsmVGjx49IuLR6DONHTvW6NChg/dR8Pfee89IT083/ud//sc7Jpqv9+DBg8aaNWuMNWvWGJKMadOmGWvWrPE+HRSIa6uqqjIyMzON0aNHGxs2bDDmzp1rJCcnh+UxYX/XW1tba1x77bVGx44djbVr1/r8/XX6U0DRdL2G0fCf8ZnOfFrKMKLvmgOBcBMgf/zjH41OnToZCQkJxqBBg4zly5eHe0qNIqnOjz//+c/eMUePHjXuvPNOo1WrVkZycrJxww03GHv37vX5Pjt27DCuuuoqo3nz5kZ6erpx//33GydOnAjx1Vh3Zrix47V++OGHRu/evY3ExESjZ8+exmuvvebzebfbbTz++ONGZmamkZiYaFxxxRXGli1bfMbs37/fGDlypNGiRQsjJSXFGDdunHHw4MFQXoYpNTU1xr333mt06tTJSEpKMrp27Wo8+uijPj/sovl6lyxZUud/r2PHjjUMI3DXtm7dOuOSSy4xEhMTjQ4dOhhTpkwJ1SX68He927dvr/fvryVLlni/RzRdr2E0/Gd8prrCTbRdcyA4DOO0ozoBAACiHHtuAACArRBuAACArRBuAACArRBuAACArRBuAACArRBuAACArRBuAACArRBuAACArRBuAMScnJwcTZ8+PdzTABAkhBsAQXXLLbfo+uuvlyRddtlluu+++0L23nPmzFFaWtpZr69YsUK33XZbyOYBILTiwz0BALCqtrZWCQkJjf76tm3bBnA2ACINd24AhMQtt9yiTz75RC+88IIcDoccDod27NghSdqwYYOuuuoqtWjRQpmZmRo9erQqKyu9X3vZZZfp7rvv1n333af09HQNGzZMkjRt2jT16dNH55xzjrKzs3XnnXfq0KFDkqSlS5dq3Lhxqq6u9r7fb3/7W0lnL0uVlpbquuuuU4sWLZSSkqKbbrpJ5eXl3s//9re/Vb9+/fTWW28pJydHqampuvnmm3Xw4MHg/kMD0CiEGwAh8cILLyg/P1/jx4/X3r17tXfvXmVnZ6uqqko//vGP1b9/f61cuVILFy5UeXm5brrpJp+vf+ONN5SQkKDPP/9cM2fOlCQ5nU69+OKL2rhxo9544w19/PHH+p//+R9J0pAhQzR9+nSlpKR43++BBx44a15ut1vXXXedDhw4oE8++USLFy/Wtm3bNGLECJ9xW7du1fvvv6/58+dr/vz5+uSTTzRlypQg/dMC0BQsSwEIidTUVCUkJCg5OVlZWVne11966SX1799fv//9772vzZ49W9nZ2frmm2907rnnSpJ69OihqVOn+nzP0/fv5OTk6KmnntLtt9+ul19+WQkJCUpNTZXD4fB5vzMVFRVp/fr12r59u7KzsyVJb775pi644AKtWLFCAwcOlHQqBM2ZM0ctW7aUJI0ePVpFRUV6+umnm/YPBkDAcecGQFitW7dOS5YsUYsWLbwfPXv2lHTqbonHgAEDzvraf//737riiivUoUMHtWzZUqNHj9b+/ft15MgR0++/adMmZWdne4ONJPXq1UtpaWnatGmT97WcnBxvsJGkdu3aqaKiwtK1AggN7twACKtDhw7pmmuu0TPPPHPW59q1a+f99TnnnOPzuR07duinP/2p7rjjDj399NNq3bq1li1bpltvvVW1tbVKTk4O6DybNWvm83uHwyG32x3Q9wAQGIQbACGTkJAgl8vl89pFF12k//f//p9ycnIUH2/+r6RVq1bJ7Xbr+eefl9N56ib0u+++2+D7nen888/Xrl27tGvXLu/dm6+//lpVVVXq1auX6fkAiBwsSwEImZycHH355ZfasWOHKisr5Xa7ddddd+nAgQMaOXKkVqxYoa1bt2rRokUaN26c32DSvXt3nThxQn/84x+1bds2vfXWW96Nxqe/36FDh1RUVKTKyso6l6sKCgrUp08fjRo1SqtXr1ZJSYnGjBmjoUOHKjc3N+D/DAAEH+EGQMg88MADiouLU69evdS2bVuVlpaqffv2+vzzz+VyuXTllVeqT58+uu+++5SWlua9I1OXvn37atq0aXrmmWfUu3dvvfPOOyosLPQZM2TIEN1+++0aMWKE2rZte9aGZOnU8tIHH3ygVq1a6Uc/+pEKCgrUtWtXzZs3L+DXDyA0HIZhGOGeBAAAQKBw5wYAANgK4QYAANgK4QYAANgK4QYAANgK4QYAANgK4QYAANgK4QYAANgK4QYAANgK4QYAANgK4QYAANgK4QYAANjK/wfHPUfVEjyvSwAAAABJRU5ErkJggg==\n", |
| "text/plain": [ |
| "<Figure size 640x480 with 1 Axes>" |
| ] |
| }, |
| "metadata": {}, |
| "output_type": "display_data" |
| } |
| ], |
| "source": [ |
| "schedule = OneCycleSchedule(start_lr=0.1, max_lr=0.5, cycle_length=500, cooldown_length=750, finish_lr=0.001)\n", |
| "plot_schedule(schedule)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "0c0d2106", |
| "metadata": {}, |
| "source": [ |
| " <!--notebook-skip-line-->\n", |
| "\n", |
| "\n", |
| "### Cyclical\n", |
| "\n", |
| "Originally proposed by [Leslie N. Smith (2015)](https://arxiv.org/abs/1506.01186), the idea of cyclically increasing and decreasing the learning rate has been shown to give faster convergence and more optimal solutions. We implement a wrapper class that loops existing cycle-based schedules such as `TriangularSchedule` and `CosineAnnealingSchedule` to provide infinitely repeating schedules. We pass the schedule class (rather than an instance) because one feature of the `CyclicalSchedule` is to vary the `cycle_length` over time as seen in [Ilya Loshchilov, Frank Hutter (2016)](https://arxiv.org/abs/1608.03983) using `cycle_length_decay`. Another feature is the ability to decay the cycle magnitude over time with `cycle_magnitude_decay`." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 13, |
| "id": "69949821", |
| "metadata": {}, |
| "outputs": [], |
| "source": [ |
| "class CyclicalSchedule():\n", |
| " def __init__(self, schedule_class, cycle_length, cycle_length_decay=1, cycle_magnitude_decay=1, **kwargs):\n", |
| " \"\"\"\n", |
| " schedule_class: class of schedule, expected to take `cycle_length` argument\n", |
| " cycle_length: iterations used for initial cycle (int)\n", |
| " cycle_length_decay: factor multiplied to cycle_length each cycle (float)\n", |
| " cycle_magnitude_decay: factor multiplied learning rate magnitudes each cycle (float)\n", |
| " kwargs: passed to the schedule_class\n", |
| " \"\"\"\n", |
| " self.schedule_class = schedule_class\n", |
| " self.length = cycle_length\n", |
| " self.length_decay = cycle_length_decay\n", |
| " self.magnitude_decay = cycle_magnitude_decay\n", |
| " self.kwargs = kwargs\n", |
| "\n", |
| " def __call__(self, iteration):\n", |
| " cycle_idx = 0\n", |
| " cycle_length = self.length\n", |
| " idx = self.length\n", |
| " while idx <= iteration:\n", |
| " cycle_length = math.ceil(cycle_length * self.length_decay)\n", |
| " cycle_idx += 1\n", |
| " idx += cycle_length\n", |
| " cycle_offset = iteration - idx + cycle_length\n", |
| "\n", |
| " schedule = self.schedule_class(cycle_length=cycle_length, **self.kwargs)\n", |
| " return schedule(cycle_offset) * self.magnitude_decay**cycle_idx" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "f5479b03", |
| "metadata": {}, |
| "source": [ |
| "As an example, we implement the triangular cyclical schedule presented in [\"Cyclical Learning Rates for Training Neural Networks\" by Leslie N. Smith (2015)](https://arxiv.org/abs/1506.01186). We use slightly different terminology to the paper here because we use `cycle_length` that is twice the 'stepsize' used in the paper. We repeat cycles, each with a length of 500 iterations and lower and upper learning rate bounds of 0.5 and 2 respectively." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 14, |
| "id": "b535c2db", |
| "metadata": {}, |
| "outputs": [ |
| { |
| "data": { |
| "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABf10lEQVR4nO3deXhU1f0/8PdkJZEshJgNAgECIosY2QygjUpBsLj9HqVKQdHqF4QK4opWoFZBaMGlKlSr4r4WcUOQRpYGo8gSNSAgBAxCEgghG4EEZu7vj/QOmWSSOXfm3rnb+/U8eVomJ8m9OJx87j3v+zkOSZIkEBEREVlEiN4HQERERKQmFjdERERkKSxuiIiIyFJY3BAREZGlsLghIiIiS2FxQ0RERJbC4oaIiIgsJUzvAwg2l8uFw4cPIyYmBg6HQ+/DISIiIgGSJKGmpgZpaWkICWn73oztipvDhw8jPT1d78MgIiIiPxw8eBCdO3duc4ztipuYmBgAjX85sbGxOh8NERERiaiurkZ6err793hbbFfcyEtRsbGxLG6IiIhMRiRSwkAxERERWQqLGyIiIrIUFjdERERkKSxuiIiIyFJY3BAREZGlsLghIiIiS2FxQ0RERJbC4oaIiIgshcUNERERWYrtOhST/pwuCZv3V+BIzSkkxbTDkG4JCA3hJqYUGL6vSCt8b5mPrsXNggULsGLFCuzatQtRUVEYNmwYFi5ciPPOO6/Nr/vggw/w6KOP4sCBA+jZsycWLlyIsWPHBumoKRCrC0sw9+MdKKupd7+WHBOJv1zTF1f2S9XxyMjMVheWYN4nO1Fafcr9WofocDxxbT+MvSBNxyMjs1v1QwkeWfkjjteddr+WEhuJeVdzzjIyXZelNmzYgGnTpuGbb77B2rVrcfr0aYwaNQonTpxo9Wu+/vpr3HTTTbj99tuxfft2XHvttbj22mtRWFgYxCMnf6wuLMGUN7d5FDYAUFZTjylvbsPqwhKdjozMTH5fNS1sAOB43Wnc9fZ2LFi1U6cjI7NbsGon7np7m0dhAwCl1ZyzjM4hSZKk90HIjh49iqSkJGzYsAGXXnqp1zHjx4/HiRMn8Nlnn7lfu/jii3HhhRdi2bJlPn9GdXU14uLiUFVVxY0zg8jpktB/3hrUNThbHXNORCh+mDeat3tJmNMlYeDja1HZ7JdPcy/cfBHGXsCrbBK36ofDuOvt7W2OiY8Ox9Y//5ZzVpAo+f1tqEBxVVUVACAhIaHVMfn5+Rg5cqTHa6NHj0Z+fr7X8fX19aiurvb4oOD7em95m4UNAJxocOLrveVBOiKygm+KjvksbADgzx8XwukyzHUcGZzTJeGRlT/6HFdZdxrf7DsWhCMipQxT3LhcLsycORPDhw9Hv379Wh1XWlqK5ORkj9eSk5NRWlrqdfyCBQsQFxfn/khPT1f1uEnMP776WWjcv7f9qvGRkJXkC/5iqTjRgM37KzQ+GrKKzfsrcLzujNDY/CJekBmRYYqbadOmobCwEO+++66q33f27Nmoqqpyfxw8eFDV70++OV0Stv5yXGhs0dFajY+GrGTvkRrhsaVVJzU8ErKS5vmttvx8hHOWERniUfDp06fjs88+w8aNG9G5c+c2x6akpKCsrMzjtbKyMqSkpHgdHxkZicjISNWOlZT7pugYnIIrArvKauF0SVzDJp+cLgnrdh8RHl9eW+97EBGAvJ/F31cb9xzlnGVAut65kSQJ06dPx0cffYSvvvoK3bp18/k12dnZyM3N9Xht7dq1yM7O1uowKUBf7xO/bdtwxsU1bBLyTdEx1J8Rz9FU1DVoeDRkFU6XhC8KvcccvDl5mnOWEela3EybNg1vvvkm3n77bcTExKC0tBSlpaU4efLs7eNJkyZh9uzZ7j/PmDEDq1evxuLFi7Fr1y7MmzcPW7ZswfTp0/U4BRLwncKsA9ewSYRo3ka29YDY0ijZ2+b9FahrcCn6Gs5ZxqNrcbN06VJUVVUhJycHqamp7o/33nvPPaa4uBglJWd7CQwbNgxvv/02XnzxRQwYMAAffvghVq5c2WYImfTjdEkoOFip6Gu4hk0ilORtAOCHX6v4xBT5pCRvI+Pbynh0zdyItNhZv359i9duuOEG3HDDDRocEantm6JjaBAN3PwP17DJF6V5GwA49b8lz+E9EzU6KrKC8hrl2azSKuUFEWnLME9LkTUpydvIuIZNvijN28g27TuqwdGQlVTUKS9uvtxZyruCBsPihjSlNG8j4xo2tUVp3ka2hbkb8sGfbFZtvZN9lAyGxQ1pxp+8jYy5G2qLBP+ukpm7obYEMmexj5KxsLghzfiTt5HJuRsib0oq6/z6ulNsNUBtCGTOyuPWMYbC4oY040/eRsbcDbXG6ZKwekeZ74GtYO6GWhPInLVmB3M3RsLihjRz6Hhgt2mZuyFv/OlD0hRzN9QafzOCAHM3RsPihjRzskFs47nW8CKIvPGnD0lTzN2QN4HkbWTM3RgHixvShNMlYf2ewG7/s3cEeaNk3x9vmLshbwLJ28iYuzEOFjekCX/7kDTF3hHUnNJ9f1rD3A01F0jeRsbcjXGwuCFN+NuHpCmuYVNzgeZtZIcreVeQPAWSt5FxzjIOFjekCaX7/rSGa9jUVKB5G9nJ005Vvg9Zgxp5GxnnLGNgcUOq82ffn9ZwDZuaCjRvI2MfJWpKjbyNrLxW+fYNpD4WN6Q6NfI2Mq5hk0ytvA3APkrkSY28jayirkG170X+Y3FDqlMjbyPjGjbJ1MrbyNhHiWSB9uRqyp+9qUh9LG5IdWrlbWRcwyZAvbyNjPuXkSzQnlxNsY+SMbC4IVUpyds4BL8nczcEAOU1YlkG0UmNuRsC1OnJ1RT7KBkDixtSlZK8Tb9OsULjmLshAKioEytu+gq+r5i7IUDZnBUqeEXGPkr6Y3FDqlKSt8k8t73QOOZuCBDPMsS2Cxf+nszdkJI5q2dSjNA47l+mPxY3pCoJYldA7SNDcf1FnYW/L3M39qakD0n/zrGIjhCb2pi7IdGMYPvIUFzeO0loLHM3+mNxQ6oqqawTGjeqTwqGZSYiMkzsPi97R9ibkj4kl2QmYUy/FKGxzN3Ym5KM4Oi+KRjeM1FoLHM3+mNxQ6pxuiSs3lEmNDYlrh1CQxzI6XWu0Hj2jrA30T4k7cJCcHGPjhjRU+wKm7kbe1OStxmRmYiLu3dEhGDwhrkbfbG4IdUo6UMS8r/5ISoiTGg8e0fYm+i+Pxd0jkNoiAMpse2EvzdzN/alJG+TEheF0BAHLkyPFxrP3I2+WNyQapT0Icnu3nh7t1OHKKHxXMO2LyV5m0EZHQAAQ7olCOdu+LayLyV5myHdEgAAg//3v75wztIXixtSjWgfkqjwxqUDABjWg2vY1DYleZvhPRqXOUNDHLiyb7LQ18RFiT9dRdbhdEnI2yu2dDS6bwpC/3e7mXOWObC4IdVs/UVs6eDSXue6J4qLu3dEZJjY25DLB/YkunQg521kqfHRQl+3/SCXD+xo8/4K1NaLLaOPyDxb0HDOMgcWN6SKxqcOxK6Ceiad7W8TGuLAZeeJhYr52K49iS4d5PQ+WzQDgEOwB/b6XXxiyo6ULKOnxJ1dPlcyZ/FtpR8WN6SKxqcOxK6C5LyNbGBXsTVsPrZrP0oe1R3YpYPHn7Ob3MVpC5cP7El0GT22XZg7byPLavZea01plbr7oZE4FjekCqWP6jaVGBMp9LV8bNd+lDyqm9je833Ex3apLaLbeWT36OhxRxAAqk+dFvraL3dy6xi9sLghVSh9VLcpPrZLrVH6qG5TfGyX2iLaXiIqPLTFa6JLntw6Rj8sbihg/jyq25SSx3aZu7EXJdt5NF86APjYLnmnZM5Ki2958SW65Alw6xi9sLihgPnzqG5ToSEOtssnr5Rs59H8jiDAx3bJu0DnrMYnpsTu3uTt5d1mPbC4oYAFkreRsV0+Nad0Ow9vmLshbwKdsxqfmBKbs9bsYO5GDyxuKGCHjovddvWWt5Exd0PN+bOdR3PM3ZA3gWQEZZlJMULfg7kbfbC4oYCdbDgjNG6gl7yNjO3yqTl/tvPwhrkbairQjKCMuRtj07W42bhxI8aNG4e0tDQ4HA6sXLnS59e89dZbGDBgAKKjo5GamorbbrsNx45xmUIvTpeE9XvEbucnREe0+jkl7fLZO8Ie8n4W62/TdDsPb5i7oaYCzdvImLsxNl2LmxMnTmDAgAF4/vnnhcZv2rQJkyZNwu23344dO3bggw8+wObNm3HHHXdofKTUmkD6kDQn2i6fvSOsz+mS8EVhqdDYptt5eMPcDTWlRkYQYO7G6HQtbsaMGYPHH38c1113ndD4/Px8ZGRk4O6770a3bt0wYsQI/N///R82b96s8ZFSawLpQ9Ice0eQTEnepul2Ht4oyd0cruRdQatTI28jY+7GuEyVucnOzsbBgwexatUqSJKEsrIyfPjhhxg7dmyrX1NfX4/q6mqPD1KP6L4/rfUhaYpr2CRTK28jayvv1dTJ007hn0vmo1beRsY5y7hMVdwMHz4cb731FsaPH4+IiAikpKQgLi6uzWWtBQsWIC4uzv2Rnp4exCO2NiX7/ozu670PSVNcwyaZWnkbWUK02BYf7KNkbWrlbWRK5qzyWrHtHkgdpipudu7ciRkzZmDOnDnYunUrVq9ejQMHDmDKlCmtfs3s2bNRVVXl/jh48GAQj9jalORtRmT6vrrmGjYB6uZtZNy/jAD18jay0BAHcnqJ7RBeUdcgNI7UEab3ASixYMECDB8+HPfffz8A4IILLsA555yDSy65BI8//jhSU1NbfE1kZCQiI8UmNlJGzbyNLDMpBhBo3CavYSu5LUzmoGbeRqa0j9LwnmJPWJG5qNGTq7moCLFfo6J7WZE6THXnpq6uDiEhnoccGtq4qZkk8So+2NTM28i4hk1q520A7l9GjdToydVcpw5iF27soxRcuhY3tbW1KCgoQEFBAQBg//79KCgoQHFxMYDGJaVJkya5x48bNw4rVqzA0qVLUVRUhE2bNuHuu+/GkCFDkJaWpscp2JbaeRsZczdUXiOWTRDN2wDcv4zU68nVHPsoGZOuxc2WLVuQlZWFrKwsAMCsWbOQlZWFOXPmAABKSkrchQ4A3HrrrViyZAmee+459OvXDzfccAPOO+88rFixQpfjtzO18zYy5m6ook6suBHN28i4f5m9qdmTqyn2UTImXTM3OTk5bS4nLV++vMVrf/rTn/CnP/1Jw6MiEVrkbWTM3dibaDYhKjxU0fdl7sbetJqz5D5KmwXet9y/LHhMlbkh45AgdgWkJG8jY+7GvpT0IUmLFy9WAOZu7E6LjKCM+5cZD4sb8ktJZZ3QuFF9xPM2MvaOsC+1+5A0xdyNfWmVEZQxd2M8LG5IMadLwmqBZSMASIlTdnUNsHeEnandh6Q55m7sSauMoIy5G+NhcUOKKelDovACyI29I+xJzX1/vFGauyFr0DIjCCjbv4y5m+BgcUOKadGHpDn2jrAftff98UZJ7oZvK+vQMm8jY+7GWFjckGJq7/vjDdew7UfLvI0sNMSBK/smC40treIO4Vagdd5GxjnLWFjckCJa7PvjDdew7UfrvI0sNT5aaNyXO9lHyQq0ztvIOGcZC4sbUkSLfX+84Rq2/Widt5E5IPa1ch8lMjet8zYyJXPW4UreFdQaixtSJBh5GxnXsO0jGHkbGfso2YuWPbmaE92T6uRpZ0A/h3xjcUOKBCNvI+Matn0EI28j4/5l9qJlT67mEqLFtm1gHyXtsbghYcHK28i4hm0fwcrbANy/zE607snVXGKMWHHDPkraY3FDwoKVt5Exd2Mfh46LLf8EmreRZSbFCI1j7sbcgtGTqyn2UTIOFjckLJh5GxlzN/ZwsuGM0DjRTIMvzN3YQ7DnLO5fZhwsbkhYeY3YPk5q5G1kzN1Yn9MlYf0esWXFhOgIVX4mczf2EMyMIMD9y4yExQ0Jq6gTK27UyNvImLuxPiV9SBLbi2UafGHuxvqCnRGUcf8yY2BxQ8JE93GKCg9V7Weyd4T1BasPSXPM3VhbsDOCMuZujIHFDQlR0ockLT7wpw6aYu8IawvGvj/eMHdjbXpkBAHmboyCxQ0JCWYfkubYO8K6grXvjzdKcjfltWJLsmQcwc7byJi7MQYWNyQkmH1ImmPvCOsK1r4/3oSGOJDTS6wQr6hrUPVnk7b0ytvImLvRH4sbEhKsfX+84Rq2demVt5FFRYQJjRPNm5Ex6JW3kXHO0h+LG/IpmPv+eMM1bOvSK28j69RBrGBiHyVz0StvI+OcpT8WN+STnnkbQNkadv6+Y/wlZBJOl4S8vWKP76udt5Gxj5I16dGTqynOWfpjcUM+iS4daJG3kYmuYVefOsPHdk1i8/4K1NaLLR2onbeRNYaKxaZBLh+Yx3HBjJQWeRsZ5yx9sbghn0SXDnJ6azdRKFnD5mO75qBk6UCLvA0gN/MTu9vI5QPzKDoq9t8qU4O8jYxzlr5Y3FCblDyqO7CL+nkb2ZBuCWgfKfZ2Zbt8cxBdOohtF6ZJ3kY2sKvY9+Zju+agZM6KjwrX7DiUzFlsNaA+FjfUJj1a43sTGuLAiEyxK2y2yzcH0e08snt01OyOIMBWA1ZjpDlruGCmi60G1Mfihtqk96O6TbFdvrXosZ2HN3xs11qMNGex1YB+WNxQm/R+VLcptsu3Dj2382iOj+1ai5HmLLYa0A+LG2qVnq3xvWG7fOvQu71AU2yXbx1Gm7PYakA/LG6oVXq2xveG7fKtQ8/tPLxhu3xrMNqcdXH3jogIFSugNu0T6/lEYljcUKuMtHYt4xq2Nei5nYc3zN1Yg9HmrNAQBy5Mjxcau4VzlqpY3FCrJIhdAQVj7VrGNWzz03s7D2+Yu7EGI+VtZIMFfw7nLHWxuKFWlVTWCY0b1Uf7tWsZ17DNz0h5GxlzN+ZntLyNjHOWPljckFdOl4TVO8qExqbEafs0S1NcwzY/o+VtZMzdmJvR8jYyzln60LW42bhxI8aNG4e0tDQ4HA6sXLnS59fU19fjkUceQdeuXREZGYmMjAy88sor2h+szWzeX4G6BrF9f4J0AQSAa9hWYLS8jYy5G3MzWt5GxjlLH7oWNydOnMCAAQPw/PPPC3/NjTfeiNzcXLz88svYvXs33nnnHZx33nkaHqU9Kdn3J7t78K6CAK5hm5kR8zYyJbkbvq2Mx4gZQRnnrOATe/REI2PGjMGYMWOEx69evRobNmxAUVEREhIa3ywZGRkaHZ295f0stnYdFR7cpQOgcQ37+XX7fI6T17CH9wxu8UWtM2LeRhYa4sCVfZOxYnuJz7GlVeLFPwWHETOCMs5ZwWeqzM0nn3yCQYMGYdGiRejUqRN69eqF++67DydPtt6Ntr6+HtXV1R4f1DanS8IXhaVCYy/tpd1O4K3hGrZ5GTVvI0uNjxYa9+VO7l9mJEbNCMo4ZwWfqYqboqIi5OXlobCwEB999BGefvppfPjhh7jrrrta/ZoFCxYgLi7O/ZGenh7EIzYnJXmbnkntNT6alriGbV5GzdvIHBD7mdy/zFiMmhGUKZmzDlfyrqAaTFXcuFwuOBwOvPXWWxgyZAjGjh2LJUuW4LXXXmv17s3s2bNRVVXl/jh48GCQj9p8jJy3kXEN23yMnLeRcf8yczLDnDVQ8D198rRT4yOxB1MVN6mpqejUqRPi4uLcr51//vmQJAm//vqr16+JjIxEbGysxwe1zch5Gxl7R5iPkfM2MiX7l+Xt5RNTRmGGOSshOlJoHPsoqcNUxc3w4cNx+PBh1Nae7RC6Z88ehISEoHPnzjoemXUYPW8j4xq2+Rg9bwM0Lh9cdp5Yv5s1O5i7MQKzzFmJMWLFDfsoqUPX4qa2thYFBQUoKCgAAOzfvx8FBQUoLi4G0LikNGnSJPf4m2++GR07dsTkyZOxc+dObNy4Effffz9uu+02REUFr2+BlRk9byNj7sZ8Dh0XW8bRK28jy0yKERrH3I0xmGXOYh+l4NK1uNmyZQuysrKQlZUFAJg1axaysrIwZ84cAEBJSYm70AGA9u3bY+3ataisrMSgQYMwYcIEjBs3Ds8++6wux29FZli7ljF3Yy4nG84IjRPNJmiFuRtzMcucxf3LgkvXPjc5OTmQpNZ/6SxfvrzFa71798batWs1PCp7K6+pFxqn59q1jL0jzMPpkrB+j9jyYEJ0hMZH0zY5dyPSyj9vbzmuu4hL4noyQ94GOLt/2b+3HfY5Vs7d6HkH0+xMlbkh7VXUiRU3eq5dy5i7MQ8l+/4kthfLJmiFuRvzMEveRsb9y4KHxQ152CqYT4kKD9X4SHxj7wjzMOq+P61h7sYczJK3kTF3EzwsbshNSR+StPjgd/n0hr0jzGHvkRqhcXrs++MNczfmYJa8jYy5m+BhcUNuZuhD0hx7Rxif0yVh3W6xXMTovsHf98cbJf1uymvFlnJJfWbJ28jk3I0IzlmBYXFDbmboQ9Ice0cYn5K8zYhM/a+ugcZfQjm9xAr4iroGjY+GvDFb3kbG3E1wsLghN6Pv++MN17CNz2x5G1lUhNjDpKI5NVKX2fI2Ms5ZwcHihgCYY98fb7iGbXxmy9vIOnUQK7TYR0kfZsvbyJTMWXxb+Y/FDQEwZ94G4Bq20ZkxbyPj/mXGZqaeXE2FhjhwZd9kobGlVXzK018sbgiAOfM2Mq5hG5cZ8zYy9lEyNjP15GouNT5aaNyXO9lHyV8sbgiAOfM2Mq5hG5dZ8zYA9y8zOjP15GrOAbE5lH2U/Mfihkybt5Exd2NcEsSuOo2Wt5Fx/zJjMmNPrqbYR0l7LG7ItHkbGXM3xlVSWSc0blQfY+VtZMzdGJPZ5ywlfZTy9vJusz9Y3JCp8zYy5m6Mx+mSsHpHmdDYlDjjXV0DzN0YldnnLO5fpj0WN4RDx8VuexoxbyNj7sZ4lPQhMejbirkbgzJzRlDG/cu0xeKGcLLhjNA40X2c9MDeEcZj1j4kzTF3YyxmzwjKmLvRFosbm3O6JGwSvMWbEB2h8dH4T0nviLiocI2PhgDz9iFpjrkbYzF73kbG/cu0xeLG5jbvr0BtvdjSQWJ7sX2c9CLaO2L7QS4fBMPWX8RupRuxD0lTjb+ExKZKLnlqT7S9gFHzNjIluZutxZyzlGJxY3NKlg6M1oekOdHeEet38YkprTV2JhYL2Bpp3x9vGn8Jid0B4NtKe6LbeeT0NnbRDIjnbjhnKcfixubyfhZrjR/bLsyQfUiaEl3D5vKB9ho7E4vdETRy3kaW1UUsu8F2+dpSsp3HQMH/ZnrinKUdFjc25nRJ+KKwVGhsdo+Ohr8K4mO7xmH2R3Wbqz51Wmgc2+VrS8l2HkZfRgc4Z2mJxY2NKXlU1+hLBwAf2zUSKzyq2xTb5RuDmbfz8EbJnHW4kncFlWBxY2NWeVS3KT62qz+rPKrbFB/bNQazb+fhjWiLjZOnnRofibWwuLEx0byN0R/VbYqP7erPKo/qNsV2+cZg9u08vEmIFls+49YxyrC4sSkleRujP6rbFNew9We1vA3AdvlGYIXtPLxJjBErbrh1jDIBFTenTnEN0KyslreRMXejPyts5+EN2+XrywrbeXjDrWO0obi4cblc+Otf/4pOnTqhffv2KCoqAgA8+uijePnll1U/QNKGFfM2MuZu9GWF7Ty8Ye5GX1ads5RsHfPzkVqNj8Y6FBc3jz/+OJYvX45FixYhIuJsO/5+/frhX//6l6oHR9qxSmt8b5i70Y/TJWH9HrHlPiNv5+ENczf6smJGEGi82zymX4rQWOZuxCkubl5//XW8+OKLmDBhAkJDQ92vDxgwALt27VL14Eg7FXVixY2Z8jYy5m70Y7U+JE0xd6Mfq2YEZSN6ir2vmLsRp7i4OXToEDIzM1u87nK5cPq0WKMr0t9WwbxJVHio70EGw94R+rFaH5LmmLvRh1UzgjLmbtSnuLjp06cP/vvf/7Z4/cMPP0RWVpYqB0XaUtKHJC3ePE8dNMXeEfoQ3ffHTH1ImmLuRh9WzdvImLtRX5jSL5gzZw5uueUWHDp0CC6XCytWrMDu3bvx+uuv47PPPtPiGEllVuxD0pzS3hFmu41tREr2/Rnd1zx9SJqSczciS2/ltWJLv+SbVfM2Mjl38+9th32O5ZwlRvGdm2uuuQaffvop/vOf/+Ccc87BnDlz8NNPP+HTTz/Fb3/7Wy2OkVRmxT4kzbF3RPApyduMyDTf1TXQ+Esop5dYwV9R16Dx0diD1fM2MuZu1KX4zg0AXHLJJVi7dq3ax0JBYrV9f7xRuoY9vKc5f9kaidXzNrKoCLFpUzTXRm2zet5GxjlLXYrv3HTv3h3HjrWcxCorK9G9e3dVDoq0Y8V9f7zhGnbwWT1vI+vUQawwYx8ldVg9byNTMmfxbeWb4uLmwIEDcDpbhjDr6+tx6NAhRd9r48aNGDduHNLS0uBwOLBy5Urhr920aRPCwsJw4YUXKvqZdmeHvA3A3hHBZoe8jYx9lILLyj25mgoNceDKvslCY0ur+JSnL8LLUp988on7/69ZswZxcXHuPzudTuTm5iIjI0PRDz9x4gQGDBiA2267Dddff73w11VWVmLSpEm44oorUFYmttcINbJD3kY2omeSUEBPXsPmbV7/2SFvI5P7KIlcJGzad5TvqwBZuSdXc6nx0ULjvtxZylCxD8LFzbXXXgsAcDgcuOWWWzw+Fx4ejoyMDCxevFjRDx8zZgzGjBmj6GsAYMqUKbj55psRGhrq825PfX096uvP/uOorq5W/POsxA55GxnXsIPHLnkb4Gwfpc0CmRruXxY4K/fkas4BsTlX7qOkpDWB3QgvS7lcLrhcLnTp0gVHjhxx/9nlcqG+vh67d+/G7373Oy2PFQDw6quvoqioCHPnzhUav2DBAsTFxbk/0tPTNT5C47JL3kbG3E3wSBC7a2P2vI2M+5cFhx16cjXFPkrqUZy52b9/PxIT9bnC/fnnn/HQQw/hzTffRFiY2E2n2bNno6qqyv1x8OBBjY/SuOySt5ExdxM8JZV1QuNG9TF33kbG3E1w2G3O4v5l6vHrUfATJ05gw4YNKC4uRkODZy+Hu+++W5UDa87pdOLmm2/GX/7yF/Tq1Uv46yIjIxEZaa49bLRip7yNjLkb7TldElbvEMu+pcSZ/+oaYO4mWOw2Z8n7l4n8e1qzoxSLmLtpleLiZvv27Rg7dizq6upw4sQJJCQkoLy8HNHR0UhKStKsuKmpqcGWLVuwfft2TJ8+HUDjUpkkSQgLC8OXX36Jyy+/XJOfbRWHjovdxrRC3kbG3I32lPQhscjbirmbILFTRlCWmRQDCBQ3zN20TfGy1D333INx48bh+PHjiIqKwjfffINffvkFAwcOxN///nctjhEAEBsbix9//BEFBQXujylTpuC8885DQUEBhg4dqtnPtoqTDWeExonuy2QG7B2hPbv0IWmOuRtt2S0jKGPuRh2Ki5uCggLce++9CAkJQWhoKOrr65Geno5Fixbh4YcfVvS9amtr3YUK0JjnKSgoQHFxMYDGvMykSZMaDzQkBP369fP4SEpKQrt27dCvXz+cc845Sk/FVpwuCev3HBUamxAdofHRBA97R2jP6vv+tIa5G23ZLW8jY+5GHYqLm/DwcISENH5ZUlKSuxCJi4tTHNbdsmULsrKy3LuJz5o1C1lZWZgzZw4AoKSkxP39KTBK+pAktrdWRklp7wgSZ5d9f7yRczciNu0Tu7Cgs+yWt5HJuRsRa3ZwzmqN4sxNVlYWvvvuO/Ts2RO/+c1vMGfOHJSXl+ONN95Av379FH2vnJwcSFLr/2GWL1/e5tfPmzcP8+bNU/Qz7cpOfUiaY+8I7dhl3x9vlORuDlfyrqBSdszbyJi7CZziOzfz589HamoqAOCJJ55Ahw4dMHXqVBw9ehT//Oc/VT9AUodd9v3xhmvY2rFr3kYmmk87ebrlljXUOrvmbWScswKn+M7NoEGD3P8/KSkJq1evVvWASH122vfHG3kNW2RZLm9vOa67qHMQjsoa7Jq3kSVEiy3hyn2UrPZvSyt2zdvIlMxZ5bVi21PYjeI7N63Ztm1bUDoUk3J22vfHG65ha8POeRtZYoxYcSP3USIxds3byEJDHMjpJVa0VdQ1+B5kQ4qKmzVr1uC+++7Dww8/jKKiIgDArl27cO2112Lw4MFwucTW3im47Jy3kWUmxQiNk9ewyTc7521kSvsokRg79uRqLipCbGFFdO8tuxEubl5++WWMGTMGy5cvx8KFC3HxxRfjzTffRHZ2NlJSUlBYWIhVq1ZpeazkJzvnbWRcw1af3fM2APcv04ode3I116mD2IUm+yh5J1zcPPPMM1i4cCHKy8vx/vvvo7y8HC+88AJ+/PFHLFu2DOeff76Wx0l+snveRsbeEeorrxFb67dq3gbg/mVasGtPrubYRykwwsXNvn37cMMNNwAArr/+eoSFheFvf/sbOndm+NLI7J63kTF3o76KOrHixqp5G9mInmLvK+ZuxNi5J1dT7KMUGOHi5uTJk4iObmyG5nA4EBkZ6X4knIyLeZuzmLtRl+haf1R4qMZHoi/mbtTFOauR3EdJBPcva0nRo+D/+te/0L59YzDwzJkzWL58ORITPa/2tdo4k/wjQewKyMp5G1l2j454bt1eobHM3bRNSR+StHhr7ATeGjl3IxKuZu7GN2YEzxrcLUGoSaScu7HyHVKlhIubLl264KWXXnL/OSUlBW+88YbHGIfDweLGYEoq64TGjepj3byNjL0j1GP3PiRNybmbf2877HMs+920jRlBT8N6JOL5dft8jpNzN8N7WjdaoJRwcXPgwAEND4O04HRJWC3QwhsAUuKsfXUNnO0dsWan78mTvSPaZvc+JM2N6JkkVNzIuRv+EvKOGUFPcu5G5EJi076jfF81oVoTPzIeJX1ILH4B5MbeEeqw874/3jB3ow7mbTwxd+M/FjcWxj4kLbF3RODsvu+PN0r63fBt1TrmbVoaLHienLM8sbixMPYhaYm9IwLHvE1LoSEOXNk3WWhsXFS4xkdjTk6XhLy9Yo802yFvI+Oc5R8WNxa29RexpQOr9yFpqjFULPa25/KBd6JLB3bJ28hS46OFxm0/yOUDbzbvr0Btvdgyuh3yNjLOWf5hcWNRjU8diF0FWXXfH28am/mJ3U3gY7veiS4d5PS2T9EMAA6Inev6XexU7I2SZXQ75G1kSuYsvq3OUlzcVFdXe/2oqalBQwOfMDGKxqcOxK6C7JK3kQ3sKraGzXb5LSl5VHdgF3vkbWSi+5dx+cA70WX02HZhtsnbyLIE/y2VVokXiFanuLiJj49Hhw4dWnzEx8cjKioKXbt2xdy5c7lDuM74qG7rEmPEWrazXX5LbI3fOrbLD4zodh7ZPTra6o4gAFSfOi007sud3DpGpri4Wb58OdLS0vDwww9j5cqVWLlyJR5++GF06tQJS5cuxZ133olnn30WTz75pBbHS4L4qG7r+Niu//iobuv42G5guJ1H60SXPLl1zFmKtl8AgNdeew2LFy/GjTfe6H5t3Lhx6N+/P/75z38iNzcXXbp0wRNPPIGHH35Y1YMlMXxUt21sl+8/bufRNrbL9w+382gbt45RTvGdm6+//hpZWVktXs/KykJ+fj4AYMSIESguLg786MgvfFS3bXK7fBHM3Xjidh5t42O7/uGc1TZ56xgReXt5txnwo7hJT0/Hyy+/3OL1l19+Genp6QCAY8eOoUMH+90RMArmbXwb0TNJaBxzN2dxOw/fmLvxD+estjU+MSU2Z63ZwdwN4Mey1N///nfccMMN+OKLLzB48GAAwJYtW7Br1y58+OGHAIDvvvsO48ePV/dISdih42K3Je2Yt5Epzd1wzxZu5yFCzt2ILE0xd3MWM4K+ZSbFAAIXF3LuRvTpPatSfOfm6quvxq5duzBmzBhUVFSgoqICY8aMwa5du/C73/0OADB16lQsWbJE9YMlMScbzgiNG2jDvI2M7fKV43YeYtguXxlmBMUoKVaYu/Hjzg0AdOvWjU9DGZTTJWH9HrHb3QnRERofjXHJ7fJXbC/xOZa9Ixrl/SzW38ZO23l4M6xHIp5ft8/nODl3Y/e7gszbiJFzNyKtGPL2luO6izoH4aiMy6/iprKyEps3b8aRI0da9LOZNGmSKgdG/mEfEnGi7fLl3hF2vR0ONBbNXxSWCo2103Ye3si5G5Ff2Jv2HbV9ccO8jRg5dyOSe1uzoxSLbD5nKS5uPv30U0yYMAG1tbWIjY2Fw3H2L8/hcLC40Rn7kIhT2jvCzmvYSvI2dtrOwxsluZvDlbwryLyNOOZuxCnO3Nx777247bbbUFtbi8rKShw/ftz9UVHB5kF6E933x659SJriGrY45m2UEc2znTzt1PhIjI15G2U4Z4lTXNwcOnQId999N6KjxW7pU/Ao2fdndF979iFpir0jxDFvo0xCtNiSr937KDFvo4ySOau8Vmw7C6tSXNyMHj0aW7Zs0eJYKEBK8jYjMnl1zd4RYpi3UY77l4lh3kaZ0BAHcnqJFXkVdfbeyFpx5uaqq67C/fffj507d6J///4IDw/3+PzVV1+t2sGRMszbKMc1bN+Yt1GOfZTEsCeXclERYr+2RffqsirFxc0dd9wBAHjsscdafM7hcMDptPcasp6Yt1GOe7b4xryNcty/TAx7cinXqYPYhand9y9TvCzlcrla/WBhox/mbfzD3I1v5TVia/fM25zF/ct8Y08u/3D/MjGKixsyJuZt/MPcjW8VdWLFDfM2nrh/WdvYk8s/3L9MjNCy1LPPPos777wT7dq1w7PPPtvm2Lvvvlv4h2/cuBF/+9vfsHXrVpSUlOCjjz7Ctdde2+r4FStWYOnSpSgoKEB9fT369u2LefPmYfTo0cI/06qYt/EfczdtE127jwoP1fhIzIW5m7ZxzvIP9y8TI1TcPPXUU5gwYQLatWuHp556qtVxDodDUXFz4sQJDBgwALfddhuuv/56n+M3btyI3/72t5g/fz7i4+Px6quvYty4cfj222+RlZUl/HOtSILYFRDzNi0xd9M6JX1I0uLtuRN4a5i7aRszgv4b3C1BqLixc+5GqLjZv3+/1/8fqDFjxmDMmDHC459++mmPP8+fPx8ff/wxPv3001aLm/r6etTXn72tXl1d7dexGl1JZZ3QuFF9mLdpTsmeLXbrHcE+JP6Tczf/3nbY51g5d2OXf5vMCAaG+5f5ZurMjcvlQk1NDRISWq/qFyxYgLi4OPdHenp6EI8wOJwuSWi/EQBIiePVdXPsHdE69iEJDHM33jEjGBjmbnxT/Ci40+nE8uXLkZub63XjzK+++kq1g/Pl73//O2pra3HjjTe2Omb27NmYNWuW+8/V1dWWK3CU9CHhBZB37B3hHff9CQxzN94xbxMY5m58U1zczJgxA8uXL8dVV12Ffv36eWycGUxvv/02/vKXv+Djjz9GUlLrV0eRkZGIjLR20p59SALH3hEtcd+fwCnJ3djpQTzmbQLH3E3bFBc37777Lt5//32MHTtWi+MRPoY//vGP+OCDDzBy5EjdjsMouO9P4LiG3RLzNoELDXHgyr7JWLG9xOfY0ip77BDOvI06OGe1TXHmJiIiApmZmVoci5B33nkHkydPxjvvvIOrrrpKt+MwCu77ow6uYbfEvI06UuPFNhn+cqc9+igxb6MOzlltU1zc3HvvvXjmmWcgSYH/I6ytrUVBQQEKCgoAND6JVVBQgOLiYgCNeZlJkya5x7/99tuYNGkSFi9ejKFDh6K0tBSlpaWoqqoK+FjMivv+qENewxZhlzVs5m3U4YDY343cR8nqmLdRh5I563ClPe4KNqV4WSovLw/r1q3DF198gb59+7bYOHPFihXC32vLli247LLL3H+Wg7+33HILli9fjpKSEnehAwAvvvgizpw5g2nTpmHatGnu1+XxdsS8jXq4hn0W8zbqYR8lT+zJpZ6BGR2E5qyTp+23NZLi4iY+Ph7XXXedKj88JyenzTtAzQuW9evXq/JzrYR5G/VwDfss5m3Uo6SPUt7eclx3UecgHJV+2JNLPQnRYg/L2K2PEqCwuDlz5gwuu+wyjBo1CikpYpvCkXaYt1GXvIYt8kt9076jli5umLdRj7x/mUgvqjU7SrHIwr+E2JNLXYkxYsWN3EfJynNWc4oyN2FhYZgyZYpHx1/SD/M26mLu5qxDx8WWR5i3EZOZFCM0zuq5G/bkUpfSPkp2ojhQPGTIEGzfvl2LYyGFmLdR32DBNX45d2NVJxvOCI0byLyNECWbrVo5d8M5S11yHyURdtu/THFxc9ddd+Hee+/Fc889h/z8fPzwww8eHxQ85TVid9CYtxE3rIfYhCrnbqzI6ZKwfo/Yo6MJ0REaH401yLkbEXl7rXuFzYyguuT9y0TIuRu7UBwo/v3vfw8AHrt/OxwOSJIEh8MBp9N+qWy9VNSJFTfM24hj7kZZH5LE9tbu/q0W5m6YEdTKiJ5JQpuz2i13o7i4UXNXcAqM6D5HUeGhGh+JdSjZs8WqvSPYh0QbmUkxgEBxI+dulCxlmQEzgtrg/mXeKS5uunbtqsVxkEJK+pCkxfOpAyXs3juC+/5ow+79bpi30YaS/cvslLtRXNzIdu7cieLiYjQ0NHi8fvXVVwd8UOQb+5Box869I7jvj3aU9Lspr7XeE6nM22hDzt2ILE1Zcc5qjeLipqioCNdddx1+/PFHd9YGgHt3cGZugoN9SLRj594R3PdHO6EhDuT0Ohdrdvr+JV9R1+BzjJkwb6Mt5m5aUvy01IwZM9CtWzccOXIE0dHR2LFjBzZu3IhBgwaxg3AQcd8f7di5dwTzNtqKihC7nhTN05kF8zbasvOc1RrFxU1+fj4ee+wxJCYmIiQkBCEhIRgxYgQWLFjg8QQVaYf7/mjLzr0jmLfRVqcOYgWh1fooMW+jLSVzloXeVm1SXNw4nU7ExDR220xMTMThw423wrp27Yrdu3ere3TkFfM22lLSOyJ/3zHL/BJyuiTk7RXrb8O8jX/s2keJPbm0FRriwJV9k4XGxkWF+x5kAYqLm379+uH7778HAAwdOhSLFi3Cpk2b8Nhjj6F79+6qHyC1JLp0wLyN/0b0TBIaV33qjGXa5W/eX4HaerGlA+Zt/NMYKhabdq20fHBcMEPEvI3/UuOjhcZtP2itJc/WKC5u/vznP8PlapwAH3vsMezfvx+XXHIJVq1ahWeffVb1A6SWRJcOcnpzovCXkjVsqzy2q2TpgHkb/zQ28xO7m2qlJc+io2Lnksm8jd8cEJvr1++yR6dixcXN6NGjcf311wMAMjMzsWvXLpSXl+PIkSO4/PLLVT9A8qTkUd2BXZi38deQbgloHyn2z8Mq7fJFlw5i24UxbxOAgV3F/u6s0i5fyZwVb5MlEy2INn202pJnaxQXN7K9e/dizZo1OHnyJBISONEFC1vjB0doiAMjMsWusNfsKLXELyHR7Tyye3TkHcEAKG01YHacs4JD3jpGxKZ9Ytk6M1Nc3Bw7dgxXXHEFevXqhbFjx6KkpAQAcPvtt+Pee+9V/QDJEx/VDZ7MpBihcXK7fLPjdh7BYbfHdjlnBYe8dYyILRZrNeCN4uLmnnvuQXh4OIqLixEdfTbANH78eKxevVrVg6OWJIhdAfFR3cAp2dvH7LkbbucRPHZrNcD2AsEzWPDvz2qtBrxRXNx8+eWXWLhwITp37uzxes+ePfHLL7+odmDkXUllndC4UX34qG6g5Hb5IszeLp/tBYJHSasBs+duuJ1HcNm11YA3ioubEydOeNyxkVVUVCAykuulWnK6JKwW2FUYAFLieHUdKLldvgizt8vndh7BJdpqwOy5G27nEVzM3ZyluLi55JJL8Prrr7v/7HA44HK5sGjRIlx22WWqHhx5UtLCnBdA6rBLu3xu5xFcdsndMG8TXMzdnKV448xFixbhiiuuwJYtW9DQ0IAHHngAO3bsQEVFBTZt2qTFMdL/sIV58Cltl2/GX/zcziP45NyNyMWKiVelmLfRweBuCdgsULiYec4S4VeH4j179mDEiBG45pprcOLECVx//fXYvn07evToocUx0v/k/Sy2ds0W5uqxwxo28zbBp6RdfmmV+EWNkTBvow87zFkiFN+5AYC4uDg88sgjHq/9+uuvuPPOO/Hiiy+qcmDkyemS8EVhqdBYtjBXj7yGLfLLf9O+oxje03x3zJi30Ydou/wvd5aa8gqbeRt92GHOEuF3E7/mjh07hpdfflmtb0fNKMnb9GQLc9XYYQ2beRt9iLbLN2sfJeZt9KFkzjpcac67giJUK25IW8zb6MfKvSOYt9GP1fsosSeXfgYK/ls9edqp8ZHoh8WNSTBvox8rr2Ezb6MfJX2UzLh/GXty6SchWqwti9n7KLWFxY0JMG+jLyv3jmDeRj+NO4SL9bsx2/5l7MmlL7vtX+aNcKBY3gm8NZWVlYEeC7WCeRt9yWvYIo9Xmi13c+i42HIH8zbayEyKAQSKADl3o2QpS0/syaUvpX2UrBgqFi5u4uLifH5+0qRJAR8QtcS8jf6s2jviZMMZoXGia/ikTHaPjnhu3V6hsWbK3XDO0peSPkpW2L/MG+Hi5tVXX9XyOKgN5TVi+xYxb6OdYT0S8fy6fT7HybkbM1wJOV0S1u8RW0ZLiI7Q+GjsSc7diDwynbe3HNdd1NnnOCNgRlBf8v5l/9522OdYOXdjlgsyUczcmEBFnVhxw7yNdqyYu1HShySxPfeN04IVczfMCBqDXfYvaw2LGxMQ3bcoKjxU4yOxLyv2jmAfEmPITIoRGmeWfjfMCBqDXfYva42uxc3GjRsxbtw4pKWlweFwYOXKlT6/Zv369bjooosQGRmJzMxMLF++XPPj1JOSPiRp8XzqQEtW6x3BfX+MwWr9bpi3MQY5dyPCirkbXYubEydOYMCAAXj++eeFxu/fvx9XXXUVLrvsMhQUFGDmzJn44x//iDVr1mh8pPphHxLjsFLvCO77YxxK+t2U14otUeuJeRtjkHM3IswwZynl195SahkzZgzGjBkjPH7ZsmXo1q0bFi9eDAA4//zzkZeXh6eeegqjR4/W6jB1xT4kxqG0d4SRQ8Xc98c4QkMcyOl1Ltbs9F0UVNQ1BOGI/Me8jbGM6JkkFCo2w5yllKkyN/n5+Rg5cqTHa6NHj0Z+fn6rX1NfX4/q6mqPDzPhvj/GYaU1bOZtjCUqQuw6UzR/pxfmbYzFSnOWUqYqbkpLS5GcnOzxWnJyMqqrq3HypPe16AULFiAuLs79kZ6eHoxDVQX3/TEWK61hM29jLJ06iBWQRt+/jHkbY1EyZxn4beUXUxU3/pg9ezaqqqrcHwcPHtT7kIQxb2MsVlnDZt7GeKyyfxl7chlLaIgDV/ZN9j0QQGmVOZ7yFGWq4iYlJQVlZZ6tysvKyhAbG4uoKO9XPpGRkYiNjfX4MAvmbYzHCr0jmLcxHqv0UWJPLuNJjY8WGvflTnP0URJlquImOzsbubm5Hq+tXbsW2dnZOh2Rtpi3MR4rrGEzb2M8SvooGXn/MvbkMh4HxH43mKWPkihdi5va2loUFBSgoKAAQOOj3gUFBSguLgbQuKTUdL+qKVOmoKioCA888AB27dqFF154Ae+//z7uuecePQ5fU8zbGJMVcjcSxK7OmLcJrsGCf9dGzd2wJ5cxWa2Pkihdi5stW7YgKysLWVlZAIBZs2YhKysLc+bMAQCUlJS4Cx0A6NatGz7//HOsXbsWAwYMwOLFi/Gvf/3Lko+BM29jTFbI3ZRU1gmNG9WHeZtgMnvuhnOWMSnpo5S315h3m/2ha5+bnJwcSFLr/xi8dR/OycnB9u3bNTwqY2DexrjM3DvC6ZKwekeZ74EAUuJ4dR1Mcu5GpEDYtO+ood5XAOcso5L3LxP5d79mRykWWWQTTVNlbuzk0HGx24PM2wSfmXM3SvqQ8G0VXGbP3TAjaFxW279MBIsbgzrZcEZonOh+R6QeM/eOYB8SYzNr7oYZQWOzY+6GxY0BOV0S1u8Re9wzITpC46Oh5szcO4L7/hibWXM3zNsYmx1zNyxuDEhJH5LE9mL7HZG6zNg7gvv+GJ9Z+90wb2Nscu5GxJodxpmzAsHixoDYh8T4zNg7gvv+GJ+S3M3hSuPcFWTexvjslrthcWNA3PfH+My4hs28jTmI5uhOnnZqfCRimLcxBzPOWYFgcWMw3PfHHMy4hs28jTkkRIstNRuljxLzNuZgxjkrECxuDIb7/piD2dawmbcxj8QYseLGKPuXMW9jDmabswLF4sZgmLcxDzOtYTNvYx5m66PEnlzmYaY5K1AsbgyGeRvzMNMaNvM25mG2/cvYk8s8zDRnBYrFjYE4XRLy9oo93sm8jf6UrGGX19ZrfDQ+fn6N2M9n3kZ/SvYvy993TNflA6dLwibBZSn25NKfmeasQLG4MZDN+ytQWy+2dMC8jf6UrGFvLda3Xf7xugahcczbGMOInmLvq+pTZ3RdPlAyZ7Enl/6UzFmVJ09rfDTaYnFjIEqWDpi3MQbRNez1u/R9sqXoqNjyRSbzNoagJHej5/IB5yzz6X6u2L/xvQZY8gwEixsDEV06iG0XxryNQYiuYevZLl9Je4H4qHCNj4ZEDOmWgPaRYtOzno/tirYX4JxlHB0ElweN0mrAXyxuDKSiTqy4ye7RkUsHBmGGdvnczsN8QkMcGJEp1hNGr8d2lbQX4JxlHGZrNeAvFjcGsvWAWC4jKjxU4yMhUWZol8/2AuZk9Md22V7AnMzWasBfLG4MQkkL87R48Tcnac/o7fLZXsCcjP7YLtsLmJPZWg34i8WNQbCFuXkZuV0+t/MwL6O3y+d2HuakpNWAmXM3LG4Mgi3MzcvIa9jczsO8jNwun9t5mJtoqwEz525Y3BjEd4Jr5mxhbjxGXsNm3sbcjJq7Yd7G3Iw8Z6mFxY0BKMnbDGILc8Mx8ho28zbmZtTcDfM25mbkOUstLG4MgHkbczPqGjbzNuZn1Hb53M7D3Iw6Z6mJxY0BMG9jfkZcw2bexvxCQxzI6SV2QVMhuMWGGkR7cjFvY1xGnLPUxOLGAJi3MT8jrmEzb2MNURFhQuNE+2SpgT25zM+Ic5aaWNzojHkbazDiGjbzNtbQqYNY4fnDr1VBWT5gTy5rUDJnmXBVisWN3pi3sQajrWEzb2Mdw3qILRkGa/8yzlnWEBriwJV9k4XGllbp0109ECxudMa8jXUYaQ2beRvrMNr+ZZyzrCM1Plpo3Jc79dm/LBAsbnTGvI11GGkNm3kb61Cyf9mWIORuOGdZhwNi/3302r8sECxudMS8jbUYKXcjQewqi3kbcxgs+N9I69wN5yxrMWofJTWwuNER166txUi5m5LKOqFxo/owb2MGRsndcM6yFqPvXxYIFjc64tq19Rghd+N0SVi9o0xobEocn2YxA6PkbjhnWYuR9y8LFIsbHR06Lnabj2vX5mGE3I2SfX/4tjIHo+RumLexHqPuXxYoFjc6OtlwRmjcQK5dm4YRekdw3x9r0jt3w7yNNVk1d8PiRidOl4T1e8RuHydER2h8NKQWI/SOyPtZrL8N9/0xF71zN8zbWJNVczeGKG6ef/55ZGRkoF27dhg6dCg2b97c5vinn34a5513HqKiopCeno577rkHp06Zq8mQkj4kie0jNT4aUpOevSOcLglfFJYKjeW+P+aid+6GeRtrsmruRvfi5r333sOsWbMwd+5cbNu2DQMGDMDo0aNx5Ij3q8+3334bDz30EObOnYuffvoJL7/8Mt577z08/PDDQT7ywLAPiXXp2TtCSd6mZ1J7VX82aUtJ7uZwpfoXe8zbWJcVcze6FzdLlizBHXfcgcmTJ6NPnz5YtmwZoqOj8corr3gd//XXX2P48OG4+eabkZGRgVGjRuGmm27yebfHaLjvj3XpuYbNvI21iebvTp52qvpzmbexNivmbnQtbhoaGrB161aMHDnS/VpISAhGjhyJ/Px8r18zbNgwbN261V3MFBUVYdWqVRg7dqzX8fX19aiurvb40Bv3/bE2PdewmbextoRosSVqtfsoMW9jbUrmrPLaeo2PRh26Fjfl5eVwOp1ITvYMYCYnJ6O01Htu4Oabb8Zjjz2GESNGIDw8HD169EBOTk6ry1ILFixAXFyc+yM9PV3181CK+/5Ym15r2MzbWF9ijFhxo3YfJeZtrC00xIGcXmJFaUVdg8ZHow7dl6WUWr9+PebPn48XXngB27Ztw4oVK/D555/jr3/9q9fxs2fPRlVVlfvj4MGDQT7ilpi3sT491rCZt7E+vfoosSeX9UVFhAmN2xqE/cvUIHY2GklMTERoaCjKyjy7qZaVlSElxXsb+0cffRQTJ07EH//4RwBA//79ceLECdx555145JFHEBLiWa9FRkYiMtJYTxsxb2N92T064rl1e4XGqrWGzbyN9cl9lESKWDX3L2NPLuvr1EHsQlruo2T0IlbXOzcREREYOHAgcnNz3a+5XC7k5uYiOzvb69fU1dW1KGBCQ0MBAJJk/EfUmLexBz1yN+U1YmvhzNuYlx77l7Enlz3o3UdJbbovS82aNQsvvfQSXnvtNfz000+YOnUqTpw4gcmTJwMAJk2ahNmzZ7vHjxs3DkuXLsW7776L/fv3Y+3atXj00Ucxbtw4d5FjZMzb2IMeuZuKOrHihnkbcwv2/mXsyWUPevdRUpuuy1IAMH78eBw9ehRz5sxBaWkpLrzwQqxevdodMi4uLva4U/PnP/8ZDocDf/7zn3Ho0CGce+65GDduHJ544gm9TkER5m3sIzMpBhDYwFLO3Sh5HNMb0bXwqHDjXwRQ65Tmbob3DOwiiXOWPch9lDYLzCNa7l+mFt2LGwCYPn06pk+f7vVz69ev9/hzWFgY5s6di7lz5wbhyNQnQewKiHkb8wtm7kZJH5K0eO4EbmbBzt0wI2gfg7slCBU3Zsjd6L4sZTcllXVC40b1Yd7G7ILZO4J9SOwjmLkbZgTtxUq5GxY3QeR0SVgtsEwBAClxvLo2u2D2jmAfEnsJVu6GGUF7sVLuhsVNECnpQ8ILIGsIVu8I7vtjL8Hqd8O8jb0o2b/M6LkbFjdBxD4k9qO0d4Q/uO+P/ci5GxGBPIjHvI39DBb87xjInBUMLG6CiH1I7CcYa9jM29hPaIgDV/ZN9j0QQFxUuF8/w+mSkLdXbOmBeRvrsEruhsVNEG39RWzpgH1IrKMxVCz2z8zf5QPRpQPmbawlNT5aaNz2g/4tH2zeX4HaerFldOZtrCMYc1YwsLgJksanDsSugrjvj3U0NvMTu1vi72O7oksHOb1ZNFuJA2L/Ldfv8u+JKSXL6MzbWIeSOcvAq1IsboKl8akDsasg5m2sZWBXsTVsfx7bVfKo7sAuzNtYiWjTR3+XD0SX0WPbhTFvYzFZgnNFaZV4ARxsLG6ChI/q2ldijFhLen8e22VrfPvS+rFd0e08snt05B1Bi6k+dVpo3Jc71dk6RgssboKEj+ral5aP7fJRXfvS+rFdbudhX6JLnvLWMUbE4iYI+KiuvSl5bFdp7obbedibVo/tcjsPe1Oyz12gW8dohcVNEPBRXXvTsl0+t/OwN60e2+WcZW9Kto7J22vMJ6ZY3AQB8zakRbt8budBWuVuOGfZW+MTU2Jz1podxszdsLgJgkPHxW7bMW9jXVrkbridB2mVu2FGkDKTYoTGGTV3w+ImCE42nBEaN5B5G8vSol0+t/MgQP3cDTOCBJg/d8PiRmNOl4T1e8RuBydER2h8NKQXJe3yRXtH5P0s1t+G23lYm9q5G+ZtCDB/7obFjcbYh4Rkou3yRXpHOF0SvigsFfp+3M7D2tTO3TBvQ4D5czcsbjTGPiQkU7N3hJK8DbfzsDYluZvDlb7vCjJvQzIz525Y3GhMdN8f9iGxPjXXsJm3oaZE83onTzvb/DzzNtSUmXM3LG40pGTfn9F92YfE6tRcw2behppKiBZb0vbVR4l5G2pKyZxVXiu2XUewsLjRkJK8zYhMXl1bnVpr2MzbUHNq7V/GvA01FRriQE4vsSK2oq5B46NRhsWNhpi3oebUWMNm3oaaU6uPEntyUXNREWFC40T3IgsWFjcaYt6GmlNjDZt5G2pOrf3L2JOLmuvUQezCW+n+ZVpjcaMR5m3IGzVyN+U1YmvbzNvYhxr7l7EnF3mj1f5lWmNxoxHmbcgbNXI3FXVixQ3zNvYS6P5l7MlF3mi1f5nWWNxohHkbak2guRvRte2o8FBFx0XmFmjuhnMWeaPV/mVaY3GjEQliV0DM29hPILkbJX1I0uK5E7idBJq7YUaQWqP2/mXBwOJGIyWVdULjRvVh3sZuAukdwT4k1JpAcjfMCFJbzJi7YXGjAadLwuodZUJjU+J4dW03gfSOYB8Saou/uRtmBKktZszdsLjRgJI+JLwAsid/e0dw3x9qi7+5G+ZtqC1mzN2wuNEA+5CQL/70juC+P+SLktxN02gE8zbki9lyNyxuNMB9f8gXf9awmbchX0JDHLiyb7LQ2NKqxosw5m1IhNlyNyxuVMZ9f0iEP2vYzNuQiNT4aKFxX+5s7KPEvA2JMFvuhsWNyrjvD4nwZw2beRsS4YDYf3u5jxLzNiRCyZx1uFI8mqEVQxQ3zz//PDIyMtCuXTsMHToUmzdvbnN8ZWUlpk2bhtTUVERGRqJXr15YtWpVkI62bczbkCgla9gNZ1zM25AQpX2U2JOLRInuKXbytFPjI/FN9+Lmvffew6xZszB37lxs27YNAwYMwOjRo3HkiPc14IaGBvz2t7/FgQMH8OGHH2L37t146aWX0KlTpyAfuXfM25AoJWvYy/P2M29DQpTuX8aeXCQqIVps243W9i8LJrHnUTW0ZMkS3HHHHZg8eTIAYNmyZfj888/xyiuv4KGHHmox/pVXXkFFRQW+/vprhIeHAwAyMjKCecitYt6GlJDXsEWKltU7S4S+J/M2JO9fJtJra3VhCSTB30HsyUWJMWLFjdxHaXhP/VYndL1z09DQgK1bt2LkyJHu10JCQjBy5Ejk5+d7/ZpPPvkE2dnZmDZtGpKTk9GvXz/Mnz8fTqf322D19fWorq72+NAK8zakhJI17F/Kxa6umbchQHz/shMNLtSdZk8uEhPo/mXBpGtxU15eDqfTieRkz0cXk5OTUVrq/Q5IUVERPvzwQzidTqxatQqPPvooFi9ejMcff9zr+AULFiAuLs79kZ6ervp5yJi3IaVEczfH6k4LjRNdEydrU5K7Ef6enLNsL9D9y4JJ98yNUi6XC0lJSXjxxRcxcOBAjB8/Ho888giWLVvmdfzs2bNRVVXl/jh48KBmx1ZeU+97EJi3obNEczeiEqIjVP1+ZE5KcjciOGcRENj+ZcGma3GTmJiI0NBQlJV5rg2XlZUhJcX7X2Bqaip69eqF0NBQ92vnn38+SktL0dDQ0GJ8ZGQkYmNjPT60UlEnVtwwb0MyJb0jRCS2F1sTJ2uTczdq4ZxFMn/3Lws2XYubiIgIDBw4ELm5ue7XXC4XcnNzkZ2d7fVrhg8fjr1798LlOrtOvGfPHqSmpiIiQt+r1ub7ALUmKjzU9yCyBSW5GxHsQ0Iy0dyNCGYESWaW3I3uy1KzZs3CSy+9hNdeew0//fQTpk6dihMnTrifnpo0aRJmz57tHj916lRUVFRgxowZ2LNnDz7//HPMnz8f06ZN0+sUACjb9yctnk8d0Flq5WTYh4SaUjN3w7wNycySu9H9UfDx48fj6NGjmDNnDkpLS3HhhRdi9erV7pBxcXExQkLO/kWmp6djzZo1uOeee3DBBRegU6dOmDFjBh588EG9TgEA9/0h/4n2jvCF+/5QU3LuRnRrhdYwb0NNybmbf2877HOsnLvRY17SvbgBgOnTp2P69OleP7d+/foWr2VnZ+Obb77R+KiU4b4/5C/R3hG+cN8faio0xIGcXudizU6xxqKtYd6GmhvRM0mouNGz343uy1JWwX1/yF9K1rDb/D7M21AzURGBX78yb0PNmSF3w+JGBUryNtz3h5pTsobdGuZtyJtOHQIveJm3oeaUzFl6PQ3O4kYFzNtQIJT0jmgN8zbkTaB9lJi3IW9CQxy4sm+y74EA4qLCNT4a71jcqCBf8Fl+5m2oNaK9I1r9euZtyIvGULH/0zzzNtSa1PhooXHbD4q1SFEbixsV7D1SIzQupzcnCvIu0NwN8zbkTWMzP//vFmcyb0OtcEDsd9n6Xfp0KmZxEyCnS8K63WJPIwzswrwNeTekWwLaR/r3zzG2XRjzNtSqgV39f2/E67SkQMYn2kfp1Bl9OhWzuAnQN0XHhPtIsDU+tSY0xIERmf5dYWf36Mg7gtSqQFoNcM6i1ijZOmbTvqMaH01LLG4CJJq3Abh0QG3zt10+t/OgtgSy5Mk5i1qjZOuYLYJbE6mJxU2AJIjdteGjuuSLv+3yuZ0HtcXfVgOcs8iXwYLvjx9+rQp67obFTYBKKuuExo3qw0d1qW1yu3yl2F6A2uJvqwG2FyBfRFsN6JG7YXETAKdLwuodZUJjU+J4dU1tk9vlK8H2AiTCn1YDbC9Avhg5d8PiJgCb91egrsElNJYXQCRCabt8budBIvzJ3TBvQ74YOXfD4iYApdWnhMeyhTmJUNoun9t5kAiluRvmbUiUUXM3LG4CUFFbLzSOLcxJlNJ2+czbkAgl7fIB5m1InFFzNyxuApBwToTQuN8PTudEQUIu7t4RYYLvlbAQB4tmEibaLh9g3obEKdniI5g7hLO4CYDomvSovqkaHwlZRWiIAxd1iRcae1GXeBbNJEy0XT7AvA2JCw1x4PLeooH14M1XLG4CMKRbAlJ9PAWVGteOa9ekyJ8u76nqOCJAvI9SwjkRnLNIkT9c3FVonL+9vPzB4iYAoSEOzB3Xp9Va1AFg7rg+vLomRYZlJiI6ou2uw+dEhGIYlw5IgYu7d0R8tO+9oh6/ph/nLFJE5L3VITocF3dncWMaV/ZLxdI/XNTiDk5qXDss/cNFuLIfl6RImdAQB5bcOKDNMYtvHMBfQKRIaIgDT17fv80x/3dpN4y9gHMWKSPy3lpwff+gzlkOSZKCvxe5jqqrqxEXF4eqqirExsaq9n2dLgmb91fgSM0pJMU0LkXxlw8FYnVhCeZ+vANlNWefykuJjcS8q/uyaCa/rS4swbxPdnq0skg4JxyPX9MPYy9I0/HIyOy8vbdS49ph7rg+qsxZSn5/s7ghMjAWzaQFvq9IK1q+t5T8/lbWDpWIgio0xBHUEB7ZA99XpBWjvLeYuSEiIiJLYXFDRERElsLihoiIiCyFxQ0RERFZCosbIiIishQWN0RERGQpLG6IiIjIUljcEBERkaWwuCEiIiJLsV2HYnm3ierqap2PhIiIiETJv7dFdo2yXXFTU1MDAEhPT9f5SIiIiEipmpoaxMXFtTnGdhtnulwuHD58GDExMXA41N0orrq6Gunp6Th48KAtNuXk+Vobz9fa7Ha+gP3O2WrnK0kSampqkJaWhpCQtlM1trtzExISgs6dO2v6M2JjYy3xRhLF87U2nq+12e18Afuds5XO19cdGxkDxURERGQpLG6IiIjIUljcqCgyMhJz585FZGSk3ocSFDxfa+P5Wpvdzhew3znb7Xybsl2gmIiIiKyNd26IiIjIUljcEBERkaWwuCEiIiJLYXFDRERElsLiRiXPP/88MjIy0K5dOwwdOhSbN2/W+5D8smDBAgwePBgxMTFISkrCtddei927d3uMOXXqFKZNm4aOHTuiffv2+H//7/+hrKzMY0xxcTGuuuoqREdHIykpCffffz/OnDkTzFPxy5NPPgmHw4GZM2e6X7Pa+R46dAh/+MMf0LFjR0RFRaF///7YsmWL+/OSJGHOnDlITU1FVFQURo4ciZ9//tnje1RUVGDChAmIjY1FfHw8br/9dtTW1gb7VHxyOp149NFH0a1bN0RFRaFHjx7461//6rE3jZnPd+PGjRg3bhzS0tLgcDiwcuVKj8+rdW4//PADLrnkErRr1w7p6elYtGiR1qfWqrbO+fTp03jwwQfRv39/nHPOOUhLS8OkSZNw+PBhj+9hpnP29d+4qSlTpsDhcODpp5/2eN1M56saiQL27rvvShEREdIrr7wi7dixQ7rjjjuk+Ph4qaysTO9DU2z06NHSq6++KhUWFkoFBQXS2LFjpS5duki1tbXuMVOmTJHS09Ol3NxcacuWLdLFF18sDRs2zP35M2fOSP369ZNGjhwpbd++XVq1apWUmJgozZ49W49TErZ582YpIyNDuuCCC6QZM2a4X7fS+VZUVEhdu3aVbr31Vunbb7+VioqKpDVr1kh79+51j3nyySeluLg4aeXKldL3338vXX311VK3bt2kkydPusdceeWV0oABA6RvvvlG+u9//ytlZmZKN910kx6n1KYnnnhC6tixo/TZZ59J+/fvlz744AOpffv20jPPPOMeY+bzXbVqlfTII49IK1askABIH330kcfn1Ti3qqoqKTk5WZowYYJUWFgovfPOO1JUVJT0z3/+M1in6aGtc66srJRGjhwpvffee9KuXbuk/Px8aciQIdLAgQM9voeZztnXf2PZihUrpAEDBkhpaWnSU0895fE5M52vWljcqGDIkCHStGnT3H92Op1SWlqatGDBAh2PSh1HjhyRAEgbNmyQJKlx8ggPD5c++OAD95iffvpJAiDl5+dLktT4jzEkJEQqLS11j1m6dKkUGxsr1dfXB/cEBNXU1Eg9e/aU1q5dK/3mN79xFzdWO98HH3xQGjFiRKufd7lcUkpKivS3v/3N/VplZaUUGRkpvfPOO5IkSdLOnTslANJ3333nHvPFF19IDodDOnTokHYH74errrpKuu222zxeu/7666UJEyZIkmSt823+i0+tc3vhhRekDh06eLyXH3zwQem8887T+Ix8a+uXvWzz5s0SAOmXX36RJMnc59za+f76669Sp06dpMLCQqlr164exY2ZzzcQXJYKUENDA7Zu3YqRI0e6XwsJCcHIkSORn5+v45Gpo6qqCgCQkJAAANi6dStOnz7tcb69e/dGly5d3Oebn5+P/v37Izk52T1m9OjRqK6uxo4dO4J49OKmTZuGq666yuO8AOud7yeffIJBgwbhhhtuQFJSErKysvDSSy+5P79//36UlpZ6nG9cXByGDh3qcb7x8fEYNGiQe8zIkSMREhKCb7/9NngnI2DYsGHIzc3Fnj17AADff/898vLyMGbMGADWO9+m1Dq3/Px8XHrppYiIiHCPGT16NHbv3o3jx48H6Wz8V1VVBYfDgfj4eADWO2eXy4WJEyfi/vvvR9++fVt83mrnK4rFTYDKy8vhdDo9frEBQHJyMkpLS3U6KnW4XC7MnDkTw4cPR79+/QAApaWliIiIcE8UsqbnW1pa6vXvQ/6c0bz77rvYtm0bFixY0OJzVjvfoqIiLF26FD179sSaNWswdepU3H333XjttdcAnD3ett7PpaWlSEpK8vh8WFgYEhISDHe+Dz30EH7/+9+jd+/eCA8PR1ZWFmbOnIkJEyYAsN75NqXWuZnp/d3cqVOn8OCDD+Kmm25ybxxptXNeuHAhwsLCcPfdd3v9vNXOV5TtdgUncdOmTUNhYSHy8vL0PhTNHDx4EDNmzMDatWvRrl07vQ9Hcy6XC4MGDcL8+fMBAFlZWSgsLMSyZctwyy236Hx06nv//ffx1ltv4e2330bfvn1RUFCAmTNnIi0tzZLnS2edPn0aN954IyRJwtKlS/U+HE1s3boVzzzzDLZt2waHw6H34RgK79wEKDExEaGhoS2enikrK0NKSopORxW46dOn47PPPsO6devQuXNn9+spKSloaGhAZWWlx/im55uSkuL170P+nJFs3boVR44cwUUXXYSwsDCEhYVhw4YNePbZZxEWFobk5GRLnW9qair69Onj8dr555+P4uJiAGePt633c0pKCo4cOeLx+TNnzqCiosJw53v//fe77970798fEydOxD333OO+S2e1821KrXMz0/tbJhc2v/zyC9auXeu+awNY65z/+9//4siRI+jSpYt7/vrll19w7733IiMjA4C1zlcJFjcBioiIwMCBA5Gbm+t+zeVyITc3F9nZ2ToemX8kScL06dPx0Ucf4auvvkK3bt08Pj9w4ECEh4d7nO/u3btRXFzsPt/s7Gz8+OOPHv+g5Amm+S9WvV1xxRX48ccfUVBQ4P4YNGgQJkyY4P7/Vjrf4cOHt3i0f8+ePejatSsAoFu3bkhJSfE43+rqanz77bce51tZWYmtW7e6x3z11VdwuVwYOnRoEM5CXF1dHUJCPKe50NBQuFwuANY736bUOrfs7Gxs3LgRp0+fdo9Zu3YtzjvvPHTo0CFIZyNOLmx+/vln/Oc//0HHjh09Pm+lc544cSJ++OEHj/krLS0N999/P9asWQPAWueriN6JZit49913pcjISGn58uXSzp07pTvvvFOKj4/3eHrGLKZOnSrFxcVJ69evl0pKStwfdXV17jFTpkyRunTpIn311VfSli1bpOzsbCk7O9v9efnR6FGjRkkFBQXS6tWrpXPPPdeQj0Z70/RpKUmy1vlu3rxZCgsLk5544gnp559/lt566y0pOjpaevPNN91jnnzySSk+Pl76+OOPpR9++EG65pprvD4+nJWVJX377bdSXl6e1LNnT0M8Gt3cLbfcInXq1Mn9KPiKFSukxMRE6YEHHnCPMfP51tTUSNu3b5e2b98uAZCWLFkibd++3f1kkBrnVllZKSUnJ0sTJ06UCgsLpXfffVeKjo7W7THhts65oaFBuvrqq6XOnTtLBQUFHnNY0yeBzHTOvv4bN9f8aSlJMtf5qoXFjUr+8Y9/SF26dJEiIiKkIUOGSN98843eh+QXAF4/Xn31VfeYkydPSnfddZfUoUMHKTo6WrruuuukkpISj+9z4MABacyYMVJUVJSUmJgo3XvvvdLp06eDfDb+aV7cWO18P/30U6lfv35SZGSk1Lt3b+nFF1/0+LzL5ZIeffRRKTk5WYqMjJSuuOIKaffu3R5jjh07Jt10001S+/btpdjYWGny5MlSTU1NME9DSHV1tTRjxgypS5cuUrt27aTu3btLjzzyiMcvOjOf77p167z+e73lllskSVLv3L7//ntpxIgRUmRkpNSpUyfpySefDNYpttDWOe/fv7/VOWzdunXu72Gmc/b137g5b8WNmc5XLQ5JatKqk4iIiMjkmLkhIiIiS2FxQ0RERJbC4oaIiIgshcUNERERWQqLGyIiIrIUFjdERERkKSxuiIiIyFJY3BAREZGlsLghItvJyMjA008/rfdhEJFGWNwQkaZuvfVWXHvttQCAnJwczJw5M2g/e/ny5YiPj2/x+nfffYc777wzaMdBRMEVpvcBEBEp1dDQgIiICL+//txzz1XxaIjIaHjnhoiC4tZbb8WGDRvwzDPPwOFwwOFw4MCBAwCAwsJCjBkzBu3bt0dycjImTpyI8vJy99fm5ORg+vTpmDlzJhITEzF69GgAwJIlS9C/f3+cc845SE9Px1133YXa2loAwPr16zF58mRUVVW5f968efMAtFyWKi4uxjXXXIP27dsjNjYWN954I8rKytyfnzdvHi688EK88cYbyMjIQFxcHH7/+9+jpqZG2780IvILixsiCopnnnkG2dnZuOOOO1BSUoKSkhKkp6ejsrISl19+ObKysrBlyxasXr0aZWVluPHGGz2+/rXXXkNERAQ2bdqEZcuWAQBCQkLw7LPPYseOHXjttdfw1Vdf4YEHHgAADBs2DE8//TRiY2PdP+++++5rcVwulwvXXHMNKioqsGHDBqxduxZFRUUYP368x7h9+/Zh5cqV+Oyzz/DZZ59hw4YNePLJJzX62yKiQHBZioiCIi4uDhEREYiOjkZKSor79eeeew5ZWVmYP3+++7VXXnkF6enp2LNnD3r16gUA6NmzJxYtWuTxPZvmdzIyMvD4449jypQpeOGFFxAREYG4uDg4HA6Pn9dcbm4ufvzxR+zfvx/p6ekAgNdffx19+/bFd999h8GDBwNoLIKWL1+OmJgYAMDEiRORm5uLJ554IrC/GCJSHe/cEJGuvv/+e6xbtw7t27d3f/Tu3RtA490S2cCBA1t87X/+8x9cccUV6NSpE2JiYjBx4kQcO3YMdXV1wj//p59+Qnp6uruwAYA+ffogPj4eP/30k/u1jIwMd2EDAKmpqThy5IiicyWi4OCdGyLSVW1tLcaNG4eFCxe2+Fxqaqr7/59zzjkenztw4AB+97vfYerUqXjiiSeQkJCAvLw83H777WhoaEB0dLSqxxkeHu7xZ4fDAZfLperPICJ1sLghoqCJiIiA0+n0eO2iiy7Cv//9b2RkZCAsTHxK2rp1K1wuFxYvXoyQkMab0O+//77Pn9fc+eefj4MHD+LgwYPuuzc7d+5EZWUl+vTpI3w8RGQcXJYioqDJyMjAt99+iwMHDqC8vBwulwvTpk1DRUUFbrrpJnz33XfYt28f1qxZg8mTJ7dZmGRmZuL06dP4xz/+gaKiIrzxxhvuoHHTn1dbW4vc3FyUl5d7Xa4aOXIk+vfvjwkTJmDbtm3YvHkzJk2ahN/85jcYNGiQ6n8HRKQ9FjdEFDT33XcfQkND0adPH5x77rkoLi5GWloaNm3aBKfTiVGjRqF///6YOXMm4uPj3XdkvBkwYACWLFmChQsXol+/fnjrrbewYMECjzHDhg3DlClTMH78eJx77rktAslA4/LSxx9/jA4dOuDSSy/FyJEj0b17d7z33nuqnz8RBYdDkiRJ74MgIiIiUgvv3BAREZGlsLghIiIiS2FxQ0RERJbC4oaIiIgshcUNERERWQqLGyIiIrIUFjdERERkKSxuiIiIyFJY3BAREZGlsLghIiIiS2FxQ0RERJby/wGWhm8VbfSOJgAAAABJRU5ErkJggg==\n", |
| "text/plain": [ |
| "<Figure size 640x480 with 1 Axes>" |
| ] |
| }, |
| "metadata": {}, |
| "output_type": "display_data" |
| } |
| ], |
| "source": [ |
| "schedule = CyclicalSchedule(TriangularSchedule, min_lr=0.5, max_lr=2, cycle_length=500)\n", |
| "plot_schedule(schedule)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "a5ddbfe1", |
| "metadata": {}, |
| "source": [ |
| " <!--notebook-skip-line-->\n", |
| "\n", |
| "\n", |
| "And lastly, we implement the scheduled used in [\"SGDR: Stochastic Gradient Descent with Warm Restarts\" by Ilya Loshchilov, Frank Hutter (2016)](https://arxiv.org/abs/1608.03983). We repeat cosine annealing schedules, but each time we halve the magnitude and double the cycle length." |
| ] |
| }, |
| { |
| "cell_type": "code", |
| "execution_count": 15, |
| "id": "c6cbbfe4", |
| "metadata": {}, |
| "outputs": [ |
| { |
| "data": { |
| "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAGwCAYAAABB4NqyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABbHUlEQVR4nO3deXxM9/4/8NfMJJMFmYjIRkjsSxBrxHLVFY3Yqu1t8VVUtf1x1VJ0iRKqKritotXqQsNtbd20JaIau8YuSBFiaYIkJCQjQZaZ8/vDzdQQyTnJnFlfz8fjPMiZzznz/lgy73zO5/35KARBEEBERETkQJSWDoCIiIjI3JgAERERkcNhAkREREQOhwkQERERORwmQERERORwmAARERGRw2ECRERERA7HydIBWCO9Xo9r166hVq1aUCgUlg6HiIiIRBAEAbdv30ZAQACUyorHeJgAlePatWsIDAy0dBhERERUBRkZGahfv36FbZgAlaNWrVoA7v8Benh4WDgaIiIiEkOr1SIwMNDwOV4RJkDlKHvs5eHhwQSIiIjIxoiZvsJJ0ERERORwmAARERGRw2ECRERERA6HCRARERE5HCZARERE5HCYABEREZHDYQJEREREDocJEBERETkcJkBERETkcLgStBkV3CvF5HVHcfKqFi5OKvxfWAO83LMR1E7MQ4mIiMzJop+8sbGx6Ny5M2rVqgUfHx8MGTIEqamplV733XffoUWLFnB1dUWbNm0QHx9v9LogCIiJiYG/vz/c3NwQERGB8+fPy9UNUQZ/shchc7YhMTUHNwqKcSXvLhZtS0WzmVvx7q+nLBobERGRo7FoArR7925MmDABBw4cwPbt21FSUoInn3wShYWFj73mjz/+wPDhwzF27FgcP34cQ4YMwZAhQ5CSkmJos2jRIixbtgwrVqzAwYMHUaNGDURGRuLevXvm6NYjBn+yFyevaB/7+tf709E99jczRkREROTYFIIgCJYOosyNGzfg4+OD3bt34x//+Ee5bYYOHYrCwkJs3rzZcK5r164IDQ3FihUrIAgCAgICMG3aNEyfPh0AkJ+fD19fX8TFxWHYsGGVxqHVaqHRaJCfn1/tzVAL7pUiZM42UW1rqJX4c25Utd6PiIjIUUn5/LaqySf5+fkAAC8vr8e2SUpKQkREhNG5yMhIJCUlAQAuXbqErKwsozYajQZhYWGGNg8rKiqCVqs1Okzl9Q3HRbctLNajw3scCSIiIpKb1SRAer0eU6ZMQffu3RESEvLYdllZWfD19TU65+vri6ysLMPrZece1+ZhsbGx0Gg0hiMwMLA6XTGSfuuupPY3C0sw5utDJnt/IiIiepTVJEATJkxASkoK1q9fb/b3jo6ORn5+vuHIyMgw2b0b1HaTfM3O1Bv49cQ1k8VARERExqwiAXrttdewefNm7Ny5E/Xr16+wrZ+fH7Kzs43OZWdnw8/Pz/B62bnHtXmYi4sLPDw8jA5T+Who+ypdN3n9cej0VjM9i4iIyK5YNAESBAGvvfYafvrpJ+zYsQPBwcGVXhMeHo7ExESjc9u3b0d4eDgAIDg4GH5+fkZttFotDh48aGhjTjVdndCmXi3J1+kFYOLaYzJERERERBZNgCZMmIBvvvkGa9euRa1atZCVlYWsrCzcvfv3vJlRo0YhOjra8PXkyZORkJCADz/8EGfPnsWcOXNw5MgRvPbaawAAhUKBKVOmYN68efjll19w6tQpjBo1CgEBARgyZIi5uwgA+HXiP9CwjvRHYfEpWSgu1csQERERkWOzaAL02WefIT8/H0888QT8/f0Nx4YNGwxt0tPTkZmZafi6W7duWLt2Lb744gu0a9cO33//PTZt2mQ0cfrNN9/ExIkT8eqrr6Jz584oKChAQkICXF1dzdq/B+1+45/46PlQyde9/cMJ0wdDRETk4KxqHSBrYcp1gB5WXKpHs5lbRbdXAEib3x8qpcKkcRAREdkbm10HyBGonZT4ZFio6PYCgKXbz8kWDxERkSNiAmQBA0ProWldd9HtP92dxoowIiIiE2ICZCFbJvcS3bZUD3ycaNnNXImIiOwJEyALUTsp0TW4tuj2K3Zf4CgQERGRiTABsqA1Y7uKbnuvVI8DF3JljIaIiMhxMAGyILWTEqGB4qvM9l+4IWM0REREjoMJkIW9EdlSdNvDl27KGAkREZHjYAJkYV0b1YGzyL+FE1fyOQ+IiIjIBJgAWZhKqUCflr6i2hbrBM4DIiIiMgEmQFZgZHiQ6LZrDlyWLQ4iIiJHwQTICkh5DLbz7HU+BiMiIqomJkBWgI/BiIiIzIsJkJXgYzAiIiLzYQJkJfgYjIiIyHyYAFkJPgYjIiIyHyZAVoSPwYiIiMyDCZAV4WMwIiIi82ACZEX4GIyIiMg8mABZGSmPwbg5KhERUdUwAbIyXRvVgZNCXFtujkpERFQ1TICsjEqpQPuGtUW1Tbmm5TwgIiKiKmACZIW6BHuJane3RI9DHAUiIiKSjAmQFerW2Ft026z8uzJGQkREZJ+YAFmhro3qwEXkRKB9aTkyR0NERGR/mABZIZVSgd7NfUS13XIyk/OAiIiIJGICZKWa+NQS1e5eqZ7rAREREUnEBMhKhTeuI7pt0kU+BiMiIpKCCZCV6tqoDtQqcW3PXy+QNxgiIiI7wwTISqmUCgxqFyCq7Z5zNzgPiIiISAImQFasR1NxE6HvlnAeEBERkRRMgKyYn4er6LacB0RERCSeRROgPXv2YNCgQQgICIBCocCmTZsqbP/iiy9CoVA8crRu3drQZs6cOY+83qJFC5l7Io8uwV5wV4v7K+ITMCIiIvEsmgAVFhaiXbt2WL58uaj2S5cuRWZmpuHIyMiAl5cXnnvuOaN2rVu3Nmq3b98+OcKXnUqpQL/WvqLaZuXfkzkaIiIi++FkyTePiopCVFSU6PYajQYajcbw9aZNm3Dr1i2MGTPGqJ2TkxP8/PxMFqcl+Xu6i2r32+ks6PQCVEqRW8kTERE5MJueA7Ry5UpERESgYcOGRufPnz+PgIAANGrUCCNGjEB6enqF9ykqKoJWqzU6rIUC4hKagiIdN0YlIiISyWYToGvXrmHr1q14+eWXjc6HhYUhLi4OCQkJ+Oyzz3Dp0iX07NkTt2/ffuy9YmNjDaNLGo0GgYGBcocvmpQFEbkxKhERkTg2mwCtXr0anp6eGDJkiNH5qKgoPPfcc2jbti0iIyMRHx+PvLw8bNy48bH3io6ORn5+vuHIyMiQOXrxpGyMmlNQJHM0RERE9sEmEyBBELBq1SqMHDkSarW6wraenp5o1qwZ0tLSHtvGxcUFHh4eRoe1kLIx6tH0WzJHQ0REZB9sMgHavXs30tLSMHbs2ErbFhQU4MKFC/D39zdDZPIQuzHqrrNcEZqIiEgMiyZABQUFSE5ORnJyMgDg0qVLSE5ONkxajo6OxqhRox65buXKlQgLC0NISMgjr02fPh27d+/G5cuX8ccff+Dpp5+GSqXC8OHDZe2LnMTOA+LO8EREROJYtAz+yJEj6N27t+HrqVOnAgBGjx6NuLg4ZGZmPlLBlZ+fjx9++AFLly4t955XrlzB8OHDkZubi7p166JHjx44cOAA6tatK19HZHZ/HpASRaX6StsmXcxB96beZoiKiIjIdikEQeAzk4dotVpoNBrk5+dbzXygcf89goQ/syttF9naF5+P7GSGiIiIiKyLlM9vm5wD5Ig6NvQS1S7pQi7nAREREVWCCZCN8K7lIqqd9l4pF0QkIiKqBBMgGyFlZ3guiEhERFQxJkA2okuwF2q6iPvr4oKIREREFWMCZCNUSgV6NBFXyZZ3t0TmaIiIiGwbEyAb0qhuTVHt0q4XyBwJERGRbWMCZENqu1e87UcZVoIRERFVjAmQDWElGBERkWkwAbIhrAQjIiIyDSZANqRLsBdquapEtb1ZWCxzNERERLaLCZANUSkVeKZ9PVFt02/ekTkaIiIi28UEyMY08Kohqt1Px69yIjQREdFjMAGyMV41ORGaiIioupgA2RhOhCYiIqo+JkA2hhOhiYiIqo8JkI3hRGgiIqLqYwJkgzgRmoiIqHqYANkgToQmIiKqHiZANogToYmIiKqHCZAN4kRoIiKi6mECZIOkTIT2FLmDPBERkSNhAmSj6td2F9XuZmGRzJEQERHZHiZANirvbomodkfTb8kcCRERke1hAmSjFFCIarfvfC5L4YmIiB7CBMhGhTeuI6pdQRFL4YmIiB7GBMhGdW1UB27O4v76WApPRERkjAmQjVIpFRjQxl9UW5bCExERGWMCZMPCG3uLasc9wYiIiIwxAbJheXfEjexwTzAiIiJjTIBsGPcEIyIiqhomQDaMe4IRERFVDRMgG8Y9wYiIiKrGognQnj17MGjQIAQEBEChUGDTpk0Vtt+1axcUCsUjR1ZWllG75cuXIygoCK6urggLC8OhQ4dk7IXlcE8wIiKiqrFoAlRYWIh27dph+fLlkq5LTU1FZmam4fDx8TG8tmHDBkydOhWzZ8/GsWPH0K5dO0RGRuL69eumDt8qNPCqIapd0oUcmSMhIiKyHU6WfPOoqChERUVJvs7Hxweenp7lvrZ48WK88sorGDNmDABgxYoV2LJlC1atWoW333673GuKiopQVPT3pqFarVZyTJYidiL072euQ6cXoFKK20KDiIjIntnkHKDQ0FD4+/ujb9++2L9/v+F8cXExjh49ioiICMM5pVKJiIgIJCUlPfZ+sbGx0Gg0hiMwMFDW+E1J7ETovLslrAQjIiL6H5tKgPz9/bFixQr88MMP+OGHHxAYGIgnnngCx44dAwDk5ORAp9PB19fX6DpfX99H5gk9KDo6Gvn5+YYjIyND1n6YUpdgL2hcxQ3ksRKMiIjoPos+ApOqefPmaN68ueHrbt264cKFC/joo4/w3//+t8r3dXFxgYuLuEdJ1kalVKBvK198f+xqpW1ZCUZERHSfTY0AladLly5IS0sDAHh7e0OlUiE7O9uoTXZ2Nvz8/CwRnlmI3RKDlWBERET32XwClJycDH//+5uCqtVqdOzYEYmJiYbX9Xo9EhMTER4ebqkQZXezsKjyRhLaERER2TuLPgIrKCgwjN4AwKVLl5CcnAwvLy80aNAA0dHRuHr1KtasWQMAWLJkCYKDg9G6dWvcu3cPX331FXbs2IHffvvNcI+pU6di9OjR6NSpE7p06YIlS5agsLDQUBVmj/LulohqdzT9Fl6RORYiIiJbYNEE6MiRI+jdu7fh66lTpwIARo8ejbi4OGRmZiI9Pd3wenFxMaZNm4arV6/C3d0dbdu2xe+//250j6FDh+LGjRuIiYlBVlYWQkNDkZCQ8MjEaHuigLjS9n3nc1kKT0REBEAhCAK3CX+IVquFRqNBfn4+PDw8LB1Opfan5WDEVwdFtV33SleEN64jc0RERETmJ+Xz2+bnABHQtVEduDmL+6tkKTwRERETILugUiowoI2/qLYshSciImICZDdYCk9ERCQeEyA7kXdH3MgON0UlIiJiAmQ3pG6KSkRE5MiYANkJbopKREQkHhMgO8FNUYmIiMRjAmQnyjZFFYOVYERE5OiYANkRVoIRERGJwwTIjrASjIiISBwmQHaElWBERETiMAGyI6wEIyIiEocJkB1hJRgREZE4TIDsCCvBiIiIxGECZGdYCUZERFQ5JkB2RmwlmNh2RERE9ogJkJ0RO7LDESAiInJkTIDsDNcCIiIiqhwTIDvDtYCIiIgqxwTIznAtICIiosoxAbIzXAuIiIiockyA7AzXAiIiIqocEyA7xLWAiIiIKsYEyA6xEoyIiKhiTIDsECvBiIiIKsYEyA6xEoyIiKhiTIDsECvBiIiIKsYEyA6xEoyIiKhiTIDsFCvBiIiIHq9aCdC9e/dMFQeZGCvBiIiIHk9yAqTX6/Hee++hXr16qFmzJi5evAgAmDVrFlauXGnyAKlqWAlGRET0eJIToHnz5iEuLg6LFi2CWv3345OQkBB89dVXku61Z88eDBo0CAEBAVAoFNi0aVOF7X/88Uf07dsXdevWhYeHB8LDw7Ft2zajNnPmzIFCoTA6WrRoISkue8BKMCIioseTnACtWbMGX3zxBUaMGAGVSmU4365dO5w9e1bSvQoLC9GuXTssX75cVPs9e/agb9++iI+Px9GjR9G7d28MGjQIx48fN2rXunVrZGZmGo59+/ZJissesBKMiIjo8cR9Qj7g6tWraNKkySPn9Xo9SkpKJN0rKioKUVFRotsvWbLE6Ov58+fj559/xq+//or27dsbzjs5OcHPz0/0fYuKilBUVGT4WqvVir7WWpVVgn1/7GqlbVkJRkREjkbyCFCrVq2wd+/eR85///33RkmIOej1ety+fRteXl5G58+fP4+AgAA0atQII0aMQHp6eoX3iY2NhUajMRyBgYFyhm02rAQjIiIqn+QRoJiYGIwePRpXr16FXq/Hjz/+iNTUVKxZswabN2+WI8bH+uCDD1BQUIDnn3/ecC4sLAxxcXFo3rw5MjMz8e6776Jnz55ISUlBrVq1yr1PdHQ0pk6davhaq9XaRRIkthJMbDsiIiJ7ITkBeuqpp/Drr79i7ty5qFGjBmJiYtChQwf8+uuv6Nu3rxwxlmvt2rV499138fPPP8PHx8dw/sFHam3btkVYWBgaNmyIjRs3YuzYseXey8XFBS4u4qqmbInYkR2OABERkaORnAABQM+ePbF9+3ZTxyLa+vXr8fLLL+O7775DREREhW09PT3RrFkzpKWlmSk66yFlLaBnO9aXORoiIiLrIXkOUKNGjZCbm/vI+by8PDRq1MgkQVVk3bp1GDNmDNatW4cBAwZU2r6goAAXLlyAv7+/7LFZG64FREREVD7JCdDly5eh0+keOV9UVISrVyuvOHpQQUEBkpOTkZycDAC4dOkSkpOTDZOWo6OjMWrUKEP7tWvXYtSoUfjwww8RFhaGrKwsZGVlIT8/39Bm+vTp2L17Ny5fvow//vgDTz/9NFQqFYYPHy61qzaPawERERGVT/QjsF9++cXw+23btkGj0Ri+1ul0SExMRFBQkKQ3P3LkCHr37m34umwi8ujRoxEXF4fMzEyjCq4vvvgCpaWlmDBhAiZMmGA4X9YeAK5cuYLhw4cjNzcXdevWRY8ePXDgwAHUrVtXUmz2oGwtoPx7pZW25VpARETkSBSCIIh69qFU3h8sUigUePgSZ2dnBAUF4cMPP8TAgQNNH6WZabVaaDQa5Ofnw8PDw9LhVMv0jcmi1gKaNaAlxvaU/xEmERGRXKR8foseAdLr9QCA4OBgHD58GN7e4taYIcsKb+wtKgFiJRgRETkSyXOALl26xOTHhnBXeCIiokdVqQy+sLAQu3fvRnp6OoqLjT9gJ02aZJLAyDSkVoKplAqZIyIiIrI8yQnQ8ePH0b9/f9y5cweFhYXw8vJCTk4O3N3d4ePjwwTIykitBAtvXEfmiIiIiCxP8iOw119/HYMGDcKtW7fg5uaGAwcO4K+//kLHjh3xwQcfyBEjVQN3hSciInqU5AQoOTkZ06ZNg1KphEqlQlFREQIDA7Fo0SLMmDFDjhipGsp2hReDu8ITEZGjkJwAOTs7G0rifXx8DOv0aDQaZGRkmDY6MgnuCk9ERGRMcgLUvn17HD58GADQq1cvxMTE4Ntvv8WUKVMQEhJi8gCp+lgJRkREZExyAjR//nzDvlrvv/8+ateujfHjx+PGjRv4/PPPTR4gVR/3BCMiIjImuQqsU6dOht/7+PggISHBpAGR6bESjIiIyJjkEaDHOXbsmF1sg2GPWAlGRERkTFICtG3bNkyfPh0zZszAxYsXAQBnz57FkCFD0LlzZ8N2GWRdWAlGRERkTHQCtHLlSkRFRSEuLg4LFy5E165d8c033yA8PBx+fn5ISUlBfHy8nLFSNXRvWldUO7HzhYiIiGyZ6ARo6dKlWLhwIXJycrBx40bk5OTg008/xalTp7BixQq0bNlSzjipmnxEJjZi2xEREdky0QnQhQsX8NxzzwEAnnnmGTg5OeE///kP6tevL1twZEJit/jiVmBEROQARCdAd+/ehbu7OwBAoVDAxcXFUA5P1u+69p5J2xEREdkySWXwX331FWrWrAkAKC0tRVxcHLy9jVcZ5mao1kns5Ob9aTl4ugNH9YiIyL4pBEEQtfJdUFAQFIqKn48oFApDdZgt02q10Gg0yM/Ph4eHh6XDMYmfjl/F6xuSK23n6eaMo7P6QqXkszAiIrItUj6/RY8AXb58ubpxkQVxMUQiIqK/mWwhRLJuXAyRiIjob0yAHAQXQyQiIvobEyAHEt7Yu/JGADzd1TJHQkREZFlMgBxI3h1xIztJF3JkjoSIiMiymAA5ELHbXPx+5jp0elHFgURERDZJ0jpAwP0Ss/KULY6oVvPxibViJRgREdF9khMgT0/PCtcDql+/Pl588UXMnj0bSiUHmKxJWSVY/r3SStuyEoyIiOyZ5AQoLi4O77zzDl588UV06dIFAHDo0CGsXr0aM2fOxI0bN/DBBx/AxcUFM2bMMHnAVHVllWDfH7taaVtWghERkT2TnACtXr0aH374IZ5//nnDuUGDBqFNmzb4/PPPkZiYiAYNGuD9999nAmSFwht7i0qAWAlGRET2TPIzqj/++APt27d/5Hz79u2RlJQEAOjRowfS09OrHx2ZHCvBiIiIqjACFBgYiJUrV2LBggVG51euXInAwEAAQG5uLmrXrm2aCMmkpFaCWWJPMJ1ewL7UG/hs93mcztSiWH9/j7LezX0QM6g13NQqs8dERET2RXIC9MEHH+C5557D1q1b0blzZwDAkSNHcPbsWXz//fcAgMOHD2Po0KGmjZRMwtorwRJSMjFpfTKKS/VG57NKirDucAbWHc4AADirFKhTQ82kiIiIqkTyI7DBgwfj7NmziIqKws2bN3Hz5k1ERUXh7NmzGDhwIABg/PjxWLx4caX32rNnDwYNGoSAgAAoFAps2rSp0mt27dqFDh06wMXFBU2aNEFcXNwjbZYvX46goCC4uroiLCwMhw4dktpNu2XNe4IlpGRi3DfHHkl+ylOiE5ClvZ8UtYxJQLN3tiD6h5O4W6wzQ6RERGTrJI8AAUBwcPAjj8CqorCwEO3atcNLL72EZ555ptL2ly5dwoABAzBu3Dh8++23SExMxMsvvwx/f39ERkYCADZs2ICpU6dixYoVCAsLw5IlSxAZGYnU1FT4+PhUO2ZbZ62VYDq9gJhNp6p8fbEOhhGiOjWcsfuNf6KmyESPiIgcj0IQBMlL/ubl5eHQoUO4fv069Hrjn9ZHjRpVtUAUCvz0008YMmTIY9u89dZb2LJlC1JSUgznhg0bhry8PCQkJAAAwsLC0LlzZ3zyyScAAL1ej8DAQEycOBFvv/12ufctKipCUVGR4WutVovAwEDk5+fDw8OjSv2xZj8dv4rXNyRX2u6joaF4un09+QMCkHQhF8O/PGDSe9ZwVuDTFzqhR9O6FpnLRERE5qXVaqHRaER9fkv+EfnXX3/FiBEjUFBQAA8PD6NFERUKRZUTIDGSkpIQERFhdC4yMhJTpkwBABQXF+Po0aOIjo42vK5UKhEREWGoUCtPbGws3n33XVlitkY+IidCi21nCtdv3zP5PQtLBIz++jAAYNnz7TC4Q32TvwcREdkmyXOApk2bhpdeegkFBQXIy8vDrVu3DMfNmzfliNEgKysLvr6+Rud8fX2h1Wpx9+5d5OTkQKfTldsmKyvrsfeNjo5Gfn6+4cjIyJAlfqshdjDEjIMmPrXETc6uqkkbTyDs/d9EzS8iIiL7JzkBunr1KiZNmgR3d3c54rEIFxcXeHh4GB327LpW3GiL2Ham0CXYCz41nWV9j+zbJWg2cyve/bXqc42IiMg+SE6AIiMjceTIETliqZSfnx+ys7ONzmVnZ8PDwwNubm7w9vaGSqUqt42fn585Q7VqYic3708z32KIKqUCc4e0Mct7fb0/HZ3e28Yd74mIHJjkOUADBgzAG2+8gdOnT6NNmzZwdjb+qX3w4MEmC+5h4eHhiI+PNzq3fft2hIeHAwDUajU6duyIxMREw2RqvV6PxMREvPbaa7LFZWusdTHEfiH+WPFCh3LXATK1nMJSNJ4Rj0+GhWJgqHkmehMRkfWQXAVW0Q7vCoUCOp34dVgKCgqQlpYG4P5WGosXL0bv3r3h5eWFBg0aIDo6GlevXsWaNWsA3C+DDwkJwYQJE/DSSy9hx44dmDRpErZs2WJUBj969Gh8/vnn6NKlC5YsWYKNGzfi7Nmzj8wNehwps8htkZSKq3WvdDX7YohlK0G/H/8nzt24I/v7/bN5Hawa01X29yEiInnJWgX2cNl7dRw5cgS9e/c2fD116lQAwOjRoxEXF4fMzEyjPcWCg4OxZcsWvP7661i6dCnq16+Pr776ypD8AMDQoUNx48YNxMTEICsrC6GhoUhISBCd/DiCssUQ8++VVtrW3IshAvcfh/Vq6YNeLX2MtsX481o+CosFmHpsaEdqLrrP/w37Zzxp4jsTEZG1qtI6QPbO3keAAGD6xmRRiyHOGtASY3s2MkNE0hTcK8WktUew+3wudCb6F+zurMDp9/qb5mZERGR2Jh8BWrZsGV599VW4urpi2bJlFbadNGmS+EjJYsIbe4tKgDzd1WaIRrqark5Y9dL9x1Z3i3X454c7kZlfVMlVFbtTIqDZO1tw5r3+XDiRiMjOiRoBCg4OxpEjR1CnTh0EBwc//mYKBS5evGjSAC3BEUaAVu69iPe2nKm03b861MMHz4fKH5AJ3C3WIeLDXbiaX/3y/RUvdEC/EH8TREVEROZi8hGgS5culft7sl3WWglWHW5qFfZH98HdYh06z9uOgmpsjDrum2P49P/ao3/bABNGSERE1kLyOkBkH/w8xK28nHe3BIcuybvCt6m5qVVImdsPL3UPqtZ9/r32ODYnXzNNUEREZFUkV4HpdDrExcUhMTGx3M1Qd+zYYbLgSD7WXglmCjGDWuPtqJZo924C7pZUbab0a+uP49S1PET3b2Xi6IiIyJIkjwBNnjwZkydPhk6nQ0hICNq1a2d0kG1QKRXo20rc0gBiV462RmonJc681x8h9ao+l+vzPZcQfzLThFEREZGlSR4BWr9+PTZu3Ij+/VkubOtsvRJMis0Te+Ln5KuYvD65Stf/e+0xXAhhdRgRkb2QPAKkVqvRpEkTOWIhM8u7I25kJ+mC+fYEk9NTofVwYX5/qKs4863t7K2mDYiIiCxG8kfBtGnTsHTpUnD9RNsntRLMHqiUCpybPwCerirJ1xaWCGg/d5sMURERkblJfgS2b98+7Ny5E1u3bkXr1q0f2Qz1xx9/NFlwJC+plWDm3hNMTslz+iH03QTk3ZVWKn/rTim6L/gd+9+OkCkyIiIyB8kjQJ6ennj66afRq1cveHt7Q6PRGB1kO8oqwcSw1UqwiiTP7ofa7pJ/BsDVvCL0X7pbhoiIiMhcJH33Ly0tRe/evfHkk0/Cz89PrpjITMoqwcRMhLblSrCKHI+JRLfY7biWL61/pzMLMOjjvfh1Yk+ZIiMiIjlJGgFycnLCuHHjUFRUvT2XyHp0b1pXVDux84Vs0R/RfRGgkV7pduqqFu9tPi1DREREJDfJj8C6dOmC48ePyxELWYCPyMRGbDtb9Ud0X9R2kz4xeuW+Sygu1VfekIiIrIrkCRD//ve/MW3aNFy5cgUdO3ZEjRo1jF5v27atyYIjMxC7rI0DLH9zfHY/tI7ZisJiaQlN2zlbcXbeAJmiIiIiOUhOgIYNGwYAmDRpkuGcQqGAIAhQKBTQ6aq+ASWZ33WtuJ3TxbazdSfn9EPjGfGSrrlXCvRYmIh9b/WRKSoiIjI1yQkQd4O3L2InN+9Py8HTHerLHI3lqZQKfPp/7fHvtdIe8165dQ8vxR3Cqhe7yBQZERGZkuQEqGHDhnLEQRYidTFER9gKon/bALyScQtf7r0s6bodZ2/g1xPXMKhdgDyBERGRyUhfBOV/Tp8+jfT0dBQXG48gDB48uNpBkfk48mKIFXlnQGuU6gV8vf8vSddNXHcc/dv4O0SiSERkyyQnQBcvXsTTTz+NU6dOGeb+APfnAQHgHCAbU7YYYv690krb2uNiiBWZPSgEl28UYOe5XEnXdX1/Ow7PelKmqIiIyBQkl8FPnjwZwcHBuH79Otzd3fHnn39iz5496NSpE3bt2iVDiCSnssUQxbDXxRAr8vVLXVFf4hpBNwpL8FLcIZkiIiIiU5CcACUlJWHu3Lnw9vaGUqmEUqlEjx49EBsba1QZRrYjvLG3qHae7tIXC7QH+6L7ws1Z2iOtsvlARERknSQnQDqdDrVq1QIAeHt749q1+9/kGzZsiNTUVNNGR2aRd0fcyE7ShRyZI7FeJ2b3k3zNxHXHodMLMkRDRETVJTkBCgkJwYkTJwAAYWFhWLRoEfbv34+5c+eiUaNGJg+Q5Ce1EswRqZ2UGNtDegVk+PzfZYiGiIiqS3ICNHPmTOj191fKnTt3Li5duoSePXsiPj4ey5YtM3mAJD+plWCOatbAELT2r1F5wwdcLyjG3F//lCkiIiKqKoVQVsZVDTdv3kTt2rUNlWC2TqvVQqPRID8/Hx4eHpYOR3Y6vYAOc38TVQn20fPtHGJBxIq0nZMA7T1p1Y7n5kVB7ST55w0iIpJAyud3lb8jp6WlYdu2bbh79y68vLyqehuyAqwEk+bITOkl7p3n/SZDJEREVFWSE6Dc3Fz06dMHzZo1Q//+/ZGZmQkAGDt2LKZNm2byAMk8WAkmXlXmA+Xf07E0nojIikhOgF5//XU4OzsjPT0d7u7uhvNDhw5FQkKCSYMj82ElmDRVmQ/E0ngiIushOQH67bffsHDhQtSvbzwPpGnTpvjrL2nbBpD1YCWYdFsmPwEPV5Wka1gaT0RkHSQnQIWFhUYjP2Vu3rwJFxdxH6JkfVgJVjVVmQ8U8eEu0wdCRESSSE6AevbsiTVr1hi+VigU0Ov1WLRoEXr37l2lIJYvX46goCC4uroiLCwMhw49fq7EE088AYVC8cgxYMAAQ5sXX3zxkdf79ZO+kJ0jKdsTTAxH2xOsImonJcZ0byDpmku5d/Bz8lWZIiIiIjEkb4a6aNEi9OnTB0eOHEFxcTHefPNN/Pnnn7h58yb2798vOYANGzZg6tSpWLFiBcLCwrBkyRJERkYiNTUVPj4+j7T/8ccfjXagz83NRbt27fDcc88ZtevXrx++/vprw9ccnapYWSXY98cq/2BmJZix2YPaYPOJTNwoKBF9zeT1yRjYNoC7xhMRWUiVVoI+d+4cevTogaeeegqFhYV45plncPz4cTRu3FhyAIsXL8Yrr7yCMWPGoFWrVlixYgXc3d2xatWqctt7eXnBz8/PcGzfvh3u7u6PJEAuLi5G7WrXrv3YGIqKiqDVao0OR8RKsKo7MKOv5Gue++wPGSIhIiIxqrQOkEajwTvvvIONGzciPj4e8+bNg06nw6uvvirpPsXFxTh69CgiIiL+DkipREREBJKSkkTdY+XKlRg2bBhq1DCuyNm1axd8fHzQvHlzjB8/Hrm5uY+9R2xsLDQajeEIDAyU1A97IbYSTGw7R6JSKvDJsFBJ1xzLyGNVGBGRhZhsadrc3FysXLlS0jU5OTnQ6XTw9TVehM/X1xdZWVmVXn/o0CGkpKTg5ZdfNjrfr18/rFmzBomJiVi4cCF2796NqKgo6HTlr94bHR2N/Px8w5GRkSGpH/ZC7MgOR4DKNzC0HtoHSls5fMp6VoUREVmCTa/Nv3LlSrRp0wZdunQxOj9s2DAMHjwYbdq0wZAhQ7B582YcPnwYu3btKvc+Li4u8PDwMDocEdcCqr7vx/eQ1F4nABPXHpMpGiIiehyLJkDe3t5QqVTIzs42Op+dnQ0/P78Kry0sLMT69esxduzYSt+nUaNG8Pb2RlpaWrXitXdcC6j6qvIoLD4lC/EnM+UJiIiIymXRBEitVqNjx45ITEw0nNPr9UhMTER4eHiF13733XcoKirCCy+8UOn7XLlyBbm5ufD39692zPaMawGZxsDQeugg8VHYxHXHmFQSEZmR6DL4Z555psLX8/LyqhTA1KlTMXr0aHTq1AldunTBkiVLUFhYiDFjxgAARo0ahXr16iE2NtboupUrV2LIkCGoU6eO0fmCggK8++67ePbZZ+Hn54cLFy7gzTffRJMmTRAZGVmlGB1F2VpAYnaF51pAFftufA80nhEvun3Zo7BPX+goY1RERFRGdAKk0WgqfX3UqFGSAxg6dChu3LiBmJgYZGVlITQ0FAkJCYaJ0enp6VAqjQeqUlNTsW/fPvz226M7bKtUKpw8eRKrV69GXl4eAgIC8OSTT+K9997jWkCV4FpApqNSKjCpd2Ms23lB9DXxKVkoLtVD7WTTU/OIiGyCQhAEjrs/RKvVQqPRID8/3+EmRP9w9AqmfXei0nYfPtcOz3asX2k7R6bTC2j6TjykPNlq6lMD26c+IVtMRET2TMrnN3/UJCOsBDMdlVKBZUNDJV1z/noh1wYiIjIDJkBkhJVgpjUwtB76tBC3wnaZyVwbiIhIdkyAyAgrwUxv5Yth8K3lLLq9nmsDERHJjgkQGeGu8PLY+1ZE5Y0eUDYhmoiI5MEEiIyUVYKJwUow8dROSvQPEffnWmbAsj0yRUNEREyA6BHcFV4eH/9fRygktOeEaCIi+TABokewEkweKqUCH0vcJoMToomI5MEEiB7BSjD5DAyth6Z13UW354RoIiJ5MAGiR7ASTF5bJveS1J4ToomITI8JED2ClWDy4oRoIiLLYwJEj2AlmPw4IZqIyLKYAFG5WAkmL06IJiKyLCZAVC6xlWBi29GjOCGaiMhymABRucSO7HAEqHo4IZqIyDKYAFG5uBaQeXBCNBGRZTABonJxLSDz4YRoIiLzYwJE5eJaQOZTlQnRr2/ghGgioupgAkTl4lpA5iV1QnSpHli6/ZyMERER2TcmQFQurgVkflInRH+yK42jQEREVcQEiB6LawGZl9QJ0XqBo0BERFXFBIgei5Vg5id1QvSynRwFIiKqCiZA9FisBDO/qkyIfu6zP+QJhojIjjEBosdiJZhlDAytBz8PccknABzLyGNZPBGRREyA6LFYCWY5//lXO0ntWRZPRCQNEyB6LFaCWU63Jt5wlvC/k2XxRETSMAGiComtBEu/eUfmSByLSqnAR8+HSrqGE6KJiMRjAkQVElsJ9tPxq/zwNbGBofXQIdBD0jWcEE1EJA4TIKqQ2Eow7b1SToSWwXfje0hqzwnRRETiMAGiComtBAM4EVoOKqUCk3o3lnTNtI3JHI0jIqoEEyCqUJdgL9RyVYlqy4nQ8pjctzmUElZHLNYJ+OM8F6ckIqqIuBpnclgqpQLPtK+H1UnplbbllhjyUCkVWDY0FK+tTxZ9zfQfknFwRl/5grJRd4t1mPPrKew4nY28O6XQ/W+gTACMVuBWKu5vNfLwaw//XqkAnFUK1KnpghFhDfFyz0ZQO/HnSiJbYBX/U5cvX46goCC4uroiLCwMhw4demzbuLg4KBQKo8PV1fgxjSAIiImJgb+/P9zc3BAREYHz58/L3Q271cCrhqh23BJDPlInRGdrizkX6CGvrDmMljEJ2HD4Km4UlqJEAPS4fwj4+/d6AKWPee3h3+sE4F6pgKt597BoWyqazdyKoLe3oNH/juC3t6Bx9Ba0mBmP7gsS8enONBSX6i3QeyJ6mMVHgDZs2ICpU6dixYoVCAsLw5IlSxAZGYnU1FT4+PiUe42HhwdSU1MNXysUxs8HFi1ahGXLlmH16tUIDg7GrFmzEBkZidOnTz+SLFHlpG6JoZLyvIZE+258DzSeES+6/eT1x9G/jT//PnA/+dl++rrZ3u/BFEcnALoHkqRF2+5/71Li/miSs7MSgbXd8GyH+nipB0eQiMzF4v/TFi9ejFdeeQVjxoxBq1atsGLFCri7u2PVqlWPvUahUMDPz89w+Pr+vVifIAhYsmQJZs6ciaeeegpt27bFmjVrcO3aNWzatMkMPbI/3BLDOkidEK0XgIlrj8kYkW24W6wza/Ijlh6ADsC9Ej3OXy/EgoS/R5CaRG9ByOwEDP8iCbtTudcekRwsOgJUXFyMo0ePIjo62nBOqVQiIiICSUlJj72uoKAADRs2hF6vR4cOHTB//ny0bt0aAHDp0iVkZWUhIiLC0F6j0SAsLAxJSUkYNmzYI/crKipCUVGR4WutVmuK7tmNsi0x8u+VVtqWlWDymty3OT7eeQFiPw7jU7JQXKp36FGF+fGnLR2CZKUCUFCkQ9LFm0i6eP+HChUANxcV/DWuHC0iMgGL/u/JycmBTqczGsEBAF9fX2RlZZV7TfPmzbFq1Sr8/PPP+Oabb6DX69GtWzdcuXIFAAzXSblnbGwsNBqN4QgMDKxu1+wKt8SwHiqlAhMllsUPWLZHpmhsw+Vc+1ilXIf7SdGDo0XBb29B+7m/4e0fTuJusc7SIRLZFJv78SE8PByjRo1CaGgoevXqhR9//BF169bF559/XuV7RkdHIz8/33BkZGSYMGL7IHZLDFaCyU9qWfz564UOPSE6qI67pUOQjQDg1p0SrD+cgZYxCQj6X0LER2dElbNoAuTt7Q2VSoXs7Gyj89nZ2fDz8xN1D2dnZ7Rv3x5paWkAYLhOyj1dXFzg4eFhdJCxm4VFlTeS0I6qrqwsXorJ6x13t/gZ/VtZOgSzunWnBEkXb2L014fReEY8mr8Tj/5L92DHGSZERA+yaAKkVqvRsWNHJCYmGs7p9XokJiYiPDxc1D10Oh1OnToFf39/AEBwcDD8/PyM7qnVanHw4EHR96RH5d0tEdXuaPotmSMh4H5ZfNO64kc2HHlCtJtahb6tyq8odQRFOgGnM2/jpdX3E6LWMQl8ZEYEK3gENnXqVHz55ZdYvXo1zpw5g/Hjx6OwsBBjxowBAIwaNcpokvTcuXPx22+/4eLFizh27BheeOEF/PXXX3j55ZcB3K8QmzJlCubNm4dffvkFp06dwqhRoxAQEIAhQ4ZYoot2QQFxz1z2nc/lT5lmsmVyL0ntyyZEO6IvR3V26CToQYXFOsMjs+C3t6Dv4l1YsYvrE5Hjsfg6QEOHDsWNGzcQExODrKwshIaGIiEhwTCJOT09HUrl33narVu38MorryArKwu1a9dGx44d8ccff6BVq7+Hud98800UFhbi1VdfRV5eHnr06IGEhASuAVQN4Y3r4JOdaZW2Kyi6vylqeOM6ZojKsamdlOgf4ov4lOzKG/9P9I8n8eHzofIFZcW+HNXZpCtB20O6IACGSdULElLholKgQ8PaGNerMXo0rcs1pMiuKQRB4I/rD9FqtdBoNMjPz+d8oP/R6QWEzE7A3ZLKv+1/9Hw7PN2hvhmiIp1eQJMZ8aLL4gHgwvz+/GAzEZ1ewL7UG/h01zmcvJqPolL7SpL8PVwwulsQS+7JZkj5/GYCVA4mQOWbvjEZ3x+7Wmm7WQNaYmzPRmaIiABgc/JVSfuEdQj0xI8TussXEBl5XJIEQFLiamnuzko8FVoPMYNaw00tboNkInOT8vlt8UdgZDvCG3uLSoDSb9rHuiu2YmBoPXzwWyou3xS3COWxjDz8euIaBrULkDkyAu5X7fVq6YNeLR+dg6TTC9hz5joW/XYGl24UolhvvaNGd0r0WHc4A+sOZ0CtUuDZDvWZDJFN4whQOTgCVL6Vey/ivS1nKm3n4eqE4zFP8jGLGe1Py8GIrw6Kbu+kBFLn8VGYtSou1ePLvWn4Jukycm6XoFSw3tEiJkNkTTgCRLIQuymq9h4nQptb10Z14OqkwL1ScR+TpXpg6fZzmBrZXObIqCrUTkpM6N0ME3o3M5wrLtVj5b4L+P5IBq7euot7VlLFXqwTHhgZAp7tEMhkiGwCR4DKwRGg8iVdyMXwLw+IasuJ0OYndS4QwAnRtu5usQ4xv5zEtlNZ0BZZ18Mzd2cFJvVpxgnUZFacBF1NTIDKp9MLCJ27DbdF/OjJidCW8czyvTiWIX4zX06Iti8PlvnnFpZazXwiTzcnjOvVmMkQyY4JUDUxAXq82T+fwuqk9ErbjQ5viHefCjFDRPQgnV5A4xnxkq75eHh7Toi2U2WPzb47nI5LufesYh6Rv4cLFjzblusMkSyYAFUTE6DH40Ro67d421ks23lBdHulAjj/Ph+FOYKyR2abkzNxV+R8MTl1b+SFr17swvlCZDJSPr85FkmSSJ0ITeYndbd4R94nzNG4qVX4z7/a48y8/rgwvz++HtkJLX1rWOyDYP/Fm2gZk4DmM+Px6U5ux0HmxSowksTPQ/x2Iln54talIdMq2y1eyoTosn3COD/DcaiUCvRu7Yvere9vO1Q2f+iX49fMPjpUVCpg0bZULNqWCq8azvjgX6Ho1ZyPyEhe/G5HknQJ9kItV3HD1TcLi2WOhh5nYGg9dAiU9vh2wLI9MkVDtsBNrcLCZ0NxZl5/nJsXhbf6NYNvLbXZ47hZWGLYuf6FLw9w13qSDRMgkkSlVOCZ9vVEtfV0N/83T/rbd+N7QMrPz+evF+LXE9dki4dsh9pJifFPNMXBd/riwvz+WD26M5rVdTd7HPsu5KJlTAJazdrKHevJ5JgAkWT1a4v7RnizsEjmSKgiKqUCHw8LlXTNtI3J0OktPzmWrEfZVh6/TettSIa6Bnma9cPjTokeCxJS0WzmVkR+tIujQmQSTIBIsry7JaLaHU2/JXMkVJmBofXQVMJP7sU6AR8nnpcxIrJlZcnQ+nHdcXHBAKTMicQ/m9Ux6wdJanYhWsYkIPTdBOxOvc6EnaqMCRBJphD5YGXf+Vx+c7ICWyb3ktT+4x3n+fdGotR0dcKql7oakqEQ/5pme++8uzqM/vr+XKEpa4/x8RhJxgSIJBO7x1dBEUvhrYHaSYn+Ib6i2+uE+/uEEUlR09UJmyf3MpTX19OIWzLDFDadzESzmVvRc2EiCu6Vmu19ybYxASLJujaqAzdncf90WApvHT7+v46SJkQv25nGUSCqkrLy+v3REYZkyMvdPCuuZNy6h5A529D8nS2cNE2VYgJEkqmUCvQP8RPVNqeAE6GtgUqpwOQ+TSRd89xnf8gUDTmKsmToWEwkzs2LwhuRTaF2kn9tnyIdOGmaKsUEiKrEz9NNVDtOhLYeE/s0g0rCZ8+xjDyWxZPJqJ2UmNC7Gc7N648zc/shvHFts7xv2aTpkNkJ2HGGk6bpb0yAqErEToTedfYGv+FYCZVSgQlPNJZ0zcR1x/n3RybnplZh3SvdDGX1/h7yrxlWUKTjAotkhAkQVYnYidD3SvU4cCFX5mhILKn7hAHAvz7bL08w5PDKyuqTZvTFuXlRGBLqb5b3LVtgMfTdbSyld2BMgKhKujaqAxeR+0YlXcyRORoSq2yfMCmOZ+TzURjJTu2kxJJhHcw6KpR3t9RQSv/6uuOcNO1gmABRlaiUCvRuXldU2/PXC2SOhqSoyj5hk9fzURiZx8OjQm/1awa1lMlrVfTTiWtoNnMrnluxj4mQg2ACRFXWsaGXqHZJF7ggorX5bnwPSe31AjBx7TGZoiEqX9meZOfe74+UOZEI9HSV/T0PX87nmkIOggkQVZl3LXELnWnvcUFEa6NSKrDs+XaSrolPyeJPxmQxNV2dsPftPjg3LwpPmWGuENcUsn9MgKjK/DzE/zTGBRGtz+AO9RFUR9pP1AOW7ZEpGiJx1E5KLH1grpCnm7yLLD64ptBzK/YzEbIjTICoyroEe6Gmi7h/QlwQ0TolTvunpPbnrxdyQjRZhbK5QsmzI3Fmbj90byzukXx1HL6cx8UV7QgTIKoylVKBHk3ETYQWu4M8mZdKqcAnw0IlXTOJawORlXFTq/DtK+GGrTdqquX9aOPiivaBCRBVS6O64nZ/TmMlmNUaGFoPTeu6i24vgNtkkHUq23ojZW4Uzszth2a+NWR9vwcXV2QZve1hAkTVUttd3FodrASzblsm95LUnttkkLVzU6vw2+tPmG2BxbIyej4esx1MgKhaWAlmH9ROSvQP8ZV0DR+FkS14eIFFuSdNlz0e6zZ/OxMhK2cVCdDy5csRFBQEV1dXhIWF4dChQ49t++WXX6Jnz56oXbs2ateujYiIiEfav/jii1AoFEZHv3795O6GQ2IlmP34+P86itzh7T4+CiNbYu5J09e0xZwnZOUsngBt2LABU6dOxezZs3Hs2DG0a9cOkZGRuH79erntd+3aheHDh2Pnzp1ISkpCYGAgnnzySVy9etWoXb9+/ZCZmWk41q1bZ47uOBwplWD70rglhjVTKRX4WOKEaD4KI1v04KTpCb2kbRAsFTdhtV4KQRAsmpaGhYWhc+fO+OSTTwAAer0egYGBmDhxIt5+++1Kr9fpdKhduzY++eQTjBo1CsD9EaC8vDxs2rSpSjFptVpoNBrk5+fDw0PalgGOaNx/jyDhz+xK29V0UeHE7EiopO7GSWbV98OdOH/jjuj2CgBp8/vz75Vslk4vYF/qDUzeeBx5d+Vf/TnAQ43E6f+Em1ol+3s5Gimf3xYdASouLsbRo0cRERFhOKdUKhEREYGkpCRR97hz5w5KSkrg5WU8nLlr1y74+PigefPmGD9+PHJzH78jeVFREbRardFB4jXxqSWqXUGRjvOAbIDUCdF8FEa27uHHYy18xVW3VhUfj1kHiyZAOTk50Ol08PU1nnzp6+uLrKwsUfd46623EBAQYJRE9evXD2vWrEFiYiIWLlyI3bt3IyoqCjpd+UOPsbGx0Gg0hiMwMLDqnXJA4Y3riG7LeUDWT+2kxNgeDSVdw0dhZC/c1CokvN7LLNVjfDxmWRafA1QdCxYswPr16/HTTz/B1fXvybjDhg3D4MGD0aZNGwwZMgSbN2/G4cOHsWvXrnLvEx0djfz8fMORkZFhph7Yh66N6sDFSdzjD84Dsg2zBoZI3iaDVWFkTx6sHjPH4or7LuTerx6L/Z2JkJlYNAHy9vaGSqVCdrbx/JHs7Gz4+flVeO0HH3yABQsW4LfffkPbtm0rbNuoUSN4e3sjLS2t3NddXFzg4eFhdJB4KqUCvZv7iGq77c8sfkjaCKnbZPBRGNkjcy+ueC2/CC1jEhD6bgJ2p/LxmJwsmgCp1Wp07NgRiYmJhnN6vR6JiYkIDw9/7HWLFi3Ce++9h4SEBHTq1KnS97ly5Qpyc3Ph7y//YliOivOA7I9KqcCk3tIqZPgojOzZg4srdgnylPW98u7qMPrr+4/Hpqw9xlWmZWDxR2BTp07Fl19+idWrV+PMmTMYP348CgsLMWbMGADAqFGjEB0dbWi/cOFCzJo1C6tWrUJQUBCysrKQlZWFgoL7Wy0UFBTgjTfewIEDB3D58mUkJibiqaeeQpMmTRAZGWmRPjoCzgOyT5P7NodKYnHXRD4KIzundlJi47juODcvCm/1awa11P8kEm06mYlmM7ei58JEFNyTv0rNUVg8ARo6dCg++OADxMTEIDQ0FMnJyUhISDBMjE5PT0dmZqah/WeffYbi4mL861//gr+/v+H44IMPAAAqlQonT57E4MGD0axZM4wdOxYdO3bE3r174eIibtVikk7KPCDuDG87VEoFPh7eXvJ14fN/lyEaIuuidlJi/BNNce79/kiZE4k67s6yvl/GrXsImbMNrWfF8/GYCVh8HSBrxHWAqub/rTmMbafLX8DyQZGtffH5yMofXZL1eG9zClbu+0vSNS91D0LMoNYyRURkne4W6/DU8r04l11olvd7ul0AFj7XDmoni49nWAWbWQeI7IubWtweO3vO3eBPLjZm1sAQNJGwYzwArNp/mfMWyOE8OE/IHI/HuAlr1TEBIpOpV9tNVLu7JXpOhLZB8RIXSASATu9tkyESIutn7sdjZZuwcnFF8ZgAkcl0a+wtui0nQtueqiyQqC3SY8CyPTJFRGQbaro64WjMk2Ypo+fiiuIxASKT4YKI9m/WwBC09pf2DfzPa7cxNu6wTBER2Y4HH4/Jvco08Pfiiq1mbcWKXWl8JP0QJkBkMlIWRNxyMpNDtDZqy+Qn4OEqbRPHxLPXuT4Q0f88uMr06tGd4S7zKtN3SvRYkJDKuUIPYQJEJiV2QcR7pXocuPD4DWrJuh2Z+aTka7g+EJGxsk1YT8+NQsqcSAR6Stt+pirK5gpxpWkmQGRiUhZEXHPgsnyBkKzUTkqM6d5A8nWh73JSNFF5aro6Ye/bfXBuXhSeMsPjsQdXmh7xRZJDjgpxHaBycB2gqtPpBbSYGY8SEY+a1SoFzrwXBZVS3jJRkk/neb/hRkGJpGtaB9TClkn/kCkiIvug0wvYl3oDkzceR95d86z+7O6sxKQ+TfFSj0Y2u64Q1wEii1EpFejT0ldU22KdwMdgNu7AjL6Sr/nz2m3M+SVFhmiI7EfZ47Hk2ZE4M7cf6mnkfzz24Fyh8Pm/2/0jMiZAZHIjw4NEt91/4YZ8gZDsVEoFPhkWKvm6uD/+wvtbTps+ICI75KZWYX90H5yZ2w/dG3uZ5T0ztUWGR2T2Wk7PBIhMrmujOhBZDY/DXBDR5g0MrYc+LcSvAVXmy72XEH8ys/KGRATgfiL07SvhuDC/P74e2Qk1Za4eK/N3OX28XZXTcw5QOTgHqPqeW/EHDl++VWk7zgOyHwOW7MKfWdL3P7owvz///omqyNx7j5Xx93DBgmfbokfTulb1/5dzgMjiugSLG6blPCD7sWXKE/CuIW4/uAe1nb1VhmiIHIO5F1cs8+AjsicX77LJ+UJMgEgWUrbFYDm8/Tj4jvT1gQpLBLSfy/J4oup4eHFFfw+12d773PVCmyyp5yOwcvARWPWxHN5xbU6+itfWJ0u+LkCjxh/R0qvKiKh8xaV6rNx3AR9tP49inXk/6p2UwL86BmL2oNZwU0tbOb46+AiMLI7l8I6rqpOir+UXo1vsdhkiInJMD+9Ib46VpsuU6oH1hzPQMiYBTWZswds/nLS6kSGOAJWDI0CmsT8tByO+OiiqbWRrX3w+spPMEZE5VXVStKerEslzomSIiIiKS/X4cm8aliamobjU/B//co8MSfn8ZgJUDiZApsHHYNQ9djuu5hdLvs7NWYEz7/WXISIiKnO3WIenl+/D2ewCi7x/nRrO2P3GP1HTVXrxxOPwERhZBT4Go/3RfVHbTfpPeXdLBDR7Z4vNVZUQ2RI3tQoJr/fCuXlReKtfM7g7mzclyC0sQcicbRj8yV6zvm8ZJkAkKymrQq9OuiRfIGQxx2f3Q40qLNhWrAMaz4hH/MlrMkRFRGXK5gqdfi8KZ+b2w9DO9UQvZmsKJ69oLZIEMQEiWXVtVAcqkf+Rfj9je+tIkDgn5/Sr8rX/Xnsc7/56yoTRENHjuKlVWPhsKNJiByBlTiRC/Gua5X1PXtGi4J55Nn0twwSIZKVSKtC6nrh5VHoB+DjxvMwRkSWolAp8+n/tq3z91/vT0f+jnSaMiIgqU9PVCZsn9zJsveHlbrq5OuV5fcNxWe//MCZAJLtBbeuJbrt8ZxpHgexU/7YB+H//CK7y9aez76DpjC12sw8Rka1QKRXo3doXx2IicW5eFN6IbAp3GfYhS7911+T3rAgTIJLd6G5BotuW6AWOAtmx6P6t8On/dajy9SV6oNnMrZj9y0kTRkVEYqmdlJjQuxlOz40yJENqE00YalDbzST3EYtl8OVgGbzpDVm+F8kZWlFtVQrg3PvcINOe6fQCWs6MR3E1BnPUSuDEnH5mXWWWiMp3t1iHmF9O4qej11DV5YVS5kRWuySeZfBkdd6IbCm6rU4Alm4/J2M0ZGkqpQLn5g+Am3PVk9xiPdAyJgF9PtjBx2JEFuamVuE//2qPtNgBODO3H57rFCCpkqxtfQ+TrgckBkeAysERINPT6QW0mrUVRRL2o7kwn6NAjiB0TgLy7lV/ifzmvjWwaUJPjggRWRExI0Nt63vgl9d6muT9uBJ0NTEBkseS7alYkpgmun0NZwX+5GrADqH93G24dcc0JbDuzkpM6tMUL/VoBLUTB7mJrEVZMvT76RsQBAEdGtTGsuEdLLYSNBOgcjABkodOL6DZO/GQsimxs/L+GjL8qd7+9Vy0Axk3TVsF4qQEmvnWwvQnW6BX87ocUSSyc0yAqokJkHwWbzuLZTsvSL6ubysffDmqswwRkTWZ++ufWLX/smz3VwLQuDujhV8tjOvVGD2aMikisidMgKqJCZB8dHoBTd+JR1WW+mES5BiKS/VoPXsrSqo/LahSLk5KLB0Win4h/vK/GRHJzuaqwJYvX46goCC4uroiLCwMhw4dqrD9d999hxYtWsDV1RVt2rRBfHy80euCICAmJgb+/v5wc3NDREQEzp/n2jLWQKVUYNnQ0Cpdu/30ddwtNsOnIlmU2kmJ8+8PQEiA/D98FJXqMe6bY0hIyZT9vYjIulg8AdqwYQOmTp2K2bNn49ixY2jXrh0iIyNx/fr1ctv/8ccfGD58OMaOHYvjx49jyJAhGDJkCFJSUgxtFi1ahGXLlmHFihU4ePAgatSogcjISNy7d89c3aIKDAyth382r1Ola+fHnzZxNGStNk/qiaXDQs3yXnN+Oc0VyIkcjMUfgYWFhaFz58745JNPAAB6vR6BgYGYOHEi3n777UfaDx06FIWFhdi8ebPhXNeuXREaGooVK1ZAEAQEBARg2rRpmD59OgAgPz8fvr6+iIuLw7BhwyqNiY/AzKPTe9uQUyit8qdnU2/8d2yYTBGRNdLpBTz76X4kX8mX9X3WvdIV4Y2rlpgTkXWwmUdgxcXFOHr0KCIiIgznlEolIiIikJSUVO41SUlJRu0BIDIy0tD+0qVLyMrKMmqj0WgQFhb22HsWFRVBq9UaHSS/g+88KfmaoDruMkRC1kylVGDTaz1wZm4/1JSxGvD6bY4QEzkSiyZAOTk50Ol08PX1NTrv6+uLrKyscq/JysqqsH3Zr1LuGRsbC41GYzgCAwOr1B+SRqVUYMUL0vaFmtG/lUzRkLVzU6uQMrcfPno+VJb7+9RyleW+RGSdLD4HyBpER0cjPz/fcGRkZFg6JIfRL8QfK17oADGFyH1b+XA9IMLTHerhwvz+mPhEY1H/bsTw83BFl2AvE92NiGyBRRMgb29vqFQqZGdnG53Pzs6Gn59fudf4+flV2L7sVyn3dHFxgYeHh9FB5tMvxB9p8/ujfaDmsW1YAk8PUikVmNavBdLm98fq0Z3h76Gu1v3mDG7F9YCIHIxFEyC1Wo2OHTsiMTHRcE6v1yMxMRHh4eHlXhMeHm7UHgC2b99uaB8cHAw/Pz+jNlqtFgcPHnzsPcnyVEoFfppwf57H0M714OfhAn8PFwzvHIgzc/sx+aFyqZQK9Grpg6QZfXFuXhTe6tcMjb3doBKZy7g4KbHihQ5cB4jIAZl369VyTJ06FaNHj0anTp3QpUsXLFmyBIWFhRgzZgwAYNSoUahXrx5iY2MBAJMnT0avXr3w4YcfYsCAAVi/fj2OHDmCL774AgCgUCgwZcoUzJs3D02bNkVwcDBmzZqFgIAADBkyxFLdJJHc1CosfDbU0mGQDVI7KTH+iaYY/0RTAPcXVFy57wK+O5yOjFv3oNMDSiVQy5UrQRORFSRAQ4cOxY0bNxATE4OsrCyEhoYiISHBMIk5PT0dSuXfA1XdunXD2rVrMXPmTMyYMQNNmzbFpk2bEBISYmjz5ptvorCwEK+++iry8vLQo0cPJCQkwNWVkxyJHMXDCRER0YMsvg6QNeI6QERERLbHZtYBIiIiIrIEJkBERETkcJgAERERkcNhAkREREQOhwkQERERORwmQERERORwmAARERGRw2ECRERERA7H4itBW6OytSG1Wq2FIyEiIiKxyj63xazxzASoHLdv3wYABAYGWjgSIiIikur27dvQaDQVtuFWGOXQ6/W4du0aatWqBYXCdBslarVaBAYGIiMjwyG22HC0/gKO12f2176xv/bNHvsrCAJu376NgIAAo31Ey8MRoHIolUrUr19ftvt7eHjYzT82MRytv4Dj9Zn9tW/sr32zt/5WNvJThpOgiYiIyOEwASIiIiKHwwTIjFxcXDB79my4uLhYOhSzcLT+Ao7XZ/bXvrG/9s3R+vswToImIiIih8MRICIiInI4TICIiIjI4TABIiIiIofDBIiIiIgcDhMgM1q+fDmCgoLg6uqKsLAwHDp0yNIhSRYbG4vOnTujVq1a8PHxwZAhQ5CammrU5t69e5gwYQLq1KmDmjVr4tlnn0V2drZRm/T0dAwYMADu7u7w8fHBG2+8gdLSUnN2pUoWLFgAhUKBKVOmGM7ZY3+vXr2KF154AXXq1IGbmxvatGmDI0eOGF4XBAExMTHw9/eHm5sbIiIicP78eaN73Lx5EyNGjICHhwc8PT0xduxYFBQUmLsrldLpdJg1axaCg4Ph5uaGxo0b47333jPaS8iW+7tnzx4MGjQIAQEBUCgU2LRpk9HrpurbyZMn0bNnT7i6uiIwMBCLFi2Su2vlqqi/JSUleOutt9CmTRvUqFEDAQEBGDVqFK5du2Z0D3vp78PGjRsHhUKBJUuWGJ23pf6alEBmsX79ekGtVgurVq0S/vzzT+GVV14RPD09hezsbEuHJklkZKTw9ddfCykpKUJycrLQv39/oUGDBkJBQYGhzbhx44TAwEAhMTFROHLkiNC1a1ehW7duhtdLS0uFkJAQISIiQjh+/LgQHx8veHt7C9HR0ZbokmiHDh0SgoKChLZt2wqTJ082nLe3/t68eVNo2LCh8OKLLwoHDx4ULl68KGzbtk1IS0sztFmwYIGg0WiETZs2CSdOnBAGDx4sBAcHC3fv3jW06devn9CuXTvhwIEDwt69e4UmTZoIw4cPt0SXKvT+++8LderUETZv3ixcunRJ+O6774SaNWsKS5cuNbSx5f7Gx8cL77zzjvDjjz8KAISffvrJ6HVT9C0/P1/w9fUVRowYIaSkpAjr1q0T3NzchM8//9xc3TSoqL95eXlCRESEsGHDBuHs2bNCUlKS0KVLF6Fjx45G97CX/j7oxx9/FNq1aycEBAQIH330kdFrttRfU2ICZCZdunQRJkyYYPhap9MJAQEBQmxsrAWjqr7r168LAITdu3cLgnD/G4yzs7Pw3XffGdqcOXNGACAkJSUJgnD/P6xSqRSysrIMbT777DPBw8NDKCoqMm8HRLp9+7bQtGlTYfv27UKvXr0MCZA99vett94SevTo8djX9Xq94OfnJ/znP/8xnMvLyxNcXFyEdevWCYIgCKdPnxYACIcPHza02bp1q6BQKISrV6/KF3wVDBgwQHjppZeMzj3zzDPCiBEjBEGwr/4+/AFpqr59+umnQu3atY3+Pb/11ltC8+bNZe5RxSpKCMocOnRIACD89ddfgiDYZ3+vXLki1KtXT0hJSREaNmxolADZcn+ri4/AzKC4uBhHjx5FRESE4ZxSqURERASSkpIsGFn15efnAwC8vLwAAEePHkVJSYlRX1u0aIEGDRoY+pqUlIQ2bdrA19fX0CYyMhJarRZ//vmnGaMXb8KECRgwYIBRvwD77O8vv/yCTp064bnnnoOPjw/at2+PL7/80vD6pUuXkJWVZdRnjUaDsLAwoz57enqiU6dOhjYRERFQKpU4ePCg+TojQrdu3ZCYmIhz584BAE6cOIF9+/YhKioKgP3190Gm6ltSUhL+8Y9/QK1WG9pERkYiNTUVt27dMlNvqiY/Px8KhQKenp4A7K+/er0eI0eOxBtvvIHWrVs/8rq99VcKJkBmkJOTA51OZ/QBCAC+vr7IysqyUFTVp9frMWXKFHTv3h0hISEAgKysLKjVasM3kzIP9jUrK6vcP4uy16zN+vXrcezYMcTGxj7ymj329+LFi/jss8/QtGlTbNu2DePHj8ekSZOwevVqAH/HXNG/56ysLPj4+Bi97uTkBC8vL6vr89tvv41hw4ahRYsWcHZ2Rvv27TFlyhSMGDECgP3190Gm6put/Rsvc+/ePbz11lsYPny4YTNQe+vvwoUL4eTkhEmTJpX7ur31VwruBk9VNmHCBKSkpGDfvn2WDkU2GRkZmDx5MrZv3w5XV1dLh2MWer0enTp1wvz58wEA7du3R0pKClasWIHRo0dbODrT27hxI7799lusXbsWrVu3RnJyMqZMmYKAgAC77C/dV1JSgueffx6CIOCzzz6zdDiyOHr0KJYuXYpjx45BoVBYOhyrwxEgM/D29oZKpXqkMig7Oxt+fn4Wiqp6XnvtNWzevBk7d+5E/fr1Def9/PxQXFyMvLw8o/YP9tXPz6/cP4uy16zJ0aNHcf36dXTo0AFOTk5wcnLC7t27sWzZMjg5OcHX19eu+gsA/v7+aNWqldG5li1bIj09HcDfMVf079nPzw/Xr183er20tBQ3b960uj6/8cYbhlGgNm3aYOTIkXj99dcNI3721t8HmapvtvZvvCz5+euvv7B9+3bD6A9gX/3du3cvrl+/jgYNGhi+f/3111+YNm0agoKCANhXf6ViAmQGarUaHTt2RGJiouGcXq9HYmIiwsPDLRiZdIIg4LXXXsNPP/2EHTt2IDg42Oj1jh07wtnZ2aivqampSE9PN/Q1PDwcp06dMvpPV/ZN6OEPXkvr06cPTp06heTkZMPRqVMnjBgxwvB7e+ovAHTv3v2RpQ3OnTuHhg0bAgCCg4Ph5+dn1GetVouDBw8a9TkvLw9Hjx41tNmxYwf0ej3CwsLM0Avx7ty5A6XS+FuhSqWCXq8HYH/9fZCp+hYeHo49e/agpKTE0Gb79u1o3rw5ateubabeiFOW/Jw/fx6///476tSpY/S6PfV35MiROHnypNH3r4CAALzxxhvYtm0bAPvqr2SWnoXtKNavXy+4uLgIcXFxwunTp4VXX31V8PT0NKoMsgXjx48XNBqNsGvXLiEzM9Nw3Llzx9Bm3LhxQoMGDYQdO3YIR44cEcLDw4Xw8HDD62Vl4U8++aSQnJwsJCQkCHXr1rXasvCHPVgFJgj2199Dhw4JTk5Owvvvvy+cP39e+PbbbwV3d3fhm2++MbRZsGCB4OnpKfz888/CyZMnhaeeeqrc0un27dsLBw8eFPbt2yc0bdrUKsrCHzZ69GihXr16hjL4H3/8UfD29hbefPNNQxtb7u/t27eF48ePC8ePHxcACIsXLxaOHz9uqHoyRd/y8vIEX19fYeTIkUJKSoqwfv16wd3d3SJl0hX1t7i4WBg8eLBQv359ITk52eh72IMVTvbS3/I8XAUmCLbVX1NiAmRGH3/8sdCgQQNBrVYLXbp0EQ4cOGDpkCQDUO7x9ddfG9rcvXtX+Pe//y3Url1bcHd3F55++mkhMzPT6D6XL18WoqKiBDc3N8Hb21uYNm2aUFJSYubeVM3DCZA99vfXX38VQkJCBBcXF6FFixbCF198YfS6Xq8XZs2aJfj6+gouLi5Cnz59hNTUVKM2ubm5wvDhw4WaNWsKHh4ewpgxY4Tbt2+bsxuiaLVaYfLkyUKDBg0EV1dXoVGjRsI777xj9IFoy/3duXNnuf9nR48eLQiC6fp24sQJoUePHoKLi4tQr149YcGCBebqopGK+nvp0qXHfg/buXOn4R720t/ylJcA2VJ/TUkhCA8sd0pERETkADgHiIiIiBwOEyAiIiJyOEyAiIiIyOEwASIiIiKHwwSIiIiIHA4TICIiInI4TICIiIjI4TABIiIiIofDBIiIqBxBQUFYsmSJpcMgIpkwASIii3vxxRcxZMgQAMATTzyBKVOmmO294+Li4Onp+cj5w4cP49VXXzVbHERkXk6WDoCISA7FxcVQq9VVvr5u3bomjIaIrA1HgIjIarz44ovYvXs3li5dCoVCAYVCgcuXLwMAUlJSEBUVhZo1a8LX1xcjR45ETk6O4donnngCr732GqZMmQJvb29ERkYCABYvXow2bdqgRo0aCAwMxL///W8UFBQAAHbt2oUxY8YgPz/f8H5z5swB8OgjsPT0dDz11FOoWbMmPDw88PzzzyM7O9vw+pw5cxAaGor//ve/CAoKgkajwbBhw3D79m15/9CIqEqYABGR1Vi6dCnCw8PxyiuvIDMzE5mZmQgMDEReXh7++c9/on379jhy5AgSEhKQnZ2N559/3uj61atXQ61WY//+/VixYgUAQKlUYtmyZfjzzz+xevVq7NixA2+++SYAoFu3bliyZAk8PDwM7zd9+vRH4tLr9Xjqqadw8+ZN7N69G9u3b8fFixcxdOhQo3YXLlzApk2bsHnzZmzevBm7d+/GggULZPrTIqLq4CMwIrIaGo0GarUa7u7u8PPzM5z/5JNP0L59e8yfP99wbtWqVQgMDMS5c+fQrFkzAEDTpk2xaNEio3s+OJ8oKCgI8+bNw7hx4/Dpp59CrVZDo9FAoVAYvd/DEhMTcerUKVy6dAmBgYEAgDVr1qB169Y4fPgwOnfuDOB+ohQXF4datWoBAEaOHInExES8//771fuDISKT4wgQEVm9EydOYOfOnahZs6bhaNGiBYD7oy5lOnbs+Mi1v//+O/r06YN69eqhVq1aGDlyJHJzc3Hnzh3R73/mzBkEBgYakh8AaNWqFTw9PXHmzBnDuaCgIEPyAwD+/v64fv26pL4SkXlwBIiIrF5BQQEGDRqEhQsXPvKav7+/4fc1atQweu3y5csYOHAgxo8fj/fffx9eXl7Yt28fxo4di+LiYri7u5s0TmdnZ6OvFQoF9Hq9Sd+DiEyDCRARWRW1Wg2dTmd0rkOHDvjhhx8QFBQEJyfx37aOHj0KvV6PDz/8EErl/QHvjRs3Vvp+D2vZsiUyMjKQkZFhGAU6ffo08vLy0KpVK9HxEJH14CMwIrIqQUFBOHjwIC5fvoycnBzo9XpMmDABN2/exPDhw3H48GFcuHAB27Ztw5gxYypMXpo0aYKSkhJ8/PHHuHjxIv773/8aJkc/+H4FBQVITExETk5OuY/GIiIi0KZNG4wYMQLHjh3DoUOHMGrUKPTq1QudOnUy+Z8BEcmPCRARWZXp06dDpVKhVatWqFu3LtLT0xEQEID9+/dDp9PhySefRJs2bTBlyhR4enoaRnbK065dOyxevBgLFy5ESEgIvv32W8TGxhq16datG8aNG4ehQ4eibt26j0yiBu4/yvr5559Ru3Zt/OMf/0BERAQaNWqEDRs2mLz/RGQeCkEQBEsHQURERGROHAEiIiIih8MEiIiIiBwOEyAiIiJyOEyAiIiIyOEwASIiIiKHwwSIiIiIHA4TICIiInI4TICIiIjI4TABIiIiIofDBIiIiIgcDhMgIiIicjj/H/EJiQF8ANbRAAAAAElFTkSuQmCC\n", |
| "text/plain": [ |
| "<Figure size 640x480 with 1 Axes>" |
| ] |
| }, |
| "metadata": {}, |
| "output_type": "display_data" |
| } |
| ], |
| "source": [ |
| "schedule = CyclicalSchedule(CosineAnnealingSchedule, min_lr=0.01, max_lr=2,\n", |
| " cycle_length=250, cycle_length_decay=2, cycle_magnitude_decay=0.5)\n", |
| "plot_schedule(schedule)" |
| ] |
| }, |
| { |
| "cell_type": "markdown", |
| "id": "ba13e9d9", |
| "metadata": {}, |
| "source": [ |
| " <!--notebook-skip-line-->\n", |
| "\n", |
| "\n", |
| "**_Want to learn more?_** Checkout the \"Learning Rate Schedules\" tutorial for a more basic overview of learning rates found in `mx.lr_scheduler`, and an example of how to use them while training your own models.\n", |
| "\n", |
| "<!-- INSERT SOURCE DOWNLOAD BUTTONS -->" |
| ] |
| } |
| ], |
| "metadata": { |
| "language_info": { |
| "name": "python" |
| } |
| }, |
| "nbformat": 4, |
| "nbformat_minor": 5 |
| } |
