versions/master/api/python/docs/_sources/tutorials/packages/gluon/training/learning_rates/learning_rate_schedules.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3d893316",
   "metadata": {},
   "source": [
    "<!--- Licensed to the Apache Software Foundation (ASF) under one -->\n",
    "<!--- or more contributor license agreements.  See the NOTICE file -->\n",
    "<!--- distributed with this work for additional information -->\n",
    "<!--- regarding copyright ownership.  The ASF licenses this file -->\n",
    "<!--- to you under the Apache License, Version 2.0 (the -->\n",
    "<!--- \"License\"); you may not use this file except in compliance -->\n",
    "<!--- with the License.  You may obtain a copy of the License at -->\n",
    "\n",
    "<!---   http://www.apache.org/licenses/LICENSE-2.0 -->\n",
    "\n",
    "<!--- Unless required by applicable law or agreed to in writing, -->\n",
    "<!--- software distributed under the License is distributed on an -->\n",
    "<!--- \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->\n",
    "<!--- KIND, either express or implied.  See the License for the -->\n",
    "<!--- specific language governing permissions and limitations -->\n",
    "<!--- under the License. -->\n",
    "\n",
    "\n",
    "# Learning Rate Schedules\n",
    "\n",
    "Setting the learning rate for stochastic gradient descent (SGD) is crucially important when training neural networks because it controls both the speed of convergence and the ultimate performance of the network. One of the simplest learning rate strategies is to have a fixed learning rate throughout the training process. Choosing a small learning rate allows the optimizer find good solutions, but this comes at the expense of limiting the initial speed of convergence. Changing the learning rate over time can overcome this tradeoff.\n",
    "\n",
    "Schedules define how the learning rate changes over time and are typically specified for each epoch or iteration (i.e. batch) of training. Schedules differ from adaptive methods (such as AdaDelta and Adam) because they:\n",
    "\n",
    "* change the global learning rate for the optimizer, rather than parameter-wise learning rates\n",
    "* don't take feedback from the training process and are specified beforehand\n",
    "\n",
    "In this tutorial, we visualize the schedules defined in `mx.lr_scheduler`, show how to implement custom schedules and see an example of using a schedule while training models. Since schedules are passed to `mx.optimizer.Optimizer` classes, these methods work with both Module and Gluon APIs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ddeb4d47",
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import math\n",
    "import matplotlib.pyplot as plt\n",
    "import mxnet as mx\n",
    "from mxnet.gluon import nn\n",
    "from mxnet.gluon.data.vision import transforms\n",
    "import numpy as np\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "39074f8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_schedule(schedule_fn, iterations=1500):\n",
    "    # Iteration count starting at 1\n",
    "    iterations = [i+1 for i in range(iterations)]\n",
    "    lrs = [schedule_fn(i) for i in iterations]\n",
    "    plt.scatter(iterations, lrs)\n",
    "    plt.xlabel(\"Iteration\")\n",
    "    plt.ylabel(\"Learning Rate\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24fc2b91",
   "metadata": {},
   "source": [
    "## Schedules\n",
    "\n",
    "In this section, we take a look at the schedules in `mx.lr_scheduler`. All of these schedules define the learning rate for a given iteration, and it is expected that iterations start at 1 rather than 0. So to find the learning rate for the 100th iteration, you can call `schedule(100)`.\n",
    "\n",
    "### Stepwise Decay Schedule\n",
    "\n",
    "One of the most commonly used learning rate schedules is called stepwise decay, where the learning rate is reduced by a factor at certain intervals. MXNet implements a `FactorScheduler` for equally spaced intervals, and `MultiFactorScheduler` for greater control. We start with an example of halving the learning rate every 250 iterations. More precisely, the learning rate will be multiplied by `factor` _after_ the `step` index and multiples thereafter. So in the example below the learning rate of the 250th iteration will be 1 and the 251st iteration will be 0.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e4e5c4e8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAzH0lEQVR4nO3deXxU9b3/8fdMVgIkAWISlsGEiiIEAVnSKNZ6jUahKNpbKZcSpIrFoqwuIBKsCgGqFqwoLihqtaBea5UiXAyLYqNhi7K7sCQiSQhIAgESmPn+/uDH1IGAGTLJyRxez8fjPB7J93zPOZ9viJm3Z/kehzHGCAAAwCacVhcAAAAQSIQbAABgK4QbAABgK4QbAABgK4QbAABgK4QbAABgK4QbAABgK6FWF1DfPB6Pvv/+ezVt2lQOh8PqcgAAQA0YY3Tw4EG1atVKTufZz82cd+Hm+++/l8vlsroMAABwDgoLC9WmTZuz9jnvwk3Tpk0lnfjhREdHW1wNAACoifLycrlcLu/n+Nmcd+Hm5KWo6Ohowg0AAEGmJreUcEMxAACwFcINAACwFcINAACwFcINAACwFcINAACwFcINAACwFcINAACwFcINAACwFcINAACwlfNuhuK6cqTKrUc+2KBlm4t14PBxuc2JdiPp5FyKdfG1JDkdkqeejheImkKcUkSoUxe2aKz7ru+gqy+5QCFOXmIKAAgMS8/cfPzxx+rXr59atWolh8Oh99577ye3WbFihS6//HJFRETooosu0rx58+q8zp8y7LXVujRrsRas3q29Fcd1zEgenViM6vZrj6Tj9Xi8QNR03CNVVHm0ec9B/f7V1eow6UMt3rinVv8GAACcZGm4qaioUJcuXTR79uwa9d+xY4f69u2ra665Rvn5+Ro9erTuvPNOLVmypI4rPbNhr63W0s0llh3fDo65jYb/bR0BBwAQEJZelrrxxht144031rj/nDlzlJycrCeffFKSdOmll2rVqlX6y1/+ooyMjLoq84yOVLkJNgH0yPubdV3HRC5RAQBqJahuKM7NzVV6erpPW0ZGhnJzc8+4TWVlpcrLy32WQJm6aHPA9gWpqPyo8nbst7oMAECQC6pwU1RUpISEBJ+2hIQElZeX68iRI9Vuk52drZiYGO/icrkCVs/OfYcDti+cUHLwqNUlAACCXFCFm3MxYcIElZWVeZfCwsKA7TupRVTA9oUT4ptGWl0CACDIBVW4SUxMVHFxsU9bcXGxoqOj1ahRo2q3iYiIUHR0tM8SKA/16RiwfUFKjI5Ur+TmVpcBAAhyQRVu0tLSlJOT49O2dOlSpaWlWVJPo/AQXdcx3pJj29EjN3XkZmIAQK1ZGm4OHTqk/Px85efnSzrxqHd+fr4KCgoknbiklJmZ6e0/fPhwbd++XQ888IC2bt2qZ599Vm+99ZbGjBljRfmSpBczexJwaiksxKE5v7tcN6S0tLoUAIANWPoo+Jo1a3TNNdd4vx87dqwkaciQIZo3b5727NnjDTqSlJycrH/9618aM2aMZs2apTZt2uill16y5DHwH3sxsyczFPtREzMUAwDqksMYY6wuoj6Vl5crJiZGZWVlAb3/BgAA1B1/Pr+D6p4bAACAn0K4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtkK4AQAAtmJ5uJk9e7aSkpIUGRmp1NRU5eXlnbX/zJkzdckll6hRo0ZyuVwaM2aMjh49Wk/VAgCAhs7ScLNgwQKNHTtWkydP1rp169SlSxdlZGSopKSk2v5vvvmmxo8fr8mTJ2vLli2aO3euFixYoIceeqieKwcAAA2VwxhjrDp4amqqevbsqWeeeUaS5PF45HK5dO+992r8+PGn9b/nnnu0ZcsW5eTkeNvGjRunzz//XKtWrar2GJWVlaqsrPR+X15eLpfLpbKyMkVHRwd4RAAAoC6Ul5crJiamRp/flp25qaqq0tq1a5Wenv6fYpxOpaenKzc3t9ptrrjiCq1du9Z76Wr79u1atGiR+vTpc8bjZGdnKyYmxru4XK7ADgQAADQooVYduLS0VG63WwkJCT7tCQkJ2rp1a7Xb/M///I9KS0vVu3dvGWN0/PhxDR8+/KyXpSZMmKCxY8d6vz955gYAANiT5TcU+2PFihWaOnWqnn32Wa1bt07vvvuu/vWvf+mxxx474zYRERGKjo72WQAAgH1ZduYmLi5OISEhKi4u9mkvLi5WYmJitdtMmjRJgwcP1p133ilJ6ty5syoqKnTXXXdp4sSJcjqDKqsBAIA6YFkaCA8PV/fu3X1uDvZ4PMrJyVFaWlq12xw+fPi0ABMSEiJJsvC+aAAA0IBYduZGksaOHashQ4aoR48e6tWrl2bOnKmKigoNHTpUkpSZmanWrVsrOztbktSvXz899dRT6tatm1JTU/XNN99o0qRJ6tevnzfkAACA85ul4WbAgAHau3evsrKyVFRUpK5du2rx4sXem4wLCgp8ztQ8/PDDcjgcevjhh7V7925dcMEF6tevn6ZMmWLVEAAAQANj6Tw3VvDnOXkAANAwBMU8NwAAAHWBcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGyFcAMAAGylVuHm6NGjgaoDAAAgIPwONx6PR4899phat26tJk2aaPv27ZKkSZMmae7cuQEvEAAAwB9+h5vHH39c8+bN04wZMxQeHu5tT0lJ0UsvvRTQ4gAAAPzld7h57bXX9MILL2jQoEEKCQnxtnfp0kVbt24NaHEAAAD+8jvc7N69WxdddNFp7R6PR8eOHQtIUQAAAOfK73DTsWNHffLJJ6e1v/POO+rWrVtAigIAADhXof5ukJWVpSFDhmj37t3yeDx69913tW3bNr322mtauHBhXdQIAABQY36fubn55pv1wQcf6KOPPlLjxo2VlZWlLVu26IMPPtB1111XFzUCAADUmMMYY6wuoj6Vl5crJiZGZWVlio6OtrocAABQA/58fvt95qZdu3bat2/fae0HDhxQu3bt/N0dAABAQPkdbnbu3Cm3231ae2VlpXbv3h2QogAAAM5VjW8ofv/9971fL1myRDExMd7v3W63cnJylJSUFNDiAAAA/FXjcNO/f39JksPh0JAhQ3zWhYWFKSkpSU8++WRAiwMAAPBXjcONx+ORJCUnJ2v16tWKi4urs6IAAADOld/33OzYsSOgwWb27NlKSkpSZGSkUlNTlZeXd9b+Bw4c0IgRI9SyZUtFRETo4osv1qJFiwJWDwAACG5+T+InSRUVFVq5cqUKCgpUVVXls27kyJE13s+CBQs0duxYzZkzR6mpqZo5c6YyMjK0bds2xcfHn9a/qqpK1113neLj4/XOO++odevW2rVrl2JjY89lGAAAwIb8nudm/fr16tOnjw4fPqyKigo1b95cpaWlioqKUnx8vLZv317jfaWmpqpnz5565plnJJ249OVyuXTvvfdq/Pjxp/WfM2eO/vznP2vr1q0KCwvzp2wv5rkBACD41Ok8N2PGjFG/fv30ww8/qFGjRvrss8+0a9cude/eXU888USN91NVVaW1a9cqPT39P8U4nUpPT1dubm6127z//vtKS0vTiBEjlJCQoJSUFE2dOrXaR9NPqqysVHl5uc8CAADsy+9wk5+fr3HjxsnpdCokJESVlZVyuVyaMWOGHnrooRrvp7S0VG63WwkJCT7tCQkJKioqqnab7du365133pHb7daiRYs0adIkPfnkk3r88cfPeJzs7GzFxMR4F5fLVeMaAQBA8PE73ISFhcnpPLFZfHy8CgoKJEkxMTEqLCwMbHWn8Hg8io+P1wsvvKDu3btrwIABmjhxoubMmXPGbSZMmKCysjLvUtc1AgAAa/l9Q3G3bt20evVqtW/fXldffbWysrJUWlqq119/XSkpKTXeT1xcnEJCQlRcXOzTXlxcrMTExGq3admypcLCwhQSEuJtu/TSS1VUVKSqqiqFh4eftk1ERIQiIiJqXBcAAAhufp+5mTp1qlq2bClJmjJlipo1a6a7775be/fu1fPPP1/j/YSHh6t79+7Kycnxtnk8HuXk5CgtLa3aba688kp988033jl3JOmrr75Sy5Ytqw02AADg/GPpW8EXLFigIUOG6Pnnn1evXr00c+ZMvfXWW9q6dasSEhKUmZmp1q1bKzs7W5JUWFioTp06aciQIbr33nv19ddf6/e//71GjhypiRMn1uiYPC0FAEDw8efz+5zmuanOunXrlJWVpYULF9Z4mwEDBmjv3r3KyspSUVGRunbtqsWLF3tvMi4oKPDe3yNJLpdLS5Ys0ZgxY3TZZZepdevWGjVqlB588MFADQMAAAQ5v87cLFmyREuXLlV4eLjuvPNOtWvXTlu3btX48eP1wQcfKCMjo8HPFsyZGwAAgk+dnLmZO3euhg0bpubNm+uHH37QSy+9pKeeekr33nuvBgwYoI0bN+rSSy+tdfEAAAC1UeMbimfNmqXp06ertLRUb731lkpLS/Xss89qw4YNmjNnDsEGAAA0CDW+LNW4cWNt2rRJSUlJMsYoIiJCy5cv15VXXlnXNQYUl6UAAAg+dfL6hSNHjigqKkqS5HA4FBER4X0kHAAAoKHw62mpl156SU2aNJEkHT9+XPPmzVNcXJxPH3/eCg4AABBoNb4slZSUJIfDcfadORx+vRXcClyWAgAg+NTJ01I7d+6sbV0AAAB1zu/XLwAAADRkhBsAAGArhBsAAGArhBsAAGArhBsAAGArfr8VvLy8vNr2kxP7hYeH17ooAACAc+V3uImNjT3rfDdt2rTR7bffrsmTJ8vp5MQQAACoX36Hm3nz5mnixIm6/fbb1atXL0lSXl6eXn31VT388MPau3evnnjiCUVEROihhx4KeMEAAABn43e4efXVV/Xkk0/qtttu87b169dPnTt31vPPP6+cnBy1bdtWU6ZMIdwAAIB65/d1o3//+9/q1q3bae3dunVTbm6uJKl3794qKCiofXUAAAB+8jvcuFwuzZ0797T2uXPnyuVySZL27dunZs2a1b46AAAAP/l9WeqJJ57Qb37zG3344Yfq2bOnJGnNmjXaunWr3nnnHUnS6tWrNWDAgMBWCgAAUAM1fiv4j+3YsUPPP/+8vvrqK0nSJZdcoj/84Q9KSkoKdH0Bx1vBAQAIPv58fp9TuAlmhBsAAIKPP5/ffl+WkqQDBw4oLy9PJSUl8ng8PusyMzPPZZcAAAAB4Xe4+eCDDzRo0CAdOnRI0dHRPhP6ORwOwg1+UtVxj+au+lZvry5Q4Q9H5fZIRtLJ36S6+FqSnA7JY+ruGHVRk0NSWJhTrmaN9OvL2+j3vdspPJTJMQHgbPy+LHXxxRerT58+mjp1qqKiouqqrjrDZSlrZS/arOc/3mF1GUHtD79I1oQ+Ha0uAwDqlT+f337/L+Du3bs1cuTIoAw2sBbBJjCe/3iHshdttroMAGiw/A43GRkZWrNmTV3UAhurOu4h2ATQi5/sUNVxz093BIDzkN/33PTt21f333+/Nm/erM6dOyssLMxn/U033RSw4mAfr+futLoEW/GYEz/TO65qZ3UpANDg+B1uhg0bJkl69NFHT1vncDjkdrtrXxVsZ9f+w1aXYDv8TAGgen6Hm1Mf/QZq4sLm3KMVaPxMAaB6PFOKejE4LcnqEmzF6eBnCgBnUqMzN08//bTuuusuRUZG6umnnz5r35EjRwakMNhLeKhTf/hFMjcVB8iwq5KZ7wYAzqBG89wkJydrzZo1atGihZKTk8+8M4dD27dvD2iBgcY8N9bicfDaY54bAOcj3i11FoQb6zFDcc1rYoZiADiBcHMWhBsAAIJPnb440+12a968ecrJyan2xZnLli3zd5cAAAAB43e4GTVqlObNm6e+ffsqJSXF58WZAAAAVvM73MyfP19vvfWW+vTpUxf1AAAA1IrfdyaGh4froosuqotaAAAAas3vcDNu3DjNmjVL59l9yAAAIEj4fVlq1apVWr58uT788EN16tTptBdnvvvuuwErDgAAwF9+h5vY2FjdcsstdVELAABArfkVbo4fP65rrrlG119/vRITE+uqJgAAgHPm1z03oaGhGj58uCorK+uqHgAAgFrx+4biXr16af369XVRCwAAQK35fc/NH//4R40bN07fffedunfvrsaNG/usv+yyywJWHAAAgL/8freU03n6yR6HwyFjjBwOh9xud8CKqwu8WwoAgOBTp++W2rFjxzkXBgAAUNf8DjcXXnhhXdQBAAAQEH6Hm5M2b96sgoICVVVV+bTfdNNNtS4KAADgXPkdbrZv365bbrlFGzZs8N5rI8n7dvCGfs8NAACwN78fBR81apSSk5NVUlKiqKgobdq0SR9//LF69OihFStW1EGJAAAANef3mZvc3FwtW7ZMcXFxcjqdcjqd6t27t7KzszVy5EjmwAEAAJby+8yN2+1W06ZNJUlxcXH6/vvvJZ240Xjbtm2BrQ4AAMBPfp+5SUlJ0RdffKHk5GSlpqZqxowZCg8P1wsvvKB27drVRY0AAAA15ne4efjhh1VRUSFJevTRR/WrX/1KV111lVq0aKEFCxYEvEAAAAB/+D1DcXX279+vZs2aeZ+YasiYoRgAgODjz+e33/fcnPTNN99oyZIlOnLkiJo3b36uuwEAAAgov8PNvn37dO211+riiy9Wnz59tGfPHknSHXfcoXHjxgW8QAAAAH/4HW7GjBmjsLAwFRQUKCoqyts+YMAALV68+JyKmD17tpKSkhQZGanU1FTl5eXVaLv58+fL4XCof//+53RcAABgP36Hm//7v//T9OnT1aZNG5/29u3ba9euXX4XsGDBAo0dO1aTJ0/WunXr1KVLF2VkZKikpOSs2+3cuVP33XefrrrqKr+PCQAA7MvvcFNRUeFzxuak/fv3KyIiwu8CnnrqKQ0bNkxDhw5Vx44dNWfOHEVFRenll18+4zZut1uDBg3Sn/70Jx4/BwAAPvwON1dddZVee+017/cOh0Mej0czZszQNddc49e+qqqqtHbtWqWnp/+nIKdT6enpys3NPeN2jz76qOLj43XHHXf85DEqKytVXl7uswAAAPvye56bGTNm6Nprr9WaNWtUVVWlBx54QJs2bdL+/fv16aef+rWv0tJSud1uJSQk+LQnJCRo69at1W6zatUqzZ07V/n5+TU6RnZ2tv70pz/5VRcAAAhefp+5SUlJ0VdffaXevXvr5ptvVkVFhW699VatX79eP/vZz+qiRq+DBw9q8ODBevHFFxUXF1ejbSZMmKCysjLvUlhYWKc1AgAAa/l95kaSYmJiNHHiRJ+27777TnfddZdeeOGFGu8nLi5OISEhKi4u9mkvLi5WYmLiaf2//fZb7dy5U/369fO2eTweSVJoaKi2bdt2WsCKiIg4p3uBAABAcDrnSfxOtW/fPs2dO9evbcLDw9W9e3fl5OR42zwej3JycpSWlnZa/w4dOmjDhg3Kz8/3LjfddJOuueYa5efny+Vy1XocAAAguJ3TmZtAGjt2rIYMGaIePXqoV69emjlzpioqKjR06FBJUmZmplq3bq3s7GxFRkYqJSXFZ/vY2FhJOq0dAACcnywPNwMGDNDevXuVlZWloqIide3aVYsXL/beZFxQUCCnM2AnmAAAgM0F5MWZkvTFF1/o8ssvl9vtDsTu6gwvzgQAIPj48/ld4zM3t95661nXHzhwoKa7AgAAqDM1DjcxMTE/uT4zM7PWBQEAANRGjcPNK6+8Upd1AAAABAR36gIAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsh3AAAAFsJtboAAGfm9hh9vKVEM/5vi3bsrVCV5z/rjCRHgL+WJKdD8pi6O0Zd1SRJ4WFOxTYK0zWXxCurXyc1Cg8RgPOPwxhjrC6iPpWXlysmJkZlZWWKjo62uhzgjBZv3KN73lyv457z6j/RgLquY7xezOxpdRkAAsCfz28uSwEN0OKNezT8b+sINrW0dHOJhr222uoyANQzwg3QwLg9RpP+8aXVZdjG0s0lOlLltroMAPWIcAM0MHk79mtvxXGry7CVqYs2W10CgHpEuAEamJKDR60uwXZ27jtsdQkA6hHhBmhg4ptGWl2C7SS1iLK6BAD1iHADNDC9kpvrgsbM0hBID/XpaHUJAOoR4QZoYEKcDj12y2VWl2Eb13WMZ74b4DxDuAEaoBtSWmrO7y5XqNPx051xRsxzA5yfOPcNNFA3pLTUtscTmaG4hjVJzFAM4ATCDdCAhTgduqZTgq7plGB1KQAQNLgsBQAAbIVwAwAAbIVwAwAAbIVwAwAAbIVwAwAAbIVwAwAAbIVwAwAAbKVBhJvZs2crKSlJkZGRSk1NVV5e3hn7vvjii7rqqqvUrFkzNWvWTOnp6WftDwAAzi+Wh5sFCxZo7Nixmjx5statW6cuXbooIyNDJSUl1fZfsWKFBg4cqOXLlys3N1cul0vXX3+9du/eXc+VAwCAhshhjDFWFpCamqqePXvqmWeekSR5PB65XC7de++9Gj9+/E9u73a71axZMz3zzDPKzMw8bX1lZaUqKyu935eXl8vlcqmsrEzR0dGBGwgAAKgz5eXliomJqdHnt6VnbqqqqrR27Vqlp6d725xOp9LT05Wbm1ujfRw+fFjHjh1T8+bNq12fnZ2tmJgY7+JyuQJSOwAAaJgsDTelpaVyu91KSPB9b05CQoKKiopqtI8HH3xQrVq18glIPzZhwgSVlZV5l8LCwlrXDQAAGq6gfnHmtGnTNH/+fK1YsUKRkZHV9omIiFBEREQ9VwYAAKxiabiJi4tTSEiIiouLfdqLi4uVmJh41m2feOIJTZs2TR999JEuu+yyuiwTAAAEEUsvS4WHh6t79+7Kycnxtnk8HuXk5CgtLe2M282YMUOPPfaYFi9erB49etRHqQAAIEhYfllq7NixGjJkiHr06KFevXpp5syZqqio0NChQyVJmZmZat26tbKzsyVJ06dPV1ZWlt58800lJSV5781p0qSJmjRpYtk4AABAw2B5uBkwYID27t2rrKwsFRUVqWvXrlq8eLH3JuOCggI5nf85wfTcc8+pqqpK//3f/+2zn8mTJ+uRRx6pz9IBAEADZPk8N/XNn+fkAQBAwxA089wAAAAEGuEGAADYCuEGAADYCuEGAADYCuEGAADYCuEGAADYCuEGAADYCuEGAADYCuEGAADYCuEGAADYCuEGAADYCuEGAADYCuEGAADYSqjVBQBAILg9Rqu27dVzK7/Wpu/LVFFlJElGkuP/9wnU15LkdEgeE9j91md9IU6pSWSYMjolanK/TmoUHiLALhzGGGN1EfXJn1emAwgOizfu0cj5+ao67rG6lKB1Xcd4vZjZ0+oygDPy5/Oby1IAgtrijXs0/G/rCDa1tHRziYa9ttrqMoCAINwACFpuj1HWexusLsM2lm4u0ZEqt9VlALVGuAEQtPJ27FfJoWNWl2ErUxdttroEoNYINwCCVsnBo1aXYDs79x22ugSg1gg3AIJWfNNIq0uwnaQWUVaXANQa4QZA0OqV3FzxTcKsLsNWHurT0eoSgFoj3AAIWiFOhx7t39nqMmzjuo7xzHcDWyDcAAhqN6S01JzfXa7wUP6c1Qbz3MBOmKEYQNC7IaWltjyayAzFftTHDMWwM8INAFsIcTp09aXxuvrSeKtLAWAxzuMCAABbIdwAAABbIdwAAABbIdwAAABbIdwAAABbIdwAAABbIdwAAABbIdwAAABbIdwAAABbIdwAAABb4fULAHAeOlLlVtb7X2rJhiIdqvQ0iPdd1ffXNa3J6ZSaRoapQ2JTDb/6Z+rd/gKFOH+8FzQ0hBsAOM8Me221lm4uOa3d1PHXnh9/Uw/HC1RNHo/0w+Fjyt2+X7nb9ysi1KlZv+2qG1JaCg0Tl6UA4DxypmCDmqs87tHwv63T4o17rC4FZ0C4AYDzxJEqN8EmgB55f7Pcp576QYNAuAGA88TURZutLsFWisqPKm/HfqvLQDUINwBwnti577DVJdhOycGjVpeAahBuAOA8kdQiyuoSbCe+aaTVJaAahBsAOE881Kej1SXYSmJ0pHolN7e6DFSDcAMA54lG4SG6rmO81WXYxiM3dWS+mwaKcAMA55EXM3sScGopItSpOb+7nHluGjAm8QOA88yLmT2ZodiPmpihOPg4jDHn1UP65eXliomJUVlZmaKjo60uBwDQwB2pcuuRDzZo2eZiHTh8XO46DGhS8IZAIynEeeLM1oUtGuu+6zvo6ksCFwT9+fwm3AAAcAbM6Fw7YSEO/XVgt4BcwvPn85t7bgAAqAbBpvaOuY0lr6og3AAAcApeVRFY9f2qCsINAACn4FUVgVXfr6og3AAAcApeVRF49fmqCsINAACn4FUVgVefr6og3AAAcApeVRFY9f2qCsINAACn4FUVgVXfr6og3AAAUA1eVVF7YSEOS15VwesXAAA4g5OvqmCGYutnKPYH4QYAgLNoFB6i6b/uKv3a6kpQUw3istTs2bOVlJSkyMhIpaamKi8v76z93377bXXo0EGRkZHq3LmzFi1aVE+VAgCAhs7ycLNgwQKNHTtWkydP1rp169SlSxdlZGSopKT6mSH//e9/a+DAgbrjjju0fv169e/fX/3799fGjRvruXIAANAQWf7izNTUVPXs2VPPPPOMJMnj8cjlcunee+/V+PHjT+s/YMAAVVRUaOHChd62n//85+ratavmzJnzk8fjxZkAAASfoHlxZlVVldauXav09HRvm9PpVHp6unJzc6vdJjc316e/JGVkZJyxf2VlpcrLy30WAABgX5aGm9LSUrndbiUkJPi0JyQkqKioqNptioqK/OqfnZ2tmJgY7+JyuQJTPAAAaJAsv+emrk2YMEFlZWXepbCw0OqSAABAHbL0UfC4uDiFhISouLjYp724uFiJiYnVbpOYmOhX/4iICEVERASmYAAA0OBZeuYmPDxc3bt3V05OjrfN4/EoJydHaWlp1W6Tlpbm01+Sli5desb+AADg/GL5JH5jx47VkCFD1KNHD/Xq1UszZ85URUWFhg4dKknKzMxU69atlZ2dLUkaNWqUrr76aj355JPq27ev5s+frzVr1uiFF16wchgAAKCBsDzcDBgwQHv37lVWVpaKiorUtWtXLV682HvTcEFBgZzO/5xguuKKK/Tmm2/q4Ycf1kMPPaT27dvrvffeU0pKSo2Od/LJd56aAgAgeJz83K7JDDaWz3NT37777juemAIAIEgVFhaqTZs2Z+1z3oUbj8ej77//Xk2bNpXDEdiXeZWXl8vlcqmwsPC8mCCQ8dob47U3xmt/dhuzMUYHDx5Uq1atfK7oVMfyy1L1zel0/mTiq63o6Ghb/CLVFOO1N8Zrb4zX/uw05piYmBr1s/08NwAA4PxCuAEAALZCuAmgiIgITZ48+byZNJDx2hvjtTfGa3/n45hPOu9uKAYAAPbGmRsAAGArhBsAAGArhBsAAGArhBsAAGArhJsAmT17tpKSkhQZGanU1FTl5eVZXdI5yc7OVs+ePdW0aVPFx8erf//+2rZtm0+fo0ePasSIEWrRooWaNGmiX//61youLvbpU1BQoL59+yoqKkrx8fG6//77dfz48focit+mTZsmh8Oh0aNHe9vsONbdu3frd7/7nVq0aKFGjRqpc+fOWrNmjXe9MUZZWVlq2bKlGjVqpPT0dH399dc++9i/f78GDRqk6OhoxcbG6o477tChQ4fqeyg/ye12a9KkSUpOTlajRo30s5/9TI899pjPu2mCebwff/yx+vXrp1atWsnhcOi9997zWR+osX355Ze66qqrFBkZKZfLpRkzZtT10Kp1tvEeO3ZMDz74oDp37qzGjRurVatWyszM1Pfff++zj2Aar/TT/8Y/Nnz4cDkcDs2cOdOnPdjGHBAGtTZ//nwTHh5uXn75ZbNp0yYzbNgwExsba4qLi60uzW8ZGRnmlVdeMRs3bjT5+fmmT58+pm3btubQoUPePsOHDzcul8vk5OSYNWvWmJ///Ofmiiuu8K4/fvy4SUlJMenp6Wb9+vVm0aJFJi4uzkyYMMGKIdVIXl6eSUpKMpdddpkZNWqUt91uY92/f7+58MILze23324+//xzs337drNkyRLzzTffePtMmzbNxMTEmPfee8988cUX5qabbjLJycnmyJEj3j433HCD6dKli/nss8/MJ598Yi666CIzcOBAK4Z0VlOmTDEtWrQwCxcuNDt27DBvv/22adKkiZk1a5a3TzCPd9GiRWbixInm3XffNZLMP/7xD5/1gRhbWVmZSUhIMIMGDTIbN240f//7302jRo3M888/X1/D9DrbeA8cOGDS09PNggULzNatW01ubq7p1auX6d69u88+gmm8xvz0v/FJ7777runSpYtp1aqV+ctf/uKzLtjGHAiEmwDo1auXGTFihPd7t9ttWrVqZbKzsy2sKjBKSkqMJLNy5UpjzIk/IGFhYebtt9/29tmyZYuRZHJzc40xJ/5jdDqdpqioyNvnueeeM9HR0aaysrJ+B1ADBw8eNO3btzdLly41V199tTfc2HGsDz74oOndu/cZ13s8HpOYmGj+/Oc/e9sOHDhgIiIizN///ndjjDGbN282kszq1au9fT788EPjcDjM7t276674c9C3b1/z+9//3qft1ltvNYMGDTLG2Gu8p37wBWpszz77rGnWrJnP7/ODDz5oLrnkkjoe0dmd7YP+pLy8PCPJ7Nq1yxgT3OM15sxj/u6770zr1q3Nxo0bzYUXXugTboJ9zOeKy1K1VFVVpbVr1yo9Pd3b5nQ6lZ6ertzcXAsrC4yysjJJUvPmzSVJa9eu1bFjx3zG26FDB7Vt29Y73tzcXHXu3FkJCQnePhkZGSovL9emTZvqsfqaGTFihPr27eszJsmeY33//ffVo0cP/eY3v1F8fLy6deumF1980bt+x44dKioq8hlzTEyMUlNTfcYcGxurHj16ePukp6fL6XTq888/r7/B1MAVV1yhnJwcffXVV5KkL774QqtWrdKNN94oyX7j/bFAjS03N1e/+MUvFB4e7u2TkZGhbdu26Ycffqin0ZybsrIyORwOxcbGSrLneD0ejwYPHqz7779fnTp1Om29HcdcE4SbWiotLZXb7fb5cJOkhIQEFRUVWVRVYHg8Ho0ePVpXXnmlUlJSJElFRUUKDw/3/rE46cfjLSoqqvbncXJdQzJ//nytW7dO2dnZp62z21glafv27XruuefUvn17LVmyRHfffbdGjhypV199VdJ/aj7b73NRUZHi4+N91oeGhqp58+YNbszjx4/Xb3/7W3Xo0EFhYWHq1q2bRo8erUGDBkmy33h/LFBjC7bf8ZOOHj2qBx98UAMHDvS+NNKO450+fbpCQ0M1cuTIatfbccw1cd69FRw1N2LECG3cuFGrVq2yupQ6UVhYqFGjRmnp0qWKjIy0upx64fF41KNHD02dOlWS1K1bN23cuFFz5szRkCFDLK4u8N566y298cYbevPNN9WpUyfl5+dr9OjRatWqlS3HixOOHTum2267TcYYPffcc1aXU2fWrl2rWbNmad26dXI4HFaX06Bw5qaW4uLiFBISctoTNMXFxUpMTLSoqtq75557tHDhQi1fvlxt2rTxticmJqqqqkoHDhzw6f/j8SYmJlb78zi5rqFYu3atSkpKdPnllys0NFShoaFauXKlnn76aYWGhiohIcE2Yz2pZcuW6tixo0/bpZdeqoKCAkn/qflsv8+JiYkqKSnxWX/8+HHt37+/wY35/vvv95696dy5swYPHqwxY8Z4z9TZbbw/FqixBdvv+Mlgs2vXLi1dutR71kay33g/+eQTlZSUqG3btt6/Ybt27dK4ceOUlJQkyX5jrinCTS2Fh4ere/fuysnJ8bZ5PB7l5OQoLS3NwsrOjTFG99xzj/7xj39o2bJlSk5O9lnfvXt3hYWF+Yx327ZtKigo8I43LS1NGzZs8PkP6uQfmVM/WK107bXXasOGDcrPz/cuPXr00KBBg7xf22WsJ1155ZWnPdr/1Vdf6cILL5QkJScnKzEx0WfM5eXl+vzzz33GfODAAa1du9bbZ9myZfJ4PEpNTa2HUdTc4cOH5XT6/pkLCQmRx+ORZL/x/ligxpaWlqaPP/5Yx44d8/ZZunSpLrnkEjVr1qyeRlMzJ4PN119/rY8++kgtWrTwWW+38Q4ePFhffvmlz9+wVq1a6f7779eSJUsk2W/MNWb1Hc12MH/+fBMREWHmzZtnNm/ebO666y4TGxvr8wRNsLj77rtNTEyMWbFihdmzZ493OXz4sLfP8OHDTdu2bc2yZcvMmjVrTFpamklLS/OuP/l49PXXX2/y8/PN4sWLzQUXXNBgH4/+sR8/LWWM/caal5dnQkNDzZQpU8zXX39t3njjDRMVFWX+9re/eftMmzbNxMbGmn/+85/myy+/NDfffHO1jw9369bNfP7552bVqlWmffv2DeLR6FMNGTLEtG7d2vso+Lvvvmvi4uLMAw884O0TzOM9ePCgWb9+vVm/fr2RZJ566imzfv1679NBgRjbgQMHTEJCghk8eLDZuHGjmT9/vomKirLkMeGzjbeqqsrcdNNNpk2bNiY/P9/n79ePnwIKpvEa89P/xqc69WkpY4JvzIFAuAmQv/71r6Zt27YmPDzc9OrVy3z22WdWl3ROJFW7vPLKK94+R44cMX/84x9Ns2bNTFRUlLnlllvMnj17fPazc+dOc+ONN5pGjRqZuLg4M27cOHPs2LF6Ho3/Tg03dhzrBx98YFJSUkxERITp0KGDeeGFF3zWezweM2nSJJOQkGAiIiLMtddea7Zt2+bTZ9++fWbgwIGmSZMmJjo62gwdOtQcPHiwPodRI+Xl5WbUqFGmbdu2JjIy0rRr185MnDjR58MumMe7fPnyav97HTJkiDEmcGP74osvTO/evU1ERIRp3bq1mTZtWn0N0cfZxrtjx44z/v1avny5dx/BNF5jfvrf+FTVhZtgG3MgOIz50VSdAAAAQY57bgAAgK0QbgAAgK0QbgAAgK0QbgAAgK0QbgAAgK0QbgAAgK0QbgAAgK0QbgAAgK0QbgCcd5KSkjRz5kyrywBQRwg3AOrU7bffrv79+0uSfvnLX2r06NH1dux58+YpNjb2tPbVq1frrrvuqrc6ANSvUKsLAAB/VVVVKTw8/Jy3v+CCCwJYDYCGhjM3AOrF7bffrpUrV2rWrFlyOBxyOBzauXOnJGnjxo268cYb1aRJEyUkJGjw4MEqLS31bvvLX/5S99xzj0aPHq24uDhlZGRIkp566il17txZjRs3lsvl0h//+EcdOnRIkrRixQoNHTpUZWVl3uM98sgjkk6/LFVQUKCbb75ZTZo0UXR0tG677TYVFxd71z/yyCPq2rWrXn/9dSUlJSkmJka//e1vdfDgwbr9oQE4J4QbAPVi1qxZSktL07Bhw7Rnzx7t2bNHLpdLBw4c0H/913+pW7duWrNmjRYvXqzi4mLddtttPtu/+uqrCg8P16effqo5c+ZIkpxOp55++mlt2rRJr776qpYtW6YHHnhAknTFFVdo5syZio6O9h7vvvvuO60uj8ejm2++Wfv379fKlSu1dOlSbd++XQMGDPDp9+233+q9997TwoULtXDhQq1cuVLTpk2ro58WgNrgshSAehETE6Pw8HBFRUUpMTHR2/7MM8+oW7dumjp1qrft5Zdflsvl0ldffaWLL75YktS+fXvNmDHDZ58/vn8nKSlJjz/+uIYPH65nn31W4eHhiomJkcPh8DneqXJycrRhwwbt2LFDLpdLkvTaa6+pU6dOWr16tXr27CnpRAiaN2+emjZtKkkaPHiwcnJyNGXKlNr9YAAEHGduAFjqiy++0PLly9WkSRPv0qFDB0knzpac1L1799O2/eijj3TttdeqdevWatq0qQYPHqx9+/bp8OHDNT7+li1b5HK5vMFGkjp27KjY2Fht2bLF25aUlOQNNpLUsmVLlZSU+DVWAPWDMzcALHXo0CH169dP06dPP21dy5YtvV83btzYZ93OnTv1q1/9SnfffbemTJmi5s2ba9WqVbrjjjtUVVWlqKiogNYZFhbm873D4ZDH4wnoMQAEBuEGQL0JDw+X2+32abv88sv1v//7v0pKSlJoaM3/JK1du1Yej0dPPvmknM4TJ6HfeuutnzzeqS699FIVFhaqsLDQe/Zm8+bNOnDggDp27FjjegA0HFyWAlBvkpKS9Pnnn2vnzp0qLS2Vx+PRiBEjtH//fg0cOFCrV6/Wt99+qyVLlmjo0KFnDSYXXXSRjh07pr/+9a/avn27Xn/9de+Nxj8+3qFDh5STk6PS0tJqL1elp6erc+fOGjRokNatW6e8vDxlZmbq6quvVo8ePQL+MwBQ9wg3AOrNfffdp5CQEHXs2FEXXHCBCgoK1KpVK3366adyu926/vrr1blzZ40ePVqxsbHeMzLV6dKli5566ilNnz5dKSkpeuONN5Sdne3T54orrtDw4cM1YMAAXXDBBafdkCyduLz0z3/+U82aNdMvfvELpaenq127dlqwYEHAxw+gfjiMMcbqIgAAAAKFMzcAAMBWCDcAAMBWCDcAAMBWCDcAAMBWCDcAAMBWCDcAAMBWCDcAAMBWCDcAAMBWCDcAAMBWCDcAAMBWCDcAAMBW/h97/vl6w5/I8wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "schedule = mx.lr_scheduler.FactorScheduler(step=250, factor=0.5)\n",
    "schedule.base_lr = 1\n",
    "plot_schedule(schedule)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d344657",
   "metadata": {},
   "source": [
    "![lr factor](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/lr_schedules/factor.png) <!--notebook-skip-line-->\n",
    "\n",
    "\n",
    "Note: the `base_lr` is used to determine the initial learning rate. It takes a default value of 0.01 since we inherit from `mx.lr_scheduler.LRScheduler`, but it can be set as a property of the schedule. We will see later in this tutorial that `base_lr` is set automatically when providing the `lr_schedule` to `Optimizer`. Also be aware that the schedules in `mx.lr_scheduler` have state (i.e. counters, etc) so calling the schedule out of order may give unexpected results.\n",
    "\n",
    "We can define non-uniform intervals with `MultiFactorScheduler` and in the example below we halve the learning rate _after_ the 250th, 750th (i.e. a step length of 500 iterations) and 900th (a step length of 150 iterations). As before, the learning rate of the 250th iteration will be 1 and the 251th iteration will be 0.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "03bb8321",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAvzElEQVR4nO3deVyVdd7/8fdhFxVQCXBBwXJcwszchqxpvKOcNG2578lxHDWnsdGs3DJTU5spxawcrSzLLLWp0ep2mla9DZeyoXCj3K1cMBMQCVBcUM7394c/z3hyiSMHLvjyej4e5/GA77V9Poic9+O6vtd1XMYYIwAAAEsEOF0AAACAPxFuAACAVQg3AADAKoQbAABgFcINAACwCuEGAABYhXADAACsEuR0AZXN7Xbrhx9+UN26deVyuZwuBwAAlIExRocPH1ajRo0UEHDxczM1Ltz88MMPio+Pd7oMAABwCfbt26cmTZpcdJ0aF27q1q0r6fQPJyIiwuFqAABAWRQVFSk+Pt7zPn4xNS7cnLkUFRERQbgBAKCaKcuUEiYUAwAAqxBuAACAVQg3AADAKoQbAABgFcINAACwCuEGAABYhXADAACsQrgBAABWIdwAAACr1LgnFFeUYyWleuz9TVqxNUcFR0+p1JweN5LOPEuxIr6WpACX5K6k4/mjpsAAKTQoQM0a1NZDN7fSDS0vU2AAH2IKAPAPR8/cfPrpp+rVq5caNWokl8uld99992e3WbVqla655hqFhobqiiuu0Pz58yu8zp8zeOFatZ60VIvX7tfB4lM6aSS3Tr+MKvZrt6RTlXg8f9R0yi0Vl7i19cBh/XHBWrWa+LGWbj5Qrn8DAADOcDTcFBcXq127dpo9e3aZ1t+9e7d69uypbt26KTMzUyNGjNCf/vQnLVu2rIIrvbDBC9dq+dZcx45vg5OlRkP+voGAAwDwC0cvS91yyy265ZZbyrz+nDlzlJiYqGeeeUaS1Lp1a61Zs0Z/+9vf1L1794oq84KOlZQSbPzosfe26qY2cVyiAgCUS7WaUJyenq6UlBSvse7duys9Pf2C25w4cUJFRUVeL3+Z+tFWv+0LUnbRcWXszne6DABANVetwk12drZiY2O9xmJjY1VUVKRjx46dd5vU1FRFRkZ6XvHx8X6rZ8+ho37bF07LPXzc6RIAANVctQo3l2LcuHEqLCz0vPbt2+e3fSc0CPfbvnBaTN0wp0sAAFRz1SrcxMXFKScnx2ssJydHERERqlWr1nm3CQ0NVUREhNfLX8b3aOO3fUGKiwhT58T6TpcBAKjmqlW4SU5OVlpamtfY8uXLlZyc7Eg9tUICdVObGEeObaPHerdhMjEAoNwcDTdHjhxRZmamMjMzJZ2+1TszM1NZWVmSTl9SGjBggGf9IUOGaNeuXXr44Ye1fft2vfDCC3rrrbc0cuRIJ8qXJM0d0ImAU07BgS7N+cM1+k1SQ6dLAQBYwNFbwdetW6du3bp5vh81apQkaeDAgZo/f74OHDjgCTqSlJiYqA8//FAjR47UrFmz1KRJE73yyiuO3AZ+trkDOvGEYh9q4gnFAICK5DLGGKeLqExFRUWKjIxUYWGhX+ffAACAiuPL+3e1mnMDAADwcwg3AADAKoQbAABgFcINAACwCuEGAABYhXADAACsQrgBAABWIdwAAACrEG4AAIBVCDcAAMAqhBsAAGAVwg0AALAK4QYAAFiFcAMAAKxCuAEAAFYh3AAAAKsQbgAAgFUINwAAwCqEGwAAYBXCDQAAsArhBgAAWIVwAwAArEK4AQAAViHcAAAAqxBuAACAVQg3AADAKoQbAABgFcINAACwCuEGAABYhXADAACsQrgBAABWIdwAAACrEG4AAIBVCDcAAMAqhBsAAGAVwg0AALAK4QYAAFiFcAMAAKxCuAEAAFYh3AAAAKsQbgAAgFUINwAAwCqEGwAAYBXCDQAAsArhBgAAWIVwAwAArEK4AQAAViHcAAAAqxBuAACAVQg3AADAKoQbAABgFcINAACwCuEGAABYhXADAACsQrgBAABWIdwAAACrEG4AAIBVCDcAAMAqhBsAAGAVwg0AALAK4QYAAFiFcAMAAKxCuAEAAFYh3AAAAKsQbgAAgFUINwAAwCqOh5vZs2crISFBYWFh6tKlizIyMi66/syZM9WyZUvVqlVL8fHxGjlypI4fP15J1QIAgKrO0XCzePFijRo1SpMnT9aGDRvUrl07de/eXbm5uedd/80339QjjzyiyZMna9u2bZo3b54WL16s8ePHV3LlAACgqnI03MyYMUODBw/WoEGD1KZNG82ZM0fh4eF69dVXz7v+v//9b3Xt2lW///3vlZCQoJtvvll9+/b92bM9AACg5nAs3JSUlGj9+vVKSUn5TzEBAUpJSVF6evp5t7n22mu1fv16T5jZtWuXPvroI/Xo0eOCxzlx4oSKioq8XgAAwF5BTh04Ly9PpaWlio2N9RqPjY3V9u3bz7vN73//e+Xl5em6666TMUanTp3SkCFDLnpZKjU1VX/5y1/8WjsAAKi6HJ9Q7ItVq1Zp6tSpeuGFF7RhwwYtWbJEH374oR5//PELbjNu3DgVFhZ6Xvv27avEigEAQGVz7MxNdHS0AgMDlZOT4zWek5OjuLi4824zceJE9e/fX3/6058kSW3btlVxcbHuvfdeTZgwQQEB52a10NBQhYaG+r8BAABQJTl25iYkJEQdOnRQWlqaZ8ztdistLU3Jycnn3ebo0aPnBJjAwEBJkjGm4ooFAADVhmNnbiRp1KhRGjhwoDp27KjOnTtr5syZKi4u1qBBgyRJAwYMUOPGjZWamipJ6tWrl2bMmKH27durS5cu+vbbbzVx4kT16tXLE3IAAEDN5mi46dOnjw4ePKhJkyYpOztbV199tZYuXeqZZJyVleV1pubRRx+Vy+XSo48+qv379+uyyy5Tr169NGXKFKdaAAAAVYzL1LDrOUVFRYqMjFRhYaEiIiKcLgcAAJSBL+/f1epuKQAAgJ9DuAEAAFYh3AAAAKsQbgAAgFUINwAAwCqEGwAAYBXCDQAAsArhBgAAWIVwAwAArEK4AQAAViHcAAAAqxBuAACAVQg3AADAKoQbAABgFcINAACwCuEGAABYhXADAACsQrgBAABWIdwAAACrEG4AAIBVCDcAAMAqhBsAAGAVwg0AALAK4QYAAFiFcAMAAKxCuAEAAFYh3AAAAKsQbgAAgFUINwAAwCqEGwAAYBXCDQAAsArhBgAAWIVwAwAArEK4AQAAViHcAAAAqxBuAACAVQg3AADAKuUKN8ePH/dXHQAAAH7hc7hxu916/PHH1bhxY9WpU0e7du2SJE2cOFHz5s3ze4EAAAC+8DncPPHEE5o/f76mT5+ukJAQz3hSUpJeeeUVvxYHAADgK5/DzcKFC/Xyyy+rX79+CgwM9Iy3a9dO27dv92txAAAAvvI53Ozfv19XXHHFOeNut1snT570S1EAAACXyudw06ZNG3322WfnjL/zzjtq3769X4oCAAC4VEG+bjBp0iQNHDhQ+/fvl9vt1pIlS7Rjxw4tXLhQH3zwQUXUCAAAUGY+n7m57bbb9P777+uTTz5R7dq1NWnSJG3btk3vv/++brrppoqoEQAAoMxcxhjjdBGVqaioSJGRkSosLFRERITT5QAAgDLw5f3b5zM3zZs316FDh84ZLygoUPPmzX3dHQAAgF/5HG727Nmj0tLSc8ZPnDih/fv3+6UoAACAS1XmCcXvvfee5+tly5YpMjLS831paanS0tKUkJDg1+IAAAB8VeZwc/vtt0uSXC6XBg4c6LUsODhYCQkJeuaZZ/xaHAAAgK/KHG7cbrckKTExUWvXrlV0dHSFFQUAAHCpfH7Oze7duyuiDgAAAL/wOdxIUnFxsVavXq2srCyVlJR4LXvwwQf9UhgAAMCl8DncbNy4UT169NDRo0dVXFys+vXrKy8vT+Hh4YqJiSHcAAAAR/l8K/jIkSPVq1cv/fjjj6pVq5a++OIL7d27Vx06dNDTTz9dETUCAACUmc/hJjMzU6NHj1ZAQIACAwN14sQJxcfHa/r06Ro/fnxF1AgAAFBmPoeb4OBgBQSc3iwmJkZZWVmSpMjISO3bt8+/1QEAAPjI5zk37du319q1a9WiRQvdcMMNmjRpkvLy8vT6668rKSmpImoEAAAoM5/P3EydOlUNGzaUJE2ZMkX16tXT0KFDdfDgQb300kt+LxAAAMAXfCo4AACo8ir0U8EvZMOGDbr11lv9tTsAAIBL4lO4WbZsmR566CGNHz9eu3btkiRt375dt99+uzp16uT5iAYAAACnlHlC8bx58zR48GDVr19fP/74o1555RXNmDFDDzzwgPr06aPNmzerdevWFVkrAADAzyrzmZtZs2bpySefVF5ent566y3l5eXphRde0KZNmzRnzhyCDQAAqBLKPKG4du3a2rJlixISEmSMUWhoqFauXKmuXbtWdI1+xYRiAACqnwqZUHzs2DGFh4dLklwul0JDQz23hAMAAFQVPj3E75VXXlGdOnUkSadOndL8+fMVHR3ttY6vH5w5e/ZsPfXUU8rOzla7du303HPPqXPnzhdcv6CgQBMmTNCSJUuUn5+vZs2aaebMmerRo4dPxwUAAHYq82WphIQEuVyui+/M5fLcRVUWixcv1oABAzRnzhx16dJFM2fO1Ntvv60dO3YoJibmnPVLSkrUtWtXxcTEaPz48WrcuLH27t2rqKgotWvXrkzH5LIUAADVjy/v344+xK9Lly7q1KmTnn/+eUmS2+1WfHy8HnjgAT3yyCPnrD9nzhw99dRT2r59u4KDgy/pmIQbAACqH0ce4uerkpISrV+/XikpKf8pJiBAKSkpSk9PP+827733npKTkzVs2DDFxsYqKSlJU6dOVWlp6QWPc+LECRUVFXm9AACAvRwLN3l5eSotLVVsbKzXeGxsrLKzs8+7za5du/TOO++otLRUH330kSZOnKhnnnlGTzzxxAWPk5qaqsjISM8rPj7er30AAICqxbFwcyncbrdiYmL08ssvq0OHDurTp48mTJigOXPmXHCbcePGqbCw0PPat29fJVYMAAAqm093S/lTdHS0AgMDlZOT4zWek5OjuLi4827TsGFDBQcHKzAw0DPWunVrZWdnq6SkRCEhIedsExoaqtDQUP8WDwAAqizHztyEhISoQ4cOSktL84y53W6lpaUpOTn5vNt07dpV3377rddnWO3cuVMNGzY8b7ABAAA1j8/h5qeTc8+8Dh8+rJKSEp/2NWrUKM2dO1cLFizQtm3bNHToUBUXF2vQoEGSpAEDBmjcuHGe9YcOHar8/HwNHz5cO3fu1IcffqipU6dq2LBhvrYBAAAs5fNlqaioqIs+76ZJkya6++67NXnyZAUEXDw79enTRwcPHtSkSZOUnZ2tq6++WkuXLvVMMs7KyvLaR3x8vJYtW6aRI0fqqquuUuPGjTV8+HCNHTvW1zYAAIClfH7OzcKFCzVhwgTdfffdnicJZ2RkaMGCBXr00Ud18OBBPf300xozZozGjx9fIUWXB8+5AQCg+vHl/dvnMzcLFizQM888o7vuussz1qtXL7Vt21YvvfSS0tLS1LRpU02ZMqVKhhsAAGA3n+fc/Pvf/1b79u3PGW/fvr3n4XvXXXedsrKyyl8dAACAj3wON/Hx8Zo3b9454/PmzfM8IO/QoUOqV69e+asDAADwkc+XpZ5++mn99re/1ccff6xOnTpJktatW6ft27frnXfekSStXbtWffr08W+lAAAAZXBJH5y5e/duvfTSS9q5c6ckqWXLlvrzn/+shIQEf9fnd0woBgCg+qk2nwruBMINAADVT4XeLSVJBQUFysjIUG5urtfTgqXTD94DAABwis/h5v3331e/fv105MgRRUREeD3Qz+VyEW4AAICjfL5bavTo0frjH/+oI0eOqKCgQD/++KPnlZ+fXxE1AgAAlJnP4Wb//v168MEHFR4eXhH1AAAAlIvP4aZ79+5at25dRdQCAABQbj7PuenZs6fGjBmjrVu3qm3btgoODvZa3rt3b78VBwAA4CufbwW/2Cd9u1wulZaWlruoisSt4AAAVD8Veiv4T2/9BgAAqEp8nnMDAABQlZXpzM2zzz6re++9V2FhYXr22Wcvuu6DDz7ol8IAAAAuRZnm3CQmJmrdunVq0KCBEhMTL7wzl0u7du3ya4H+xpwbAACqH7/Pudm9e/d5vwYAAKhqmHMDAACs4vPdUqWlpZo/f77S0tLO+8GZK1as8FtxAAAAvvI53AwfPlzz589Xz549lZSU5PXBmQAAAE7zOdwsWrRIb731lnr06FER9QAAAJSLz3NuQkJCdMUVV1RELQAAAOXmc7gZPXq0Zs2aJR8/tQEAAKBS+HxZas2aNVq5cqU+/vhjXXnlled8cOaSJUv8VhwAAICvfA43UVFRuuOOOyqiFgAAgHLzKdycOnVK3bp1080336y4uLiKqgmWKznl1rw13+nttVna9+NxlbolI+nMfXcV8bUkBbgkt6m4Y1CTszVJUkhwgKJqBatbyxhN6nWlaoUECkDNU6aPXzhbeHi4tm3bpmbNmlVUTRWKj19wVupHW/XSpzzlGpXjpjYxmjugk9NlAPADX96/fZ5Q3LlzZ23cuPGSi0PNRbBBZVu+NVeDF651ugwAlcznOTf33XefRo8ere+//14dOnRQ7dq1vZZfddVVfisO9ig55SbYwBHLt+bqWEkpl6iAGsTny1IBAeee7HG5XDLGyOVyqbS01G/FVQQuSzlj3me79PiH25wuAzVU/1821eO3t3W6DADl4PdPBT8bnwqOS7E3/6jTJaAG23OI3z+gJvE53FTXicRwVrP64U6XgBosoQG/f0BN4nO4OWPr1q3KyspSSUmJ13jv3r3LXRTs0z85gctScMz4Hm2cLgFAJfI53OzatUt33HGHNm3a5JlrI8nz6eBVfc4NnBESFKA//yqRScWodDe1iWEyMVDD+Hwr+PDhw5WYmKjc3FyFh4dry5Yt+vTTT9WxY0etWrWqAkqELcb1aKM//yrR6TJQg/CcG6Bm8vluqejoaK1YsUJXXXWVIiMjlZGRoZYtW2rFihUaPXp0lX8GDndLOY8nFFMTTygG4KsKvVuqtLRUdevWlXQ66Pzwww9q2bKlmjVrph07dlxaxahRQoICNPTXLTT01y2cLgUAYCGfw01SUpK++uorJSYmqkuXLpo+fbpCQkL08ssvq3nz5hVRIwAAQJn5HG4effRRFRcXS5L++te/6tZbb9X111+vBg0aaPHixX4vEAAAwBc+z7k5n/z8fNWrV89zx1RVxpwbAACqnwr94Mwzvv32Wy1btkzHjh1T/fr1L3U3AAAAfuVzuDl06JBuvPFG/eIXv1CPHj104MABSdI999yj0aNH+71AAAAAX/gcbkaOHKng4GBlZWUpPPw/jzTv06ePli5d6tfiAAAAfOXzhOL/+7//07Jly9SkSROv8RYtWmjv3r1+KwwAAOBS+Hzmpri42OuMzRn5+fkKDQ31S1EAAACXyudwc/3112vhwoWe710ul9xut6ZPn65u3br5tTgAAABf+XxZavr06brxxhu1bt06lZSU6OGHH9aWLVuUn5+vzz//vCJqBAAAKDOfz9wkJSVp586duu6663TbbbepuLhYd955pzZu3KjLL7+8ImoEAAAoM5/P3EhSZGSkJkyY4DX2/fff695779XLL7/sl8IAAAAuxSU/xO+nDh06pHnz5vlrdwAAAJfEb+EGAACgKiDcAAAAqxBuAACAVco8ofjOO++86PKCgoLy1gIAAFBuZQ43kZGRP7t8wIAB5S4IAACgPMocbl577bWKrAMAAMAvmHMDAACsQrgBAABWIdwAAACrEG4AAIBVCDcAAMAqhBsAAGAVwg0AALAK4QYAAFiFcAMAAKxSJcLN7NmzlZCQoLCwMHXp0kUZGRll2m7RokVyuVy6/fbbK7ZAAABQbTgebhYvXqxRo0Zp8uTJ2rBhg9q1a6fu3bsrNzf3otvt2bNHDz30kK6//vpKqhQAAFQHjoebGTNmaPDgwRo0aJDatGmjOXPmKDw8XK+++uoFtyktLVW/fv30l7/8Rc2bN7/o/k+cOKGioiKvFwAAsJej4aakpETr169XSkqKZywgIEApKSlKT0+/4HZ//etfFRMTo3vuuednj5GamqrIyEjPKz4+3i+1AwCAqsnRcJOXl6fS0lLFxsZ6jcfGxio7O/u826xZs0bz5s3T3Llzy3SMcePGqbCw0PPat29fuesGAABVV5DTBfji8OHD6t+/v+bOnavo6OgybRMaGqrQ0NAKrgwAAFQVjoab6OhoBQYGKicnx2s8JydHcXFx56z/3Xffac+ePerVq5dnzO12S5KCgoK0Y8cOXX755RVbNAAAqNIcvSwVEhKiDh06KC0tzTPmdruVlpam5OTkc9Zv1aqVNm3apMzMTM+rd+/e6tatmzIzM5lPAwAAnL8sNWrUKA0cOFAdO3ZU586dNXPmTBUXF2vQoEGSpAEDBqhx48ZKTU1VWFiYkpKSvLaPioqSpHPGAQBAzeR4uOnTp48OHjyoSZMmKTs7W1dffbWWLl3qmWSclZWlgADH71gHAADVhMsYY5wuojIVFRUpMjJShYWFioiIcLocAABQBr68f3NKBAAAWIVwAwAArEK4AQAAViHcAAAAqxBuAACAVQg3AADAKoQbAABgFcINAACwCuEGAABYhXADAACsQrgBAABWIdwAAACrEG4AAIBVCDcAAMAqhBsAAGAVwg0AALAK4QYAAFiFcAMAAKxCuAEAAFYh3AAAAKsQbgAAgFUINwAAwCqEGwAAYBXCDQAAsArhBgAAWIVwAwAArEK4AQAAViHcAAAAqxBuAACAVQg3AADAKoQbAABgFcINAACwCuEGAABYhXADAACsQrgBAABWIdwAAACrEG4AAIBVCDcAAMAqhBsAAGAVwg0AALAK4QYAAFiFcAMAAKxCuAEAAFYh3AAAAKsQbgAAgFWCnC4AAPyh1G20ZsdBvbj6G235oVDFJUaSZCS5/v86/vpakgJcktuUf1+BgS41qB2ibi1jNKnXlaoVEuhj5wB+ymWMMU4XUZmKiooUGRmpwsJCRUREOF0OAD9YuvmAHlyUqZJTbqdLKbeb2sRo7oBOTpcBVDm+vH9zWQpAtbZ08wEN+fsGK4KNJC3fmqvBC9c6XQZQrRFuAFRbpW6jSe9ucroMv1u+NVfHSkqdLgOotgg3AKqtjN35yj1y0ukyKsTUj7Y6XQJQbRFuAFRbuYePO11Chdlz6KjTJQDVFuEGQLUVUzfM6RIqTEKDcKdLAKotwg2AaqtzYn3F1Al2uowKMb5HG6dLAKotwg2AaiswwKW/3t7W6TL87qY2MTzvBigHwg2Aau03SQ015w/XKCTIjj9nPOcGKD+eUAyg2vtNUkNt+2scTygGIIlwA8ASgQEu3dA6Rje0jnG6FAAOs+M8LgAAwP9HuAEAAFYh3AAAAKsQbgAAgFUINwAAwCqEGwAAYBXCDQAAsArhBgAAWIVwAwAArFIlws3s2bOVkJCgsLAwdenSRRkZGRdcd+7cubr++utVr1491atXTykpKRddHwAA1CyOh5vFixdr1KhRmjx5sjZs2KB27dqpe/fuys3NPe/6q1atUt++fbVy5Uqlp6crPj5eN998s/bv31/JlQMAgKrIZYwxThbQpUsXderUSc8//7wkye12Kz4+Xg888IAeeeSRn92+tLRU9erV0/PPP68BAwacs/zEiRM6ceKE5/uioiLFx8ersLBQERER/msEAABUmKKiIkVGRpbp/dvRMzclJSVav369UlJSPGMBAQFKSUlRenp6mfZx9OhRnTx5UvXr1z/v8tTUVEVGRnpe8fHxfqkdAABUTY6Gm7y8PJWWlio2NtZrPDY2VtnZ2WXax9ixY9WoUSOvgHS2cePGqbCw0PPat29fuesGAABVV5DTBZTHtGnTtGjRIq1atUphYWHnXSc0NFShoaGVXBkAAHCKo+EmOjpagYGBysnJ8RrPyclRXFzcRbd9+umnNW3aNH3yySe66qqrKrJMAABQjTh6WSokJEQdOnRQWlqaZ8ztdistLU3JyckX3G769Ol6/PHHtXTpUnXs2LEySgUAANWE45elRo0apYEDB6pjx47q3LmzZs6cqeLiYg0aNEiSNGDAADVu3FipqamSpCeffFKTJk3Sm2++qYSEBM/cnDp16qhOnTqO9QEAAKoGx8NNnz59dPDgQU2aNEnZ2dm6+uqrtXTpUs8k46ysLAUE/OcE04svvqiSkhL9z//8j9d+Jk+erMcee6wySwcAAFWQ48+5qWy+3CcPAACqhmrznBsAAAB/I9wAAACrEG4AAIBVCDcAAMAqhBsAAGAVwg0AALAK4QYAAFiFcAMAAKxCuAEAAFYh3AAAAKsQbgAAgFUINwAAwCqEGwAAYBXCDQAAsArhBgAAWIVwAwAArBLkdAEAUJMdOX5KD765Thm783X0pPGMG0muS/xakgJcktuUf1/++pqaakZNgQFSaFCAmjWorYdubqUbWl6mwICz91Q5XMYY8/Or2aOoqEiRkZEqLCxURESE0+UAqMF6P/+Zvv6+yOkygAoTHOjSc33b6zdJDcu9L1/ev7ksBQAOINigJjhZajTk7xu0dPOBSj0u4QYAKtmR46cINqhRHntvq0rdlXehiHADAJVs5OKNTpcAVKrsouPK2J1faccj3ABAJcv68ZjTJQCVLvfw8Uo7FuEGACpZ03q1nC4BqHQxdcMq7ViEGwCoZH/r097pEoBKFRcRps6J9SvteIQbAKhkdcKCdFUTHkWBmuOx3m0q9Xk3hBsAcMB7919PwIH1ggNdmvOHa/zynBtf8IRiAHDIe/dfzxOKqcmqmqrKE4oJNwDgoDphQXr1j790ugzAKlyWAgAAViHcAAAAqxBuAACAVQg3AADAKoQbAABgFcINAACwCuEGAABYhXADAACsQrgBAABWqXFPKDbm9POji4qKHK4EAACU1Zn37TPv4xdT48LN4cOHJUnx8fEOVwIAAHx1+PBhRUZGXnQdlylLBLKI2+3WDz/8oLp168rl8u+HeRUVFSk+Pl779u1TRIT9n/ZLv3ajX7vRr/1s69kYo8OHD6tRo0YKCLj4rJoad+YmICBATZo0qdBjREREWPGLVFb0azf6tRv92s+mnn/ujM0ZTCgGAABWIdwAAACrEG78KDQ0VJMnT1ZoaKjTpVQK+rUb/dqNfu1XE3s+o8ZNKAYAAHbjzA0AALAK4QYAAFiFcAMAAKxCuAEAAFYh3PjJ7NmzlZCQoLCwMHXp0kUZGRlOl3RJUlNT1alTJ9WtW1cxMTG6/fbbtWPHDq91jh8/rmHDhqlBgwaqU6eO/vu//1s5OTle62RlZalnz54KDw9XTEyMxowZo1OnTlVmKz6bNm2aXC6XRowY4Rmzsdf9+/frD3/4gxo0aKBatWqpbdu2WrdunWe5MUaTJk1Sw4YNVatWLaWkpOibb77x2kd+fr769euniIgIRUVF6Z577tGRI0cqu5WfVVpaqokTJyoxMVG1atXS5Zdfrscff9zrs2mqc7+ffvqpevXqpUaNGsnlcundd9/1Wu6v3r7++mtdf/31CgsLU3x8vKZPn17RrZ3Xxfo9efKkxo4dq7Zt26p27dpq1KiRBgwYoB9++MFrH9WpX+nn/43PNmTIELlcLs2cOdNrvLr17BcG5bZo0SITEhJiXn31VbNlyxYzePBgExUVZXJycpwuzWfdu3c3r732mtm8ebPJzMw0PXr0ME2bNjVHjhzxrDNkyBATHx9v0tLSzLp168wvf/lLc+2113qWnzp1yiQlJZmUlBSzceNG89FHH5no6Ggzbtw4J1oqk4yMDJOQkGCuuuoqM3z4cM+4bb3m5+ebZs2ambvvvtt8+eWXZteuXWbZsmXm22+/9awzbdo0ExkZad59913z1Vdfmd69e5vExERz7Ngxzzq/+c1vTLt27cwXX3xhPvvsM3PFFVeYvn37OtHSRU2ZMsU0aNDAfPDBB2b37t3m7bffNnXq1DGzZs3yrFOd+/3oo4/MhAkTzJIlS4wk889//tNruT96KywsNLGxsaZfv35m8+bN5h//+IepVauWeemllyqrTY+L9VtQUGBSUlLM4sWLzfbt2016errp3Lmz6dChg9c+qlO/xvz8v/EZS5YsMe3atTONGjUyf/vb37yWVbee/YFw4wedO3c2w4YN83xfWlpqGjVqZFJTUx2syj9yc3ONJLN69WpjzOk/IMHBwebtt9/2rLNt2zYjyaSnpxtjTv9nDAgIMNnZ2Z51XnzxRRMREWFOnDhRuQ2UweHDh02LFi3M8uXLzQ033OAJNzb2OnbsWHPdddddcLnb7TZxcXHmqaee8owVFBSY0NBQ849//MMYY8zWrVuNJLN27VrPOh9//LFxuVxm//79FVf8JejZs6f54x//6DV25513mn79+hlj7Or3p298/urthRdeMPXq1fP6fR47dqxp2bJlBXd0cRd7oz8jIyPDSDJ79+41xlTvfo25cM/ff/+9ady4sdm8ebNp1qyZV7ip7j1fKi5LlVNJSYnWr1+vlJQUz1hAQIBSUlKUnp7uYGX+UVhYKEmqX7++JGn9+vU6efKkV7+tWrVS06ZNPf2mp6erbdu2io2N9azTvXt3FRUVacuWLZVYfdkMGzZMPXv29OpJsrPX9957Tx07dtRvf/tbxcTEqH379po7d65n+e7du5Wdne3Vc2RkpLp06eLVc1RUlDp27OhZJyUlRQEBAfryyy8rr5kyuPbaa5WWlqadO3dKkr766iutWbNGt9xyiyT7+j2bv3pLT0/Xr371K4WEhHjW6d69u3bs2KEff/yxkrq5NIWFhXK5XIqKipJkZ79ut1v9+/fXmDFjdOWVV56z3Maey4JwU055eXkqLS31enOTpNjYWGVnZztUlX+43W6NGDFCXbt2VVJSkiQpOztbISEhnj8WZ5zdb3Z29nl/HmeWVSWLFi3Shg0blJqaes4y23qVpF27dunFF19UixYttGzZMg0dOlQPPvigFixYIOk/NV/s9zk7O1sxMTFey4OCglS/fv0q1/Mjjzyi3/3ud2rVqpWCg4PVvn17jRgxQv369ZNkX79n81dv1e13/Izjx49r7Nix6tu3r+dDI23s98knn1RQUJAefPDB8y63seeyqHGfCo6yGzZsmDZv3qw1a9Y4XUqF2Ldvn4YPH67ly5crLCzM6XIqhdvtVseOHTV16lRJUvv27bV582bNmTNHAwcOdLg6/3vrrbf0xhtv6M0339SVV16pzMxMjRgxQo0aNbKyX5x28uRJ3XXXXTLG6MUXX3S6nAqzfv16zZo1Sxs2bJDL5XK6nCqFMzflFB0drcDAwHPuoMnJyVFcXJxDVZXf/fffrw8++EArV65UkyZNPONxcXEqKSlRQUGB1/pn9xsXF3fen8eZZVXF+vXrlZubq2uuuUZBQUEKCgrS6tWr9eyzzyooKEixsbHW9HpGw4YN1aZNG6+x1q1bKysrS9J/ar7Y73NcXJxyc3O9lp86dUr5+flVrucxY8Z4zt60bdtW/fv318iRIz1n6mzr92z+6q26/Y6fCTZ79+7V8uXLPWdtJPv6/eyzz5Sbm6umTZt6/obt3btXo0ePVkJCgiT7ei4rwk05hYSEqEOHDkpLS/OMud1upaWlKTk52cHKLo0xRvfff7/++c9/asWKFUpMTPRa3qFDBwUHB3v1u2PHDmVlZXn6TU5O1qZNm7z+Q535I/PTN1Yn3Xjjjdq0aZMyMzM9r44dO6pfv36er23p9YyuXbuec2v/zp071axZM0lSYmKi4uLivHouKirSl19+6dVzQUGB1q9f71lnxYoVcrvd6tKlSyV0UXZHjx5VQID3n7nAwEC53W5J9vV7Nn/1lpycrE8//VQnT570rLN8+XK1bNlS9erVq6RuyuZMsPnmm2/0ySefqEGDBl7Lbeu3f//++vrrr73+hjVq1EhjxozRsmXLJNnXc5k5PaPZBosWLTKhoaFm/vz5ZuvWrebee+81UVFRXnfQVBdDhw41kZGRZtWqVebAgQOe19GjRz3rDBkyxDRt2tSsWLHCrFu3ziQnJ5vk5GTP8jO3R998880mMzPTLF261Fx22WVV9vbos519t5Qx9vWakZFhgoKCzJQpU8w333xj3njjDRMeHm7+/ve/e9aZNm2aiYqKMv/617/M119/bW677bbz3j7cvn178+WXX5o1a9aYFi1aVIlbo39q4MCBpnHjxp5bwZcsWWKio6PNww8/7FmnOvd7+PBhs3HjRrNx40YjycyYMcNs3LjRc3eQP3orKCgwsbGxpn///mbz5s1m0aJFJjw83JHbhC/Wb0lJiendu7dp0qSJyczM9Pr7dfZdQNWpX2N+/t/4p356t5Qx1a9nfyDc+Mlzzz1nmjZtakJCQkznzp3NF1984XRJl0TSeV+vvfaaZ51jx46Z++67z9SrV8+Eh4ebO+64wxw4cMBrP3v27DG33HKLqVWrlomOjjajR482J0+erORufPfTcGNjr++//75JSkoyoaGhplWrVubll1/2Wu52u83EiRNNbGysCQ0NNTfeeKPZsWOH1zqHDh0yffv2NXXq1DERERFm0KBB5vDhw5XZRpkUFRWZ4cOHm6ZNm5qwsDDTvHlzM2HCBK83u+rc78qVK8/7/3XgwIHGGP/19tVXX5nrrrvOhIaGmsaNG5tp06ZVVoteLtbv7t27L/j3a+XKlZ59VKd+jfn5f+OfOl+4qW49+4PLmLMe1QkAAFDNMecGAABYhXADAACsQrgBAABWIdwAAACrEG4AAIBVCDcAAMAqhBsAAGAVwg0AALAK4QZAjZOQkKCZM2c6XQaACkK4AVCh7r77bt1+++2SpF//+tcaMWJEpR17/vz5ioqKOmd87dq1uvfeeyutDgCVK8jpAgDAVyUlJQoJCbnk7S+77DI/VgOgquHMDYBKcffdd2v16tWaNWuWXC6XXC6X9uzZI0navHmzbrnlFtWpU0exsbHq37+/8vLyPNv++te/1v33368RI0YoOjpa3bt3lyTNmDFDbdu2Ve3atRUfH6/77rtPR44ckSStWrVKgwYNUmFhoed4jz32mKRzL0tlZWXptttuU506dRQREaG77rpLOTk5nuWPPfaYrr76ar3++utKSEhQZGSkfve73+nw4cMV+0MDcEkINwAqxaxZs5ScnKzBgwfrwIEDOnDggOLj41VQUKD/+q//Uvv27bVu3TotXbpUOTk5uuuuu7y2X7BggUJCQvT5559rzpw5kqSAgAA9++yz2rJlixYsWKAVK1bo4YcfliRde+21mjlzpiIiIjzHe+ihh86py+1267bbblN+fr5Wr16t5cuXa9euXerTp4/Xet99953effddffDBB/rggw+0evVqTZs2rYJ+WgDKg8tSACpFZGSkQkJCFB4erri4OM/4888/r/bt22vq1KmesVdffVXx8fHauXOnfvGLX0iSWrRooenTp3vt8+z5OwkJCXriiSc0ZMgQvfDCCwoJCVFkZKRcLpfX8X4qLS1NmzZt0u7duxUfHy9JWrhwoa688kqtXbtWnTp1knQ6BM2fP19169aVJPXv319paWmaMmVK+X4wAPyOMzcAHPXVV19p5cqVqlOnjufVqlUrSafPlpzRoUOHc7b95JNPdOONN6px48aqW7eu+vfvr0OHDuno0aNlPv62bdsUHx/vCTaS1KZNG0VFRWnbtm2esYSEBE+wkaSGDRsqNzfXp14BVA7O3ABw1JEjR9SrVy89+eST5yxr2LCh5+vatWt7LduzZ49uvfVWDR06VFOmTFH9+vW1Zs0a3XPPPSopKVF4eLhf6wwODvb63uVyye12+/UYAPyDcAOg0oSEhKi0tNRr7JprrtH//u//KiEhQUFBZf+TtH79erndbj3zzDMKCDh9Evqtt9762eP9VOvWrbVv3z7t27fPc/Zm69atKigoUJs2bcpcD4Cqg8tSACpNQkKCvvzyS+3Zs0d5eXlyu90aNmyY8vPz1bdvX61du1bfffedli1bpkGDBl00mFxxxRU6efKknnvuOe3atUuvv/66Z6Lx2cc7cuSI0tLSlJeXd97LVSkpKWrbtq369eunDRs2KCMjQwMGDNANN9ygjh07+v1nAKDiEW4AVJqHHnpIgYGBatOmjS677DJlZWWpUaNG+vzzz1VaWqqbb75Zbdu21YgRIxQVFeU5I3M+7dq104wZM/Tkk08qKSlJb7zxhlJTU73WufbaazVkyBD16dNHl1122TkTkqXTl5f+9a9/qV69evrVr36llJQUNW/eXIsXL/Z7/wAqh8sYY5wuAgAAwF84cwMAAKxCuAEAAFYh3AAAAKsQbgAAgFUINwAAwCqEGwAAYBXCDQAAsArhBgAAWIVwAwAArEK4AQAAViHcAAAAq/w/Ru12ALR9J+kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "schedule = mx.lr_scheduler.MultiFactorScheduler(step=[250, 750, 900], factor=0.5)\n",
    "schedule.base_lr = 1\n",
    "plot_schedule(schedule)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "041caa91",
   "metadata": {},
   "source": [
    "![lr multifactor](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/lr_schedules/multifactor.png) <!--notebook-skip-line-->\n",
    "\n",
    "\n",
    "### Polynomial Schedule\n",
    "\n",
    "Stepwise schedules and the discontinuities they introduce may sometimes lead to instability in the optimization, so in some cases smoother schedules are preferred. `PolyScheduler` gives a smooth decay using a polynomial function and reaches a learning rate of 0 after `max_update` iterations. In the example below, we have a quadratic function (`pwr=2`) that falls from 0.998 at iteration 1 to 0 at iteration 1000. After this the learning rate stays at 0, so nothing will be learnt from `max_update` iterations onwards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "83ee0daf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABDg0lEQVR4nO3de1yUZf7/8ffMICAqqKEghGF5KNOE1DxkWyZFoVbbbpn5VTO3VrPUtDIttbU81Fqbmma5mrVbq9XaSU0zPJSH8oBY5Kk8pKlAaojiAWXu3x/+mJVEnBvmnhlmXs/HYx6PnLmumc9lyry97uu6L5thGIYAAAAChN3XBQAAAHgS4QYAAAQUwg0AAAgohBsAABBQCDcAACCgEG4AAEBAIdwAAICAEuLrArzN6XRq//79qlGjhmw2m6/LAQAAbjAMQ0ePHlVcXJzs9rLnZoIu3Ozfv18JCQm+LgMAAJTD3r17demll5bZJujCTY0aNSSd/c2JjIz0cTUAAMAd+fn5SkhIcH2PlyXowk3xpajIyEjCDQAAlYw7S0pYUAwAAAIK4QYAAAQUwg0AAAgohBsAABBQCDcAACCgEG4AAEBAIdwAAICAQrgBAAABhXADAAACStDdodgqRU5Da3cdVu7Rk6pbI1zXNagth52DOQEA8Dafztx89dVX6tq1q+Li4mSz2fTxxx9ftM/y5ct17bXXKiwsTA0bNtTs2bMtr/NiFmUdUPvxX6r7jG80aE6mus/4Ru3Hf6lFWQd8XRoAAEHHp+GmoKBALVq00NSpU91qv2vXLnXu3FkdO3ZUZmamBg8erL/85S9avHixxZVe2KKsA+r37wzlHC0s8XzO0UL1+3cGAQcAAC+zGYZh+LoI6exBWB999JHuuuuuC7YZNmyYFixYoKysLNdz9913n/Ly8rRo0aJS+5w6dUqnTp1y/br4VNEjR45U+ODMIqeh5s8t1vHCogu2CQ2xa8uY27hEBQBABeTn5ysqKsqt7+9KtaB4zZo1SklJKfFcamqq1qxZc8E+48ePV1RUlOuRkJDgsXpW/3iwzGAjSYVnnJq0ZLvHPhMAAJStUoWb7OxsxcTElHguJiZG+fn5OnHiRKl9hg8friNHjrgee/fu9Vg9/934i1vtXlv+k4qcfjFBBgBAwKtU4aY8wsLCFBkZWeLhKRebtSnmNMTsDQAAXlKpwk1sbKxycnJKPJeTk6PIyEhVrVrV6/W0TqztdltmbwAA8I5KFW7atWun9PT0Es8tWbJE7dq180k9vdsnut2W2RsAALzDp+Hm2LFjyszMVGZmpqSzW70zMzO1Z88eSWfXy/Tq1cvVvl+/ftq5c6eeeuopbd26VdOmTdP777+vxx9/3BflKzTErrYNarndftoKZm8AALCaT8PN+vXrlZycrOTkZEnSkCFDlJycrFGjRkmSDhw44Ao6ktSgQQMtWLBAS5YsUYsWLfTyyy/rn//8p1JTU31SvyS907et223POM/usAIAANbxm/vceIuZffLuuu+N1fpm129utb2iToTSh3b0yOcCABAsAvY+N/7KzOzNjl+Pq/CM08JqAAAIboQbDwgNsathnWput+8181sLqwEAILgRbjxkdNer3W77za7DzN4AAGARwo2HtG8YLYeJ46OYvQEAwBqEGw9x2G0acNMVbrdn9gYAAGsQbjxo0C1NZObs7+HzvrOsFgAAghXhxoMcdpse6+j+7M1HG/dxUz8AADyMcONhg25p4nZbpyFNSf/RwmoAAAg+hBsPc9htSm1a1+32U5b+yOwNAAAeRLixQK/2DdxuW8SBmgAAeBThxgJtL79EYSb2hXOgJgAAnkO4sYDDblN/E9vCOVATAADPIdxY5LFOjVXFxO/uc/OzrCsGAIAgQrixiMNu0z/uTXK7PQdqAgDgGYQbC3VJildsZJjb7TmSAQCAiiPcWOzvf27hdluOZAAAoOIINxbjQE0AALyLcGMxDtQEAMC7CDdewIGaAAB4D+HGCzhQEwAA7yHceAkHagIA4B2EGy/hQE0AALyDcONFHKgJAID1CDdexIGaAABYj3DjRRyoCQCA9Qg3XsaBmgAAWItw42UcqAkAgLUINz5g9kDNLpO/trAaAAACC+HGR8wcqLk995g+27TfwmoAAAgchBsfMXug5lMfbmLnFAAAbiDc+IjZAzVPnHbqmx2HLKwIAIDAQLjxIbMHar69ZpdltQAAECgINz5k9kDNLzbncmkKAICLINz4mNnZm8fey7CsFgAAAgHhxsccdpvuvjbO7fYLs7K57w0AAGUg3PiB8Xe7vy1cknrN/NaiSgAAqPwIN34gNMSutg1qud3+m12Hmb0BAOACCDd+4p2+bU21Hz7vO4sqAQCgciPc+InQELvSmsW43X5exj52TgEAUArCjR+Zcn9Lt9sakiYt2W5dMQAAVFKEGz/isNuU2rSu2+1fW/4TszcAAPwO4cbP9GrfwO22ToPZGwAAfo9w42faXn6JwkPcv60fszcAAJREuPEzDrtNE//s/n1vmL0BAKAkwo0f6pIUr0Z1q7ndftoKZm8AAChGuPFTCwb+we22Z5zSlPQfLawGAIDKg3Djp0JD7GpYx/3Zm6nLmL0BAEAi3Pi10V2vdrvtaafB7A0AACLc+LX2DaNlYuMUszcAAIhw49ccdpsevbmh2+1POw2t/vGghRUBAOD/CDd+7rFOjU3N3jzx30zLagEAoDIg3Pg5s7M3OfmF+mzTfgsrAgDAvxFuKoHHOjWWw8TszdD3M1l7AwAIWoSbSsBht2nATVe43b6wiJ1TAIDgRbipJAbd0kQmJm80ZemPzN4AAIIS4aaScNhteqyj+7M3RZw5BQAIUj4PN1OnTlViYqLCw8PVpk0brV27tsz2r776qpo0aaKqVasqISFBjz/+uE6ePOmlan1r0C1NZDcxfcOJ4QCAYOTTcDN37lwNGTJEo0ePVkZGhlq0aKHU1FTl5uaW2v69997T008/rdGjR2vLli2aOXOm5s6dqxEjRni5ct9w2G0aaGLnFCeGAwCCkU/DzSuvvKKHHnpIffr0UdOmTTV9+nRFRERo1qxZpbZfvXq1rr/+et1///1KTEzUrbfequ7du190tieQPNapsaqY+L/GieEAgGDjs3BTWFioDRs2KCUl5X/F2O1KSUnRmjVrSu3Tvn17bdiwwRVmdu7cqYULFyotLe2Cn3Pq1Cnl5+eXeFRmDrtN/7g3ye32nBgOAAg2Pgs3Bw8eVFFRkWJiYko8HxMTo+zs7FL73H///RozZow6dOigKlWq6IorrtBNN91U5mWp8ePHKyoqyvVISEjw6Dh8oUtSvGIjw9xuz84pAEAw8fmCYjOWL1+ucePGadq0acrIyNC8efO0YMECPf/88xfsM3z4cB05csT12Lt3rxcrts7f/9zC7bbsnAIABJMQX31wdHS0HA6HcnJySjyfk5Oj2NjYUvuMHDlSPXv21F/+8hdJUvPmzVVQUKCHH35YzzzzjOz287NaWFiYwsLcn+WoLIpPDD/j5oTMa8t/0qBbGsthZrsVAACVkM9mbkJDQ9WyZUulp6e7nnM6nUpPT1e7du1K7XP8+PHzAozD4ZAkGUZwXXYxe+YUO6cAAMHCp5elhgwZohkzZujtt9/Wli1b1L9/fxUUFKhPnz6SpF69emn48OGu9l27dtXrr7+uOXPmaNeuXVqyZIlGjhyprl27ukJOMGHnFAAA5/PZZSlJ6tatm3799VeNGjVK2dnZSkpK0qJFi1yLjPfs2VNipubZZ5+VzWbTs88+q3379qlOnTrq2rWrxo4d66sh+FTxzqlH52S61b5459TgWxpbWxgAAD5kM4Lsek5+fr6ioqJ05MgRRUZG+rocj2g77ktl559yq63DJm0fm8baGwBApWLm+7tS7ZZC6dg5BQDA/xBuAkDxzil3ceYUACCQEW4CADunAAD4H8JNgGDnFAAAZxFuAgRnTgEAcBbhJoCYPXNqcjpnTgEAAg/hJsCY2TnllPTYexnWFQMAgA8QbgKM2Z1TC7OyVXjGaV1BAAB4GeEmwJjdOSVJw+d9Z1E1AAB4H+EmAJndOfXfjH2svQEABAzCTQAyu3NKYu0NACBwEG4CVJekeCXWrup2e9beAAACBeEmgI29+xpT7XvN/NaiSgAA8B7CTQBre/klCjexdeqbXYeZvQEAVHqEmwDmsNs00cR9byTp6f9usqgaAAC8g3AT4LokxevahEi328/buJ+dUwCASo1wEwQ+6N/BVPt7Xl9tUSUAAFiPcBMEHHab7k6Kc7t9xt48fbZpv4UVAQBgHcJNkJhgcu3NUx9u4vIUAKBSItwEidAQu9o2qOV2+xOnnfpmxyELKwIAwBqEmyDyTt+2ptqP+JgzpwAAlQ/hJoiEhtiV1izG7fY/HzrB2hsAQKVDuAkyU+5vaar943M3svYGAFCpEG6CjMNu05+udX/n1BmnNGnJdgsrAgDAswg3QWj83eZ2Tr22/CdmbwAAlQbhJgiZXXvjNJi9AQBUHoSbIDXl/pZy/0hNZm8AAJUH4SZIOew2Tbkvye32zN4AACoLwk0Q65IUr9jIMLfbT1nG7A0AwP8RboLc300cy2BIeuy9DOuKAQDAAwg3Qa59w2hVMfGnYGFWtgrPOK0rCACACiLcBDmH3aZ/3Jtkqk/Pmd9YUwwAAB5AuIG6JMWrUd1qbrf/dtdvzN4AAPwW4QaSpAUD/2CqfZfJX1tUCQAAFUO4gaSzN/Zrk1jL7fbbc49xqCYAwC8RbuDyr7+0NdWeQzUBAP6IcAMXs8cycKgmAMAfEW5QAscyAAAqO8INSnDYbXqs4xVut+dYBgCAvyHc4DyDbmliavaGYxkAAP6EcIPzmJ294VgGAIA/IdygVINuaSK7iekbjmUAAPgLwg1K5bDbNLlbkqk+nSd/ZU0xAACYQLjBBZk9luHH3AJu7AcA8DnCDcpk9liGoe9nsrgYAOBThBuUyeyxDIVFhqak/2hhRQAAlI1wg4syeyzDlKU/MnsDAPCZCoWbkydPeqoO+DGzxzIUcWM/AIAPmQ43TqdTzz//vOLj41W9enXt3LlTkjRy5EjNnDnT4wXCP5g9lmEyN/YDAPiI6XDzwgsvaPbs2XrppZcUGhrqer5Zs2b65z//6dHi4D8cdpsGdWpoqs89r6+2qBoAAC7MdLh555139Oabb6pHjx5yOByu51u0aKGtW7d6tDj4l8c6NZbDxPRNxt48toYDALzOdLjZt2+fGjY8/1/wTqdTp0+f9khR8E8Ou00DbnL/WAaJreEAAO8zHW6aNm2qr7/++rznP/zwQyUnJ3ukKPgvs8cyFBYZWv3jQesKAgDgd0LMdhg1apR69+6tffv2yel0at68edq2bZveeecdzZ8/34oa4UeKj2V4dE6m232e+G+mvh1xi3VFAQBwDtMzN3feeac+++wzffnll6pWrZpGjRqlLVu26LPPPtMtt/AFFgy6JMXr2oRIt9vn5Bey9gYA4DU2wzCCakFEfn6+oqKidOTIEUVGuv8FjZKKnIauGLHQ7fY2ST+NS5PDzDUtAAD+PzPf36Znbi6//HIdOnTovOfz8vJ0+eWXm307TZ06VYmJiQoPD1ebNm20du3aMtvn5eVpwIABqlevnsLCwtS4cWMtXOj+lyw8w2G3aWBH9xcXG2JrOADAO0yHm927d6uoqOi850+dOqV9+/aZeq+5c+dqyJAhGj16tDIyMtSiRQulpqYqNze31PaFhYW65ZZbtHv3bn344Yfatm2bZsyYofj4eLPDgAcMuqWJqRv7sTUcAOANbi8o/vTTT13/vXjxYkVFRbl+XVRUpPT0dCUmJpr68FdeeUUPPfSQ+vTpI0maPn26FixYoFmzZunpp58+r/2sWbN0+PBhrV69WlWqVJEk058Jz3HYbXqs4xWavGyH230en7tRac3rcXkKAGAZt9fc2O1nJ3lsNpt+36VKlSpKTEzUyy+/rC5durj1wYWFhYqIiNCHH36ou+66y/V87969lZeXp08++eS8Pmlpaapdu7YiIiL0ySefqE6dOrr//vs1bNiwEjcUPNepU6d06tQp16/z8/OVkJDAmhsPKXIaavTMQpm5lc3Ajg01JLWJdUUBAAKOJWtunE6nnE6n6tevr9zcXNevnU6nTp06pW3btrkdbCTp4MGDKioqUkxMyQMZY2JilJ2dXWqfnTt36sMPP1RRUZEWLlyokSNH6uWXX9YLL7xwwc8ZP368oqKiXI+EhAS3a8TFFW8NN4NzpwAAVjK95mbXrl2Kjo62opaLcjqdqlu3rt588021bNlS3bp10zPPPKPp06dfsM/w4cN15MgR12Pv3r1erDg4mN0aLrG4GABgHdM38ZOkgoICrVixQnv27FFhYWGJ1wYOHOjWe0RHR8vhcCgnJ6fE8zk5OYqNjS21T7169VSlSpUSl6CuuuoqZWdnq7CwsMRBnsXCwsIUFhbmVk0ovw/6dzC1Nbx4cXHXFnEWVgUACEamw83GjRuVlpam48ePq6CgQLVr19bBgwcVERGhunXruh1uQkND1bJlS6Wnp7vW3DidTqWnp+vRRx8ttc/111+v9957T06n07UGaPv27apXr16pwQbeU7w1nMXFAABfM31Z6vHHH1fXrl3122+/qWrVqvrmm2/0888/q2XLlpo4caKp9xoyZIhmzJiht99+W1u2bFH//v1VUFDg2j3Vq1cvDR8+3NW+f//+Onz4sAYNGqTt27drwYIFGjdunAYMGGB2GLCA2XOnzjilSUu2W1cQACAomQ43mZmZGjp0qOx2uxwOh06dOqWEhAS99NJLGjFihKn36tatmyZOnKhRo0YpKSlJmZmZWrRokWuR8Z49e3TgwAFX+4SEBC1evFjr1q3TNddco4EDB2rQoEGlbhuH95VncfFry1lcDADwLNPHL9SpU0erV69Wo0aN1LhxY02ZMkWpqanaunWrWrZsqYKCAqtq9QiOX7De3VO/VsbefLfbszUcAHAxlh6/kJycrHXr1kmSbrzxRo0aNUrvvvuuBg8erGbNmpWvYgSUD/p3MNV+ClvDAQAeZDrcjBs3TvXq1ZMkjR07VrVq1VL//v3166+/6o033vB4gah8ynPu1GPvZVhXEAAgqHAqOCxRnjsXb3/hdoWGmM7bAIAgYOllqQvJyMgwdYdiBLbyLC7uPPkra4oBAAQVU+Fm8eLFeuKJJzRixAjt3LlTkrR161bdddddat26tZxOpyVFonLqkhSvRnWrud3+x9wCTg0HAFSY2+Fm5syZuv322zV79my9+OKLatu2rf7973+rXbt2io2NVVZWlhYudP8OtQgOCwb+wVT7QXM2srgYAFAhboebSZMm6cUXX9TBgwf1/vvv6+DBg5o2bZq+//57TZ8+XVdddZWVdaKSCg2xq01iLbfbOw0WFwMAKsbtcLNjxw7dc889kqS7775bISEh+vvf/65LL73UsuIQGP71l7am2i/MylbhGS5xAgDKx+1wc+LECUVEREiSbDabwsLCXFvCgbKEhtiV1izGVB8WFwMAysvUwZn//Oc/Vb16dUnSmTNnNHv2bEVHR5do4+7BmQguU+5vqc9HLJS7q2mKFxdzajgAwCy373OTmJgom63sUxFtNptrF5W/4j43vjM/c58enZPpdnu7TfpxbBqnhgMATH1/uz1zs3v37orWhSDXJSlek9K368dfj7vVvnhx8bT/a2lxZQCAQMLtYOFVCwbdaKo9i4sBAGYRbuBV5Vlc3PqFLyyqBgAQiAg38Lop97eUmVU0R04W6cHZay2rBwAQWAg38DqH3aYp9yWZ6rN0668czQAAcAvhBj7RJSlejepEmOrz+FyOZgAAXJzpcJOfn1/q4+jRoyosLLSiRgQos4uLzzilSUu2W1QNACBQmA43NWvWVK1atc571KxZU1WrVtVll12m0aNHc0I4Lio0xK4+19c31Wfysp+YvQEAlMl0uJk9e7bi4uI0YsQIffzxx/r44481YsQIxcfH6/XXX9fDDz+syZMna8KECVbUiwAzumtz1alexVSfe15fbVE1AIBA4PYdiot16tRJf/3rX3XvvfeWeP7999/XG2+8ofT0dP3rX//S2LFjtXXrVo8W6wncodj/FDkNXTFioak+U7onczQDAAQRM9/fpmduVq9ereTk5POeT05O1po1ayRJHTp00J49e8y+NYKUw27TwI5XmOrD4mIAwIWYDjcJCQmaOXPmec/PnDlTCQkJkqRDhw6pVq1aFa8OQWPQLU1k5ggpFhcDAC7E1KngkjRx4kTdc889+vzzz9W6dWtJ0vr167V161Z9+OGHkqR169apW7dunq0UAc1ht2lytyRTB2tOXvaTBt3SmIM1AQAlmF5zI0m7du3SG2+8oe3bz/7LuUmTJvrrX/+qxMRET9fncay58W93T/1aGXvz3W5/bUJNzRtwvYUVAQD8gZnv73KFm8qMcOPfWFwMACiNme9v05elJCkvL09r165Vbm7uefez6dWrV3neEpD0v8XFk5ftcLvPY//ZqLTm9bg8BQCQVI6Zm88++0w9evTQsWPHFBkZKZvtf18oNptNhw8f9niRnsTMjf8rchpq9MxCmdkMlZwQpY8GdLCuKACAT1m6FXzo0KF68MEHdezYMeXl5em3335zPfw92KByKF5cbMbGvUc4WBMAIKkc4Wbfvn0aOHCgIiLMHXoImNElKV7XJpibWRs0h3vfAADKEW5SU1O1fv16K2oBSvigv7nLTE5Deuy9DIuqAQBUFqYXFHfu3FlPPvmkNm/erObNm6tKlZLnAt1xxx0eKw7BzWG3afK9LTTw/U1u91mYla3CM06FhpjO7QCAAGF6QbHdfuEvDZvNpqKiogoXZSUWFFc+N/09XbsPnXS7fWyNUH3zzC0WVgQA8DZLFxQ7nc4LPvw92KBySh96s6n22UcL1Xf2OouqAQD4O+bu4fccdpteuy/JVJ/0rbnsngKAIOXWmpvJkyfr4YcfVnh4uCZPnlxm24EDB3qkMOBcXZLiNSl9u3789bjbfR6fy839ACAYubXmpkGDBlq/fr0uueQSNWjQ4MJvZrNp586dHi3Q01hzU3kVnnGq8bOfm+ozsGNDDUltYlFFAABv8fjxC7t27Sr1vwFvCg2xK61ZjBZm5bjdh5PDASD4sOYGlcqU+1ua/kOb8vJyK0oBAPgp0/e5KSoq0uzZs5Wenl7qwZlLly71WHHA7znsNr12f7IeeW+j2312HTqu5+dv1sguTS2sDADgL0yHm0GDBmn27Nnq3LmzmjVrVuLgTMAb0q6JU9p3+01dnpq5cpeG3XYlN/cDgCBg+iZ+0dHReuedd5SWlmZVTZZiQXFgKM/J4Q3rROjLoR2tKwoAYBlLb+IXGhqqhg0blrs4wBPKc3L4T7+evTwFAAhspsPN0KFDNWnSJJmc8AE8rjwnh89cuUsLvztgUUUAAH9g+rLUH//4Ry1btky1a9fW1Vdffd7BmfPmzfNogZ7GZanAUuQ01HDEQpn5Q1zFbtPWF25nezgAVCKWXpaqWbOm/vjHP+rGG29UdHS0oqKiSjwAb3LYbZpi8miG005Dk5Zst6YgAIDPmdotdebMGXXs2FG33nqrYmNjraoJMKVLUrxmrdqpjL35bvfh5n4AELhMzdyEhISoX79+OnXqlFX1AOXyQf8OMhtT2o5dYkktAADfMn1Z6rrrrtPGje7fQA3whvJcnvq14LQenL3WmoIAAD5j+iZ+jzzyiIYOHapffvlFLVu2VLVq1Uq8fs0113isOMCM8lyeWrr1V322ab+6toizsDIAgDeZ3i1lt58/2WOz2WQYhmw2m4qKijxWnBXYLRXYyrN7yibpp3FprL8BAD/m8VPBz8Wp4PBnxZenHp2T6XYfQ9I9r6/WvAHXW1YXAMB7TIebyy67zIo6AI/pkhSveRv3aum2Q273ydibx+UpAAgQpi9LFdu8ebP27NmjwsLCEs/fcccdHinMKlyWCh6tn/9CvxacNtVnB5enAMAvmfn+Nh1udu7cqT/+8Y/6/vvvXWttJLlOB2fNDfxFkdPQFSMWmurT4JIILXuSwzUBwN9YeofiQYMGqUGDBsrNzVVERIR++OEHffXVV2rVqpWWL19e3poBj3PYbRrY8QpTfXYdOq5PMvdZVBEAwBtMh5s1a9ZozJgxio6Olt1ul91uV4cOHTR+/HgNHDjQihqBcht0SxM5TF5lGjQnU0VODoYFgMrKdLgpKipSjRo1JEnR0dHav3+/pLMLjbdt21auIqZOnarExESFh4erTZs2WrvWvRurzZkzRzabTXfddVe5PheBz2G3aUr3ZNP92o370oJqAADeYDrcNGvWTJs2bZIktWnTRi+99JJWrVqlMWPG6PLLLzddwNy5czVkyBCNHj1aGRkZatGihVJTU5Wbm1tmv927d+uJJ57QDTfcYPozEVzSrolT3w7mdvnlHivUmM9+sKgiAICVTIebZ599Vk6nU5I0ZswY7dq1SzfccIMWLlyoyZMnmy7glVde0UMPPaQ+ffqoadOmmj59uiIiIjRr1qwL9ikqKlKPHj30t7/97aKB6tSpU8rPzy/xQPAZ2aWZki41t4B81qrdKjzjtKgiAIBVTIeb1NRU3X333ZKkhg0bauvWrTp48KByc3N18803m3qvwsJCbdiwQSkpKf8ryG5XSkqK1qxZc8F+Y8aMUd26ddW3b9+Lfsb48eMVFRXleiQkJJiqEYHjv490MN2n9QtfWFAJAMBKpsNNsZ9++kmLFy/WiRMnVLt27XK9x8GDB1VUVKSYmJgSz8fExCg7O7vUPitXrtTMmTM1Y8YMtz5j+PDhOnLkiOuxd+/ectWKys9ht+k1k4drHjlZxOGaAFDJmA43hw4dUqdOndS4cWOlpaXpwIEDkqS+fftq6NChHi/wXEePHlXPnj01Y8YMRUdHu9UnLCxMkZGRJR4IXl2S4nVzk0tM9Sk+XBMAUDmYDjePP/64qlSpoj179igiIsL1fLdu3bRo0SJT7xUdHS2Hw6GcnJwSz+fk5Cg2Nva89jt27NDu3bvVtWtXhYSEKCQkRO+8844+/fRThYSEaMeOHWaHgyA0q09bRYU7TPV57D8b2R4OAJWE6XDzxRdf6MUXX9Sll15a4vlGjRrp559/NvVeoaGhatmypdLT013POZ1Opaenq127due1v/LKK/X9998rMzPT9bjjjjvUsWNHZWZmsp4Gblv37K2m+3SauMyCSgAAnmb64MyCgoISMzbFDh8+rLCwMNMFDBkyRL1791arVq103XXX6dVXX1VBQYH69OkjSerVq5fi4+M1fvx4hYeHq1mzZiX616xZU5LOex4oS2iIXX2ur6+3Vu1xu8/uwyfUd/Y6zXygtYWVAQAqyvTMzQ033KB33nnH9WubzSan06mXXnpJHTuaP5OnW7dumjhxokaNGqWkpCRlZmZq0aJFrkXGe/bsca3rATxpdNfmqlO9iqk+6VtzWX8DAH7O9MGZWVlZ6tSpk6699lotXbpUd9xxh3744QcdPnxYq1at0hVXmDvLx9s4OBPnKs/hmhKnhwOAt1l6cGazZs20fft2dejQQXfeeacKCgp09913a+PGjX4fbIDfc9htmnxvC9P9/jRtpQXVAAA8wfTMzYX88ssvGjNmjN58801PvJ1lmLlBaW76e7p2Hzppqk/fDg00sktTiyoCAJzL0pmbCzl06JBmzpzpqbcDvCp9qLm7a0vSzJW7tPA71oMBgL/xWLgBKjOH3aZp95s/PfzR9zK4/w0A+BnCDfD/pV0Tpz7X1zfVxynpz9NWWVMQAKBcCDfAOUZ3ba7E2uGm+mz85Yien7/ZoooAAGa5fRO/4pPALyQvL6+itQB+If2Jm01vD5+5cpda1q+ltGvqWVQVAMBdbs/cREVFlfm47LLL1KtXLytrBbyiPKeHS9IA1t8AgF/w2FbwyoKt4HDX3VO/VsbefFN9EmtX1fKnzO+8AgCUzSdbwYFA80H/Dqb/ghSfPwUA8B3CDXABDrtNr5VjezjnTwGAbxFugDKkXROnvh0uM93vsf9sZP0NAPgI4Qa4iJFdmik5wfz6rE4Tl1lQDQDgYgg3gBs+ZP0NAFQahBvADay/AYDKg3ADuCntmjg9dEOi6X6svwEA7yLcACY80/lq9W5v7vwpSWrzwhcWVAMAKA3hBjDpb3eYP3/q4PEz6jz5K4sqAgCci3ADlEP6E+bvQvzD/qMsMAYALyDcAOXgsNs0jQXGAOCXCDdAOaVdE6c+15tff8MCYwCwFuEGqIDRXZvr6thqpvsl/W2xBdUAACTCDVBhCwbfpMgwc3+Vjp4qYoExAFiEcAN4wMbRt5nu88P+o3ru0ywLqgGA4Ea4ATzAYbfptfuSTPebvfpnjV2w2fMFAUAQI9wAHtIlKV6drow23W/G17u08LsDFlQEAMGJcAN40MwH2pRrgfEj72WwgwoAPIRwA3jYgsE3KbpaiOl+N/99qQXVAEDwIdwAFvj2mVtN9/n5t5PqPGmFBdUAQHAh3AAWKO8C4x8OHFMXtogDQIUQbgCLlHeBcRZnUAFAhRBuAAuVd4ExZ1ABQPkRbgCLnV1gXMV0P86gAoDyIdwAXvDtM7eUq1+L5xZ5uBIACHyEG8ALHHabpt2fbLrfsUKnrp/wpQUVAUDgItwAXpJ2TZweuiHRdL99eaeUxhZxAHAb4Qbwomc6X60+119mut9mtogDgNsIN4CXje7aTDc3Kd8W8QffWmtBRQAQWAg3gA/M6lO+LeJLt/2qv332gwUVAUDgINwAPlLeM6jeWrVbYxdstqAiAAgMhBvAh8pzBpUkzfh6lxZ+d8DD1QBAYCDcAD5U3i3ikvTIexnc5A8ASkG4AXysvFvEJema0Z97thgACACEG8APlHeLeMFpQ8ljFltQEQBUXoQbwE+Ud4v4b8fPcBdjADgH4QbwI7P6tFGzetVN9+MuxgDwP4QbwM/MH3SjmpbjHjjcxRgAziLcAH5o4eCbFB8VarofdzEGAMIN4LdWDb9Ftao6TPdbuu1Xjf40y4KKAKByINwAfmzj6NtULdT8X9O3V/+svrOZwQEQnAg3gJ/77rnbytUvfeuvBBwAQYlwA/i5itzFOH0rB20CCD6EG6ASqMhdjDloE0CwIdwAlUR572IsnT1oc37mfg9XBAD+iXADVCLlvYuxJD06ZyMBB0BQINwAlUx572IsnQ044xdyiQpAYPOLcDN16lQlJiYqPDxcbdq00dq1F97hMWPGDN1www2qVauWatWqpZSUlDLbA4Fo/qAbdXU5A84bX3GJCkBg83m4mTt3roYMGaLRo0crIyNDLVq0UGpqqnJzc0ttv3z5cnXv3l3Lli3TmjVrlJCQoFtvvVX79u3zcuWAby2oQMDhEhWAQGYzDMPwZQFt2rRR69at9dprr0mSnE6nEhIS9Nhjj+npp5++aP+ioiLVqlVLr732mnr16nXR9vn5+YqKitKRI0cUGRlZ4foBX+s8aYV+OHCsXH3/+ocGGp7W1MMVAYDnmfn+9unMTWFhoTZs2KCUlBTXc3a7XSkpKVqzZo1b73H8+HGdPn1atWvXLvX1U6dOKT8/v8QDCCQLynnQpsQlKgCByafh5uDBgyoqKlJMTEyJ52NiYpSdne3WewwbNkxxcXElAtK5xo8fr6ioKNcjISGhwnUD/qa8B21KXKICEHh8vuamIiZMmKA5c+boo48+Unh4eKlthg8friNHjrgee/fu9XKVgHesGn6LakeElKvvo3M2auwC7mQMIDD4NNxER0fL4XAoJyenxPM5OTmKjY0ts+/EiRM1YcIEffHFF7rmmmsu2C4sLEyRkZElHkCgyhiVqrhyzuDM+Hq3np9PwAFQ+fk03ISGhqply5ZKT093Ped0OpWenq527dpdsN9LL72k559/XosWLVKrVq28USpQaawefku5A87Mlbv1t8+yPFwRAHiXzy9LDRkyRDNmzNDbb7+tLVu2qH///iooKFCfPn0kSb169dLw4cNd7V988UWNHDlSs2bNUmJiorKzs5Wdna1jx8q3WwQIRKsrcInqrVU/68G3vvVwRQDgPT4PN926ddPEiRM1atQoJSUlKTMzU4sWLXItMt6zZ48OHDjgav/666+rsLBQf/7zn1WvXj3XY+LEib4aAuCXKnKJaum2g+oyaYWHKwIA7/D5fW68jfvcINi0H79E+48Ulqtv09hqWjj4Js8WBADlUGnucwPAehW5RLU5u0DXj1/i4YoAwFqEGyAIVOQS1b4jhUr+2yIPVwQA1iHcAEFi9fBbFF8zrFx9fztRpKtHfa4iZ1BdxQZQSRFugCCy6ukUXVrOgFNQ6NQVIxZq4XfczRiAfyPcAEFm5dMpalrO08Ql6ZH3Nur5+dwLB4D/ItwAQWjhoBvVLK5GufvPXMm9cAD4L8INEKTmD/yDbm5Sp9z9uRcOAH9FuAGC2Kw+16lvhwbl7p914JjSXl3uuYIAwAMIN0CQG9mlqV67L7nc/TdnF6j9uC88WBEAVAzhBoC6JMVp2v3Xlrv//vzTavLMAraKA/ALhBsAkqS0a+pp+v+VP+CcKpKuGLFQ8zP3ebAqADCPcAPA5bZm9bRjXJqqh5b/R8OjczLVdzY7qQD4DuEGQAkOu01ZY27XpbXCy/0e6VsP6qYX07lMBcAnCDcASrVyWCfd1PiScvff/dtJ7mgMwCcINwAuaPaDbSt0LxyJOxoD8D7CDYAyzepznVKuqluh95i58mf1mfWNhyoCgLIRbgBc1D97t9aU7uW/F44kLdt+SK2eX8w6HACWI9wAcEvXFnHaMS5NYY7yv8fBgjNsFwdgOcINALc57DZtG9tZ8VFhFXqfR+dk6sG3uEwFwBqEGwCmrRqeoqsrcKq4JC3dxmUqANYg3AAolwUD/6BOV1ZsoTGXqQBYgXADoNxmPlDxhcYSl6kAeBbhBkCFFC80rlOtSoXeZ+m2Q2o1ZhGXqQBUGOEGQIU57DatG3mrOjaJrtD7HDxepCtGLNSnGb94qDIAwYhwA8Bj3urTRn07NKjw+wx8f5NufPFLZnEAlAvhBoBHjezSVNPuv7bC7/Pzb6eYxQFQLoQbAB6Xdk097RiXpgaXRFT4vZjFAWAW4QaAJRx2m5Y92dEjl6mYxQFghs0wjKD651B+fr6ioqJ05MgRRUZG+rocICgUnnGqxd8W6cTpiv+4ialRRV8PS1FoCP82A4KJme9vfjoAsFxoiF1bnk9Ts/iK/4Mi5+hpNX72c/X79zouVQEoFeEGgNfMf+wGTbovySPvtSgrl0tVAEpFuAHgVXcmxWvHuDRFhod45P0Gvr9JbcZ+ocIzTo+8H4DKj3ADwOscdpu+ey5VfdoneuT9ii9Vjf70O4+8H4DKjQXFAHyq8IxTSWMW63ihZ2ZeQu3SpuduU9VQh0feD4B/YEExgEojNMSuzWNu99gsTqFTumrUInWauJRLVUCQYuYGgN/w9CyOJLVOrKl3/9KOreNAJcfMDYBKydOzOJK0bneeGj/7ue6ZvoqZHCBIMHMDwC8VnnHqurFLlHfijEffN/XqOprWo7UcdptH3xeAtZi5AVDphYbYlTk6VQ9en+jR9138w6+6YsRCPf6fjczkAAGKmRsAfq/wjFNpk1bop1+Pe/y9b2tWV1Pvb8VMDuDnmLkBEFBCQ+z6cmhHTeme7PEfWsV3Oh78XgYzOUCAYOYGQKVS5DT06hfbNGX5Dkvev0lMNX084AbukwP4GTPf34QbAJVSkdPQn6atUuYvRyx5/7ioMKUP7UjIAfwE4aYMhBsgsJwoLFLrF5boWGGRJe9fs6pDk+67Vh0a1WFdDuBDhJsyEG6AwPRRxj49/n6mpZ9x/eW19c8HrmM2B/ABwk0ZCDdA4CpyGnr03Q36/IccSz+nephDk++7Vjc2YTYH8BbCTRkIN0DgKzzj1FMfZurjzAOWf1aTutU1ovNVXLYCLEa4KQPhBggeRU5DA/69QYs2WzuTU4ygA1iHcFMGwg0QfArPOPV//1yjtbvzvPaZBB3Aswg3ZSDcAMHLFyFHkmpWDdHtzeppVNerWYwMlBPhpgyEGwCFZ5x68sNMfeKFNTm/Z5d0Zb0aeuLWK1mQDJhAuCkD4QZAseK7Hb+2fId89YOwit2mxOgI/enaS/Vgh8sVGsKpOEBpCDdlINwA+L0ip6GV235V//9s0PFC354vZZMUEWrXZZdUY3YHOAfhpgyEGwBlOXbyjG5/dYX25p30dSklhNilsBBCD4IX4aYMhBsA7ig849SMr3/SpPSfVHjGf39MFl/EMiSFOGyKCHWoZf1amtz9WlUPD/FlaYBHEW7KQLgBYNaJwiLdOfVrbc8p8HUpptl1NvgUz/H46r8lyW6TnIZv66Ama2tyWDjDaOb72y9Wrk2dOlWJiYkKDw9XmzZttHbt2jLbf/DBB7ryyisVHh6u5s2ba+HChV6qFEAwqhrq0BeP36TtL9yuYbc1VkQVv/jR6Ranzn7pOH38305JZwzf10FN1tZ0xikVFDq1+cBRPfj2Ol058nMtyvLNrkSfmjt3roYMGaLRo0crIyNDLVq0UGpqqnJzc0ttv3r1anXv3l19+/bVxo0bddddd+muu+5SVlaWlysHEGxCQ+zqf1MjbX7+dm0Zc5u6tY5XqM9/igL+63SRoX7/zvB6wPH5Zak2bdqodevWeu211yRJTqdTCQkJeuyxx/T000+f175bt24qKCjQ/PnzXc+1bdtWSUlJmj59+kU/j8tSADztRGGRnvvse320YZ98vNkK8EuxkeFa9fTNFbpEVWkuSxUWFmrDhg1KSUlxPWe325WSkqI1a9aU2mfNmjUl2ktSamrqBdufOnVK+fn5JR4A4ElVQx168U9J2j6us2tGJzKMKR2gWHb+Sa3dddhrn+fTv30HDx5UUVGRYmJiSjwfExOj7OzsUvtkZ2ebaj9+/HhFRUW5HgkJCZ4pHgBKURx0vvvb7doxLk1v926ttok1VYmW6QCWyD3qvdsrBPw+weHDh2vIkCGuX+fn5xNwAHiFw27TjVfV1Y1X1ZV0dnv5zJU79MG6Pfr58EkVBdVeVQS7ujXCvfZZPg030dHRcjgcysnJKfF8Tk6OYmNjS+0TGxtrqn1YWJjCwsI8UzAAVEDxguT+NzWSdPbOyF9tydVLX2zRrl8LdJL1OghQsZHhuq5Bba99nk8nSkNDQ9WyZUulp6e7nnM6nUpPT1e7du1K7dOuXbsS7SVpyZIlF2wPAP7KYbep49Ux+vzxm7R1XGftntDZtd388kvCVcXuB1taAQ947o6mXr2jts8vSw0ZMkS9e/dWq1atdN111+nVV19VQUGB+vTpI0nq1auX4uPjNX78eEnSoEGDdOONN+rll19W586dNWfOHK1fv15vvvmmL4cBAB7x+9mdYude0tr720kVOUveQI1JH/ijKg6bpnRP1m3N6nn1c30ebrp166Zff/1Vo0aNUnZ2tpKSkrRo0SLXouE9e/bIbv/fv13at2+v9957T88++6xGjBihRo0a6eOPP1azZs18NQQAsNyFQs+5irekL92co7zjZ1xreirbXW6pqfLWZOUdis3w+X1uvI373AAAUPlUmvvcAAAAeBrhBgAABBTCDQAACCiEGwAAEFAINwAAIKAQbgAAQEAh3AAAgIBCuAEAAAGFcAMAAAKKz49f8LbiGzLn5+f7uBIAAOCu4u9tdw5WCLpwc/ToUUlSQkKCjysBAABmHT16VFFRUWW2CbqzpZxOp/bv368aNWrIZvPsYV75+flKSEjQ3r17g+LcKsYb2BhvYGO8gS/QxmwYho4ePaq4uLgSB2qXJuhmbux2uy699FJLPyMyMjIg/iC5i/EGNsYb2Bhv4AukMV9sxqYYC4oBAEBAIdwAAICAQrjxoLCwMI0ePVphYWG+LsUrGG9gY7yBjfEGvmAcc7GgW1AMAAACGzM3AAAgoBBuAABAQCHcAACAgEK4AQAAAYVw4yFTp05VYmKiwsPD1aZNG61du9bXJZXL+PHj1bp1a9WoUUN169bVXXfdpW3btpVoc/LkSQ0YMECXXHKJqlevrj/96U/Kyckp0WbPnj3q3LmzIiIiVLduXT355JM6c+aMN4di2oQJE2Sz2TR48GDXc4E41n379un//u//dMkll6hq1apq3ry51q9f73rdMAyNGjVK9erVU9WqVZWSkqIff/yxxHscPnxYPXr0UGRkpGrWrKm+ffvq2LFj3h7KRRUVFWnkyJFq0KCBqlatqiuuuELPP/98ibNpKvN4v/rqK3Xt2lVxcXGy2Wz6+OOPS7zuqbF99913uuGGGxQeHq6EhAS99NJLVg+tVGWN9/Tp0xo2bJiaN2+uatWqKS4uTr169dL+/ftLvEdlGq908f/H5+rXr59sNpteffXVEs9XtjF7hIEKmzNnjhEaGmrMmjXL+OGHH4yHHnrIqFmzppGTk+Pr0kxLTU013nrrLSMrK8vIzMw00tLSjPr16xvHjh1ztenXr5+RkJBgpKenG+vXrzfatm1rtG/f3vX6mTNnjGbNmhkpKSnGxo0bjYULFxrR0dHG8OHDfTEkt6xdu9ZITEw0rrnmGmPQoEGu5wNtrIcPHzYuu+wy44EHHjC+/fZbY+fOncbixYuNn376ydVmwoQJRlRUlPHxxx8bmzZtMu644w6jQYMGxokTJ1xtbrvtNqNFixbGN998Y3z99ddGw4YNje7du/tiSGUaO3ascckllxjz5883du3aZXzwwQdG9erVjUmTJrnaVObxLly40HjmmWeMefPmGZKMjz76qMTrnhjbkSNHjJiYGKNHjx5GVlaW8Z///MeoWrWq8cYbb3hrmC5ljTcvL89ISUkx5s6da2zdutVYs2aNcd111xktW7Ys8R6VabyGcfH/x8XmzZtntGjRwoiLizP+8Y9/lHitso3ZEwg3HnDdddcZAwYMcP26qKjIiIuLM8aPH+/DqjwjNzfXkGSsWLHCMIyzP0CqVKlifPDBB642W7ZsMSQZa9asMQzj7F9Gu91uZGdnu9q8/vrrRmRkpHHq1CnvDsANR48eNRo1amQsWbLEuPHGG13hJhDHOmzYMKNDhw4XfN3pdBqxsbHG3//+d9dzeXl5RlhYmPGf//zHMAzD2Lx5syHJWLdunavN559/bthsNmPfvn3WFV8OnTt3Nh588MESz919991Gjx49DMMIrPH+/ovPU2ObNm2aUatWrRJ/nocNG2Y0adLE4hGVrawv+mJr1641JBk///yzYRiVe7yGceEx//LLL0Z8fLyRlZVlXHbZZSXCTWUfc3lxWaqCCgsLtWHDBqWkpLies9vtSklJ0Zo1a3xYmWccOXJEklS7dm1J0oYNG3T69OkS473yyitVv35913jXrFmj5s2bKyYmxtUmNTVV+fn5+uGHH7xYvXsGDBigzp07lxiTFJhj/fTTT9WqVSvdc889qlu3rpKTkzVjxgzX67t27VJ2dnaJMUdFRalNmzYlxlyzZk21atXK1SYlJUV2u13ffvut9wbjhvbt2ys9PV3bt2+XJG3atEkrV67U7bffLinwxnsuT41tzZo1+sMf/qDQ0FBXm9TUVG3btk2//fabl0ZTPkeOHJHNZlPNmjUlBeZ4nU6nevbsqSeffFJXX331ea8H4pjdQbipoIMHD6qoqKjEl5skxcTEKDs720dVeYbT6dTgwYN1/fXXq1mzZpKk7OxshYaGun5YFDt3vNnZ2aX+fhS/5k/mzJmjjIwMjR8//rzXAm2skrRz5069/vrratSokRYvXqz+/ftr4MCBevvttyX9r+ay/jxnZ2erbt26JV4PCQlR7dq1/W7MTz/9tO677z5deeWVqlKlipKTkzV48GD16NFDUuCN91yeGltl+zNe7OTJkxo2bJi6d+/uOjQyEMf74osvKiQkRAMHDiz19UAcszuC7lRwuG/AgAHKysrSypUrfV2KJfbu3atBgwZpyZIlCg8P93U5XuF0OtWqVSuNGzdOkpScnKysrCxNnz5dvXv39nF1nvf+++/r3Xff1Xvvvaerr75amZmZGjx4sOLi4gJyvDjr9OnTuvfee2UYhl5//XVfl2OZDRs2aNKkScrIyJDNZvN1OX6FmZsKio6OlsPhOG8HTU5OjmJjY31UVcU9+uijmj9/vpYtW6ZLL73U9XxsbKwKCwuVl5dXov25442NjS3196P4NX+xYcMG5ebm6tprr1VISIhCQkK0YsUKTZ48WSEhIYqJiQmYsRarV6+emjZtWuK5q666Snv27JH0v5rL+vMcGxur3NzcEq+fOXNGhw8f9rsxP/nkk67Zm+bNm6tnz556/PHHXTN1gTbec3lqbJXtz3hxsPn555+1ZMkS16yNFHjj/frrr5Wbm6v69eu7fob9/PPPGjp0qBITEyUF3pjdRbipoNDQULVs2VLp6emu55xOp9LT09WuXTsfVlY+hmHo0Ucf1UcffaSlS5eqQYMGJV5v2bKlqlSpUmK827Zt0549e1zjbdeunb7//vsSf6GKf8j8/ovVlzp16qTvv/9emZmZrkerVq3Uo0cP138HyliLXX/99edt7d++fbsuu+wySVKDBg0UGxtbYsz5+fn69ttvS4w5Ly9PGzZscLVZunSpnE6n2rRp44VRuO/48eOy20v+mHM4HHI6nZICb7zn8tTY2rVrp6+++kqnT592tVmyZImaNGmiWrVqeWk07ikONj/++KO+/PJLXXLJJSVeD7Tx9uzZU999912Jn2FxcXF68skntXjxYkmBN2a3+XpFcyCYM2eOERYWZsyePdvYvHmz8fDDDxs1a9YssYOmsujfv78RFRVlLF++3Dhw4IDrcfz4cVebfv36GfXr1zeWLl1qrF+/3mjXrp3Rrl071+vF26NvvfVWIzMz01i0aJFRp04dv90efa5zd0sZRuCNde3atUZISIgxduxY48cffzTeffddIyIiwvj3v//tajNhwgSjZs2axieffGJ89913xp133lnq9uHk5GTj22+/NVauXGk0atTIL7ZG/17v3r2N+Ph411bwefPmGdHR0cZTTz3lalOZx3v06FFj48aNxsaNGw1JxiuvvGJs3LjRtTvIE2PLy8szYmJijJ49expZWVnGnDlzjIiICJ9sEy5rvIWFhcYdd9xhXHrppUZmZmaJn1/n7gKqTOM1jIv/P/693++WMozKN2ZPINx4yJQpU4z69esboaGhxnXXXWd88803vi6pXCSV+njrrbdcbU6cOGE88sgjRq1atYyIiAjjj3/8o3HgwIES77N7927j9ttvN6pWrWpER0cbQ4cONU6fPu3l0Zj3+3ATiGP97LPPjGbNmhlhYWHGlVdeabz55pslXnc6ncbIkSONmJgYIywszOjUqZOxbdu2Em0OHTpkdO/e3ahevboRGRlp9OnTxzh69Kg3h+GW/Px8Y9CgQUb9+vWN8PBw4/LLLzeeeeaZEl92lXm8y5YtK/Xva+/evQ3D8NzYNm3aZHTo0MEICwsz4uPjjQkTJnhriCWUNd5du3Zd8OfXsmXLXO9RmcZrGBf/f/x7pYWbyjZmT7AZxjm36gQAAKjkWHMDAAACCuEGAAAEFMINAAAIKIQbAAAQUAg3AAAgoBBuAABAQCHcAACAgEK4AQAAAYVwAyDoJCYm6tVXX/V1GQAsQrgBYKkHHnhAd911lyTppptu0uDBg7322bNnz1bNmjXPe37dunV6+OGHvVYHAO8K8XUBAGBWYWGhQkNDy92/Tp06HqwGgL9h5gaAVzzwwANasWKFJk2aJJvNJpvNpt27d0uSsrKydPvtt6t69eqKiYlRz549dfDgQVffm266SY8++qgGDx6s6OhopaamSpJeeeUVNW/eXNWqVVNCQoIeeeQRHTt2TJK0fPly9enTR0eOHHF93nPPPSfp/MtSe/bs0Z133qnq1asrMjJS9957r3JyclyvP/fcc0pKStK//vUvJSYmKioqSvfdd5+OHj1q7W8agHIh3ADwikmTJqldu3Z66KGHdODAAR04cEAJCQnKy8vTzTffrOTkZK1fv16LFi1STk6O7r333hL93377bYWGhmrVqlWaPn26JMlut2vy5Mn64Ycf9Pbbb2vp0qV66qmnJEnt27fXq6++qsjISNfnPfHEE+fV5XQ6deedd+rw4cNasWKFlixZop07d6pbt24l2u3YsUMff/yx5s+fr/nz52vFihWaMGGCRb9bACqCy1IAvCIqKkqhoaGKiIhQbGys6/nXXntNycnJGjdunOu5WbNmKSEhQdu3b1fjxo0lSY0aNdJLL71U4j3PXb+TmJioF154Qf369dO0adMUGhqqqKgo2Wy2Ep/3e+np6fr++++1a9cuJSQkSJLeeecdXX311Vq3bp1at24t6WwImj17tmrUqCFJ6tmzp9LT0zV27NiK/cYA8DhmbgD41KZNm7Rs2TJVr17d9bjyyislnZ0tKdayZcvz+n755Zfq1KmT4uPjVaNGDfXs2VOHDh3S8ePH3f78LVu2KCEhwRVsJKlp06aqWbOmtmzZ4nouMTHRFWwkqV69esrNzTU1VgDewcwNAJ86duyYunbtqhdffPG81+rVq+f672rVqpV4bffu3erSpYv69++vsWPHqnbt2lq5cqX69u2rwsJCRUREeLTOKlWqlPi1zWaT0+n06GcA8AzCDQCvCQ0NVVFRUYnnrr32Wv33v/9VYmKiQkLc/5G0YcMGOZ1Ovfzyy7Lbz05Cv//++xf9vN+76qqrtHfvXu3du9c1e7N582bl5eWpadOmbtcDwH9wWQqA1yQmJurbb7/V7t27dfDgQTmdTg0YMECHDx9W9+7dtW7dOu3YsUOLFy9Wnz59ygwmDRs21OnTpzVlyhTt3LlT//rXv1wLjc/9vGPHjik9PV0HDx4s9XJVSkqKmjdvrh49eigjI0Nr165Vr169dOONN6pVq1Ye/z0AYD3CDQCveeKJJ+RwONS0aVPVqVNHe/bsUVxcnFatWqWioiLdeuutat68uQYPHqyaNWu6ZmRK06JFC73yyit68cUX1axZM7377rsaP358iTbt27dXv3791K1bN9WpU+e8BcnS2ctLn3zyiWrVqqU//OEPSklJ0eWXX665c+d6fPwAvMNmGIbh6yIAAAA8hZkbAAAQUAg3AAAgoBBuAABAQCHcAACAgEK4AQAAAYVwAwAAAgrhBgAABBTCDQAACCiEGwAAEFAINwAAIKAQbgAAQED5f8IR0mAWaQX/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "schedule = mx.lr_scheduler.PolyScheduler(max_update=1000, base_lr=1, pwr=2)\n",
    "plot_schedule(schedule)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "122a6ea7",
   "metadata": {},
   "source": [
    "![lr poly](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/lr_schedules/polynomial.png) <!--notebook-skip-line-->\n",
    "\n",
    "\n",
    "Note: unlike `FactorScheduler`, the `base_lr` is set as an argument when instantiating the schedule.\n",
    "\n",
    "And we don't evaluate at `iteration=0` (to get `base_lr`) since we are working with schedules starting at `iteration=1`.\n",
    "\n",
    "### Custom Schedules\n",
    "\n",
    "You can implement your own custom schedule with a function or callable class, that takes an integer denoting the iteration index (starting at 1) and returns a float representing the learning rate to be used for that iteration. We implement the Cosine Annealing Schedule in the example below as a callable class (see `__call__` method)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0eae412b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABClUlEQVR4nO3de1yUZf7/8ffMIOAJUEkQRLHyjImiEWlbJuWG2mF/W2aumrm1miUeak1LbSsPHddTarmatbutWt9qS0nXUCuL8kiF5zyEqYBogKKJMvfvj5ZZSVTuYQ7M8Ho+HvPIbq5r5nOZwdv7/tz3ZTEMwxAAAICfsHq7AAAAAFci3AAAAL9CuAEAAH6FcAMAAPwK4QYAAPgVwg0AAPArhBsAAOBXArxdgKfZ7XYdPnxY9evXl8Vi8XY5AACgEgzD0IkTJxQVFSWr9dLnZmpcuDl8+LBiYmK8XQYAAHDCwYMH1bRp00uOqXHhpn79+pJ++c0JCQnxcjUAAKAyioqKFBMT4/g5fik1LtyUXYoKCQkh3AAA4GMq01JCQzEAAPArhBsAAOBXCDcAAMCvEG4AAIBfIdwAAAC/QrgBAAB+hXADAAD8CuEGAAD4FcINAADwKzXuCcXucrqkVE9/9J3WbM9VwalzKjV+OW5IskiyWaV6wbXUq32kJvdtr9qBNm+WCwCA3/LqmZvPPvtMffv2VVRUlCwWiz744IPLzlm3bp06d+6soKAgXX311Vq8eLHb67ycB9/aqLaTVmrpxkM6WnxOZw3Jrl9exn//edYu/XTqrJZsPKi2k1Yq9okVavlkmjr+ZZUeeGODTv58zruLAADAT3g13BQXF6tjx4569dVXKzV+//796t27t3r06KHMzEyNGjVKf/zjH7Vq1So3V3pxD761Uau35zk192ypocLT57Rm11HFPb1KsU+sULfp6Zq79nuVnLO7uFIAAGoGi2EYhreLkH7ZCOv999/XnXfeedEx48aN04oVK5SVleU4du+996qgoEArV66s1OcUFRUpNDRUhYWFVd4483RJqdpOqtznOqNukE0jbrpaf7zhSgUG0B4FAKi5zPz89qmfmBkZGUpOTi53rFevXsrIyLjonDNnzqioqKjcy1Wmpm132XtVpPhMqV5YtUutnvpYnZ/9j9bsyFOpvVpkUQAAqi2fCjc5OTmKiIgodywiIkJFRUU6ffp0hXOmTZum0NBQxysmJsZl9Rw4dspl73U5x4vP6oE3N+qqCWn6w4KvdLqk1GOfDQCAL/GpcOOM8ePHq7Cw0PE6ePCgy947tlEdl72XGev3HlPbSSt1/bRPCDkAAPyKT4WbyMhI5ebmljuWm5urkJAQ1a5du8I5QUFBCgkJKfdylQkp7Vz2Xs44XHhGbSetVM+X1tCADADAf/lUuElKSlJ6enq5Y6tXr1ZSUpJX6qkdaNMt7Rp75bPPtzf/tFo99bF6/XUdZ3IAADWeV8PNyZMnlZmZqczMTEm/3OqdmZmp7OxsSb9cUho0aJBj/LBhw7Rv3z79+c9/1s6dOzV37lwtW7ZMo0eP9kb5kqQFg7pWi4AjSbtyizmTAwCo8bx6K/i6devUo0ePC44PHjxYixcv1v33368DBw5o3bp15eaMHj1a27dvV9OmTTVx4kTdf//9lf5MV94Kfr5LPaHYW7/Bv41rrFfv6yKb1eKlCgAAcA0zP7+rzXNuPMVd4eZyTpeUatKH32rVdzk6ecYuT55XmXNvvPrER3vwEwEAcC3CzSV4K9xUpOxsT9o3R3SixL1xp1NMiN4d3p2zOAAAn0S4uYTqFG7OV2o3tH7XUT2Xtk17jrrv+TmcxQEA+CLCzSVU13BzvrKg88T73+hIUYnL35+zOAAAX0O4uQRfCDfnKzln15/fzdQHmUdc/t6cxQEA+ArCzSX4WrgpU3Y258F/blLJOdf9J7u5dSMtGnKdy94PAAB38NuNM2sym9WiG9s21u7nUvTXe+LlqgtKa3YdU7ep/3HRuwEA4H2EGx90V+dofT81RW8O7qrAgKrHnENFZ9Vywgoe/AcA8AuEGx/16zM5VXXWLrV66mP95aPvql4cAABeRLjxA3d1jtbeqSlq4YJdyt/4Ilvdp692QVUAAHgH4cZP2KwWrX28h2beG1/l9/qxoERxkz5Wqb1G9ZoDAPwE4cbP3BHvmrM4J0vsumpCmlZmuf4WdAAA3Ilw44dceRZn2D+2KO3bw1UvCgAADyHc+DFXncV5+O2t+nDLIRdVBQCAexFu/JyrzuKMXJapP765wTVFAQDgRoSbGqLsLE5wFZ6L88mOo3rgja9dWBUAAK5HuKlBbFaLdj6XovZN6jv9Hmt25WvIIgIOAKD6ItzUQCtSf6MHusU6PX/t7nz1mfmp6woCAMCFCDc11KS+7TX3vs5Oz886clK9CTgAgGqIcFODpVzTRHunpqhuLef6cLYdOamUGetcWxQAAFVEuKnhbFaLtj2bogZ1Apyavz2nWN2nf+LiqgAAcB7hBpKkrZN6KTosyKm5PxacIeAAAKoNwg0cvngiWe2a1HNq7o8FZ5RCDw4AoBog3KCctNQbFRfl3K3i24+cVJ9Zn7m4IgAAzCHc4ALLR/5GPVqFOzU36/AJnoMDAPAqwg0q9MYDibq59RVOzV27mwf9AQC8h3CDi1o05Fr1bON8wOnLJSoAgBcQbnBJC++/VkOcfJrxd4dP6IE32GwTAOBZhBtc1uS+7TW0e6xTc9fsOqq/fLTNtQUBAHAJhBtUysQ+zgecN744oCkrtru2IAAALoJwg0qb2Ke9hnRr7tTcBZ/vV9q3R1xcEQAAFyLcwJTJfeN0c2vnbhN/+O0tKrUbLq4IAIDyCDcwbdGQRHVw8kF/ic/9x8XVAABQHuEGTvnIyQf95Z86xzYNAAC3ItzAaW88kOhUwGGbBgCAOxFuUCVvPJCoOCc228w6fEJDF290Q0UAgJqOcIMqW556o9o7EXDSd+bpo28Ou6EiAEBNRriBS6xIvVHNGwSbnvfov7ZyBxUAwKUIN3CZNY/f7NS8+L+scnElAICajHADl7FZLZp7XyfT806cKVVvGowBAC5CuIFLpVwTpQdviDU9b9vhE3r6wyzXFwQAqHEIN3C5J3s7t03D4i9/YA8qAECVEW7gFs5u08AeVACAqiLcwG0WDUlU+8i6pueNYA8qAEAVEG7gVitG3aTwugGm5hiSer601j0FAQD8HuEGbvf1k7eannPg+GmeYAwAcArhBm5ns1o059540/N4gjEAwBmEG3hEn/ho9WxjvsGYJxgDAMwi3MBjFt6fqNiG5rdooP8GAGAG4QYelf7Yzab/0NF/AwAwg3ADj7JZLZrjxBYN9N8AACqLcAOPc3aLBvpvAACVQbiBVzzZu70GX9/M9Dz6bwAAl0O4gdf85fYOphuM6b8BAFwO4QZelf7Yzebn0H8DALgEwg28yma1aK4TDcYj6b8BAFwE4QZel3JNlIZ0M9d/Y0j6/dwv3FMQAMCneT3cvPrqq4qNjVVwcLASExO1YcOGS46fMWOGWrdurdq1aysmJkajR4/Wzz//7KFq4S6T+5rvv9n6Y6GeXb7dTRUBAHyVV8PN0qVLNWbMGE2ePFlbtmxRx44d1atXL+Xl5VU4/u2339YTTzyhyZMna8eOHVq4cKGWLl2qCRMmeLhyuIMz/TcL1+9X2rdH3FANAMBXeTXcvPLKK3rwwQc1ZMgQtWvXTvPnz1edOnW0aNGiCsd/+eWX6tatm+677z7Fxsbq1ltvVf/+/S97tge+wfn+my303wAAHLwWbkpKSrR582YlJyf/rxirVcnJycrIyKhwzvXXX6/Nmzc7wsy+ffuUlpamlJSUi37OmTNnVFRUVO6F6ivlmiilxEWYmnPOkB59e4ubKgIA+BqvhZv8/HyVlpYqIqL8D7KIiAjl5ORUOOe+++7TM888o+7du6tWrVq66qqrdNNNN13ystS0adMUGhrqeMXExLh0HXC92fclyGYxNyctK4fLUwAASdWgodiMdevWaerUqZo7d662bNmi9957TytWrNCzzz570Tnjx49XYWGh43Xw4EEPVgxn2KwWze5v/vLUI29zeQoAIAV464PDw8Nls9mUm5tb7nhubq4iIyMrnDNx4kQNHDhQf/zjHyVJHTp0UHFxsR566CE9+eSTslovzGpBQUEKCgpy/QLgVinXRGlo9nEtXP9DpefYJd0970u9N6Kb+woDAFR7XjtzExgYqISEBKWnpzuO2e12paenKykpqcI5p06duiDA2Gw2SZJh8Dd2fzOxT5w6xYSYmrPlYAFPLwaAGs6rl6XGjBmjBQsW6M0339SOHTs0fPhwFRcXa8iQIZKkQYMGafz48Y7xffv21bx587RkyRLt379fq1ev1sSJE9W3b19HyIF/eXd4d5lsv+HpxQBQw3ntspQk9evXT0ePHtWkSZOUk5Oj+Ph4rVy50tFknJ2dXe5MzVNPPSWLxaKnnnpKhw4d0hVXXKG+fftqypQp3loC3MxmtWj2vfF6ZElmpecY4vIUANRkFqOGXc8pKipSaGioCgsLFRJi7pIHvGfo4q+VvjPf1JzZ/Tupb8coN1UEAPAkMz+/fepuKdRcC+9PVET9WqbmPMrlKQCokQg38Bmfj0u+/KBf+f08NtcEgJqGcAOfERhg1dDuzU3N2XqwkLunAKCGIdzAp0zsE6fYRuZ2D+fyFADULIQb+Jz0seZ3D09+eZ3rCwEAVEuEG/gcm9WiWfd0NDVn/7FT+nfmITdVBACoTgg38Em3d25q+vJU6pJMLk8BQA1AuIHPcuby1N3zvnRDJQCA6oRwA59ls1o05954U3PYewoA/B/hBj6tT3y06c012XsKAPwb4QY+793h3U2NL9t7CgDgnwg38HlcngIAnI9wA7/QJz5anbk8BQAQ4QZ+5B0uTwEARLiBH7FZLRrZ4ypTc7g8BQD+h3ADv5J6S2vZLObmjFrC5SkA8CeEG/gVm9Wi2f07mZpTakiPvr3FTRUBADyNcAO/k3JNlIZ2b25qTlpWjtK+PeKmigAAnkS4gV+a2CfO9MP9Hv3XFi5PAYAfINzAb5l9uB+XpwDAPxBu4LecuXsqLStHJefsbqoIAOAJhBv4tdRbWstq8u6p3rM+c08xAACPINzAr9msFs3qF29qzp68Yp59AwA+jHADv9cnPlo924SbmpPKs28AwGcRblAjLLw/URH1a1V6vJ3mYgDwWYQb1Bifj0s2NZ7mYgDwTYQb1BiBAValxEWYmkNzMQD4HsINapTZ9yXIzM1TNBcDgO8h3KBGsVktmn1vvKk5NBcDgG8h3KDG6RMfrZZX1Kn0eJqLAcC3EG5QI61IvdHUeJqLAcB3EG5QI9FcDAD+i3CDGovmYgDwT4Qb1Fg0FwOAfyLcoEajuRgA/A/hBjUezcUA4F8IN6jxaC4GAP9CuAFEczEA+BPCDSCaiwHAnxBugP+iuRgA/APhBjgPzcUA4PsIN8B5aC4GAN9HuAF+heZiAPBthBvgV2guBgDfRrgBKkBzMQD4LsINcBE0FwOAbyLcABdBczEA+KYqhZuff/7ZVXUA1RLNxQDge0yHG7vdrmeffVbR0dGqV6+e9u3bJ0maOHGiFi5c6PICAW+iuRgAfI/pcPPcc89p8eLFeuGFFxQYGOg4HhcXp7/97W8uLQ6oDmguBgDfYjrcvPXWW3r99dc1YMAA2Ww2x/GOHTtq586dLi0OqC5oLgYA32E63Bw6dEhXX331BcftdrvOnj3rkqKA6obmYgDwHabDTbt27fT5559fcPzdd99Vp06dXFIUUB3RXAwAviHA7IRJkyZp8ODBOnTokOx2u9577z3t2rVLb731lpYvX+6OGoFqoay5+JElmZWek7pkq1I6NJHNaiYWAQCqwvSZmzvuuEMfffSRPvnkE9WtW1eTJk3Sjh079NFHH+mWW25xR41AtUFzMQBUfxbDMGrUPatFRUUKDQ1VYWGhQkJCvF0OfFDJObtaPfWxqTm7n7tNgQE8MxMAnGXm57fp77ZXXnmljh07dsHxgoICXXnllWbfDvA5NBcDQPVmOtwcOHBApaWlFxw/c+aMDh06ZLqAV199VbGxsQoODlZiYqI2bNhwyfEFBQUaMWKEmjRpoqCgILVq1UppaWmmPxeoCpqLAaD6qnRD8Ycffuj49apVqxQaGur499LSUqWnpys2NtbUhy9dulRjxozR/PnzlZiYqBkzZqhXr17atWuXGjdufMH4kpIS3XLLLWrcuLHeffddRUdH64cfflBYWJipzwWqiuZiAKi+Kt1zY7X+cpLHYrHo11Nq1aql2NhYvfzyy+rTp0+lPzwxMVFdu3bVnDlzJP3yrJyYmBg9+uijeuKJJy4YP3/+fL344ovauXOnatWqVanPOHPmjM6cOeP496KiIsXExNBzA5e45eW12nP0VKXHp8RFau4fEtxYEQD4J7f03NjtdtntdjVr1kx5eXmOf7fb7Tpz5ox27dplKtiUlJRo8+bNSk5O/l8xVquSk5OVkZFR4ZwPP/xQSUlJGjFihCIiIhQXF6epU6dWeJmszLRp0xQaGup4xcTEVLpG4HJ4cjEAVD+me27279+v8PDwKn9wfn6+SktLFRFRvjEzIiJCOTk5Fc7Zt2+f3n33XZWWliotLU0TJ07Uyy+/rOeee+6inzN+/HgVFhY6XgcPHqxy7UAZZ5qLn/i/b9xUDQBAcuIhfpJUXFysTz/9VNnZ2SopKSn3tZEjR7qksIrY7XY1btxYr7/+umw2mxISEnTo0CG9+OKLmjx5coVzgoKCFBQU5LaagNn3JejjCWmq7DMV3tt6WC/eHU/vDQC4ielws3XrVqWkpOjUqVMqLi5Ww4YNlZ+frzp16qhx48aVDjfh4eGy2WzKzc0tdzw3N1eRkZEVzmnSpIlq1apVbsPOtm3bKicnRyUlJeV2KQc8xZnm4rvnfan3RnRzX1EAUIOZviw1evRo9e3bVz/99JNq166tr776Sj/88IMSEhL00ksvVfp9AgMDlZCQoPT0dMcxu92u9PR0JSUlVTinW7du+v7772W3/69nYffu3WrSpAnBBl5l9snFWw4WcGs4ALiJ6XCTmZmpsWPHymq1ymaz6cyZM4qJidELL7ygCRMmmHqvMWPGaMGCBXrzzTe1Y8cODR8+XMXFxRoyZIgkadCgQRo/frxj/PDhw3X8+HGlpqZq9+7dWrFihaZOnaoRI0aYXQbgcmabi0cv3apSe416QDgAeITpy1K1atVy3BbeuHFjZWdnq23btgoNDTXdrNuvXz8dPXpUkyZNUk5OjuLj47Vy5UpHk3F2drbjsyQpJiZGq1at0ujRo3XNNdcoOjpaqampGjdunNllAC4XGGDVdS0a6Kv9P1Vq/Dm7NHP1bo3p1drNlQFAzWJ6b6lbb71V999/v+677z49+OCD+vbbbzVy5Ej9/e9/108//aSvv/7aXbW6BHtLwZ2c2Xdq79QUmosB4DLcurfU1KlT1aRJE0nSlClT1KBBAw0fPlxHjx7Va6+95lzFgJ9w5tbwu+d96aZqAKBmYldwwMVK7YaummBuv7PZ/Tupb8coN1UEAL7PrWduLmbLli2mnlAM+Cub1aKRPa4yNYfmYgBwHVPhZtWqVXrsscc0YcIE7du3T5K0c+dO3XnnneratWu5W7SBmiz1ltYy00ZT1lwMAKi6SoebhQsX6rbbbtPixYv1/PPP67rrrtM//vEPJSUlKTIyUllZWUpLM3cqHvBXNqtFs/rFm5ozZ933nL0BABeodLiZOXOmnn/+eeXn52vZsmXKz8/X3Llz9d1332n+/Plq27atO+sEfE6f+Gh1jql8X5fd4OwNALhCpRuK69atq23btik2NlaGYSgoKEhr165Vt26+9Qh5GorhSc40F3NrOABcyC0NxadPn1adOr88Xt5isSgoKMhxSziAijnTXMyt4QBQNaaeUPy3v/1N9erVkySdO3dOixcvVnh4eLkx7twVHPBFqbe01px1e1XZdpqyfae4NRwAnFPpy1KxsbGyWC59qtxisTjuoqquuCwFb1ieecjUruEBVmnXc1yeAoAyZn5+V/rMzYEDB6paF1Bj9YmP1sw1e7Qnr7hS49l3CgCc57KH+AG4tBUjf2Nq/Ky13BoOAM4g3AAewr5TAOAZhBvAg2bfl2BqfFlzMQCg8gg3gAc5c2v42GWZXJ4CABMIN4CHmd13qqTU0Jd78t1XEAD4GdPhpqioqMLXiRMnVFJS4o4aAb/izL5Tj/1fpltqAQB/ZDrchIWFqUGDBhe8wsLCVLt2bTVv3lyTJ09mh3DgEszuO5VbVELvDQBUkulws3jxYkVFRWnChAn64IMP9MEHH2jChAmKjo7WvHnz9NBDD2nWrFmaPn26O+oF/MY7w7ubGp+6ZCu9NwBQCZV+QnGZnj176k9/+pPuueeecseXLVum1157Tenp6fr73/+uKVOmaOfOnS4t1hV4QjGqk1dW7dSstXsrPT4lLlJz/2DujisA8Adu2TizzJdffqlOnTpdcLxTp07KyMiQJHXv3l3Z2dlm3xqocVJvaS0zGyykZeWo5ByXfAHgUkyHm5iYGC1cuPCC4wsXLlRMTIwk6dixY2rQoEHVqwP8nM1q0aMmbw3vPeszN1UDAP7B1K7gkvTSSy/p7rvv1scff6yuXbtKkjZt2qSdO3fq3XfflSRt3LhR/fr1c22lgJ8yu2v4nrxidg0HgEsw3XMjSfv379drr72m3bt3S5Jat26tP/3pT4qNjXV1fS5Hzw2qI3YNB4BLM/Pz26lw48sIN6iubnl5rfYcPVXp8SN7XM2u4QBqDLeHm4KCAm3YsEF5eXkXPM9m0KBBZt/Oowg3qK5KztnV6qmPKz3eapH2TOHsDYCawczPb9M9Nx999JEGDBigkydPKiQkRBbL/76xWiyWah9ugOqqbNfwtKzcSo23G9LM1bs5ewMAv2L6bqmxY8fqgQce0MmTJ1VQUKCffvrJ8Tp+/Lg7agRqjNn3JZi6NXzW2u95sB8A/IrpcHPo0CGNHDlSderUcUc9QI1ms1o0+954U3Punvele4oBAB9lOtz06tVLmzZtckctAPTLvlORIUGVHr/lYAH7TgHAeUz33PTu3VuPP/64tm/frg4dOqhWrVrlvn777be7rDigpnrx9x01cNGGSo8fvXSrUjo0obkYAOTE3VJW68VP9lgsFpWWlla5KHfibin4glK7oTZPpemsiZ0WuDUcgD9z695Sdrv9oq/qHmwAX2GzWvTXe+JNzZlNczEASHIi3ADwjD7x0eocU/mzi4akR9/e4r6CAMBHVKrnZtasWXrooYcUHBysWbNmXXLsyJEjXVIYAOmd4d111YS0So8v2zU8MIC/twCouSrVc9OiRQtt2rRJjRo1UosWLS7+ZhaL9u3b59ICXY2eG/iaV1bt1Ky1eys9/roWDbXkT0lurAgAPM/lTyjev39/hb8G4H5mdw3/av9xzt4AqNH47gdUczarRbP6xZua03vWZ+4pBgB8gOnn3JSWlmrx4sVKT0+vcOPMNWvWuKw4AL/oEx+tRV/s05aDRZUavyevWB99c1h9O0a5uTIAqH5Mh5vU1FQtXrxYvXv3VlxcXLmNMwG4j9nm4tQlPNgPQM1kOtwsWbJEy5YtU0pKijvqAXARNqtFv4uP0nuZldtqwW78cmv43D8kuLkyAKheTPfcBAYG6uqrr3ZHLQAuY/rvO5oaX3ZrOADUJKbDzdixYzVz5kyZ3LUBgAsEBliVEhdhag7NxQBqGtN7S911111au3atGjZsqPbt21+wceZ7773n0gJdjefcwNeV2g1dPSFNZv7Hnd2/E83FAHyay59zc76wsDDdddddThcHoGpsVotm3xuvR5ZkVnrO2GWZNBcDqDFMhZtz586pR48euvXWWxUZGemumgBcRp/4aM1M3609R09VanxJqaHZ6Xs06pZWbq4MALzPVM9NQECAhg0bpjNnzrirHgCVtCL1RlPjZ6/Zw67hAGoE0w3F1157rbZu3eqOWgCYYLa5uNSQZq7e7caKAKB6MN1z8/DDD2vs2LH68ccflZCQoLp165b7+jXXXOOy4gBc2uz7EvSxiebiWWu/V+otrei9AeDXTN8tZbVeeLLHYrHIMAxZLBaVlpa6rDh34G4p+JsZq3dpRvr3lR7fOSZM743o5saKAMD13Hq3FLuCA9XLoz1bafaa71Vayb+mbDlYwL5TAPya6XDTvHlzd9QBwEk2q0UjbrpKs9burfSc0UvZdwqA/zIdbsps375d2dnZKikpKXf89ttvr3JRAMxJvaW15qzbq8reDHXO/ktz8Zherd1bGAB4gelws2/fPt1111367rvvHL02khy7g1f3nhvAH9msFs3qZ+7BfjQXA/BXpm8FT01NVYsWLZSXl6c6depo27Zt+uyzz9SlSxetW7fODSUCqIw+8dHqHGOuSf7ueV+6qRoA8B7T4SYjI0PPPPOMwsPDZbVaZbVa1b17d02bNk0jR450R40AKumd4d1NjS9rLgYAf2I63JSWlqp+/fqSpPDwcB0+/Ms3xubNm2vXrl1OFfHqq68qNjZWwcHBSkxM1IYNGyo1b8mSJbJYLLrzzjud+lzA39isFo3scZWpOWOXZfLkYgB+xXS4iYuL0zfffCNJSkxM1AsvvKAvvvhCzzzzjK688krTBSxdulRjxozR5MmTtWXLFnXs2FG9evVSXl7eJecdOHBAjz32mG644QbTnwn4s9RbWstMG01JqaEv9+S7ryAA8DDT4eapp56S3W6XJD3zzDPav3+/brjhBqWlpWnWrFmmC3jllVf04IMPasiQIWrXrp3mz5+vOnXqaNGiRRedU1paqgEDBugvf/mLU4EK8GdlzcVmPPZ/mW6pBQC8wXS46dWrl373u99Jkq6++mrt3LlT+fn5ysvL080332zqvUpKSrR582YlJyf/ryCrVcnJycrIyLjovGeeeUaNGzfW0KFDL/sZZ86cUVFRUbkX4O/MNhfnFpXQewPAb5gON2W+//57rVq1SqdPn1bDhg2deo/8/HyVlpYqIqL85n8RERHKycmpcM769eu1cOFCLViwoFKfMW3aNIWGhjpeMTExTtUK+BqzzcWpS7bSewPAL5gON8eOHVPPnj3VqlUrpaSk6MiRI5KkoUOHauzYsS4v8HwnTpzQwIEDtWDBAoWHh1dqzvjx41VYWOh4HTx40K01AtWF2eZiuyE9+vYWN1YEAJ5hOtyMHj1atWrVUnZ2turUqeM43q9fP61cudLUe4WHh8tmsyk3N7fc8dzcXEVGRl4wfu/evTpw4ID69u2rgIAABQQE6K233tKHH36ogIAA7d174ePng4KCFBISUu4F1BSpt7SWmUf0pWXlqOSc3W31AIAnmA43//nPf/T888+radOm5Y63bNlSP/zwg6n3CgwMVEJCgtLT0x3H7Ha70tPTlZSUdMH4Nm3a6LvvvlNmZqbjdfvtt6tHjx7KzMzkkhPwKzarRY+avDW896zP3FQNAHiG6e0XiouLy52xKXP8+HEFBQWZLmDMmDEaPHiwunTpomuvvVYzZsxQcXGxhgwZIkkaNGiQoqOjNW3aNAUHBysuLq7c/LCwMEm64DiAX5jdd2pPXjG7hgPwaabP3Nxwww166623HP9usVhkt9v1wgsvqEePHqYL6Nevn1566SVNmjRJ8fHxyszM1MqVKx1NxtnZ2Y6+HgDmOXNrOM3FAHyZxSjb+bKSsrKy1LNnT3Xu3Flr1qzR7bffrm3btun48eP64osvdNVV5k6Be1pRUZFCQ0NVWFhI/w1qlFteXqs9R09VenxKXKTm/iHBjRUBQOWZ+fnt1BOKd+/ere7du+uOO+5QcXGxfve732nr1q3VPtgANdmK1BtNjae5GICvMn3m5mJ+/PFHPfPMM3r99ddd8XZuw5kb1GQP/2OT0rJyLz/wv1o2rqvVY25yX0EAUEluPXNzMceOHdPChQtd9XYA3GD2fQmmbg0vay4GAF/isnADoPqzWS2afW+8qTkj/0VzMQDfQrgBapg+8dFqecWFj3O4GEPS3fO+dF9BAOBihBugBjLbXLzlYAGXpwD4jEo/xK9sJ/CLKSgoqGotADwkMMCqlLgIU83Fo5duVUqHJrJZzXTtAIDnVfrMzfk7a1f0at68uQYNGuTOWgG4kNnm4nN2aebq3W6rBwBcxWW3gvsKbgUH/md55iE9siTT1Jy9U1M4ewPA47xyKzgA39MnPlqdY8yFfJqLAVR3hBughntneHdT42kuBlDdEW6AGs5mtWhkD3Nbp4xeyrNvAFRfhBsASr2ltcy00dBcDKA6I9wAkM1q0ax+8abmzFr7PWdvAFRLhBsAkmguBuA/CDcAHGguBuAPCDcAHGguBuAPCDcAyqG5GICvI9wAKIfmYgC+jnAD4AI0FwPwZYQbABWiuRiAryLcAKgQzcUAfBXhBsBF0VwMwBcRbgBcFM3FAHwR4QbAJdFcDMDXEG4AXBbNxQB8CeEGwGU501z86L9oLgbgHYQbAJVitrlYkn4/7wv3FAMAl0C4AVApzjQXbz1YyOUpAB5HuAFQac40F6cu4fIUAM8i3AAwxWxzsd2QHn17i5uqAYALEW4AmGKzWjTrno6m5qRl5ajknN1NFQFAeYQbAKbd3rmpYhsFm5rTe9ZnbqoGAMoj3ABwSvrYm02N35NXTHMxAI8g3ABwis1q0Zx7403N4dk3ADyBcAPAaX3io9Xyijqm5iS/vM49xQDAfxFuAFTJitQbTY3ff+yU/p15yE3VAADhBkAVBQZYlRIXYWpO6pJMLk8BcBvCDYAqm31fgkzuzMDO4QDchnADoMpsVotmm2wuZudwAO5CuAHgEn3io9XJ5NYMI7l7CoAbEG4AuMy7JrdmMMTlKQCuR7gB4DLOPPuGy1MAXI1wA8ClnNk5nMtTAFyJcAPA5czuHM7lKQCuRLgB4HI2q0Uje1xlag6XpwC4CuEGgFuk3tJaNpMPv+HyFABXINwAcAub1aLZ/TuZmsPlKQCuQLgB4DYp10RpaPfmpuZweQpAVRFuALjVxD5xutrkzuGPcnkKQBUQbgC4XZrJncMl6ffzvnBDJQBqAsINALcLDLCavjy19WAhl6cAOIVwA8AjJvaJU2yjYFNzuDwFwBmEGwAekz72ZtNzkl9e5/pCAPg1wg0Aj7FZLZp1T0dTc/YfO6V/Zx5yU0UA/BHhBoBH3d65qenLU6lLMrk8BaDSqkW4efXVVxUbG6vg4GAlJiZqw4YNFx27YMEC3XDDDWrQoIEaNGig5OTkS44HUP04c3nquimr3VAJAH/k9XCzdOlSjRkzRpMnT9aWLVvUsWNH9erVS3l5eRWOX7dunfr376+1a9cqIyNDMTExuvXWW3XoEKetAV9hs1o05954U3OOFp/VA4v5iwyAy7MYhuHVc72JiYnq2rWr5syZI0my2+2KiYnRo48+qieeeOKy80tLS9WgQQPNmTNHgwYNuuz4oqIihYaGqrCwUCEhIVWuH4Dz7nr1c209WGRqzuz+ndS3Y5SbKgJQXZn5+e3VMzclJSXavHmzkpOTHcesVquSk5OVkZFRqfc4deqUzp49q4YNG1b49TNnzqioqKjcC0D18O7w7qbncHs4gMvxarjJz89XaWmpIiIiyh2PiIhQTk5Opd5j3LhxioqKKheQzjdt2jSFhoY6XjExMVWuG4BrOHN5SpJ6vrTW9cUA8Bte77mpiunTp2vJkiV6//33FRxc8d0X48ePV2FhoeN18OBBD1cJ4FL6xEfr5taNTM05cPy0hi7e6KaKAPg6r4ab8PBw2Ww25ebmljuem5uryMjIS8596aWXNH36dP3nP//RNddcc9FxQUFBCgkJKfcCUL0sGnKdQoNtpuak78xjewYAFfJquAkMDFRCQoLS09Mdx+x2u9LT05WUlHTReS+88IKeffZZrVy5Ul26dPFEqQDcbONTt5qeQ/8NgIp4/bLUmDFjtGDBAr355pvasWOHhg8fruLiYg0ZMkSSNGjQII0fP94x/vnnn9fEiRO1aNEixcbGKicnRzk5OTp58qS3lgDABZzZXFOi/wbAhbwebvr166eXXnpJkyZNUnx8vDIzM7Vy5UpHk3F2draOHDniGD9v3jyVlJTo97//vZo0aeJ4vfTSS95aAgAXcWZzTfpvAPya159z42k85wao3krthlpOSJPd5DyefwP4N595zg0A/JrNatGc+zqZnkf/DYAyhBsA1U7KNVEa0q2Z6Xn03wCQCDcAqqnJfTsotiH9NwDMI9wAqLbSHzO/ezjPvwFAuAFQbdmsFs2l/waASYQbANWas/03ic/9xw3VAPAFhBsA1Z4z/Tf5p86p96zP3FQRgOqMcAPAJzjTf7Pt8AkajIEaiHADwCc4239DgzFQ8xBuAPgMZ/tvaDAGahbCDQCfMrlvB7WPrGt6XsenV7qhGgDVEeEGgM9ZMeomhQSZ+/Z1ssSu7s+nu6kiANUJ4QaAT9o6+bem5/z4088a8sbXbqgGQHVCuAHgk2xWi+bcG2963tpd+Xp2+XbXFwSg2iDcAPBZfeKj1bNNuOl5C9fvV9q3R9xQEYDqgHADwKctvD/RqQbjh9/ewh1UgJ8i3ADweStG3aTwugGm510z+WM3VAPA2wg3APzC10/eanpO8VlDnZ5Z5YZqAHgT4QaAX3D2CcY/nTqnbtM/cUNFALyFcAPAb6RcE6Wh3Zubnneo4IxSZn7qhooAeAPhBoBfmdgnTj1aNTI9b/uRk+rDLuKAXyDcAPA7bzxwnZqGBpqel3X4hB54Y4MbKgLgSYQbAH5p/fhb1KC2zfS8NbuO6i8fbXNDRQA8hXADwG9tnfxb1Q00/23ujS8O6NnlBBzAVxFuAPi1b582vweVJC1cT8ABfBXhBoBfc/YWcYmAA/gqwg0Av5dyTZQevCHWqbkEHMD3EG4A1AhP9m6vId3MPwNHIuAAvoZwA6DGmNw3Tje3Nr+LuPRLwPnLR1kurgiAOxBuANQoi4YkKq5JPafmvvHFDxq6mOfgANUd4QZAjbM89Ua1i6zr1Nz0nUf1wBtfu7giAK5EuAFQI6WNuklNw4KcmrtmV76GLCLgANUV4QZAjbX+iWSnA87a3fnqw2abQLVEuAFQo61/IlltnbxElXXkpFJmrHNtQQCqjHADoMb7uAqXqLbnFKvLs6tUajdcXBUAZxFuAEBVu0SVX3xOV01IU9q3h11cFQBnEG4A4L/WP5Gsdk7eJi5JD7+9Vc8u51k4gLcRbgDgPGmpN6p9FQLOwvU/aMiir1xYEQCzCDcA8CsrUm9UXFR9p+ev3X2MPhzAiwg3AFCB5SN/ox6tnNuqQfpfH87yzEMurApAZRBuAOAi3nggUTe3vqJK7/HIkkw98AaXqQBPItwAwCUsGnKtkts2rtJ7rNnFZSrAkwg3AHAZfxvcVbP7d6rSe3CZCvAcwg0AVELfjlHaOzVF9QKr9m3zkSWZuuvVzzmLA7gR4QYAKslmtSjrmdvUtEFwld5n68EiXTUhTR9u+dFFlQE4H+EGAExaP66nerR2/k6qMiOXfaPEKf9RyTm7C6oCUIZwAwBOeGNIooZ2b1Hl98k9cVatnvpYw/6xkUtVgIsQbgDASRP7tNPc+zq75L1WZuXpqglpejFtByEHqCKLYRg16v+ioqIihYaGqrCwUCEhId4uB4AfKLUbSn55nfYfO+Wy93z0pqs06tbWslktLntPwJeZ+fnNmRsAqCKb1aK1j/dwyWWqMrPX7dVVE9L08sqdnMkBTOLMDQC4UMk5uzr+ZaVOn3Xtt9Y7r2miF+6JV2AAfydFzWTm5zfhBgDcoM/sz5V1qMjl79uwbi299Pt43dj6Ci5ZoUYh3FwC4QaAp/w785BSl2S67f2bhtXWM3fEEXRQIxBuLoFwA8CTSu2Gfj/vC209WOjWzwkOsKpTszANu/EqdW9J2IH/IdxcAuEGgDecLilV1+dW62RJqUc+zyIpwGZRnUCbEpo10Kz+nVUvOMAjnw24A+HmEgg3ALzpmY+2adEXB7z2+Zb/viTJ8OCvJclqkcpu/PLkZ1OT52qyWaWgAKuaN6qrx25t49JLpj53K/irr76q2NhYBQcHKzExURs2bLjk+HfeeUdt2rRRcHCwOnTooLS0NA9VCgBVM6lve+1+7jZdGxvmlc83JNn/+/Lkr+2Szhne+Wxq8lxN5+xScYld24+c0ANvblSbiR9rZdYReZrXw83SpUs1ZswYTZ48WVu2bFHHjh3Vq1cv5eXlVTj+yy+/VP/+/TV06FBt3bpVd955p+68805lZWV5uHIAcE5ggFXLhnXT7uduU1cvhRzAE86WGhr2jy0eDzhevyyVmJiorl27as6cOZIku92umJgYPfroo3riiScuGN+vXz8VFxdr+fLljmPXXXed4uPjNX/+/Mt+HpelAFQ3JefsSpn5qb4/6ronHAPVSWRIsL544uYqXaLymctSJSUl2rx5s5KTkx3HrFarkpOTlZGRUeGcjIyMcuMlqVevXhcdf+bMGRUVFZV7AUB1Ehhg1Sdje2jHM79Vm4h63i4HcLmcop+1Yf9xj32eV8NNfn6+SktLFRERUe54RESEcnJyKpyTk5Njavy0adMUGhrqeMXExLimeABwsdqBNq0cfaN2P3eb7ohv4u1yAJfKO/Gzxz7L6z037jZ+/HgVFhY6XgcPHvR2SQBwSYEBVs28t7P2Tk3Rm4O7qklIoLdLAqqscf1gj32WVx96EB4eLpvNptzc3HLHc3NzFRkZWeGcyMhIU+ODgoIUFBTkmoIBwINsVotubNtYGW1vUck5uxau36v56/aq8GfPPCsHcJXIkGBd26Khxz7Pq2duAgMDlZCQoPT0dMcxu92u9PR0JSUlVTgnKSmp3HhJWr169UXHA4A/CAywavhNLfXN07/V7udu07jfttKVjYL9//Q7/MLTt7fz6FOzvf64yjFjxmjw4MHq0qWLrr32Ws2YMUPFxcUaMmSIJGnQoEGKjo7WtGnTJEmpqam68cYb9fLLL6t3795asmSJNm3apNdff92bywAAjykLOsNvainpl7utFnz+vf6RcUD5J87Kbkic20F1UMtm0ez+nfTbOM/2kHk93PTr109Hjx7VpEmTlJOTo/j4eK1cudLRNJydnS2r9X9/N7n++uv19ttv66mnntKECRPUsmVLffDBB4qLi/PWEgDAqwIDrBrRo5VG9GhV7vjpklI9/dF3St+Wo59OlcqQbzzllpp8tyZ3PqHYDK8/58bTeM4NAAC+x2eecwMAAOBqhBsAAOBXCDcAAMCvEG4AAIBfIdwAAAC/QrgBAAB+hXADAAD8CuEGAAD4FcINAADwK17ffsHTyh7IXFRU5OVKAABAZZX93K7Mxgo1LtycOHFCkhQTE+PlSgAAgFknTpxQaGjoJcfUuL2l7Ha7Dh8+rPr168tice1mXkVFRYqJidHBgwdrxL5VrNe/sV7/xnr9n7+t2TAMnThxQlFRUeU21K5IjTtzY7Va1bRpU7d+RkhIiF/8Qaos1uvfWK9/Y73+z5/WfLkzNmVoKAYAAH6FcAMAAPwK4caFgoKCNHnyZAUFBXm7FI9gvf6N9fo31uv/auKay9S4hmIAAODfOHMDAAD8CuEGAAD4FcINAADwK4QbAADgVwg3LvLqq68qNjZWwcHBSkxM1IYNG7xdklOmTZumrl27qn79+mrcuLHuvPNO7dq1q9yYn3/+WSNGjFCjRo1Ur149/b//9/+Um5tbbkx2drZ69+6tOnXqqHHjxnr88cd17tw5Ty7FtOnTp8tisWjUqFGOY/641kOHDukPf/iDGjVqpNq1a6tDhw7atGmT4+uGYWjSpElq0qSJateureTkZO3Zs6fcexw/flwDBgxQSEiIwsLCNHToUJ08edLTS7ms0tJSTZw4US1atFDt2rV11VVX6dlnny23N40vr/ezzz5T3759FRUVJYvFog8++KDc1121tm+//VY33HCDgoODFRMToxdeeMHdS6vQpdZ79uxZjRs3Th06dFDdunUVFRWlQYMG6fDhw+Xew5fWK13+v/H5hg0bJovFohkzZpQ77mtrdgkDVbZkyRIjMDDQWLRokbFt2zbjwQcfNMLCwozc3Fxvl2Zar169jDfeeMPIysoyMjMzjZSUFKNZs2bGyZMnHWOGDRtmxMTEGOnp6camTZuM6667zrj++usdXz937pwRFxdnJCcnG1u3bjXS0tKM8PBwY/z48d5YUqVs2LDBiI2NNa655hojNTXVcdzf1nr8+HGjefPmxv333298/fXXxr59+4xVq1YZ33//vWPM9OnTjdDQUOODDz4wvvnmG+P22283WrRoYZw+fdox5re//a3RsWNH46uvvjI+//xz4+qrrzb69+/vjSVd0pQpU4xGjRoZy5cvN/bv32+88847Rr169YyZM2c6xvjyetPS0ownn3zSeO+99wxJxvvvv1/u665YW2FhoREREWEMGDDAyMrKMv71r38ZtWvXNl577TVPLdPhUustKCgwkpOTjaVLlxo7d+40MjIyjGuvvdZISEgo9x6+tF7DuPx/4zLvvfee0bFjRyMqKsr461//Wu5rvrZmVyDcuMC1115rjBgxwvHvpaWlRlRUlDFt2jQvVuUaeXl5hiTj008/NQzjl28gtWrVMt555x3HmB07dhiSjIyMDMMwfvmf0Wq1Gjk5OY4x8+bNM0JCQowzZ854dgGVcOLECaNly5bG6tWrjRtvvNERbvxxrePGjTO6d+9+0a/b7XYjMjLSePHFFx3HCgoKjKCgIONf//qXYRiGsX37dkOSsXHjRseYjz/+2LBYLMahQ4fcV7wTevfubTzwwAPljv3ud78zBgwYYBiGf6331z/4XLW2uXPnGg0aNCj353ncuHFG69at3byiS7vUD/oyGzZsMCQZP/zwg2EYvr1ew7j4mn/88UcjOjrayMrKMpo3b14u3Pj6mp3FZakqKikp0ebNm5WcnOw4ZrValZycrIyMDC9W5hqFhYWSpIYNG0qSNm/erLNnz5Zbb5s2bdSsWTPHejMyMtShQwdFREQ4xvTq1UtFRUXatm2bB6uvnBEjRqh3797l1iT551o//PBDdenSRXfffbcaN26sTp06acGCBY6v79+/Xzk5OeXWHBoaqsTExHJrDgsLU5cuXRxjkpOTZbVa9fXXX3tuMZVw/fXXKz09Xbt375YkffPNN1q/fr1uu+02Sf633vO5am0ZGRn6zW9+o8DAQMeYXr16adeuXfrpp588tBrnFBYWymKxKCwsTJJ/rtdut2vgwIF6/PHH1b59+wu+7o9rrgzCTRXl5+ertLS03A83SYqIiFBOTo6XqnINu92uUaNGqVu3boqLi5Mk5eTkKDAw0PHNosz5683Jyanw96Psa9XJkiVLtGXLFk2bNu2Cr/nbWiVp3759mjdvnlq2bKlVq1Zp+PDhGjlypN58801J/6v5Un+ec3Jy1Lhx43JfDwgIUMOGDavdmp944gnde++9atOmjWrVqqVOnTpp1KhRGjBggCT/W+/5XLU2X/szXubnn3/WuHHj1L9/f8emkf643ueff14BAQEaOXJkhV/3xzVXRo3bFRyVN2LECGVlZWn9+vXeLsUtDh48qNTUVK1evVrBwcHeLscj7Ha7unTpoqlTp0qSOnXqpKysLM2fP1+DBw/2cnWut2zZMv3zn//U22+/rfbt2yszM1OjRo1SVFSUX64Xvzh79qzuueceGYahefPmebsct9m8ebNmzpypLVu2yGKxeLucaoUzN1UUHh4um812wR00ubm5ioyM9FJVVffII49o+fLlWrt2rZo2beo4HhkZqZKSEhUUFJQbf/56IyMjK/z9KPtadbF582bl5eWpc+fOCggIUEBAgD799FPNmjVLAQEBioiI8Ju1lmnSpInatWtX7ljbtm2VnZ0t6X81X+rPc2RkpPLy8sp9/dy5czp+/Hi1W/Pjjz/uOHvToUMHDRw4UKNHj3acqfO39Z7PVWvztT/jZcHmhx9+0OrVqx1nbST/W+/nn3+uvLw8NWvWzPE97IcfftDYsWMVGxsryf/WXFmEmyoKDAxUQkKC0tPTHcfsdrvS09OVlJTkxcqcYxiGHnnkEb3//vtas2aNWrRoUe7rCQkJqlWrVrn17tq1S9nZ2Y71JiUl6bvvviv3P1TZN5lf/2D1pp49e+q7775TZmam49WlSxcNGDDA8Wt/WWuZbt26XXBr/+7du9W8eXNJUosWLRQZGVluzUVFRfr666/LrbmgoECbN292jFmzZo3sdrsSExM9sIrKO3XqlKzW8t/mbDab7Ha7JP9b7/lctbakpCR99tlnOnv2rGPM6tWr1bp1azVo0MBDq6mcsmCzZ88effLJJ2rUqFG5r/vbegcOHKhvv/223PewqKgoPf7441q1apUk/1tzpXm7o9kfLFmyxAgKCjIWL15sbN++3XjooYeMsLCwcnfQ+Irhw4cboaGhxrp164wjR444XqdOnXKMGTZsmNGsWTNjzZo1xqZNm4ykpCQjKSnJ8fWy26NvvfVWIzMz01i5cqVxxRVXVNvbo893/t1ShuF/a92wYYMREBBgTJkyxdizZ4/xz3/+06hTp47xj3/8wzFm+vTpRlhYmPHvf//b+Pbbb4077rijwtuHO3XqZHz99dfG+vXrjZYtW1aLW6N/bfDgwUZ0dLTjVvD33nvPCA8PN/785z87xvjyek+cOGFs3brV2Lp1qyHJeOWVV4ytW7c67g5yxdoKCgqMiIgIY+DAgUZWVpaxZMkSo06dOl65TfhS6y0pKTFuv/12o2nTpkZmZma571/n3wXkS+s1jMv/N/61X98tZRi+t2ZXINy4yOzZs41mzZoZgYGBxrXXXmt89dVX3i7JKZIqfL3xxhuOMadPnzYefvhho0GDBkadOnWMu+66yzhy5Ei59zlw4IBx2223GbVr1zbCw8ONsWPHGmfPnvXwasz7dbjxx7V+9NFHRlxcnBEUFGS0adPGeP3118t93W63GxMnTjQiIiKMoKAgo2fPnsauXbvKjTl27JjRv39/o169ekZISIgxZMgQ48SJE55cRqUUFRUZqampRrNmzYzg4GDjyiuvNJ588slyP+x8eb1r166t8P/XwYMHG4bhurV98803Rvfu3Y2goCAjOjramD59uqeWWM6l1rt///6Lfv9au3at4z18ab2Gcfn/xr9WUbjxtTW7gsUwzntUJwAAgI+j5wYAAPgVwg0AAPArhBsAAOBXCDcAAMCvEG4AAIBfIdwAAAC/QrgBAAB+hXADAAD8CuEGQI0TGxurGTNmeLsMAG5CuAHgVvfff7/uvPNOSdJNN92kUaNGeeyzFy9erLCwsAuOb9y4UQ899JDH6gDgWQHeLgAAzCopKVFgYKDT86+44goXVgOguuHMDQCPuP/++/Xpp59q5syZslgsslgsOnDggCQpKytLt912m+rVq6eIiAgNHDhQ+fn5jrk33XSTHnnkEY0aNUrh4eHq1auXJOmVV15Rhw4dVLduXcXExOjhhx/WyZMnJUnr1q3TkCFDVFhY6Pi8p59+WtKFl6Wys7N1xx13qF69egoJCdE999yj3Nxcx9effvppxcfH6+9//7tiY2MVGhqqe++9VydOnHDvbxoApxBuAHjEzJkzlZSUpAcffFBHjhzRkSNHFBMTo4KCAt18883q1KmTNm3apJUrVyo3N1f33HNPuflvvvmmAgMD9cUXX2j+/PmSJKvVqlmzZmnbtm168803tWbNGv35z3+WJF1//fWaMWOGQkJCHJ/32GOPXVCX3W7XHXfcoePHj+vTTz/V6tWrtW/fPvXr16/cuL179+qDDz7Q8uXLtXz5cn366aeaPn26m363AFQFl6UAeERoaKgCAwNVp04dRUZGOo7PmTNHnTp10tSpUx3HFi1apJiYGO3evVutWrWSJLVs2VIvvPBCufc8v38nNjZWzz33nIYNG6a5c+cqMDBQoaGhslgs5T7v19LT0/Xdd99p//79iomJkSS99dZbat++vTZu3KiuXbtK+iUELV68WPXr15ckDRw4UOnp6ZoyZUrVfmMAuBxnbgB41TfffKO1a9eqXr16jlebNm0k/XK2pExCQsIFcz/55BP17NlT0dHRql+/vgYOHKhjx47p1KlTlf78HTt2KCYmxhFsJKldu3YKCwvTjh07HMdiY2MdwUaSmjRpory8PFNrBeAZnLkB4FUnT55U37599fzzz1/wtSZNmjh+Xbdu3XJfO3DggPr06aPhw4drypQpatiwodavX6+hQ4eqpKREderUcWmdtWrVKvfvFotFdrvdpZ8BwDUINwA8JjAwUKWlpeWOde7cWf/3f/+n2NhYBQRU/lvS5s2bZbfb9fLLL8tq/eUk9LJlyy77eb/Wtm1bHTx4UAcPHnScvdm+fbsKCgrUrl27StcDoPrgshQAj4mNjdXXX3+tAwcOKD8/X3a7XSNGjNDx48fVv39/bdy4UXv37tWqVas0ZMiQSwaTq6++WmfPntXs2bO1b98+/f3vf3c0Gp//eSdPnlR6erry8/MrvFyVnJysDh06aMCAAdqyZYs2bNigQYMG6cYbb1SXLl1c/nsAwP0INwA85rHHHpPNZlO7du10xRVXKDs7W1FRUfriiy9UWlqqW2+9VR06dNCoUaMUFhbmOCNTkY4dO+qVV17R888/r7i4OP3zn//UtGnTyo25/vrrNWzYMPXr109XXHHFBQ3J0i+Xl/7973+rQYMG+s1vfqPk5GRdeeWVWrp0qcvXD8AzLIZhGN4uAgAAwFU4cwMAAPwK4QYAAPgVwg0AAPArhBsAAOBXCDcAAMCvEG4AAIBfIdwAAAC/QrgBAAB+hXADAAD8CuEGAAD4FcINAADwK/8fGZhRq3HwsCYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class CosineAnnealingSchedule():\n",
    "    def __init__(self, min_lr, max_lr, cycle_length):\n",
    "        self.min_lr = min_lr\n",
    "        self.max_lr = max_lr\n",
    "        self.cycle_length = cycle_length\n",
    "\n",
    "    def __call__(self, iteration):\n",
    "        if iteration <= self.cycle_length:\n",
    "            unit_cycle = (1 + math.cos(iteration * math.pi / self.cycle_length)) / 2\n",
    "            adjusted_cycle = (unit_cycle * (self.max_lr - self.min_lr)) + self.min_lr\n",
    "            return adjusted_cycle\n",
    "        else:\n",
    "            return self.min_lr\n",
    "\n",
    "\n",
    "schedule = CosineAnnealingSchedule(min_lr=0, max_lr=1, cycle_length=1000)\n",
    "plot_schedule(schedule)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d2b0360",
   "metadata": {},
   "source": [
    "![lr cosine](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/lr_schedules/cosine.png) <!--notebook-skip-line-->\n",
    "\n",
    "\n",
    "## Using Schedules\n",
    "\n",
    "While training a simple handwritten digit classifier on the MNIST dataset, we take a look at how to use a learning rate schedule during training. Our demonstration model is a basic convolutional neural network. We start by preparing our `DataLoader` and defining the network.\n",
    "\n",
    "As discussed above, the schedule should return a learning rate given an (1-based) iteration index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8a983108",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[04:49:42] /work/mxnet/src/storage/storage.cc:202: Using Pooled (Naive) StorageManager for CPU\n"
     ]
    }
   ],
   "source": [
    "# Use GPU if one exists, else use CPU\n",
    "device = mx.gpu() if mx.device.num_gpus() else mx.cpu()\n",
    "\n",
    "# MNIST images are 28x28. Total pixels in input layer is 28x28 = 784\n",
    "num_inputs = 784\n",
    "# Clasify the images into one of the 10 digits\n",
    "num_outputs = 10\n",
    "# 64 images in a batch\n",
    "batch_size = 64\n",
    "\n",
    "# Load the training data\n",
    "train_dataset = mx.gluon.data.vision.MNIST(train=True).transform_first(transforms.ToTensor())\n",
    "train_dataloader = mx.gluon.data.DataLoader(train_dataset, batch_size, shuffle=True, num_workers=5)\n",
    "\n",
    "# Build a simple convolutional network\n",
    "def build_cnn():\n",
    "    net = nn.HybridSequential()\n",
    "    # First convolution\n",
    "    net.add(nn.Conv2D(channels=10, kernel_size=5, activation='relu'))\n",
    "    net.add(nn.MaxPool2D(pool_size=2, strides=2))\n",
    "    # Second convolution\n",
    "    net.add(nn.Conv2D(channels=20, kernel_size=5, activation='relu'))\n",
    "    net.add(nn.MaxPool2D(pool_size=2, strides=2))\n",
    "    # Flatten the output before the fully connected layers\n",
    "    net.add(nn.Flatten())\n",
    "    # First fully connected layers with 512 neurons\n",
    "    net.add(nn.Dense(512, activation=\"relu\"))\n",
    "    # Second fully connected layer with as many neurons as the number of classes\n",
    "    net.add(nn.Dense(num_outputs))\n",
    "    return net\n",
    "\n",
    "net = build_cnn()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a4bf9d9",
   "metadata": {},
   "source": [
    "We then initialize our network (technically deferred until we pass the first batch) and define the loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "08a24595",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[04:49:45] /work/mxnet/src/storage/storage.cc:202: Using Pooled (Naive) StorageManager for GPU\n"
     ]
    }
   ],
   "source": [
    "# Initialize the parameters with Xavier initializer\n",
    "net.initialize(mx.init.Xavier(), device=device)\n",
    "# Use cross entropy loss\n",
    "softmax_cross_entropy = mx.gluon.loss.SoftmaxCrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "203be59b",
   "metadata": {},
   "source": [
    "We're now ready to create our schedule, and in this example we opt for a stepwise decay schedule using `MultiFactorScheduler`. Since we're only training a demonstration model for a limited number of epochs (10 in total) we will exaggerate the schedule and drop the learning rate by 90% after the 4th, 7th and 9th epochs. We call these steps, and the drop occurs _after_ the step index. Schedules are defined for iterations (i.e. training batches), so we must represent our steps in iterations too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "cdbaccbc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Learning rate drops after iterations: [3752, 6566, 8442]\n"
     ]
    }
   ],
   "source": [
    "steps_epochs = [4, 7, 9]\n",
    "# assuming we keep partial batches, see `last_batch` parameter of DataLoader\n",
    "iterations_per_epoch = math.ceil(len(train_dataset) / batch_size)\n",
    "# iterations just before starts of epochs (iterations are 1-indexed)\n",
    "steps_iterations = [s*iterations_per_epoch for s in steps_epochs]\n",
    "print(\"Learning rate drops after iterations: {}\".format(steps_iterations))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2cf8e4b",
   "metadata": {},
   "source": [
    "```\n",
    "Learning rate drops after iterations: [3752, 6566, 8442]\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "615b1269",
   "metadata": {},
   "outputs": [],
   "source": [
    "schedule = mx.lr_scheduler.MultiFactorScheduler(step=steps_iterations, factor=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90aabb4e",
   "metadata": {},
   "source": [
    "**We create our `Optimizer` and pass the schedule via the `lr_scheduler` parameter.** In this example we're using Stochastic Gradient Descent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bc76af37",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "learning rate from ``lr_scheduler`` has been overwritten by ``learning_rate`` in optimizer.\n"
     ]
    }
   ],
   "source": [
    "sgd_optimizer = mx.optimizer.SGD(learning_rate=0.03, lr_scheduler=schedule)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb517046",
   "metadata": {},
   "source": [
    "And we use this optimizer (with schedule) in our `Trainer` and train for 10 epochs. Alternatively, we could have set the `optimizer` to the string `sgd`, and pass a dictionary of the optimizer parameters directly to the trainer using `optimizer_params`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1d139094",
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer = mx.gluon.Trainer(params=net.collect_params(), optimizer=sgd_optimizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "324929da",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[04:49:47] /work/mxnet/src/operator/cudnn_ops.cc:421: Auto-tuning cuDNN op, set MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[04:49:47] /work/mxnet/src/operator/cudnn_ops.cc:421: Auto-tuning cuDNN op, set MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1; Batch 1; Loss 2.300970; LR 0.030000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 2; Batch 1; Loss 0.101566; LR 0.030000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 3; Batch 1; Loss 0.061599; LR 0.030000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 4; Batch 1; Loss 0.030203; LR 0.030000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 5; Batch 1; Loss 0.063257; LR 0.003000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 6; Batch 1; Loss 0.035521; LR 0.003000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 7; Batch 1; Loss 0.039717; LR 0.003000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 8; Batch 1; Loss 0.021303; LR 0.000300\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 9; Batch 1; Loss 0.028846; LR 0.000300\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 10; Batch 1; Loss 0.012560; LR 0.000030\n"
     ]
    }
   ],
   "source": [
    "num_epochs = 10\n",
    "# epoch and batch counts starting at 1\n",
    "for epoch in range(1, num_epochs+1):\n",
    "    # Iterate through the images and labels in the training data\n",
    "    for batch_num, (data, label) in enumerate(train_dataloader, start=1):\n",
    "        # get the images and labels\n",
    "        data = data.to_device(device)\n",
    "        label = label.to_device(device)\n",
    "        # Ask autograd to record the forward pass\n",
    "        with mx.autograd.record():\n",
    "            # Run the forward pass\n",
    "            output = net(data)\n",
    "            # Compute the loss\n",
    "            loss = softmax_cross_entropy(output, label)\n",
    "        # Compute gradients\n",
    "        loss.backward()\n",
    "        # Update parameters\n",
    "        trainer.step(data.shape[0])\n",
    "\n",
    "        # Show loss and learning rate after first iteration of epoch\n",
    "        if batch_num == 1:\n",
    "            curr_loss = mx.np.mean(loss).item()\n",
    "            curr_lr = trainer.learning_rate\n",
    "            print(\"Epoch: %d; Batch %d; Loss %f; LR %f\" % (epoch, batch_num, curr_loss, curr_lr))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa874417",
   "metadata": {},
   "source": [
    "Epoch: 1; Batch 1; Loss 2.304071; LR 0.030000 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 2; Batch 1; Loss 0.059640; LR 0.030000 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 3; Batch 1; Loss 0.072601; LR 0.030000 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 4; Batch 1; Loss 0.042228; LR 0.030000 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 5; Batch 1; Loss 0.025745; LR 0.003000 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 6; Batch 1; Loss 0.027391; LR 0.003000 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 7; Batch 1; Loss 0.048237; LR 0.003000 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 8; Batch 1; Loss 0.024213; LR 0.000300 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 9; Batch 1; Loss 0.008892; LR 0.000300 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 10; Batch 1; Loss 0.006875; LR 0.000030 <!--notebook-skip-line-->\n",
    "\n",
    "\n",
    "We see that the learning rate starts at 0.03, and falls to 0.00003 by the end of training as per the schedule we defined.\n",
    "\n",
    "### Manually setting the learning rate: Gluon API only\n",
    "\n",
    "When using the method above you don't need to manually keep track of iteration count and set the learning rate, so this is the recommended approach for most cases. Sometimes you might want more fine-grained control over setting the learning rate though, so Gluon's `Trainer` provides the `set_learning_rate` method for this.\n",
    "\n",
    "We replicate the example above, but now keep track of the `iteration_idx`, call the schedule and set the learning rate appropriately using `set_learning_rate`. We also use `schedule.base_lr` to set the initial learning rate for the schedule since we are calling the schedule directly and not using it as part of the `Optimizer`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "2657b2ae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1; Batch 1; Loss 2.317546; LR 0.030000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 2; Batch 1; Loss 0.145374; LR 0.030000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 3; Batch 1; Loss 0.016087; LR 0.030000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 4; Batch 1; Loss 0.047587; LR 0.030000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 5; Batch 1; Loss 0.035325; LR 0.003000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 6; Batch 1; Loss 0.107048; LR 0.003000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 7; Batch 1; Loss 0.004735; LR 0.003000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 8; Batch 1; Loss 0.014305; LR 0.000300\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 9; Batch 1; Loss 0.012392; LR 0.000300\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 10; Batch 1; Loss 0.005573; LR 0.000030\n"
     ]
    }
   ],
   "source": [
    "net = build_cnn()\n",
    "net.initialize(mx.init.Xavier(), device=device)\n",
    "\n",
    "schedule = mx.lr_scheduler.MultiFactorScheduler(step=steps_iterations, factor=0.1)\n",
    "schedule.base_lr = 0.03\n",
    "sgd_optimizer = mx.optimizer.SGD()\n",
    "trainer = mx.gluon.Trainer(params=net.collect_params(), optimizer=sgd_optimizer)\n",
    "\n",
    "iteration_idx = 1\n",
    "num_epochs = 10\n",
    "# epoch and batch counts starting at 1\n",
    "for epoch in range(1, num_epochs + 1):\n",
    "    # Iterate through the images and labels in the training data\n",
    "    for batch_num, (data, label) in enumerate(train_dataloader, start=1):\n",
    "        # get the images and labels\n",
    "        data = data.to_device(device)\n",
    "        label = label.to_device(device)\n",
    "        # Ask autograd to record the forward pass\n",
    "        with mx.autograd.record():\n",
    "            # Run the forward pass\n",
    "            output = net(data)\n",
    "            # Compute the loss\n",
    "            loss = softmax_cross_entropy(output, label)\n",
    "        # Compute gradients\n",
    "        loss.backward()\n",
    "        # Update the learning rate\n",
    "        lr = schedule(iteration_idx)\n",
    "        trainer.set_learning_rate(lr)\n",
    "        # Update parameters\n",
    "        trainer.step(data.shape[0])\n",
    "        # Show loss and learning rate after first iteration of epoch\n",
    "        if batch_num == 1:\n",
    "            curr_loss = mx.np.mean(loss).item()\n",
    "            curr_lr = trainer.learning_rate\n",
    "            print(\"Epoch: %d; Batch %d; Loss %f; LR %f\" % (epoch, batch_num, curr_loss, curr_lr))\n",
    "        iteration_idx += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b5000a2",
   "metadata": {},
   "source": [
    "Epoch: 1; Batch 1; Loss 2.334119; LR 0.030000 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 2; Batch 1; Loss 0.178930; LR 0.030000 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 3; Batch 1; Loss 0.142640; LR 0.030000 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 4; Batch 1; Loss 0.041116; LR 0.030000 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 5; Batch 1; Loss 0.051049; LR 0.003000 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 6; Batch 1; Loss 0.027170; LR 0.003000 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 7; Batch 1; Loss 0.083776; LR 0.003000 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 8; Batch 1; Loss 0.082553; LR 0.000300 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 9; Batch 1; Loss 0.027984; LR 0.000300 <!--notebook-skip-line-->\n",
    "\n",
    "Epoch: 10; Batch 1; Loss 0.030896; LR 0.000030 <!--notebook-skip-line-->\n",
    "\n",
    "\n",
    "Once again, we see the learning rate start at 0.03, and fall to 0.00003 by the end of training as per the schedule we defined.\n",
    "\n",
    "## Advanced Schedules\n",
    "\n",
    "We have a related tutorial on Advanced Learning Rate Schedules that shows reference implementations of schedules that give state-of-the-art results. We look at cyclical schedules applied to a variety of cycle shapes, and many other techniques such as warm-up and cool-down.\n",
    "\n",
    "<!-- INSERT SOURCE DOWNLOAD BUTTONS -->"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}