| <!DOCTYPE html> |
| |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"/> |
| <meta content="IE=edge" http-equiv="X-UA-Compatible"/> |
| <meta content="width=device-width, initial-scale=1" name="viewport"/> |
| <meta content="mxnet.optimizer" property="og:title"> |
| <meta content="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/og-logo.png" property="og:image"> |
| <meta content="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/og-logo.png" property="og:image:secure_url"> |
| <meta content="mxnet.optimizer" property="og:description"/> |
| <title>mxnet.optimizer — mxnet documentation</title> |
| <link crossorigin="anonymous" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css" integrity="sha384-1q8mTJOASx8j1Au+a5WDVnPi2lkFfwwEAa8hDDdjZlpLegxhjVME1fgjWPGmkzs7" rel="stylesheet"/> |
| <link href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css" rel="stylesheet"/> |
| <link href="../../_static/basic.css" rel="stylesheet" type="text/css"> |
| <link href="../../_static/pygments.css" rel="stylesheet" type="text/css"> |
| <link href="../../_static/mxnet.css" rel="stylesheet" type="text/css"/> |
| <script type="text/javascript"> |
| var DOCUMENTATION_OPTIONS = { |
| URL_ROOT: '../../', |
| VERSION: '', |
| COLLAPSE_INDEX: false, |
| FILE_SUFFIX: '.html', |
| HAS_SOURCE: true, |
| SOURCELINK_SUFFIX: '.txt' |
| }; |
| </script> |
| <script src="https://code.jquery.com/jquery-1.11.1.min.js" type="text/javascript"></script> |
| <script src="../../_static/underscore.js" type="text/javascript"></script> |
| <script src="../../_static/searchtools_custom.js" type="text/javascript"></script> |
| <script src="../../_static/doctools.js" type="text/javascript"></script> |
| <script src="../../_static/selectlang.js" type="text/javascript"></script> |
| <script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script> |
| <script type="text/javascript"> jQuery(function() { Search.loadIndex("/searchindex.js"); Search.init();}); </script> |
| <script> |
| (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ |
| (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new |
| Date();a=s.createElement(o), |
| m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) |
| })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); |
| |
| ga('create', 'UA-96378503-1', 'auto'); |
| ga('send', 'pageview'); |
| |
| </script> |
| <!-- --> |
| <!-- <script type="text/javascript" src="../../_static/jquery.js"></script> --> |
| <!-- --> |
| <!-- <script type="text/javascript" src="../../_static/underscore.js"></script> --> |
| <!-- --> |
| <!-- <script type="text/javascript" src="../../_static/doctools.js"></script> --> |
| <!-- --> |
| <!-- <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script> --> |
| <!-- --> |
| <link href="../../genindex.html" rel="index" title="Index"> |
| <link href="../../search.html" rel="search" title="Search"/> |
| <link href="../index.html" rel="up" title="Module code"/> |
| <link href="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet-icon.png" rel="icon" type="image/png"/> |
| </link></link></link></meta></meta></meta></head> |
| <body background="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet-background-compressed.jpeg" role="document"> |
| <div class="content-block"><div class="navbar navbar-fixed-top"> |
| <div class="container" id="navContainer"> |
| <div class="innder" id="header-inner"> |
| <h1 id="logo-wrap"> |
| <a href="../../" id="logo"><img src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/mxnet_logo.png"/></a> |
| </h1> |
| <nav class="nav-bar" id="main-nav"> |
| <a class="main-nav-link" href="../../install/index.html">Install</a> |
| <span id="dropdown-menu-position-anchor"> |
| <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Gluon <span class="caret"></span></a> |
| <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu"> |
| <li><a class="main-nav-link" href="../../gluon/index.html">About</a></li> |
| <li><a class="main-nav-link" href="http://gluon.mxnet.io">Tutorials</a></li> |
| </ul> |
| </span> |
| <span id="dropdown-menu-position-anchor"> |
| <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">API <span class="caret"></span></a> |
| <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu"> |
| <li><a class="main-nav-link" href="../../api/python/index.html">Python</a></li> |
| <li><a class="main-nav-link" href="../../api/c++/index.html">C++</a></li> |
| <li><a class="main-nav-link" href="../../api/clojure/index.html">Clojure</a></li> |
| <li><a class="main-nav-link" href="../../api/julia/index.html">Julia</a></li> |
| <li><a class="main-nav-link" href="../../api/perl/index.html">Perl</a></li> |
| <li><a class="main-nav-link" href="../../api/r/index.html">R</a></li> |
| <li><a class="main-nav-link" href="../../api/scala/index.html">Scala</a></li> |
| </ul> |
| </span> |
| <span id="dropdown-menu-position-anchor-docs"> |
| <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Docs <span class="caret"></span></a> |
| <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu-docs"> |
| <li><a class="main-nav-link" href="../../faq/index.html">FAQ</a></li> |
| <li><a class="main-nav-link" href="../../tutorials/index.html">Tutorials</a> |
| <li><a class="main-nav-link" href="https://github.com/apache/incubator-mxnet/tree/0.11.0/example">Examples</a></li> |
| <li><a class="main-nav-link" href="../../architecture/index.html">Architecture</a></li> |
| <li><a class="main-nav-link" href="../../api/python/gluon/model_zoo.html">Model Zoo</a></li> |
| <li><a class="main-nav-link" href="../../api/python/contrib/onnx.html">ONNX</a></li> |
| </li></ul> |
| </span> |
| <span id="dropdown-menu-position-anchor-community"> |
| <a aria-expanded="true" aria-haspopup="true" class="main-nav-link dropdown-toggle" data-toggle="dropdown" href="#" role="button">Community <span class="caret"></span></a> |
| <ul class="dropdown-menu navbar-menu" id="package-dropdown-menu-community"> |
| <li><a class="main-nav-link" href="http://discuss.mxnet.io">Forum</a></li> |
| <li><a class="main-nav-link" href="https://github.com/apache/incubator-mxnet">Github</a></li> |
| <li><a class="main-nav-link" href="../../community/contribute.html">Contribute</a></li> |
| <li><a class="main-nav-link" href="../../community/ecosystem.html">Ecosystem</a></li> |
| <li><a class="main-nav-link" href="../../community/powered_by.html">Powered By</a></li> |
| </ul> |
| </span> |
| <span id="dropdown-menu-position-anchor-version" style="position: relative"><a href="#" class="main-nav-link dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="true">Versions(0.11.0)<span class="caret"></span></a><ul id="package-dropdown-menu" class="dropdown-menu"><li><a class="main-nav-link" href=http://mxnet.incubator.apache.org/>master</a></li><li><a class="main-nav-link" href=http://mxnet.incubator.apache.org/versions/1.2.1/index.html>1.2.1</a></li><li><a class="main-nav-link" href=http://mxnet.incubator.apache.org/versions/1.1.0/index.html>1.1.0</a></li><li><a class="main-nav-link" href=http://mxnet.incubator.apache.org/versions/1.0.0/index.html>1.0.0</a></li><li><a class="main-nav-link" href=http://mxnet.incubator.apache.org/versions/0.12.1/index.html>0.12.1</a></li><li><a class="main-nav-link" href=http://mxnet.incubator.apache.org/versions/0.11.0/index.html>0.11.0</a></li></ul></span></nav> |
| <script> function getRootPath(){ return "../../" } </script> |
| <div class="burgerIcon dropdown"> |
| <a class="dropdown-toggle" data-toggle="dropdown" href="#" role="button">☰</a> |
| <ul class="dropdown-menu" id="burgerMenu"> |
| <li><a href="../../install/index.html">Install</a></li> |
| <li><a class="main-nav-link" href="../../tutorials/index.html">Tutorials</a></li> |
| <li class="dropdown-submenu dropdown"> |
| <a aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" role="button" tabindex="-1">Community</a> |
| <ul class="dropdown-menu"> |
| <li><a href="http://discuss.mxnet.io" tabindex="-1">Forum</a></li> |
| <li><a href="https://github.com/apache/incubator-mxnet" tabindex="-1">Github</a></li> |
| <li><a href="../../community/contribute.html" tabindex="-1">Contribute</a></li> |
| <li><a href="../../community/ecosystem.html" tabindex="-1">Ecosystem</a></li> |
| <li><a href="../../community/powered_by.html" tabindex="-1">Powered By</a></li> |
| </ul> |
| </li> |
| <li class="dropdown-submenu"> |
| <a aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" role="button" tabindex="-1">API</a> |
| <ul class="dropdown-menu"> |
| <li><a href="../../api/python/index.html" tabindex="-1">Python</a> |
| </li> |
| <li><a href="../../api/c++/index.html" tabindex="-1">C++</a> |
| </li> |
| <li><a href="../../api/clojure/index.html" tabindex="-1">Clojure</a> |
| </li> |
| <li><a href="../../api/julia/index.html" tabindex="-1">Julia</a> |
| </li> |
| <li><a href="../../api/perl/index.html" tabindex="-1">Perl</a> |
| </li> |
| <li><a href="../../api/r/index.html" tabindex="-1">R</a> |
| </li> |
| <li><a href="../../api/scala/index.html" tabindex="-1">Scala</a> |
| </li> |
| </ul> |
| </li> |
| <li class="dropdown-submenu"> |
| <a aria-expanded="true" aria-haspopup="true" class="dropdown-toggle burger-link" data-toggle="dropdown" href="#" tabindex="-1">Docs</a> |
| <ul class="dropdown-menu"> |
| <li><a href="../../tutorials/index.html" tabindex="-1">Tutorials</a></li> |
| <li><a href="../../faq/index.html" tabindex="-1">FAQ</a></li> |
| <li><a href="../../architecture/index.html" tabindex="-1">Architecture</a></li> |
| <li><a href="https://github.com/apache/incubator-mxnet/tree/0.11.0/example" tabindex="-1">Examples</a></li> |
| <li><a href="../../api/python/gluon/model_zoo.html" tabindex="-1">Gluon Model Zoo</a></li> |
| </ul> |
| </li> |
| <li><a class="main-nav-link" href="https://github.com/dmlc/mxnet">Github</a></li> |
| <li id="dropdown-menu-position-anchor-version-mobile" class="dropdown-submenu" style="position: relative"><a href="#" tabindex="-1">Versions(0.11.0)</a><ul class="dropdown-menu"><li><a tabindex="-1" href=http://mxnet.incubator.apache.org/>master</a></li><li><a tabindex="-1" href=http://mxnet.incubator.apache.org/versions/1.2.1/index.html>1.2.1</a></li><li><a tabindex="-1" href=http://mxnet.incubator.apache.org/versions/1.1.0/index.html>1.1.0</a></li><li><a tabindex="-1" href=http://mxnet.incubator.apache.org/versions/1.0.0/index.html>1.0.0</a></li><li><a tabindex="-1" href=http://mxnet.incubator.apache.org/versions/0.12.1/index.html>0.12.1</a></li><li><a tabindex="-1" href=http://mxnet.incubator.apache.org/versions/0.11.0/index.html>0.11.0</a></li></ul></li></ul> |
| </div> |
| <div class="plusIcon dropdown"> |
| <a class="dropdown-toggle" data-toggle="dropdown" href="#" role="button"><span aria-hidden="true" class="glyphicon glyphicon-plus"></span></a> |
| <ul class="dropdown-menu dropdown-menu-right" id="plusMenu"></ul> |
| </div> |
| <div id="search-input-wrap"> |
| <form action="../../search.html" autocomplete="off" class="" method="get" role="search"> |
| <div class="form-group inner-addon left-addon"> |
| <i class="glyphicon glyphicon-search"></i> |
| <input class="form-control" name="q" placeholder="Search" type="text"/> |
| </div> |
| <input name="check_keywords" type="hidden" value="yes"> |
| <input name="area" type="hidden" value="default"/> |
| </input></form> |
| <div id="search-preview"></div> |
| </div> |
| <div id="searchIcon"> |
| <span aria-hidden="true" class="glyphicon glyphicon-search"></span> |
| </div> |
| <!-- <div id="lang-select-wrap"> --> |
| <!-- <label id="lang-select-label"> --> |
| <!-- <\!-- <i class="fa fa-globe"></i> -\-> --> |
| <!-- <span></span> --> |
| <!-- </label> --> |
| <!-- <select id="lang-select"> --> |
| <!-- <option value="en">Eng</option> --> |
| <!-- <option value="zh">中文</option> --> |
| <!-- </select> --> |
| <!-- </div> --> |
| <!-- <a id="mobile-nav-toggle"> |
| <span class="mobile-nav-toggle-bar"></span> |
| <span class="mobile-nav-toggle-bar"></span> |
| <span class="mobile-nav-toggle-bar"></span> |
| </a> --> |
| </div> |
| </div> |
| </div> |
| <script type="text/javascript"> |
| $('body').css('background', 'white'); |
| </script> |
| <div class="container"> |
| <div class="row"> |
| <div aria-label="main navigation" class="sphinxsidebar leftsidebar" role="navigation"> |
| <div class="sphinxsidebarwrapper"> |
| <ul> |
| <li class="toctree-l1"><a class="reference internal" href="../../api/python/index.html">Python Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../api/r/index.html">R Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../api/julia/index.html">Julia Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../api/c++/index.html">C++ Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../api/scala/index.html">Scala Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../api/perl/index.html">Perl Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../faq/index.html">HowTo Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../architecture/index.html">System Documents</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../tutorials/index.html">Tutorials</a></li> |
| <li class="toctree-l1"><a class="reference internal" href="../../community/contribute.html">Community</a></li> |
| </ul> |
| </div> |
| </div> |
| <div class="content"> |
| <div class="page-tracker"></div> |
| <h1>Source code for mxnet.optimizer</h1><div class="highlight"><pre> |
| <span></span><span class="c1"># coding: utf-8</span> |
| <span class="c1"># Licensed to the Apache Software Foundation (ASF) under one</span> |
| <span class="c1"># or more contributor license agreements. See the NOTICE file</span> |
| <span class="c1"># distributed with this work for additional information</span> |
| <span class="c1"># regarding copyright ownership. The ASF licenses this file</span> |
| <span class="c1"># to you under the Apache License, Version 2.0 (the</span> |
| <span class="c1"># "License"); you may not use this file except in compliance</span> |
| <span class="c1"># with the License. You may obtain a copy of the License at</span> |
| <span class="c1">#</span> |
| <span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span> |
| <span class="c1">#</span> |
| <span class="c1"># Unless required by applicable law or agreed to in writing,</span> |
| <span class="c1"># software distributed under the License is distributed on an</span> |
| <span class="c1"># "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY</span> |
| <span class="c1"># KIND, either express or implied. See the License for the</span> |
| <span class="c1"># specific language governing permissions and limitations</span> |
| <span class="c1"># under the License.</span> |
| |
| <span class="c1"># pylint: disable=too-many-lines</span> |
| <span class="sd">"""Weight updating functions."""</span> |
| <span class="kn">import</span> <span class="nn">logging</span> |
| <span class="kn">import</span> <span class="nn">math</span> |
| <span class="kn">import</span> <span class="nn">pickle</span> |
| <span class="kn">import</span> <span class="nn">warnings</span> |
| <span class="kn">import</span> <span class="nn">numpy</span> |
| <span class="kn">from</span> <span class="nn">.base</span> <span class="k">import</span> <span class="n">py_str</span> |
| <span class="kn">from</span> <span class="nn">.ndarray</span> <span class="k">import</span> <span class="p">(</span><span class="n">NDArray</span><span class="p">,</span> <span class="n">zeros</span><span class="p">,</span> <span class="n">clip</span><span class="p">,</span> <span class="n">sqrt</span><span class="p">,</span> <span class="n">cast</span><span class="p">,</span> <span class="n">maximum</span><span class="p">,</span> <span class="nb">abs</span> <span class="k">as</span> <span class="n">NDabs</span><span class="p">,</span> <span class="n">array</span><span class="p">,</span> <span class="n">multiply</span><span class="p">)</span> |
| <span class="kn">from</span> <span class="nn">.ndarray</span> <span class="k">import</span> <span class="p">(</span><span class="n">sgd_update</span><span class="p">,</span> <span class="n">sgd_mom_update</span><span class="p">,</span> <span class="n">adam_update</span><span class="p">,</span> <span class="n">rmsprop_update</span><span class="p">,</span> <span class="n">rmspropalex_update</span><span class="p">,</span> |
| <span class="n">mp_sgd_update</span><span class="p">,</span> <span class="n">mp_sgd_mom_update</span><span class="p">,</span> <span class="n">square</span><span class="p">,</span> <span class="n">ftrl_update</span><span class="p">,</span> <span class="n">ftml_update</span><span class="p">,</span> |
| <span class="n">signsgd_update</span><span class="p">,</span> <span class="n">signum_update</span><span class="p">)</span> |
| <span class="kn">from</span> <span class="nn">.ndarray</span> <span class="k">import</span> <span class="n">sparse</span> |
| <span class="kn">from</span> <span class="nn">.random</span> <span class="k">import</span> <span class="n">normal</span> |
| |
| |
| <div class="viewcode-block" id="Optimizer"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Optimizer">[docs]</a><span class="k">class</span> <span class="nc">Optimizer</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> |
| <span class="sd">"""The base class inherited by all optimizers.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> rescale_grad : float, optional</span> |
| <span class="sd"> Multiply the gradient with `rescale_grad` before updating. Often</span> |
| <span class="sd"> choose to be ``1.0/batch_size``.</span> |
| |
| <span class="sd"> param_idx2name : dict from int to string, optional</span> |
| <span class="sd"> A dictionary that maps int index to string name.</span> |
| |
| <span class="sd"> clip_gradient : float, optional</span> |
| <span class="sd"> Clip the gradient by projecting onto the box ``[-clip_gradient, clip_gradient]``.</span> |
| |
| <span class="sd"> learning_rate : float, optional</span> |
| <span class="sd"> The initial learning rate.</span> |
| |
| <span class="sd"> lr_scheduler : LRScheduler, optional</span> |
| <span class="sd"> The learning rate scheduler.</span> |
| |
| <span class="sd"> wd : float, optional</span> |
| <span class="sd"> The weight decay (or L2 regularization) coefficient. Modifies objective</span> |
| <span class="sd"> by adding a penalty for having large weights.</span> |
| |
| <span class="sd"> sym: Symbol, optional</span> |
| <span class="sd"> The Symbol this optimizer is applying to.</span> |
| |
| <span class="sd"> begin_num_update : int, optional</span> |
| <span class="sd"> The initial number of updates.</span> |
| |
| <span class="sd"> multi_precision : bool, optional</span> |
| <span class="sd"> Flag to control the internal precision of the optimizer.</span> |
| <span class="sd"> ``False`` results in using the same precision as the weights (default),</span> |
| <span class="sd"> ``True`` makes internal 32-bit copy of the weights and applies gradients</span> |
| <span class="sd"> in 32-bit precision even if actual weights used in the model have lower precision.</span> |
| <span class="sd"> Turning this on can improve convergence and accuracy when training with float16.</span> |
| |
| <span class="sd"> Properties</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> learning_rate : float</span> |
| <span class="sd"> The current learning rate of the optimizer. Given an Optimizer object</span> |
| <span class="sd"> optimizer, its learning rate can be accessed as optimizer.learning_rate.</span> |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">rescale_grad</span><span class="o">=</span><span class="mf">1.</span><span class="p">,</span> <span class="n">param_idx2name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="mf">0.</span><span class="p">,</span> |
| <span class="n">clip_gradient</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.01</span><span class="p">,</span> |
| <span class="n">lr_scheduler</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">sym</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">begin_num_update</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> |
| <span class="n">multi_precision</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">param_dict</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">rescale_grad</span> <span class="o">=</span> <span class="n">rescale_grad</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">lr</span> <span class="o">=</span> <span class="n">learning_rate</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">lr_scheduler</span> <span class="o">=</span> <span class="n">lr_scheduler</span> |
| <span class="k">if</span> <span class="n">lr_scheduler</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">lr_scheduler</span><span class="o">.</span><span class="n">base_lr</span> <span class="o">=</span> <span class="n">learning_rate</span> |
| |
| <span class="bp">self</span><span class="o">.</span><span class="n">wd</span> <span class="o">=</span> <span class="n">wd</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">lr_mult</span> <span class="o">=</span> <span class="p">{}</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">wd_mult</span> <span class="o">=</span> <span class="p">{}</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">begin_num_update</span> <span class="o">=</span> <span class="n">begin_num_update</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">num_update</span> <span class="o">=</span> <span class="n">begin_num_update</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_index_update_count</span> <span class="o">=</span> <span class="p">{}</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span> <span class="o">=</span> <span class="n">clip_gradient</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">multi_precision</span> <span class="o">=</span> <span class="n">multi_precision</span> |
| |
| <span class="k">if</span> <span class="n">param_idx2name</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="n">param_idx2name</span> <span class="o">=</span> <span class="p">{}</span> |
| <span class="k">assert</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">param_idx2name</span><span class="p">,</span> <span class="nb">dict</span><span class="p">),</span> \ |
| <span class="s1">'param_idx2name should be a dict of param indexes to names.'</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">idx2name</span> <span class="o">=</span> <span class="n">param_idx2name</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">sym_info</span> <span class="o">=</span> <span class="p">(</span><span class="n">sym</span><span class="o">.</span><span class="n">attr_dict</span><span class="p">(),</span> <span class="n">sym</span><span class="o">.</span><span class="n">list_arguments</span><span class="p">())</span> <span class="k">if</span> <span class="n">sym</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="k">else</span> <span class="p">()</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">param_dict</span> <span class="o">=</span> <span class="n">param_dict</span> <span class="k">if</span> <span class="n">param_dict</span> <span class="k">else</span> <span class="p">{}</span> |
| |
| <span class="bp">self</span><span class="o">.</span><span class="n">set_lr_mult</span><span class="p">({})</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">set_wd_mult</span><span class="p">({})</span> |
| |
| <span class="n">opt_registry</span> <span class="o">=</span> <span class="p">{}</span> |
| |
| <span class="nd">@staticmethod</span> |
| <div class="viewcode-block" id="Optimizer.register"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Optimizer.register">[docs]</a> <span class="k">def</span> <span class="nf">register</span><span class="p">(</span><span class="n">klass</span><span class="p">):</span> |
| <span class="sd">"""Registers a new optimizer.</span> |
| |
| <span class="sd"> Once an optimizer is registered, we can create an instance of this</span> |
| <span class="sd"> optimizer with `create_optimizer` later.</span> |
| |
| <span class="sd"> Examples</span> |
| <span class="sd"> --------</span> |
| |
| <span class="sd"> >>> @mx.optimizer.Optimizer.register</span> |
| <span class="sd"> ... class MyOptimizer(mx.optimizer.Optimizer):</span> |
| <span class="sd"> ... pass</span> |
| <span class="sd"> >>> optim = mx.optimizer.Optimizer.create_optimizer('MyOptimizer')</span> |
| <span class="sd"> >>> print(type(optim))</span> |
| <span class="sd"> <class '__main__.MyOptimizer'></span> |
| <span class="sd"> """</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">klass</span><span class="p">,</span> <span class="nb">type</span><span class="p">))</span> |
| <span class="n">name</span> <span class="o">=</span> <span class="n">klass</span><span class="o">.</span><span class="vm">__name__</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> |
| <span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">Optimizer</span><span class="o">.</span><span class="n">opt_registry</span><span class="p">:</span> |
| <span class="n">warnings</span><span class="o">.</span><span class="n">warn</span><span class="p">(</span><span class="s1">'WARNING: New optimizer </span><span class="si">%s</span><span class="s1">.</span><span class="si">%s</span><span class="s1"> is overriding '</span> |
| <span class="s1">'existing optimizer </span><span class="si">%s</span><span class="s1">.</span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> |
| <span class="p">(</span><span class="n">klass</span><span class="o">.</span><span class="vm">__module__</span><span class="p">,</span> <span class="n">klass</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span> |
| <span class="n">Optimizer</span><span class="o">.</span><span class="n">opt_registry</span><span class="p">[</span><span class="n">name</span><span class="p">]</span><span class="o">.</span><span class="vm">__module__</span><span class="p">,</span> |
| <span class="n">Optimizer</span><span class="o">.</span><span class="n">opt_registry</span><span class="p">[</span><span class="n">name</span><span class="p">]</span><span class="o">.</span><span class="vm">__name__</span><span class="p">))</span> |
| <span class="n">Optimizer</span><span class="o">.</span><span class="n">opt_registry</span><span class="p">[</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">klass</span> |
| <span class="k">return</span> <span class="n">klass</span></div> |
| |
| <span class="nd">@staticmethod</span> |
| <div class="viewcode-block" id="Optimizer.create_optimizer"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Optimizer.create_optimizer">[docs]</a> <span class="k">def</span> <span class="nf">create_optimizer</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> |
| <span class="sd">"""Instantiates an optimizer with a given name and kwargs.</span> |
| |
| <span class="sd"> .. note:: We can use the alias `create` for ``Optimizer.create_optimizer``.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> name: str</span> |
| <span class="sd"> Name of the optimizer. Should be the name</span> |
| <span class="sd"> of a subclass of Optimizer. Case insensitive.</span> |
| |
| <span class="sd"> kwargs: dict</span> |
| <span class="sd"> Parameters for the optimizer.</span> |
| |
| <span class="sd"> Returns</span> |
| <span class="sd"> -------</span> |
| <span class="sd"> Optimizer</span> |
| <span class="sd"> An instantiated optimizer.</span> |
| |
| <span class="sd"> Examples</span> |
| <span class="sd"> --------</span> |
| <span class="sd"> >>> sgd = mx.optimizer.Optimizer.create_optimizer('sgd')</span> |
| <span class="sd"> >>> type(sgd)</span> |
| <span class="sd"> <class 'mxnet.optimizer.SGD'></span> |
| <span class="sd"> >>> adam = mx.optimizer.create('adam', learning_rate=.1)</span> |
| <span class="sd"> >>> type(adam)</span> |
| <span class="sd"> <class 'mxnet.optimizer.Adam'></span> |
| <span class="sd"> """</span> |
| <span class="k">if</span> <span class="n">name</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="ow">in</span> <span class="n">Optimizer</span><span class="o">.</span><span class="n">opt_registry</span><span class="p">:</span> |
| <span class="k">return</span> <span class="n">Optimizer</span><span class="o">.</span><span class="n">opt_registry</span><span class="p">[</span><span class="n">name</span><span class="o">.</span><span class="n">lower</span><span class="p">()](</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'Cannot find optimizer </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">name</span><span class="p">)</span></div> |
| |
| <span class="nd">@property</span> |
| <span class="k">def</span> <span class="nf">learning_rate</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">lr_scheduler</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">lr_scheduler</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">num_update</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">lr</span> |
| |
| <div class="viewcode-block" id="Optimizer.create_state"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Optimizer.create_state">[docs]</a> <span class="k">def</span> <span class="nf">create_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="sd">"""Creates auxiliary state for a given weight.</span> |
| |
| <span class="sd"> Some optimizers require additional states, e.g. as momentum, in addition</span> |
| <span class="sd"> to gradients in order to update weights. This function creates state</span> |
| <span class="sd"> for a given weight which will be used in `update`. This function is</span> |
| <span class="sd"> called only once for each weight.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> index : int</span> |
| <span class="sd"> An unique index to identify the weight.</span> |
| <span class="sd"> weight : NDArray</span> |
| <span class="sd"> The weight.</span> |
| |
| <span class="sd"> Returns</span> |
| <span class="sd"> -------</span> |
| <span class="sd"> state : any obj</span> |
| <span class="sd"> The state associated with the weight.</span> |
| <span class="sd"> """</span></div> |
| |
| <div class="viewcode-block" id="Optimizer.create_state_multi_precision"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Optimizer.create_state_multi_precision">[docs]</a> <span class="k">def</span> <span class="nf">create_state_multi_precision</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="sd">"""Creates auxiliary state for a given weight, including FP32 high</span> |
| <span class="sd"> precision copy if original weight is FP16.</span> |
| |
| <span class="sd"> This method is provided to perform automatic mixed precision training</span> |
| <span class="sd"> for optimizers that do not support it themselves.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> index : int</span> |
| <span class="sd"> An unique index to identify the weight.</span> |
| <span class="sd"> weight : NDArray</span> |
| <span class="sd"> The weight.</span> |
| |
| <span class="sd"> Returns</span> |
| <span class="sd"> -------</span> |
| <span class="sd"> state : any obj</span> |
| <span class="sd"> The state associated with the weight.</span> |
| <span class="sd"> """</span> |
| <span class="n">weight_master_copy</span> <span class="o">=</span> <span class="kc">None</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">multi_precision</span> <span class="ow">and</span> <span class="n">weight</span><span class="o">.</span><span class="n">dtype</span> <span class="o">==</span> <span class="n">numpy</span><span class="o">.</span><span class="n">float16</span><span class="p">:</span> |
| <span class="n">weight_master_copy</span> <span class="o">=</span> <span class="n">weight</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="n">numpy</span><span class="o">.</span><span class="n">float32</span><span class="p">)</span> |
| <span class="k">return</span> <span class="p">(</span><span class="n">weight_master_copy</span><span class="p">,)</span> <span class="o">+</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">create_state</span><span class="p">(</span><span class="n">index</span><span class="p">,</span> <span class="n">weight_master_copy</span><span class="p">),)</span> |
| <span class="k">if</span> <span class="n">weight</span><span class="o">.</span><span class="n">dtype</span> <span class="o">==</span> <span class="n">numpy</span><span class="o">.</span><span class="n">float16</span> <span class="ow">and</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">multi_precision</span><span class="p">:</span> |
| <span class="n">warnings</span><span class="o">.</span><span class="n">warn</span><span class="p">(</span><span class="s2">"Accumulating with float16 in optimizer can lead to "</span> |
| <span class="s2">"poor accuracy or slow convergence. "</span> |
| <span class="s2">"Consider using multi_precision=True option of the "</span> |
| <span class="s2">"optimizer"</span><span class="p">)</span> |
| <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">create_state</span><span class="p">(</span><span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">)</span></div> |
| |
| <div class="viewcode-block" id="Optimizer.update"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Optimizer.update">[docs]</a> <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="sd">"""Updates the given parameter using the corresponding gradient and state.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> index : int</span> |
| <span class="sd"> The unique index of the parameter into the individual learning</span> |
| <span class="sd"> rates and weight decays. Learning rates and weight decay</span> |
| <span class="sd"> may be set via `set_lr_mult()` and `set_wd_mult()`, respectively.</span> |
| <span class="sd"> weight : NDArray</span> |
| <span class="sd"> The parameter to be updated.</span> |
| <span class="sd"> grad : NDArray</span> |
| <span class="sd"> The gradient of the objective with respect to this parameter.</span> |
| <span class="sd"> state : any obj</span> |
| <span class="sd"> The state returned by `create_state()`.</span> |
| <span class="sd"> """</span> |
| <span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">()</span></div> |
| |
| <div class="viewcode-block" id="Optimizer.update_multi_precision"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Optimizer.update_multi_precision">[docs]</a> <span class="k">def</span> <span class="nf">update_multi_precision</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="sd">"""Updates the given parameter using the corresponding gradient and state.</span> |
| <span class="sd"> Mixed precision version.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> index : int</span> |
| <span class="sd"> The unique index of the parameter into the individual learning</span> |
| <span class="sd"> rates and weight decays. Learning rates and weight decay</span> |
| <span class="sd"> may be set via `set_lr_mult()` and `set_wd_mult()`, respectively.</span> |
| <span class="sd"> weight : NDArray</span> |
| <span class="sd"> The parameter to be updated.</span> |
| <span class="sd"> grad : NDArray</span> |
| <span class="sd"> The gradient of the objective with respect to this parameter.</span> |
| <span class="sd"> state : any obj</span> |
| <span class="sd"> The state returned by `create_state()`.</span> |
| <span class="sd"> """</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">multi_precision</span> <span class="ow">and</span> <span class="n">weight</span><span class="o">.</span><span class="n">dtype</span> <span class="o">==</span> <span class="n">numpy</span><span class="o">.</span><span class="n">float16</span><span class="p">:</span> |
| <span class="c1"># Wrapper for mixed precision</span> |
| <span class="n">weight_master_copy</span> <span class="o">=</span> <span class="n">state</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> |
| <span class="n">original_state</span> <span class="o">=</span> <span class="n">state</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> |
| <span class="n">grad32</span> <span class="o">=</span> <span class="n">grad</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="n">numpy</span><span class="o">.</span><span class="n">float32</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">index</span><span class="p">,</span> <span class="n">weight_master_copy</span><span class="p">,</span> <span class="n">grad32</span><span class="p">,</span> <span class="n">original_state</span><span class="p">)</span> |
| <span class="n">cast</span><span class="p">(</span><span class="n">weight_master_copy</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">dtype</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">)</span></div> |
| |
| <div class="viewcode-block" id="Optimizer.set_learning_rate"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Optimizer.set_learning_rate">[docs]</a> <span class="k">def</span> <span class="nf">set_learning_rate</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">lr</span><span class="p">):</span> |
| <span class="sd">"""Sets a new learning rate of the optimizer.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> lr : float</span> |
| <span class="sd"> The new learning rate of the optimizer.</span> |
| <span class="sd"> """</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">lr_scheduler</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="k">raise</span> <span class="ne">UserWarning</span><span class="p">(</span><span class="s2">"LRScheduler of the optimizer has already been "</span> |
| <span class="s2">"defined. Note that set_learning_rate can mutate "</span> |
| <span class="s2">"the value of the learning rate of the optimizer "</span> |
| <span class="s2">"only when the LRScheduler of the optimizer is "</span> |
| <span class="s2">"undefined."</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">lr</span> <span class="o">=</span> <span class="n">lr</span></div> |
| |
| <div class="viewcode-block" id="Optimizer.set_lr_scale"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Optimizer.set_lr_scale">[docs]</a> <span class="k">def</span> <span class="nf">set_lr_scale</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">args_lrscale</span><span class="p">):</span> <span class="c1"># pylint: disable=unused-argument</span> |
| <span class="sd">"""[DEPRECATED] Sets lr scale. Use set_lr_mult instead."""</span> |
| <span class="k">raise</span> <span class="ne">DeprecationWarning</span></div> |
| |
| <div class="viewcode-block" id="Optimizer.set_lr_mult"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Optimizer.set_lr_mult">[docs]</a> <span class="k">def</span> <span class="nf">set_lr_mult</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">args_lr_mult</span><span class="p">):</span> |
| <span class="sd">"""Sets an individual learning rate multiplier for each parameter.</span> |
| |
| <span class="sd"> If you specify a learning rate multiplier for a parameter, then</span> |
| <span class="sd"> the learning rate for the parameter will be set as the product of</span> |
| <span class="sd"> the global learning rate `self.lr` and its multiplier.</span> |
| |
| <span class="sd"> .. note:: The default learning rate multiplier of a `Variable`</span> |
| <span class="sd"> can be set with `lr_mult` argument in the constructor.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> args_lr_mult : dict of str/int to float</span> |
| <span class="sd"> For each of its key-value entries, the learning rate multipler for the</span> |
| <span class="sd"> parameter specified in the key will be set as the given value.</span> |
| |
| <span class="sd"> You can specify the parameter with either its name or its index.</span> |
| <span class="sd"> If you use the name, you should pass `sym` in the constructor,</span> |
| <span class="sd"> and the name you specified in the key of `args_lr_mult` should match</span> |
| <span class="sd"> the name of the parameter in `sym`. If you use the index, it should</span> |
| <span class="sd"> correspond to the index of the parameter used in the `update` method.</span> |
| |
| <span class="sd"> Specifying a parameter by its index is only supported for backward</span> |
| <span class="sd"> compatibility, and we recommend to use the name instead.</span> |
| <span class="sd"> """</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">lr_mult</span> <span class="o">=</span> <span class="p">{}</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">sym_info</span><span class="p">:</span> |
| <span class="n">attr</span><span class="p">,</span> <span class="n">arg_names</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">sym_info</span> |
| <span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">arg_names</span><span class="p">:</span> |
| <span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">attr</span> <span class="ow">and</span> <span class="s1">'__lr_mult__'</span> <span class="ow">in</span> <span class="n">attr</span><span class="p">[</span><span class="n">name</span><span class="p">]:</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">lr_mult</span><span class="p">[</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="n">attr</span><span class="p">[</span><span class="n">name</span><span class="p">][</span><span class="s1">'__lr_mult__'</span><span class="p">])</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">lr_mult</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">args_lr_mult</span><span class="p">)</span></div> |
| |
| <div class="viewcode-block" id="Optimizer.set_wd_mult"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Optimizer.set_wd_mult">[docs]</a> <span class="k">def</span> <span class="nf">set_wd_mult</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">args_wd_mult</span><span class="p">):</span> |
| <span class="sd">"""Sets an individual weight decay multiplier for each parameter.</span> |
| |
| <span class="sd"> By default, if `param_idx2name` was provided in the</span> |
| <span class="sd"> constructor, the weight decay multipler is set as 0 for all</span> |
| <span class="sd"> parameters whose name don't end with ``_weight`` or</span> |
| <span class="sd"> ``_gamma``.</span> |
| |
| <span class="sd"> .. note:: The default weight decay multiplier for a `Variable`</span> |
| <span class="sd"> can be set with its `wd_mult` argument in the constructor.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> args_wd_mult : dict of string/int to float</span> |
| <span class="sd"> For each of its key-value entries, the weight decay multipler for the</span> |
| <span class="sd"> parameter specified in the key will be set as the given value.</span> |
| |
| <span class="sd"> You can specify the parameter with either its name or its index.</span> |
| <span class="sd"> If you use the name, you should pass `sym` in the constructor,</span> |
| <span class="sd"> and the name you specified in the key of `args_lr_mult` should match</span> |
| <span class="sd"> the name of the parameter in `sym`. If you use the index, it should</span> |
| <span class="sd"> correspond to the index of the parameter used in the `update` method.</span> |
| |
| <span class="sd"> Specifying a parameter by its index is only supported for backward</span> |
| <span class="sd"> compatibility, and we recommend to use the name instead.</span> |
| <span class="sd"> """</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">wd_mult</span> <span class="o">=</span> <span class="p">{}</span> |
| <span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">idx2name</span><span class="o">.</span><span class="n">values</span><span class="p">():</span> |
| <span class="k">if</span> <span class="ow">not</span> <span class="p">(</span><span class="n">n</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s1">'_weight'</span><span class="p">)</span> <span class="ow">or</span> <span class="n">n</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s1">'_gamma'</span><span class="p">)):</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">wd_mult</span><span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="o">=</span> <span class="mf">0.0</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">sym_info</span><span class="p">:</span> |
| <span class="n">attr</span><span class="p">,</span> <span class="n">arg_names</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">sym_info</span> |
| <span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">arg_names</span><span class="p">:</span> |
| <span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">attr</span> <span class="ow">and</span> <span class="s1">'__wd_mult__'</span> <span class="ow">in</span> <span class="n">attr</span><span class="p">[</span><span class="n">name</span><span class="p">]:</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">wd_mult</span><span class="p">[</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="n">attr</span><span class="p">[</span><span class="n">name</span><span class="p">][</span><span class="s1">'__wd_mult__'</span><span class="p">])</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">wd_mult</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">args_wd_mult</span><span class="p">)</span></div> |
| |
| <span class="k">def</span> <span class="nf">_update_count</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">):</span> |
| <span class="sd">"""Updates num_update.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> index : int</span> |
| <span class="sd"> The index to be updated.</span> |
| <span class="sd"> """</span> |
| <span class="k">if</span> <span class="n">index</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_index_update_count</span><span class="p">:</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_index_update_count</span><span class="p">[</span><span class="n">index</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">begin_num_update</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_index_update_count</span><span class="p">[</span><span class="n">index</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">num_update</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_index_update_count</span><span class="p">[</span><span class="n">index</span><span class="p">],</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_update</span><span class="p">)</span> |
| |
| <span class="k">def</span> <span class="nf">_get_lr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">):</span> |
| <span class="sd">"""Gets the learning rate given the index of the weight.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> index : int</span> |
| <span class="sd"> The index corresponding to the weight.</span> |
| |
| <span class="sd"> Returns</span> |
| <span class="sd"> -------</span> |
| <span class="sd"> lr : float</span> |
| <span class="sd"> Learning rate for this index.</span> |
| <span class="sd"> """</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">lr_scheduler</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="n">lr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">lr_scheduler</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">num_update</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="n">lr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">lr</span> |
| |
| <span class="k">if</span> <span class="n">index</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">param_dict</span><span class="p">:</span> |
| <span class="n">lr</span> <span class="o">*=</span> <span class="bp">self</span><span class="o">.</span><span class="n">param_dict</span><span class="p">[</span><span class="n">index</span><span class="p">]</span><span class="o">.</span><span class="n">lr_mult</span> |
| <span class="k">elif</span> <span class="n">index</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">lr_mult</span><span class="p">:</span> |
| <span class="n">lr</span> <span class="o">*=</span> <span class="bp">self</span><span class="o">.</span><span class="n">lr_mult</span><span class="p">[</span><span class="n">index</span><span class="p">]</span> |
| <span class="k">elif</span> <span class="n">index</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">idx2name</span><span class="p">:</span> |
| <span class="n">lr</span> <span class="o">*=</span> <span class="bp">self</span><span class="o">.</span><span class="n">lr_mult</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">idx2name</span><span class="p">[</span><span class="n">index</span><span class="p">],</span> <span class="mf">1.0</span><span class="p">)</span> |
| <span class="k">return</span> <span class="n">lr</span> |
| |
| <span class="k">def</span> <span class="nf">_get_wd</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">):</span> |
| <span class="sd">"""Gets weight decay for index.</span> |
| <span class="sd"> Returns 0 for non-weights if the name of weights are provided for `__init__`.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> index : int</span> |
| <span class="sd"> The index for weight.</span> |
| |
| <span class="sd"> Returns</span> |
| <span class="sd"> -------</span> |
| <span class="sd"> wd : float</span> |
| <span class="sd"> Weight decay for this index.</span> |
| <span class="sd"> """</span> |
| <span class="n">wd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">wd</span> |
| <span class="k">if</span> <span class="n">index</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">param_dict</span><span class="p">:</span> |
| <span class="n">wd</span> <span class="o">*=</span> <span class="bp">self</span><span class="o">.</span><span class="n">param_dict</span><span class="p">[</span><span class="n">index</span><span class="p">]</span><span class="o">.</span><span class="n">wd_mult</span> |
| <span class="k">elif</span> <span class="n">index</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">wd_mult</span><span class="p">:</span> |
| <span class="n">wd</span> <span class="o">*=</span> <span class="bp">self</span><span class="o">.</span><span class="n">wd_mult</span><span class="p">[</span><span class="n">index</span><span class="p">]</span> |
| <span class="k">elif</span> <span class="n">index</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">idx2name</span><span class="p">:</span> |
| <span class="n">wd</span> <span class="o">*=</span> <span class="bp">self</span><span class="o">.</span><span class="n">wd_mult</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">idx2name</span><span class="p">[</span><span class="n">index</span><span class="p">],</span> <span class="mf">1.0</span><span class="p">)</span> |
| <span class="k">return</span> <span class="n">wd</span> |
| |
| <span class="k">def</span> <span class="nf">__getstate__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> |
| <span class="n">ret</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="vm">__dict__</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span> |
| <span class="c1"># do not include param_dict in the state</span> |
| <span class="k">del</span> <span class="n">ret</span><span class="p">[</span><span class="s1">'param_dict'</span><span class="p">]</span> |
| <span class="k">return</span> <span class="n">ret</span> |
| |
| <span class="k">def</span> <span class="nf">__setstate__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="bp">self</span><span class="o">.</span><span class="vm">__dict__</span> <span class="o">=</span> <span class="n">state</span> |
| <span class="c1"># param_dict needs to be explicitly set by the trainer</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">param_dict</span> <span class="o">=</span> <span class="p">{}</span></div> |
| |
| <span class="c1"># convenience wrapper for Optimizer.Register</span> |
| <span class="n">register</span> <span class="o">=</span> <span class="n">Optimizer</span><span class="o">.</span><span class="n">register</span> <span class="c1"># pylint: disable=invalid-name</span> |
| |
| <span class="c1"># pylint: disable=line-too-long</span> |
| <span class="nd">@register</span> |
| <div class="viewcode-block" id="SGD"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.SGD">[docs]</a><span class="k">class</span> <span class="nc">SGD</span><span class="p">(</span><span class="n">Optimizer</span><span class="p">):</span> |
| <span class="sd">"""The SGD optimizer with momentum and weight decay.</span> |
| |
| <span class="sd"> If the storage types of grad is ``row_sparse`` and ``lazy_update`` is True, \</span> |
| <span class="sd"> **lazy updates** are applied by::</span> |
| |
| <span class="sd"> for row in grad.indices:</span> |
| <span class="sd"> rescaled_grad[row] = lr * (rescale_grad * clip(grad[row], clip_gradient) + wd * weight[row])</span> |
| <span class="sd"> state[row] = momentum[row] * state[row] + rescaled_grad[row]</span> |
| <span class="sd"> weight[row] = weight[row] - state[row]</span> |
| |
| <span class="sd"> The sparse update only updates the momentum for the weights whose row_sparse</span> |
| <span class="sd"> gradient indices appear in the current batch, rather than updating it for all</span> |
| <span class="sd"> indices. Compared with the original update, it can provide large</span> |
| <span class="sd"> improvements in model training throughput for some applications. However, it</span> |
| <span class="sd"> provides slightly different semantics than the original update, and</span> |
| <span class="sd"> may lead to different empirical results.</span> |
| |
| <span class="sd"> Otherwise, **standard updates** are applied by::</span> |
| |
| <span class="sd"> rescaled_grad = lr * (rescale_grad * clip(grad, clip_gradient) + wd * weight)</span> |
| <span class="sd"> state = momentum * state + rescaled_grad</span> |
| <span class="sd"> weight = weight - state</span> |
| |
| <span class="sd"> For details of the update algorithm see</span> |
| <span class="sd"> :class:`~mxnet.ndarray.sgd_update` and :class:`~mxnet.ndarray.sgd_mom_update`.</span> |
| |
| <span class="sd"> This optimizer accepts the following parameters in addition to those accepted</span> |
| <span class="sd"> by :class:`.Optimizer`.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> momentum : float, optional</span> |
| <span class="sd"> The momentum value.</span> |
| <span class="sd"> lazy_update : bool, optional</span> |
| <span class="sd"> Default is True. If True, lazy updates are applied \</span> |
| <span class="sd"> if the storage types of weight and grad are both ``row_sparse``.</span> |
| <span class="sd"> multi_precision: bool, optional</span> |
| <span class="sd"> Flag to control the internal precision of the optimizer.</span> |
| <span class="sd"> ``False`` results in using the same precision as the weights (default),</span> |
| <span class="sd"> ``True`` makes internal 32-bit copy of the weights and applies gradients \</span> |
| <span class="sd"> in 32-bit precision even if actual weights used in the model have lower precision.\</span> |
| <span class="sd"> Turning this on can improve convergence and accuracy when training with float16.</span> |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">momentum</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">lazy_update</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">SGD</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> <span class="o">=</span> <span class="n">momentum</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">lazy_update</span> <span class="o">=</span> <span class="n">lazy_update</span> |
| |
| <span class="k">def</span> <span class="nf">create_state_multi_precision</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="n">weight_master_copy</span> <span class="o">=</span> <span class="kc">None</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">multi_precision</span> <span class="ow">and</span> <span class="n">weight</span><span class="o">.</span><span class="n">dtype</span> <span class="o">==</span> <span class="n">numpy</span><span class="o">.</span><span class="n">float16</span><span class="p">:</span> |
| <span class="n">weight_master_copy</span> <span class="o">=</span> <span class="n">weight</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="n">numpy</span><span class="o">.</span><span class="n">float32</span><span class="p">)</span> |
| <span class="k">return</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">create_state</span><span class="p">(</span><span class="n">index</span><span class="p">,</span> <span class="n">weight_master_copy</span><span class="p">),</span> <span class="n">weight_master_copy</span><span class="p">)</span> |
| <span class="k">if</span> <span class="n">weight</span><span class="o">.</span><span class="n">dtype</span> <span class="o">==</span> <span class="n">numpy</span><span class="o">.</span><span class="n">float16</span> <span class="ow">and</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">multi_precision</span><span class="p">:</span> |
| <span class="n">warnings</span><span class="o">.</span><span class="n">warn</span><span class="p">(</span><span class="s2">"Accumulating with float16 in optimizer can lead to "</span> |
| <span class="s2">"poor accuracy or slow convergence. "</span> |
| <span class="s2">"Consider using multi_precision=True option of the "</span> |
| <span class="s2">"SGD optimizer"</span><span class="p">)</span> |
| <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">create_state</span><span class="p">(</span><span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">)</span> |
| |
| <span class="k">def</span> <span class="nf">create_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="n">momentum</span> <span class="o">=</span> <span class="kc">None</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> <span class="o">!=</span> <span class="mf">0.0</span><span class="p">:</span> |
| <span class="n">stype</span> <span class="o">=</span> <span class="n">weight</span><span class="o">.</span><span class="n">stype</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">lazy_update</span> <span class="k">else</span> <span class="s1">'default'</span> |
| <span class="n">momentum</span> <span class="o">=</span> <span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">dtype</span><span class="p">,</span> <span class="n">stype</span><span class="o">=</span><span class="n">stype</span><span class="p">)</span> |
| <span class="k">return</span> <span class="n">momentum</span> |
| |
| <span class="k">def</span> <span class="nf">_update_impl</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">,</span> <span class="n">multi_precision</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_update_count</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">lr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_lr</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">wd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_wd</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| |
| <span class="n">kwargs</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'rescale_grad'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">rescale_grad</span><span class="p">}</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span> |
| <span class="n">kwargs</span><span class="p">[</span><span class="s1">'momentum'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">:</span> |
| <span class="n">kwargs</span><span class="p">[</span><span class="s1">'clip_gradient'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span> |
| |
| <span class="k">if</span> <span class="ow">not</span> <span class="n">multi_precision</span><span class="p">:</span> |
| <span class="k">if</span> <span class="n">state</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="n">sgd_mom_update</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span> |
| <span class="n">lazy_update</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">lazy_update</span><span class="p">,</span> <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="n">wd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="n">sgd_update</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span> <span class="n">lazy_update</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">lazy_update</span><span class="p">,</span> |
| <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="n">wd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="k">if</span> <span class="n">state</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="n">mp_sgd_mom_update</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">state</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span> |
| <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="n">wd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="n">mp_sgd_update</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span> |
| <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="n">wd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| |
| <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_update_impl</span><span class="p">(</span><span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">,</span> <span class="n">multi_precision</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span> |
| |
| <span class="k">def</span> <span class="nf">update_multi_precision</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="n">use_multi_precision</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">multi_precision</span> <span class="ow">and</span> <span class="n">weight</span><span class="o">.</span><span class="n">dtype</span> <span class="o">==</span> <span class="n">numpy</span><span class="o">.</span><span class="n">float16</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_update_impl</span><span class="p">(</span><span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">,</span> |
| <span class="n">multi_precision</span><span class="o">=</span><span class="n">use_multi_precision</span><span class="p">)</span></div> |
| |
| <span class="nd">@register</span> |
| <div class="viewcode-block" id="Signum"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Signum">[docs]</a><span class="k">class</span> <span class="nc">Signum</span><span class="p">(</span><span class="n">Optimizer</span><span class="p">):</span> |
| <span class="sa">r</span><span class="sd">"""The Signum optimizer that takes the sign of gradient or momentum.</span> |
| |
| <span class="sd"> The optimizer updates the weight by::</span> |
| |
| <span class="sd"> rescaled_grad = rescale_grad * clip(grad, clip_gradient) + wd * weight</span> |
| <span class="sd"> state = momentum * state + (1-momentum)*rescaled_grad</span> |
| <span class="sd"> weight = (1 - lr * wd_lh) * weight - lr * sign(state)</span> |
| |
| <span class="sd"> Reference:</span> |
| <span class="sd"> Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli & Anima Anandkumar. (2018).</span> |
| <span class="sd"> signSGD: Compressed Optimisation for Non-Convex Problems. In ICML'18.</span> |
| |
| <span class="sd"> See: https://arxiv.org/abs/1802.04434</span> |
| |
| <span class="sd"> For details of the update algorithm see</span> |
| <span class="sd"> :class:`~mxnet.ndarray.signsgd_update` and :class:`~mxnet.ndarray.signum_update`.</span> |
| |
| <span class="sd"> This optimizer accepts the following parameters in addition to those accepted</span> |
| <span class="sd"> by :class:`.Optimizer`.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> momentum : float, optional</span> |
| <span class="sd"> The momentum value.</span> |
| <span class="sd"> wd_lh : float, optional</span> |
| <span class="sd"> The amount of decoupled weight decay regularization, see details in the original paper at:\</span> |
| <span class="sd"> https://arxiv.org/abs/1711.05101</span> |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.01</span><span class="p">,</span> <span class="n">momentum</span><span class="o">=</span><span class="mf">0.9</span><span class="p">,</span> <span class="n">wd_lh</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">Signum</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">learning_rate</span><span class="o">=</span><span class="n">learning_rate</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> <span class="o">=</span> <span class="n">momentum</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">wd_lh</span> <span class="o">=</span> <span class="n">wd_lh</span> |
| |
| <span class="k">def</span> <span class="nf">create_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="n">momentum</span> <span class="o">=</span> <span class="kc">None</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> <span class="o">!=</span> <span class="mf">0.0</span><span class="p">:</span> |
| <span class="n">momentum</span> <span class="o">=</span> <span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">dtype</span><span class="p">,</span> <span class="n">stype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">stype</span><span class="p">)</span> |
| <span class="k">return</span> <span class="n">momentum</span> |
| |
| <span class="k">def</span> <span class="nf">_update_impl</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_update_count</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">lr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_lr</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">wd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_wd</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| |
| <span class="n">kwargs</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'rescale_grad'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">rescale_grad</span><span class="p">}</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span> |
| <span class="n">kwargs</span><span class="p">[</span><span class="s1">'momentum'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">:</span> |
| <span class="n">kwargs</span><span class="p">[</span><span class="s1">'clip_gradient'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">wd_lh</span><span class="p">:</span> |
| <span class="n">kwargs</span><span class="p">[</span><span class="s1">'wd_lh'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">wd_lh</span> |
| |
| <span class="k">if</span> <span class="n">state</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="n">signum_update</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span> |
| <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="n">wd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="n">signsgd_update</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span> |
| <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="n">wd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| |
| <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_update_impl</span><span class="p">(</span><span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">)</span></div> |
| |
| <span class="nd">@register</span> |
| <div class="viewcode-block" id="FTML"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.FTML">[docs]</a><span class="k">class</span> <span class="nc">FTML</span><span class="p">(</span><span class="n">Optimizer</span><span class="p">):</span> |
| <span class="sd">"""The FTML optimizer.</span> |
| |
| <span class="sd"> This class implements the optimizer described in</span> |
| <span class="sd"> *FTML - Follow the Moving Leader in Deep Learning*,</span> |
| <span class="sd"> available at http://proceedings.mlr.press/v70/zheng17a/zheng17a.pdf.</span> |
| |
| <span class="sd"> Denote time step by t. The optimizer updates the weight by::</span> |
| |
| <span class="sd"> rescaled_grad = clip(grad * rescale_grad + wd * weight, clip_gradient)</span> |
| <span class="sd"> v = beta2 * v + (1 - beta2) * square(rescaled_grad)</span> |
| <span class="sd"> d_t = (1 - power(beta1, t)) / lr * square_root(v / (1 - power(beta2, t))) + epsilon)</span> |
| <span class="sd"> z = beta1 * z + (1 - beta1) * rescaled_grad - (d_t - beta1 * d_(t-1)) * weight</span> |
| <span class="sd"> weight = - z / d_t</span> |
| |
| <span class="sd"> This optimizer accepts the following parameters in addition to those accepted</span> |
| <span class="sd"> by :class:`.Optimizer`.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> beta1 : float, optional</span> |
| <span class="sd"> 0 < beta1 < 1. Generally close to 0.5.</span> |
| <span class="sd"> beta2 : float, optional</span> |
| <span class="sd"> 0 < beta2 < 1. Generally close to 1.</span> |
| <span class="sd"> epsilon : float, optional</span> |
| <span class="sd"> Small value to avoid division by 0.</span> |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">beta1</span><span class="o">=</span><span class="mf">0.6</span><span class="p">,</span> <span class="n">beta2</span><span class="o">=</span><span class="mf">0.999</span><span class="p">,</span> <span class="n">epsilon</span><span class="o">=</span><span class="mf">1e-8</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">FTML</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">beta1</span> <span class="o">=</span> <span class="n">beta1</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">beta2</span> <span class="o">=</span> <span class="n">beta2</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">epsilon</span> <span class="o">=</span> <span class="n">epsilon</span> |
| |
| <span class="k">def</span> <span class="nf">create_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="k">return</span> <span class="p">(</span><span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">dtype</span><span class="p">),</span> <span class="c1"># d_0</span> |
| <span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">dtype</span><span class="p">),</span> <span class="c1"># v_0</span> |
| <span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">dtype</span><span class="p">))</span> <span class="c1"># z_0</span> |
| |
| <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_update_count</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">lr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_lr</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">wd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_wd</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">t</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_index_update_count</span><span class="p">[</span><span class="n">index</span><span class="p">]</span> |
| |
| <span class="n">kwargs</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'beta1'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">beta1</span><span class="p">,</span> <span class="s1">'beta2'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">beta2</span><span class="p">,</span> <span class="s1">'epsilon'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">epsilon</span><span class="p">,</span> |
| <span class="s1">'rescale_grad'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">rescale_grad</span><span class="p">,</span> <span class="s1">'t'</span><span class="p">:</span> <span class="n">t</span><span class="p">}</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">:</span> |
| <span class="n">kwargs</span><span class="p">[</span><span class="s1">'clip_grad'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span> |
| |
| <span class="n">prev_d</span><span class="p">,</span> <span class="n">prev_v</span><span class="p">,</span> <span class="n">prev_z</span> <span class="o">=</span> <span class="n">state</span> |
| <span class="n">ftml_update</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">prev_d</span><span class="p">,</span> <span class="n">prev_v</span><span class="p">,</span> <span class="n">prev_z</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span> |
| <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="n">wd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span></div> |
| |
| <span class="nd">@register</span> |
| <div class="viewcode-block" id="LBSGD"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.LBSGD">[docs]</a><span class="k">class</span> <span class="nc">LBSGD</span><span class="p">(</span><span class="n">Optimizer</span><span class="p">):</span> |
| <span class="sd">"""The Large Batch SGD optimizer with momentum and weight decay.</span> |
| |
| <span class="sd"> The optimizer updates the weight by::</span> |
| |
| <span class="sd"> state = momentum * state + lr * rescale_grad * clip(grad, clip_gradient) + wd * weight</span> |
| <span class="sd"> weight = weight - state</span> |
| |
| <span class="sd"> For details of the update algorithm see :class:`~mxnet.ndarray.lbsgd_update` and</span> |
| <span class="sd"> :class:`~mxnet.ndarray.lbsgd_mom_update`.</span> |
| |
| <span class="sd"> This optimizer accepts the following parameters in addition to those accepted</span> |
| <span class="sd"> by :class:`.Optimizer`.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> momentum : float, optional</span> |
| <span class="sd"> The momentum value.</span> |
| <span class="sd"> multi_precision: bool, optional</span> |
| <span class="sd"> Flag to control the internal precision of the optimizer.</span> |
| <span class="sd"> ``False`` results in using the same precision as the weights (default),</span> |
| <span class="sd"> ``True`` makes internal 32-bit copy of the weights and applies gradients</span> |
| <span class="sd"> in 32-bit precision even if actual weights used in the model have lower precision.`<</span> |
| <span class="sd"> Turning this on can improve convergence and accuracy when training with float16.</span> |
| <span class="sd"> warmup_strategy: string ('linear', 'power2', 'sqrt'. , 'lars' default : 'linear')</span> |
| <span class="sd"> warmup_epochs: unsigned, default: 5</span> |
| <span class="sd"> batch_scale: unsigned, default: 1 (same as batch size*numworkers)</span> |
| <span class="sd"> updates_per_epoch: updates_per_epoch (default: 32, Default might not reflect true number batches per epoch. Used for warmup.)</span> |
| <span class="sd"> begin_epoch: unsigned, default 0, starting epoch.</span> |
| <span class="sd"> """</span> |
| |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">momentum</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">multi_precision</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">warmup_strategy</span><span class="o">=</span><span class="s1">'linear'</span><span class="p">,</span> |
| <span class="n">warmup_epochs</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">batch_scale</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">updates_per_epoch</span><span class="o">=</span><span class="mi">32</span><span class="p">,</span> <span class="n">begin_epoch</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">num_epochs</span><span class="o">=</span><span class="mi">60</span><span class="p">,</span> |
| <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">LBSGD</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Running Large-Batch SGD Algorithm'</span><span class="p">)</span> |
| <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'(Batch_scale=</span><span class="si">%f</span><span class="s1">, warmup_epochs=</span><span class="si">%d</span><span class="s1">, warmup_strategy=</span><span class="si">%s</span><span class="s1">, updates_per_epoch=</span><span class="si">%d</span><span class="s1">)'</span><span class="p">,</span> |
| <span class="n">batch_scale</span><span class="p">,</span> <span class="n">warmup_epochs</span><span class="p">,</span> <span class="n">warmup_strategy</span><span class="p">,</span> <span class="n">updates_per_epoch</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> <span class="o">=</span> <span class="n">momentum</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">multi_precision</span> <span class="o">=</span> <span class="n">multi_precision</span> |
| <span class="c1"># new user parameters for large batch</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">warmup_strategy</span> <span class="o">=</span> <span class="n">warmup_strategy</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">warmup_epochs</span> <span class="o">=</span> <span class="n">warmup_epochs</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">batch_scale</span> <span class="o">=</span> <span class="n">batch_scale</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">updates_per_epoch</span> <span class="o">=</span> <span class="n">updates_per_epoch</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">init_updates</span> <span class="o">=</span> <span class="n">begin_epoch</span> <span class="o">*</span> <span class="n">updates_per_epoch</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">num_epochs</span> <span class="o">=</span> <span class="n">num_epochs</span> |
| <span class="c1"># addl internal usage parameters and storage</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">lbmult</span> <span class="o">=</span> <span class="mi">1</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">cumgrads</span> <span class="o">=</span> <span class="p">{}</span> |
| <span class="c1"># for adaptive lr</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">adaptive</span> <span class="o">=</span> <span class="kc">False</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">admult</span> <span class="o">=</span> <span class="mi">1</span> <span class="c1"># adaptation constant</span> |
| |
| <span class="k">def</span> <span class="nf">create_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="n">momentum</span> <span class="o">=</span> <span class="kc">None</span> |
| <span class="n">weight_master_copy</span> <span class="o">=</span> <span class="kc">None</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">multi_precision</span> <span class="ow">and</span> <span class="n">weight</span><span class="o">.</span><span class="n">dtype</span> <span class="o">==</span> <span class="n">numpy</span><span class="o">.</span><span class="n">float16</span><span class="p">:</span> |
| <span class="n">weight_master_copy</span> <span class="o">=</span> <span class="n">array</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">ctx</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">numpy</span><span class="o">.</span><span class="n">float32</span><span class="p">)</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> <span class="o">!=</span> <span class="mf">0.0</span><span class="p">:</span> |
| <span class="n">momentum</span> <span class="o">=</span> <span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">numpy</span><span class="o">.</span><span class="n">float32</span><span class="p">,</span> |
| <span class="n">stype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">stype</span><span class="p">)</span> |
| <span class="k">return</span> <span class="p">(</span><span class="n">momentum</span><span class="p">,</span> <span class="n">weight_master_copy</span><span class="p">)</span> |
| <span class="k">if</span> <span class="n">weight</span><span class="o">.</span><span class="n">dtype</span> <span class="o">==</span> <span class="n">numpy</span><span class="o">.</span><span class="n">float16</span> <span class="ow">and</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">multi_precision</span><span class="p">:</span> |
| <span class="n">warnings</span><span class="o">.</span><span class="n">warn</span><span class="p">(</span><span class="s2">"Accumulating with float16 in optimizer can lead to "</span> |
| <span class="s2">"poor accuracy or slow convergence. "</span> |
| <span class="s2">"Consider using multi_precision=True option of the "</span> |
| <span class="s2">"SGD optimizer"</span><span class="p">)</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> <span class="o">!=</span> <span class="mf">0.0</span><span class="p">:</span> |
| <span class="n">momentum</span> <span class="o">=</span> <span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">dtype</span><span class="p">,</span> <span class="n">stype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">stype</span><span class="p">)</span> |
| <span class="k">return</span> <span class="n">momentum</span> |
| |
| <span class="k">def</span> <span class="nf">_get_lbmult</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">nup</span><span class="p">):</span> |
| <span class="sd">"""Returns lr scaling factor for large batch according to warmup schedule</span> |
| <span class="sd"> (to be implemented)</span> |
| <span class="sd"> """</span> |
| <span class="n">nwup</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">warmup_epochs</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">updates_per_epoch</span> |
| <span class="n">strategy</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">warmup_strategy</span> |
| <span class="n">maxmult</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">batch_scale</span><span class="p">)</span> |
| <span class="k">if</span> <span class="n">nup</span> <span class="o">>=</span> <span class="n">nwup</span><span class="p">:</span> |
| <span class="n">mult</span> <span class="o">=</span> <span class="n">maxmult</span> |
| <span class="k">elif</span> <span class="n">nwup</span> <span class="o"><=</span> <span class="mi">1</span><span class="p">:</span> |
| <span class="n">mult</span> <span class="o">=</span> <span class="mf">1.0</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="k">if</span> <span class="p">(</span><span class="n">strategy</span> <span class="o">==</span> <span class="s1">'linear'</span><span class="p">):</span> |
| <span class="n">mult</span> <span class="o">=</span> <span class="mf">1.0</span> <span class="o">+</span> <span class="p">(</span><span class="n">maxmult</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">*</span> <span class="n">nup</span> <span class="o">/</span> <span class="n">nwup</span> |
| <span class="k">elif</span> <span class="p">(</span><span class="n">strategy</span> <span class="o">==</span> <span class="s1">'power2'</span><span class="p">):</span> |
| <span class="n">mult</span> <span class="o">=</span> <span class="mf">1.0</span> <span class="o">+</span> <span class="p">(</span><span class="n">maxmult</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="o">*</span> <span class="p">(</span><span class="n">nup</span><span class="o">*</span><span class="n">nup</span><span class="p">)</span><span class="o">/</span><span class="p">(</span><span class="n">nwup</span><span class="o">*</span><span class="n">nwup</span><span class="p">)</span> |
| <span class="k">elif</span> <span class="p">(</span><span class="n">strategy</span> <span class="o">==</span> <span class="s1">'sqrt'</span><span class="p">):</span> |
| <span class="n">mult</span> <span class="o">=</span> <span class="mf">1.0</span> <span class="o">+</span> <span class="p">(</span><span class="n">maxmult</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="o">*</span> <span class="n">math</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">nup</span><span class="p">)</span> <span class="o">/</span> <span class="n">nwup</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="n">mult</span> <span class="o">=</span> <span class="mf">1.0</span> |
| <span class="k">return</span> <span class="n">mult</span> |
| |
| <span class="k">def</span> <span class="nf">_get_lars</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">g</span><span class="p">,</span> <span class="n">wd</span><span class="p">):</span> |
| <span class="sd">"""Returns a scaling factor for the learning rate for this layer</span> |
| <span class="sd"> default is 1</span> |
| <span class="sd"> """</span> |
| <span class="n">weight2</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_l2norm</span><span class="p">(</span><span class="n">weight</span><span class="p">)</span> |
| <span class="n">grad2</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_l2norm</span><span class="p">(</span><span class="n">g</span><span class="p">)</span> |
| <span class="n">lars</span> <span class="o">=</span> <span class="n">math</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">weight2</span> <span class="o">/</span> <span class="p">(</span><span class="n">grad2</span> <span class="o">+</span> <span class="n">wd</span> <span class="o">*</span> <span class="n">weight2</span> <span class="o">+</span> <span class="mf">1e-18</span><span class="p">))</span> |
| <span class="k">if</span> <span class="n">lars</span> <span class="o"><</span> <span class="mf">0.01</span><span class="p">:</span> |
| <span class="n">lars</span> <span class="o">=</span> <span class="mf">0.01</span> |
| <span class="k">elif</span> <span class="n">lars</span> <span class="o">></span> <span class="mi">100</span><span class="p">:</span> |
| <span class="n">lars</span> <span class="o">=</span> <span class="mi">100</span> |
| <span class="k">return</span> <span class="n">lars</span> |
| |
| <span class="k">def</span> <span class="nf">_l2norm</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">v</span><span class="p">):</span> |
| <span class="s2">"inner product implementation"</span> |
| <span class="n">norm</span> <span class="o">=</span> <span class="n">multiply</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span><span class="o">.</span><span class="n">asnumpy</span><span class="p">()</span><span class="o">.</span><span class="n">sum</span><span class="p">()</span> |
| <span class="k">return</span> <span class="n">norm</span> |
| |
| <span class="k">def</span> <span class="nf">_reset_cum_gradient</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">):</span> |
| <span class="s2">"called every macro-batch to reset cumulated gradients to 0 for a given index"</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">cumgrads</span><span class="p">[</span><span class="n">index</span><span class="p">][</span><span class="s1">'cum_grad'</span><span class="p">]</span> <span class="o">=</span> <span class="mi">0</span> |
| |
| <span class="k">def</span> <span class="nf">_get_cum_gradient</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">):</span> |
| <span class="s2">"get the cumulated gradient for index"</span> |
| <span class="k">if</span> <span class="n">index</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">cumgrads</span><span class="p">:</span> |
| <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">cumgrads</span><span class="p">[</span><span class="n">index</span><span class="p">]</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="k">return</span> <span class="p">{}</span> |
| |
| <span class="k">def</span> <span class="nf">_put_cum_gradient</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">cgrad</span><span class="p">):</span> |
| <span class="s2">"store cumulated gradient for index"</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">cumgrads</span><span class="p">[</span><span class="n">index</span><span class="p">]</span> <span class="o">=</span> <span class="n">cgrad</span> |
| |
| <span class="k">def</span> <span class="nf">_cumulate_gradient</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">index</span><span class="p">):</span> |
| <span class="s2">"Cumulate gradients for large-batch emulation. Cumulated by index (layer)"</span> |
| <span class="n">cgrad</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_cum_gradient</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="k">if</span> <span class="n">cgrad</span><span class="p">:</span> |
| <span class="n">num_cums</span> <span class="o">=</span> <span class="n">cgrad</span><span class="p">[</span><span class="s1">'num_cums'</span><span class="p">]</span> |
| <span class="k">if</span> <span class="n">num_cums</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span> |
| <span class="n">cum_grad</span> <span class="o">=</span> <span class="n">cgrad</span><span class="p">[</span><span class="s1">'cum_grad'</span><span class="p">]</span> <span class="o">+</span> <span class="n">grad</span> |
| <span class="n">num_cums</span> <span class="o">+=</span> <span class="mi">1</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="n">cum_grad</span> <span class="o">=</span> <span class="n">grad</span> |
| <span class="n">num_cums</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">init_updates</span> <span class="o">+</span> <span class="mi">1</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="n">cum_grad</span> <span class="o">=</span> <span class="n">grad</span> |
| <span class="n">num_cums</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">init_updates</span> <span class="o">+</span> <span class="mi">1</span> |
| <span class="n">cgrad</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'cum_grad'</span><span class="p">:</span> <span class="n">cum_grad</span><span class="p">,</span> <span class="s1">'num_cums'</span><span class="p">:</span> <span class="n">num_cums</span><span class="p">}</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_put_cum_gradient</span><span class="p">(</span><span class="n">index</span><span class="p">,</span> <span class="n">cgrad</span><span class="p">)</span> |
| <span class="k">return</span> <span class="n">cgrad</span> |
| |
| <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="k">assert</span> <span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="k">assert</span> <span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| |
| <span class="n">lr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_lr</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">wd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_wd</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_update_count</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| |
| <span class="c1"># new stuff for large batch</span> |
| <span class="n">cgrad</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_cumulate_gradient</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="n">index</span><span class="p">)</span> |
| <span class="k">if</span> <span class="p">(</span><span class="n">cgrad</span><span class="p">[</span><span class="s1">'num_cums'</span><span class="p">]</span> <span class="o">%</span> <span class="bp">self</span><span class="o">.</span><span class="n">batch_scale</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> |
| <span class="n">grad</span> <span class="o">=</span> <span class="n">cgrad</span><span class="p">[</span><span class="s1">'cum_grad'</span><span class="p">]</span> <span class="o">/</span> <span class="bp">self</span><span class="o">.</span><span class="n">batch_scale</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">warmup_strategy</span> <span class="o">==</span> <span class="s1">'lars'</span><span class="p">:</span> |
| <span class="n">lbmult</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_lars</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">wd</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="n">lbmult</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_lbmult</span><span class="p">(</span><span class="n">cgrad</span><span class="p">[</span><span class="s1">'num_cums'</span><span class="p">])</span> |
| <span class="n">lr</span> <span class="o">=</span> <span class="n">lr</span> <span class="o">*</span> <span class="n">lbmult</span> |
| <span class="c1"># do the regular sgd update flow</span> |
| <span class="n">kwargs</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'rescale_grad'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">rescale_grad</span><span class="p">}</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span> |
| <span class="n">kwargs</span><span class="p">[</span><span class="s1">'momentum'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">:</span> |
| <span class="n">kwargs</span><span class="p">[</span><span class="s1">'clip_gradient'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span> |
| <span class="n">use_multi_precision</span> <span class="o">=</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">state</span><span class="p">,</span> <span class="p">(</span><span class="nb">list</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">))</span> |
| |
| <span class="k">if</span> <span class="ow">not</span> <span class="n">use_multi_precision</span><span class="p">:</span> |
| <span class="k">if</span> <span class="n">state</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="n">sgd_mom_update</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span> <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="n">wd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="n">sgd_update</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span> <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="n">wd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="k">if</span> <span class="n">state</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="n">mp_sgd_mom_update</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">state</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span> <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="n">wd</span><span class="p">,</span> |
| <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="n">mp_sgd_update</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span> <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="n">wd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="c1"># reset update count and cumulated gradient per large batch</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_reset_cum_gradient</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="n">lr</span> <span class="o">=</span> <span class="mf">0.0</span> |
| <span class="n">kwargs</span> <span class="o">=</span> <span class="p">{}</span> |
| <span class="n">sgd_update</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span> <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="n">wd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span></div> |
| |
| <span class="c1"># pylint: enable=line-too-long</span> |
| <span class="nd">@register</span> |
| <div class="viewcode-block" id="DCASGD"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.DCASGD">[docs]</a><span class="k">class</span> <span class="nc">DCASGD</span><span class="p">(</span><span class="n">Optimizer</span><span class="p">):</span> |
| <span class="sd">"""The DCASGD optimizer.</span> |
| |
| <span class="sd"> This class implements the optimizer described in *Asynchronous Stochastic Gradient Descent</span> |
| <span class="sd"> with Delay Compensation for Distributed Deep Learning*,</span> |
| <span class="sd"> available at https://arxiv.org/abs/1609.08326.</span> |
| |
| <span class="sd"> This optimizer accepts the following parameters in addition to those accepted</span> |
| <span class="sd"> by :class:`.Optimizer`.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> momentum : float, optional</span> |
| <span class="sd"> The momentum value.</span> |
| |
| <span class="sd"> lamda : float, optional</span> |
| <span class="sd"> Scale DC value.</span> |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">momentum</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">lamda</span><span class="o">=</span><span class="mf">0.04</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">DCASGD</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> <span class="o">=</span> <span class="n">momentum</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">weight_previous</span> <span class="o">=</span> <span class="p">{}</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">lamda</span> <span class="o">=</span> <span class="n">lamda</span> |
| |
| <span class="k">def</span> <span class="nf">create_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> <span class="o">==</span> <span class="mf">0.0</span><span class="p">:</span> |
| <span class="k">return</span> <span class="p">(</span><span class="kc">None</span><span class="p">,</span> |
| <span class="n">weight</span><span class="o">.</span><span class="n">copy</span><span class="p">())</span> <span class="c1"># previous weight</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="k">return</span> <span class="p">(</span><span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">dtype</span><span class="p">),</span> <span class="c1"># momentum</span> |
| <span class="n">weight</span><span class="o">.</span><span class="n">copy</span><span class="p">())</span> <span class="c1"># previous weight</span> |
| |
| <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_update_count</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">lr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_lr</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">wd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_wd</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| |
| <span class="n">grad</span> <span class="o">=</span> <span class="n">grad</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">rescale_grad</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="n">grad</span> <span class="o">=</span> <span class="n">clip</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="o">-</span><span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">)</span> |
| |
| <span class="n">mom</span><span class="p">,</span> <span class="n">previous_weight</span> <span class="o">=</span> <span class="n">state</span> |
| <span class="k">if</span> <span class="n">mom</span><span class="p">:</span> |
| <span class="n">mom</span><span class="p">[:]</span> <span class="o">*=</span> <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> |
| <span class="n">mom</span><span class="p">[:]</span> <span class="o">+=</span> <span class="o">-</span><span class="n">lr</span> <span class="o">*</span> <span class="p">(</span><span class="n">grad</span> <span class="o">+</span> <span class="n">wd</span> <span class="o">*</span> <span class="n">weight</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">lamda</span> \ |
| <span class="o">*</span> <span class="n">grad</span> <span class="o">*</span> <span class="n">grad</span> <span class="o">*</span> <span class="p">(</span><span class="n">weight</span> <span class="o">-</span> <span class="n">previous_weight</span><span class="p">))</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="k">assert</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> <span class="o">==</span> <span class="mf">0.0</span><span class="p">)</span> |
| <span class="n">mom</span> <span class="o">=</span> <span class="o">-</span><span class="n">lr</span> <span class="o">*</span> <span class="p">(</span><span class="n">grad</span> <span class="o">+</span> <span class="n">wd</span> <span class="o">*</span> <span class="n">weight</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">lamda</span> \ |
| <span class="o">*</span> <span class="n">grad</span> <span class="o">*</span> <span class="n">grad</span> <span class="o">*</span> <span class="p">(</span><span class="n">weight</span> <span class="o">-</span> <span class="n">previous_weight</span><span class="p">))</span> |
| <span class="n">previous_weight</span><span class="p">[:]</span> <span class="o">=</span> <span class="n">weight</span> |
| <span class="n">weight</span><span class="p">[:]</span> <span class="o">+=</span> <span class="n">mom</span></div> |
| |
| <span class="nd">@register</span> |
| <div class="viewcode-block" id="NAG"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.NAG">[docs]</a><span class="k">class</span> <span class="nc">NAG</span><span class="p">(</span><span class="n">Optimizer</span><span class="p">):</span> |
| <span class="sd">"""Nesterov accelerated SGD.</span> |
| |
| <span class="sd"> This optimizer updates each weight by::</span> |
| |
| <span class="sd"> state = momentum * state + grad + wd * weight</span> |
| <span class="sd"> weight = weight - (lr * (grad + momentum * state))</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> momentum : float, optional</span> |
| <span class="sd"> The momentum value.</span> |
| <span class="sd"> multi_precision: bool, optional</span> |
| <span class="sd"> Flag to control the internal precision of the optimizer.</span> |
| <span class="sd"> ``False`` results in using the same precision as the weights (default),</span> |
| <span class="sd"> ``True`` makes internal 32-bit copy of the weights and applies gradients \</span> |
| <span class="sd"> in 32-bit precision even if actual weights used in the model have lower precision.\</span> |
| <span class="sd"> Turning this on can improve convergence and accuracy when training with float16.</span> |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">momentum</span><span class="o">=</span><span class="mf">0.0</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">NAG</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> <span class="o">=</span> <span class="n">momentum</span> |
| |
| <span class="k">def</span> <span class="nf">create_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="n">momentum</span> <span class="o">=</span> <span class="kc">None</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> <span class="o">!=</span> <span class="mf">0.0</span><span class="p">:</span> |
| <span class="n">momentum</span> <span class="o">=</span> <span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">dtype</span><span class="p">)</span> |
| <span class="k">return</span> <span class="n">momentum</span> |
| |
| <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_update_count</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">lr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_lr</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">wd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_wd</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| |
| <span class="n">grad</span> <span class="o">=</span> <span class="n">grad</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">rescale_grad</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="n">grad</span> <span class="o">=</span> <span class="n">clip</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="o">-</span><span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">)</span> |
| |
| <span class="k">if</span> <span class="n">state</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="n">mom</span> <span class="o">=</span> <span class="n">state</span> |
| <span class="n">mom</span><span class="p">[:]</span> <span class="o">*=</span> <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> |
| <span class="n">grad</span> <span class="o">+=</span> <span class="n">wd</span> <span class="o">*</span> <span class="n">weight</span> |
| <span class="n">mom</span><span class="p">[:]</span> <span class="o">+=</span> <span class="n">grad</span> |
| <span class="n">grad</span><span class="p">[:]</span> <span class="o">+=</span> <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> <span class="o">*</span> <span class="n">mom</span> |
| <span class="n">weight</span><span class="p">[:]</span> <span class="o">+=</span> <span class="o">-</span><span class="n">lr</span> <span class="o">*</span> <span class="n">grad</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="k">assert</span> <span class="bp">self</span><span class="o">.</span><span class="n">momentum</span> <span class="o">==</span> <span class="mf">0.0</span> |
| <span class="n">weight</span><span class="p">[:]</span> <span class="o">+=</span> <span class="o">-</span><span class="n">lr</span> <span class="o">*</span> <span class="p">(</span><span class="n">grad</span> <span class="o">+</span> <span class="n">wd</span> <span class="o">*</span> <span class="n">weight</span><span class="p">)</span></div> |
| |
| <span class="nd">@register</span> |
| <div class="viewcode-block" id="SGLD"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.SGLD">[docs]</a><span class="k">class</span> <span class="nc">SGLD</span><span class="p">(</span><span class="n">Optimizer</span><span class="p">):</span> |
| <span class="sd">"""Stochastic Gradient Riemannian Langevin Dynamics.</span> |
| |
| <span class="sd"> This class implements the optimizer described in the paper *Stochastic Gradient</span> |
| <span class="sd"> Riemannian Langevin Dynamics on the Probability Simplex*, available at</span> |
| <span class="sd"> https://papers.nips.cc/paper/4883-stochastic-gradient-riemannian-langevin-dynamics-on-the-probability-simplex.pdf.</span> |
| |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">SGLD</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| |
| <span class="k">def</span> <span class="nf">create_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="k">return</span> <span class="kc">None</span> |
| |
| <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_update_count</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">lr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_lr</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">wd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_wd</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| |
| <span class="n">grad</span> <span class="o">=</span> <span class="n">grad</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">rescale_grad</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="n">grad</span> <span class="o">=</span> <span class="n">clip</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="o">-</span><span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">)</span> |
| <span class="n">weight</span><span class="p">[:]</span> <span class="o">+=</span> <span class="o">-</span> <span class="n">lr</span><span class="o">/</span><span class="mi">2</span> <span class="o">*</span> <span class="p">(</span><span class="n">grad</span> <span class="o">+</span> <span class="n">wd</span> <span class="o">*</span> <span class="n">weight</span><span class="p">)</span> <span class="o">+</span> <span class="n">normal</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">math</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">lr</span><span class="p">),</span> <span class="n">shape</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> |
| <span class="n">dtype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">dtype</span><span class="p">,</span> <span class="n">ctx</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">)</span></div> |
| |
| |
| |
| <span class="nd">@register</span> <span class="c1"># pylint: disable=invalid-name</span> |
| <div class="viewcode-block" id="ccSGD"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.ccSGD">[docs]</a><span class="k">class</span> <span class="nc">ccSGD</span><span class="p">(</span><span class="n">SGD</span><span class="p">):</span> |
| <span class="sd">"""[DEPRECATED] Same as `SGD`. Left here for backward compatibility."""</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">ccSGD</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span></div> |
| |
| <span class="nd">@register</span> |
| <div class="viewcode-block" id="Adam"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Adam">[docs]</a><span class="k">class</span> <span class="nc">Adam</span><span class="p">(</span><span class="n">Optimizer</span><span class="p">):</span> |
| <span class="sd">"""The Adam optimizer.</span> |
| |
| <span class="sd"> This class implements the optimizer described in *Adam: A Method for</span> |
| <span class="sd"> Stochastic Optimization*, available at http://arxiv.org/abs/1412.6980.</span> |
| |
| <span class="sd"> If the storage types of grad is ``row_sparse``, and ``lazy_update`` is True, \</span> |
| <span class="sd"> **lazy updates** are applied by::</span> |
| |
| <span class="sd"> for row in grad.indices:</span> |
| <span class="sd"> rescaled_grad[row] = clip(grad[row] * rescale_grad + wd * weight[row], clip_gradient)</span> |
| <span class="sd"> m[row] = beta1 * m[row] + (1 - beta1) * rescaled_grad[row]</span> |
| <span class="sd"> v[row] = beta2 * v[row] + (1 - beta2) * (rescaled_grad[row]**2)</span> |
| <span class="sd"> w[row] = w[row] - learning_rate * m[row] / (sqrt(v[row]) + epsilon)</span> |
| |
| <span class="sd"> The lazy update only updates the mean and var for the weights whose row_sparse</span> |
| <span class="sd"> gradient indices appear in the current batch, rather than updating it for all indices.</span> |
| <span class="sd"> Compared with the original update, it can provide large improvements in model training</span> |
| <span class="sd"> throughput for some applications. However, it provides slightly different semantics than</span> |
| <span class="sd"> the original update, and may lead to different empirical results.</span> |
| |
| <span class="sd"> Otherwise, **standard updates** are applied by::</span> |
| |
| <span class="sd"> rescaled_grad = clip(grad * rescale_grad + wd * weight, clip_gradient)</span> |
| <span class="sd"> m = beta1 * m + (1 - beta1) * rescaled_grad</span> |
| <span class="sd"> v = beta2 * v + (1 - beta2) * (rescaled_grad**2)</span> |
| <span class="sd"> w = w - learning_rate * m / (sqrt(v) + epsilon)</span> |
| |
| <span class="sd"> This optimizer accepts the following parameters in addition to those accepted</span> |
| <span class="sd"> by :class:`.Optimizer`.</span> |
| |
| <span class="sd"> For details of the update algorithm, see :class:`~mxnet.ndarray.adam_update`.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> beta1 : float, optional</span> |
| <span class="sd"> Exponential decay rate for the first moment estimates.</span> |
| <span class="sd"> beta2 : float, optional</span> |
| <span class="sd"> Exponential decay rate for the second moment estimates.</span> |
| <span class="sd"> epsilon : float, optional</span> |
| <span class="sd"> Small value to avoid division by 0.</span> |
| <span class="sd"> lazy_update : bool, optional</span> |
| <span class="sd"> Default is True. If True, lazy updates are applied \</span> |
| <span class="sd"> if the storage types of weight and grad are both ``row_sparse``.</span> |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.001</span><span class="p">,</span> <span class="n">beta1</span><span class="o">=</span><span class="mf">0.9</span><span class="p">,</span> <span class="n">beta2</span><span class="o">=</span><span class="mf">0.999</span><span class="p">,</span> <span class="n">epsilon</span><span class="o">=</span><span class="mf">1e-8</span><span class="p">,</span> |
| <span class="n">lazy_update</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">Adam</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">learning_rate</span><span class="o">=</span><span class="n">learning_rate</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">beta1</span> <span class="o">=</span> <span class="n">beta1</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">beta2</span> <span class="o">=</span> <span class="n">beta2</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">epsilon</span> <span class="o">=</span> <span class="n">epsilon</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">lazy_update</span> <span class="o">=</span> <span class="n">lazy_update</span> |
| |
| <span class="k">def</span> <span class="nf">create_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="n">stype</span> <span class="o">=</span> <span class="n">weight</span><span class="o">.</span><span class="n">stype</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">lazy_update</span> <span class="k">else</span> <span class="s1">'default'</span> |
| <span class="k">return</span> <span class="p">(</span><span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">dtype</span><span class="p">,</span> |
| <span class="n">stype</span><span class="o">=</span><span class="n">stype</span><span class="p">),</span> <span class="c1"># mean</span> |
| <span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">dtype</span><span class="p">,</span> |
| <span class="n">stype</span><span class="o">=</span><span class="n">stype</span><span class="p">))</span> <span class="c1"># variance</span> |
| |
| <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_update_count</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">lr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_lr</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">wd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_wd</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| |
| <span class="n">t</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_index_update_count</span><span class="p">[</span><span class="n">index</span><span class="p">]</span> |
| <span class="n">coef1</span> <span class="o">=</span> <span class="mf">1.</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">beta1</span><span class="o">**</span><span class="n">t</span> |
| <span class="n">coef2</span> <span class="o">=</span> <span class="mf">1.</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">beta2</span><span class="o">**</span><span class="n">t</span> |
| <span class="n">lr</span> <span class="o">*=</span> <span class="n">math</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">coef2</span><span class="p">)</span><span class="o">/</span><span class="n">coef1</span> |
| |
| <span class="n">kwargs</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'beta1'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">beta1</span><span class="p">,</span> <span class="s1">'beta2'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">beta2</span><span class="p">,</span> <span class="s1">'epsilon'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">epsilon</span><span class="p">,</span> |
| <span class="s1">'rescale_grad'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">rescale_grad</span><span class="p">}</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">:</span> |
| <span class="n">kwargs</span><span class="p">[</span><span class="s1">'clip_gradient'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span> |
| |
| <span class="n">mean</span><span class="p">,</span> <span class="n">var</span> <span class="o">=</span> <span class="n">state</span> |
| <span class="n">adam_update</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">mean</span><span class="p">,</span> <span class="n">var</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span> |
| <span class="n">lazy_update</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">lazy_update</span><span class="p">,</span> <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="n">wd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span></div> |
| |
| <span class="nd">@register</span> |
| <div class="viewcode-block" id="AdaGrad"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.AdaGrad">[docs]</a><span class="k">class</span> <span class="nc">AdaGrad</span><span class="p">(</span><span class="n">Optimizer</span><span class="p">):</span> |
| <span class="sd">"""AdaGrad optimizer.</span> |
| |
| <span class="sd"> This class implements the AdaGrad optimizer described in *Adaptive Subgradient</span> |
| <span class="sd"> Methods for Online Learning and Stochastic Optimization*, and available at</span> |
| <span class="sd"> http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf.</span> |
| |
| <span class="sd"> This optimizer updates each weight by::</span> |
| |
| <span class="sd"> grad = clip(grad * rescale_grad, clip_gradient)</span> |
| <span class="sd"> history += square(grad)</span> |
| <span class="sd"> div = grad / sqrt(history + float_stable_eps)</span> |
| <span class="sd"> weight += (div + weight * wd) * -lr</span> |
| |
| <span class="sd"> This optimizer accepts the following parameters in addition to those accepted</span> |
| <span class="sd"> by :class:`.Optimizer`.</span> |
| |
| <span class="sd"> See Also</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> :meth:`mxnet.ndarray.sparse.adagrad_update`.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> eps: float, optional</span> |
| <span class="sd"> Initial value of the history accumulator. Avoids division by 0.</span> |
| |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">eps</span><span class="o">=</span><span class="mf">1e-7</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">AdaGrad</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">float_stable_eps</span> <span class="o">=</span> <span class="n">eps</span> |
| |
| <span class="k">def</span> <span class="nf">create_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="k">return</span> <span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">stype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">stype</span><span class="p">)</span> <span class="c1"># history</span> |
| |
| <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_update_count</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">lr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_lr</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">wd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_wd</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| |
| <span class="n">is_sparse</span> <span class="o">=</span> <span class="n">grad</span><span class="o">.</span><span class="n">stype</span> <span class="o">==</span> <span class="s1">'row_sparse'</span> |
| <span class="n">history</span> <span class="o">=</span> <span class="n">state</span> |
| |
| <span class="k">if</span> <span class="n">is_sparse</span><span class="p">:</span> |
| <span class="n">kwargs</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'epsilon'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">float_stable_eps</span><span class="p">,</span> |
| <span class="s1">'rescale_grad'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">rescale_grad</span><span class="p">}</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">:</span> |
| <span class="n">kwargs</span><span class="p">[</span><span class="s1">'clip_gradient'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span> |
| <span class="n">sparse</span><span class="o">.</span><span class="n">adagrad_update</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">history</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span> <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="n">wd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="n">grad</span> <span class="o">=</span> <span class="n">grad</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">rescale_grad</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="n">grad</span> <span class="o">=</span> <span class="n">clip</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="o">-</span><span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">)</span> |
| <span class="n">history</span><span class="p">[:]</span> <span class="o">+=</span> <span class="n">square</span><span class="p">(</span><span class="n">grad</span><span class="p">)</span> |
| <span class="n">div</span> <span class="o">=</span> <span class="n">grad</span> <span class="o">/</span> <span class="n">sqrt</span><span class="p">(</span><span class="n">history</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">float_stable_eps</span><span class="p">)</span> |
| <span class="n">weight</span><span class="p">[:]</span> <span class="o">+=</span> <span class="p">(</span><span class="n">div</span> <span class="o">+</span> <span class="n">weight</span> <span class="o">*</span> <span class="n">wd</span><span class="p">)</span> <span class="o">*</span> <span class="o">-</span><span class="n">lr</span></div> |
| |
| <span class="nd">@register</span> |
| <div class="viewcode-block" id="RMSProp"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.RMSProp">[docs]</a><span class="k">class</span> <span class="nc">RMSProp</span><span class="p">(</span><span class="n">Optimizer</span><span class="p">):</span> |
| <span class="sd">"""The RMSProp optimizer.</span> |
| |
| <span class="sd"> Two versions of RMSProp are implemented:</span> |
| |
| <span class="sd"> If ``centered=False``, we follow</span> |
| <span class="sd"> http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf by</span> |
| <span class="sd"> Tieleman & Hinton, 2012.</span> |
| <span class="sd"> For details of the update algorithm see :class:`~mxnet.ndarray.rmsprop_update`.</span> |
| |
| <span class="sd"> If ``centered=True``, we follow http://arxiv.org/pdf/1308.0850v5.pdf (38)-(45)</span> |
| <span class="sd"> by Alex Graves, 2013.</span> |
| <span class="sd"> For details of the update algorithm see :class:`~mxnet.ndarray.rmspropalex_update`.</span> |
| |
| <span class="sd"> This optimizer accepts the following parameters in addition to those accepted</span> |
| <span class="sd"> by :class:`.Optimizer`.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> gamma1: float, optional</span> |
| <span class="sd"> A decay factor of moving average over past squared gradient.</span> |
| <span class="sd"> gamma2: float, optional</span> |
| <span class="sd"> A "momentum" factor. Only used if `centered`=``True``.</span> |
| <span class="sd"> epsilon : float, optional</span> |
| <span class="sd"> Small value to avoid division by 0.</span> |
| <span class="sd"> centered : bool, optional</span> |
| <span class="sd"> Flag to control which version of RMSProp to use.</span> |
| <span class="sd"> ``True`` will use Graves's version of `RMSProp`,</span> |
| <span class="sd"> ``False`` will use Tieleman & Hinton's version of `RMSProp`.</span> |
| <span class="sd"> clip_weights : float, optional</span> |
| <span class="sd"> Clips weights into range ``[-clip_weights, clip_weights]``.</span> |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.001</span><span class="p">,</span> <span class="n">gamma1</span><span class="o">=</span><span class="mf">0.9</span><span class="p">,</span> <span class="n">gamma2</span><span class="o">=</span><span class="mf">0.9</span><span class="p">,</span> |
| <span class="n">epsilon</span><span class="o">=</span><span class="mf">1e-8</span><span class="p">,</span> <span class="n">centered</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">clip_weights</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">RMSProp</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">learning_rate</span><span class="o">=</span><span class="n">learning_rate</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">gamma1</span> <span class="o">=</span> <span class="n">gamma1</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">gamma2</span> <span class="o">=</span> <span class="n">gamma2</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">centered</span> <span class="o">=</span> <span class="n">centered</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">epsilon</span> <span class="o">=</span> <span class="n">epsilon</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">clip_weights</span> <span class="o">=</span> <span class="n">clip_weights</span> |
| |
| <span class="k">def</span> <span class="nf">create_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">centered</span><span class="p">:</span> |
| <span class="k">return</span> <span class="p">(</span> |
| <span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">stype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">stype</span><span class="p">),</span> <span class="c1"># n</span> |
| <span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">stype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">stype</span><span class="p">),</span> <span class="c1"># g</span> |
| <span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">stype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">stype</span><span class="p">))</span> <span class="c1"># delta</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="k">return</span> <span class="p">(</span><span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">stype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">stype</span><span class="p">),)</span> <span class="c1"># n</span> |
| |
| <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_update_count</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">lr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_lr</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">wd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_wd</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| |
| <span class="n">kwargs</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'gamma1'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">gamma1</span><span class="p">,</span> <span class="s1">'epsilon'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">epsilon</span><span class="p">,</span> |
| <span class="s1">'rescale_grad'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">rescale_grad</span><span class="p">}</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">centered</span><span class="p">:</span> |
| <span class="n">kwargs</span><span class="p">[</span><span class="s1">'gamma2'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">gamma2</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">:</span> |
| <span class="n">kwargs</span><span class="p">[</span><span class="s1">'clip_gradient'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_weights</span><span class="p">:</span> |
| <span class="n">kwargs</span><span class="p">[</span><span class="s1">'clip_weights'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_weights</span> |
| |
| <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">centered</span><span class="p">:</span> |
| <span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="p">)</span> <span class="o">=</span> <span class="n">state</span> |
| <span class="n">rmsprop_update</span><span class="p">(</span> |
| <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span> <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="n">wd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="n">n</span><span class="p">,</span> <span class="n">g</span><span class="p">,</span> <span class="n">delta</span> <span class="o">=</span> <span class="n">state</span> |
| <span class="n">rmspropalex_update</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">g</span><span class="p">,</span> <span class="n">delta</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span> |
| <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="n">wd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span></div> |
| |
| <span class="nd">@register</span> |
| <div class="viewcode-block" id="AdaDelta"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.AdaDelta">[docs]</a><span class="k">class</span> <span class="nc">AdaDelta</span><span class="p">(</span><span class="n">Optimizer</span><span class="p">):</span> |
| <span class="sd">"""The AdaDelta optimizer.</span> |
| |
| <span class="sd"> This class implements AdaDelta, an optimizer described in *ADADELTA: An adaptive</span> |
| <span class="sd"> learning rate method*, available at https://arxiv.org/abs/1212.5701.</span> |
| |
| <span class="sd"> This optimizer updates each weight by::</span> |
| |
| <span class="sd"> grad = clip(grad * rescale_grad + wd * weight, clip_gradient)</span> |
| <span class="sd"> acc_grad = rho * acc_grad + (1. - rho) * grad * grad</span> |
| <span class="sd"> delta = sqrt(acc_delta + epsilon) / sqrt(acc_grad + epsilon) * grad</span> |
| <span class="sd"> acc_delta = rho * acc_delta + (1. - rho) * delta * delta</span> |
| <span class="sd"> weight -= (delta + wd * weight)</span> |
| |
| <span class="sd"> This optimizer accepts the following parameters in addition to those accepted</span> |
| <span class="sd"> by :class:`.Optimizer`.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> rho: float</span> |
| <span class="sd"> Decay rate for both squared gradients and delta.</span> |
| <span class="sd"> epsilon : float</span> |
| <span class="sd"> Small value to avoid division by 0.</span> |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">rho</span><span class="o">=</span><span class="mf">0.90</span><span class="p">,</span> <span class="n">epsilon</span><span class="o">=</span><span class="mf">1e-5</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">AdaDelta</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">rho</span> <span class="o">=</span> <span class="n">rho</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">epsilon</span> <span class="o">=</span> <span class="n">epsilon</span> |
| |
| <span class="k">def</span> <span class="nf">create_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="k">return</span> <span class="p">(</span><span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">),</span> <span class="c1"># accumulated g</span> |
| <span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">))</span> <span class="c1"># accumulated delta</span> |
| |
| <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="n">wd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_wd</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_update_count</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| |
| <span class="c1"># preprocess grad</span> |
| <span class="n">grad</span> <span class="o">*=</span> <span class="bp">self</span><span class="o">.</span><span class="n">rescale_grad</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="n">grad</span> <span class="o">=</span> <span class="n">clip</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="o">-</span><span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">)</span> |
| |
| <span class="c1"># accumulated g and delta initlization</span> |
| <span class="n">acc_g</span><span class="p">,</span> <span class="n">acc_delta</span> <span class="o">=</span> <span class="n">state</span> |
| |
| <span class="c1"># update g, delta</span> |
| <span class="n">acc_g</span><span class="p">[:]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">rho</span> <span class="o">*</span> <span class="n">acc_g</span> <span class="o">+</span> <span class="p">(</span><span class="mf">1.</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">rho</span><span class="p">)</span> <span class="o">*</span> <span class="n">grad</span> <span class="o">*</span> <span class="n">grad</span> |
| <span class="n">current_delta</span> <span class="o">=</span> <span class="n">sqrt</span><span class="p">(</span><span class="n">acc_delta</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">epsilon</span><span class="p">)</span> <span class="o">/</span> <span class="n">sqrt</span><span class="p">(</span><span class="n">acc_g</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">epsilon</span><span class="p">)</span> <span class="o">*</span> <span class="n">grad</span> |
| <span class="n">acc_delta</span><span class="p">[:]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">rho</span> <span class="o">*</span> <span class="n">acc_delta</span> <span class="o">+</span> <span class="p">(</span><span class="mf">1.</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">rho</span><span class="p">)</span> <span class="o">*</span> <span class="n">current_delta</span> <span class="o">*</span> <span class="n">current_delta</span> |
| |
| <span class="c1"># update weight</span> |
| <span class="n">weight</span><span class="p">[:]</span> <span class="o">-=</span> <span class="n">current_delta</span> <span class="o">+</span> <span class="n">wd</span> <span class="o">*</span> <span class="n">weight</span></div> |
| |
| <span class="c1">#pylint: disable=invalid-name</span> |
| <span class="c1">#pylint: disable=line-too-long</span> |
| <span class="nd">@register</span> |
| <div class="viewcode-block" id="Ftrl"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Ftrl">[docs]</a><span class="k">class</span> <span class="nc">Ftrl</span><span class="p">(</span><span class="n">Optimizer</span><span class="p">):</span> |
| <span class="sd">"""The Ftrl optimizer.</span> |
| |
| <span class="sd"> Referenced from *Ad Click Prediction: a View from the Trenches*, available at</span> |
| <span class="sd"> http://dl.acm.org/citation.cfm?id=2488200.</span> |
| |
| <span class="sd"> eta :</span> |
| <span class="sd"> .. math::</span> |
| <span class="sd"> \\eta_{t,i} = \\frac{learningrate}{\\beta+\\sqrt{\\sum_{s=1}^tg_{s,i}^2}}</span> |
| |
| <span class="sd"> The optimizer updates the weight by::</span> |
| |
| <span class="sd"> rescaled_grad = clip(grad * rescale_grad, clip_gradient)</span> |
| <span class="sd"> z += rescaled_grad - (sqrt(n + rescaled_grad**2) - sqrt(n)) * weight / learning_rate</span> |
| <span class="sd"> n += rescaled_grad**2</span> |
| <span class="sd"> w = (sign(z) * lamda1 - z) / ((beta + sqrt(n)) / learning_rate + wd) * (abs(z) > lamda1)</span> |
| |
| <span class="sd"> If the storage types of weight, state and grad are all ``row_sparse``, \</span> |
| <span class="sd"> **sparse updates** are applied by::</span> |
| |
| <span class="sd"> for row in grad.indices:</span> |
| <span class="sd"> rescaled_grad[row] = clip(grad[row] * rescale_grad, clip_gradient)</span> |
| <span class="sd"> z[row] += rescaled_grad[row] - (sqrt(n[row] + rescaled_grad[row]**2) - sqrt(n[row])) * weight[row] / learning_rate</span> |
| <span class="sd"> n[row] += rescaled_grad[row]**2</span> |
| <span class="sd"> w[row] = (sign(z[row]) * lamda1 - z[row]) / ((beta + sqrt(n[row])) / learning_rate + wd) * (abs(z[row]) > lamda1)</span> |
| |
| <span class="sd"> The sparse update only updates the z and n for the weights whose row_sparse</span> |
| <span class="sd"> gradient indices appear in the current batch, rather than updating it for all</span> |
| <span class="sd"> indices. Compared with the original update, it can provide large</span> |
| <span class="sd"> improvements in model training throughput for some applications. However, it</span> |
| <span class="sd"> provides slightly different semantics than the original update, and</span> |
| <span class="sd"> may lead to different empirical results.</span> |
| |
| <span class="sd"> For details of the update algorithm, see :class:`~mxnet.ndarray.ftrl_update`.</span> |
| |
| <span class="sd"> This optimizer accepts the following parameters in addition to those accepted</span> |
| <span class="sd"> by :class:`.Optimizer`.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> lamda1 : float, optional</span> |
| <span class="sd"> L1 regularization coefficient.</span> |
| <span class="sd"> learning_rate : float, optional</span> |
| <span class="sd"> The initial learning rate.</span> |
| <span class="sd"> beta : float, optional</span> |
| <span class="sd"> Per-coordinate learning rate correlation parameter.</span> |
| <span class="sd"> """</span> |
| |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">lamda1</span><span class="o">=</span><span class="mf">0.01</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.1</span><span class="p">,</span> <span class="n">beta</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">Ftrl</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">lamda1</span> <span class="o">=</span> <span class="n">lamda1</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">beta</span> <span class="o">=</span> <span class="n">beta</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">lr</span> <span class="o">=</span> <span class="n">learning_rate</span> |
| |
| <span class="k">def</span> <span class="nf">create_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="k">return</span> <span class="p">(</span><span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">stype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">stype</span><span class="p">),</span> <span class="c1"># z</span> |
| <span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">stype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">stype</span><span class="p">))</span> <span class="c1"># n</span> |
| |
| <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_update_count</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">wd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_wd</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">lr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_lr</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| |
| <span class="n">kwargs</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'lamda1'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">lamda1</span><span class="p">,</span> <span class="s1">'beta'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">beta</span><span class="p">,</span> <span class="s1">'rescale_grad'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">rescale_grad</span><span class="p">}</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">:</span> |
| <span class="n">kwargs</span><span class="p">[</span><span class="s1">'clip_gradient'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span> |
| |
| <span class="c1"># accumulated g and delta initialization</span> |
| <span class="n">z</span><span class="p">,</span> <span class="n">n</span> <span class="o">=</span> <span class="n">state</span> |
| <span class="n">ftrl_update</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">z</span><span class="p">,</span> <span class="n">n</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="n">weight</span><span class="p">,</span> |
| <span class="n">lr</span><span class="o">=</span><span class="n">lr</span><span class="p">,</span> <span class="n">wd</span><span class="o">=</span><span class="n">wd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span></div> |
| |
| <span class="c1"># pylint: enable=line-too-long</span> |
| <span class="nd">@register</span> |
| <div class="viewcode-block" id="Adamax"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Adamax">[docs]</a><span class="k">class</span> <span class="nc">Adamax</span><span class="p">(</span><span class="n">Optimizer</span><span class="p">):</span> |
| <span class="sd">"""The AdaMax optimizer.</span> |
| |
| <span class="sd"> It is a variant of Adam based on the infinity norm</span> |
| <span class="sd"> available at http://arxiv.org/abs/1412.6980 Section 7.</span> |
| |
| <span class="sd"> The optimizer updates the weight by::</span> |
| |
| <span class="sd"> grad = clip(grad * rescale_grad + wd * weight, clip_gradient)</span> |
| <span class="sd"> m = beta1 * m_t + (1 - beta1) * grad</span> |
| <span class="sd"> u = maximum(beta2 * u, abs(grad))</span> |
| <span class="sd"> weight -= lr / (1 - beta1**t) * m / u</span> |
| |
| <span class="sd"> This optimizer accepts the following parameters in addition to those accepted</span> |
| <span class="sd"> by :class:`.Optimizer`.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> beta1 : float, optional</span> |
| <span class="sd"> Exponential decay rate for the first moment estimates.</span> |
| <span class="sd"> beta2 : float, optional</span> |
| <span class="sd"> Exponential decay rate for the second moment estimates.</span> |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.002</span><span class="p">,</span> <span class="n">beta1</span><span class="o">=</span><span class="mf">0.9</span><span class="p">,</span> <span class="n">beta2</span><span class="o">=</span><span class="mf">0.999</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">Adamax</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">learning_rate</span><span class="o">=</span><span class="n">learning_rate</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">beta1</span> <span class="o">=</span> <span class="n">beta1</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">beta2</span> <span class="o">=</span> <span class="n">beta2</span> |
| |
| <span class="k">def</span> <span class="nf">create_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="k">return</span> <span class="p">(</span><span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">dtype</span><span class="p">),</span> <span class="c1"># mean</span> |
| <span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">dtype</span><span class="p">))</span> <span class="c1"># variance</span> |
| |
| <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_update_count</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">lr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_lr</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">wd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_wd</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| |
| <span class="n">t</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_index_update_count</span><span class="p">[</span><span class="n">index</span><span class="p">]</span> |
| <span class="n">lr</span> <span class="o">/=</span> <span class="p">(</span><span class="mf">1.</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">beta1</span><span class="o">**</span><span class="n">t</span><span class="p">)</span> |
| |
| <span class="c1"># preprocess grad</span> |
| <span class="n">grad</span> <span class="o">=</span> <span class="n">grad</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">rescale_grad</span> <span class="o">+</span> <span class="n">wd</span> <span class="o">*</span> <span class="n">weight</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="n">grad</span> <span class="o">=</span> <span class="n">clip</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="o">-</span><span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">)</span> |
| |
| <span class="c1"># update m_t and u_t</span> |
| <span class="n">m_t</span><span class="p">,</span> <span class="n">u_t</span> <span class="o">=</span> <span class="n">state</span> |
| <span class="n">m_t</span><span class="p">[:]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">beta1</span> <span class="o">*</span> <span class="n">m_t</span> <span class="o">+</span> <span class="p">(</span><span class="mf">1.</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">beta1</span><span class="p">)</span> <span class="o">*</span> <span class="n">grad</span> |
| <span class="n">u_t</span><span class="p">[:]</span> <span class="o">=</span> <span class="n">maximum</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">beta2</span> <span class="o">*</span> <span class="n">u_t</span><span class="p">,</span> <span class="n">NDabs</span><span class="p">(</span><span class="n">grad</span><span class="p">))</span> |
| |
| <span class="c1"># update weight</span> |
| <span class="n">weight</span><span class="p">[:]</span> <span class="o">-=</span> <span class="n">lr</span> <span class="o">*</span> <span class="n">m_t</span> <span class="o">/</span> <span class="n">u_t</span></div> |
| |
| <span class="nd">@register</span> |
| <div class="viewcode-block" id="Nadam"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Nadam">[docs]</a><span class="k">class</span> <span class="nc">Nadam</span><span class="p">(</span><span class="n">Optimizer</span><span class="p">):</span> |
| <span class="sd">"""The Nesterov Adam optimizer.</span> |
| |
| <span class="sd"> Much like Adam is essentially RMSprop with momentum,</span> |
| <span class="sd"> Nadam is Adam RMSprop with Nesterov momentum available</span> |
| <span class="sd"> at http://cs229.stanford.edu/proj2015/054_report.pdf.</span> |
| |
| <span class="sd"> This optimizer accepts the following parameters in addition to those accepted</span> |
| <span class="sd"> by :class:`.Optimizer`.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> beta1 : float, optional</span> |
| <span class="sd"> Exponential decay rate for the first moment estimates.</span> |
| <span class="sd"> beta2 : float, optional</span> |
| <span class="sd"> Exponential decay rate for the second moment estimates.</span> |
| <span class="sd"> epsilon : float, optional</span> |
| <span class="sd"> Small value to avoid division by 0.</span> |
| <span class="sd"> schedule_decay : float, optional</span> |
| <span class="sd"> Exponential decay rate for the momentum schedule</span> |
| <span class="sd"> """</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.001</span><span class="p">,</span> <span class="n">beta1</span><span class="o">=</span><span class="mf">0.9</span><span class="p">,</span> <span class="n">beta2</span><span class="o">=</span><span class="mf">0.999</span><span class="p">,</span> <span class="n">epsilon</span><span class="o">=</span><span class="mf">1e-8</span><span class="p">,</span> |
| <span class="n">schedule_decay</span><span class="o">=</span><span class="mf">0.004</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">Nadam</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">learning_rate</span><span class="o">=</span><span class="n">learning_rate</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">beta1</span> <span class="o">=</span> <span class="n">beta1</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">beta2</span> <span class="o">=</span> <span class="n">beta2</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">epsilon</span> <span class="o">=</span> <span class="n">epsilon</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">schedule_decay</span> <span class="o">=</span> <span class="n">schedule_decay</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">m_schedule</span> <span class="o">=</span> <span class="mf">1.</span> |
| |
| <span class="k">def</span> <span class="nf">create_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="k">return</span> <span class="p">(</span><span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">dtype</span><span class="p">),</span> <span class="c1"># mean</span> |
| <span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">,</span> <span class="n">dtype</span><span class="o">=</span><span class="n">weight</span><span class="o">.</span><span class="n">dtype</span><span class="p">))</span> <span class="c1"># variance</span> |
| |
| <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">weight</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="k">assert</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">))</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">_update_count</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">lr</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_lr</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="n">wd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_wd</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| |
| <span class="n">t</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_index_update_count</span><span class="p">[</span><span class="n">index</span><span class="p">]</span> |
| |
| <span class="c1"># preprocess grad</span> |
| <span class="n">grad</span> <span class="o">=</span> <span class="n">grad</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">rescale_grad</span> <span class="o">+</span> <span class="n">wd</span> <span class="o">*</span> <span class="n">weight</span> |
| <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span> |
| <span class="n">grad</span> <span class="o">=</span> <span class="n">clip</span><span class="p">(</span><span class="n">grad</span><span class="p">,</span> <span class="o">-</span><span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">clip_gradient</span><span class="p">)</span> |
| |
| <span class="c1"># warming momentum schedule</span> |
| <span class="n">momentum_t</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">beta1</span> <span class="o">*</span> <span class="p">(</span><span class="mf">1.</span> <span class="o">-</span> <span class="mf">0.5</span> <span class="o">*</span> <span class="p">(</span><span class="nb">pow</span><span class="p">(</span><span class="mf">0.96</span><span class="p">,</span> <span class="n">t</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">schedule_decay</span><span class="p">)))</span> |
| <span class="n">momentum_t_1</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">beta1</span> <span class="o">*</span> <span class="p">(</span><span class="mf">1.</span> <span class="o">-</span> <span class="mf">0.5</span> <span class="o">*</span> <span class="p">(</span><span class="nb">pow</span><span class="p">(</span><span class="mf">0.96</span><span class="p">,</span> <span class="p">(</span><span class="n">t</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">schedule_decay</span><span class="p">)))</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">m_schedule</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">m_schedule</span> <span class="o">*</span> <span class="n">momentum_t</span> |
| <span class="n">m_schedule_next</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">m_schedule</span> <span class="o">*</span> <span class="n">momentum_t_1</span> |
| |
| <span class="c1"># update m_t and v_t</span> |
| <span class="n">m_t</span><span class="p">,</span> <span class="n">v_t</span> <span class="o">=</span> <span class="n">state</span> |
| <span class="n">m_t</span><span class="p">[:]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">beta1</span> <span class="o">*</span> <span class="n">m_t</span> <span class="o">+</span> <span class="p">(</span><span class="mf">1.</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">beta1</span><span class="p">)</span> <span class="o">*</span> <span class="n">grad</span> |
| <span class="n">v_t</span><span class="p">[:]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">beta2</span> <span class="o">*</span> <span class="n">v_t</span> <span class="o">+</span> <span class="p">(</span><span class="mf">1.</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">beta2</span><span class="p">)</span> <span class="o">*</span> <span class="n">grad</span> <span class="o">*</span> <span class="n">grad</span> |
| |
| <span class="n">grad_prime</span> <span class="o">=</span> <span class="n">grad</span> <span class="o">/</span> <span class="p">(</span><span class="mf">1.</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">m_schedule</span><span class="p">)</span> |
| <span class="n">m_t_prime</span> <span class="o">=</span> <span class="n">m_t</span> <span class="o">/</span> <span class="p">(</span><span class="mf">1.</span> <span class="o">-</span> <span class="n">m_schedule_next</span><span class="p">)</span> |
| <span class="n">v_t_prime</span> <span class="o">=</span> <span class="n">v_t</span> <span class="o">/</span> <span class="p">(</span><span class="mf">1.</span> <span class="o">-</span> <span class="nb">pow</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">beta2</span><span class="p">,</span> <span class="n">t</span><span class="p">))</span> |
| <span class="n">m_t_bar</span> <span class="o">=</span> <span class="p">(</span><span class="mf">1.</span> <span class="o">-</span> <span class="n">momentum_t</span><span class="p">)</span> <span class="o">*</span> <span class="n">grad_prime</span> <span class="o">+</span> <span class="n">momentum_t_1</span> <span class="o">*</span> <span class="n">m_t_prime</span> |
| |
| <span class="c1"># update weight</span> |
| <span class="n">weight</span><span class="p">[:]</span> <span class="o">-=</span> <span class="n">lr</span> <span class="o">*</span> <span class="n">m_t_bar</span> <span class="o">/</span> <span class="p">(</span><span class="n">sqrt</span><span class="p">(</span><span class="n">v_t_prime</span><span class="p">)</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">epsilon</span><span class="p">)</span></div> |
| |
| <span class="nd">@register</span> |
| <div class="viewcode-block" id="Test"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Test">[docs]</a><span class="k">class</span> <span class="nc">Test</span><span class="p">(</span><span class="n">Optimizer</span><span class="p">):</span> |
| <span class="sd">"""The Test optimizer"""</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> |
| <span class="nb">super</span><span class="p">(</span><span class="n">Test</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> |
| |
| <div class="viewcode-block" id="Test.create_state"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Test.create_state">[docs]</a> <span class="k">def</span> <span class="nf">create_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="sd">"""Creates a state to duplicate weight."""</span> |
| <span class="k">return</span> <span class="n">zeros</span><span class="p">(</span><span class="n">weight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">)</span></div> |
| |
| <div class="viewcode-block" id="Test.update"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Test.update">[docs]</a> <span class="k">def</span> <span class="nf">update</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">state</span><span class="p">):</span> |
| <span class="sd">"""Performs w += rescale_grad * grad."""</span> |
| <span class="n">weight</span><span class="p">[:]</span> <span class="o">+=</span> <span class="n">grad</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">rescale_grad</span> |
| <span class="n">state</span><span class="p">[:]</span> <span class="o">=</span> <span class="n">weight</span></div></div> |
| |
| <span class="c1"># backward compatibility wrapper for Optimizer.CreateOptimizer</span> |
| <span class="n">create</span> <span class="o">=</span> <span class="n">Optimizer</span><span class="o">.</span><span class="n">create_optimizer</span> <span class="c1"># pylint: disable=invalid-name</span> |
| |
| <div class="viewcode-block" id="Updater"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Updater">[docs]</a><span class="k">class</span> <span class="nc">Updater</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> |
| <span class="sd">"""Updater for kvstore."""</span> |
| <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">optimizer</span><span class="p">):</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">optimizer</span> <span class="o">=</span> <span class="n">optimizer</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">states</span> <span class="o">=</span> <span class="p">{}</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">states_synced</span> <span class="o">=</span> <span class="p">{}</span> |
| |
| <span class="k">def</span> <span class="nf">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">index</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="n">weight</span><span class="p">):</span> |
| <span class="sd">"""Updates weight given gradient and index."""</span> |
| <span class="c1"># convert ctypes.char_p.value back to python str if needed</span> |
| <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">index</span><span class="p">,</span> <span class="nb">bytes</span><span class="p">):</span> |
| <span class="n">index</span> <span class="o">=</span> <span class="n">py_str</span><span class="p">(</span><span class="n">index</span><span class="p">)</span> |
| <span class="k">if</span> <span class="n">index</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">states</span><span class="p">:</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">states</span><span class="p">[</span><span class="n">index</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">optimizer</span><span class="o">.</span><span class="n">create_state_multi_precision</span><span class="p">(</span><span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">states_synced</span><span class="p">[</span><span class="n">index</span><span class="p">]</span> <span class="o">=</span> <span class="kc">True</span> |
| <span class="k">elif</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">states_synced</span><span class="p">[</span><span class="n">index</span><span class="p">]:</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">states</span><span class="p">[</span><span class="n">index</span><span class="p">]</span> <span class="o">=</span> \ |
| <span class="bp">self</span><span class="o">.</span><span class="n">sync_state_context</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">states</span><span class="p">[</span><span class="n">index</span><span class="p">],</span> <span class="n">weight</span><span class="o">.</span><span class="n">context</span><span class="p">)</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">states_synced</span><span class="p">[</span><span class="n">index</span><span class="p">]</span> <span class="o">=</span> <span class="kc">True</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">optimizer</span><span class="o">.</span><span class="n">update_multi_precision</span><span class="p">(</span><span class="n">index</span><span class="p">,</span> <span class="n">weight</span><span class="p">,</span> <span class="n">grad</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">states</span><span class="p">[</span><span class="n">index</span><span class="p">])</span> |
| |
| <div class="viewcode-block" id="Updater.sync_state_context"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Updater.sync_state_context">[docs]</a> <span class="k">def</span> <span class="nf">sync_state_context</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">state</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span> |
| <span class="sd">"""sync state context."""</span> |
| <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">state</span><span class="p">,</span> <span class="n">NDArray</span><span class="p">):</span> |
| <span class="k">return</span> <span class="n">state</span><span class="o">.</span><span class="n">as_in_context</span><span class="p">(</span><span class="n">context</span><span class="p">)</span> |
| <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">state</span><span class="p">,</span> <span class="p">(</span><span class="nb">tuple</span><span class="p">,</span> <span class="nb">list</span><span class="p">)):</span> |
| <span class="n">synced_state</span> <span class="o">=</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sync_state_context</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">context</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">state</span><span class="p">)</span> |
| <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">state</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">):</span> |
| <span class="k">return</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">synced_state</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="k">return</span> <span class="nb">list</span><span class="p">(</span><span class="n">synced_state</span><span class="p">)</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="k">return</span> <span class="n">state</span></div> |
| |
| <div class="viewcode-block" id="Updater.set_states"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Updater.set_states">[docs]</a> <span class="k">def</span> <span class="nf">set_states</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">states</span><span class="p">):</span> |
| <span class="sd">"""Sets updater states."""</span> |
| <span class="n">states</span> <span class="o">=</span> <span class="n">pickle</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">states</span><span class="p">)</span> |
| <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">states</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">)</span> <span class="ow">and</span> <span class="nb">len</span><span class="p">(</span><span class="n">states</span><span class="p">)</span> <span class="o">==</span> <span class="mi">2</span><span class="p">:</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">states</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">optimizer</span> <span class="o">=</span> <span class="n">states</span> |
| <span class="k">else</span><span class="p">:</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">states</span> <span class="o">=</span> <span class="n">states</span> |
| <span class="bp">self</span><span class="o">.</span><span class="n">states_synced</span> <span class="o">=</span> <span class="nb">dict</span><span class="o">.</span><span class="n">fromkeys</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">states</span><span class="o">.</span><span class="n">keys</span><span class="p">(),</span> <span class="kc">False</span><span class="p">)</span></div> |
| |
| <div class="viewcode-block" id="Updater.get_states"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.Updater.get_states">[docs]</a> <span class="k">def</span> <span class="nf">get_states</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">dump_optimizer</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span> |
| <span class="sd">"""Gets updater states.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> dump_optimizer : bool, default False</span> |
| <span class="sd"> Whether to also save the optimizer itself. This would also save optimizer</span> |
| <span class="sd"> information such as learning rate and weight decay schedules.</span> |
| <span class="sd"> """</span> |
| <span class="k">return</span> <span class="n">pickle</span><span class="o">.</span><span class="n">dumps</span><span class="p">((</span><span class="bp">self</span><span class="o">.</span><span class="n">states</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">optimizer</span><span class="p">)</span> <span class="k">if</span> <span class="n">dump_optimizer</span> <span class="k">else</span> <span class="bp">self</span><span class="o">.</span><span class="n">states</span><span class="p">)</span></div></div> |
| |
| <div class="viewcode-block" id="get_updater"><a class="viewcode-back" href="../../api/python/optimization/optimization.html#mxnet.optimizer.get_updater">[docs]</a><span class="k">def</span> <span class="nf">get_updater</span><span class="p">(</span><span class="n">optimizer</span><span class="p">):</span> |
| <span class="sd">"""Returns a closure of the updater needed for kvstore.</span> |
| |
| <span class="sd"> Parameters</span> |
| <span class="sd"> ----------</span> |
| <span class="sd"> optimizer: Optimizer</span> |
| <span class="sd"> The optimizer.</span> |
| |
| <span class="sd"> Returns</span> |
| <span class="sd"> -------</span> |
| <span class="sd"> updater: function</span> |
| <span class="sd"> The closure of the updater.</span> |
| <span class="sd"> """</span> |
| <span class="k">return</span> <span class="n">Updater</span><span class="p">(</span><span class="n">optimizer</span><span class="p">)</span></div> |
| </pre></div> |
| </div> |
| </div> |
| <div aria-label="main navigation" class="sphinxsidebar rightsidebar" role="navigation"> |
| <div class="sphinxsidebarwrapper"> |
| </div> |
| </div> |
| </div><div class="footer"> |
| <div class="section-disclaimer"> |
| <div class="container"> |
| <div> |
| <img height="60" src="https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/apache_incubator_logo.png"/> |
| <p> |
| Apache MXNet is an effort undergoing incubation at The Apache Software Foundation (ASF), <strong>sponsored by the <i>Apache Incubator</i></strong>. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. |
| </p> |
| <p> |
| "Copyright © 2017-2018, The Apache Software Foundation |
| Apache MXNet, MXNet, Apache, the Apache feather, and the Apache MXNet project logo are either registered trademarks or trademarks of the Apache Software Foundation." |
| </p> |
| </div> |
| </div> |
| </div> |
| </div> <!-- pagename != index --> |
| </div> |
| <script crossorigin="anonymous" integrity="sha384-0mSbJDEHialfmuBBQP6A4Qrprq5OVfW37PRR3j5ELqxss1yVqOtnepnHVP9aJ7xS" src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"></script> |
| <script src="../../_static/js/sidebar.js" type="text/javascript"></script> |
| <script src="../../_static/js/search.js" type="text/javascript"></script> |
| <script src="../../_static/js/navbar.js" type="text/javascript"></script> |
| <script src="../../_static/js/clipboard.min.js" type="text/javascript"></script> |
| <script src="../../_static/js/copycode.js" type="text/javascript"></script> |
| <script src="../../_static/js/page.js" type="text/javascript"></script> |
| <script type="text/javascript"> |
| $('body').ready(function () { |
| $('body').css('visibility', 'visible'); |
| }); |
| </script> |
| </body> |
| </html> |