<!DOCTYPE html PUBLIC ""
    "">
<html><head><meta charset="UTF-8" /><title>org.apache.clojure-mxnet.optimizer documentation</title><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="css/highlight.css" /><script type="text/javascript" src="js/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a></h2><h1><a href="index.html"><span class="project-title"><span class="project-name">Clojure-mxnet</span> <span class="project-version">1.6.0-SNAPSHOT</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>org</span></div></div></li><li class="depth-2"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>apache</span></div></div></li><li class="depth-3"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>clojure-mxnet</span></div></div></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.base.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>base</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.callback.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>callback</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.context.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>context</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.dtype.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>dtype</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.eval-metric.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>eval-metric</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.executor.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>executor</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.image.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>image</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.infer.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>infer</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.initializer.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>initializer</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.io.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>io</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.kvstore.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>kvstore</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.kvstore-server.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>kvstore-server</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.layout.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>layout</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.lr-scheduler.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>lr-scheduler</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.module.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>module</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.monitor.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>monitor</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.ndarray.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>ndarray</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.ndarray-api.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>ndarray-api</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.ndarray-random-api.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>ndarray-random-api</span></div></a></li><li class="depth-4 branch current"><a href="org.apache.clojure-mxnet.optimizer.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>optimizer</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.primitives.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>primitives</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.profiler.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>profiler</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.random.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>random</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.resource-scope.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>resource-scope</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.shape.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>shape</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.symbol.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>symbol</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.symbol-api.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>symbol-api</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.symbol-random-api.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>symbol-random-api</span></div></a></li><li class="depth-4 branch"><a href="org.apache.clojure-mxnet.util.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>util</span></div></a></li><li class="depth-4"><a href="org.apache.clojure-mxnet.visualization.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>visualization</span></div></a></li></ul></div><div class="sidebar secondary"><h3><a href="#top"><span class="inner">Public Vars</span></a></h3><ul><li class="depth-1"><a href="org.apache.clojure-mxnet.optimizer.html#var-ada-delta"><div class="inner"><span>ada-delta</span></div></a></li><li class="depth-1"><a href="org.apache.clojure-mxnet.optimizer.html#var-ada-grad"><div class="inner"><span>ada-grad</span></div></a></li><li class="depth-1"><a href="org.apache.clojure-mxnet.optimizer.html#var-adam"><div class="inner"><span>adam</span></div></a></li><li class="depth-1"><a href="org.apache.clojure-mxnet.optimizer.html#var-create-state"><div class="inner"><span>create-state</span></div></a></li><li class="depth-1"><a href="org.apache.clojure-mxnet.optimizer.html#var-dcasgd"><div class="inner"><span>dcasgd</span></div></a></li><li class="depth-1"><a href="org.apache.clojure-mxnet.optimizer.html#var-nag"><div class="inner"><span>nag</span></div></a></li><li class="depth-1"><a href="org.apache.clojure-mxnet.optimizer.html#var-rms-prop"><div class="inner"><span>rms-prop</span></div></a></li><li class="depth-1"><a href="org.apache.clojure-mxnet.optimizer.html#var-sgd"><div class="inner"><span>sgd</span></div></a></li><li class="depth-1"><a href="org.apache.clojure-mxnet.optimizer.html#var-sgld"><div class="inner"><span>sgld</span></div></a></li><li class="depth-1"><a href="org.apache.clojure-mxnet.optimizer.html#var-update"><div class="inner"><span>update</span></div></a></li></ul></div><div class="namespace-docs" id="content"><h1 class="anchor" id="top">org.apache.clojure-mxnet.optimizer</h1><div class="doc"><pre class="plaintext"></pre></div><div class="public anchor" id="var-ada-delta"><h3>ada-delta</h3><div class="usage"><code>(ada-delta {:keys [rho rescale-gradient epsilon wd clip-gradient], :as opts, :or {rho 0.05, rescale-gradient 1.0, epsilon 1.0E-8, wd 0.0, clip-gradient 0}})</code><code>(ada-delta)</code></div><div class="doc"><pre class="plaintext">AdaDelta optimizer as described in Matthew D. Zeiler, 2012.
<a href="http://arxiv.org/abs/1212.5701">http://arxiv.org/abs/1212.5701</a></pre></div></div><div class="public anchor" id="var-ada-grad"><h3>ada-grad</h3><div class="usage"><code>(ada-grad {:keys [learning-rate rescale-gradient epsilon wd], :as opts, :or {learning-rate 0.05, rescale-gradient 1.0, epsilon 1.0E-7, wd 0.0}})</code><code>(ada-grad)</code></div><div class="doc"><pre class="plaintext"> AdaGrad optimizer as described in Duchi, Hazan and Singer, 2011.
<a href="http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf">http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf</a>

- learning-rate Step size.
- epsilon A small number to make the updating processing stable.
             Default value is set to 1e-7.
- rescale-gradient rescaling factor of gradient.
- wd L2 regularization coefficient add to all the weights</pre></div></div><div class="public anchor" id="var-adam"><h3>adam</h3><div class="usage"><code>(adam {:keys [learning-rate beta1 beta2 epsilon decay-factor wd clip-gradient lr-scheduler], :as opts, :or {learning-rate 0.002, beta1 0.9, beta2 0.999, epsilon 1.0E-8, decay-factor (- 1 1.0E-8), wd 0, clip-gradient 0}})</code><code>(adam)</code></div><div class="doc"><pre class="plaintext">Adam optimizer as described in [King2014]

[King2014] Diederik Kingma, Jimmy Ba,
Adam: A Method for Stochastic Optimization,
<a href="http://arxiv.org/abs/1412.6980">http://arxiv.org/abs/1412.6980</a>

 - learning-rate  Step size.
 - beta1  Exponential decay rate for the first moment estimates.
 - beta2  Exponential decay rate for the second moment estimates.
 -  epsilon
 - decay-factor
 - wd L2 regularization coefficient add to all the weights
 - clip-gradient  clip gradient in range [-clip_gradient, clip_gradient]
 - lr-scheduler The learning rate scheduler</pre></div></div><div class="public anchor" id="var-create-state"><h3>create-state</h3><div class="usage"><code>(create-state optimizer index weight)</code></div><div class="doc"><pre class="plaintext">Create additional optimizer state such as momentum.
</pre></div></div><div class="public anchor" id="var-dcasgd"><h3>dcasgd</h3><div class="usage"><code>(dcasgd {:keys [learning-rate momentum lambda wd clip-gradient lr-scheduler], :as opts, :or {learning-rate 0.01, momentum 0.0, lambda 0.04, wd 0.0, clip-gradient 0}})</code><code>(dcasgd)</code></div><div class="doc"><pre class="plaintext">DCASGD optimizer with momentum and weight regularization.
Implementation of paper 'Asynchronous Stochastic Gradient Descent with
Delay Compensation for Distributed Deep Learning'</pre></div></div><div class="public anchor" id="var-nag"><h3>nag</h3><div class="usage"><code>(nag {:keys [learning-rate momentum wd clip-gradient lr-scheduler], :as opts, :or {learning-rate 0.01, momentum 0.0, wd 1.0E-4, clip-gradient 0}})</code><code>(nag)</code></div><div class="doc"><pre class="plaintext">SGD with nesterov.
It is implemented according to
<a href="https://github.com/torch/optim/blob/master/sgd.lua">https://github.com/torch/optim/blob/master/sgd.lua</a></pre></div></div><div class="public anchor" id="var-rms-prop"><h3>rms-prop</h3><div class="usage"><code>(rms-prop {:keys [learning-rate rescale-gradient gamma1 gamma2 wd lr-scheduler clip-gradient], :as opts, :or {learning-rate 0.002, rescale-gradient 1.0, gamma1 0.95, gamma2 0.9, wd 0.0, clip-gradient 0}})</code><code>(rms-prop)</code></div><div class="doc"><pre class="plaintext">RMSProp optimizer as described in Tieleman &amp; Hinton, 2012.
<a href="http://arxiv.org/pdf/1308.0850v5.pdf">http://arxiv.org/pdf/1308.0850v5.pdf</a> Eq(38) - Eq(45) by Alex Graves, 2013.
- learningRate Step size.
- gamma1  decay factor of moving average for gradient, gradient^^2.
-  gamma2  momentum factor of moving average for gradient.
-  rescale-gradient rescaling factor of gradient.
-  wd L2 regularization coefficient add to all the weights
-  clip-gradient clip gradient in range [-clip_gradient, clip_gradient]
-  lr-scheduler The learning rate scheduler</pre></div></div><div class="public anchor" id="var-sgd"><h3>sgd</h3><div class="usage"><code>(sgd {:keys [learning-rate momentum wd clip-gradient lr-scheduler], :as opts, :or {learning-rate 0.01, momentum 0.0, wd 1.0E-4, clip-gradient 0}})</code><code>(sgd)</code></div><div class="doc"><pre class="plaintext">A very simple SGD optimizer with momentum and weight regularization.
</pre></div></div><div class="public anchor" id="var-sgld"><h3>sgld</h3><div class="usage"><code>(sgld {:keys [learning-rate rescale-gradient wd clip-gradient lr-scheduler], :as opts, :or {learning-rate 0.01, rescale-gradient 1, wd 1.0E-4, clip-gradient 0}})</code><code>(sgld)</code></div><div class="doc"><pre class="plaintext">Stochastic Langevin Dynamics Updater to sample from a distribution.

- learning-rate Step size.
- rescale-gradient rescaling factor of gradient.
- wd L2 regularization coefficient add to all the weights
- clip-gradient Float, clip gradient in range [-clip_gradient, clip_gradient]
- lr-scheduler The learning rate scheduler</pre></div></div><div class="public anchor" id="var-update"><h3>update</h3><div class="usage"><code>(update optimizer index weight grad state)</code></div><div class="doc"><pre class="plaintext">Update the parameters.
- optimizer - the optimizer
-  index An unique integer key used to index the parameters
-  weight weight ndarray
-  grad grad ndarray
 -  state NDArray or other objects returned by initState
          The auxiliary state used in optimization.
</pre></div></div></div></body></html>