tree 401789579686239bf9e6eb9bd941d9a132a49af5
parent 77f3fd7ef6fdc0eb54656d73ae0d2c06244062ad
author Orhan Kislal <okislal@apache.org> 1617031727 -0400
committer Orhan Kislal <okislal@pivotal.io> 1620405964 +0300

MLP: Add rmsprop and Adam optimization techniques

JIRA: MADLIB-1434, MADLIB-1435

This commit adds two new variants on the gradient descent algorithms.
They are paried together since they both involve adaptive learning rates.

rmsprop: Keep a moving average of the squared gradient for each weight
Adam: Keep an exponential moving average of the gradient and the squared gradient