tree 6c781d8e105bcba5957cff9edb5648e0fd1d2a5e
parent 91923af3c3aa71fe24c85679cbbc30933e06fd97
author Yuhao Zhang <yzhang@pivotal.io> 1571089767 -0400
committer Orhan Kislal <okislal@apache.org> 1571089937 -0400

DL: Add training for multiple models

This commit adds a new function to train multiple models in parallel
with model hopping method supported for Greenplum DB only.

Model hopping method involves the following steps:

- Train models in parallel for a single epoch using the data local
to each segment.
- Move the models to the next segment in round-robin fashion.

This method ensures that all of the models visit the entire dataset,
which eliminates the need to average the model at the end.

This commit also fixes the following issue:
-  In the regular fit function, excessive amounts of threads were being
created and left over by keras sessions. This issue was fixed by reusing
the same session and the same computational graph throughout the
process.
- While deserializing weights, if the model shape expected less elements
than present in the weights, the excessive weights would get dropped.
This is fixed by adding an explicit check for validating number of
elements in model weights matches the model.

Closes #443

Co-authored-by: Ekta Khanna <ekhanna@pivotal.io>
Co-authored-by: Nandish Jayaram <njayaram@apache.org>
Co-authored-by: Nikhil Kak <nkak@pivotal.io>
Co-authored-by: Orhan Kislal <okislal@apache.org>