tree 092940ff4679b54b494053f6e1bb7afd25908dc0
parent 51d8b29f6c0891568da9ec8d489fd5de7565699d
author Matthias Boehm <mboehm7@gmail.com> 1597410506 +0200
committer Matthias Boehm <mboehm7@gmail.com> 1597410506 +0200

[SYSTEMDS-2618] Rework federated backend (UDF support for custom ops)

This patch finalizes the rework of the federated runtime backend to only
rely on few meta operations. So far we supported read, put, get, and
instructions. Unfortunately, relying purely on instruction is not
possible for more complex operations. Inspired by Spark's RDD
operations, we extend this by a simple UDF mechanism where local UDF
functions objects for custom operations are constructed at the driver,
serialized, and shipped to the federated workers. This patch introduces
the related UDF framework and re-implements the recently added transform
encode via such UDFs.