tree f887b05ef56d33239d3b62b381e1a58c293bdb78
parent 7af2ae04f28ddcb36158719a25a7fa34b22d3266
author Matthias Boehm <mboehm7@gmail.com> 1596912685 +0200
committer Matthias Boehm <mboehm7@gmail.com> 1596912715 +0200

[SYSTEMDS-2600] Rework federated runtime backend (framework, ops)

This patch makes a major rework of the exiting federated runtime backend
and operations in order to simplify the joint development of all
remaining federated operations.

The new design has only four command types: read, put, get, exec_inst,
which allows to read federated matrices, put and get variables, and
execute arbitrary instructions over these variables. With this approach,
we can reuse the existing symbol table and CP/Spark instructions and
only need to handle their orchestration and global compensations.
Furthermore, the new design adds several primitives like broadcast,
broadcastSliced, aggregations, and rbind/cbind and more convenient data
structures.

Finally, this patch also includes minor reworks of the execution
context, and reblock rewrite to allow for specific characteristics of
federated execution.