tree 3765987b17c5c381a5064aa61ff08f96094abfaa
parent a0835010840346a260029eaebe3813d8f7a05a0f
author Matthias Boehm <mboehm7@gmail.com> 1597517887 +0200
committer Matthias Boehm <mboehm7@gmail.com> 1597517887 +0200

[SYSTEMDS-2624] Rework acquireRead and cleanup of federated matrices

The integration of federated matrices and frames into our caching and
buffer pool framework was suboptimal because (1) repeated pinning of a
federated data into memory of the coordinator caused repeated
consolidation of the federated data, and (2) missing cleanup of
federated data intermediates. This rework fixes both issues and thus,
reduces unnecessary memory pressure at the workers and unnecessary data
transfer at the coordinator.

For the cleanup of federated data, we added special guards to ensure
only federated intermediates (and files thereof) are deleted but NOT the
original federated input files.