tree 8c787b577bdbf3e755b3ff5f1f28e87d842c3fdd
parent 205c67aaf1778cb62f91de415ab677ad8be7f2dd
author Matthias Boehm <mboehm7@gmail.com> 1591877293 +0200
committer Matthias Boehm <mboehm7@gmail.com> 1591877293 +0200

[SYSTEMDS-412] Fix robustness lineage DAGs, parfor integration

This patch makes further robustness improvements to the handling of
large lineage DAGs via non-recursive primitives. In this context,
explain needed special treatment to preserve the previous output in DFS
order w/ post-append.

Furthermore, this also fixes a number of issues of the parfor
integration such as (1) invalid cached hashes after sub-DAG replacement,
(2) introduced cycles during parfor lineage merge, (3) steplm script
improvements (disabled parfor dependency analysis was hiding the issue
that introduced the cycles), and (4) some debugging functionality to
reliably detect cycles in lineage DAGs.