[SYSTEMML-2204] Fix deregistration of parallelized RDDs on cleanup
This patch fixes the meta data handling of parallelized RDDs to allow
for proper deregistration during cleanup (e.g., on rmvar). With this
patch, we avoid unnecessarily conservative guarded parallelize by
correctly reflecting the currently consumed memory of live RDDs in the
driver.
diff --git a/src/main/java/org/apache/sysml/runtime/controlprogram/context/SparkExecutionContext.java b/src/main/java/org/apache/sysml/runtime/controlprogram/context/SparkExecutionContext.java
index 86c3adc..179c24a 100644
--- a/src/main/java/org/apache/sysml/runtime/controlprogram/context/SparkExecutionContext.java
+++ b/src/main/java/org/apache/sysml/runtime/controlprogram/context/SparkExecutionContext.java
@@ -380,6 +380,7 @@
//keep rdd handle for future operations on it
RDDObject rddhandle = new RDDObject(rdd);
rddhandle.setHDFSFile(fromFile);
+ rddhandle.setParallelizedRDD(!fromFile);
mo.setRDDHandle(rddhandle);
}
//CASE 3: non-dirty (file exists on HDFS)
