| <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> |
| <!-- NewPage --> |
| <html lang="en"> |
| <head> |
| <!-- Generated by javadoc --> |
| <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> |
| <title>Overview (SystemDS 2.1.0-SNAPSHOT API)</title> |
| <link rel="stylesheet" type="text/css" href="stylesheet.css" title="Style"> |
| <script type="text/javascript" src="script.js"></script> |
| </head> |
| <body> |
| <script type="text/javascript"><!-- |
| try { |
| if (location.href.indexOf('is-external=true') == -1) { |
| parent.document.title="Overview (SystemDS 2.1.0-SNAPSHOT API)"; |
| } |
| } |
| catch(err) { |
| } |
| //--> |
| </script> |
| <noscript> |
| <div>JavaScript is disabled on your browser.</div> |
| </noscript> |
| <!-- ========= START OF TOP NAVBAR ======= --> |
| <div class="topNav"><a name="navbar.top"> |
| <!-- --> |
| </a> |
| <div class="skipNav"><a href="#skip.navbar.top" title="Skip navigation links">Skip navigation links</a></div> |
| <a name="navbar.top.firstrow"> |
| <!-- --> |
| </a> |
| <ul class="navList" title="Navigation"> |
| <li class="navBarCell1Rev">Overview</li> |
| <li>Package</li> |
| <li>Class</li> |
| <li>Use</li> |
| <li><a href="overview-tree.html">Tree</a></li> |
| <li><a href="deprecated-list.html">Deprecated</a></li> |
| <li><a href="index-all.html">Index</a></li> |
| <li><a href="help-doc.html">Help</a></li> |
| </ul> |
| </div> |
| <div class="subNav"> |
| <ul class="navList"> |
| <li>Prev</li> |
| <li>Next</li> |
| </ul> |
| <ul class="navList"> |
| <li><a href="index.html?overview-summary.html" target="_top">Frames</a></li> |
| <li><a href="overview-summary.html" target="_top">No Frames</a></li> |
| </ul> |
| <ul class="navList" id="allclasses_navbar_top"> |
| <li><a href="allclasses-noframe.html">All Classes</a></li> |
| </ul> |
| <div> |
| <script type="text/javascript"><!-- |
| allClassesLink = document.getElementById("allclasses_navbar_top"); |
| if(window==top) { |
| allClassesLink.style.display = "block"; |
| } |
| else { |
| allClassesLink.style.display = "none"; |
| } |
| //--> |
| </script> |
| </div> |
| <a name="skip.navbar.top"> |
| <!-- --> |
| </a></div> |
| <!-- ========= END OF TOP NAVBAR ========= --> |
| <div class="header"> |
| <h1 class="title">SystemDS 2.1.0-SNAPSHOT API</h1> |
| </div> |
| <div class="header"> |
| <div class="subTitle"> |
| <div class="block">SystemDS Architecture</div> |
| </div> |
| <p>See: <a href="#overview.description">Description</a></p> |
| </div> |
| <div class="contentContainer"> |
| <table class="overviewSummary" border="0" cellpadding="3" cellspacing="0" summary="Packages table, listing packages, and an explanation"> |
| <caption><span>Packages</span><span class="tabEnd"> </span></caption> |
| <tr> |
| <th class="colFirst" scope="col">Package</th> |
| <th class="colLast" scope="col">Description</th> |
| </tr> |
| <tbody> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/api/package-summary.html">org.apache.sysds.api</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/api/jmlc/package-summary.html">org.apache.sysds.api.jmlc</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/api/mlcontext/package-summary.html">org.apache.sysds.api.mlcontext</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/common/package-summary.html">org.apache.sysds.common</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/conf/package-summary.html">org.apache.sysds.conf</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/hops/package-summary.html">org.apache.sysds.hops</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/hops/codegen/package-summary.html">org.apache.sysds.hops.codegen</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/hops/codegen/cplan/package-summary.html">org.apache.sysds.hops.codegen.cplan</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/hops/codegen/cplan/cuda/package-summary.html">org.apache.sysds.hops.codegen.cplan.cuda</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/hops/codegen/cplan/java/package-summary.html">org.apache.sysds.hops.codegen.cplan.java</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/hops/codegen/opt/package-summary.html">org.apache.sysds.hops.codegen.opt</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/hops/codegen/template/package-summary.html">org.apache.sysds.hops.codegen.template</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/hops/cost/package-summary.html">org.apache.sysds.hops.cost</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/hops/estim/package-summary.html">org.apache.sysds.hops.estim</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/hops/ipa/package-summary.html">org.apache.sysds.hops.ipa</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/hops/recompile/package-summary.html">org.apache.sysds.hops.recompile</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/hops/rewrite/package-summary.html">org.apache.sysds.hops.rewrite</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/lops/package-summary.html">org.apache.sysds.lops</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/lops/compile/package-summary.html">org.apache.sysds.lops.compile</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/parser/package-summary.html">org.apache.sysds.parser</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/parser/dml/package-summary.html">org.apache.sysds.parser.dml</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/package-summary.html">org.apache.sysds.runtime</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/codegen/package-summary.html">org.apache.sysds.runtime.codegen</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/compress/package-summary.html">org.apache.sysds.runtime.compress</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/compress/cocode/package-summary.html">org.apache.sysds.runtime.compress.cocode</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/compress/colgroup/package-summary.html">org.apache.sysds.runtime.compress.colgroup</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/compress/colgroup/mapping/package-summary.html">org.apache.sysds.runtime.compress.colgroup.mapping</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/compress/colgroup/pre/package-summary.html">org.apache.sysds.runtime.compress.colgroup.pre</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/compress/colgroup/tree/package-summary.html">org.apache.sysds.runtime.compress.colgroup.tree</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/compress/estim/package-summary.html">org.apache.sysds.runtime.compress.estim</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/compress/estim/sample/package-summary.html">org.apache.sysds.runtime.compress.estim.sample</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/compress/lib/package-summary.html">org.apache.sysds.runtime.compress.lib</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/compress/readers/package-summary.html">org.apache.sysds.runtime.compress.readers</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/compress/utils/package-summary.html">org.apache.sysds.runtime.compress.utils</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/controlprogram/package-summary.html">org.apache.sysds.runtime.controlprogram</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/controlprogram/caching/package-summary.html">org.apache.sysds.runtime.controlprogram.caching</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/controlprogram/context/package-summary.html">org.apache.sysds.runtime.controlprogram.context</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/controlprogram/federated/package-summary.html">org.apache.sysds.runtime.controlprogram.federated</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/controlprogram/paramserv/package-summary.html">org.apache.sysds.runtime.controlprogram.paramserv</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/controlprogram/paramserv/dp/package-summary.html">org.apache.sysds.runtime.controlprogram.paramserv.dp</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/controlprogram/paramserv/rpc/package-summary.html">org.apache.sysds.runtime.controlprogram.paramserv.rpc</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/controlprogram/parfor/package-summary.html">org.apache.sysds.runtime.controlprogram.parfor</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/controlprogram/parfor/opt/package-summary.html">org.apache.sysds.runtime.controlprogram.parfor.opt</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/controlprogram/parfor/stat/package-summary.html">org.apache.sysds.runtime.controlprogram.parfor.stat</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/controlprogram/parfor/util/package-summary.html">org.apache.sysds.runtime.controlprogram.parfor.util</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/data/package-summary.html">org.apache.sysds.runtime.data</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/functionobjects/package-summary.html">org.apache.sysds.runtime.functionobjects</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/instructions/package-summary.html">org.apache.sysds.runtime.instructions</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/instructions/cp/package-summary.html">org.apache.sysds.runtime.instructions.cp</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/instructions/cpfile/package-summary.html">org.apache.sysds.runtime.instructions.cpfile</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/instructions/fed/package-summary.html">org.apache.sysds.runtime.instructions.fed</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/instructions/gpu/package-summary.html">org.apache.sysds.runtime.instructions.gpu</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/instructions/gpu/context/package-summary.html">org.apache.sysds.runtime.instructions.gpu.context</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/instructions/spark/package-summary.html">org.apache.sysds.runtime.instructions.spark</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/instructions/spark/data/package-summary.html">org.apache.sysds.runtime.instructions.spark.data</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/instructions/spark/functions/package-summary.html">org.apache.sysds.runtime.instructions.spark.functions</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/instructions/spark/utils/package-summary.html">org.apache.sysds.runtime.instructions.spark.utils</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/io/package-summary.html">org.apache.sysds.runtime.io</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/lineage/package-summary.html">org.apache.sysds.runtime.lineage</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/matrix/data/package-summary.html">org.apache.sysds.runtime.matrix.data</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/matrix/operators/package-summary.html">org.apache.sysds.runtime.matrix.operators</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/meta/package-summary.html">org.apache.sysds.runtime.meta</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/privacy/package-summary.html">org.apache.sysds.runtime.privacy</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/privacy/finegrained/package-summary.html">org.apache.sysds.runtime.privacy.finegrained</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/privacy/propagation/package-summary.html">org.apache.sysds.runtime.privacy.propagation</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/transform/package-summary.html">org.apache.sysds.runtime.transform</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/transform/decode/package-summary.html">org.apache.sysds.runtime.transform.decode</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/transform/encode/package-summary.html">org.apache.sysds.runtime.transform.encode</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/transform/meta/package-summary.html">org.apache.sysds.runtime.transform.meta</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/transform/tokenize/package-summary.html">org.apache.sysds.runtime.transform.tokenize</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="altColor"> |
| <td class="colFirst"><a href="org/apache/sysds/runtime/util/package-summary.html">org.apache.sysds.runtime.util</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| <tr class="rowColor"> |
| <td class="colFirst"><a href="org/apache/sysds/utils/package-summary.html">org.apache.sysds.utils</a></td> |
| <td class="colLast"> </td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
| <div class="contentContainer"><a name="overview.description"> |
| <!-- --> |
| </a> |
| <div class="block"><h1>SystemDS Architecture</h1> |
| Algorithms in Apache SystemDS are written in a high-level R-like language called Declarative Machine learning Language (DML) |
| or a high-level Python-like language called PyDML. |
| SystemDS compiles and optimizes these algorithms into hybrid runtime |
| plans of multi-threaded, in-memory operations on a single node (scale-up) and distributed Spark operations on |
| a cluster of nodes (scale-out). SystemDS's high-level architecture consists of the following components: |
| |
| <h2>Language</h2> |
| DML (with either R- or Python-like syntax) provides linear algebra primitives, a rich set of statistical |
| functions and matrix manipulations, |
| as well as user-defined and external functions, control structures including parfor loops, and recursion. |
| The user provides the DML script through one of the following APIs: |
| <ul> |
| <li>Command-line interface ( <a href="org/apache/sysds/api/DMLScript.html" title="class in org.apache.sysds.api"><code>DMLScript</code></a> )</li> |
| <li>Convenient programmatic interface for Spark users ( <a href="org/apache/sysds/api/mlcontext/MLContext.html" title="class in org.apache.sysds.api.mlcontext"><code>MLContext</code></a> )</li> |
| <li>Java Machine Learning Connector API ( <a href="org/apache/sysds/api/jmlc/Connection.html" title="class in org.apache.sysds.api.jmlc"><code>Connection</code></a> )</li> |
| </ul> |
| |
| <a href="org/apache/sysds/parser/ParserWrapper.html" title="class in org.apache.sysds.parser"><code>ParserWrapper</code></a> performs syntactic validation and |
| parses the input DML script using ANTLR into a |
| a hierarchy of <a href="org/apache/sysds/parser/StatementBlock.html" title="class in org.apache.sysds.parser"><code>StatementBlock</code></a> and |
| <a href="org/apache/sysds/parser/Statement.html" title="class in org.apache.sysds.parser"><code>Statement</code></a> as defined by control structures. |
| Another important class of the language component is <a href="org/apache/sysds/parser/DMLTranslator.html" title="class in org.apache.sysds.parser"><code>DMLTranslator</code></a> |
| which performs live variable analysis and semantic validation. |
| During that process we also retrieve input data characteristics -- i.e., format, |
| number of rows, columns, and non-zero values -- as well as |
| infrastructure characteristics, which are used for subsequent |
| optimizations. Finally, we construct directed acyclic graphs (DAGs) |
| of high-level operators ( <a href="org/apache/sysds/hops/Hop.html" title="class in org.apache.sysds.hops"><code>Hop</code></a> ) per statement block. |
| |
| <h2>Optimizer</h2> |
| The SystemDS optimizer works over programs of HOP DAGs, where HOPs are operators on |
| matrices or scalars, and are categorized according to their |
| access patterns. Examples are matrix multiplications, unary |
| aggregates like rowSums(), binary operations like cell-wise |
| matrix additions, reorganization operations like transpose or |
| sort, and more specific operations. We perform various optimizations |
| on these HOP DAGs, including algebraic simplification rewrites ( <a href="org/apache/sysds/hops/rewrite/ProgramRewriter.html" title="class in org.apache.sysds.hops.rewrite"><code>ProgramRewriter</code></a> ), |
| intra-/<a href="org/apache/sysds/hops/ipa/InterProceduralAnalysis.html" title="class in org.apache.sysds.hops.ipa"><code>InterProceduralAnalysis</code></a> |
| for statistics propagation into functions and over entire programs, and |
| operator ordering of matrix multiplication chains. We compute |
| memory estimates for all HOPs, reflecting the memory |
| requirements of in-memory single-node operations and |
| intermediates. Each HOP DAG is compiled to a DAG of |
| low-level operators ( <a href="org/apache/sysds/lops/Lop.html" title="class in org.apache.sysds.lops"><code>Lop</code></a> ) such as grouping and aggregate, |
| which are backend-specific physical operators. Operator selection |
| picks the best physical operators for a given HOP |
| based on memory estimates, data, and cluster characteristics. |
| Individual LOPs have corresponding runtime implementations, |
| called instructions, and the optimizer generates |
| an executable runtime program of instructions. |
| |
| <h2>Runtime</h2> |
| We execute the generated runtime program locally |
| in CP (control program), i.e., within a driver process. |
| This driver handles recompilation, runs in-memory single-node |
| <a href="org/apache/sysds/runtime/instructions/cp/CPInstruction.html" title="class in org.apache.sysds.runtime.instructions.cp"><code>CPInstruction</code></a> (some of which are multi-threaded ), |
| maintains an in-memory buffer pool, and launches Spark jobs if the runtime plan contains distributed computations |
| in the form of Spark instructions ( <a href="org/apache/sysds/runtime/instructions/spark/SPInstruction.html" title="class in org.apache.sysds.runtime.instructions.spark"><code>SPInstruction</code></a> ). |
| For the Spark backend, we rely on Spark's lazy evaluation and stage construction. |
| CP instructions may also be backed by GPU kernels ( <a href="org/apache/sysds/runtime/instructions/gpu/GPUInstruction.html" title="class in org.apache.sysds.runtime.instructions.gpu"><code>GPUInstruction</code></a> ). |
| The multi-level buffer pool caches local matrices in-memory, |
| evicts them if necessary, and handles data exchange between |
| local and distributed runtime backends. |
| The core of SystemDS's runtime instructions is an adaptive matrix block library, |
| which is sparsity-aware and operates on the entire matrix in CP, or blocks of a matrix in a distributed setting. Further |
| key features include parallel for-loops for task-parallel |
| computations, and dynamic recompilation for runtime plan adaptation addressing initial unknowns.</div> |
| </div> |
| <!-- ======= START OF BOTTOM NAVBAR ====== --> |
| <div class="bottomNav"><a name="navbar.bottom"> |
| <!-- --> |
| </a> |
| <div class="skipNav"><a href="#skip.navbar.bottom" title="Skip navigation links">Skip navigation links</a></div> |
| <a name="navbar.bottom.firstrow"> |
| <!-- --> |
| </a> |
| <ul class="navList" title="Navigation"> |
| <li class="navBarCell1Rev">Overview</li> |
| <li>Package</li> |
| <li>Class</li> |
| <li>Use</li> |
| <li><a href="overview-tree.html">Tree</a></li> |
| <li><a href="deprecated-list.html">Deprecated</a></li> |
| <li><a href="index-all.html">Index</a></li> |
| <li><a href="help-doc.html">Help</a></li> |
| </ul> |
| </div> |
| <div class="subNav"> |
| <ul class="navList"> |
| <li>Prev</li> |
| <li>Next</li> |
| </ul> |
| <ul class="navList"> |
| <li><a href="index.html?overview-summary.html" target="_top">Frames</a></li> |
| <li><a href="overview-summary.html" target="_top">No Frames</a></li> |
| </ul> |
| <ul class="navList" id="allclasses_navbar_bottom"> |
| <li><a href="allclasses-noframe.html">All Classes</a></li> |
| </ul> |
| <div> |
| <script type="text/javascript"><!-- |
| allClassesLink = document.getElementById("allclasses_navbar_bottom"); |
| if(window==top) { |
| allClassesLink.style.display = "block"; |
| } |
| else { |
| allClassesLink.style.display = "none"; |
| } |
| //--> |
| </script> |
| </div> |
| <a name="skip.navbar.bottom"> |
| <!-- --> |
| </a></div> |
| <!-- ======== END OF BOTTOM NAVBAR ======= --> |
| <p class="legalCopy"><small>Copyright © 2021 <a href="https://www.apache.org/">The Apache Software Foundation</a>. All rights reserved.</small></p> |
| </body> |
| </html> |
