content/release-notes/systemds-release-notes-2.0.0.html - systemds-website - Git at Google

 <!--

 -->
 <!--

 -->
 <!DOCTYPE html>
 <html lang="en">
   <!--

 -->

 <head>
   <meta charset="utf-8">
   <meta http-equiv="X-UA-Compatible" content="IE=edge">

   <title>SystemML 1.2.0 Release Notes</title>

   <meta name="description" content="Project Release Notes">
   <meta name="author" content="Apache SystemDS">

   <!-- Enable responsive viewport -->
   <meta name="HandheldFriendly" content="True">
   <meta name="viewport" content="width=device-width, initial-scale=1.0">

   <!-- You can use Open Graph tags to customize link previews.
   Learn more: https://developers.facebook.com/docs/sharing/webmasters -->
   <meta property="og:url" content="https://systemds.apache.org/" />
   <meta property="og:type" content="website" />
   <meta property="og:title" content="SystemML 1.2.0 Release Notes" />
   <meta property="og:description" content="Apache SystemDS is an open source ML system for the end-to-end data science lifecycle" />
   <meta property="og:image" content="" />

   <!-- Le HTML5 shim, for IE6-8 support of HTML elements -->
   <!--[if lt IE 9]>
   <script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
   <![endif]-->

   <!-- Le styles -->
   <link rel="stylesheet" href="/assets/css/main.css">

   <!-- favicons -->
   <link rel="shortcut icon" href="/assets/img/favicon.png">
 </head> <!-- META -->
   <body class="vcard">
     <!--

 -->
 <header class="site-header site-header--not-home">
   <h1 class="logo"><a class="url" href="/"><i class="logo-mark"></i><span class="fn org">Apache SystemDS<sup
           id="trademark">&trade;</sup></span></a></h1>
   <nav class="main-nav">
     <ul>

       <li role="presentation">


         <a href="/download" target="_self">Download</a>

       </li>

       <li role="presentation">


         <a href="/get-started" target="_self">Get Started</a>

       </li>

       <li role="presentation">

         <a class="nav-link--hover">Docs <i class="icon icon-chevron-down"></i></a>
         <ul>


           <li><a href="https://apache.github.io/systemds/" target="_blank">latest</a></li>


           <li><a href="https://systemds.apache.org/docs/3.2.0/" target="_blank">3.2.0 (stable)</a></li>


           <li><a href="https://systemds.apache.org/release-notes/systemds-release-notes-3.2.0" target="_blank">Release Notes 3.2.0</a></li>

         </ul>

       </li>

       <li role="presentation">

         <a class="nav-link--hover">Community <i class="icon icon-chevron-down"></i></a>
         <ul>


           <li><a href="/community" target="_self">Get Involved</a></li>


           <li><a href="https://issues.apache.org/jira/browse/SYSTEMDS" target="_blank">Issue Tracker</a></li>


           <li><a href="https://github.com/apache/systemds" target="_blank">Source Code</a></li>


           <li><a href="https://github.com/apache/systemds-website" target="_blank">Website Source Code</a></li>

         </ul>

       </li>

       <li role="presentation">

         <a class="nav-link--hover">Apache <i class="icon icon-chevron-down"></i></a>
         <ul>


           <li><a href="https://www.apache.org/foundation/how-it-works" target="_blank">Apache Software Foundation</a></li>


           <li><a href="https://www.apache.org/licenses/" target="_blank">Apache License</a></li>


           <li><a href="https://www.apache.org/foundation/sponsorship" target="_blank">Sponsorship</a></li>


           <li><a href="https://www.apache.org/foundation/thanks" target="_blank">Thanks</a></li>


           <li><a href="/privacy-policy" target="_self">Privacy Policy</a></li>


           <li><a href="/security" target="_self">Security</a></li>

         </ul>

       </li>

     </ul>
   </nav>
 </header> <!-- GLOBAL HEADER -->
     <!--

 -->
 <!--

 -->
 <div>
   <!--

 -->

 <section class="full-stripe full-stripe--subpage-header clear-header">
   <div class="ml-container ml-container--horizontally-center">
     <div class="col col-12 content-group content-group--center-content content-group--center-align">
       <h1>Apache SystemDS<sup id="trademark">&trade;</sup> 2.0.0 Release Notes</h1>
     </div>
   </div>
 </section>

 <section class="full-stripe full-stripe--alternate">
   <div class="ml-container">
     <div class="col col-12 content-group content-group--medium-bottom-margin">

       <h2 id="release-notes-for-systemds-20">Release notes for SystemDS 2.0.</h2>

       <p>SystemDS 2.0 is the first major release under the new name. This release contains a major refactoring, a few major
 features, a large number of improvements and fixes, and some experimental features to better support the end-to-end
 data science lifecycle. In addition to that, this release also removes several features that are not up date and
 outdated.</p>

       <p>The major changes (compared to SystemML 1.2) include</p>

       <ul>
         <li>New mechanism for DML-bodied (script-level) builtin functions, and a wealth of new built-in functions for data preprocessing including data cleaning, augmentation and feature engineering techniques, new ML algorithms, and model debugging.</li>
         <li>Several methods for data cleaning have been implemented including multiple imputations with multivariate imputation by chained equations (MICE) and other techniques, SMOTE, an oversampling technique for class imbalance, forward and backward NA filling, cleaning using schema and length information, support for outlier detection using standard deviation and inter-quartile range, and functional dependency discovery.</li>
         <li>A complete framework for lineage tracing and reuse including support for loop deduplication, full and partial reuse, compiler assisted reuse, several new rewrites to facilitate reuse.</li>
         <li>New federated runtime backend including support for federated matrices and frames, federated builtins (transform-encode, decode etc.).</li>
         <li>Refactor compression package and add functionalities including quantization for lossy compression, binary cell operations, left matrix multiplication. [experimental]</li>
         <li>New python bindings with supports for several builtins, matrix operations, federated tensors and lineage traces.</li>
         <li>Cuda implementation of cumulative aggregate operators (cumsum, cumprod etc.)</li>
         <li>New model debugging technique with slice finder.</li>
         <li>New tensor data model (basic tensors of different value types, data tensors with schema) [experimental]</li>
         <li>Cloud deployment scripts for AWS and scripts to set up and start federated operations.</li>
         <li>Performance improvements with parallel sort, gpu cum agg, append cbind etc.</li>
         <li>Various compiler and runtime improvements including new and improved rewrites, reduced Spark context creation, new eval framework, list operations, updated native kernel libraries to name a few.</li>
         <li>New data reader/writer for json frames and support for sql as a data source.</li>
         <li>Miscellaneous improvements: improved documentation, better testing, run/release scripts, improved packaging, Docker container for systemds, support for lambda expressions, bug fixes.</li>
         <li>Removed MapReduce compiler and runtime backend, pydml parser, Java-UDF framework, script-level debugger.</li>
         <li>Deprecated ./scripts/algorithms, as those algorithms gradually will be part of SystemDS builtins.</li>
       </ul>
     </div></div></section>

 </div>


     <!--

 -->
 <footer class="site-footer">
   <div class="ml-container ml-container--horizontally-center">
     <div class="footer-col-wrapper">
       <div style="margin-top: 20px; margin-bottom: 20px;">
         <a href="https://www.apache.org"><img src="/assets/img/apache-foundation-logo.png" alt="Apache Incubator" /></a>
       </div>
       <!--Footer copyright year update credit: http://updateyourfooter.com/-->
       <div>
         Copyright &copy; 2015
         <script>new Date().getFullYear() > 2015 && document.write("-" + new Date().getFullYear());</script>
         <a href="https://www.apache.org">The Apache Software Foundation</a>.
         Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.
         <br>
         Apache and the Apache Feather logo are trademarks of The Apache Software Foundation.
       </div>
     </div>
   </div>
 </footer> <!-- GLOBAL FOOTER-->
     <script src="assets/js/bundle.min.js"></script>
   </body>
 </html>
	<!--

	-->
	<!--

	-->
	<!DOCTYPE html>
	<html lang="en">
	<!--

	-->

	<head>
	<meta charset="utf-8">
	<meta http-equiv="X-UA-Compatible" content="IE=edge">

	<title>SystemML 1.2.0 Release Notes</title>

	<meta name="description" content="Project Release Notes">
	<meta name="author" content="Apache SystemDS">

	<!-- Enable responsive viewport -->
	<meta name="HandheldFriendly" content="True">
	<meta name="viewport" content="width=device-width, initial-scale=1.0">

	<!-- You can use Open Graph tags to customize link previews.
	Learn more: https://developers.facebook.com/docs/sharing/webmasters -->
	<meta property="og:url" content="https://systemds.apache.org/" />
	<meta property="og:type" content="website" />
	<meta property="og:title" content="SystemML 1.2.0 Release Notes" />
	<meta property="og:description" content="Apache SystemDS is an open source ML system for the end-to-end data science lifecycle" />
	<meta property="og:image" content="" />

	<!-- Le HTML5 shim, for IE6-8 support of HTML elements -->
	<!--[if lt IE 9]>
	<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
	<![endif]-->

	<!-- Le styles -->
	<link rel="stylesheet" href="/assets/css/main.css">

	<!-- favicons -->
	<link rel="shortcut icon" href="/assets/img/favicon.png">
	</head> <!-- META -->
	<body class="vcard">
	<!--

	-->
	<header class="site-header site-header--not-home">
	<h1 class="logo"><a class="url" href="/"><i class="logo-mark"></i><span class="fn org">Apache SystemDS<sup
	id="trademark">™</sup></span></a></h1>
	<nav class="main-nav">
	<ul>

	<li role="presentation">


	<a href="/download" target="_self">Download</a>

	</li>

	<li role="presentation">


	<a href="/get-started" target="_self">Get Started</a>

	</li>

	<li role="presentation">

	<a class="nav-link--hover">Docs <i class="icon icon-chevron-down"></i></a>
	<ul>


	<li><a href="https://apache.github.io/systemds/" target="_blank">latest</a></li>


	<li><a href="https://systemds.apache.org/docs/3.2.0/" target="_blank">3.2.0 (stable)</a></li>


	<li><a href="https://systemds.apache.org/release-notes/systemds-release-notes-3.2.0" target="_blank">Release Notes 3.2.0</a></li>

	</ul>

	</li>

	<li role="presentation">

	<a class="nav-link--hover">Community <i class="icon icon-chevron-down"></i></a>
	<ul>


	<li><a href="/community" target="_self">Get Involved</a></li>


	<li><a href="https://issues.apache.org/jira/browse/SYSTEMDS" target="_blank">Issue Tracker</a></li>


	<li><a href="https://github.com/apache/systemds" target="_blank">Source Code</a></li>


	<li><a href="https://github.com/apache/systemds-website" target="_blank">Website Source Code</a></li>

	</ul>

	</li>

	<li role="presentation">

	<a class="nav-link--hover">Apache <i class="icon icon-chevron-down"></i></a>
	<ul>


	<li><a href="https://www.apache.org/foundation/how-it-works" target="_blank">Apache Software Foundation</a></li>


	<li><a href="https://www.apache.org/licenses/" target="_blank">Apache License</a></li>


	<li><a href="https://www.apache.org/foundation/sponsorship" target="_blank">Sponsorship</a></li>


	<li><a href="https://www.apache.org/foundation/thanks" target="_blank">Thanks</a></li>


	<li><a href="/privacy-policy" target="_self">Privacy Policy</a></li>


	<li><a href="/security" target="_self">Security</a></li>

	</ul>

	</li>

	</ul>
	</nav>
	</header> <!-- GLOBAL HEADER -->
	<!--

	-->
	<!--

	-->
	<div>
	<!--

	-->

	<section class="full-stripe full-stripe--subpage-header clear-header">
	<div class="ml-container ml-container--horizontally-center">
	<div class="col col-12 content-group content-group--center-content content-group--center-align">
	<h1>Apache SystemDS<sup id="trademark">™</sup> 2.0.0 Release Notes</h1>
	</div>
	</div>
	</section>

	<section class="full-stripe full-stripe--alternate">
	<div class="ml-container">
	<div class="col col-12 content-group content-group--medium-bottom-margin">

	<h2 id="release-notes-for-systemds-20">Release notes for SystemDS 2.0.</h2>

	<p>SystemDS 2.0 is the first major release under the new name. This release contains a major refactoring, a few major
	features, a large number of improvements and fixes, and some experimental features to better support the end-to-end
	data science lifecycle. In addition to that, this release also removes several features that are not up date and
	outdated.</p>

	<p>The major changes (compared to SystemML 1.2) include</p>

	<ul>
	<li>New mechanism for DML-bodied (script-level) builtin functions, and a wealth of new built-in functions for data preprocessing including data cleaning, augmentation and feature engineering techniques, new ML algorithms, and model debugging.</li>
	<li>Several methods for data cleaning have been implemented including multiple imputations with multivariate imputation by chained equations (MICE) and other techniques, SMOTE, an oversampling technique for class imbalance, forward and backward NA filling, cleaning using schema and length information, support for outlier detection using standard deviation and inter-quartile range, and functional dependency discovery.</li>
	<li>A complete framework for lineage tracing and reuse including support for loop deduplication, full and partial reuse, compiler assisted reuse, several new rewrites to facilitate reuse.</li>
	<li>New federated runtime backend including support for federated matrices and frames, federated builtins (transform-encode, decode etc.).</li>
	<li>Refactor compression package and add functionalities including quantization for lossy compression, binary cell operations, left matrix multiplication. [experimental]</li>
	<li>New python bindings with supports for several builtins, matrix operations, federated tensors and lineage traces.</li>
	<li>Cuda implementation of cumulative aggregate operators (cumsum, cumprod etc.)</li>
	<li>New model debugging technique with slice finder.</li>
	<li>New tensor data model (basic tensors of different value types, data tensors with schema) [experimental]</li>
	<li>Cloud deployment scripts for AWS and scripts to set up and start federated operations.</li>
	<li>Performance improvements with parallel sort, gpu cum agg, append cbind etc.</li>
	<li>Various compiler and runtime improvements including new and improved rewrites, reduced Spark context creation, new eval framework, list operations, updated native kernel libraries to name a few.</li>
	<li>New data reader/writer for json frames and support for sql as a data source.</li>
	<li>Miscellaneous improvements: improved documentation, better testing, run/release scripts, improved packaging, Docker container for systemds, support for lambda expressions, bug fixes.</li>
	<li>Removed MapReduce compiler and runtime backend, pydml parser, Java-UDF framework, script-level debugger.</li>
	<li>Deprecated ./scripts/algorithms, as those algorithms gradually will be part of SystemDS builtins.</li>
	</ul>
	</div></div></section>

	</div>


	<!--

	-->
	<footer class="site-footer">
	<div class="ml-container ml-container--horizontally-center">
	<div class="footer-col-wrapper">
	<div style="margin-top: 20px; margin-bottom: 20px;">
	<a href="https://www.apache.org"><img src="/assets/img/apache-foundation-logo.png" alt="Apache Incubator" /></a>
	</div>
	<!--Footer copyright year update credit: http://updateyourfooter.com/-->
	<div>
	Copyright © 2015
	<script>new Date().getFullYear() > 2015 && document.write("-" + new Date().getFullYear());</script>
	<a href="https://www.apache.org">The Apache Software Foundation</a>.
	Licensed under the <a href="https://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.
	<br>
	Apache and the Apache Feather logo are trademarks of The Apache Software Foundation.
	</div>
	</div>
	</div>
	</footer> <!-- GLOBAL FOOTER-->
	<script src="assets/js/bundle.min.js"></script>
	</body>
	</html>