blob: 00af4e31f66ea56dd8c82c79abf165f974d0d32e [file] [log] [blame]
<!DOCTYPE html>
<html lang="en">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>SystemDS 3.1.0 Release Notes</title>
<meta name="description" content="Project Release Notes">
<meta name="author" content="Apache SystemDS">
<!-- Enable responsive viewport -->
<meta name="HandheldFriendly" content="True">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<!-- You can use Open Graph tags to customize link previews.
Learn more: -->
<meta property="og:url" content="" />
<meta property="og:type" content="website" />
<meta property="og:title" content="SystemDS 3.1.0 Release Notes" />
<meta property="og:description" content="Apache SystemDS is an open source ML system for the end-to-end data science lifecycle" />
<meta property="og:image" content="" />
<!-- Le HTML5 shim, for IE6-8 support of HTML elements -->
<!--[if lt IE 9]>
<script src=""></script>
<!-- Le styles -->
<link rel="stylesheet" href="/assets/css/main.css">
<!-- favicons -->
<link rel="shortcut icon" href="/assets/img/favicon.png">
</head> <!-- META -->
<body class="vcard">
<header class="site-header site-header--not-home">
<h1 class="logo"><a class="url" href="/"><i class="logo-mark"></i><span class="fn org">Apache SystemDS<sup
<nav class="main-nav">
<li role="presentation">
<a href="/download" target="_self">Download</a>
<li role="presentation">
<a href="/get-started" target="_self">Get Started</a>
<li role="presentation">
<a class="nav-link--hover">Docs <i class="icon icon-chevron-down"></i></a>
<li><a href="" target="_blank">latest</a></li>
<li><a href="" target="_blank">3.1.0 (stable)</a></li>
<li><a href="" target="_blank">Release Notes 3.1.0</a></li>
<li role="presentation">
<a class="nav-link--hover">Community <i class="icon icon-chevron-down"></i></a>
<li><a href="/community" target="_self">Get Involved</a></li>
<li><a href="" target="_blank">Issue Tracker</a></li>
<li><a href="" target="_blank">Source Code</a></li>
<li><a href="" target="_blank">Website Source Code</a></li>
<li role="presentation">
<a class="nav-link--hover">Apache <i class="icon icon-chevron-down"></i></a>
<li><a href="" target="_blank">Apache Software Foundation</a></li>
<li><a href="" target="_blank">Apache License</a></li>
<li><a href="" target="_blank">Sponsorship</a></li>
<li><a href="" target="_blank">Thanks</a></li>
<li><a href="/privacy-policy" target="_self">Privacy Policy</a></li>
<li><a href="/security" target="_self">Security</a></li>
</header> <!-- GLOBAL HEADER -->
<section class="full-stripe full-stripe--subpage-header clear-header">
<div class="ml-container ml-container--horizontally-center">
<div class="col col-12 content-group content-group--center-content content-group--center-align">
<h1>Apache SystemDS<sup id="trademark">&trade;</sup> 3.1.0 Release Notes</h1>
<section class="full-stripe full-stripe--alternate">
<div class="ml-container">
<div class="col col-12 content-group content-group--medium-bottom-margin">
<h2 id="release-notes-for-systemds-310">Release notes for SystemDS 3.1.0</h2>
<p>SystemDS 3.1 is a minor release. Release 3.1 contains new features and major improvements to existing features.</p>
<h3 id="features-and-improvements">Features and Improvements</h3>
<li>Performance codegen kmeans mnist80m w/ compression</li>
<li>Prefetch instruction</li>
<li>Broadcast instruction</li>
<li>Create apply functions for cleaning primitives</li>
<li>LogicalEnumerator change with transitions concept and cleanups</li>
<li>Flatten the nested loop for parallel pipelines execution</li>
<li>Adding apply_pipeline() builtin for cleaning pipelines API</li>
<li>Release docker images with GitHub actions</li>
<li>Add monitoring tool testing workflows</li>
<li>Asynchronous Execution and Persist Spark Transformations</li>
<li>Future-based asynchronous execution of Spark actions</li>
<li>New operator linearization order to maximize inter-operator parallelism</li>
<li>Lineage-based reuse of Spark actions</li>
<li>Push down rmvar instructions for asynchronous instructions</li>
<li>Lineage-based reuse of asynchronous operators</li>
<li>Persist and reuse Spark RDDs</li>
<li>Refactor to add LOP rewrite step in compilation</li>
<li>Federated Compression Instruction</li>
<li>CLA IO Compressed Matrices</li>
<li>Compressed Max/Min Index support.</li>
<li>Federated async compression</li>
<li>Federated Workload-aware Compression</li>
<li>Python 3.9 support</li>
<li>Parallel Compressed Encode</li>
<li>New builtin function auc (area under ROC curve)</li>
<li>Unique() function for performance</li>
<h3 id="bug">Bug</h3>
<li>Fix memory configuration in</li>
<li>OOM Error On Binary Write</li>
<li>Out of memory error</li>
<li>CLA Improved Run estimation</li>
<li>AttributeError: Function definition not found</li>
<li>applySchema built-in to set the schema of frame from DML</li>
<li>CSR TSMM left with filled rows bug</li>
<li>Sparse TSMM dense row blocks CSR</li>
<li>py4j.Py4JException: Method exceptionString([class org.apache.spark.SparkConf]) does not exist</li>
<li>MatrixBlock size using CSR when allowed</li>
<li>Federated Nan Values</li>
<li>countDistinctApprox() operation in AggregateUnaryCPInstruction is inefficient for row/col aggregations</li>
<li>Correct the release artifact generation date</li>
<li>Log4j incompatible dependencies</li>
<li>ConcurrentModificationException in federated execution</li>
<li>Jackson Core missing for json writing and reading in reduced binary</li>
<li>Fix Java doc warnings</li>
<li>Enque output not UTF-8 python</li>
<li>Read CSV directly without mtd python</li>
<li>Python configuration not loading defaults</li>
<li>Matrix Multiplication crash in Spark</li>
<li>Pipelines failing in Hybrid execution</li>
<li>Built-in tests failure in Git actions</li>
<li>Cleaning Pipelines failed with No space left on device</li>
<li>IndexOutOfBounds due to int overflow on replace</li>
<li>Cleaning Pipelines: Replace function failure in hybrid execution</li>
<li>Cleaning Pipelines: Block Sizes mismatch</li>
<li>Cleaning Pipelines in hybrid mode: Invalid block dimensions error</li>
<li>Federated Statistics print in non federated scenario</li>
<li>Spark Aggregate Binary operations parse to Fed instruction</li>
<li>FederationUtils.bindResponses causes out of memory because of sparse matrices.</li>
<li>Python IDE test Docs fail</li>
<li>MSVM robustness for non-existing classes</li>
<li>CLA ArrayOutOfBounds in sample</li>
<li>CLA Invalid Unique estimate DDC</li>
<li>Federated read cache cannot be disabled</li>
<li>Monitoring Heavy hitters not always correct list</li>
<li>Slow Federated Mlogreg on Criteo (dummy-coded)</li>
<li>Incorrect warning when reading scalars</li>
<li>Spark with default settings</li>
<li>Cleaning Pipelines: Task Parallel Experiments failing in spark mode</li>
<li>Unique() crashes with iterator EOF on vectors with &gt;1K distinct items</li>
<li>Perftest: Mlogreg on 1M_1k_dense w/ unnecessary spark jobs</li>
<li>Perftest: lmDS on 1M_1k_dense with unnecessary spark tsmm</li>
<li>Java doc warnings</li>
<footer class="site-footer">
<div class="ml-container ml-container--horizontally-center">
<div class="footer-col-wrapper">
<div style="margin-top: 20px; margin-bottom: 20px;">
<a href=""><img src="/assets/img/apache-foundation-logo.png" alt="Apache Incubator" /></a>
<!--Footer copyright year update credit:>
Copyright &copy; 2015
<script>new Date().getFullYear() > 2015 && document.write("-" + new Date().getFullYear());</script>
<a href="">The Apache Software Foundation</a>.
Licensed under the <a href="">Apache License, Version 2.0</a>.
Apache and the Apache Feather logo are trademarks of The Apache Software Foundation.
</footer> <!-- GLOBAL FOOTER-->
<script src="assets/js/bundle.min.js"></script>