| commit | 3f8ff967da5e8b01d9e04604290646d1b748d20d | [log] [tgz] |
|---|---|---|
| author | Matthias Boehm <mboehm7@gmail.com> | Sat Oct 31 23:05:47 2020 +0100 |
| committer | Matthias Boehm <mboehm7@gmail.com> | Sat Oct 31 23:06:44 2020 +0100 |
| tree | a5e963c9cc61668a6d7051a6addd4d97c2c9fc81 | |
| parent | e96e1cbafc53e90f64c08899cccbc1f02a9e46b3 [diff] |
[SYSTEMDS-2710] Fix K-Means and GMM builtin functions, robustness IPA This patch fixes various issues related to the recent addition of seeds to the Kmeans builtin function. First, the GMM built-in function was failing in IPA because the GMM call to Kmeans used by-position arguments and thus missed the new seed argument. We now use named parameters to guard against future additions. Second, the error handling in such cases of mismatching numbers of arguments was horrible (failing in inter-procedural analysis index out of bounds exceptions). We now use a more robust handling in IPA such that the user get the intended, error message explaining the problem. Third, the kmeans addition of seeds used a uniform random matrix and incrementally added uniform random matrices per centroid. Adding multiple uniformly distributed random variables, gives a normally distributed random variable. For K-Means initialization this is not desirable and ultimately led to the flaky python test failures.
Overview: SystemDS is a versatile system for the end-to-end data science lifecycle from data integration, cleaning, and feature engineering, over efficient, local and distributed ML model training, to deployment and serving. To this end, we aim to provide a stack of declarative languages with R-like syntax for (1) the different tasks of the data-science lifecycle, and (2) users with different expertise. These high-level scripts are compiled into hybrid execution plans of local, in-memory CPU and GPU operations, as well as distributed operations on Apache Spark. In contrast to existing systems - that either provide homogeneous tensors or 2D Datasets - and in order to serve the entire data science lifecycle, the underlying data model are DataTensors, i.e., tensors (multi-dimensional arrays) whose first dimension may have a heterogeneous and nested schema.
Quick Start Install, Quick Start and Hello World
Documentation: SystemDS Documentation
Python Documentation Python SystemDS Documentation
Issue Tracker Jira Dashboard
Status and Build: SystemDS is renamed from SystemML which is an Apache Top Level Project. To build from source visit SystemDS Install from source
