| commit | e73d1964bf6ba78fbc23541dd8bb3f87131a1d6d | [log] [tgz] |
|---|---|---|
| author | Baunsgaard <baunsgaard@tugraz.at> | Tue Jul 14 11:51:46 2020 +0200 |
| committer | baunsgaard <baunsgaard@tugraz.at> | Sun Sep 06 12:18:48 2020 +0200 |
| tree | 42e80ddd7f8bad3372f016c45299d6a5f59d56dd | |
| parent | 592039e17bdc2b6ba5e328651b15862ac5410f98 [diff] |
[SYSTEMDS-373] Fix error in new compression Remove Quan ColGroup because of new Dictionary technique. - FIX Quantile and CentralMoment for DDC ColGroups - compression tests remove redundant tests in Par Compressed Matrix and Remove dependency on AutomatedTestBase - Compare matrix now only count the first 50 discrepancies - Rounding improvement in Quantized representation - Out commented some debugging - Fix count distinct compressed test works, but is not comparable to count distinct non lossy. since the number of distinct values can both grow and shrink - Update Dictionary to not copy a copy for lossy dictionaries on scalar operations - Dictionary constructor in all colGroups - QDictionary Optimize scalar operator - Improve scalar operation - Fix leftMultByRowVector OLE in case of only one value! - Improve ColSum - Improvements in performance of unary aggregates - ColGroupValue fix for single column colGroups - disallowing ColGroup construction with double array - Dictionary documentation - Fixed docs for ColGroupValue - Lossy Bitmap fix and improved performance - revert DMLScript entry file - call BitMap lossy for 8Bit bitmap - DDC rework, moving dublicate code to DDC from DDC1 and 2, furthermore general fixes and optimizations in tests and all ColGroups for better lossy handling - DDC docs fix - common sum and ColSum without performance degradation - Fix bug in Unary aggregate where output from uncompressed ColGroup was empty - Move SkipList boolean to compression Settings - Dictionary correcting Override keyword - A dictionary not I Dictionary - in sum all rows to Double dictionary always reuse double array - Allowing more tests to execute. Compression - Parallel scalar operations - shallow serialize - rename AbstractBitmap to ABitmap
Overview: SystemDS is a versatile system for the end-to-end data science lifecycle from data integration, cleaning, and feature engineering, over efficient, local and distributed ML model training, to deployment and serving. To this end, we aim to provide a stack of declarative languages with R-like syntax for (1) the different tasks of the data-science lifecycle, and (2) users with different expertise. These high-level scripts are compiled into hybrid execution plans of local, in-memory CPU and GPU operations, as well as distributed operations on Apache Spark. In contrast to existing systems - that either provide homogeneous tensors or 2D Datasets - and in order to serve the entire data science lifecycle, the underlying data model are DataTensors, i.e., tensors (multi-dimensional arrays) whose first dimension may have a heterogeneous and nested schema.
Quick Start Install, Quick Start and Hello World
Documentation: SystemDS Documentation
Python Documentation Python SystemDS Documentation
Issue Tracker Jira Dashboard
Status and Build: SystemDS is still in pre-alpha status. The original code base was forked from Apache SystemML 1.2 in September 2018. We will continue to support linear algebra programs over matrices, while replacing the underlying data model and compiler, as well as substantially extending the supported functionalities. Until the first release, you can build your own snapshot via Apache Maven: mvn clean package -P distribution.
