| commit | 12cdc89a31d1b33728bf074005aa426f1beac11d | [log] [tgz] |
|---|---|---|
| author | baunsgaard <baunsgaard@tugraz.at> | Tue Dec 08 13:46:47 2020 +0100 |
| committer | baunsgaard <baunsgaard@tugraz.at> | Fri Jan 08 17:34:51 2021 +0100 |
| tree | eb15d687d52dd935572487258f315a3cd4065da5 | |
| parent | d4839b42c5c071df34ba25201d0708595494af06 [diff] |
[SYSTEMDS-2787] Compression Steps Reorganization This commit contains various changes (some massive ones). The biggest change is the ordering of compression steps, of which now we classify first on a sample of the data. Since this was experimented to be 10-30% faster. Furthermore this allows us to try compression at a lower cost if the compression is not valid to perform. Overall Compression time for covtype went from - ~1.0 to 0.36 sec (including read from disk) 0.11 sec compression Furthermore now unlike before the transpose is heuristically chosen, Since it is more efficient not to transpose the matrix for compression in some cases. - Compressed Sparse matrix multiplication fix - modified matrix multiplication to push down information of transposing to the ba+* op. to allow not decompressing the matrix. - Configuration option of enabling and disabling overlapping compression. - decompress row section direct access to the matrix block not using quick set/get. - adding safe boolean to decompress to specify if management of nnz should be done. This allows the decompression of intermediates at near half the computation cost. - Add configuration for sampling ratio default 0.01 but with a minimum sample size of 2000 elements. - DML Config settings for Cocode-Compression method default to COST - add support for right sparse matrix multiplication with overlapping output. Further improvements are on the way. - Compression statistics are added when statistics and compression is enabled - Readers for extracting bitmaps are optimized for either transposed or untransposed matrices giving 5-15% improved performance. - Hashmaps are modified to improve insertion time since previously they would hash values twice 10% improved performance. furthermore the default sizes are modified to start smaller. - Additional tests for multipication to cover different edge cases.
Overview: SystemDS is a versatile system for the end-to-end data science lifecycle from data integration, cleaning, and feature engineering, over efficient, local and distributed ML model training, to deployment and serving. To this end, we aim to provide a stack of declarative languages with R-like syntax for (1) the different tasks of the data-science lifecycle, and (2) users with different expertise. These high-level scripts are compiled into hybrid execution plans of local, in-memory CPU and GPU operations, as well as distributed operations on Apache Spark. In contrast to existing systems - that either provide homogeneous tensors or 2D Datasets - and in order to serve the entire data science lifecycle, the underlying data model are DataTensors, i.e., tensors (multi-dimensional arrays) whose first dimension may have a heterogeneous and nested schema.
Quick Start Install, Quick Start and Hello World
Documentation: SystemDS Documentation
Python Documentation Python SystemDS Documentation
Issue Tracker Jira Dashboard
Status and Build: SystemDS is renamed from SystemML which is an Apache Top Level Project. To build from source visit SystemDS Install from source
