[SYSTEMDS-3444][SYSTEMDS-2699] Compressed I/O This commit is a major overhaul of the writing and reading of compressed matrices. The design is now changed to write dictionaries separately and reading in both local and spark is working. Where a spark read will combine the dictionaries written in a distributed execution. Also contained in this PR is updates and refinements of the schema apply that now in a fused manner of update and apply can compress a matrix single-threaded at around 669MiB/s and multi-threaded 2GiB. This is done via first a full materialization of the compressed format in memory meaning that there is potential for further speedup if we relocate this compression on the IO path. But this is left for future work. One major improvement that makes our default compression faster as well is ACountHashMap.java now generalize the counting hashmap between the co-coded columns and single columns and optimized the increment calls for improved performance. The Co-Coding algorithm has also been slightly modified in this PR to add a small fraction to the cost of column groups depending on their column indexes. this makes it so that columns with the same cost are sorted based on their average column indexes, and in turn, improve the compression time of highly compressible data such as binary or ultra-sparse data. The PR also fixed the Nan Compression to not be treated specially to allow us to compress matrices containing Nan and then afterward we can replace Nan in an already compressed representation. Before the behavior was to replace all Nan Values with 0. Future work is to parallelize the reading of compressed matrices, which currently only is single threaded in the CP case. In the serialization performance benchmark, this commit moves the size calculation outside of the timed part. and improves the general code evaluation of individual functions. Closes #1880
Overview: SystemDS is an open source ML system for the end-to-end data science lifecycle from data integration, cleaning, and feature engineering, over efficient, local and distributed ML model training, to deployment and serving. To this end, we aim to provide a stack of declarative languages with R-like syntax for (1) the different tasks of the data-science lifecycle, and (2) users with different expertise. These high-level scripts are compiled into hybrid execution plans of local, in-memory CPU and GPU operations, as well as distributed operations on Apache Spark. In contrast to existing systems - that either provide homogeneous tensors or 2D Datasets - and in order to serve the entire data science lifecycle, the underlying data model are DataTensors, i.e., tensors (multi-dimensional arrays) whose first dimension may have a heterogeneous and nested schema.
Quick Start Install, Quick Start and Hello World
Documentation: SystemDS Documentation
Python Documentation Python SystemDS Documentation
Issue Tracker Jira Dashboard
Status and Build: SystemDS is renamed from SystemML which is an Apache Top Level Project. To build from source visit SystemDS Install from source