[SYSTEMDS-3444][SYSTEMDS-2699] Compressed I/O
This commit is a major overhaul of the writing and reading of compressed
matrices.

The design is now changed to write dictionaries separately and reading
in both local and spark is working. Where a spark read will combine the
dictionaries written in a distributed execution.

Also contained in this PR is updates and refinements of the schema apply
that now in a fused manner of update and apply can compress a matrix
single-threaded at around 669MiB/s and multi-threaded 2GiB.
This is done via first a full materialization of the compressed format
in memory meaning that there is potential for further speedup if we
relocate this compression on the IO path. But this is left for future
work.

One major improvement that makes our default compression faster as well is
ACountHashMap.java now generalize the counting hashmap between the
co-coded columns and single columns and optimized the increment calls
for improved performance.

The Co-Coding algorithm has also been slightly modified in this PR
to add a small fraction to the cost of column groups depending on their
column indexes. this makes it so that columns with the same cost are sorted
based on their average column indexes, and in turn, improve the compression
time of highly compressible data such as binary or ultra-sparse data.

The PR also fixed the Nan Compression to not be treated specially to
allow us to compress matrices containing Nan and then afterward we can
replace Nan in an already compressed representation. Before the behavior
was to replace all Nan Values with 0.

Future work is to parallelize the reading of compressed matrices, which
currently only is single threaded in the CP case.

In the serialization performance benchmark, this commit moves the size
calculation outside of the timed part. and improves the general code
evaluation of individual functions.

Closes #1880
137 files changed
tree: e5e0c50dcc9f9674edcbf500674ba71c6e5de967
  1. .github/
  2. .mvn/
  3. bin/
  4. conf/
  5. dev/
  6. docker/
  7. docs/
  8. scripts/
  9. src/
  10. .asf.yaml
  11. .gitattributes
  12. .gitignore
  13. .gitmodules
  14. CITATION
  15. CONTRIBUTING.md
  16. LICENSE
  17. NOTICE
  18. pom.xml
  19. README.md
README.md

Apache SystemDS

Overview: SystemDS is an open source ML system for the end-to-end data science lifecycle from data integration, cleaning, and feature engineering, over efficient, local and distributed ML model training, to deployment and serving. To this end, we aim to provide a stack of declarative languages with R-like syntax for (1) the different tasks of the data-science lifecycle, and (2) users with different expertise. These high-level scripts are compiled into hybrid execution plans of local, in-memory CPU and GPU operations, as well as distributed operations on Apache Spark. In contrast to existing systems - that either provide homogeneous tensors or 2D Datasets - and in order to serve the entire data science lifecycle, the underlying data model are DataTensors, i.e., tensors (multi-dimensional arrays) whose first dimension may have a heterogeneous and nested schema.

Quick Start Install, Quick Start and Hello World

Documentation: SystemDS Documentation

Python Documentation Python SystemDS Documentation

Issue Tracker Jira Dashboard

Status and Build: SystemDS is renamed from SystemML which is an Apache Top Level Project. To build from source visit SystemDS Install from source

Build Documentation LicenseCheck Java Tests Python Test