tree eb15d687d52dd935572487258f315a3cd4065da5
parent d4839b42c5c071df34ba25201d0708595494af06
author baunsgaard <baunsgaard@tugraz.at> 1607431607 +0100
committer baunsgaard <baunsgaard@tugraz.at> 1610123691 +0100

[SYSTEMDS-2787] Compression Steps Reorganization

This commit contains various changes (some massive ones).
The biggest change is the ordering of compression steps, of which
now we classify first on a sample of the data. Since this was experimented
to be 10-30% faster. Furthermore this allows us to try compression at a
lower cost if the compression is not valid to perform.

Overall Compression time for covtype went from
 - ~1.0 to 0.36 sec (including read from disk) 0.11 sec compression

Furthermore now unlike before the transpose is heuristically chosen, Since
it is more efficient not to transpose the matrix for compression in some
cases.

- Compressed Sparse matrix multiplication fix
- modified matrix multiplication to push down information of
  transposing to the ba+* op. to allow not decompressing the matrix.
- Configuration option of enabling and disabling overlapping compression.
- decompress row section direct access to the matrix block not using
  quick set/get.
- adding safe boolean to decompress to specify if management of
  nnz should be done. This allows the decompression of intermediates at
  near half the computation cost.
- Add configuration for sampling ratio default 0.01 but with a minimum
  sample size of 2000 elements.
- DML Config settings for Cocode-Compression method default to COST
- add support for right sparse matrix multiplication with overlapping
  output. Further improvements are on the way.
- Compression statistics are added when statistics and compression is
  enabled
- Readers for extracting bitmaps are optimized for either transposed or
  untransposed matrices giving 5-15% improved performance.
- Hashmaps are modified to improve insertion time since previously they
  would hash values twice 10% improved performance. furthermore the
  default sizes are modified to start smaller.
- Additional tests for multipication to cover different edge cases.