| <!-- |
| {% comment %} |
| Licensed to the Apache Software Foundation (ASF) under one or more |
| contributor license agreements. See the NOTICE file distributed with |
| this work for additional information regarding copyright ownership. |
| The ASF licenses this file to you under the Apache License, Version 2.0 |
| (the "License"); you may not use this file except in compliance with |
| the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| {% end comment %} |
| --> |
| |
| --- |
| GENERAL NOTES: |
| * Assign IDs for epics by default with a key range of 10 |
| * Keep epic tasks in the key range, otherwise create new epic |
| --- |
| |
| SYSTEMDS-10 Compiler Rework / Misc |
| * 11 Support DML-bodied builtin functions OK |
| * 12 Remove unnecessary HOP/LOP indirections OK |
| * 13 Refactoring test cases into component/integration OK |
| * 14 Complete removal of external functions from all scripts |
| * 15 Travis integration w/ subset of tests OK (removed for Github Actions) |
| * 16 Remove instruction patching |
| * 17 Refactoring of program block hierarchy OK |
| * 18 Improve API for new dml-bodied builtin functions OK |
| * 19 Cleanup pass cpvar+rmvar to mvvar instructions OK |
| |
| SYSTEMDS-20 New Data Model |
| * 21 Finalize dense tensor blocks OK |
| * 22 Sparse double/float tensor blocks |
| * 23 Sparse int/bool tensor blocks |
| * 24 Initial tensor dml integration OK |
| * 25 Initial data tensor implementation |
| * 26 Non-zero default value for sparse (row/col) |
| * 27 Tensor buffer pool integration |
| * 28 Tensor local readers and writers (textcell, binary) OK |
| * 29 Tensor local readers and writers (linearize csv) |
| |
| SYSTEMDS-30 Builtin and Packaging |
| * 31 Shell script for local runs OK |
| * 32 Shell script for spark runs OK |
| * 33 Cleanup hadoop dependency for local runs |
| * 34 Wrapper blocks for sequence files |
| * 35 Replace unnecessary dependencies w/ custom |
| * 36 Shell script for AWS execution OK |
| * 37 Cleanup mvn package, compile, test OK |
| * 38 Runscript rename, improve, fix cygwin OK |
| |
| SYSTEMDS-40 Preprocessing builtins |
| * 41 Add new winsorize builtin function OK |
| * 42 Add new outlier builtin function OK |
| * 43 Add new scale builtin function OK |
| * 44 SotA normalization primitives |
| * 45 SotA outlier detection primitives |
| * 46 Generalization of quantiles |
| * 47 Add new normalize builtin function OK |
| * 48 Extended transform encode/apply: feature hashing OK |
| |
| SYSTEMDS-50 I/O Formats |
| * 51 Support for homogeneous JSON (local/distributed) |
| * 52 Support for libsvm files (local/distributed) |
| * 53 New sql data source (local, distributed) |
| * 54 Support for is.na, is.nan, is.infinite OK |
| * 55 Cleanup file formats, binary cell, input/output infos OK |
| * 56 Consolidate single/multi-threaded readers |
| * 57 Support for hdf5 files (local/distributed) |
| |
| SYSTEMDS-60 Update SystemML improvements |
| * 61 Take over cumulative aggregate improvements OK |
| * 62 Take over sparsity estimation improvements OK |
| * 63 Take over fixes for transform w/ binning |
| |
| SYSTEMDS-70 Lineage Tracing and Reuse OK |
| * 71 Basic collection of lineage traces OK |
| * 72 Serialization and deserialization of lineage traces OK |
| * 73 Deduplication of lineage traces OK |
| (loops, nested loops, branchless loops) OK |
| * 74 Performance features lineage tracing OK |
| * 75 Reuse cache based on lineage traces OK |
| * 76 Generate runtime plan from lineage trace OK |
| * 77 New builtin function for obtaining lineage OK |
| * 78 Extended lineage tracing (parfor, funs, codegen) OK |
| * 79 Reuse cache configuration options (cmd line options) OK |
| |
| SYSTEMDS-80 Improved distributed operations |
| * 81 Avoid unnecessary replication on rmm |
| * 82 Avoid unnecessary replication on binary |
| |
| SYSTEMDS-100 Various Fixes |
| * 101 Fix spark quantiles w/ multiple queries OK |
| * 102 Fix parser issue after refactoring OK |
| * 103 Fix handling of builtin functions w/ matching udfs OK |
| * 104 Fix failing tests due to incorrect parfor parameters OK |
| * 105 Fix all application/function tests (various issues) OK |
| * 106 Fix correctness of as.integer for negative numbers OK |
| * 107 Fix correctness IPA check dimension-preserving OK |
| * 108 Fix codegen optimizer (early costing abort) OK |
| * 109 Fix update-in-place w/ udf function calls OK |
| |
| SYSTEMDS-110 New Builtin Functions |
| * 111 Time builtin function for script-level measurements OK |
| * 112 Image data augmentation builtin functions OK |
| * 113 Builtin functions for linear regression algorithms OK |
| * 114 Builtin function for stepwise regression OK |
| * 115 Builtin function for model debugging (slice finder) OK |
| * 116 Builtin function for kmeans OK |
| * 117 Builtin function for lm cross validation OK |
| * 118 Builtin function for hyperparameter grid search |
| * 119 Builtin functions for l2svm and msvm OK |
| |
| SYSTEMDS-120 Performance Features |
| * 121 Avoid spark context creation on parfor result merge OK |
| * 122 Reduce thread contention on parfor left indexing OK |
| * 123 Avoid unnecessary spark context creation on explain OK |
| * 124 Improved cbind performance (small rhs) OK |
| * 125 Parallel Sort (previously 191) OK |
| |
| SYSTEMDS-130 IPA and Size Propagation |
| * 131 New IPA pass for function call forwarding OK |
| * 132 Size propagation via data characteristics |
| * 133 Hop Size Propagation for Tensors |
| * 134 Increased repretition and fixpoint detection OK |
| * 135 Fix IPA scalar propagation w/ named arguments OK |
| |
| SYSTEMDS-140 Distributed Tensor Operations |
| * 141 Infrastructure for spark tensor operations |
| * 142 Basis tensor operations (datagen, aggregate, elementwise) |
| |
| SYSTEMDS-150 Releases |
| * 151 Fixing native BLAS instructions (MKL,OpenBLAS) OK |
| * 152 Run script documentation OK |
| * 153 convert all occurrences SystemML/sysml to SystemDS/sysds OK |
| * 154 fix license headers in all touched files OK |
| * 155 import release scripts from SystemML repo OK |
| * 156 initial codestyle eclipse template OK |
| * 157 Shell script simplification, cleanups and fixes for 0.2 OK |
| |
| SYSTEMDS-160 Tensor Compiler/Runtime |
| * 161 Local readers and writers for tensors (all formats) |
| * 162 Spark binary tensor instructions |
| |
| SYSTEMDS-170 Lineage full and partial reuse |
| * 171 Initial version of partial rewrites OK |
| * 172 Parfor integration (blocked waiting for results) OK |
| * 173 Improved cost estimates OK |
| * 174 Reuse rewrite for rbind/cbind-tsmm/ba+* OK |
| * 175 Refactoring of lineage rewrite code OK |
| * 176 Reuse rewrite for cbind/rbind-elementwise */+ |
| * 177 Reuse rewrite for aggregate OK |
| * 178 Compiler assisted reuse (eg. CV, lmCG) |
| * 179 Lineage handling for lists OK |
| |
| SYSTEMDS-180 New Builtin Functions II OK |
| * 181 Builtin function for naive bayes OK |
| * 182 Builtin function for typeof (frame schema detection) OK |
| * 183 Builtin function detectSchema OK |
| * 184 Builtin function GNMF OK |
| * 185 Builtin function PNMF OK |
| * 186 Builtin function ALS-DS OK |
| * 187 Builtin function ALS-CG OK |
| * 188 Builtin function ALS OK |
| * 189 Builtin function mice (chained equation imputation) OK |
| |
| SYSTEMDS-190 New Builtin Functions III |
| * 191 Builtin function multi logreg OK |
| * 192 Extended cmdline interface -b for calling builtin functions |
| * 193 New outlierBySds builtin function (delete row/cell) OK |
| * 194 New outlierByIQR builtin function (delete row/cell) OK |
| * 195 Builtin function mice for nominal features OK |
| * 196 Builtin function intersect (set intersection) OK |
| * 197 Builtin function for functional dependency discovery OK |
| * 198 Extended slice finding (classification) OK |
| * 199 Builtin function Multinominal Logistic Regression Predict OK |
| * 200 Builtin function Gaussian Mixture Model OK |
| |
| SYSTEMDS-200 Various Fixes |
| * 201 Fix spark append instruction zero columns OK |
| * 202 Fix rewrite split DAG multiple removeEmpty (pending op) OK |
| * 203 Fix codegen outer compilation (MV operations) OK |
| * 204 Fix rewrite simplify sequences of binary comparisons OK |
| * 205 Fix scoping of builtin dml-bodied functions (vs user-defined) |
| * 206 Fix codegen outer template compilation (tsmm) OK |
| * 207 Fix builtin function call hoisting from expressions OK |
| * 208 Fix bufferpool leak (live var analysis and createvar) OK |
| * 209 Fix performance sparse M-CV elementwise multiply OK |
| |
| SYSTEMDS-210 Extended lists Operations |
| * 211 Cbind and Rbind over lists of matrices OK |
| * 212 Creation of empty lists OK |
| * 213 Add entries to lists OK |
| * 214 Removal of entries from lists OK |
| * 215 List expansion during dynamic recompilation OK |
| * 216 New rewrite for tsmm/mm over list of folds OK |
| |
| SYSTEMDS-220 Federated Tensors and Instructions |
| * 221 Initial infrastructure federated operations OK |
| * 222 Federated matrix-vector multiplication OK |
| * 223 Federated unary aggregates OK |
| * 224 Federated frames OK |
| * 225 Federated elementwise operations OK |
| * 226 Federated rbind and cbind OK |
| * 227 Federated worker setup and infrastructure OK |
| * 228 Federated matrix-matrix multiplication OK |
| * 229 Federated transform functionality |
| |
| SYSTEMDS-230 Lineage Integration |
| * 231 Use lineage in buffer pool |
| * 232 Lineage of code generated operators OK |
| * 233 Lineage cache and reuse of function results OK |
| * 234 Lineage tracing for spark instructions OK |
| * 235 Lineage tracing for remote-spark parfor OK |
| * 236 Extended multi-level lineage cache for statement blocks OK |
| * 237 Unmark loop dependent operations OK |
| * 238 Robustness parfor lineage merge (flaky tests) OK |
| * 239 Make lineage accessible through python bindings OK |
| |
| SYSTEMDS-240 GPU Backend Improvements |
| * 241 Dense GPU cumulative aggregates OK |
| |
| SYSTEMDS-250 Extended Slice Finding |
| * 251 Alternative slice enumeration approach OK |
| * 252 Initial data slicing implementation Python OK |
| * 253 Distributed slicing algorithms (task/data parallel) OK |
| * 254 Consolidation and fixes distributed slice finding OK |
| |
| SYSTEMDS-260 Misc Tools |
| * 261 Stable marriage algorithm OK |
| * 262 Data augmentation tool for data cleaning OK |
| * 263 ONNX graph importer (Python API, docs, tests) OK |
| * 264 ONNX graph exporter |
| * 265 Entity resolution pipelines and primitives OK |
| |
| SYSTEMDS-270 Compressed Matrix Blocks |
| * 271 Reintroduce compressed matrix blocks from SystemML OK |
| * 272 Simplify and speedup compression tests OK |
| * 273 Refactor compressed Matrix Block to simplify responsibilities OK |
| * 273a Redesign allocation of ColGroups in ColGroupFactory |
| * 274 Make the DDC Compression dictionary share correctly OK |
| * 275 Include compressionSettings in DMLConfiguration |
| * 276 Allow Uncompressed Columns to be in sparse formats |
| * 277 Sampling based estimators fix |
| * 278 Compression-CoCode algorithm optimization |
| * 278a Return ColGroups estimated compression ratio to Factory |
| * 279 Add missing standard lossless compression techniques |
| * 279a ColGroup FOR (Frame of reference) encoding |
| * 279b ColGroup DEL (Delta) encoding |
| * MINOR Reduce memory usage for compression statistics. |
| * MINOR Make ContainsAZero() method in UncompressedBitMap |
| |
| SYSTEMDS-280 New Rewrites |
| * 281 New rewrites sum(rmEmpty), sum(table) OK |
| * 282 New rewrite binary add chain to nary plus OK |
| |
| SYSTEMDS-290 Misc Compiler Features |
| * 291 Eval function calls of dml-bodied builtin functions OK |
| * 292 Eval function calls of native builtin functions (genate fun) |
| * 293 Compiler rewrites for eval to normal function calls |
| * 294 DML-bodied builtin functions with frames as main input |
| |
| SYSTEMDS-300 Github Workflow/Actions |
| * 301 Initial Setup of GitHub Actions for automatic testing OK |
| * 302 Workflow for Application tests OK |
| * 303 Workflow for Function tests OK |
| * 304 Workflow make documentation OK |
| * 305 Workflow run docker containers for speedup OK |
| |
| SYSTEMDS-310 Python Bindings |
| * 311 Initial Python Binding for federated execution OK |
| * 312 Python 3.6 compatibility OK |
| * 313 Python Documentation upload via Github Actions OK |
| * 314 Python SystemDS context manager OK |
| * 314a Change py4j interaction using py4j for termination not stdin |
| * 315 Python Federated Matrices Tests OK |
| * 316 Extended Python API (rand, lm, mm) OK |
| * 317 Python Rerun with different inputs without recompilation |
| * 318 Python Multiple outputs |
| * 319 Python Caching of intermediates |
| |
| SYSTEMDS-320 Merge SystemDS into Apache SystemML OK |
| * 321 Merge histories of SystemDS and SystemML OK |
| * 322 Change global package names OK |
| * 323 Fix licenses and notice file OK |
| |
| SYSTEMDS-330 Lineage Tracing, Reuse and Integration |
| * 331 Cache and reuse scalar outputs (instruction and multi-level) OK |
| * 332 Parfor integration with multi-level reuse OK |
| * 333 Improve cache eviction with actual compute time OK |
| * 334 Cache scalars only with atleast one matrix inputs |
| * 335 Weighted eviction policy (function(size,computetime,LRU time)) OK |
| * 336 Better use of cache status to handle multithreading |
| * 337 Adjust disk I/O speed by recording actual time taken OK |
| * 338 Extended lineage tracing (rmEmpty, lists), partial rewrites OK |
| * 339 Lineage tracing robustness (indexed updates, algorithms) |
| |
| SYSTEMDS-340 Compiler Assisted Lineage Caching and Reuse |
| * 341 Finalize unmarking of loop dependent operations |
| * 342 Mark functions as last-use to enable early eviction |
| * 343 Identify equal last level HOPs to ensure SB-level reuse |
| * 344 Unmark functions/SBs containing non-determinism for caching OK |
| * 345 Compiler assisted cache configuration |
| |
| SYSTEMDS-350 Data Cleaning Framework |
| * 351 New builtin function for error correction by schema OK |
| * 352 New builtin function for error correction by length Ok |
| |
| SYSTEMDS-360 Privacy/Data Exchange Constraints |
| * 361 Initial privacy meta data (compiler/runtime) OK |
| * 362 Runtime privacy propagation |
| * 363 Compile-time privacy propagation |
| * 364 Error handling violated privacy constraints OK |
| * 365 Extended privacy/data exchange constraints OK |
| |
| SYSTEMDS-370 Lossy Compression Blocks |
| * 371 ColGroup Quantization OK (Naive Q8) |
| * 321 ColGroup Base Data change (from Double to ??) |
| |
| SYSTEMDS-380 Memory Footprint |
| * 381 Matrix Block Memory footprint update |
| |
| SYSTEMDS-390 New Builtin Functions IV |
| * 391 New GLM builtin function (from algorithms) OK |
| * 392 Builtin function for missing value imputation via FDs OK |
| * 393 Builtin to find Connected Components of a graph OK |
| * 394 Builtin for one-hot encoding of matrix (not frame), see table OK |
| * 395 SVM rework and utils (confusionMatrix, msvmPredict) OK |
| * 396 Builtin for counting number of distinct values OK |
| * 397 Algorithm for neural collaborative filtering (NCF) OK |
| |
| SYSTEMDS-400 Spark Backend Improvements |
| * 401 Fix output block indexes of rdiag (diagM2V) OK |
| |
| SYSTEMDS-410 Lineage Tracing, Reuse and Integration II |
| * 411 Improved handling of multi-level cache duplicates OK |
| * 412 Robust lineage tracing (non-recursive, parfor) OK |
| * 413 Cache and reuse MultiReturnBuiltin instructions OK |
| * 414 New rewrite for PCA --> lmDS pipeline OK |
| * 415 MLContext lineage support (tracing and reuse correctness) OK |
| * 416 Lineage deduplication while, nested if, loop sequences OK |
| * 417 New rewrite for partial reuse in StepLM OK |
| * 418 Performance lineage tracing and reuse probing small data OK |
| * 419 Performance and robustness partial rewrites |
| |
| SYSTEMDS-420 Compiler Improvements |
| * 421 Fix invalid IPA scalar propagation into functions OK |
| |
| SYSTEMDS-500 Documentation Webpage Reintroduction |
| * 501 Make Documentation webpage framework OK |
| |
| Others: |
| * Break append instruction to cbind and rbind |
| |
| SYSTEMDS-510 IO formats |
| * 511 Add protobuf support to write and read FrameBlocks to HDFS OK |
| |
| SYSTEMDS-520 Lineage Tracing, Reuse and Integration III |
| * 521 New lineage option exposing cache policies OK |
| |
| SYSTEMDS-610 Cleaning Pipelines |
| * 611 Initial Brute force execution OK |
| |
| |
