| --- |
| layout: page |
| title: Roadmap |
| description: Apache SystemDS Roadmap |
| group: nav-right |
| --- |
| <!-- |
| {% comment %} |
| Licensed to the Apache Software Foundation (ASF) under one or more |
| contributor license agreements. See the NOTICE file distributed with |
| this work for additional information regarding copyright ownership. |
| The ASF licenses this file to you under the Apache License, Version 2.0 |
| (the "License"); you may not use this file except in compliance with |
| the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| {% endcomment %} |
| --> |
| |
| <!-- Hero --> |
| <section class="full-stripe full-stripe--subpage-header clear-header"> |
| <div class="ml-container ml-container--horizontally-center"> |
| <div class="col col-12 content-group content-group--center-content content-group--center-align"> |
| <h1>{{ site.data.project.name }} Roadmap</h1> |
| </div> |
| </div> |
| </section> |
| |
| |
| <section class="full-stripe full-stripe--alternate"> |
| <div class="ml-container ml-container--narrow content-group"> |
| <div class="col col-12"> |
| <h2>Planned for Future Releases</h2> |
| <ul> |
| <li>Completion of Prior Experimental Features</li> |
| <li>Algorithms & builtin functions |
| <ul> |
| <li>NN layers-based factorization machines with regression & classification capabilities</li> |
| <li>NN optimization test suite with well known optimization test functions</li> |
| <li>Model selection & hyper parameter tuning</li> |
| <li>Additional distribution functions, e.g. weibull, gamma</li> |
| <li>Time Series Analysis and Anomaly Detection</li> |
| <li>Scalable Decomposition Algorithms</li> |
| </ul> |
| </li> |
| <li>Enhanced Deep Learning support |
| <ul> |
| <li>Coherent sparse operations on CPU/GPU</li> |
| <li>Coherent single-precision support on CPU/GPU</li> |
| <li>Distributed DL operations</li> |
| </ul> |
| </li> |
| <li>GPU Support |
| <ul> |
| <li>Full compiler integration (cost-based, automatic placement)</li> |
| <li>Multi GPUs</li> |
| <li>Distributed GPUs</li> |
| </ul> |
| </li> |
| <li>Code generation |
| <ul> |
| <li>Deep learning operations</li> |
| <li>Heterogeneous HW, incl GPUs</li> |
| </ul> |
| </li> |
| <li>Compressed Linear Algebra |
| <ul> |
| <li>Deep learning operations</li> |
| <li>Ultra-sparse datasets</li> |
| </ul> |
| </li> |
| <li>Model Serving</li> |
| <li>Misc Runtime |
| <ul> |
| <li>NUMA-awareness (thread pools, matrix partitioning)</li> |
| <li>Unified memory management (ops, bufferpool, RDDs/broadcasts)</li> |
| <li>Support additional external formats such as feather format for matrices and frames</li> |
| <li>Parfor support for broadcasts</li> |
| </ul> |
| </li> |
| <li>Misc Compiler |
| <ul> |
| <li>Holistic sum-product optimization and operator fusion</li> |
| <li>SPOOF compiler improvement</li> |
| <li>Tree-based Data Structures</li> |
| <li>Global Dataflow Optimizations</li> |
| </ul> |
| </li> |
| <li>Tooling</li> |
| <li>Common DSL Architecture</li> |
| </ul> |
| </div> |
| |
| <div class="col col-12"> |
| <h2>Current Release</h2> |
| <ul> |
| <li> |
| <strong>SystemML 1.2.0 (<a href="release-notes/systemml-release-notes-1.2.0.html">released</a> in Aug, 2018) |
| <a |
| href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342959&projectId=12319522">details</a></strong> |
| <ul> |
| <li>New data type list for lists and structs</li> |
| <li>Built in deep learning functions, e.g. convolution, maxpooling, bias</li> |
| <li>DML eval function</li> |
| <li>Multiple new builtin functions, e.g. exists, lower.tri, upper.tri</li> |
| <li>Code motion, global subexpression elimination, rewrites for codegen</li> |
| <li>Performance improvements for ultra-sparse data and instruction generation |
| </ul> |
| </li> |
| <li> |
| Experimental Features |
| <ul> |
| <li>Parameter server</li> |
| <li>Sparsity estimators</li> |
| </ul> |
| </li> |
| <li> |
| Removals |
| <ul> |
| <li>Opt level 4</li> |
| </ul> |
| </li> |
| </ul> |
| </div> |
| |
| <div class="col col-12"> |
| <h2>Prior Releases</h2> |
| <ul> |
| <li> |
| <strong>SystemML 1.1.0 (<a href="release-notes/systemml-release-notes-1.1.0.html">released</a> in March, 2018) |
| <a |
| href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319522&version=12342282">details</a></strong> |
| <ul> |
| <li>New Builtin Functions: `ifelse`, `assert`, `eval`, `avg_pool`, `avg_pool_backward`</li> |
| <li>Additional Layers in NN library: average pooling, upsampling, low-rank fully connected</li> |
| <li>New Capabilities/Features such as dense matrix blocks >16GB, additional ParFor result aggregation |
| operations, UDFs callable in expressions, zero rows/columns matrices, matrix-matrix multiplication over |
| compressed matrices</li> |
| <li>Extended Caffe2DML and Keras2DML APIs</li> |
| <li>Compiler & Runtime enhancements</> |
| <li>Performance improvements</li> |
| </ul> |
| </li> |
| <li> |
| <strong>SystemML 1.0.0 (<a href="release-notes/systemml-release-notes-1.0.0.html">released</a> in December, |
| 2017) |
| <a |
| href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319522&version=12338328">details</a></strong> |
| <ul> |
| <li>Enhanced Deep Learning support with enhanced NN layers and functions, Caffe2DML, and operator |
| implementation</li> |
| <li>Native BLAS support</li> |
| <li>Additional algorithms: autoencoder, enhanced PCA</li> |
| <li>Enhanced rewrites, IPA, vectorization, and instruction generation</li> |
| <li>Enhanced JMLC API, e.g. prepared scripts with thread affinity for outputs and configs, script cloning, |
| configuration management</li> |
| <li>SystemML Lite artifact</li> |
| <li>Compression on by default</li> |
| </ul> |
| </li> |
| <li> |
| Experimental Features |
| <ul> |
| <li>Keras2DML.</li> |
| <li>Enhanced code generation, code gen optimizer, and multi-threaded codegen operators/li> |
| <li>Enhanced GPU support</li> |
| </ul> |
| </li> |
| <li> |
| Removals |
| <ul> |
| <li>Dropped JDK 7 support</li> |
| </ul> |
| </li> |
| <li> |
| <strong>SystemML 0.15.0 (<a href="release-notes/systemml-release-notes-0.15.0.html">released</a> in September, |
| 2017) |
| <a |
| href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319522&version=12341587">details</a></strong> |
| <ul> |
| <li>Added several new 2D convolution layers</li> |
| <li>Graduated `nn` library from staging to `scripts/nn`</li> |
| </ul> |
| </li> |
| <li> |
| Experimental Features |
| <ul> |
| <li>Expanded Code Generation for broader performance improvements</li> |
| <li>Enhanced GPU support and scalability</li> |
| </ul> |
| </li> |
| <li> |
| Removals |
| <ul> |
| <li>Removed file-based transform</li> |
| <li>Removed original MLContext API</li> |
| </ul> |
| </li> |
| </ul> |
| |
| <ul> |
| <li> |
| <strong>SystemML 0.14.0-incubating (<a |
| href="release-notes/systemml-release-notes-0.14.0-incubating.html">released</a> in May, 2017) |
| <a |
| href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319522&version=12340322">details</a></strong> |
| <ul> |
| <li>Runtime feature extensions (new libsvm-binary data converters, parfor spark buffer pool handling, parfor |
| block partitioning of fixed size batches of rows or columns, native dataset support in parfor spark |
| datapartition-execute)</li> |
| <li>Compiler feature extensions (improved parfor execution type selection, improved literal replacement for |
| nrow/ncol, simplified instruction generation across back-ends, consolidated static/dynamic rewrite |
| utilities)</li> |
| </ul> |
| </li> |
| <li> |
| Experimental Features |
| <ul> |
| <li>New Code Generation capabilities for automatic operator fusion (basic code generator, compiler |
| integration, runtime integration, in-memory source code compilation, extended explain tool, support for |
| right indexing and replace in cellwise and |
| row aggregate templates, support for row, column, or no aggregation in rowwise template). Note code |
| generation provides significant performance gains with fewer read/write intermediates, reduced scans of |
| inputs and intermediates, and enhanced |
| sparsity exploitation. To enable this feature, set codegen.enabled property to true in SystemML-config.xml |
| file.</li> |
| <li>New instructions and operators for GPU support (relu_maxpooling, conv2d_bias_add, bias_multiply)</li> |
| </ul> |
| </li> |
| <li> |
| Removals |
| <ul> |
| <li>Removed support for Java 6 and Java 7</li> |
| <li>Removed parfor perftesttool and cost estimator</li> |
| </ul> |
| </li> |
| </ul> |
| |
| <ul> |
| <li> |
| <strong>SystemML 0.13.0-incubating (<a |
| href="release-notes/systemml-release-notes-0.13.0-incubating.html">released</a> in March, 2017) |
| <a |
| href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319522&version=12339247">details</a></strong> |
| <ul> |
| <li>Updated build for Spark 2.1.0</li> |
| <li>New simplification rewrites for stratstats</li> |
| <li>New fused operator tack+* in CP and Spark</li> |
| <li>New dmlFromResource capability in Python (equivalent to Scala)</li> |
| <li>Add input float support to MLContext</li> |
| </ul> |
| </li> |
| <li> |
| Documentation Enhancements |
| <ul> |
| <li>Deploy versioned documentation to main project website</li> |
| <li>Add python mlcontext example to engine dev guide</li> |
| <li>Add MLContext info functionality to docs</li> |
| <li>Update DML Language Reference for write description parameter</li> |
| </ul> |
| </li> |
| <li> |
| Deprecations, Removals |
| <ul> |
| <li>Deprecate old MLContext API</li> |
| <li>Deprecate parfor perftesttool</li> |
| <li>Deprecate SQLContext methods</li> |
| <li>Replace deprecated Accumulator with AccumulatorV2</li> |
| <li>Replace append with cbind for matrices</li> |
| <li>Migrate Vector and LabeledPoint classes from mllib to ml</li> |
| </ul> |
| </li> |
| <li> |
| Experimental Features / Algorithms |
| <ul> |
| <li>Compressed Linear Algebra v2 (new DDC encoding format, hardened sample-based estimators, debugging |
| tools, new column grouping algorithm, additional operations)</li> |
| </ul> |
| </li> |
| </ul> |
| |
| <ul> |
| <li> |
| <strong>SystemML 0.12.0-incubating (<a |
| href="release-notes/systemml-release-notes-0.12.0-incubating.html">released</a> in February, 2017) |
| <a |
| href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319522&version=12339246">details</a></strong> |
| <ul> |
| <li>Support pip install of new python package</li> |
| <li>Allow NumPy arrays, Pandas DataFrame and SciPy matrices as input to MLContext</li> |
| <li>Improve SystemML Python DSL for NumPy</li> |
| <li>Updated build for Spark 1.6.0</li> |
| <li>DML utility script to shuffle input dataset</li> |
| </ul> |
| </li> |
| <li> |
| Experimental Features / Algorithms |
| <ul> |
| <li>GPU Enhancements</li> |
| </ul> |
| </li> |
| </ul> |
| |
| <ul> |
| <li> |
| <strong>SystemML 0.11.0-incubating (<a |
| href="release-notes/systemml-release-notes-0.11.0-incubating.html">released</a> in November, 2016) |
| <a |
| href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319522&version=12335820">details</a></strong> |
| <ul> |
| <li>SystemML frames</li> |
| <li>New MLContext API</li> |
| <li>Transform functions based on SystemML frames</li> |
| </ul> |
| </li> |
| <li> |
| Experimental Features / Algorithms |
| <ul> |
| <li>New built-in functions for deep learning (convolution and pooling)</li> |
| <li>Deep learning library (DML bodied functions)</li> |
| <li>Python DSL Integration</li> |
| <li>GPU Support</li> |
| <li>Compressed Linear Algebra</li> |
| </ul> |
| </li> |
| <li> |
| New Algorithms |
| <ul> |
| <li>Lasso</li> |
| <li>kNN</li> |
| <li>Lanczos</li> |
| <li>PPCA</li> |
| </ul> |
| </li> |
| <li> |
| Deep Learning Algorithms |
| <ul> |
| <li>CNN (Lenet)</li> |
| <li>RBM</li> |
| </ul> |
| </li> |
| </ul> |
| |
| <ul> |
| <li> |
| <strong>SystemML 0.10.0-incubating (<a |
| href="release-notes/systemml-release-notes-0.10.0-incubating.html">released</a> in June, 2016) |
| <a |
| href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319522&version=12334651">details</a></strong> |
| <ul> |
| <li>Different types of Spark Matrix Blocks: MCSR, CSR, COO</li> |
| <li>SystemML Frame support in JMLC/CP</li> |
| <li>Initial Deep Learning support</li> |
| <li>API/Scripts: parser error handling, SystemML configuration handling,</li> |
| </ul> |
| </li> |
| <li> |
| Include Algorithms in SystemML jar, print matrix |
| <ul> |
| <li>New fused operator: wdivmm with variations</li> |
| <li>Performance Features: cache-conscious operations, more multithreaded</li> |
| </ul> |
| </li> |
| <li> |
| Operations, New Simplification Rewrites |
| <ul> |
| <li>New Algorithms: kNN</li> |
| <li>Documentation: javadocs, Jupyter/Zeppelin notebook examples</li> |
| </ul> |
| </li> |
| </ul> |
| |
| <ul> |
| <li> |
| <strong>SystemML 0.9.0-incubating (<a |
| href="release-notes/systemml-release-notes-0.9.0-incubating.html">released</a> in January, 2016) |
| <a |
| href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319522&version=12334650">details</a></strong> |
| <ul> |
| <li>Improvements to MLContext and MLPipeline wrappers</li> |
| <li>New converter utilities for RDDs and DataFrames</li> |
| <li>New Optimizations for Spark Backend, e.g. eager RDD caching and</li> |
| </ul> |
| </li> |
| <li> |
| Repartitioning, RDD Checkpointing, On-Demand Creation of SparkContext |
| <ul> |
| <li>New Runtime Operators for mmult, multihreaded readers and operators.</li> |
| <li>New Algorithms: ALS, Cubic Splines</li> |
| <li>Online Documentation</li> |
| </ul> |
| </li> |
| </ul> |
| </div> |
| </div> |
| </section> |