docs/engine-dev-guide.md

layout: global displayTitle: SystemML Engine Developer Guide title: SystemML Engine Developer Guide description: SystemML Engine Developer Guide

This will become a table of contents (this text will be scraped). {:toc}

Building SystemML

SystemML is built using Apache Maven. SystemML will build on Linux, MacOS, or Windows, and requires Maven 3 and Java 7 (or higher). To build SystemML, run:

mvn clean package

To build the SystemML distributions, run:

mvn clean package -P distribution

Testing SystemML

SystemML features a comprehensive set of integration tests. To perform these tests, run:

mvn verify

Note: these tests require R to be installed and available as part of the PATH variable on the machine on which you are running these tests.

If required, please install the following packages in R:

install.packages(c("batch", "bitops", "boot", "caTools", "data.table", "doMC", "doSNOW", "ggplot2", "glmnet", "lda", "Matrix", "matrixStats", "moments", "plotrix", "psych", "reshape", "topicmodels", "wordcloud"), dependencies=TRUE)

Development Environment

SystemML itself is written in Java and is managed using Maven. As a result, SystemML can readily be imported into a standard development environment such as Eclipse and IntelliJ IDEA. The DMLScript class serves as the main entrypoint to SystemML. Executing DMLScript with no arguments displays usage information. A script file can be specified using the -f argument.

In Eclipse, a Debug Configuration can be created with DMLScript as the Main class and any arguments specified as Program arguments. A PyDML script requires the addition of a -python switch.

Suppose that we have a hello.dml script containing the following:

print('hello ' + $1)

This SystemML script can be debugged in Eclipse using a Debug Configuration such as the following:

Python MLContext API

When working with the Python MLContext API (see src/main/python/systemml/mlcontext.py) during development, it can be useful to install the Python MLContext API in editable mode (-e). This allows Python updates to take effect without requiring the SystemML python artifact to be built and installed.

{% highlight bash %} mvn clean pip3 install -e src/main/python mvn clean package PYSPARK_PYTHON=python3 pyspark --driver-class-path target/SystemML.jar {% endhighlight %}

Using Python version 3.5.2 (default, Jul 28 2016 21:28:07) SparkSession available as ‘spark’.

from systemml import MLContext, dml ml = MLContext(sc)

Welcome to Apache SystemML!

script = dml(“print(‘hello world’)”) ml.execute(script) hello world MLResults {% endhighlight %}

Matrix Multiplication Operators

In the following, we give an overview of backend-specific physical matrix multiplication operators in SystemML as well as their internally used matrix multiplication block operations.

Basic Matrix Multiplication Operators

An AggBinaryOp hop can be compiled into the following physical operators.

1. Physical Operators in CP (single node, control program)

Name	Description	Operation
`MM`	basic matrix multiplication	`mm`
`MMChain`	matrix multiplication chain	`mmchain`
`TSMM`	transpose-self matrix multiplication	`tsmm`
`PMM`	permutation matrix multiplication	`pmm`

2. Physical Operator in MR (distributed, mapreduce)

Name	Description	Operation
`MapMM`	map-side matrix multiplication, w/ or w/o agg	`mm`
`MapMMChain`	map-side matrix chain multiplication	`mmchain`
`TSMM`	map-side transpose-self matrix multiplication	`tsmm`
`PMM`	map-side permutation matrix multiplication	`pmm`
`CPMM`	cross-product matrix multiplication, 2 jobs	`mm`
`RMM`	replication-based matrix multiplication, 1 job	`mm`

3. Physical Operators in SPARK (distributed, spark)

Name	Description	Operation
`MapMM`	see MR, flatmap/mappartitions/maptopair + reduce/reducebykey/no_aggregation	`mm`
`MapMMChain`	see MR, mapvalues/maptopair + reduce	`mmchain`
`TSMM`	see MR, mapvalues + reduce	`tsmm`
`PMM`	see MR, flatmaptopair + reducebykey	`pmm`
`CPMM`	see MR, 2 x maptopair + join + maptopair + reduce/reducebykey	`mm`
`RMM`	see MR, 2 x flatmap + join + maptopair + reducebykey	`mm`
`ZIPMM`	partitioning-preserving 1-1 zipping mm, join + mapvalues + reduce	`mm`

Complex Matrix Multiplication Operators

A QuaternaryOp hop can be compiled into the following physical operators. Note that wsloss, wsigmoid, wdivmm have different semantics though. The main goal of these operators is to prevent the creation of dense “outer” products via selective computation over a sparse driver (sparse matrix and sparse-safe operation).