Scripts and demo notebooks for deploying and running SystemDS on a Databricks cluster. Tested against DBR 16.4 LTS (Spark 3.5.2 / Scala 2.12), where the SystemDS jar runs unchanged.
| File | Purpose |
|---|---|
deploy.sh | Create a UC volume, upload SystemDS.jar, create a single-user cluster, install the Delta Kernel libraries, and import the demo notebooks. |
SystemDS_MLContext_Demo.scala | Notebook: Unity Catalog round-trip using the SystemDS MLContext (Scala) API. Reads a table, runs a configurable DML script, writes the result back. |
SystemDS_Delta_E2E.scala | Notebook: end-to-end Delta → linear regression on one Delta table. SystemDS reads it natively as a frame (read(format="delta")) → transformencode → lm; Spark ML reads the same table → OneHotEncoder → LinearRegression. Times read + encode + train for both. |
demo.dml | Standalone DML smoke test: reads a matrix from storage, computes column sums and a Gram-matrix trace. |
.env.example | Template for your local configuration. |
The Databricks CLI installed and authenticated once interactively:
databricks auth login --profile <your-profile>
A built SystemDS jar at <repo-root>/target/SystemDS.jar (mvn -q -DskipTests package), or set JAR_LOCAL to point at one.
python3 on your PATH (used to parse CLI JSON output).
All settings are read from environment variables. The easiest way is a .env file:
cp scripts/databricks/.env.example scripts/databricks/.env # then edit scripts/databricks/.env
deploy.sh looks for a .env file by:
ENV_FILE=/abs/path/to/.env if you set it explicitly, otherwise.env it finds.Anything already exported in your shell overrides values from .env.
| Variable | Default | Description |
|---|---|---|
PROFILE | DEFAULT | Databricks CLI profile. |
CATALOG / SCHEMA / VOLUME | main / default / systemds | UC location for the jar volume (and notebook defaults). |
POLICY_ID | (empty) | Compute policy id; leave empty for none. |
SPARK_VERSION | 16.4.x-scala2.12 | DBR runtime. |
NODE_TYPE | i3.xlarge | Node type. |
NUM_WORKERS | 0 | Worker count (0 = single node). |
AUTOTERMINATION_MINUTES | 30 | Auto-terminate idle minutes. |
CLUSTER_NAME | systemds | Cluster name. |
JAR_LOCAL | <repo-root>/target/SystemDS.jar | Jar to upload. |
NB_DIR | /Users/<you> | Workspace folder to import notebooks into. |
NB_FILES | (the 2 demo notebooks) | Space-separated notebooks to import; language detected from extension. |
DELTA_KERNEL_VERSION | 3.3.2 | Delta Kernel Maven library version installed by deploy.sh libs (>= 3.3.2; must match pom.xml). |
USER_NAME | (auto-detected) | Databricks user. |
NODE_TYPE is a cloud instance type. The default i3.xlarge is a small, storage-optimized AWS node (4 vCPU / 30.5 GiB RAM / 950 GB local NVMe SSD); the fast local disk is handy because the notebook spills SystemDS scratch to /local_disk0. With NUM_WORKERS=0 this single node is both driver and executor, so it bounds the total memory available to SystemDS.
Some common AWS options (pick more cores/RAM for larger workloads):
| Node type | vCPU | RAM | Local SSD | Notes |
|---|---|---|---|---|
i3.xlarge | 4 | 30.5 GiB | 950 GB | Default; storage-optimized. |
i3.2xlarge | 8 | 61 GiB | 1900 GB | Same family, 2× bigger. |
i4i.xlarge | 4 | 32 GiB | 937 GB | Newer gen, faster storage. |
m5d.xlarge | 4 | 16 GiB | 150 GB | General purpose w/ local SSD. |
r5d.2xlarge | 8 | 64 GiB | 300 GB | Memory-optimized w/ local SSD. |
The exact set depends on your cloud (AWS / Azure / GCP) and workspace. List what your workspace actually offers with:
databricks clusters list-node-types -p "$PROFILE" -o json
References: AWS EC2 instance types, Azure VM sizes, GCP machine families.
cd scripts/databricks ./deploy.sh upload # create UC volume + copy SystemDS.jar into it ./deploy.sh cluster # create the single-user cluster + install the jar ./deploy.sh libs # install the Delta Kernel Maven libraries on the cluster ./deploy.sh import # import the demo notebooks ./deploy.sh all # all of the above
The created cluster id is written to scripts/databricks/.cluster_id.
The Delta notebook (SystemDS_Delta_E2E) reads Delta tables natively through the Spark-free Delta Kernel, which is not on the DBR classpath. ./deploy.sh libs installs io.delta:delta-kernel-defaults (version DELTA_KERNEL_VERSION, default 3.3.2) as a cluster Maven library. Use >= 3.3.2: earlier releases trip a classloader conflict with DBR's bundled parquet. The version must match the delta-kernel.version property in the SystemDS pom.xml.
SystemDS_MLContext_Demo)The Scala notebook is driven by widgets, so nothing is hardcoded — set them in the notebook UI or pass them as job parameters:
| Widget | Default | Description |
|---|---|---|
catalog / schema | main / default | Where the input/output tables live. |
input_table / output_table | systemds_input / systemds_output | Table names. |
dml_path | (blank) | DML script to run. Blank uses the built-in z-score demo; otherwise a path readable from the driver (UC volume, /Workspace, or dbfs:). |
exec_type | DEFAULT | SystemDS execution mode. DEFAULT lets SystemDS choose the plan; DRIVER, SPARK, or DRIVER_AND_SPARK force a mode. |
Custom DML contract: the script receives the input matrix as X and must produce a matrix Y and a scalar checksum.
SystemDS_Delta_E2E)| Widget | Default | Description |
|---|---|---|
catalog / schema / volume | main / default / systemds | UC location; the Delta table is written under the volume. |
rows | 1000000 | Rows in the generated Delta table. |
num_numeric / num_categorical / cardinality | 100 / 20 / 30 | Feature shape. Defaults are deliberately encode-heavy (700 features) so the SystemDS-vs-Spark difference is visible. |
reg | 1e-3 | L2 regularization for lm. |
recreate | true | Rewrite the Delta table before running. |
statistics | true | Print the SystemDS per-instruction breakdown. |
The encode complexity (categoricals × cardinality), not the row count, drives the gap: more categoricals blow up Spark's StringIndexer + OneHotEncoder (each a shuffle stage), while SystemDS dummycodes in-memory. On a single node, raw rows instead favor Spark, and very large tables can exhaust driver memory.
Indicative single-node (i3.xlarge, 1M rows) numbers — single cold run, no warmup:
| workload | Spark ML | SystemDS | speedup |
|---|---|---|---|
SystemDS_Delta_E2E, 700 features (read + encode + train) | 116.6 s | 55.4 s | ~2.1× |
The Spark side is the same speed on Spark 3.5.2 (DBR 16.4) and Spark 4.0.0 (DBR 17.3 LTS), so the comparison is not an artifact of an old runtime.
/Workspace).--add-opens set at JVM launch (configured via spark.{driver,executor}.extraJavaOptions), and an absolute scratch dir (the notebook pins sysds.scratch)..env and .cluster_id are git-ignored — they hold personal config and local state and should never be committed.