To use docker and systemDS there are two options. The first is to build your own docker image. The other option is to download the already build image from docker.
To Build the docker image simply run the build script.
./docker/build.sh
Afterwards you should have a local image with the id systemds/sysds:latest. To execute any given DML script follow the step Run.
Running SystemDS in a docker container is as simple as constructing any DML script. Then download the docker image systemds/sysds:latest or build your own.
docker pull systemds/sysds:latest
Verify that the docker image correctly works simply by running it.
docker run --rm systemds/sysds:latest
It should respond with something like:
Hello, World!
SystemDS Statistics:
Total execution time: 0.010 sec.
To run specific scripts mount a folder(s) containing the scripts and data, and execute the script inside the folder using the docker container.
You can mount any such folder and execute systemds by changing the first part of the -v argument of the following command:
docker run \ -v $(pwd)/docker/mountFolder:/input \ --rm systemds/sysds:latest
Default behavior is to run the script located at /input/main.dml. To run any other script use:
docker run \ -v $(pwd)/folder/to/share:/any/path/in/docker/instance \ --rm systemds/sysds:latest \ systemds /any/path/to/a/script.dml
To run a federated worker in a docker container simply use:
docker run -p 8000:8000 --rm systemds/sysds:latest systemds WORKER 8000
This port forwards the worker to port 8000 on the host and starts a worker in the instance on port 8000.
Note that the worker does not have any data, since no data is mounted in the worker image. To add a folder containing the data needed in the worker do the following:
docker run \ -p 8000:8000 \ -v $(pwd)/data/folder/path/locally:/data/folder/path/in/container \ --rm systemds/sysds:latest systemds WORKER 8000
To run a python script the pythonsysds image is used.
docker run --rm systemds/pythonsysds:latest
User provided scripts have to be mounted into the image.
docker run \ -v $(pwd)/data/folder/path/locally:/data/folder/path/in/container \ --rm systemds/pythonsystds:latest \ python3 path/to/script/to/execute.py
We also have a docker image for execution of tests. This enables faster test execution on the github actions. To build this image simply run the same command as above.
./docker/build.sh
Because the github action pulls the image from docker hub the image has to be pushed to docker hub to produce any change in the behavior of the testing.
docker push systemds/testingsysds:latest
For each of the tests that require R, this image is simply used, because it skips the installation of the R packages, since they are installed in this image.
Test your testing image locally by running the following command:
docker run \ -v $(pwd):/github/workspace \ systemds/testingsysds:latest \ org.apache.sysds.test.component.**