Google Git
Sign in
apache / systemds / refs/heads/branch-0.12 / . / docs / hadoop-batch-mode.md
blob: ddc1c1ff7f37cff21199c3fd87fecf09243f9245 [file] [log] [blame] [view]
---
layout: global
title: Invoking SystemML in Hadoop Batch Mode
description: Invoking SystemML in Hadoop Batch Mode
---
<!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to you under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
* This will become a table of contents (this text will be scraped).
{:toc}
<br/>
# Overview
Given that a primary purpose of SystemML is to perform machine learning on large distributed data sets,
two of the most important ways to invoke SystemML are Hadoop Batch and Spark Batch modes.
Here, we will look at SystemML's Hadoop Batch mode in more depth.
We will look at running SystemML with Standalone Hadoop, Pseudo-Distributed Hadoop, and Distributed Hadoop.
We will first run SystemML on a single machine with Hadoop running in Standalone mode. Next, we'll run SystemML on HDFS
in Hadoop's Pseudo-Distributed mode on a single machine, followed by Pseudo-Distributed mode with YARN.
After that, we'll set up a 4-node Hadoop cluster and run SystemML on Distributed Hadoop with YARN.
Note that this tutorial does not address security. For security considerations with regards to Hadoop, please
refer to the Hadoop documentation.
* * *
# Hadoop Batch Mode Invocation Syntax
SystemML can be invoked in Hadoop Batch mode using the following syntax:
hadoop jar SystemML.jar [-? | -help | -f <filename>] (-config=<config_filename>) ([-args | -nvargs] <args-list>)
The `SystemML.jar` file is specified to Hadoop using the `jar` option.
The DML script to invoke is specified after the `-f` argument. Configuration settings can be passed to SystemML
using the optional `-config=` argument. DML scripts can optionally take named arguments (`-nvargs`) or positional
arguments (`-args`). Named arguments are preferred over positional arguments. Positional arguments are considered
to be deprecated. All the primary algorithm scripts included with SystemML use named arguments.
**Example #1: DML Invocation with Named Arguments**
hadoop jar systemml/SystemML.jar -f systemml/algorithms/Kmeans.dml -nvargs X=X.mtx k=5
**Example #2: DML Invocation with Positional Arguments**
hadoop jar systemml/SystemML.jar -f example/test/LinearRegression.dml -args "v" "y" 0.00000001 "w"
In a clustered environment, it is *highly* recommended that SystemML configuration settings are specified
in a `SystemML-config.xml` file. By default, SystemML will look for this file in the current working
directory (`./SystemML-config.xml`). This location can be overridden by the `-config=` argument.
**Example #3: DML Invocation with Configuration File Explicitly Specified and Named Arguments**
hadoop jar systemml/SystemML.jar -f systemml/algorithms/Kmeans.dml -config=/conf/SystemML-config.xml -nvargs X=X.mtx k=5
For recommended SystemML configuration settings in a clustered environment, please see
[Recommended Hadoop Cluster Configuration Settings](hadoop-batch-mode.html#recommended-hadoop-cluster-configuration-settings).
* * *
# SystemML with Standalone Hadoop
In Standalone mode, Hadoop runs on a single machine as a single Java process.
To begin, I connected to my Linux server as root and created a hadoop user.
$ ssh root@host1.example.com
[root@host1 ~]# useradd hadoop
[root@host1 ~]# passwd hadoop
Next, I logged on as the hadoop user. I downloaded the version of Hadoop that I wanted to use from an Apache mirror.
A list of Hadoop releases can be found at the [Apache Hadoop Releases](http://hadoop.apache.org/releases.html) website.
After downloading the Hadoop binary release, I unpacked it.
$ ssh hadoop@host1.example.com
[hadoop@host1 ~]$ wget http://mirror.sdunix.com/apache/hadoop/common/hadoop-2.6.2/hadoop-2.6.2.tar.gz
[hadoop@host1 ~]$ tar -xvzf hadoop-2.6.2.tar.gz
My Linux server already had a JDK (Java Development Kit) installed. If you haven't done so already, you will need Java
installed in order to use Hadoop.
I updated my `.bash_profile` file to export a `JAVA_HOME` environment variable, which I pointed to my JDK installation
directory. I also exported a `HADOOP_HOME` environment variable, which points to the root directory of the Hadoop release
that I unpacked. I updated the `PATH` variable to include the `JAVA_HOME` `bin` directory, the `HADOOP_HOME` `bin` directory,
and the `HADOOP_HOME` `sbin` directory.
[hadoop@host1 ~]# vi .bash_profile
...
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64
export HADOOP_HOME=/home/hadoop/hadoop-2.6.2
PATH=$JAVA_HOME/bin:$PATH:$HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export PATH
...
[hadoop@host1 ~]$ source ~/.bash_profile
To verify that Java and Hadoop were on the path, I used the `java -version` and `hadoop version` commands.
[hadoop@host1 ~]$ java -version
java version "1.7.0_79"
OpenJDK Runtime Environment (rhel-2.5.5.1.el6_6-x86_64 u79-b14)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
[hadoop@host1 ~]$ hadoop version
Hadoop 2.6.2
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 0cfd050febe4a30b1ee1551dcc527589509fb681
Compiled by jenkins on 2015-10-22T00:42Z
Compiled with protoc 2.5.0
From source with checksum f9ebb94bf5bf9bec892825ede28baca
This command was run using /home/hadoop/hadoop-2.6.2/share/hadoop/common/hadoop-common-2.6.2.jar
Next, I downloaded a SystemML release from the [downloads](http://systemml.apache.org/download.html) page.
Following this, I unpacked it.
[hadoop@host1 ~]$ tar -xvzf systemml-{{site.SYSTEMML_VERSION}}.tar.gz
**Alternatively**, we could have built the SystemML distributed release using [Apache Maven](http://maven.apache.org) and unpacked it.
[hadoop@host1 ~]$ git clone https://github.com/apache/incubator-systemml.git
[hadoop@host1 ~]$ cd incubator-systemml
[hadoop@host1 incubator-systemml]$ mvn clean package -P distribution
[hadoop@host1 incubator-systemml]$ tar -xvzf target/systemml-{{site.SYSTEMML_VERSION}}.tar.gz -C ..
[hadoop@host1 ~]$ cd ..
I downloaded the `genLinearRegressionData.dml` script that is used in the SystemML README example.
[hadoop@host1 ~]$ wget https://raw.githubusercontent.com/apache/incubator-systemml/master/scripts/datagen/genLinearRegressionData.dml
Next, I invoked the `genLinearRegressionData.dml` DML script in Hadoop Batch mode.
Hadoop was executed with the `SystemML.jar` file specified by the hadoop `jar` option.
The `genLinearRegressionData.dml` was specified using the `-f` option. Named input
arguments to the DML script were specified following the `-nvargs` option.
[hadoop@host1 ~]$ hadoop jar systemml-{{site.SYSTEMML_VERSION}}/SystemML.jar -f genLinearRegressionData.dml -nvargs numSamples=1000 numFeatures=50 maxFeatureValue=5 maxWeight=5 addNoise=FALSE b=0 sparsity=0.7 output=linRegData.csv format=csv perc=0.5
15/11/11 15:56:21 INFO api.DMLScript: BEGIN DML run 11/11/2015 15:56:21
15/11/11 15:56:21 INFO api.DMLScript: HADOOP_HOME: /home/hadoop/hadoop-2.6.2
15/11/11 15:56:21 WARN conf.DMLConfig: No default SystemML config file (./SystemML-config.xml) found
15/11/11 15:56:21 WARN conf.DMLConfig: Using default settings in DMLConfig
15/11/11 15:56:22 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
15/11/11 15:56:22 WARN hops.OptimizerUtils: Auto-disable multi-threaded text read for 'text' and 'csv' due to thread contention on JRE < 1.8 (java.version=1.7.0_79).
15/11/11 15:56:22 INFO api.DMLScript: SystemML Statistics:
Total execution time: 0.288 sec.
Number of executed MR Jobs: 0.
15/11/11 15:56:22 INFO api.DMLScript: END DML run 11/11/2015 15:56:22
In the console output, we see a warning that no default SystemML config file was found in the current working directory.
In a distributed environment on a large data set, it is highly advisable to specify configuration settings in a SystemML config file for
optimal performance. The location of the SystemML config file can be explicitly specified using the `-config=` argument.
The OptimizerUtils warning occurs because parallel multi-threaded text reads in Java versions less than 1.8 result
in thread contention issues, so only a single thread reads matrix data in text formats.
If we examine the contents of the directory, we see that `linRegData.csv` and `perc.csv` were written to the file system,
along with their corresponding metadata files. The `scratch_space` directory is used to write temporary matrix files.
[hadoop@host1 ~]$ ls -l
total 197500
-rw-rw-r-- 1 hadoop hadoop 2208 Nov 11 15:45 genLinearRegressionData.dml
drwxr-xr-x 9 hadoop hadoop 4096 Oct 21 17:53 hadoop-2.6.2
-rw-rw-r-- 1 hadoop hadoop 195515434 Oct 30 14:04 hadoop-2.6.2.tar.gz
drwxrwxrwx 2 hadoop hadoop 4096 Nov 11 15:56 linRegData.csv
-rw-r--r-- 1 hadoop hadoop 214 Nov 11 15:56 linRegData.csv.mtd
drwxrwxrwx 2 hadoop hadoop 4096 Nov 11 15:56 perc.csv
-rw-r--r-- 1 hadoop hadoop 206 Nov 11 15:56 perc.csv.mtd
drwxrwxrwx 2 hadoop hadoop 4096 Nov 11 15:56 scratch_space
drwxrwxr-x 4 hadoop hadoop 4096 Nov 11 15:42 systemml-{{site.SYSTEMML_VERSION}}
-rw-rw-r-- 1 hadoop hadoop 6683281 Oct 27 21:13 systemml-{{site.SYSTEMML_VERSION}}.tar.gz
To clean things up, I'll delete the files that were generated.
[hadoop@host1 ~]$ rm -r *.csv
[hadoop@host1 ~]$ rm *.csv.mtd
[hadoop@host1 ~]$ rmdir scratch_space/
* * *
# SystemML with Pseudo-Distributed Hadoop
Next, we'll look at running SystemML with Hadoop in Pseudo-Distributed mode. In Pseudo-Distributed mode, each Hadoop daemon
(such as NameNode and DataNode) runs in a separate Java process on a single machine.
In the previous section about Hadoop Standalone mode, we set up the `JAVA_HOME` and `HADOOP_HOME` environment variables
and added `JAVA_HOME/bin`, `HADOOP_HOME/bin`, and `HADOOP_HOME/sbin` to the `PATH` in `.bash_profile`.
We also need to set the `JAVA_HOME` value in the `hadoop-env.sh` file in the Hadoop configuration directory (`etc/hadoop`).
[hadoop@host1 hadoop]$ pwd
/home/hadoop/hadoop-2.6.2/etc/hadoop
[hadoop@host1 hadoop]$ vi hadoop-env.sh
...
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64
...
We need to be able to passwordlessly `ssh` to localhost. To do so, I'll generate a public key/private key pair and add
the public key to the hadoop user's `authorized_keys`. We can `ssh` to localhost to verify that we can connect without
a password.
[hadoop@host1 ~]$ ssh-keygen -t rsa -b 4096 -C "hadoop example"
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
[hadoop@host1 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@host1 ~]$ chmod 600 ~/.ssh/authorized_keys
[hadoop@host1 ~]$ ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 6b:86:78:86:13:0a:49:d4:c7:a7:15:10:d1:27:88:9e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
[hadoop@host1 ~]$ exit
logout
Connection to localhost closed.
[hadoop@host1 ~]$ ls -l .ssh
total 16
-rw------- 1 hadoop hadoop 736 Nov 11 16:44 authorized_keys
-rw------- 1 hadoop hadoop 3243 Nov 11 16:41 id_rsa
-rw-r--r-- 1 hadoop hadoop 736 Nov 11 16:41 id_rsa.pub
-rw-r--r-- 1 hadoop hadoop 391 Nov 11 16:46 known_hosts
In the Hadoop configuration directory (`etc/hadoop`), in the `core-site.xml` file, we specify the `fs.defaultFS`
property to be `localhost` with port `9000`.
[hadoop@host1 hadoop]$ vi core-site.xml
...
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
...
By default, HDFS replicates data on three nodes. Since we're running on a single machine, we'll change this to one.
We'll add a `dfs.replication` property to `hdfs-site.xml` and set its value to `1`.
[hadoop@host1 hadoop]$ vi hdfs-site.xml
...
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
...
Next, we'll format HDFS.
[hadoop@host1 ~]$ hdfs namenode -format
15/11/11 17:23:33 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = host1.example.com/9.30.252.15
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.6.2
...
STARTUP_MSG: java = 1.7.0_79
************************************************************/
...
15/11/11 17:23:34 INFO common.Storage: Storage directory /tmp/hadoop-hadoop/dfs/name has been successfully formatted.
...
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at host1.example.com/9.30.252.15
************************************************************/
We'll start up HDFS using the `start-dfs.sh` script. This starts the NameNode, DataNode, and SecondaryNameNode daemons
on the single machine.
[hadoop@host1 ~]$ start-dfs.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/hadoop/hadoop-2.6.2/logs/hadoop-hadoop-namenode-host1.out
localhost: starting datanode, logging to /home/hadoop/hadoop-2.6.2/logs/hadoop-hadoop-datanode-host1.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
RSA key fingerprint is 6b:86:78:86:13:0a:49:d4:c7:a7:15:10:d1:27:88:9e.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop-2.6.2/logs/hadoop-hadoop-secondarynamenode-host1.out
We can see the running Java processes using the `jps` command.
[hadoop@host1 ~]$ jps
36128 Jps
35844 DataNode
36007 SecondaryNameNode
35722 NameNode
Here, we can see detailed information about the Java processes that were started.
[hadoop@host1 ~]$ ps -C java -f -ww
UID PID PPID C STIME TTY TIME CMD
hadoop 35722 1 5 17:38 ? 00:00:05 /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/java -Dproc_namenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/hadoop/hadoop-2.6.2/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/hadoop/hadoop-2.6.2 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/hadoop/hadoop-2.6.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/hadoop/hadoop-2.6.2/logs -Dhadoop.log.file=hadoop-hadoop-namenode-host1.log -Dhadoop.home.dir=/home/hadoop/hadoop-2.6.2 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/home/hadoop/hadoop-2.6.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode
hadoop 35844 1 4 17:38 ? 00:00:04 /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/java -Dproc_datanode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/hadoop/hadoop-2.6.2/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/hadoop/hadoop-2.6.2 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/hadoop/hadoop-2.6.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/hadoop/hadoop-2.6.2/logs -Dhadoop.log.file=hadoop-hadoop-datanode-host1.log -Dhadoop.home.dir=/home/hadoop/hadoop-2.6.2 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/home/hadoop/hadoop-2.6.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -server -Dhadoop.security.logger=ERROR,RFAS -Dhadoop.security.logger=ERROR,RFAS -Dhadoop.security.logger=ERROR,RFAS -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.datanode.DataNode
hadoop 36007 1 5 17:38 ? 00:00:04 /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/java -Dproc_secondarynamenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/hadoop/hadoop-2.6.2/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/hadoop/hadoop-2.6.2 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,console -Djava.library.path=/home/hadoop/hadoop-2.6.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/home/hadoop/hadoop-2.6.2/logs -Dhadoop.log.file=hadoop-hadoop-secondarynamenode-host1.log -Dhadoop.home.dir=/home/hadoop/hadoop-2.6.2 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/home/hadoop/hadoop-2.6.2/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
Useful log information is created by default in the hadoop `logs` directory.
If everything worked correctly, we can hit port 50070 in a browser (http://host1.example.com:50070) to see Hadoop information.
If we look at our HDFS file system, we see that it currently doesn't contain any files.
[hadoop@host1 ~]$ hdfs dfs -ls
ls: `.': No such file or directory
Let's go ahead and execute the `genLinearRegressionData.dml` script in Hadoop Pseudo-Distributed mode.
[hadoop@host1 ~]$ hadoop jar systemml-{{site.SYSTEMML_VERSION}}/SystemML.jar -f genLinearRegressionData.dml -nvargs numSamples=1000 numFeatures=50 maxFeatureValue=5 maxWeight=5 addNoise=FALSE b=0 sparsity=0.7 output=linRegData.csv format=csv perc=0.5
15/11/11 18:16:33 INFO api.DMLScript: BEGIN DML run 11/11/2015 18:16:33
15/11/11 18:16:33 INFO api.DMLScript: HADOOP_HOME: /home/hadoop/hadoop-2.6.2
15/11/11 18:16:33 WARN conf.DMLConfig: No default SystemML config file (./SystemML-config.xml) found
15/11/11 18:16:33 WARN conf.DMLConfig: Using default settings in DMLConfig
15/11/11 18:16:33 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
15/11/11 18:16:33 WARN hops.OptimizerUtils: Auto-disable multi-threaded text read for 'text' and 'csv' due to thread contention on JRE < 1.8 (java.version=1.7.0_79).
15/11/11 18:16:35 INFO api.DMLScript: SystemML Statistics:
Total execution time: 1.484 sec.
Number of executed MR Jobs: 0.
15/11/11 18:16:35 INFO api.DMLScript: END DML run 11/11/2015 18:16:35
If we list the contents of the current directory in our regular file system, we see that no files have been written
to the regular file system.
[hadoop@host1 ~]$ ls
genLinearRegressionData.dml hadoop-2.6.2 hadoop-2.6.2.tar.gz systemml-{{site.SYSTEMML_VERSION}} systemml-{{site.SYSTEMML_VERSION}}.tar.gz
If we list the contents of the HDFS file system, we see that HDFS contains our data files and the corresponding metadata files.
[hadoop@host1 ~]$ hdfs dfs -ls
Found 5 items
drwxr-xr-x - hadoop supergroup 0 2015-11-11 18:16 linRegData.csv
-rw-r--r-- 1 hadoop supergroup 214 2015-11-11 18:16 linRegData.csv.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-11 18:16 perc.csv
-rw-r--r-- 1 hadoop supergroup 206 2015-11-11 18:16 perc.csv.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-11 18:16 scratch_space
If we examine the Hadoop web interface mentioned previously, we see that the files, directories, and blocks in HDFS have
increased in number.
Now that we're done with this example, I'll clean things up and delete the generated files from HDFS.
[hadoop@host1 hadoop]$ hdfs dfs -rm -r *.csv
[hadoop@host1 hadoop]$ hdfs dfs -rm *.mtd
[hadoop@host1 hadoop]$ hdfs dfs -rmdir scratch_space
I'll stop HDFS using the `stop-dfs.sh` script and then verify that the Java processes have stopped.
[hadoop@host1 ~]$ stop-dfs.sh
Stopping namenodes on [localhost]
localhost: stopping namenode
localhost: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
[hadoop@host1 ~]$ jps
37337 Jps
* * *
# SystemML with Pseudo-Distributed Hadoop and YARN
To add YARN to Pseudo-Distributed Hadoop on the single machine, we need to take our setup from the
previous example and update two configuration
files and start the ResourceManager and NodeManager daemons.
In the `mapred-site.xml` configuration file, we specify the
`mapreduce.framework.name` property as `yarn`.
[hadoop@host1 hadoop]$ pwd
/home/hadoop/hadoop-2.6.2/etc/hadoop
[hadoop@host1 hadoop]$ cp mapred-site.xml.template mapred-site.xml
[hadoop@host1 hadoop]$ vi mapred-site.xml
...
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
...
In the `yarn-site.xml` configuration file, we specify the `yarn.nodemanager.aux-services` property
to be `mapreduce_shuffle`.
[hadoop@host1 hadoop]$ vi yarn-site.xml
...
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
...
Next, we'll start HDFS using the `start-dfs.sh` script.
[hadoop@host1 hadoop]$ start-dfs.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/hadoop/hadoop-2.6.2/logs/hadoop-hadoop-namenode-host1.out
localhost: starting datanode, logging to /home/hadoop/hadoop-2.6.2/logs/hadoop-hadoop-datanode-host1.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop-2.6.2/logs/hadoop-hadoop-secondarynamenode-host1.out
After that, we'll start YARN using the `start-yarn.sh` script.
[hadoop@host1 hadoop]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/hadoop-2.6.2/logs/yarn-hadoop-resourcemanager-host1.out
localhost: starting nodemanager, logging to /home/hadoop/hadoop-2.6.2/logs/yarn-hadoop-nodemanager-host1.out
We can use the `jps` command to verify that the HDFS daemons (NameNode, DataNode, and SecondaryNameNode) and YARN
daemons (ResourceManager and NodeManager) are running.
[hadoop@host1 hadoop]$ jps
52046 ResourceManager
52482 Jps
52149 NodeManager
51582 NameNode
51712 DataNode
51880 SecondaryNameNode
We can now view YARN information via the web interface on port 8088 (http://host1.example.com:8088).
I'll execute the `genLinearRegressionData.dml` example that we've previously considered.
[hadoop@host1 hadoop]$ cd ~
[hadoop@host1 ~]$ hadoop jar systemml-{{site.SYSTEMML_VERSION}}/SystemML.jar -f genLinearRegressionData.dml -nvargs numSamples=1000 numFeatures=50 maxFeatureValue=5 maxWeight=5 addNoise=FALSE b=0 sparsity=0.7 output=linRegData.csv format=csv perc=0.5
15/11/12 11:57:04 INFO api.DMLScript: BEGIN DML run 11/12/2015 11:57:04
15/11/12 11:57:04 INFO api.DMLScript: HADOOP_HOME: /home/hadoop/hadoop-2.6.2
15/11/12 11:57:04 WARN conf.DMLConfig: No default SystemML config file (./SystemML-config.xml) found
15/11/12 11:57:04 WARN conf.DMLConfig: Using default settings in DMLConfig
15/11/12 11:57:05 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
15/11/12 11:57:06 WARN hops.OptimizerUtils: Auto-disable multi-threaded text read for 'text' and 'csv' due to thread contention on JRE < 1.8 (java.version=1.7.0_79).
15/11/12 11:57:07 INFO api.DMLScript: SystemML Statistics:
Total execution time: 1.265 sec.
Number of executed MR Jobs: 0.
15/11/12 11:57:07 INFO api.DMLScript: END DML run 11/12/2015 11:57:07
If we examine the HDFS file system, we see the files generated by the execution of the DML script by SystemML on Hadoop.
[hadoop@host1 ~]$ hdfs dfs -ls
Found 5 items
drwxr-xr-x - hadoop supergroup 0 2015-11-12 11:57 linRegData.csv
-rw-r--r-- 1 hadoop supergroup 214 2015-11-12 11:57 linRegData.csv.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-12 11:57 perc.csv
-rw-r--r-- 1 hadoop supergroup 206 2015-11-12 11:57 perc.csv.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-12 11:57 scratch_space
I'll go ahead and delete the generated example files from HDFS.
[hadoop@host1 ~]$ hdfs dfs -rm -r *.csv
[hadoop@host1 ~]$ hdfs dfs -rm *.mtd
[hadoop@host1 ~]$ hdfs dfs -rmdir scratch_space
We'll stop the YARN daemons using the `stop-yarn.sh` script.
[hadoop@host1 ~]$ stop-yarn.sh
stopping yarn daemons
stopping resourcemanager
localhost: stopping nodemanager
no proxyserver to stop
We can stop HDFS with the `stop-dfs.sh` script.
[hadoop@host1 ~]$ stop-dfs.sh
Stopping namenodes on [localhost]
localhost: stopping namenode
localhost: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
If we list the running Java processes, we see all the YARN daemons and HDFS daemons have stopped.
[hadoop@host1 ~]$ jps
53459 Jps
For cleanliness, I'll also delete the `/tmp/hadoop-hadoop` files created by Hadoop before proceeding to
the next example.
* * *
# SystemML with Distributed Hadoop and YARN
In our previous example, we ran SystemML on Hadoop in Pseudo-Distributed mode with YARN on a single machine.
This example will look at Distributed Hadoop with YARN on a 4-node cluster. Each server is running
Red Hat Enterprise Linux Server, release 6.6.
I have 4 nodes: host1, host2, host3, and host4. The host1 node
that we previously set up will act as the master for both HDFS and YARN,
and host2, bd150, and host4 will be slaves. For more information regarding
network configurations, please see the Hadoop documentation.
First, I created a hadoop user on each slave node.
[root@host1 ~]$ ssh root@host2.example.com
[root@host2 ~]# useradd hadoop
[root@host2 ~]# passwd hadoop
[root@host2 ~]# exit
[root@host1 ~]$ ssh root@host3.example.com
[root@host2 ~]# useradd hadoop
[root@host2 ~]# passwd hadoop
[root@host2 ~]# exit
[root@host1 ~]$ ssh root@host4.example.com
[root@host2 ~]# useradd hadoop
[root@host2 ~]# passwd hadoop
[root@host2 ~]# exit
Next, I set up passwordless login from the hadoop user on the master node (host1)
to each of the slave nodes. The `ssh-copy-id` command copied the master node's hadoop user's
public key value to the ~/.ssh/authorized_keys file of each of the slave nodes. I
tested the passwordless login from the master node to each of the slave nodes for the hadoop
user.
$ ssh hadoop@host1.example.com
[hadoop@host1 ~]$ ssh-copy-id host2.example.com
[hadoop@host1 ~]$ ssh hadoop@host2.example.com
Last login: Thu Nov 12 14:16:21 2015
[hadoop@host2 ~]$ exit
[hadoop@host1 ~]$ ssh-copy-id host3.example.com
[hadoop@host1 ~]$ ssh hadoop@host3.example.com
Last login: Thu Nov 12 14:16:40 2015
[hadoop@host3 ~]$ exit
[hadoop@host1 ~]$ ssh-copy-id host4.example.com
[hadoop@host1 ~]$ ssh hadoop@host4.example.com
Last login: Thu Nov 12 14:17:10 2015
[hadoop@host4 ~]$ exit
On the master node, I specified the slave nodes in the Hadoop `slaves` configuration file.
[hadoop@host1 hadoop]$ pwd
/home/hadoop/hadoop-2.6.2/etc/hadoop
[hadoop@host1 hadoop]$ more slaves
host2.example.com
host3.example.com
host4.example.com
In the `core-site.xml` file, I specified the `fs.defaultFS` property to reference the master node.
[hadoop@host1 hadoop]$ more core-site.xml
...
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://host1.example.com:9000</value>
</property>
</configuration>
...
In the `hdfs-site.xml` configuration file, I removed the previous `dfs.replication` property, since we
will use the default replication value (of 3).
[hadoop@host1 hadoop]$ more hdfs-site.xml
...
<configuration>
</configuration>
...
We'll be using YARN, so our `mapred-site.xml` will have the `mapreduce.framework.name`
property set to `yarn`, as in the previous example. Additionally, we'll set the `mapreduce.map.java.opts` and
`mapreduce.reduce.java.opts` properties to `-Xmx2g -Xms2g -Xmn200m`. The `-Xmn` parameter fixes the
size of the young generation and typically is set to 10% of the maximum heap, which we have set to 2g.
Furthermore, we'll set `mapreduce.map.memory.mb` and `mapreduce.reduce.memory.mb` to `3072`. Typically these
values are set to at least 1.5 times the value of the maximum heap size.
[hadoop@host1 hadoop]$ more mapred-site.xml
...
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx2g -Xms2g -Xmn200m</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx2g -Xms2g -Xmn200m</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>3072</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>3072</value>
</property>
</configuration>
...
In the `yarn-site.xml` configuration file, I added a `yarn.resourcemanager.hostname` property and specified
the master node as the host.
[hadoop@host1 hadoop]$ more yarn-site.xml
...
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>host1.example.com</value>
</property>
</configuration>
...
In the previous example, we specified the `JAVA_HOME` in the `hadoop-env.sh` configuration script.
We will use that same value.
[hadoop@host1 hadoop]$ more hadoop-env.sh
...
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64
...
Next, I copied my hadoop installation (which includes all of the mentioned configuration settings)
to each slave node.
[hadoop@host1 ~]$ pwd
/home/hadoop
[hadoop@host1 ~]$ scp -r hadoop-2.6.2 hadoop@host2.example.com:~/
[hadoop@host1 ~]$ scp -r hadoop-2.6.2 hadoop@host3.example.com:~/
[hadoop@host1 ~]$ scp -r hadoop-2.6.2 hadoop@host4.example.com:~/
My master node `.bash_profile` contains `JAVA_HOME` and `HADOOP_HOME` environment variables
and adds `JAVA_HOME/bin`, `HADOOP_HOME/bin` and `HADOOP_HOME/sbin` to the `PATH`.
...
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64
export HADOOP_HOME=/home/hadoop/hadoop-2.6.2
PATH=$JAVA_HOME/bin:$PATH:$HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export PATH
...
I copied the `.bash_profile` file to the slave nodes.
[hadoop@host1 ~]$ pwd
/home/hadoop
[hadoop@host1 ~]$ scp .bash_profile hadoop@host2.example.com:~/.bash_profile
[hadoop@host1 ~]$ scp .bash_profile hadoop@host3.example.com:~/.bash_profile
[hadoop@host1 ~]$ scp .bash_profile hadoop@host4.example.com:~/.bash_profile
On the master, I formatted HDFS.
[hadoop@host1 ~]$ hdfs namenode -format
Next, on the master, I started HDFS using `start-dfs.sh`. We can see that the master NameNode
and the slave DataNodes started up.
[hadoop@host1 ~]$ start-dfs.sh
Starting namenodes on [host1.example.com]
host1.example.com: starting namenode, logging to /home/hadoop/hadoop-2.6.2/logs/hadoop-hadoop-namenode-host1.out
host4.example.com: starting datanode, logging to /home/hadoop/hadoop-2.6.2/logs/hadoop-hadoop-datanode-host4.out
host2.example.com: starting datanode, logging to /home/hadoop/hadoop-2.6.2/logs/hadoop-hadoop-datanode-host2.out
host3.example.com: starting datanode, logging to /home/hadoop/hadoop-2.6.2/logs/hadoop-hadoop-datanode-host3.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop-2.6.2/logs/hadoop-hadoop-secondarynamenode-host1.out
Next I started YARN using the `start-yarn.sh` script. We see the master ResourceManager and the
slave NodeManagers started up.
[hadoop@host1 ~]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/hadoop-2.6.2/logs/yarn-hadoop-resourcemanager-host1.out
host3.example.com: starting nodemanager, logging to /home/hadoop/hadoop-2.6.2/logs/yarn-hadoop-nodemanager-host3.out
host2.example.com: starting nodemanager, logging to /home/hadoop/hadoop-2.6.2/logs/yarn-hadoop-nodemanager-host2.out
host4.example.com: starting nodemanager, logging to /home/hadoop/hadoop-2.6.2/logs/yarn-hadoop-nodemanager-host4.out
On the master, we see that the NameNode, SecondaryNameNode, and ResourceManager daemons are running.
[hadoop@host1 ~]$ jps
1563 NameNode
1775 SecondaryNameNode
2240 Jps
1978 ResourceManager
On the slaves, we see that the DataNode and NodeManager daemons are running.
[hadoop@host2 ~]$ jps
29096 Jps
28974 NodeManager
28821 DataNode
[hadoop@host3 ~]$ jps
5950 Jps
5706 DataNode
5819 NodeManager
[hadoop@host4 ~]$ jps
16388 Jps
16153 DataNode
16266 NodeManager
If we look at the Hadoop (on port 50070) and YARN (on port 8088) web interfaces, we can see information about our running cluster.
* * *
## SystemML with Distributed Hadoop and YARN: Linear Regression Example
Let's go ahead and run the SystemML example from the GitHub README.
[hadoop@host1 ~]$ hadoop jar systemml-{{site.SYSTEMML_VERSION}}/SystemML.jar -f genLinearRegressionData.dml -nvargs numSamples=1000 numFeatures=50 maxFeatureValue=5 maxWeight=5 addNoise=FALSE b=0 sparsity=0.7 output=linRegData.csv format=csv perc=0.5
[hadoop@host1 ~]$ hadoop jar systemml-{{site.SYSTEMML_VERSION}}/SystemML.jar -f systemml-{{site.SYSTEMML_VERSION}}/algorithms/utils/sample.dml -nvargs X=linRegData.csv sv=perc.csv O=linRegDataParts ofmt=csv
[hadoop@host1 ~]$ hadoop jar systemml-{{site.SYSTEMML_VERSION}}/SystemML.jar -f systemml-{{site.SYSTEMML_VERSION}}/algorithms/utils/splitXY.dml -nvargs X=linRegDataParts/1 y=51 OX=linRegData.train.data.csv OY=linRegData.train.labels.csv ofmt=csv
[hadoop@host1 ~]$ hadoop jar systemml-{{site.SYSTEMML_VERSION}}/SystemML.jar -f systemml-{{site.SYSTEMML_VERSION}}/algorithms/utils/splitXY.dml -nvargs X=linRegDataParts/2 y=51 OX=linRegData.test.data.csv OY=linRegData.test.labels.csv ofmt=csv
[hadoop@host1 ~]$ hadoop jar systemml-{{site.SYSTEMML_VERSION}}/SystemML.jar -f systemml-{{site.SYSTEMML_VERSION}}/algorithms/LinearRegDS.dml -nvargs X=linRegData.train.data.csv Y=linRegData.train.labels.csv B=betas.csv fmt=csv
...
BEGIN LINEAR REGRESSION SCRIPT
Reading X and Y...
Calling the Direct Solver...
Computing the statistics...
AVG_TOT_Y,-0.051722694902638956
STDEV_TOT_Y,54.132787822718356
AVG_RES_Y,1.5905895170230406E-10
STDEV_RES_Y,2.0668015575844624E-8
DISPERSION,4.262683023432828E-16
PLAIN_R2,1.0
ADJUSTED_R2,1.0
PLAIN_R2_NOBIAS,1.0
ADJUSTED_R2_NOBIAS,1.0
PLAIN_R2_VS_0,1.0
ADJUSTED_R2_VS_0,1.0
Writing the output matrix...
END LINEAR REGRESSION SCRIPT
15/11/17 15:50:34 INFO api.DMLScript: SystemML Statistics:
Total execution time: 0.480 sec.
...
[hadoop@host1 ~]$ hadoop jar systemml-{{site.SYSTEMML_VERSION}}/SystemML.jar -f systemml-{{site.SYSTEMML_VERSION}}/algorithms/GLM-predict.dml -nvargs X=linRegData.test.data.csv Y=linRegData.test.labels.csv B=betas.csv fmt=csv
...
LOGLHOOD_Z,,FALSE,NaN
LOGLHOOD_Z_PVAL,,FALSE,NaN
PEARSON_X2,,FALSE,2.5039962709907123E-13
PEARSON_X2_BY_DF,,FALSE,5.703863943031236E-16
PEARSON_X2_PVAL,,FALSE,1.0
DEVIANCE_G2,,FALSE,0.0
DEVIANCE_G2_BY_DF,,FALSE,0.0
DEVIANCE_G2_PVAL,,FALSE,1.0
LOGLHOOD_Z,,TRUE,NaN
LOGLHOOD_Z_PVAL,,TRUE,NaN
PEARSON_X2,,TRUE,2.5039962709907123E-13
PEARSON_X2_BY_DF,,TRUE,5.703863943031236E-16
PEARSON_X2_PVAL,,TRUE,1.0
DEVIANCE_G2,,TRUE,0.0
DEVIANCE_G2_BY_DF,,TRUE,0.0
DEVIANCE_G2_PVAL,,TRUE,1.0
AVG_TOT_Y,1,,0.9381218622147646
STDEV_TOT_Y,1,,55.6116696631821
AVG_RES_Y,1,,2.5577864570734575E-10
STDEV_RES_Y,1,,2.390848397359923E-8
PRED_STDEV_RES,1,TRUE,1.0
PLAIN_R2,1,,1.0
ADJUSTED_R2,1,,1.0
PLAIN_R2_NOBIAS,1,,1.0
ADJUSTED_R2_NOBIAS,1,,1.0
15/11/17 15:51:17 INFO api.DMLScript: SystemML Statistics:
Total execution time: 0.269 sec.
...
If we look at HDFS, we can see the files that were generated by the SystemML DML script executions.
[hadoop@host1 ~]$ hdfs dfs -ls
Found 16 items
drwxr-xr-x - hadoop supergroup 0 2015-11-17 15:50 betas.csv
-rw-r--r-- 3 hadoop supergroup 208 2015-11-17 15:50 betas.csv.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-17 15:21 linRegData.csv
-rw-r--r-- 3 hadoop supergroup 214 2015-11-17 15:21 linRegData.csv.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-17 15:50 linRegData.test.data.csv
-rw-r--r-- 3 hadoop supergroup 213 2015-11-17 15:50 linRegData.test.data.csv.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-17 15:50 linRegData.test.labels.csv
-rw-r--r-- 3 hadoop supergroup 210 2015-11-17 15:50 linRegData.test.labels.csv.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-17 15:49 linRegData.train.data.csv
-rw-r--r-- 3 hadoop supergroup 213 2015-11-17 15:49 linRegData.train.data.csv.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-17 15:49 linRegData.train.labels.csv
-rw-r--r-- 3 hadoop supergroup 210 2015-11-17 15:49 linRegData.train.labels.csv.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-17 15:49 linRegDataParts
drwxr-xr-x - hadoop supergroup 0 2015-11-17 15:21 perc.csv
-rw-r--r-- 3 hadoop supergroup 206 2015-11-17 15:21 perc.csv.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-17 15:21 scratch_space
Before the next example, I'll delete the files created in HDFS by this example.
[hadoop@host1 ~]$ hdfs dfs -rm -r linRegData*
[hadoop@host1 ~]$ hdfs dfs -rm -r *.csv
[hadoop@host1 ~]$ hdfs dfs -rm -r *.mtd
* * *
## SystemML with Distributed Hadoop and YARN: K-Means Clustering Example
Our previous example showed SystemML running in Hadoop Batch mode on a 4-node cluster with YARN.
However, the size of the data used was trivial. In this example, we'll generate a slightly larger set
of data and then analyze that data with the `Kmeans.dml` and `Kmeans-predict.dml` scripts.
Information about the SystemML K-means clustering algorithm can be found in the
[K-Means Clustering](algorithms-clustering.html#k-means-clustering) section of the [SystemML
Algorithms Reference](algorithms-reference.html).
I'm going to modify my `SystemML-config.xml` file.
I updated the `numreducers` property to be 6, which is twice my number of data nodes.
The `numreducers` property specifies the number of reduce tasks per MR job.
<numreducers>6</numreducers>
To begin, I'll download the `genRandData4Kmeans.dml` script that I'll use to generate a set of data.
[hadoop@host1 ~]$ wget https://raw.githubusercontent.com/apache/incubator-systemml/master/scripts/datagen/genRandData4Kmeans.dml
A description of the named arguments that can be passed in to this script can be found in the comment section at the top of the
`genRandData4Kmeans.dml` file. For data, I'll generate a matrix `X.mtx` consisting of 1 million rows and 100 features. I'll explicitly reference my `SystemML-config.xml` file, since I'm
executing SystemML in Hadoop from my home directory rather than from the SystemML project root directory.
[hadoop@host1 ~]$ hadoop jar systemml-{{site.SYSTEMML_VERSION}}/SystemML.jar -f genRandData4Kmeans.dml -config=systemml-{{site.SYSTEMML_VERSION}}/SystemML-config.xml -nvargs nr=1000000 nf=100 nc=10 dc=10.0 dr=1.0 fbf=100.0 cbf=100.0 X=X.mtx C=C.mtx Y=Y.mtx YbyC=YbyC.mtx
After the data generation has finished, I'll check HDFS for the amount of space used. The 1M-row matrix `X.mtx`
requires about 2.8GB of space.
[hadoop@host1 ~]$ hdfs dfs -df -h
Filesystem Size Used Available Use%
hdfs://host1.example.com:9000 400.7 G 2.8 G 318.7 G 1%
Here we can see the data files that were generated.
[hadoop@host1 ~]$ hdfs dfs -ls
Found 9 items
drwxr-xr-x - hadoop supergroup 0 2015-11-19 11:53 C.mtx
-rw-r--r-- 3 hadoop supergroup 176 2015-11-19 11:53 C.mtx.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-19 11:56 X.mtx
-rw-r--r-- 3 hadoop supergroup 186 2015-11-19 11:56 X.mtx.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-19 11:53 Y.mtx
-rw-r--r-- 3 hadoop supergroup 182 2015-11-19 11:53 Y.mtx.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-19 11:56 YbyC.mtx
-rw-r--r-- 3 hadoop supergroup 182 2015-11-19 11:56 YbyC.mtx.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-19 11:56 scratch_space
Here we can see the `X.mtx` data files.
[hadoop@host1 ~]$ hdfs dfs -ls X.mtx
Found 6 items
-rw-r--r-- 1 hadoop supergroup 484418384 2015-11-19 11:56 X.mtx/2-r-00000
-rw-r--r-- 1 hadoop supergroup 481626112 2015-11-19 11:56 X.mtx/2-r-00001
-rw-r--r-- 1 hadoop supergroup 475834931 2015-11-19 11:56 X.mtx/2-r-00002
-rw-r--r-- 1 hadoop supergroup 478519922 2015-11-19 11:56 X.mtx/2-r-00003
-rw-r--r-- 1 hadoop supergroup 481624723 2015-11-19 11:56 X.mtx/2-r-00004
-rw-r--r-- 1 hadoop supergroup 481624048 2015-11-19 11:56 X.mtx/2-r-00005
Next, I'll run the `Kmeans.dml` algorithm on the 1M-row matrix `X.mtx`.
[hadoop@host1 ~]$ hadoop jar systemml-{{site.SYSTEMML_VERSION}}/SystemML.jar -f systemml-{{site.SYSTEMML_VERSION}}/algorithms/Kmeans.dml -config=/systemml-{{site.SYSTEMML_VERSION}}/SystemML-config.xml -nvargs X=X.mtx k=5 C=Centroids.mtx
We can see the `Centroids.mtx` data file has been written to HDFS.
[hadoop@host1 ~]$ hdfs dfs -ls
Found 11 items
drwxr-xr-x - hadoop supergroup 0 2015-11-19 11:53 C.mtx
-rw-r--r-- 3 hadoop supergroup 176 2015-11-19 11:53 C.mtx.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-19 12:10 Centroids.mtx
-rw-r--r-- 3 hadoop supergroup 174 2015-11-19 12:10 Centroids.mtx.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-19 11:56 X.mtx
-rw-r--r-- 3 hadoop supergroup 186 2015-11-19 11:56 X.mtx.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-19 11:53 Y.mtx
-rw-r--r-- 3 hadoop supergroup 182 2015-11-19 11:53 Y.mtx.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-19 11:56 YbyC.mtx
-rw-r--r-- 3 hadoop supergroup 182 2015-11-19 11:56 YbyC.mtx.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-19 12:10 scratch_space
Now that we have trained our model, next we will test our model. We can do this with
the `Kmeans-predict.dml` script.
[hadoop@host1 ~]$ hadoop jar systemml-{{site.SYSTEMML_VERSION}}/SystemML.jar -f systemml-{{site.SYSTEMML_VERSION}}/algorithms/Kmeans-predict.dml -config=systemml-{{site.SYSTEMML_VERSION}}/SystemML-config.xml -nvargs X=X.mtx C=Centroids.mtx prY=PredY.mtx O=stats.txt
In the file system, we can see that the `PredY.mtx` matrix was created.
The `stats.txt` file lists statistics about the results.
[hadoop@host1 ~]$ hdfs dfs -ls
Found 15 items
drwxr-xr-x - hadoop supergroup 0 2015-11-19 11:53 C.mtx
-rw-r--r-- 3 hadoop supergroup 176 2015-11-19 11:53 C.mtx.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-19 12:10 Centroids.mtx
-rw-r--r-- 3 hadoop supergroup 174 2015-11-19 12:10 Centroids.mtx.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-19 13:20 PredY.mtx
-rw-r--r-- 3 hadoop supergroup 182 2015-11-19 13:20 PredY.mtx.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-19 11:56 X.mtx
-rw-r--r-- 3 hadoop supergroup 186 2015-11-19 11:56 X.mtx.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-19 11:53 Y.mtx
-rw-r--r-- 3 hadoop supergroup 182 2015-11-19 11:53 Y.mtx.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-19 11:56 YbyC.mtx
-rw-r--r-- 3 hadoop supergroup 182 2015-11-19 11:56 YbyC.mtx.mtd
drwxr-xr-x - hadoop supergroup 0 2015-11-19 13:21 scratch_space
-rw-r--r-- 3 hadoop supergroup 261 2015-11-19 13:21 stats.txt
-rw-r--r-- 3 hadoop supergroup 127 2015-11-19 13:21 stats.txt.mtd
The `PredY.mtx` matrix consists of a single column of a million rows of doubles, as we can
see in the resulting metadata file.
[hadoop@host1 ~]$ hdfs dfs -cat PredY.mtx.mtd
{
"data_type": "matrix"
,"value_type": "double"
,"rows": 1000000
,"cols": 1
,"nnz": 1000000
,"format": "text"
,"description": { "author": "SystemML" }
}
The statistics generated from testing the method are displayed below.
[hadoop@host1 ~]$ hdfs dfs -cat stats.txt
TSS,,1.1262427174414966E11
WCSS_M,,9.77022617396343E10
WCSS_M_PC,,86.75062686450579
BCSS_M,,1.4922010004515366E10
BCSS_M_PC,,13.249373135494215
WCSS_C,,9.770230517014426E10
WCSS_C_PC,,86.75066542680617
BCSS_C,,1.4921964103415842E10
BCSS_C_PC,,13.249332379537428
* * *
# Recommended Hadoop Cluster Configuration Settings
Below are some recommended Hadoop configuration file settings that may be of assistance when running SystemML on Hadoop
in a clustered environment.
<table>
<thead>
<tr>
<th>Configuration File</th>
<th>Setting</th>
<th>Value</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td nowrap rowspan="5" style="vertical-align: top; padding-top: 22px;"><code>mapred-site.xml</code></td>
<td nowrap><code>mapreduce.map.java.opts</code></td>
<td nowrap><code>-Xmx2g -Xms2g -Xmn200m</code></td>
<td>Increase memory of child JVMs of Maps, max/min heap size of 2GB, -Xmn specifies young generation which is typically 10% of maximum heap size</td>
</tr>
<tr>
<td nowrap><code>mapreduce.reduce.java.opts</code></td>
<td nowrap><code>-Xmx2g -Xms2g -Xmn200m</code></td>
<td>Increase memory of child JVMs of Reduces, max/min heap size of 2GB, -Xmn specifies young generation which is typically 10% of maximum heap size</td>
</tr>
<tr>
<td nowrap><code>mapreduce.map.memory.mb</code></td>
<td nowrap><code>3072</code></td>
<td>Set to at least 1.5 times the value of the Map max heap size</td>
</tr>
<tr>
<td nowrap><code>mapreduce.reduce.memory.mb</code></td>
<td nowrap><code>3072</code></td>
<td>Set to at least 1.5 times the value of the Reduce max heap size</td>
</tr>
<tr>
<td nowrap><code>io.sort.mb</code> (deprecated) /<br/> <code>mapreduce.task.io.sort.mb</code></td>
<td nowrap><code>384</code></td>
<td>Memory limit while sorting data</td>
</tr>
<tr>
<td nowrap><code>yarn-site.xml</code></td>
<td nowrap><code>yarn.nodemanager.vmem-pmem-ratio</code></td>
<td nowrap><code>2</code> to <code>5</code></td>
<td>Maximum ratio of virtual memory to physical memory</td>
</tr>
</tbody>
</table>