This section provides information for setting up a cluster in a virtual machine. Setting up a cluster in a virtual machine involves the following:
To start the cluster run following:
vagrant up
At this point the VM will have a single node Mesos cluster running and a single node HDFS cluster running. You can modify the Mesos and Hadoop versions by editing the corresponding variables in the Vagrantfile
.
Check that Mesos and Hadoop are running with a browser:
Hadoop/HDFS namenode Mesos Master
To ssh in the cluster, run following:
vagrant ssh
The password for vagrant user is vagrant if prompted You can switch to the hadoop user when needed to launch hadoop processes
sudo su - hduser
To build Myriad Scheduler inside VM, run the gradlew build:
cd /vagrant ./gradlew build
NOTE: If build failure failure occurs, the issue is not with the build itself, but a failure to write to disk. This can happen when you built outside the vagrant instance first. Exit the user hduser
by typing exit
and build again as the vagrant
user.
The Myriad Schedule and Executer jar files and all the runtime dependences as well as the Myriad configuration file must be copied to $YARN_HOME.
/vagrant/myriad-scheduler/build/libs/*
/vagrant/myriad-scheduler/src/main/resources/myriad-config-default.yml
/vagrant/myriad-scheduler/build/libs/myriad-executor-0.1.0.jar
For example, the files are copied to the following locations:
cp /vagrant/myriad-scheduler/build/libs/* $YARN_HOME/share/hadoop/yarn/lib/ cp /vagrant/myriad-executor/build/libs/myriad-executor-0.1.0.jar $YARN_HOME/share/hadoop/yarn/lib/ cp /vagrant/myriad-scheduler/src/main/resources/myriad-config-default.yml $YARN_HOME/etc/hadoop/
As a minimum, the following Myriad configuration parameters must be set:
NOTE: Enabling Cgroups involves modifying the yarn-site.xml and myriad-config-default.yml files. If you plan on using Cgroups, you could set that property at this time. See Configuring Cgroups for more information.
To configure Myriad itself, update $YARN_HOME/etc/hadoop/myriad-default-config.yml with the following content:
mesosMaster: <mesos Master IP address>:5050 checkpoint: false frameworkFailoverTimeout: 43200000 frameworkName: MyriadAlpha nativeLibrary: /usr/local/lib/libmesos.so zkServers: localhost:2181 zkTimeout: 20000 profiles: small: cpu: 2 mem: 2048 medium: cpu: 4 mem: 4096 large: cpu: 10 mem: 12288 rebalancer: true nodemanager: jvmMaxMemoryMB: 1024 user: hduser cpus: 0.2 cgroups: false executor: jvmMaxMemoryMB: 256 path: file://localhost/usr/local/libexec/mesos/myriad-executor-runnable-0.1.0.jar
To configure YARN to use Myriad, update $YARN_HOME/etc/hadoop/yarn-site.xml with following content:
<property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>${nodemanager.resource.cpu-vcores}</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>${nodemanager.resource.memory-mb}</value> </property> <!-- The following properties enable dynamic port assignment by Mesos --> <property> <name>yarn.nodemanager.address</name> <value>${myriad.yarn.nodemanager.address}</value> </property> <property> <name>yarn.nodemanager.webapp.address</name> <value>${myriad.yarn.nodemanager.webapp.address}</value> </property> <property> <name>yarn.nodemanager.webapp.https.address</name> <value>${myriad.yarn.nodemanager.webapp.address}</value> </property> <property> <name>yarn.nodemanager.localizer.address</name> <value>${myriad.yarn.nodemanager.localizer.address}</value> </property> <!-- The following properties configure Myriad Scheduler --> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.myriad.scheduler.yarn.MyriadFairScheduler</value> <description>One can configure other schedulers as well from following list: org.apache.myriad.scheduler.yarn.MyriadCapacityScheduler, org.apache.myriad.scheduler.yarn.MyriadFifoScheduler</description> </property>
To launch Myriad, run the following:
sudo su hduser yarn-daemon.sh start resourcemanager
To check that things are running, from a browser on the host check out the following urls:
To shut down, from the vagrant ssh console, run the following:
yarn-daemon.sh stop resourcemanager sh ./vagrant/shutdown.sh exit exit vagrant halt