These instructions apply to source release downloads only. If Myriad sources are checked out from the git repository, the gradle wrapper script will already be available.
Myriad‘s source release tarball does not include gradle wrapper script. So you’ll need to:
gradle wrapper
command from the $PROJECT_ROOT of the Myriad project. This generates a $PROJECT_ROOT/gradlew
script, and also places a supporting jar at $PROJECT_ROOT/gradle/wrapper/gradle-wrapper.jar
.Ensure $PROJECT_ROOT/gradlew
script is available. Run the following command from $PROJECT_ROOT to build Myriad (both Scheduler and Executor components):
./gradlew build
At this point the jars will be located in the following directories:
# scheduler jars $PROJECT_HOME/myriad-scheduler/build/libs/ # executor jar $PROJECT_HOME/myriad-executor/build/libs/
To build only the Scheduler, from $PROJECT_HOME, run:
./gradlew :myriad-scheduler:build
To build only the Executor, from $PROJECT_HOME, run:
./gradlew :myriad-executor:build
To deploy Myriad Scheduler and Executor Binaries:
For example:
cp myriad-scheduler/build/libs/*.jar /opt/hadoop-2.7.0/share/hadoop/yarn/lib/ cp myriad-executor/build/libs/myriad-executor-<version>.jar /opt/hadoop-2.7.0/share/hadoop/yarn/lib/ cp myriad-scheduler/build/resources/main/myriad-config-default.yml /opt/hadoop-2.7.0/etc/hadoop/
NOTE: For advanced users, you can also copy myriad-executor-.jar to any other directory on a slave filesystem or it can be copied to HDFS as well. In either case, you need to update the executor's path property in the myriad-config-default.yml file and prepend the path with either file:// or hdfs://, as appropriate.
As a minimum, the following Myriad configuration parameters must be set:
Enabling Cgroups involves modifying the yarn-site.xml and myriad-config-default.yml files. If you plan on using Cgroups, you could set that property at this time. See Configuring Cgroup for more information.
NOTE: By copying the myriad-config-default.yml file to the /etc/hadoop directory, you can make changes to the configuration file without having to rebuild Myriad. If you specify the Myriad configuration parameters before building Myriad, you must rebuild Myriad and redeploy the jar files. This is required because the myriad-config-default.yml file is embedded into the Myriad Scheduler jar.
In order to run Myriad, the following YARN properties must be modified on each node in the cluster:
export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
<property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>${nodemanager.resource.cpu-vcores}</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>${nodemanager.resource.memory-mb}</value> </property> <!--These options enable dynamic port assignment by mesos --> <property> <name>yarn.nodemanager.address</name> <value>${myriad.yarn.nodemanager.address}</value> </property> <property> <name>yarn.nodemanager.webapp.address</name> <value>${myriad.yarn.nodemanager.webapp.address}</value> </property> <property> <name>yarn.nodemanager.webapp.https.address</name> <value>${myriad.yarn.nodemanager.webapp.address}</value> </property> <property> <name>yarn.nodemanager.localizer.address</name> <value>${myriad.yarn.nodemanager.localizer.address}</value> </property> <!-- Configure Myriad Scheduler here --> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.myriad.scheduler.yarn.MyriadFairScheduler</value> <description>One can configure other scehdulers as well from following list: org.apache.myriad.scheduler.yarn.MyriadCapacityScheduler, org.apache.myriad.scheduler.yarn.MyriadFifoScheduler</description> </property>
Edit the $YARN_HOME/etc/hadoop/mapred-site.xml file and add the dynamic port assignment properties.
// Add following to $YARN_HOME/etc/hadoop/mapred-site.xml: <!--This option enables dynamic port assignment by mesos --> <property> <name>mapreduce.shuffle.port</name> <value>${myriad.mapreduce.shuffle.port}</value> </property>
Myriad Scheduler runs inside Resource Manager as a plugin. To start the Resource Manager:
./sbin/yarn-daemon.sh start resourcemanager
Myriad Executor and Node Managers are launched automatically by Myriad Scheduler as a response to flexup and flex down behavior. See the Myriad Cluster API.
You are welcome to use any editor you like, however, many of the myriad developers use IntelliJ IDEA. The build process described below uses gradle. The build script is configured with the idea plugin to create an intellij project structure. This is accomplished with the following command:
./gradlew idea # to open on a mac open myriad.ipr
NOTE: If you are interested in using eclipse, there is a plugin for that. Either send in a pull request, or send a message on the dev mail list that you would make use of it.