Docker is the easiest way to from 0 to Yarn on Mesos within minutes.
YARN runs as a user and group, and expects consistent uid and gids in HDFS and accross the cluster. Hence it is necessary to edit docker/Dockerfile and modify the following:
ENV YARN_USER="yarn" ENV YARN_UID="107" ENV HADOOP_GROUP="hadoop" ENV HADOOP_GID="112" ENV YARN_GROUP="yarn" ENV YARN_UID="113"
The run the following commands:
./gradlew -P dockerTag=username/myriad buildRMDocker docker push username/myriad
This will build the ResourceManager from src, save, and push the image as username/myriad.
It is assumed you already have hdfs and Mesos running. For more information about Apache Mesos visit the website. If you need to setup hdfs, consider using the hdfs-mesos framework.
In order for the ResourceManager to operate correctly, you will need to provide 5 configuration files. These files will need to mounted from a directory into /usr/local/hadoo/etc
of the docker container.
Edit and copy the file templates along with the hdfs.xml and core-site.xml from your hdfs configuration into a directory named config
. Additional, files maybe necessary such as rack.sh, log4j.properties, hadoop-env.sh, and yarn-env.sh depending on your configuration. Create a second directoy called dist
. Then tar the files using tar -zcf dist\config.tgz config
.
From the directories containing dist
and config
execute:
docker run --net=host -v $PWD/dist -v $PWD/config:/usr/local/hadoop/etc/hadoop --name='myriad-resourcemanager' -t \ <username>/myriad
There is no need to to rebuild the docker image to update YARN's configuration, simply edit the files in config and repackage using tar -zcf dist\config.tgz config
. To update the configuration on the Nodemanagers they will need to be flexed down and flexed up, to update the resource manager the docker container will need to be restarted:
docker kill myriad-resourcemanager docker run --net=host -v $PWD/dist -v $PWD/config:/usr/local/hadoop/etc/hadoop --name='myriad-resourcemanager' -t \ <username>/myriad