Myriad allows Mesos and YARN to co-exist and share resources with Mesos as the resource manager for the datacenter. Sharing resources between these two resource allocation systems improves overall cluster utilization and avoids statically partitioning resources among separate clusters and separate resource managers.
The following diagram shows two resource managers running independently which results in a statically partitioned datacenter:
The Mesos Slave and YARN’s Node Manager are processes that run on the host OS. Both processes advertise available resources to Mesos Master and YARN Resource Manager respectively. Each process can be configured to advertise a subset of resources. This ability is leveraged, in conjunction with Cgroups, to allow Mesos Slave and YARN Node Manager to co-exist on a node.
/sys/fs/cgroup/cpu/mesos/node-manager-task-id/container-1
The following diagram shows a node running YARN NodeManager as a Mesos Slave task:
One way to avoid static partitioning and to enable resource sharing when running two resource managers, is to let one resource manager be in absolute control of the datacenter’s resources. The other resource manager then manages a subset of resources, allocated to it through the primary resource manager.
The following diagram shows a scenario where Mesos is used as the resource manager for the datacenter which allows both Mesos and YARN to schedule tasks on any node.
Each node in the cluster has both daemons, Mesos Slave and YARN Node Manager, installed. By default, the Mesos slave daemon is started on each node and advertises all available resources to the Mesos Master.
The following diagram shows how Myriad launches a YARN Node Manager as a task under Mesos Slave: