Overview of Mesos

Mesos is a cluster management platform that provides resource sharing and isolation across distributed applications. For example, Hadoop MapReduce, MPI, Hypertable, and Spark (a new MapReduce-like framework from the Mesos team that supports low-latency interactive and iterative jobs) can run on Mesos.

Mesos Use-cases

  • Run Hadoop, MPI, Spark and other cluster applications on a dynamically shared pool of machines
  • Run multiple instances of Hadoop on the same cluster, e.g. separate instances for production and experimental jobs, or even multiple versions of Hadoop
  • Build new cluster applications without having to reinvent low-level facilities for running tasks on different nodes, monitoring them, etc., and have your applications coexist with existing ones

Features of Mesos

  • Master fault tolerance using ZooKeeper
  • Isolation between tasks using Linux Containers
  • Memory and CPU aware allocation
  • Efficient fine-grained resource sharing abstraction (resource offers) that allows applications to achieve app-specific scheduling goals (e.g. Hadoop optimizes for data locality)
    Visit mesosproject.org for more details about Mesos.

Please note that Mesos is still in beta. Though the current version is in use in production at Twitter, it may have some stability issues in certain environments.

What you'll find on this page

  1. Quick-start Guides
  2. System Requirements
  3. Downloading Mesos
  4. Building Mesos
  5. Testing the Build
  6. Deploying to a Cluster
  7. Where to Go From Here

System Requirements

Mesos runs on Linux and Mac OS X, and has previously also been tested on OpenSolaris. You will need the following packages to run it:

  • g++ 4.1 or higher.
  • Python 2.6 (for the Mesos web UI). On Mac OS X 10.6 or earlier, get Python from MacPorts via port install python26, because OS X's version of Python is not 64-bit.
  • Python 2.6 developer packages (on red-hat - sudo yum install python26-devel python-devel)
  • cppunit (for building zookeeper) (on red-hat - sudo yum install cppunit-devel)
  • Java JDK 1.6 or higher. Mac OS X 10.6 users will need to install the JDK from http://connect.apple.com/ and set JAVA_HEADERS=/System/Library/Frameworks/JavaVM.framework/Versions/A/Headers.

Downloading and Building Mesos

Mesos uses the standard GNU build tools. See the section farther below for instructions for checking out and building the source code via source control.

To get running with Mesos version 0.9.0-incubating, our most recently release:

  1. Download Mesos 0.9.0-incubating
  2. Run configure (there are a few different helper scripts that wrap the configure script called configure.)
    1. In OS X: run ./configure.macosx. If you're running Mountain Lion, you may need to follow the instructions here and here.
    2. In Linux, you can probably just run ./configure, which includes the web UI and included Zookeeper. Advanced users may want to exclude these options or include other options, see Mesos Command-Line Flags.
  3. run make

NOTES:

  • In the SVN trunk branch since Jan 19 2012 (when Mesos switched fully to the GNU Autotools build system), the build process attempts to guess where your Java include directory is, but if you have set the $JAVA_HOME environment variable, it will use $JAVA_HOME/include, which may not be correct (or exist) on your machine (in which case you will see an error such as: “configure: error: failed to build against JDK (using libtool)”). If this is the case, we suggest you unset the JAVA_HOME environment variable.
  • configure may print a warning at the end about “unrecognized options: --with-java-home, ...”. This comes from one of the nested configure scripts that we call, so it doesn't mean that your options were ignored.
  • (NOT SURE IF THIS IS STILL RELEVANT) On 32-bit platforms, you should set CXXFLAGS="-march=i486" when running configure to ensure certain atomic instructions can be used.

Checking Mesos Source Code out of Git or SVN

Currently, you can obtain the current Mesos development HEAD by checking it out from either the Apache SVN or Apache Git repository (the git repo is a mirror of the SVN repo)

  • For SVN, use: svn co https://svn.apache.org/repos/asf/incubator/mesos/trunk mesos-trunk
  • For git, use: git clone git://git.apache.org/mesos.git

Running Example Frameworks and Testing the Build

Currently, in order to run the example frameworks (in src/examples), you must first build the test suite, as instructed below.

After you build Mesos by running the make command, you can build and run its example frameworks and unit tests (which use the example frameworks) by issuing the make check command from the directory where you ran the make command.

After you have done this, you can also set up a small Mesos cluster and run a job on it as follows:

  1. Go into the directory where you built Mesos.
  2. Type bin/mesos-master.sh to launch the master.
  3. Take note of the IP and port that the master is running on, which will look something like 192.168.0.1:5050. Note: In this example the IP address of master is 192.168.0.1, and the port is 5050. We will continue to use the URL shown here for the rest of this example; however when you run the following commands replace all instances of it with the URL of your master.
  4. URL of master: 192.168.0.1:5050
  5. View the master's web UI at http://[hostname of master]:5050.
  6. Launch a slave by typing bin/mesos-slave.sh --master=192.168.0.1:5050. The slave will show up on the master's web UI if you refresh it.
  7. Run the C++ test framework (a sample that just runs five tasks on the cluster) using src/test-framework 192.168.0.1:5050. It should successfully exit after running five tasks.
  8. You can also try the example python or Java frameworks, with commands like the following:
  9. src/examples/java/test-framework 192.168.0.1:5050
  10. src/examples/python/test-framework 192.168.0.1:5050

Deploying to a Cluster

To run Mesos on more than one machine, you have multiple options:

  • To launch a cluster on a private cluster that you own, use Mesos's deploy scripts
  • To launch a cluster on Amazon EC2, you can use the Mesos EC2 scripts

Where to Go From Here