publish/documentation/latest/high-availability/index.html - mesos-site - Git at Google

 <!DOCTYPE html>
 <html>
   <head>
     <meta charset="utf-8">
     <title>Apache Mesos - High-Availability Mode</title>
     <meta name="viewport" content="width=device-width, initial-scale=1.0">

     <meta property="og:locale" content="en_US"/>
     <meta property="og:type" content="website"/>
     <meta property="og:title" content="Apache Mesos"/>
     <meta property="og:site_name" content="Apache Mesos"/>
     <meta property="og:url" content="http://mesos.apache.org/"/>
     <meta property="og:image" content="http://mesos.apache.org/assets/img/mesos_logo_fb_preview.png"/>
     <meta property="og:description"
           content="Apache Mesos abstracts resources away from machines,
                    enabling fault-tolerant and elastic distributed systems
                    to easily be built and run effectively."/>

     <meta name="twitter:card" content="summary"/>
     <meta name="twitter:site" content="@ApacheMesos"/>
     <meta name="twitter:title" content="Apache Mesos"/>
     <meta name="twitter:image" content="http://mesos.apache.org/assets/img/mesos_logo_fb_preview.png"/>
     <meta name="twitter:description"
           content="Apache Mesos abstracts resources away from machines,
                    enabling fault-tolerant and elastic distributed systems
                    to easily be built and run effectively."/>

     <link href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
     <link rel="alternate" type="application/atom+xml" title="Apache Mesos Blog" href="/blog/feed.xml">
     <link href="../../../assets/css/main.css" media="screen" rel="stylesheet" type="text/css" />


     <!-- Google Analytics Magic -->
     <script type="text/javascript">
     var _gaq = _gaq || [];
     _gaq.push(['_setAccount', 'UA-20226872-1']);
     _gaq.push(['_setDomainName', 'apache.org']);
     _gaq.push(['_trackPageview']);

     (function() {
       var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
       ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
       var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
     })();
     </script>

   </head>
   <body>
     <!-- magical breadcrumbs -->
     <div class="topnav">
       <div class="container">
         <ul class="breadcrumb">
           <li>
             <div class="dropdown">
               <a data-toggle="dropdown" href="#">Apache Software Foundation <span class="caret"></span></a>
               <ul class="dropdown-menu" role="menu" aria-labelledby="dLabel">
                 <li><a href="http://www.apache.org">Apache Homepage</a></li>
                 <li><a href="http://www.apache.org/licenses/">License</a></li>
                 <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
                 <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
                 <li><a href="http://www.apache.org/security/">Security</a></li>
               </ul>
             </div>
           </li>

           <li><a href="http://mesos.apache.org">Apache Mesos</a></li>


           <li><a href="/documentation
 /">Documentation
 </a></li>


         </ul><!-- /.breadcrumb -->
       </div><!-- /.container -->
     </div><!-- /.topnav -->

     <!-- navbar excitement -->
 <div class="navbar navbar-default navbar-static-top" role="navigation">
   <div class="container">
     <div class="navbar-header">
       <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#mesos-menu" aria-expanded="false">
       <span class="sr-only">Toggle navigation</span>
       <span class="icon-bar"></span>
       <span class="icon-bar"></span>
       <span class="icon-bar"></span>
       </button>
       <a class="navbar-brand" href="/"><img src="/assets/img/mesos_logo.png" alt="Apache Mesos logo"/></a>
     </div><!-- /.navbar-header -->

     <div class="navbar-collapse collapse" id="mesos-menu">
       <ul class="nav navbar-nav navbar-right">
         <li><a href="/gettingstarted/">Getting Started</a></li>
         <li><a href="/blog/">Blog</a></li>
         <li><a href="/documentation/latest/">Documentation</a></li>
         <li><a href="/downloads/">Downloads</a></li>
         <li><a href="/community/">Community</a></li>
       </ul>
     </div><!-- /#mesos-menu -->
   </div><!-- /.container -->
 </div><!-- /.navbar -->

 <div class="content">
   <div class="container">
     <div class="row-fluid">
   <div class="col-md-4">
     <h4>If you're new to Mesos</h4>
     <p>See the <a href="/gettingstarted/">getting started</a> page for more
        information about downloading, building, and deploying Mesos.</p>

     <h4>If you'd like to get involved or you're looking for support</h4>
     <p>See our <a href="/community/">community</a> page for more details.</p>
   </div>
   <div class="col-md-8">
     <h1>Mesos High-Availability Mode</h1>

 <p>If the Mesos master is unavailable, existing tasks can continue to execute, but new resources cannot be allocated and new tasks cannot be launched. To reduce the chance of this situation occurring, Mesos has a high-availability mode that uses multiple Mesos masters: one active master (called the <em>leader</em> or leading master) and several <em>backups</em> in case it fails. The masters elect the leader, with <a href="http://zookeeper.apache.org/">Apache ZooKeeper</a> both coordinating the election and handling leader detection by masters, agents, and scheduler drivers. More information regarding <a href="http://zookeeper.apache.org/doc/trunk/recipes.html#sc_leaderElection">how leader election works</a> is available on the Apache Zookeeper website.</p>

 <p>This document describes how to configure Mesos to run in high-availability mode. For more information on developing highly available frameworks, see a <a href="/documentation/latest/./high-availability-framework-guide/">companion document</a>.</p>

 <p><strong>Note</strong>: This document assumes you know how to start, run, and work with ZooKeeper, whose client library is included in the standard Mesos build.</p>

 <h2>Usage</h2>

 <p>To put Mesos into high-availability mode:</p>

 <ol>
 <li><p>Ensure that the ZooKeeper cluster is up and running.</p></li>
 <li><p>Provide the znode path to all masters, agents, and framework schedulers as follows:</p>

 <ul>
 <li><p>Start the mesos-master binaries using the <code>--zk</code> flag, e.g. <code>--zk=zk://host1:port1,host2:port2,.../path</code></p></li>
 <li><p>Start the mesos-agent binaries with <code>--master=zk://host1:port1,host2:port2,.../path</code></p></li>
 <li><p>Start any framework schedulers using the same <code>zk</code> path as in the last two steps. The SchedulerDriver must be constructed with this path, as shown in the <a href="/documentation/latest/./app-framework-development-guide/">Framework Development Guide</a>.</p></li>
 </ul>
 </li>
 </ol>


 <p>From now on, the Mesos masters and agents all communicate with ZooKeeper to find out which master is the current leading master. This is in addition to the usual communication between the leading master and the agents.</p>

 <p>In addition to ZooKeeper, one can get the location of the leading master by sending an HTTP request to <a href="/documentation/latest/./endpoints/master/redirect/">/redirect</a> endpoint on any master.</p>

 <p>For HTTP endpoints that only work at the leading master, requests made to endpoints at a non-leading master will result in either a <code>307 Temporary Redirect</code> (with the location of the leading master) or <code>503 Service Unavailable</code> (if the master does not know who the current leader is).</p>

 <p>Refer to the <a href="/documentation/latest/./app-framework-development-guide/">Scheduler API</a> for how to deal with leadership changes.</p>

 <h2>Component Disconnection Handling</h2>

 <p>When a network partition disconnects a component (master, agent, or scheduler driver) from ZooKeeper, the component&rsquo;s Master Detector induces a timeout event. This notifies the component that it has no leading master. Depending on the component, the following happens. (Note that while a component is disconnected from ZooKeeper, a master may still be in communication with agents or schedulers and vice versa.)</p>

 <ul>
 <li><p>Agents disconnected from ZooKeeper no longer know which master is the leader. They ignore messages from masters to ensure they don&rsquo;t act on a non-leader&rsquo;s decisions. When an agent reconnects to ZooKeeper, ZooKeeper informs it of the current leader and the agent stops ignoring messages from the leader.</p></li>
 <li><p>Masters enter leaderless state irrespective of whether they are a leader or not before the disconnection.</p>

 <ul>
 <li><p>If the leader was disconnected from ZooKeeper, it aborts its process. The user/developer/administrator can then start a new master instance which will try to reconnect to ZooKeeper.</p>

 <ul>
 <li>Note that many production deployments of Mesos use a process supervisor (such as systemd or supervisord) that is configured to automatically restart the Mesos master if the process aborts unexpectedly.</li>
 </ul>
 </li>
 <li><p>Otherwise, the disconnected backup waits to reconnect with ZooKeeper and possibly get elected as the new leading master.</p></li>
 </ul>
 </li>
 <li><p>Scheduler drivers disconnected from the leading master notify the scheduler about their disconnection from the leader.</p></li>
 </ul>


 <p>When a network partition disconnects an agent from the leader:</p>

 <ul>
 <li><p>The agent fails health checks from the leader.</p></li>
 <li><p>The leader marks the agent as deactivated and sends its tasks to the LOST state. The  <a href="/documentation/latest/./app-framework-development-guide/">Framework Development Guide</a> describes these various task states.</p></li>
 <li><p>Deactivated agents may not re-register with the leader and are told to shut down upon any post-deactivation communication.</p></li>
 </ul>


 <h2>Implementation Details</h2>

 <p>Mesos implements two levels of ZooKeeper leader election abstractions, one in <code>src/zookeeper</code> and the other in <code>src/master</code> (look for <code>contender|detector.hpp|cpp</code>).</p>

 <ul>
 <li><p>The lower level <code>LeaderContender</code> and <code>LeaderDetector</code> implement a generic ZooKeeper election algorithm loosely modeled after this
 <a href="http://zookeeper.apache.org/doc/trunk/recipes.html#sc_leaderElection">recipe</a> (sans herd effect handling due to the master group&rsquo;s small size, which is often 3).</p></li>
 <li><p>The higher level <code>MasterContender</code> and <code>MasterDetector</code> wrap around ZooKeeper&rsquo;s contender and detector abstractions as adapters to provide/interpret the ZooKeeper data.</p></li>
 <li><p>Each Mesos master simultaneously uses both a contender and a detector to try to elect themselves and detect who the current leader is. A separate detector is necessary because each master&rsquo;s WebUI redirects browser traffic to the current leader when that master is not elected. Other Mesos components (i.e., agents and scheduler drivers) use the detector to find the current leader and connect to it.</p></li>
 </ul>


 <p>The notion of the group of leader candidates is implemented in <code>Group</code>. This abstraction handles reliable (through queues and retries of retryable errors under the covers) ZooKeeper group membership registration, cancellation, and monitoring. It watches for several ZooKeeper session events:</p>

 <ul>
 <li>Connection</li>
 <li>Reconnection</li>
 <li>Session Expiration</li>
 <li>ZNode creation, deletion, updates</li>
 </ul>


 <p>We also explicitly timeout our sessions when disconnected from ZooKeeper for a specified amount of time. See <code>MASTER_CONTENDER_ZK_SESSION_TIMEOUT</code> and <code>MASTER_DETECTOR_ZK_SESSION_TIMEOUT</code>. This is because the ZooKeeper client libraries only notify of session expiration upon reconnection. These timeouts are of particular interest for network partitions.</p>

   </div>
 </div>

   </div><!-- /.container -->
 </div><!-- /.content -->

 <hr>


     <!-- footer -->
     <div class="footer">
       <div class="container">
         <div class="col-md-4 social-blk">
           <span class="social">
             <a href="https://twitter.com/ApacheMesos"
               class="twitter-follow-button"
               data-show-count="false" data-size="large">Follow @ApacheMesos</a>
             <script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script>
             <a href="https://twitter.com/intent/tweet?button_hashtag=mesos"
               class="twitter-hashtag-button"
               data-size="large"
               data-related="ApacheMesos">Tweet #mesos</a>
             <script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script>
           </span>
         </div>

         <div class="col-md-8 trademark">
           <p>&copy; 2012-2017 <a href="http://apache.org">The Apache Software Foundation</a>.
             Apache Mesos, the Apache feather logo, and the Apache Mesos project logo are trademarks of The Apache Software Foundation.
           <p>
         </div>
       </div><!-- /.container -->
     </div><!-- /.footer -->

     <!-- JS -->
     <script src="//code.jquery.com/jquery-1.11.0.min.js" type="text/javascript"></script>
     <script src="//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js" type="text/javascript"></script>
   </body>
 </html>
	<!DOCTYPE html>
	<html>
	<head>
	<meta charset="utf-8">
	<title>Apache Mesos - High-Availability Mode</title>
	<meta name="viewport" content="width=device-width, initial-scale=1.0">

	<meta property="og:locale" content="en_US"/>
	<meta property="og:type" content="website"/>
	<meta property="og:title" content="Apache Mesos"/>
	<meta property="og:site_name" content="Apache Mesos"/>
	<meta property="og:url" content="http://mesos.apache.org/"/>
	<meta property="og:image" content="http://mesos.apache.org/assets/img/mesos_logo_fb_preview.png"/>
	<meta property="og:description"
	content="Apache Mesos abstracts resources away from machines,
	enabling fault-tolerant and elastic distributed systems
	to easily be built and run effectively."/>

	<meta name="twitter:card" content="summary"/>
	<meta name="twitter:site" content="@ApacheMesos"/>
	<meta name="twitter:title" content="Apache Mesos"/>
	<meta name="twitter:image" content="http://mesos.apache.org/assets/img/mesos_logo_fb_preview.png"/>
	<meta name="twitter:description"
	content="Apache Mesos abstracts resources away from machines,
	enabling fault-tolerant and elastic distributed systems
	to easily be built and run effectively."/>

	<link href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
	<link rel="alternate" type="application/atom+xml" title="Apache Mesos Blog" href="/blog/feed.xml">
	<link href="../../../assets/css/main.css" media="screen" rel="stylesheet" type="text/css" />



	<!-- Google Analytics Magic -->
	<script type="text/javascript">
	var _gaq = _gaq \|\| [];
	_gaq.push(['_setAccount', 'UA-20226872-1']);
	_gaq.push(['_setDomainName', 'apache.org']);
	_gaq.push(['_trackPageview']);

	(function() {
	var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
	ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
	var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
	})();
	</script>

	</head>
	<body>
	<!-- magical breadcrumbs -->
	<div class="topnav">
	<div class="container">
	<ul class="breadcrumb">
	<li>
	<div class="dropdown">
	<a data-toggle="dropdown" href="#">Apache Software Foundation <span class="caret"></span></a>
	<ul class="dropdown-menu" role="menu" aria-labelledby="dLabel">
	<li><a href="http://www.apache.org">Apache Homepage</a></li>
	<li><a href="http://www.apache.org/licenses/">License</a></li>
	<li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
	<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
	<li><a href="http://www.apache.org/security/">Security</a></li>
	</ul>
	</div>
	</li>

	<li><a href="http://mesos.apache.org">Apache Mesos</a></li>


	<li><a href="/documentation
	/">Documentation
	</a></li>


	</ul><!-- /.breadcrumb -->
	</div><!-- /.container -->
	</div><!-- /.topnav -->

	<!-- navbar excitement -->
	<div class="navbar navbar-default navbar-static-top" role="navigation">
	<div class="container">
	<div class="navbar-header">
	<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#mesos-menu" aria-expanded="false">
	<span class="sr-only">Toggle navigation</span>
	<span class="icon-bar"></span>
	<span class="icon-bar"></span>
	<span class="icon-bar"></span>
	</button>
	<a class="navbar-brand" href="/"><img src="/assets/img/mesos_logo.png" alt="Apache Mesos logo"/></a>
	</div><!-- /.navbar-header -->

	<div class="navbar-collapse collapse" id="mesos-menu">
	<ul class="nav navbar-nav navbar-right">
	<li><a href="/gettingstarted/">Getting Started</a></li>
	<li><a href="/blog/">Blog</a></li>
	<li><a href="/documentation/latest/">Documentation</a></li>
	<li><a href="/downloads/">Downloads</a></li>
	<li><a href="/community/">Community</a></li>
	</ul>
	</div><!-- /#mesos-menu -->
	</div><!-- /.container -->
	</div><!-- /.navbar -->

	<div class="content">
	<div class="container">
	<div class="row-fluid">
	<div class="col-md-4">
	<h4>If you're new to Mesos</h4>
	<p>See the <a href="/gettingstarted/">getting started</a> page for more
	information about downloading, building, and deploying Mesos.</p>

	<h4>If you'd like to get involved or you're looking for support</h4>
	<p>See our <a href="/community/">community</a> page for more details.</p>
	</div>
	<div class="col-md-8">
	<h1>Mesos High-Availability Mode</h1>

	<p>If the Mesos master is unavailable, existing tasks can continue to execute, but new resources cannot be allocated and new tasks cannot be launched. To reduce the chance of this situation occurring, Mesos has a high-availability mode that uses multiple Mesos masters: one active master (called the <em>leader</em> or leading master) and several <em>backups</em> in case it fails. The masters elect the leader, with <a href="http://zookeeper.apache.org/">Apache ZooKeeper</a> both coordinating the election and handling leader detection by masters, agents, and scheduler drivers. More information regarding <a href="http://zookeeper.apache.org/doc/trunk/recipes.html#sc_leaderElection">how leader election works</a> is available on the Apache Zookeeper website.</p>

	<p>This document describes how to configure Mesos to run in high-availability mode. For more information on developing highly available frameworks, see a <a href="/documentation/latest/./high-availability-framework-guide/">companion document</a>.</p>

	<p><strong>Note</strong>: This document assumes you know how to start, run, and work with ZooKeeper, whose client library is included in the standard Mesos build.</p>

	<h2>Usage</h2>

	<p>To put Mesos into high-availability mode:</p>

	<ol>
	<li><p>Ensure that the ZooKeeper cluster is up and running.</p></li>
	<li><p>Provide the znode path to all masters, agents, and framework schedulers as follows:</p>

	<ul>
	<li><p>Start the mesos-master binaries using the <code>--zk</code> flag, e.g. <code>--zk=zk://host1:port1,host2:port2,.../path</code></p></li>
	<li><p>Start the mesos-agent binaries with <code>--master=zk://host1:port1,host2:port2,.../path</code></p></li>
	<li><p>Start any framework schedulers using the same <code>zk</code> path as in the last two steps. The SchedulerDriver must be constructed with this path, as shown in the <a href="/documentation/latest/./app-framework-development-guide/">Framework Development Guide</a>.</p></li>
	</ul>
	</li>
	</ol>


	<p>From now on, the Mesos masters and agents all communicate with ZooKeeper to find out which master is the current leading master. This is in addition to the usual communication between the leading master and the agents.</p>

	<p>In addition to ZooKeeper, one can get the location of the leading master by sending an HTTP request to <a href="/documentation/latest/./endpoints/master/redirect/">/redirect</a> endpoint on any master.</p>

	<p>For HTTP endpoints that only work at the leading master, requests made to endpoints at a non-leading master will result in either a <code>307 Temporary Redirect</code> (with the location of the leading master) or <code>503 Service Unavailable</code> (if the master does not know who the current leader is).</p>

	<p>Refer to the <a href="/documentation/latest/./app-framework-development-guide/">Scheduler API</a> for how to deal with leadership changes.</p>

	<h2>Component Disconnection Handling</h2>

	<p>When a network partition disconnects a component (master, agent, or scheduler driver) from ZooKeeper, the component’s Master Detector induces a timeout event. This notifies the component that it has no leading master. Depending on the component, the following happens. (Note that while a component is disconnected from ZooKeeper, a master may still be in communication with agents or schedulers and vice versa.)</p>

	<ul>
	<li><p>Agents disconnected from ZooKeeper no longer know which master is the leader. They ignore messages from masters to ensure they don’t act on a non-leader’s decisions. When an agent reconnects to ZooKeeper, ZooKeeper informs it of the current leader and the agent stops ignoring messages from the leader.</p></li>
	<li><p>Masters enter leaderless state irrespective of whether they are a leader or not before the disconnection.</p>

	<ul>
	<li><p>If the leader was disconnected from ZooKeeper, it aborts its process. The user/developer/administrator can then start a new master instance which will try to reconnect to ZooKeeper.</p>

	<ul>
	<li>Note that many production deployments of Mesos use a process supervisor (such as systemd or supervisord) that is configured to automatically restart the Mesos master if the process aborts unexpectedly.</li>
	</ul>
	</li>
	<li><p>Otherwise, the disconnected backup waits to reconnect with ZooKeeper and possibly get elected as the new leading master.</p></li>
	</ul>
	</li>
	<li><p>Scheduler drivers disconnected from the leading master notify the scheduler about their disconnection from the leader.</p></li>
	</ul>


	<p>When a network partition disconnects an agent from the leader:</p>

	<ul>
	<li><p>The agent fails health checks from the leader.</p></li>
	<li><p>The leader marks the agent as deactivated and sends its tasks to the LOST state. The <a href="/documentation/latest/./app-framework-development-guide/">Framework Development Guide</a> describes these various task states.</p></li>
	<li><p>Deactivated agents may not re-register with the leader and are told to shut down upon any post-deactivation communication.</p></li>
	</ul>


	<h2>Implementation Details</h2>

	<p>Mesos implements two levels of ZooKeeper leader election abstractions, one in <code>src/zookeeper</code> and the other in <code>src/master</code> (look for <code>contender\|detector.hpp\|cpp</code>).</p>

	<ul>
	<li><p>The lower level <code>LeaderContender</code> and <code>LeaderDetector</code> implement a generic ZooKeeper election algorithm loosely modeled after this
	<a href="http://zookeeper.apache.org/doc/trunk/recipes.html#sc_leaderElection">recipe</a> (sans herd effect handling due to the master group’s small size, which is often 3).</p></li>
	<li><p>The higher level <code>MasterContender</code> and <code>MasterDetector</code> wrap around ZooKeeper’s contender and detector abstractions as adapters to provide/interpret the ZooKeeper data.</p></li>
	<li><p>Each Mesos master simultaneously uses both a contender and a detector to try to elect themselves and detect who the current leader is. A separate detector is necessary because each master’s WebUI redirects browser traffic to the current leader when that master is not elected. Other Mesos components (i.e., agents and scheduler drivers) use the detector to find the current leader and connect to it.</p></li>
	</ul>


	<p>The notion of the group of leader candidates is implemented in <code>Group</code>. This abstraction handles reliable (through queues and retries of retryable errors under the covers) ZooKeeper group membership registration, cancellation, and monitoring. It watches for several ZooKeeper session events:</p>

	<ul>
	<li>Connection</li>
	<li>Reconnection</li>
	<li>Session Expiration</li>
	<li>ZNode creation, deletion, updates</li>
	</ul>


	<p>We also explicitly timeout our sessions when disconnected from ZooKeeper for a specified amount of time. See <code>MASTER_CONTENDER_ZK_SESSION_TIMEOUT</code> and <code>MASTER_DETECTOR_ZK_SESSION_TIMEOUT</code>. This is because the ZooKeeper client libraries only notify of session expiration upon reconnection. These timeouts are of particular interest for network partitions.</p>

	</div>
	</div>

	</div><!-- /.container -->
	</div><!-- /.content -->

	<hr>



	<!-- footer -->
	<div class="footer">
	<div class="container">
	<div class="col-md-4 social-blk">
	<span class="social">
	<a href="https://twitter.com/ApacheMesos"
	class="twitter-follow-button"
	data-show-count="false" data-size="large">Follow @ApacheMesos</a>
	<script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script>
	<a href="https://twitter.com/intent/tweet?button_hashtag=mesos"
	class="twitter-hashtag-button"
	data-size="large"
	data-related="ApacheMesos">Tweet #mesos</a>
	<script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script>
	</span>
	</div>

	<div class="col-md-8 trademark">
	<p>© 2012-2017 <a href="http://apache.org">The Apache Software Foundation</a>.
	Apache Mesos, the Apache feather logo, and the Apache Mesos project logo are trademarks of The Apache Software Foundation.
	<p>
	</div>
	</div><!-- /.container -->
	</div><!-- /.footer -->

	<!-- JS -->
	<script src="//code.jquery.com/jquery-1.11.0.min.js" type="text/javascript"></script>
	<script src="//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js" type="text/javascript"></script>
	</body>
	</html>