content/documentation/0.21.0/features/resource-isolation/index.html - aurora-website - Git at Google

 <!DOCTYPE html>
 <html lang="en">
   <head>
     <meta charset="utf-8">
     <meta name="viewport" content="width=device-width, initial-scale=1">
 	<title>Apache Aurora</title>
     <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css">
     <link href="/assets/css/main.css" rel="stylesheet">
 	<!-- Analytics -->
 	<script type="text/javascript">
 		  var _gaq = _gaq || [];
 		  _gaq.push(['_setAccount', 'UA-45879646-1']);
 		  _gaq.push(['_setDomainName', 'apache.org']);
 		  _gaq.push(['_trackPageview']);

 		  (function() {
 		    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
 		    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
 		    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
 		  })();
 	</script>
   </head>
   <body>
     <div class="container-fluid section-header">
   <div class="container">
     <div class="nav nav-bar">
     <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a>
     <ul class="nav navbar-nav navbar-right">
       <li><a href="/documentation/latest/">Documentation</a></li>
       <li><a href="/community/">Community</a></li>
       <li><a href="/downloads/">Downloads</a></li>
       <li><a href="/blog/">Blog</a></li>
     </ul>
     </div>
   </div>
 </div>

     <div class="container-fluid">
       <div class="container content">
         <div class="col-md-12 documentation">
 <h5 class="page-header text-uppercase">Documentation
 <select onChange="window.location.href='/documentation/' + this.value + '/features/resource-isolation/'"
         value="0.21.0">
   <option value="0.22.0"
     >
     0.22.0
       (latest)
   </option>
   <option value="0.21.0"
     selected="selected">
     0.21.0
   </option>
   <option value="0.20.0"
     >
     0.20.0
   </option>
   <option value="0.19.1"
     >
     0.19.1
   </option>
   <option value="0.19.0"
     >
     0.19.0
   </option>
   <option value="0.18.1"
     >
     0.18.1
   </option>
   <option value="0.18.0"
     >
     0.18.0
   </option>
   <option value="0.17.0"
     >
     0.17.0
   </option>
   <option value="0.16.0"
     >
     0.16.0
   </option>
   <option value="0.15.0"
     >
     0.15.0
   </option>
   <option value="0.14.0"
     >
     0.14.0
   </option>
   <option value="0.13.0"
     >
     0.13.0
   </option>
   <option value="0.12.0"
     >
     0.12.0
   </option>
   <option value="0.11.0"
     >
     0.11.0
   </option>
   <option value="0.10.0"
     >
     0.10.0
   </option>
   <option value="0.9.0"
     >
     0.9.0
   </option>
   <option value="0.8.0"
     >
     0.8.0
   </option>
   <option value="0.7.0-incubating"
     >
     0.7.0-incubating
   </option>
   <option value="0.6.0-incubating"
     >
     0.6.0-incubating
   </option>
   <option value="0.5.0-incubating"
     >
     0.5.0-incubating
   </option>
 </select>
 </h5>
 <h1 id="resources-isolation-and-sizing">Resources Isolation and Sizing</h1>

 <p>This document assumes Aurora and Mesos have been configured
 using our <a href="../../operations/configuration/#resource-isolation">recommended resource isolation settings</a>.</p>

 <ul>
 <li><a href="#isolation">Isolation</a></li>
 <li><a href="#sizing">Sizing</a></li>
 <li><a href="#oversubscription">Oversubscription</a></li>
 </ul>

 <h2 id="isolation">Isolation</h2>

 <p>Aurora is a multi-tenant system; a single software instance runs on a
 server, serving multiple clients/tenants. To share resources among
 tenants, it leverages Mesos for isolation of:</p>

 <ul>
 <li>CPU</li>
 <li>GPU</li>
 <li>memory</li>
 <li>disk space</li>
 <li>ports</li>
 </ul>

 <p>CPU is a soft limit, and handled differently from memory and disk space.
 Too low a CPU value results in throttling your application and
 slowing it down. Memory and disk space are both hard limits; when your
 application goes over these values, it&rsquo;s killed.</p>

 <h3 id="cpu-isolation">CPU Isolation</h3>

 <p>Mesos can be configured to use a quota based CPU scheduler (the <em>Completely</em>
 <em>Fair Scheduler</em>) to provide consistent and predictable performance.
 This is effectively a guarantee of resources &ndash; you receive at least what
 you requested, but also no more than you&rsquo;ve requested.</p>

 <p>The scheduler gives applications a CPU quota for every 100 ms interval.
 When an application uses its quota for an interval, it is throttled for
 the rest of the 100 ms. Usage resets for each interval and unused
 quota does not carry over.</p>

 <p>For example, an application specifying 4.0 CPU has access to 400 ms of
 CPU time every 100 ms. This CPU quota can be used in different ways,
 depending on the application and available resources. Consider the
 scenarios shown in this diagram.</p>

 <p><img alt="CPU Availability" src="../../images/CPUavailability.png" /></p>

 <ul>
 <li><p><em>Scenario A</em>: the application can use up to 4 cores continuously for
 every 100 ms interval. It is never throttled and starts processing
 new requests immediately.</p></li>
 <li><p><em>Scenario B</em> : the application uses up to 8 cores (depending on
 availability) but is throttled after 50 ms. The CPU quota resets at the
 start of each new 100 ms interval.</p></li>
 <li><p><em>Scenario C</em> : is like Scenario A, but there is a garbage collection
 event in the second interval that consumes all CPU quota. The
 application throttles for the remaining 75 ms of that interval and
 cannot service requests until the next interval. In this example, the
 garbage collection finished in one interval but, depending on how much
 garbage needs collecting, it may take more than one interval and further
 delay service of requests.</p></li>
 </ul>

 <p><em>Technical Note</em>: Mesos considers logical cores, also known as
 hyperthreading or SMT cores, as the unit of CPU.</p>

 <h3 id="memory-isolation">Memory Isolation</h3>

 <p>Mesos uses dedicated memory allocation. Your application always has
 access to the amount of memory specified in your configuration. The
 application&rsquo;s memory use is defined as the sum of the resident set size
 (RSS) of all processes in a shard. Each shard is considered
 independently.</p>

 <p>In other words, say you specified a memory size of 10GB. Each shard
 would receive 10GB of memory. If an individual shard&rsquo;s memory demands
 exceed 10GB, that shard is killed, but the other shards continue
 working.</p>

 <p><em>Technical note</em>: Total memory size is not enforced at allocation time,
 so your application can request more than its allocation without getting
 an ENOMEM. However, it will be killed shortly after.</p>

 <h3 id="disk-space">Disk Space</h3>

 <p>Disk space used by your application is defined as the sum of the files&rsquo;
 disk space in your application&rsquo;s directory, including the <code>stdout</code> and
 <code>stderr</code> logged from your application. Each shard is considered
 independently. You should use off-node storage for your application&rsquo;s
 data whenever possible.</p>

 <p>In other words, say you specified disk space size of 100MB. Each shard
 would receive 100MB of disk space. If an individual shard&rsquo;s disk space
 demands exceed 100MB, that shard is killed, but the other shards
 continue working.</p>

 <p>After your application finishes running, its allocated disk space is
 reclaimed. Thus, your job&rsquo;s final action should move any disk content
 that you want to keep, such as logs, to your home file system or other
 less transitory storage. Disk reclamation takes place an undefined
 period after the application finish time; until then, the disk contents
 are still available but you shouldn&rsquo;t count on them being so.</p>

 <p><em>Technical note</em> : Disk space is not enforced at write so your
 application can write above its quota without getting an ENOSPC, but it
 will be killed shortly after. This is subject to change.</p>

 <h3 id="gpu-isolation">GPU Isolation</h3>

 <p>GPU isolation will be supported for Nvidia devices starting from Mesos 1.0.
 Access to the allocated units will be exclusive with no sharing between tasks
 allowed (e.g. no fractional GPU allocation). For more details, see the
 <a href="https://docs.google.com/document/d/10GJ1A80x4nIEo8kfdeo9B11PIbS1xJrrB4Z373Ifkpo/edit#heading=h.w84lz7p4eexl">Mesos design document</a>
 and the <a href="http://mesos.apache.org/documentation/latest/configuration/">Mesos agent configuration</a>.</p>

 <h3 id="other-resources">Other Resources</h3>

 <p>Other resources, such as network bandwidth, do not have any performance
 guarantees. For some resources, such as memory bandwidth, there are no
 practical sharing methods so some application combinations collocated on
 the same host may cause contention.</p>

 <h2 id="sizing">Sizing</h2>

 <h3 id="cpu-sizing">CPU Sizing</h3>

 <p>To correctly size Aurora-run Mesos tasks, specify a per-shard CPU value
 that lets the task run at its desired performance when at peak load
 distributed across all shards. Include reserve capacity of at least 50%,
 possibly more, depending on how critical your service is (or how
 confident you are about your original estimate : -)), ideally by
 increasing the number of shards to also improve resiliency. When running
 your application, observe its CPU stats over time. If consistently at or
 near your quota during peak load, you should consider increasing either
 per-shard CPU or the number of shards.</p>

 <h2 id="memory-sizing">Memory Sizing</h2>

 <p>Size for your application&rsquo;s peak requirement. Observe the per-instance
 memory statistics over time, as memory requirements can vary over
 different periods. Remember that if your application exceeds its memory
 value, it will be killed, so you should also add a safety margin of
 around 10-20%. If you have the ability to do so, you may also want to
 put alerts on the per-instance memory.</p>

 <h2 id="disk-space-sizing">Disk Space Sizing</h2>

 <p>Size for your application&rsquo;s peak requirement. Rotate and discard log
 files as needed to stay within your quota. When running a Java process,
 add the maximum size of the Java heap to your disk space requirement, in
 order to account for an out of memory error dumping the heap
 into the application&rsquo;s sandbox space.</p>

 <h2 id="gpu-sizing">GPU Sizing</h2>

 <p>GPU is highly dependent on your application requirements and is only limited
 by the number of physical GPU units available on a target box.</p>

 <h2 id="oversubscription">Oversubscription</h2>

 <p>Mesos supports <a href="http://mesos.apache.org/documentation/latest/oversubscription/">oversubscription of machine resources</a>
 via the concept of revocable tasks. In contrast to non-revocable tasks, revocable tasks are best-effort.
 Mesos reserves the right to throttle or even kill them if they might affect existing high-priority
 user-facing services.</p>

 <p>As of today, the only revocable resource supported by Aurora are CPU and RAM resources. A job can
 opt-in to use those by specifying the <code>revocable</code> <a href="../../features/multitenancy/#configuration-tiers">Configuration Tier</a>.
 A revocable job will only be scheduled using revocable resources, even if there are plenty of
 non-revocable resources available.</p>

 <p>The Aurora scheduler must be <a href="../../operations/configuration/#resource-isolation">configured to receive revocable offers</a>
 from Mesos and accept revocable jobs. If not configured properly revocable tasks will never get
 assigned to hosts and will stay in <code>PENDING</code>.</p>

 <p>For details on how to mark a job as being revocable, see the
 <a href="../../reference/configuration/">Configuration Reference</a>.</p>

 </div>

       </div>
     </div>
   	<div class="container-fluid section-footer buffer">
       <div class="container">
         <div class="row">
 		  <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3>
 		  <ul>
 		    <li><a href="/downloads/">Downloads</a></li>
             <li><a href="/community/">Mailing Lists</a></li>
 			<li><a href="http://issues.apache.org/jira/browse/AURORA">Issue Tracking</a></li>
 			<li><a href="/documentation/latest/contributing/">How To Contribute</a></li>
 		  </ul>
 	      </div>
 		  <div class="col-md-2"><h3>The ASF</h3>
           <ul>
             <li><a href="http://www.apache.org/licenses/">License</a></li>
             <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
             <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
             <li><a href="http://www.apache.org/security/">Security</a></li>
           </ul>
 		  </div>
 		  <div class="col-md-6">
 			<p class="disclaimer">&copy; 2014-2017 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/">Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/">Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p>
         </div>
       </div>
     </div>

   </body>
 </html>
	<!DOCTYPE html>
	<html lang="en">
	<head>
	<meta charset="utf-8">
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<title>Apache Aurora</title>
	<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css">
	<link href="/assets/css/main.css" rel="stylesheet">
	<!-- Analytics -->
	<script type="text/javascript">
	var _gaq = _gaq \|\| [];
	_gaq.push(['_setAccount', 'UA-45879646-1']);
	_gaq.push(['_setDomainName', 'apache.org']);
	_gaq.push(['_trackPageview']);

	(function() {
	var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
	ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
	var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
	})();
	</script>
	</head>
	<body>
	<div class="container-fluid section-header">
	<div class="container">
	<div class="nav nav-bar">
	<a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a>
	<ul class="nav navbar-nav navbar-right">
	<li><a href="/documentation/latest/">Documentation</a></li>
	<li><a href="/community/">Community</a></li>
	<li><a href="/downloads/">Downloads</a></li>
	<li><a href="/blog/">Blog</a></li>
	</ul>
	</div>
	</div>
	</div>

	<div class="container-fluid">
	<div class="container content">
	<div class="col-md-12 documentation">
	<h5 class="page-header text-uppercase">Documentation
	<select onChange="window.location.href='/documentation/' + this.value + '/features/resource-isolation/'"
	value="0.21.0">
	<option value="0.22.0"
	>
	0.22.0
	(latest)
	</option>
	<option value="0.21.0"
	selected="selected">
	0.21.0
	</option>
	<option value="0.20.0"
	>
	0.20.0
	</option>
	<option value="0.19.1"
	>
	0.19.1
	</option>
	<option value="0.19.0"
	>
	0.19.0
	</option>
	<option value="0.18.1"
	>
	0.18.1
	</option>
	<option value="0.18.0"
	>
	0.18.0
	</option>
	<option value="0.17.0"
	>
	0.17.0
	</option>
	<option value="0.16.0"
	>
	0.16.0
	</option>
	<option value="0.15.0"
	>
	0.15.0
	</option>
	<option value="0.14.0"
	>
	0.14.0
	</option>
	<option value="0.13.0"
	>
	0.13.0
	</option>
	<option value="0.12.0"
	>
	0.12.0
	</option>
	<option value="0.11.0"
	>
	0.11.0
	</option>
	<option value="0.10.0"
	>
	0.10.0
	</option>
	<option value="0.9.0"
	>
	0.9.0
	</option>
	<option value="0.8.0"
	>
	0.8.0
	</option>
	<option value="0.7.0-incubating"
	>
	0.7.0-incubating
	</option>
	<option value="0.6.0-incubating"
	>
	0.6.0-incubating
	</option>
	<option value="0.5.0-incubating"
	>
	0.5.0-incubating
	</option>
	</select>
	</h5>
	<h1 id="resources-isolation-and-sizing">Resources Isolation and Sizing</h1>

	<p>This document assumes Aurora and Mesos have been configured
	using our <a href="../../operations/configuration/#resource-isolation">recommended resource isolation settings</a>.</p>

	<ul>
	<li><a href="#isolation">Isolation</a></li>
	<li><a href="#sizing">Sizing</a></li>
	<li><a href="#oversubscription">Oversubscription</a></li>
	</ul>

	<h2 id="isolation">Isolation</h2>

	<p>Aurora is a multi-tenant system; a single software instance runs on a
	server, serving multiple clients/tenants. To share resources among
	tenants, it leverages Mesos for isolation of:</p>

	<ul>
	<li>CPU</li>
	<li>GPU</li>
	<li>memory</li>
	<li>disk space</li>
	<li>ports</li>
	</ul>

	<p>CPU is a soft limit, and handled differently from memory and disk space.
	Too low a CPU value results in throttling your application and
	slowing it down. Memory and disk space are both hard limits; when your
	application goes over these values, it’s killed.</p>

	<h3 id="cpu-isolation">CPU Isolation</h3>

	<p>Mesos can be configured to use a quota based CPU scheduler (the <em>Completely</em>
	<em>Fair Scheduler</em>) to provide consistent and predictable performance.
	This is effectively a guarantee of resources – you receive at least what
	you requested, but also no more than you’ve requested.</p>

	<p>The scheduler gives applications a CPU quota for every 100 ms interval.
	When an application uses its quota for an interval, it is throttled for
	the rest of the 100 ms. Usage resets for each interval and unused
	quota does not carry over.</p>

	<p>For example, an application specifying 4.0 CPU has access to 400 ms of
	CPU time every 100 ms. This CPU quota can be used in different ways,
	depending on the application and available resources. Consider the
	scenarios shown in this diagram.</p>

	<p><img alt="CPU Availability" src="../../images/CPUavailability.png" /></p>

	<ul>
	<li><p><em>Scenario A</em>: the application can use up to 4 cores continuously for
	every 100 ms interval. It is never throttled and starts processing
	new requests immediately.</p></li>
	<li><p><em>Scenario B</em> : the application uses up to 8 cores (depending on
	availability) but is throttled after 50 ms. The CPU quota resets at the
	start of each new 100 ms interval.</p></li>
	<li><p><em>Scenario C</em> : is like Scenario A, but there is a garbage collection
	event in the second interval that consumes all CPU quota. The
	application throttles for the remaining 75 ms of that interval and
	cannot service requests until the next interval. In this example, the
	garbage collection finished in one interval but, depending on how much
	garbage needs collecting, it may take more than one interval and further
	delay service of requests.</p></li>
	</ul>

	<p><em>Technical Note</em>: Mesos considers logical cores, also known as
	hyperthreading or SMT cores, as the unit of CPU.</p>

	<h3 id="memory-isolation">Memory Isolation</h3>

	<p>Mesos uses dedicated memory allocation. Your application always has
	access to the amount of memory specified in your configuration. The
	application’s memory use is defined as the sum of the resident set size
	(RSS) of all processes in a shard. Each shard is considered
	independently.</p>

	<p>In other words, say you specified a memory size of 10GB. Each shard
	would receive 10GB of memory. If an individual shard’s memory demands
	exceed 10GB, that shard is killed, but the other shards continue
	working.</p>

	<p><em>Technical note</em>: Total memory size is not enforced at allocation time,
	so your application can request more than its allocation without getting
	an ENOMEM. However, it will be killed shortly after.</p>

	<h3 id="disk-space">Disk Space</h3>

	<p>Disk space used by your application is defined as the sum of the files’
	disk space in your application’s directory, including the <code>stdout</code> and
	<code>stderr</code> logged from your application. Each shard is considered
	independently. You should use off-node storage for your application’s
	data whenever possible.</p>

	<p>In other words, say you specified disk space size of 100MB. Each shard
	would receive 100MB of disk space. If an individual shard’s disk space
	demands exceed 100MB, that shard is killed, but the other shards
	continue working.</p>

	<p>After your application finishes running, its allocated disk space is
	reclaimed. Thus, your job’s final action should move any disk content
	that you want to keep, such as logs, to your home file system or other
	less transitory storage. Disk reclamation takes place an undefined
	period after the application finish time; until then, the disk contents
	are still available but you shouldn’t count on them being so.</p>

	<p><em>Technical note</em> : Disk space is not enforced at write so your
	application can write above its quota without getting an ENOSPC, but it
	will be killed shortly after. This is subject to change.</p>

	<h3 id="gpu-isolation">GPU Isolation</h3>

	<p>GPU isolation will be supported for Nvidia devices starting from Mesos 1.0.
	Access to the allocated units will be exclusive with no sharing between tasks
	allowed (e.g. no fractional GPU allocation). For more details, see the
	<a href="https://docs.google.com/document/d/10GJ1A80x4nIEo8kfdeo9B11PIbS1xJrrB4Z373Ifkpo/edit#heading=h.w84lz7p4eexl">Mesos design document</a>
	and the <a href="http://mesos.apache.org/documentation/latest/configuration/">Mesos agent configuration</a>.</p>

	<h3 id="other-resources">Other Resources</h3>

	<p>Other resources, such as network bandwidth, do not have any performance
	guarantees. For some resources, such as memory bandwidth, there are no
	practical sharing methods so some application combinations collocated on
	the same host may cause contention.</p>

	<h2 id="sizing">Sizing</h2>

	<h3 id="cpu-sizing">CPU Sizing</h3>

	<p>To correctly size Aurora-run Mesos tasks, specify a per-shard CPU value
	that lets the task run at its desired performance when at peak load
	distributed across all shards. Include reserve capacity of at least 50%,
	possibly more, depending on how critical your service is (or how
	confident you are about your original estimate : -)), ideally by
	increasing the number of shards to also improve resiliency. When running
	your application, observe its CPU stats over time. If consistently at or
	near your quota during peak load, you should consider increasing either
	per-shard CPU or the number of shards.</p>

	<h2 id="memory-sizing">Memory Sizing</h2>

	<p>Size for your application’s peak requirement. Observe the per-instance
	memory statistics over time, as memory requirements can vary over
	different periods. Remember that if your application exceeds its memory
	value, it will be killed, so you should also add a safety margin of
	around 10-20%. If you have the ability to do so, you may also want to
	put alerts on the per-instance memory.</p>

	<h2 id="disk-space-sizing">Disk Space Sizing</h2>

	<p>Size for your application’s peak requirement. Rotate and discard log
	files as needed to stay within your quota. When running a Java process,
	add the maximum size of the Java heap to your disk space requirement, in
	order to account for an out of memory error dumping the heap
	into the application’s sandbox space.</p>

	<h2 id="gpu-sizing">GPU Sizing</h2>

	<p>GPU is highly dependent on your application requirements and is only limited
	by the number of physical GPU units available on a target box.</p>

	<h2 id="oversubscription">Oversubscription</h2>

	<p>Mesos supports <a href="http://mesos.apache.org/documentation/latest/oversubscription/">oversubscription of machine resources</a>
	via the concept of revocable tasks. In contrast to non-revocable tasks, revocable tasks are best-effort.
	Mesos reserves the right to throttle or even kill them if they might affect existing high-priority
	user-facing services.</p>

	<p>As of today, the only revocable resource supported by Aurora are CPU and RAM resources. A job can
	opt-in to use those by specifying the <code>revocable</code> <a href="../../features/multitenancy/#configuration-tiers">Configuration Tier</a>.
	A revocable job will only be scheduled using revocable resources, even if there are plenty of
	non-revocable resources available.</p>

	<p>The Aurora scheduler must be <a href="../../operations/configuration/#resource-isolation">configured to receive revocable offers</a>
	from Mesos and accept revocable jobs. If not configured properly revocable tasks will never get
	assigned to hosts and will stay in <code>PENDING</code>.</p>

	<p>For details on how to mark a job as being revocable, see the
	<a href="../../reference/configuration/">Configuration Reference</a>.</p>

	</div>

	</div>
	</div>
	<div class="container-fluid section-footer buffer">
	<div class="container">
	<div class="row">
	<div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3>
	<ul>
	<li><a href="/downloads/">Downloads</a></li>
	<li><a href="/community/">Mailing Lists</a></li>
	<li><a href="http://issues.apache.org/jira/browse/AURORA">Issue Tracking</a></li>
	<li><a href="/documentation/latest/contributing/">How To Contribute</a></li>
	</ul>
	</div>
	<div class="col-md-2"><h3>The ASF</h3>
	<ul>
	<li><a href="http://www.apache.org/licenses/">License</a></li>
	<li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
	<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
	<li><a href="http://www.apache.org/security/">Security</a></li>
	</ul>
	</div>
	<div class="col-md-6">
	<p class="disclaimer">© 2014-2017 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/">Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/">Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p>
	</div>
	</div>
	</div>

	</body>
	</html>