Google Git
Sign in
apache / aurora-website / refs/heads/asf-site / . / content / documentation / latest / reference / configuration-tutorial / index.html
blob: 5b0b0d7d43c1772637b6f5b8005a5d59d885cd8a [file] [log] [blame]
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Apache Aurora</title>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css">
<link href="/assets/css/main.css" rel="stylesheet">
<!-- Analytics -->
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-45879646-1']);
_gaq.push(['_setDomainName', 'apache.org']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
</head>
<body>
<div class="container-fluid section-header">
<div class="container">
<div class="nav nav-bar">
<a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a>
<ul class="nav navbar-nav navbar-right">
<li><a href="/documentation/latest/">Documentation</a></li>
<li><a href="/community/">Community</a></li>
<li><a href="/downloads/">Downloads</a></li>
<li><a href="/blog/">Blog</a></li>
</ul>
</div>
</div>
</div>
<div class="container-fluid">
<div class="container content">
<div class="col-md-12 documentation">
<h5 class="page-header text-uppercase">Documentation
<select onChange="window.location.href='/documentation/' + this.value + '/reference/configuration-tutorial/'"
value="latest">
<option value="0.22.0"
>
0.22.0
(latest)
</option>
<option value="0.21.0"
>
0.21.0
</option>
<option value="0.20.0"
>
0.20.0
</option>
<option value="0.19.1"
>
0.19.1
</option>
<option value="0.19.0"
>
0.19.0
</option>
<option value="0.18.1"
>
0.18.1
</option>
<option value="0.18.0"
>
0.18.0
</option>
<option value="0.17.0"
>
0.17.0
</option>
<option value="0.16.0"
>
0.16.0
</option>
<option value="0.15.0"
>
0.15.0
</option>
<option value="0.14.0"
>
0.14.0
</option>
<option value="0.13.0"
>
0.13.0
</option>
<option value="0.12.0"
>
0.12.0
</option>
<option value="0.11.0"
>
0.11.0
</option>
<option value="0.10.0"
>
0.10.0
</option>
<option value="0.9.0"
>
0.9.0
</option>
<option value="0.8.0"
>
0.8.0
</option>
<option value="0.7.0-incubating"
>
0.7.0-incubating
</option>
<option value="0.6.0-incubating"
>
0.6.0-incubating
</option>
<option value="0.5.0-incubating"
>
0.5.0-incubating
</option>
</select>
</h5>
<h1 id="aurora-configuration-tutorial">Aurora Configuration Tutorial</h1>
<p>How to write Aurora configuration files, including feature descriptions
and best practices. When writing a configuration file, make use of
<code>aurora job inspect</code>. It takes the same job key and configuration file
arguments as <code>aurora job create</code> or <code>aurora update start</code>. It first ensures the
configuration parses, then outputs it in human-readable form.</p>
<p>You should read this after going through the general <a href="../../getting-started/tutorial/">Aurora Tutorial</a>.</p>
<ul>
<li><a href="#the-basics">The Basics</a>
<ul>
<li><a href="#use-bottom-to-top-object-ordering">Use Bottom-To-Top Object Ordering</a></li>
</ul></li>
<li><a href="#an-example-configuration-file">An Example Configuration File</a></li>
<li><a href="#defining-process-objects">Defining Process Objects</a></li>
<li><a href="#getting-your-code-into-the-sandbox">Getting Your Code Into The Sandbox</a></li>
<li><a href="#defining-task-objects">Defining Task Objects</a>
<ul>
<li><a href="#sequentialtask-running-processes-in-parallel-or-sequentially">SequentialTask: Running Processes in Parallel or Sequentially</a></li>
<li><a href="#simpletask">SimpleTask</a></li>
<li><a href="#combining-tasks">Combining tasks</a></li>
</ul></li>
<li><a href="#defining-job-objects">Defining Job Objects</a></li>
<li><a href="#the-jobs-list">The jobs List</a></li>
<li><a href="#basic-examples">Basic Examples</a></li>
</ul>
<h2 id="the-basics">The Basics</h2>
<p>To run a job on Aurora, you must specify a configuration file that tells
Aurora what it needs to know to schedule the job, what Mesos needs to
run the tasks the job is made up of, and what Thermos needs to run the
processes that make up the tasks. This file must have
a<code>.aurora</code> suffix.</p>
<p>A configuration file defines a collection of objects, along with parameter
values for their attributes. An Aurora configuration file contains the
following three types of objects:</p>
<ul>
<li>Job</li>
<li>Task</li>
<li>Process</li>
</ul>
<p>A configuration also specifies a list of <code>Job</code> objects assigned
to the variable <code>jobs</code>.</p>
<ul>
<li>jobs (list of defined Jobs to run)</li>
</ul>
<p>The <code>.aurora</code> file format is just Python. However, <code>Job</code>, <code>Task</code>,
<code>Process</code>, and other classes are defined by a type-checked dictionary
templating library called <em>Pystachio</em>, a powerful tool for
configuration specification and reuse. Pystachio objects are tailored
via {{}} surrounded templates.</p>
<p>When writing your <code>.aurora</code> file, you may use any Pystachio datatypes, as
well as any objects shown in the <a href="../configuration/"><em>Aurora Configuration
Reference</em></a>, without <code>import</code> statements - the
Aurora config loader injects them automatically. Other than that, an <code>.aurora</code>
file works like any other Python script.</p>
<p><a href="../configuration/"><em>Aurora Configuration Reference</em></a>
has a full reference of all Aurora/Thermos defined Pystachio objects.</p>
<h3 id="use-bottom-to-top-object-ordering">Use Bottom-To-Top Object Ordering</h3>
<p>A well-structured configuration starts with structural templates (if
any). Structural templates encapsulate in their attributes all the
differences between Jobs in the configuration that are not directly
manipulated at the <code>Job</code> level, but typically at the <code>Process</code> or <code>Task</code>
level. For example, if certain processes are invoked with slightly
different settings or input.</p>
<p>After structural templates, define, in order, <code>Process</code>es, <code>Task</code>s, and
<code>Job</code>s.</p>
<p>Structural template names should be <em>UpperCamelCased</em> and their
instantiations are typically <em>UPPER_SNAKE_CASED</em>. <code>Process</code>, <code>Task</code>,
and <code>Job</code> names are typically <em>lower_snake_cased</em>. Indentation is typically 2
spaces.</p>
<h2 id="an-example-configuration-file">An Example Configuration File</h2>
<p>The following is a typical configuration file. Don&rsquo;t worry if there are
parts you don&rsquo;t understand yet, but you may want to refer back to this
as you read about its individual parts. Note that names surrounded by
curly braces {{}} are template variables, which the system replaces with
bound values for the variables.</p>
<pre class="highlight plaintext"><code># --- templates here ---
class Profile(Struct):
package_version = Default(String, 'live')
java_binary = Default(String, '/usr/lib/jvm/java-1.7.0-openjdk/bin/java')
extra_jvm_options = Default(String, '')
parent_environment = Default(String, 'prod')
parent_serverset = Default(String,
'/foocorp/service/bird/{{parent_environment}}/bird')
# --- processes here ---
main = Process(
name = 'application',
cmdline = '{{profile.java_binary}} -server -Xmx1792m '
'{{profile.extra_jvm_options}} '
'-jar application.jar '
'-upstreamService {{profile.parent_serverset}}'
)
# --- tasks ---
base_task = SequentialTask(
name = 'application',
processes = [
Process(
name = 'fetch',
cmdline = 'curl -O
https://packages.foocorp.com/{{profile.package_version}}/application.jar'),
]
)
# not always necessary but often useful to have separate task
# resource classes
staging_task = base_task(resources =
Resources(cpu = 1.0,
ram = 2048*MB,
disk = 1*GB))
production_task = base_task(resources =
Resources(cpu = 4.0,
ram = 2560*MB,
disk = 10*GB))
# --- job template ---
job_template = Job(
name = 'application',
role = 'myteam',
contact = 'myteam-team@foocorp.com',
instances = 20,
service = True,
task = production_task
)
# -- profile instantiations (if any) ---
PRODUCTION = Profile()
STAGING = Profile(
extra_jvm_options = '-Xloggc:gc.log',
parent_environment = 'staging'
)
# -- job instantiations --
jobs = [
job_template(cluster = 'cluster1', environment = 'prod')
.bind(profile = PRODUCTION),
job_template(cluster = 'cluster2', environment = 'prod')
.bind(profile = PRODUCTION),
job_template(cluster = 'cluster1',
environment = 'staging',
service = False,
task = staging_task,
instances = 2)
.bind(profile = STAGING),
]
</code></pre>
<h2 id="defining-process-objects">Defining Process Objects</h2>
<p>Processes are handled by the Thermos system. A process is a single
executable step run as a part of an Aurora task, which consists of a
bash-executable statement.</p>
<p>The key (and required) <code>Process</code> attributes are:</p>
<ul>
<li> <code>name</code>: Any string which is a valid Unix filename (no slashes,
NULLs, or leading periods). The <code>name</code> value must be unique relative
to other Processes in a <code>Task</code>.</li>
<li> <code>cmdline</code>: A command line run in a bash subshell, so you can use
bash scripts. Nothing is supplied for command-line arguments,
so <code>$*</code> is unspecified.</li>
</ul>
<p>Many tiny processes make managing configurations more difficult. For
example, the following is a bad way to define processes.</p>
<pre class="highlight plaintext"><code>copy = Process(
name = 'copy',
cmdline = 'curl -O https://packages.foocorp.com/app.zip'
)
unpack = Process(
name = 'unpack',
cmdline = 'unzip app.zip'
)
remove = Process(
name = 'remove',
cmdline = 'rm -f app.zip'
)
run = Process(
name = 'app',
cmdline = 'java -jar app.jar'
)
run_task = Task(
processes = [copy, unpack, remove, run],
constraints = order(copy, unpack, remove, run)
)
</code></pre>
<p>Since <code>cmdline</code> runs in a bash subshell, you can chain commands
with <code>&amp;&amp;</code> or <code>||</code>.</p>
<p>When defining a <code>Task</code> that is just a list of Processes run in a
particular order, use <code>SequentialTask</code>, as described in the <a href="#Task"><em>Defining</em>
<code>Task</code> <em>Objects</em></a> section. The following simplifies and combines the
above multiple <code>Process</code> definitions into just two.</p>
<pre class="highlight plaintext"><code>stage = Process(
name = 'stage',
cmdline = 'curl -O https://packages.foocorp.com/app.zip &amp;&amp; '
'unzip app.zip &amp;&amp; rm -f app.zip')
run = Process(name = 'app', cmdline = 'java -jar app.jar')
run_task = SequentialTask(processes = [stage, run])
</code></pre>
<p><code>Process</code> also has optional attributes to customize its behaviour. Details can be found in the <a href="../configuration/#process-objects">Aurora Configuration Reference</a>.</p>
<h2 id="getting-your-code-into-the-sandbox">Getting Your Code Into The Sandbox</h2>
<p>When using Aurora, you need to get your executable code into its &ldquo;sandbox&rdquo;, specifically
the Task sandbox where the code executes for the Processes that make up that Task.</p>
<p>Each Task has a sandbox created when the Task starts and garbage
collected when it finishes. All of a Task&rsquo;s processes run in its
sandbox, so processes can share state by using a shared current
working directory.</p>
<p>Typically, you save this code somewhere. You then need to define a Process
in your <code>.aurora</code> configuration file that fetches the code from that somewhere
to where the agent can see it. For a public cloud, that can be anywhere public on
the Internet, such as S3. For a private cloud internal storage, you need to put in
on an accessible HDFS cluster or similar storage.</p>
<p>The template for this Process is:</p>
<pre class="highlight plaintext"><code>&lt;name&gt; = Process(
name = '&lt;name&gt;'
cmdline = '&lt;command to copy and extract code archive into current working directory&gt;'
)
</code></pre>
<p>Note: Be sure the extracted code archive has an executable.</p>
<h2 id="getting-environment-variables-into-the-sandbox">Getting Environment Variables Into The Sandbox</h2>
<p>Every time a process is forked the Thermos executor checks for the existence of the
<code>.thermos_profile</code> file, if the <code>.thermos_profile</code> file exists it will be sourced.
You can utilize this process to pass environment variables to the sandbox.</p>
<p>An example for this Process is:</p>
<pre class="highlight plaintext"><code>setup_env = Process(
name = 'setup',
cmdline = (
'cat &lt;&lt;EOF &gt; .thermos_profile\n'
'export RESULT=hello\n'
'EOF\n'
)
)
read_env = Process(
name = 'read'
cmdline = 'echo $RESULT'
)
</code></pre>
<h2 id="defining-task-objects">Defining Task Objects</h2>
<p>Tasks are handled by Mesos. A task is a collection of processes that
runs in a shared sandbox. It&rsquo;s the fundamental unit Aurora uses to
schedule the datacenter; essentially what Aurora does is find places
in the cluster to run tasks.</p>
<p>The key (and required) parts of a Task are:</p>
<ul>
<li><p><code>name</code>: A string giving the Task&rsquo;s name. By default, if a Task is
not given a name, it inherits the first name in its Process list.</p></li>
<li><p><code>processes</code>: An unordered list of Process objects bound to the Task.
The value of the optional <code>constraints</code> attribute affects the
contents as a whole. Currently, the only constraint, <code>order</code>, determines if
the processes run in parallel or sequentially.</p></li>
<li><p><code>resources</code>: A <code>Resource</code> object defining the Task&rsquo;s resource
footprint. A <code>Resource</code> object has three attributes:
- <code>cpu</code>: A Float, the fractional number of cores the Task
requires.
- <code>ram</code>: An Integer, RAM bytes the Task requires.
- <code>disk</code>: An integer, disk bytes the Task requires.</p></li>
</ul>
<p>A basic Task definition looks like:</p>
<pre class="highlight plaintext"><code>Task(
name="hello_world",
processes=[Process(name = "hello_world", cmdline = "echo hello world")],
resources=Resources(cpu = 1.0,
ram = 1*GB,
disk = 1*GB))
</code></pre>
<p>A Task has optional attributes to customize its behaviour. Details can be found in the <a href="../configuration/#task-object">Aurora Configuration Reference</a></p>
<h3 id="sequentialtask-running-processes-in-parallel-or-sequentially">SequentialTask: Running Processes in Parallel or Sequentially</h3>
<p>By default, a Task with several Processes runs them in parallel. There
are two ways to run Processes sequentially:</p>
<ul>
<li><p>Include an <code>order</code> constraint in the Task definition&rsquo;s <code>constraints</code>
attribute whose arguments specify the processes&rsquo; run order:</p>
<pre class="highlight plaintext"><code>Task( ... processes=[process1, process2, process3],
constraints = order(process1, process2, process3), ...)
</code></pre></li>
<li><p>Use <code>SequentialTask</code> instead of <code>Task</code>; it automatically runs
processes in the order specified in the <code>processes</code> attribute. No
<code>constraint</code> parameter is needed:</p>
<pre class="highlight plaintext"><code>SequentialTask( ... processes=[process1, process2, process3] ...)
</code></pre></li>
</ul>
<h3 id="simpletask">SimpleTask</h3>
<p>For quickly creating simple tasks, use the <code>SimpleTask</code> helper. It
creates a basic task from a provided name and command line using a
default set of resources. For example, in a .<code>aurora</code> configuration
file:</p>
<pre class="highlight plaintext"><code>SimpleTask(name="hello_world", command="echo hello world")
</code></pre>
<p>is equivalent to</p>
<pre class="highlight plaintext"><code>Task(name="hello_world",
processes=[Process(name = "hello_world", cmdline = "echo hello world")],
resources=Resources(cpu = 1.0,
ram = 1*GB,
disk = 1*GB))
</code></pre>
<p>The simplest idiomatic Job configuration thus becomes:</p>
<pre class="highlight plaintext"><code>import os
hello_world_job = Job(
task=SimpleTask(name="hello_world", command="echo hello world"),
role=os.getenv('USER'),
cluster="cluster1")
</code></pre>
<p>When written to <code>hello_world.aurora</code>, you invoke it with a simple
<code>aurora job create cluster1/$USER/test/hello_world hello_world.aurora</code>.</p>
<h3 id="combining-tasks">Combining tasks</h3>
<p><code>Tasks.concat</code>(synonym,<code>concat_tasks</code>) and
<code>Tasks.combine</code>(synonym,<code>combine_tasks</code>) merge multiple Task definitions
into a single Task. It may be easier to define complex Jobs
as smaller constituent Tasks. But since a Job only includes a single
Task, the subtasks must be combined before using them in a Job.
Smaller Tasks can also be reused between Jobs, instead of having to
repeat their definition for multiple Jobs.</p>
<p>With both methods, the merged Task takes the first Task&rsquo;s name. The
difference between the two is the result Task&rsquo;s process ordering.</p>
<ul>
<li><p><code>Tasks.combine</code> runs its subtasks&rsquo; processes in no particular order.
The new Task&rsquo;s resource consumption is the sum of all its subtasks&rsquo;
consumption.</p></li>
<li><p><code>Tasks.concat</code> runs its subtasks in the order supplied, with each
subtask&rsquo;s processes run serially between tasks. It is analogous to
the <code>order</code> constraint helper, except at the Task level instead of
the Process level. The new Task&rsquo;s resource consumption is the
maximum value specified by any subtask for each Resource attribute
(cpu, ram and disk).</p></li>
</ul>
<p>For example, given the following:</p>
<pre class="highlight plaintext"><code>setup_task = Task(
...
processes=[download_interpreter, update_zookeeper],
# It is important to note that {{Tasks.concat}} has
# no effect on the ordering of the processes within a task;
# hence the necessity of the {{order}} statement below
# (otherwise, the order in which {{download_interpreter}}
# and {{update_zookeeper}} run will be non-deterministic)
constraints=order(download_interpreter, update_zookeeper),
...
)
run_task = SequentialTask(
...
processes=[download_application, start_application],
...
)
combined_task = Tasks.concat(setup_task, run_task)
</code></pre>
<p>The <code>Tasks.concat</code> command merges the two Tasks into a single Task and
ensures all processes in <code>setup_task</code> run before the processes
in <code>run_task</code>. Conceptually, the task is reduced to:</p>
<pre class="highlight plaintext"><code>task = Task(
...
processes=[download_interpreter, update_zookeeper,
download_application, start_application],
constraints=order(download_interpreter, update_zookeeper,
download_application, start_application),
...
)
</code></pre>
<p>In the case of <code>Tasks.combine</code>, the two schedules run in parallel:</p>
<pre class="highlight plaintext"><code>task = Task(
...
processes=[download_interpreter, update_zookeeper,
download_application, start_application],
constraints=order(download_interpreter, update_zookeeper) +
order(download_application, start_application),
...
)
</code></pre>
<p>In the latter case, each of the two sequences may operate in parallel.
Of course, this may not be the intended behavior (for example, if
the <code>start_application</code> Process implicitly relies
upon <code>download_interpreter</code>). Make sure you understand the difference
between using one or the other.</p>
<h2 id="defining-job-objects">Defining Job Objects</h2>
<p>A job is a group of identical tasks that Aurora can run in a Mesos cluster.</p>
<p>A <code>Job</code> object is defined by the values of several attributes, some
required and some optional. The required attributes are:</p>
<ul>
<li><p><code>task</code>: Task object to bind to this job. Note that a Job can
only take a single Task.</p></li>
<li><p><code>role</code>: Job&rsquo;s role account; in other words, the user account to run
the job as on a Mesos cluster machine. A common value is
<code>os.getenv(&#39;USER&#39;)</code>; using a Python command to get the user who
submits the job request. The other common value is the service
account that runs the job, e.g. <code>www-data</code>.</p></li>
<li><p><code>environment</code>: Job&rsquo;s environment, typical values
are <code>devel</code>, <code>test</code>, or <code>prod</code>.</p></li>
<li><p><code>cluster</code>: Aurora cluster to schedule the job in, defined in
<code>/etc/aurora/clusters.json</code> or <code>~/.clusters.json</code>. You can specify
jobs where the only difference is the <code>cluster</code>, then at run time
only run the Job whose job key includes your desired cluster&rsquo;s name.</p></li>
</ul>
<p>You usually see a <code>name</code> parameter. By default, <code>name</code> inherits its
value from the Job&rsquo;s associated Task object, but you can override this
default. For these four parameters, a Job definition might look like:</p>
<pre class="highlight plaintext"><code>foo_job = Job( name = 'foo', cluster = 'cluster1',
role = os.getenv('USER'), environment = 'prod',
task = foo_task)
</code></pre>
<p>In addition to the required attributes, there are several optional
attributes. Details can be found in the <a href="../configuration/#job-objects">Aurora Configuration Reference</a>.</p>
<h2 id="the-jobs-list">The jobs List</h2>
<p>At the end of your <code>.aurora</code> file, you need to specify a list of the
file&rsquo;s defined Jobs. For example, the following exports the jobs <code>job1</code>,
<code>job2</code>, and <code>job3</code>.</p>
<pre class="highlight plaintext"><code>jobs = [job1, job2, job3]
</code></pre>
<p>This allows the aurora client to invoke commands on those jobs, such as
starting, updating, or killing them.</p>
<h1 id="basic-examples">Basic Examples</h1>
<p>These are provided to give a basic understanding of simple Aurora jobs.</p>
<h3 id="hello_world-aurora">hello_world.aurora</h3>
<p>Put the following in a file named <code>hello_world.aurora</code>, substituting your own values
for values such as <code>cluster</code>s.</p>
<pre class="highlight plaintext"><code>import os
hello_world_process = Process(name = 'hello_world', cmdline = 'echo hello world')
hello_world_task = Task(
resources = Resources(cpu = 0.1, ram = 16 * MB, disk = 16 * MB),
processes = [hello_world_process])
hello_world_job = Job(
cluster = 'cluster1',
role = os.getenv('USER'),
task = hello_world_task)
jobs = [hello_world_job]
</code></pre>
<p>Then issue the following commands to create and kill the job, using your own values for the job key.</p>
<pre class="highlight plaintext"><code>aurora job create cluster1/$USER/test/hello_world hello_world.aurora
aurora job kill cluster1/$USER/test/hello_world
</code></pre>
<h3 id="environment-tailoring">Environment Tailoring</h3>
<p>Put the following in a file named <code>hello_world_productionized.aurora</code>, substituting your own values
for values such as <code>cluster</code>s.</p>
<pre class="highlight plaintext"><code>include('hello_world.aurora')
production_resources = Resources(cpu = 1.0, ram = 512 * MB, disk = 2 * GB)
staging_resources = Resources(cpu = 0.1, ram = 32 * MB, disk = 512 * MB)
hello_world_template = hello_world(
name = "hello_world-{{cluster}}"
task = hello_world(resources=production_resources))
jobs = [
# production jobs
hello_world_template(cluster = 'cluster1', instances = 25),
hello_world_template(cluster = 'cluster2', instances = 15),
# staging jobs
hello_world_template(
cluster = 'local',
instances = 1,
task = hello_world(resources=staging_resources)),
]
</code></pre>
<p>Then issue the following commands to create and kill the job, using your own values for the job key</p>
<pre class="highlight plaintext"><code>aurora job create cluster1/$USER/test/hello_world-cluster1 hello_world_productionized.aurora
aurora job kill cluster1/$USER/test/hello_world-cluster1
</code></pre>
</div>
</div>
</div>
<div class="container-fluid section-footer buffer">
<div class="container">
<div class="row">
<div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3>
<ul>
<li><a href="/downloads/">Downloads</a></li>
<li><a href="/community/">Mailing Lists</a></li>
<li><a href="http://issues.apache.org/jira/browse/AURORA">Issue Tracking</a></li>
<li><a href="/documentation/latest/contributing/">How To Contribute</a></li>
</ul>
</div>
<div class="col-md-2"><h3>The ASF</h3>
<ul>
<li><a href="http://www.apache.org/licenses/">License</a></li>
<li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
<li><a href="http://www.apache.org/security/">Security</a></li>
</ul>
</div>
<div class="col-md-6">
<p class="disclaimer">&copy; 2014-2017 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/">Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/">Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p>
</div>
</div>
</div>
</body>
</html>