| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>Apache Aurora</title> |
| <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css"> |
| <link href="/assets/css/main.css" rel="stylesheet"> |
| <!-- Analytics --> |
| <script type="text/javascript"> |
| var _gaq = _gaq || []; |
| _gaq.push(['_setAccount', 'UA-45879646-1']); |
| _gaq.push(['_setDomainName', 'apache.org']); |
| _gaq.push(['_trackPageview']); |
| |
| (function() { |
| var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; |
| ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; |
| var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); |
| })(); |
| </script> |
| </head> |
| <body> |
| <div class="container-fluid section-header"> |
| <div class="container"> |
| <div class="nav nav-bar"> |
| <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> |
| <ul class="nav navbar-nav navbar-right"> |
| <li><a href="/documentation/latest/">Documentation</a></li> |
| <li><a href="/community/">Community</a></li> |
| <li><a href="/downloads/">Downloads</a></li> |
| <li><a href="/blog/">Blog</a></li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| |
| <div class="container-fluid"> |
| <div class="container content"> |
| <div class="col-md-12 documentation"> |
| <h5 class="page-header text-uppercase">Documentation |
| <select onChange="window.location.href='/documentation/' + this.value + '/getting-started/overview/'" |
| value="0.19.1"> |
| <option value="0.22.0" |
| > |
| 0.22.0 |
| (latest) |
| </option> |
| <option value="0.21.0" |
| > |
| 0.21.0 |
| </option> |
| <option value="0.20.0" |
| > |
| 0.20.0 |
| </option> |
| <option value="0.19.1" |
| selected="selected"> |
| 0.19.1 |
| </option> |
| <option value="0.19.0" |
| > |
| 0.19.0 |
| </option> |
| <option value="0.18.1" |
| > |
| 0.18.1 |
| </option> |
| <option value="0.18.0" |
| > |
| 0.18.0 |
| </option> |
| <option value="0.17.0" |
| > |
| 0.17.0 |
| </option> |
| <option value="0.16.0" |
| > |
| 0.16.0 |
| </option> |
| <option value="0.15.0" |
| > |
| 0.15.0 |
| </option> |
| <option value="0.14.0" |
| > |
| 0.14.0 |
| </option> |
| <option value="0.13.0" |
| > |
| 0.13.0 |
| </option> |
| <option value="0.12.0" |
| > |
| 0.12.0 |
| </option> |
| <option value="0.11.0" |
| > |
| 0.11.0 |
| </option> |
| <option value="0.10.0" |
| > |
| 0.10.0 |
| </option> |
| <option value="0.9.0" |
| > |
| 0.9.0 |
| </option> |
| <option value="0.8.0" |
| > |
| 0.8.0 |
| </option> |
| <option value="0.7.0-incubating" |
| > |
| 0.7.0-incubating |
| </option> |
| <option value="0.6.0-incubating" |
| > |
| 0.6.0-incubating |
| </option> |
| <option value="0.5.0-incubating" |
| > |
| 0.5.0-incubating |
| </option> |
| </select> |
| </h5> |
| <h1 id="aurora-system-overview">Aurora System Overview</h1> |
| |
| <p>Apache Aurora is a service scheduler that runs on top of Apache Mesos, enabling you to run |
| long-running services, cron jobs, and ad-hoc jobs that take advantage of Apache Mesos’ scalability, |
| fault-tolerance, and resource isolation.</p> |
| |
| <h2 id="components">Components</h2> |
| |
| <p>It is important to have an understanding of the components that make up |
| a functioning Aurora cluster.</p> |
| |
| <p><img alt="Aurora Components" src="../../images/components.png" /></p> |
| |
| <ul> |
| <li><p><strong>Aurora scheduler</strong> |
| The scheduler is your primary interface to the work you run in your cluster. You will |
| instruct it to run jobs, and it will manage them in Mesos for you. You will also frequently use |
| the scheduler’s read-only web interface as a heads-up display for what’s running in your cluster.</p></li> |
| <li><p><strong>Aurora client</strong> |
| The client (<code>aurora</code> command) is a command line tool that exposes primitives that you can use to |
| interact with the scheduler. The client operates on</p></li> |
| </ul> |
| |
| <p>Aurora also provides an admin client (<code>aurora_admin</code> command) that contains commands built for |
| cluster administrators. You can use this tool to do things like manage user quotas and manage |
| graceful maintenance on machines in cluster.</p> |
| |
| <ul> |
| <li><p><strong>Aurora executor</strong> |
| The executor (a.k.a. Thermos executor) is responsible for carrying out the workloads described in |
| the Aurora DSL (<code>.aurora</code> files). The executor is what actually executes user processes. It will |
| also perform health checking of tasks and register tasks in ZooKeeper for the purposes of dynamic |
| service discovery.</p></li> |
| <li><p><strong>Aurora observer</strong> |
| The observer provides browser-based access to the status of individual tasks executing on worker |
| machines. It gives insight into the processes executing, and facilitates browsing of task sandbox |
| directories.</p></li> |
| <li><p><strong>ZooKeeper</strong> |
| <a href="http://zookeeper.apache.org">ZooKeeper</a> is a distributed consensus system. In an Aurora cluster |
| it is used for reliable election of the leading Aurora scheduler and Mesos master. It is also |
| used as a vehicle for service discovery, see <a href="../../features/service-discovery/">Service Discovery</a></p></li> |
| <li><p><strong>Mesos master</strong> |
| The master is responsible for tracking worker machines and performing accounting of their |
| resources. The scheduler interfaces with the master to control the cluster.</p></li> |
| <li><p><strong>Mesos agent</strong> |
| The agent receives work assigned by the scheduler and executes them. It interfaces with Linux |
| isolation systems like cgroups, namespaces and Docker to manage the resource consumption of tasks. |
| When a user task is launched, the agent will launch the executor (in the context of a Linux cgroup |
| or Docker container depending upon the environment), which will in turn fork user processes.</p></li> |
| </ul> |
| |
| <p>In earlier versions of Mesos and Aurora, the Mesos agent was known as the Mesos slave.</p> |
| |
| <h2 id="jobs-tasks-and-processes">Jobs, Tasks and Processes</h2> |
| |
| <p>Aurora is a Mesos framework used to schedule <em>jobs</em> onto Mesos. Mesos |
| cares about individual <em>tasks</em>, but typical jobs consist of dozens or |
| hundreds of task replicas. Aurora provides a layer on top of Mesos with |
| its <code>Job</code> abstraction. An Aurora <code>Job</code> consists of a task template and |
| instructions for creating near-identical replicas of that task (modulo |
| things like “instance id” or specific port numbers which may differ from |
| machine to machine).</p> |
| |
| <p>How many tasks make up a Job is complicated. On a basic level, a Job consists of |
| one task template and instructions for creating near-identical replicas of that task |
| (otherwise referred to as “instances” or “shards”).</p> |
| |
| <p>A task can merely be a single <em>process</em> corresponding to a single |
| command line, such as <code>python2.7 my_script.py</code>. However, a task can also |
| consist of many separate processes, which all run within a single |
| sandbox. For example, running multiple cooperating agents together, |
| such as <code>logrotate</code>, <code>installer</code>, master, or agent processes. This is |
| where Thermos comes in. While Aurora provides a <code>Job</code> abstraction on |
| top of Mesos <code>Tasks</code>, Thermos provides a <code>Process</code> abstraction |
| underneath Mesos <code>Task</code>s and serves as part of the Aurora framework’s |
| executor.</p> |
| |
| <p>You define <code>Job</code>s,<code>Task</code>s, and <code>Process</code>es in a configuration file. |
| Configuration files are written in Python, and make use of the |
| <a href="https://github.com/wickman/pystachio">Pystachio</a> templating language, |
| along with specific Aurora, Mesos, and Thermos commands and methods. |
| The configuration files typically end with a <code>.aurora</code> extension.</p> |
| |
| <p>Summary:</p> |
| |
| <ul> |
| <li>Aurora manages jobs made of tasks.</li> |
| <li>Mesos manages tasks made of processes.</li> |
| <li>Thermos manages processes.</li> |
| <li>All that is defined in <code>.aurora</code> configuration files</li> |
| </ul> |
| |
| <p><img alt="Aurora hierarchy" src="../../images/aurora_hierarchy.png" /></p> |
| |
| <p>Each <code>Task</code> has a <em>sandbox</em> created when the <code>Task</code> starts and garbage |
| collected when it finishes. All of a <code>Task'</code>s processes run in its |
| sandbox, so processes can share state by using a shared current working |
| directory.</p> |
| |
| <p>The sandbox garbage collection policy considers many factors, most |
| importantly age and size. It makes a best-effort attempt to keep |
| sandboxes around as long as possible post-task in order for service |
| owners to inspect data and logs, should the <code>Task</code> have completed |
| abnormally. But you can’t design your applications assuming sandboxes |
| will be around forever, e.g. by building log saving or other |
| checkpointing mechanisms directly into your application or into your |
| <code>Job</code> description.</p> |
| |
| </div> |
| |
| </div> |
| </div> |
| <div class="container-fluid section-footer buffer"> |
| <div class="container"> |
| <div class="row"> |
| <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> |
| <ul> |
| <li><a href="/downloads/">Downloads</a></li> |
| <li><a href="/community/">Mailing Lists</a></li> |
| <li><a href="http://issues.apache.org/jira/browse/AURORA">Issue Tracking</a></li> |
| <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> |
| </ul> |
| </div> |
| <div class="col-md-2"><h3>The ASF</h3> |
| <ul> |
| <li><a href="http://www.apache.org/licenses/">License</a></li> |
| <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li> |
| <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li> |
| <li><a href="http://www.apache.org/security/">Security</a></li> |
| </ul> |
| </div> |
| <div class="col-md-6"> |
| <p class="disclaimer">© 2014-2017 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/">Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/">Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> |
| </div> |
| </div> |
| </div> |
| |
| </body> |
| </html> |
