Google Git
Sign in
apache / aurora-website / refs/heads/asf-site / . / content / documentation / 0.19.1 / operations / configuration / index.html
blob: 8bba5f088ef85cb1175c4949b866f91f6df2aceb [file] [log] [blame]
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Apache Aurora</title>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css">
<link href="/assets/css/main.css" rel="stylesheet">
<!-- Analytics -->
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-45879646-1']);
_gaq.push(['_setDomainName', 'apache.org']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
</head>
<body>
<div class="container-fluid section-header">
<div class="container">
<div class="nav nav-bar">
<a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a>
<ul class="nav navbar-nav navbar-right">
<li><a href="/documentation/latest/">Documentation</a></li>
<li><a href="/community/">Community</a></li>
<li><a href="/downloads/">Downloads</a></li>
<li><a href="/blog/">Blog</a></li>
</ul>
</div>
</div>
</div>
<div class="container-fluid">
<div class="container content">
<div class="col-md-12 documentation">
<h5 class="page-header text-uppercase">Documentation
<select onChange="window.location.href='/documentation/' + this.value + '/operations/configuration/'"
value="0.19.1">
<option value="0.22.0"
>
0.22.0
(latest)
</option>
<option value="0.21.0"
>
0.21.0
</option>
<option value="0.20.0"
>
0.20.0
</option>
<option value="0.19.1"
selected="selected">
0.19.1
</option>
<option value="0.19.0"
>
0.19.0
</option>
<option value="0.18.1"
>
0.18.1
</option>
<option value="0.18.0"
>
0.18.0
</option>
<option value="0.17.0"
>
0.17.0
</option>
<option value="0.16.0"
>
0.16.0
</option>
<option value="0.15.0"
>
0.15.0
</option>
<option value="0.14.0"
>
0.14.0
</option>
<option value="0.13.0"
>
0.13.0
</option>
<option value="0.12.0"
>
0.12.0
</option>
<option value="0.11.0"
>
0.11.0
</option>
<option value="0.10.0"
>
0.10.0
</option>
<option value="0.9.0"
>
0.9.0
</option>
<option value="0.8.0"
>
0.8.0
</option>
<option value="0.7.0-incubating"
>
0.7.0-incubating
</option>
<option value="0.6.0-incubating"
>
0.6.0-incubating
</option>
<option value="0.5.0-incubating"
>
0.5.0-incubating
</option>
</select>
</h5>
<h1 id="scheduler-configuration">Scheduler Configuration</h1>
<p>The Aurora scheduler can take a variety of configuration options through command-line arguments.
Examples are available under <code>examples/scheduler/</code>. For a list of available Aurora flags and their
documentation, see <a href="../../reference/scheduler-configuration/">Scheduler Configuration Reference</a>.</p>
<h2 id="a-note-on-configuration">A Note on Configuration</h2>
<p>Like Mesos, Aurora uses command-line flags for runtime configuration. As such the Aurora
&ldquo;configuration file&rdquo; is typically a <code>scheduler.sh</code> shell script of the form.</p>
<pre class="highlight shell"><code><span style="color: #999988;font-style: italic">#!/bin/bash</span>
<span style="color: #008080">AURORA_HOME</span><span style="color: #000000;font-weight: bold">=</span>/usr/local/aurora-scheduler
<span style="color: #999988;font-style: italic"># Flags controlling the JVM.</span>
<span style="color: #008080">JAVA_OPTS</span><span style="color: #000000;font-weight: bold">=(</span>
-Xmx2g
-Xms2g
<span style="color: #999988;font-style: italic"># GC tuning, etc.</span>
<span style="color: #000000;font-weight: bold">)</span>
<span style="color: #999988;font-style: italic"># Flags controlling the scheduler.</span>
<span style="color: #008080">AURORA_FLAGS</span><span style="color: #000000;font-weight: bold">=(</span>
<span style="color: #999988;font-style: italic"># Port for client RPCs and the web UI</span>
-http_port<span style="color: #000000;font-weight: bold">=</span>8081
<span style="color: #999988;font-style: italic"># Log configuration, etc.</span>
<span style="color: #000000;font-weight: bold">)</span>
<span style="color: #999988;font-style: italic"># Environment variables controlling libmesos</span>
<span style="color: #0086B3">export </span><span style="color: #008080">JAVA_HOME</span><span style="color: #000000;font-weight: bold">=</span>...
<span style="color: #0086B3">export </span><span style="color: #008080">GLOG_v</span><span style="color: #000000;font-weight: bold">=</span>1
<span style="color: #0086B3">export </span><span style="color: #008080">LIBPROCESS_PORT</span><span style="color: #000000;font-weight: bold">=</span>8083
<span style="color: #0086B3">export </span><span style="color: #008080">LIBPROCESS_IP</span><span style="color: #000000;font-weight: bold">=</span>192.168.33.7
<span style="color: #008080">JAVA_OPTS</span><span style="color: #000000;font-weight: bold">=</span><span style="color: #d14">"</span><span style="color: #000000;font-weight: bold">${</span><span style="color: #008080">JAVA_OPTS</span><span style="background-color: #f8f8f8">[*]</span><span style="color: #000000;font-weight: bold">}</span><span style="color: #d14">"</span> <span style="color: #0086B3">exec</span> <span style="color: #d14">"</span><span style="color: #008080">$AURORA_HOME</span><span style="color: #d14">/bin/aurora-scheduler"</span> <span style="color: #d14">"</span><span style="color: #000000;font-weight: bold">${</span><span style="color: #008080">AURORA_FLAGS</span><span style="background-color: #f8f8f8">[@]</span><span style="color: #000000;font-weight: bold">}</span><span style="color: #d14">"</span>
</code></pre>
<p>That way Aurora&rsquo;s current flags are visible in <code>ps</code> and in the <code>/vars</code> admin endpoint.</p>
<h2 id="jvm-configuration">JVM Configuration</h2>
<p>JVM settings are dependent on your environment and cluster size. They might require
custom tuning. As a starting point, we recommend:</p>
<ul>
<li>Ensure the initial (<code>-Xms</code>) and maximum (<code>-Xmx</code>) heap size are idential to prevent heap resizing
at runtime.</li>
<li>Either <code>-XX:+UseConcMarkSweepGC</code> or <code>-XX:+UseG1GC -XX:+UseStringDeduplication</code> are
sane defaults for the garbage collector.</li>
<li><code>-Djava.net.preferIPv4Stack=true</code> makes sense in most cases as well.</li>
</ul>
<h2 id="network-configuration">Network Configuration</h2>
<p>By default, Aurora binds to all interfaces and auto-discovers its hostname. To reduce ambiguity
it helps to hardcode them though:</p>
<pre class="highlight plaintext"><code>-http_port=8081
-ip=192.168.33.7
-hostname="aurora1.us-east1.example.org"
</code></pre>
<p>Two environment variables control the ip and port for the communication with the Mesos master
and for the replicated log used by Aurora:</p>
<pre class="highlight plaintext"><code>export LIBPROCESS_PORT=8083
export LIBPROCESS_IP=192.168.33.7
</code></pre>
<p>It is important that those can be reached from all Mesos master and Aurora scheduler instances.</p>
<h2 id="replicated-log-configuration">Replicated Log Configuration</h2>
<p>Aurora schedulers use ZooKeeper to discover log replicas and elect a leader. Only one scheduler is
leader at a given time - the other schedulers follow log writes and prepare to take over as leader
but do not communicate with the Mesos master. Either 3 or 5 schedulers are recommended in a
production deployment depending on failure tolerance and they must have persistent storage.</p>
<p>Below is a summary of scheduler storage configuration flags that either don&rsquo;t have default values
or require attention before deploying in a production environment.</p>
<h3 id="native_log_quorum_size"><code>-native_log_quorum_size</code></h3>
<p>Defines the Mesos replicated log quorum size. In a cluster with <code>N</code> schedulers, the flag
<code>-native_log_quorum_size</code> should be set to <code>floor(N/2) + 1</code>. So in a cluster with 1 scheduler
it should be set to <code>1</code>, in a cluster with 3 it should be set to <code>2</code>, and in a cluster of 5 it
should be set to <code>3</code>.</p>
<table><thead>
<tr>
<th>Number of schedulers (N)</th>
<th><code>-native_log_quorum_size</code> setting (<code>floor(N/2) + 1</code>)</th>
</tr>
</thead><tbody>
<tr>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td>3</td>
<td>2</td>
</tr>
<tr>
<td>5</td>
<td>3</td>
</tr>
<tr>
<td>7</td>
<td>4</td>
</tr>
</tbody></table>
<p><em>Incorrectly setting this flag will cause data corruption to occur!</em></p>
<h3 id="native_log_file_path"><code>-native_log_file_path</code></h3>
<p>Location of the Mesos replicated log files. For optimal and consistent performance, consider
allocating a dedicated disk (preferably SSD) for the replicated log. Ensure that this disk is not
used by anything else (e.g. no process logging) and in particular that it is a real disk
and not just a partition.</p>
<p>Even when a dedicated disk is used, switching from <code>CFQ</code> to <code>deadline</code> I/O scheduler of Linux kernel
can furthermore help with storage performance in Aurora (<a href="https://issues.apache.org/jira/browse/AURORA-1211">see this ticket for details</a>).</p>
<h3 id="native_log_zk_group_path"><code>-native_log_zk_group_path</code></h3>
<p>ZooKeeper path used for Mesos replicated log quorum discovery.</p>
<p>See <a href="https://github.com/apache/aurora/blob/rel/0.19.1/src/main/java/org/apache/aurora/scheduler/log/mesos/MesosLogStreamModule.java">code</a> for
other available Mesos replicated log configuration options and default values.</p>
<h3 id="changing-the-quorum-size">Changing the Quorum Size</h3>
<p>Special care needs to be taken when changing the size of the Aurora scheduler quorum.
Since Aurora uses a Mesos replicated log, similar steps need to be followed as when
<a href="http://mesos.apache.org/documentation/latest/operational-guide">changing the Mesos quorum size</a>.</p>
<p>As a preparation, increase <code>-native_log_quorum_size</code> on each existing scheduler and restart them.
When updating from 3 to 5 schedulers, the quorum size would grow from 2 to 3.</p>
<p>When starting the new schedulers, use the <code>-native_log_quorum_size</code> set to the new value. Failing to
first increase the quorum size on running schedulers can in some cases result in corruption
or truncating of the replicated log used by Aurora. In that case, see the documentation on
<a href="../backup-restore/">recovering from backup</a>.</p>
<h2 id="backup-configuration">Backup Configuration</h2>
<p>Configuration options for the Aurora scheduler backup manager.</p>
<ul>
<li><code>-backup_interval</code>: The interval on which the scheduler writes local storage backups.
The default is every hour.</li>
<li><code>-backup_dir</code>: Directory to write backups to. As stated above, this should not be co-located on the
same disk as the replicated log.</li>
<li><code>-max_saved_backups</code>: Maximum number of backups to retain before deleting the oldest backup(s).</li>
</ul>
<h2 id="resource-isolation">Resource Isolation</h2>
<p>For proper CPU, memory, and disk isolation as mentioned in our <a href="../../features/resource-isolation/">enduser documentation</a>,
we recommend to add the following isolators to the <code>--isolation</code> flag of the Mesos agent:</p>
<ul>
<li><code>cgroups/cpu</code></li>
<li><code>cgroups/mem</code></li>
<li><code>disk/du</code></li>
</ul>
<p>In addition, we recommend to set the following <a href="http://mesos.apache.org/documentation/latest/configuration/">agent flags</a>:</p>
<ul>
<li><code>--cgroups_limit_swap</code> to enable memory limits on both memory and swap instead of just memory.
Alternatively, you could disable swap on your agent hosts.</li>
<li><code>--cgroups_enable_cfs</code> to enable hard limits on CPU resources via the CFS bandwidth limiting
feature.</li>
<li><code>--enforce_container_disk_quota</code> to enable disk quota enforcement for containers.</li>
</ul>
<p>To enable the optional GPU support in Mesos, please see the GPU related flags in the
<a href="http://mesos.apache.org/documentation/latest/configuration/">Mesos configuration</a>.
To enable the corresponding feature in Aurora, you have to start the scheduler with the
flag</p>
<pre class="highlight plaintext"><code>-allow_gpu_resource=true
</code></pre>
<p>If you want to use revocable resources, first follow the
<a href="http://mesos.apache.org/documentation/latest/oversubscription/">Mesos oversubscription documentation</a>
and then set set this Aurora scheduler flag to allow receiving revocable Mesos offers:</p>
<pre class="highlight plaintext"><code>-receive_revocable_resources=true
</code></pre>
<p>Both CPUs and RAM are supported as revocable resources. The former is enabled by the default,
the latter needs to be enabled via:</p>
<pre class="highlight plaintext"><code>-enable_revocable_ram=true
</code></pre>
<p>Unless you want to use the <a href="https://github.com/apache/aurora/blob/rel/0.19.1/src/main/resources/org/apache/aurora/scheduler/tiers.json">default</a>
tier configuration, you will also have to specify a file path:</p>
<pre class="highlight plaintext"><code>-tier_config=path/to/tiers/config.json
</code></pre>
<h2 id="multi-framework-setup">Multi-Framework Setup</h2>
<p>Aurora holds onto Mesos offers in order to provide efficient scheduling and
<a href="../../features/multitenancy/#preemption">preemption</a>. This is problematic in multi-framework
environments as Aurora might starve other frameworks.</p>
<p>With a downside of increased scheduling latency, Aurora can be configured to be more cooperative:</p>
<ul>
<li>Lowering <code>-min_offer_hold_time</code> (e.g. to <code>1mins</code>) can ensure unused offers are returned back to
Mesos more frequently.</li>
<li>Increasing <code>-offer_filter_duration</code> (e.g to <code>30secs</code>) will instruct Mesos
not to re-offer rejected resources for the given duration.</li>
</ul>
<p>Setting a <a href="http://mesos.apache.org/documentation/latest/quota/">minimum amount of resources</a> for
each Mesos role can furthermore help to ensure no framework is starved entirely.</p>
<h2 id="containers">Containers</h2>
<p>Both the Mesos and Docker containerizers require configuration of the Mesos agent.</p>
<h3 id="mesos-containerizer">Mesos Containerizer</h3>
<p>The minimal agent configuration requires to enable Docker and Appc image support for the Mesos
containerizer:</p>
<pre class="highlight plaintext"><code>--containerizers=mesos
--image_providers=appc,docker
--isolation=filesystem/linux,docker/runtime # as an addition to your other isolators
</code></pre>
<p>Further details can be found in the corresponding <a href="http://mesos.apache.org/documentation/latest/container-image/">Mesos documentation</a>.</p>
<h3 id="docker-containerizer">Docker Containerizer</h3>
<p>The <a href="http://mesos.apache.org/documentation/latest/docker-containerizer/">Docker containerizer</a>
requires the Docker engine is installed on each agent host. In addition, it must be enabled on the
Mesos agents by launching them with the option:</p>
<pre class="highlight plaintext"><code>--containerizers=mesos,docker
</code></pre>
<p>If you would like to run a container with a read-only filesystem, it may also be necessary to use
the scheduler flag <code>-thermos_home_in_sandbox</code> in order to set HOME to the sandbox
before the executor runs. This will make sure that the executor/runner PEX extractions happens
inside of the sandbox instead of the container filesystem root.</p>
<p>If you would like to supply your own parameters to <code>docker run</code> when launching jobs in docker
containers, you may use the following flags:</p>
<pre class="highlight plaintext"><code>-allow_docker_parameters
-default_docker_parameters
</code></pre>
<p><code>-allow_docker_parameters</code> controls whether or not users may pass their own configuration parameters
through the job configuration files. If set to <code>false</code> (the default), the scheduler will reject
jobs with custom parameters. <em>NOTE</em>: this setting should be used with caution as it allows any job
owner to specify any parameters they wish, including those that may introduce security concerns
(<code>privileged=true</code>, for example).</p>
<p><code>-default_docker_parameters</code> allows a cluster operator to specify a universal set of parameters that
should be used for every container that does not have parameters explicitly configured at the job
level. The argument accepts a multimap format:</p>
<pre class="highlight plaintext"><code>-default_docker_parameters="read-only=true,tmpfs=/tmp,tmpfs=/run"
</code></pre>
<h3 id="common-options">Common Options</h3>
<p>The following Aurora options work for both containerizers.</p>
<p>A scheduler flag, <code>-global_container_mounts</code> allows mounting paths from the host (i.e the agent machine)
into all containers on that host. The format is a comma separated list of host<em>path:container</em>path[:mode]
tuples. For example <code>-global_container_mounts=/opt/secret_keys_dir:/mnt/secret_keys_dir:ro</code> mounts
<code>/opt/secret_keys_dir</code> from the agents into all launched containers. Valid modes are <code>ro</code> and <code>rw</code>.</p>
<h2 id="thermos-process-logs">Thermos Process Logs</h2>
<h3 id="log-destination">Log destination</h3>
<p>By default, Thermos will write process stdout/stderr to log files in the sandbox. Process object
configuration allows specifying alternate log file destinations like streamed stdout/stderr or
suppression of all log output. Default behavior can be configured for the entire cluster with the
following flag (through the <code>-thermos_executor_flags</code> argument to the Aurora scheduler):</p>
<pre class="highlight plaintext"><code>--runner-logger-destination=both
</code></pre>
<p><code>both</code> configuration will send logs to files and stream to parent stdout/stderr outputs.</p>
<p>See <a href="../../reference/configuration/#logger">Configuration Reference</a> for all destination options.</p>
<h3 id="log-rotation">Log rotation</h3>
<p>By default, Thermos will not rotate the stdout/stderr logs from child processes and they will grow
without bound. An individual user may change this behavior via configuration on the Process object,
but it may also be desirable to change the default configuration for the entire cluster.
In order to enable rotation by default, the following flags can be applied to Thermos (through the
<code>-thermos_executor_flags</code> argument to the Aurora scheduler):</p>
<pre class="highlight plaintext"><code>--runner-logger-mode=rotate
--runner-rotate-log-size-mb=100
--runner-rotate-log-backups=10
</code></pre>
<p>In the above example, each instance of the Thermos runner will rotate stderr/stdout logs once they
reach 100 MiB in size and keep a maximum of 10 backups. If a user has provided a custom setting for
their process, it will override these default settings.</p>
<h2 id="thermos-executor-wrapper">Thermos Executor Wrapper</h2>
<p>If you need to do computation before starting the Thermos executor (for example, setting a different
<code>--announcer-hostname</code> parameter for every executor), then the Thermos executor should be invoked
inside a wrapper script. In such a case, the aurora scheduler should be started with
<code>-thermos_executor_path</code> pointing to the wrapper script and <code>-thermos_executor_resources</code> set to a
comma separated string of all the resources that should be copied into the sandbox (including the
original Thermos executor). Ensure the wrapper script does not access resources outside of the
sandbox, as when the script is run from within a Docker container those resources may not exist.</p>
<p>For example, to wrap the executor inside a simple wrapper, the scheduler will be started like this
<code>-thermos_executor_path=/path/to/wrapper.sh -thermos_executor_resources=/usr/share/aurora/bin/thermos_executor.pex</code></p>
<h2 id="custom-executors">Custom Executors</h2>
<p>The scheduler can be configured to utilize a custom executor by specifying the <code>-custom_executor_config</code> flag.
The flag must be set to the path of a valid executor configuration file.</p>
<p>For more information on this feature please see the custom executors <a href="../../features/custom-executors/">documentation</a>.</p>
<h2 id="a-note-on-increasing-executor-overhead">A note on increasing executor overhead</h2>
<p>Increasing executor overhead on an existing cluster, whether it be for custom executors or for Thermos,
will result in degraded preemption performance until all task which began life with the previous
executor configuration with less overhead are preempted/restarted.</p>
<h2 id="controlling-mtta-via-update-affinity">Controlling MTTA via Update Affinity</h2>
<p>When there is high resource contention in your cluster you may experience noticably elevated job update
times, as well as high task churn across the cluster. This is due to Aurora&rsquo;s first-fit scheduling
algorithm. To alleviate this, you can enable update affinity where the Scheduler will make a best-effort
attempt to reuse the same agent for the updated task (so long as the resources for the job are not being
increased).</p>
<p>To enable this in the Scheduler, you can set the following options:</p>
<pre class="highlight plaintext"><code>--enable_update_affinity=true
--update_affinity_reservation_hold_time=3mins
</code></pre>
<p>You will need to tune the hold time to match the behavior you see in your cluster. If you have extremely
high update throughput, you might have to extend it as processing updates could easily add significant
delays between scheduling attempts. You may also have to tune scheduling parameters to achieve the
throughput you need in your cluster. Some relevant settings (with defaults) are:</p>
<pre class="highlight plaintext"><code>--max_schedule_attempts_per_sec=40
--initial_schedule_penalty=1secs
--max_schedule_penalty=1mins
--scheduling_max_batch_size=3
--max_tasks_per_schedule_attempt=5
</code></pre>
<p>There are metrics exposed by the Scheduler which can provide guidance on where the bottleneck is.
Example metrics to look at:</p>
<pre class="highlight plaintext"><code>- schedule_attempts_blocks (if this number is greater than 0, then task throughput is hitting
limits controlled by --max_scheduler_attempts_per_sec)
- scheduled_task_penalty_* (metrics around scheduling penalties for tasks, if the numbers here are high
then you could have high contention for resources)
</code></pre>
<p>Most likely you&rsquo;ll run into limits with the number of update instances that can be processed per minute
before you run into any other limits. So if your total work done per minute starts to exceed 2k instances,
you may need to extend the update<em>affinity</em>reservation<em>hold</em>time.</p>
</div>
</div>
</div>
<div class="container-fluid section-footer buffer">
<div class="container">
<div class="row">
<div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3>
<ul>
<li><a href="/downloads/">Downloads</a></li>
<li><a href="/community/">Mailing Lists</a></li>
<li><a href="http://issues.apache.org/jira/browse/AURORA">Issue Tracking</a></li>
<li><a href="/documentation/latest/contributing/">How To Contribute</a></li>
</ul>
</div>
<div class="col-md-2"><h3>The ASF</h3>
<ul>
<li><a href="http://www.apache.org/licenses/">License</a></li>
<li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
<li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
<li><a href="http://www.apache.org/security/">Security</a></li>
</ul>
</div>
<div class="col-md-6">
<p class="disclaimer">&copy; 2014-2017 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/">Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/">Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p>
</div>
</div>
</div>
</body>
</html>