| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>Apache Aurora</title> |
| <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css"> |
| <link href="/assets/css/main.css" rel="stylesheet"> |
| <!-- Analytics --> |
| <script type="text/javascript"> |
| var _gaq = _gaq || []; |
| _gaq.push(['_setAccount', 'UA-45879646-1']); |
| _gaq.push(['_setDomainName', 'apache.org']); |
| _gaq.push(['_trackPageview']); |
| |
| (function() { |
| var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; |
| ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; |
| var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); |
| })(); |
| </script> |
| </head> |
| <body> |
| <div class="container-fluid section-header"> |
| <div class="container"> |
| <div class="nav nav-bar"> |
| <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> |
| <ul class="nav navbar-nav navbar-right"> |
| <li><a href="/documentation/latest/">Documentation</a></li> |
| <li><a href="/community/">Community</a></li> |
| <li><a href="/downloads/">Downloads</a></li> |
| <li><a href="/blog/">Blog</a></li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| |
| <div class="container-fluid"> |
| <div class="container content"> |
| <div class="col-md-12 documentation"> |
| <h5 class="page-header text-uppercase">Documentation |
| <select onChange="window.location.href='/documentation/' + this.value + '/operations/configuration/'" |
| value="0.20.0"> |
| <option value="0.22.0" |
| > |
| 0.22.0 |
| (latest) |
| </option> |
| <option value="0.21.0" |
| > |
| 0.21.0 |
| </option> |
| <option value="0.20.0" |
| selected="selected"> |
| 0.20.0 |
| </option> |
| <option value="0.19.1" |
| > |
| 0.19.1 |
| </option> |
| <option value="0.19.0" |
| > |
| 0.19.0 |
| </option> |
| <option value="0.18.1" |
| > |
| 0.18.1 |
| </option> |
| <option value="0.18.0" |
| > |
| 0.18.0 |
| </option> |
| <option value="0.17.0" |
| > |
| 0.17.0 |
| </option> |
| <option value="0.16.0" |
| > |
| 0.16.0 |
| </option> |
| <option value="0.15.0" |
| > |
| 0.15.0 |
| </option> |
| <option value="0.14.0" |
| > |
| 0.14.0 |
| </option> |
| <option value="0.13.0" |
| > |
| 0.13.0 |
| </option> |
| <option value="0.12.0" |
| > |
| 0.12.0 |
| </option> |
| <option value="0.11.0" |
| > |
| 0.11.0 |
| </option> |
| <option value="0.10.0" |
| > |
| 0.10.0 |
| </option> |
| <option value="0.9.0" |
| > |
| 0.9.0 |
| </option> |
| <option value="0.8.0" |
| > |
| 0.8.0 |
| </option> |
| <option value="0.7.0-incubating" |
| > |
| 0.7.0-incubating |
| </option> |
| <option value="0.6.0-incubating" |
| > |
| 0.6.0-incubating |
| </option> |
| <option value="0.5.0-incubating" |
| > |
| 0.5.0-incubating |
| </option> |
| </select> |
| </h5> |
| <h1 id="scheduler-configuration">Scheduler Configuration</h1> |
| |
| <p>The Aurora scheduler can take a variety of configuration options through command-line arguments. |
| Examples are available under <code>examples/scheduler/</code>. For a list of available Aurora flags and their |
| documentation, see <a href="../../reference/scheduler-configuration/">Scheduler Configuration Reference</a>.</p> |
| |
| <h2 id="a-note-on-configuration">A Note on Configuration</h2> |
| |
| <p>Like Mesos, Aurora uses command-line flags for runtime configuration. As such the Aurora |
| “configuration file” is typically a <code>scheduler.sh</code> shell script of the form.</p> |
| <pre class="highlight shell"><code><span style="color: #999988;font-style: italic">#!/bin/bash</span> |
| <span style="color: #008080">AURORA_HOME</span><span style="color: #000000;font-weight: bold">=</span>/usr/local/aurora-scheduler |
| |
| <span style="color: #999988;font-style: italic"># Flags controlling the JVM.</span> |
| <span style="color: #008080">JAVA_OPTS</span><span style="color: #000000;font-weight: bold">=(</span> |
| -Xmx2g |
| -Xms2g |
| <span style="color: #999988;font-style: italic"># GC tuning, etc.</span> |
| <span style="color: #000000;font-weight: bold">)</span> |
| |
| <span style="color: #999988;font-style: italic"># Flags controlling the scheduler.</span> |
| <span style="color: #008080">AURORA_FLAGS</span><span style="color: #000000;font-weight: bold">=(</span> |
| <span style="color: #999988;font-style: italic"># Port for client RPCs and the web UI</span> |
| -http_port<span style="color: #000000;font-weight: bold">=</span>8081 |
| <span style="color: #999988;font-style: italic"># Log configuration, etc.</span> |
| <span style="color: #000000;font-weight: bold">)</span> |
| |
| <span style="color: #999988;font-style: italic"># Environment variables controlling libmesos</span> |
| <span style="color: #0086B3">export </span><span style="color: #008080">JAVA_HOME</span><span style="color: #000000;font-weight: bold">=</span>... |
| <span style="color: #0086B3">export </span><span style="color: #008080">GLOG_v</span><span style="color: #000000;font-weight: bold">=</span>1 |
| <span style="color: #0086B3">export </span><span style="color: #008080">LIBPROCESS_PORT</span><span style="color: #000000;font-weight: bold">=</span>8083 |
| <span style="color: #0086B3">export </span><span style="color: #008080">LIBPROCESS_IP</span><span style="color: #000000;font-weight: bold">=</span>192.168.33.7 |
| |
| <span style="color: #008080">JAVA_OPTS</span><span style="color: #000000;font-weight: bold">=</span><span style="color: #d14">"</span><span style="color: #000000;font-weight: bold">${</span><span style="color: #008080">JAVA_OPTS</span><span style="background-color: #f8f8f8">[*]</span><span style="color: #000000;font-weight: bold">}</span><span style="color: #d14">"</span> <span style="color: #0086B3">exec</span> <span style="color: #d14">"</span><span style="color: #008080">$AURORA_HOME</span><span style="color: #d14">/bin/aurora-scheduler"</span> <span style="color: #d14">"</span><span style="color: #000000;font-weight: bold">${</span><span style="color: #008080">AURORA_FLAGS</span><span style="background-color: #f8f8f8">[@]</span><span style="color: #000000;font-weight: bold">}</span><span style="color: #d14">"</span> |
| </code></pre> |
| |
| <p>That way Aurora’s current flags are visible in <code>ps</code> and in the <code>/vars</code> admin endpoint.</p> |
| |
| <h2 id="jvm-configuration">JVM Configuration</h2> |
| |
| <p>JVM settings are dependent on your environment and cluster size. They might require |
| custom tuning. As a starting point, we recommend:</p> |
| |
| <ul> |
| <li>Ensure the initial (<code>-Xms</code>) and maximum (<code>-Xmx</code>) heap size are idential to prevent heap resizing |
| at runtime.</li> |
| <li>Either <code>-XX:+UseConcMarkSweepGC</code> or <code>-XX:+UseG1GC -XX:+UseStringDeduplication</code> are |
| sane defaults for the garbage collector.</li> |
| <li><code>-Djava.net.preferIPv4Stack=true</code> makes sense in most cases as well.</li> |
| </ul> |
| |
| <h2 id="network-configuration">Network Configuration</h2> |
| |
| <p>By default, Aurora binds to all interfaces and auto-discovers its hostname. To reduce ambiguity |
| it helps to hardcode them though:</p> |
| <pre class="highlight plaintext"><code>-http_port=8081 |
| -ip=192.168.33.7 |
| -hostname="aurora1.us-east1.example.org" |
| </code></pre> |
| |
| <p>Two environment variables control the ip and port for the communication with the Mesos master |
| and for the replicated log used by Aurora:</p> |
| <pre class="highlight plaintext"><code>export LIBPROCESS_PORT=8083 |
| export LIBPROCESS_IP=192.168.33.7 |
| </code></pre> |
| |
| <p>It is important that those can be reached from all Mesos master and Aurora scheduler instances.</p> |
| |
| <h2 id="replicated-log-configuration">Replicated Log Configuration</h2> |
| |
| <p>Aurora schedulers use ZooKeeper to discover log replicas and elect a leader. Only one scheduler is |
| leader at a given time - the other schedulers follow log writes and prepare to take over as leader |
| but do not communicate with the Mesos master. Either 3 or 5 schedulers are recommended in a |
| production deployment depending on failure tolerance and they must have persistent storage.</p> |
| |
| <p>Below is a summary of scheduler storage configuration flags that either don’t have default values |
| or require attention before deploying in a production environment.</p> |
| |
| <h3 id="native_log_quorum_size"><code>-native_log_quorum_size</code></h3> |
| |
| <p>Defines the Mesos replicated log quorum size. In a cluster with <code>N</code> schedulers, the flag |
| <code>-native_log_quorum_size</code> should be set to <code>floor(N/2) + 1</code>. So in a cluster with 1 scheduler |
| it should be set to <code>1</code>, in a cluster with 3 it should be set to <code>2</code>, and in a cluster of 5 it |
| should be set to <code>3</code>.</p> |
| |
| <table><thead> |
| <tr> |
| <th>Number of schedulers (N)</th> |
| <th><code>-native_log_quorum_size</code> setting (<code>floor(N/2) + 1</code>)</th> |
| </tr> |
| </thead><tbody> |
| <tr> |
| <td>1</td> |
| <td>1</td> |
| </tr> |
| <tr> |
| <td>3</td> |
| <td>2</td> |
| </tr> |
| <tr> |
| <td>5</td> |
| <td>3</td> |
| </tr> |
| <tr> |
| <td>7</td> |
| <td>4</td> |
| </tr> |
| </tbody></table> |
| |
| <p><em>Incorrectly setting this flag will cause data corruption to occur!</em></p> |
| |
| <h3 id="native_log_file_path"><code>-native_log_file_path</code></h3> |
| |
| <p>Location of the Mesos replicated log files. For optimal and consistent performance, consider |
| allocating a dedicated disk (preferably SSD) for the replicated log. Ensure that this disk is not |
| used by anything else (e.g. no process logging) and in particular that it is a real disk |
| and not just a partition.</p> |
| |
| <p>Even when a dedicated disk is used, switching from <code>CFQ</code> to <code>deadline</code> I/O scheduler of Linux kernel |
| can furthermore help with storage performance in Aurora (<a href="https://issues.apache.org/jira/browse/AURORA-1211">see this ticket for details</a>).</p> |
| |
| <h3 id="native_log_zk_group_path"><code>-native_log_zk_group_path</code></h3> |
| |
| <p>ZooKeeper path used for Mesos replicated log quorum discovery.</p> |
| |
| <p>See <a href="https://github.com/apache/aurora/blob/rel/0.20.0/src/main/java/org/apache/aurora/scheduler/log/mesos/MesosLogStreamModule.java">code</a> for |
| other available Mesos replicated log configuration options and default values.</p> |
| |
| <h3 id="changing-the-quorum-size">Changing the Quorum Size</h3> |
| |
| <p>Special care needs to be taken when changing the size of the Aurora scheduler quorum. |
| Since Aurora uses a Mesos replicated log, similar steps need to be followed as when |
| <a href="http://mesos.apache.org/documentation/latest/operational-guide">changing the Mesos quorum size</a>.</p> |
| |
| <p>As a preparation, increase <code>-native_log_quorum_size</code> on each existing scheduler and restart them. |
| When updating from 3 to 5 schedulers, the quorum size would grow from 2 to 3.</p> |
| |
| <p>When starting the new schedulers, use the <code>-native_log_quorum_size</code> set to the new value. Failing to |
| first increase the quorum size on running schedulers can in some cases result in corruption |
| or truncating of the replicated log used by Aurora. In that case, see the documentation on |
| <a href="../backup-restore/">recovering from backup</a>.</p> |
| |
| <h2 id="backup-configuration">Backup Configuration</h2> |
| |
| <p>Configuration options for the Aurora scheduler backup manager.</p> |
| |
| <ul> |
| <li><code>-backup_interval</code>: The interval on which the scheduler writes local storage backups. |
| The default is every hour.</li> |
| <li><code>-backup_dir</code>: Directory to write backups to. As stated above, this should not be co-located on the |
| same disk as the replicated log.</li> |
| <li><code>-max_saved_backups</code>: Maximum number of backups to retain before deleting the oldest backup(s).</li> |
| </ul> |
| |
| <h2 id="resource-isolation">Resource Isolation</h2> |
| |
| <p>For proper CPU, memory, and disk isolation as mentioned in our <a href="../../features/resource-isolation/">enduser documentation</a>, |
| we recommend to add the following isolators to the <code>--isolation</code> flag of the Mesos agent:</p> |
| |
| <ul> |
| <li><code>cgroups/cpu</code></li> |
| <li><code>cgroups/mem</code></li> |
| <li><code>disk/du</code></li> |
| </ul> |
| |
| <p>In addition, we recommend to set the following <a href="http://mesos.apache.org/documentation/latest/configuration/">agent flags</a>:</p> |
| |
| <ul> |
| <li><code>--cgroups_limit_swap</code> to enable memory limits on both memory and swap instead of just memory. |
| Alternatively, you could disable swap on your agent hosts.</li> |
| <li><code>--cgroups_enable_cfs</code> to enable hard limits on CPU resources via the CFS bandwidth limiting |
| feature.</li> |
| <li><code>--enforce_container_disk_quota</code> to enable disk quota enforcement for containers.</li> |
| </ul> |
| |
| <p>To enable the optional GPU support in Mesos, please see the GPU related flags in the |
| <a href="http://mesos.apache.org/documentation/latest/configuration/">Mesos configuration</a>. |
| To enable the corresponding feature in Aurora, you have to start the scheduler with the |
| flag</p> |
| <pre class="highlight plaintext"><code>-allow_gpu_resource=true |
| </code></pre> |
| |
| <p>If you want to use revocable resources, first follow the |
| <a href="http://mesos.apache.org/documentation/latest/oversubscription/">Mesos oversubscription documentation</a> |
| and then set set this Aurora scheduler flag to allow receiving revocable Mesos offers:</p> |
| <pre class="highlight plaintext"><code>-receive_revocable_resources=true |
| </code></pre> |
| |
| <p>Both CPUs and RAM are supported as revocable resources. The former is enabled by the default, |
| the latter needs to be enabled via:</p> |
| <pre class="highlight plaintext"><code>-enable_revocable_ram=true |
| </code></pre> |
| |
| <p>Unless you want to use the <a href="https://github.com/apache/aurora/blob/rel/0.20.0/src/main/resources/org/apache/aurora/scheduler/tiers.json">default</a> |
| tier configuration, you will also have to specify a file path:</p> |
| <pre class="highlight plaintext"><code>-tier_config=path/to/tiers/config.json |
| </code></pre> |
| |
| <h2 id="multi-framework-setup">Multi-Framework Setup</h2> |
| |
| <p>Aurora holds onto Mesos offers in order to provide efficient scheduling and |
| <a href="../../features/multitenancy/#preemption">preemption</a>. This is problematic in multi-framework |
| environments as Aurora might starve other frameworks.</p> |
| |
| <p>With a downside of increased scheduling latency, Aurora can be configured to be more cooperative:</p> |
| |
| <ul> |
| <li>Lowering <code>-min_offer_hold_time</code> (e.g. to <code>1mins</code>) can ensure unused offers are returned back to |
| Mesos more frequently.</li> |
| <li>Increasing <code>-offer_filter_duration</code> (e.g to <code>30secs</code>) will instruct Mesos |
| not to re-offer rejected resources for the given duration.</li> |
| </ul> |
| |
| <p>Setting a <a href="http://mesos.apache.org/documentation/latest/quota/">minimum amount of resources</a> for |
| each Mesos role can furthermore help to ensure no framework is starved entirely.</p> |
| |
| <h2 id="containers">Containers</h2> |
| |
| <p>Both the Mesos and Docker containerizers require configuration of the Mesos agent.</p> |
| |
| <h3 id="mesos-containerizer">Mesos Containerizer</h3> |
| |
| <p>The minimal agent configuration requires to enable Docker and Appc image support for the Mesos |
| containerizer:</p> |
| <pre class="highlight plaintext"><code>--containerizers=mesos |
| --image_providers=appc,docker |
| --isolation=filesystem/linux,docker/runtime # as an addition to your other isolators |
| </code></pre> |
| |
| <p>Further details can be found in the corresponding <a href="http://mesos.apache.org/documentation/latest/container-image/">Mesos documentation</a>.</p> |
| |
| <h3 id="docker-containerizer">Docker Containerizer</h3> |
| |
| <p>The <a href="http://mesos.apache.org/documentation/latest/docker-containerizer/">Docker containerizer</a> |
| requires the Docker engine is installed on each agent host. In addition, it must be enabled on the |
| Mesos agents by launching them with the option:</p> |
| <pre class="highlight plaintext"><code>--containerizers=mesos,docker |
| </code></pre> |
| |
| <p>If you would like to run a container with a read-only filesystem, it may also be necessary to use |
| the scheduler flag <code>-thermos_home_in_sandbox</code> in order to set HOME to the sandbox |
| before the executor runs. This will make sure that the executor/runner PEX extractions happens |
| inside of the sandbox instead of the container filesystem root.</p> |
| |
| <p>If you would like to supply your own parameters to <code>docker run</code> when launching jobs in docker |
| containers, you may use the following flags:</p> |
| <pre class="highlight plaintext"><code>-allow_docker_parameters |
| -default_docker_parameters |
| </code></pre> |
| |
| <p><code>-allow_docker_parameters</code> controls whether or not users may pass their own configuration parameters |
| through the job configuration files. If set to <code>false</code> (the default), the scheduler will reject |
| jobs with custom parameters. <em>NOTE</em>: this setting should be used with caution as it allows any job |
| owner to specify any parameters they wish, including those that may introduce security concerns |
| (<code>privileged=true</code>, for example).</p> |
| |
| <p><code>-default_docker_parameters</code> allows a cluster operator to specify a universal set of parameters that |
| should be used for every container that does not have parameters explicitly configured at the job |
| level. The argument accepts a multimap format:</p> |
| <pre class="highlight plaintext"><code>-default_docker_parameters="read-only=true,tmpfs=/tmp,tmpfs=/run" |
| </code></pre> |
| |
| <h3 id="common-options">Common Options</h3> |
| |
| <p>The following Aurora options work for both containerizers.</p> |
| |
| <p>A scheduler flag, <code>-global_container_mounts</code> allows mounting paths from the host (i.e the agent machine) |
| into all containers on that host. The format is a comma separated list of host<em>path:container</em>path[:mode] |
| tuples. For example <code>-global_container_mounts=/opt/secret_keys_dir:/mnt/secret_keys_dir:ro</code> mounts |
| <code>/opt/secret_keys_dir</code> from the agents into all launched containers. Valid modes are <code>ro</code> and <code>rw</code>.</p> |
| |
| <h2 id="thermos-process-logs">Thermos Process Logs</h2> |
| |
| <h3 id="log-destination">Log destination</h3> |
| |
| <p>By default, Thermos will write process stdout/stderr to log files in the sandbox. Process object |
| configuration allows specifying alternate log file destinations like streamed stdout/stderr or |
| suppression of all log output. Default behavior can be configured for the entire cluster with the |
| following flag (through the <code>-thermos_executor_flags</code> argument to the Aurora scheduler):</p> |
| <pre class="highlight plaintext"><code>--runner-logger-destination=both |
| </code></pre> |
| |
| <p><code>both</code> configuration will send logs to files and stream to parent stdout/stderr outputs.</p> |
| |
| <p>See <a href="../../reference/configuration/#logger">Configuration Reference</a> for all destination options.</p> |
| |
| <h3 id="log-rotation">Log rotation</h3> |
| |
| <p>By default, Thermos will not rotate the stdout/stderr logs from child processes and they will grow |
| without bound. An individual user may change this behavior via configuration on the Process object, |
| but it may also be desirable to change the default configuration for the entire cluster. |
| In order to enable rotation by default, the following flags can be applied to Thermos (through the |
| <code>-thermos_executor_flags</code> argument to the Aurora scheduler):</p> |
| <pre class="highlight plaintext"><code>--runner-logger-mode=rotate |
| --runner-rotate-log-size-mb=100 |
| --runner-rotate-log-backups=10 |
| </code></pre> |
| |
| <p>In the above example, each instance of the Thermos runner will rotate stderr/stdout logs once they |
| reach 100 MiB in size and keep a maximum of 10 backups. If a user has provided a custom setting for |
| their process, it will override these default settings.</p> |
| |
| <h2 id="thermos-executor-wrapper">Thermos Executor Wrapper</h2> |
| |
| <p>If you need to do computation before starting the Thermos executor (for example, setting a different |
| <code>--announcer-hostname</code> parameter for every executor), then the Thermos executor should be invoked |
| inside a wrapper script. In such a case, the aurora scheduler should be started with |
| <code>-thermos_executor_path</code> pointing to the wrapper script and <code>-thermos_executor_resources</code> set to a |
| comma separated string of all the resources that should be copied into the sandbox (including the |
| original Thermos executor). Ensure the wrapper script does not access resources outside of the |
| sandbox, as when the script is run from within a Docker container those resources may not exist.</p> |
| |
| <p>For example, to wrap the executor inside a simple wrapper, the scheduler will be started like this |
| <code>-thermos_executor_path=/path/to/wrapper.sh -thermos_executor_resources=/usr/share/aurora/bin/thermos_executor.pex</code></p> |
| |
| <h2 id="custom-executors">Custom Executors</h2> |
| |
| <p>The scheduler can be configured to utilize a custom executor by specifying the <code>-custom_executor_config</code> flag. |
| The flag must be set to the path of a valid executor configuration file.</p> |
| |
| <p>For more information on this feature please see the custom executors <a href="../../features/custom-executors/">documentation</a>.</p> |
| |
| <h2 id="a-note-on-increasing-executor-overhead">A note on increasing executor overhead</h2> |
| |
| <p>Increasing executor overhead on an existing cluster, whether it be for custom executors or for Thermos, |
| will result in degraded preemption performance until all task which began life with the previous |
| executor configuration with less overhead are preempted/restarted.</p> |
| |
| <h2 id="controlling-mtta-via-update-affinity">Controlling MTTA via Update Affinity</h2> |
| |
| <p>When there is high resource contention in your cluster you may experience noticably elevated job update |
| times, as well as high task churn across the cluster. This is due to Aurora’s first-fit scheduling |
| algorithm. To alleviate this, you can enable update affinity where the Scheduler will make a best-effort |
| attempt to reuse the same agent for the updated task (so long as the resources for the job are not being |
| increased).</p> |
| |
| <p>To enable this in the Scheduler, you can set the following options:</p> |
| <pre class="highlight plaintext"><code>--enable_update_affinity=true |
| --update_affinity_reservation_hold_time=3mins |
| </code></pre> |
| |
| <p>You will need to tune the hold time to match the behavior you see in your cluster. If you have extremely |
| high update throughput, you might have to extend it as processing updates could easily add significant |
| delays between scheduling attempts. You may also have to tune scheduling parameters to achieve the |
| throughput you need in your cluster. Some relevant settings (with defaults) are:</p> |
| <pre class="highlight plaintext"><code>--max_schedule_attempts_per_sec=40 |
| --initial_schedule_penalty=1secs |
| --max_schedule_penalty=1mins |
| --scheduling_max_batch_size=3 |
| --max_tasks_per_schedule_attempt=5 |
| </code></pre> |
| |
| <p>There are metrics exposed by the Scheduler which can provide guidance on where the bottleneck is. |
| Example metrics to look at:</p> |
| <pre class="highlight plaintext"><code>- schedule_attempts_blocks (if this number is greater than 0, then task throughput is hitting |
| limits controlled by --max_scheduler_attempts_per_sec) |
| - scheduled_task_penalty_* (metrics around scheduling penalties for tasks, if the numbers here are high |
| then you could have high contention for resources) |
| </code></pre> |
| |
| <p>Most likely you’ll run into limits with the number of update instances that can be processed per minute |
| before you run into any other limits. So if your total work done per minute starts to exceed 2k instances, |
| you may need to extend the update<em>affinity</em>reservation<em>hold</em>time.</p> |
| |
| </div> |
| |
| </div> |
| </div> |
| <div class="container-fluid section-footer buffer"> |
| <div class="container"> |
| <div class="row"> |
| <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> |
| <ul> |
| <li><a href="/downloads/">Downloads</a></li> |
| <li><a href="/community/">Mailing Lists</a></li> |
| <li><a href="http://issues.apache.org/jira/browse/AURORA">Issue Tracking</a></li> |
| <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> |
| </ul> |
| </div> |
| <div class="col-md-2"><h3>The ASF</h3> |
| <ul> |
| <li><a href="http://www.apache.org/licenses/">License</a></li> |
| <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li> |
| <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li> |
| <li><a href="http://www.apache.org/security/">Security</a></li> |
| </ul> |
| </div> |
| <div class="col-md-6"> |
| <p class="disclaimer">© 2014-2017 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/">Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/">Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> |
| </div> |
| </div> |
| </div> |
| |
| </body> |
| </html> |