| <!DOCTYPE html> |
| <html> |
| <head> |
| <meta charset="utf-8"> |
| <title>Apache Mesos - Quota</title> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| <meta property="og:locale" content="en_US"/> |
| <meta property="og:type" content="website"/> |
| <meta property="og:title" content="Apache Mesos"/> |
| <meta property="og:site_name" content="Apache Mesos"/> |
| <meta property="og:url" content="http://mesos.apache.org/"/> |
| <meta property="og:image" content="http://mesos.apache.org/assets/img/mesos_logo_fb_preview.png"/> |
| <meta property="og:description" |
| content="Apache Mesos abstracts resources away from machines, |
| enabling fault-tolerant and elastic distributed systems |
| to easily be built and run effectively."/> |
| |
| <meta name="twitter:card" content="summary"/> |
| <meta name="twitter:site" content="@ApacheMesos"/> |
| <meta name="twitter:title" content="Apache Mesos"/> |
| <meta name="twitter:image" content="http://mesos.apache.org/assets/img/mesos_logo_fb_preview.png"/> |
| <meta name="twitter:description" |
| content="Apache Mesos abstracts resources away from machines, |
| enabling fault-tolerant and elastic distributed systems |
| to easily be built and run effectively."/> |
| |
| <link href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet"> |
| <link rel="alternate" type="application/atom+xml" title="Apache Mesos Blog" href="/blog/feed.xml"> |
| <link href="../../../assets/css/main.css" media="screen" rel="stylesheet" type="text/css" /> |
| |
| |
| |
| <!-- Google Analytics Magic --> |
| <script type="text/javascript"> |
| var _gaq = _gaq || []; |
| _gaq.push(['_setAccount', 'UA-20226872-1']); |
| _gaq.push(['_setDomainName', 'apache.org']); |
| _gaq.push(['_trackPageview']); |
| |
| (function() { |
| var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; |
| ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; |
| var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); |
| })(); |
| </script> |
| |
| </head> |
| <body> |
| <!-- magical breadcrumbs --> |
| <div class="topnav"> |
| <div class="container"> |
| <ul class="breadcrumb"> |
| <li> |
| <div class="dropdown"> |
| <a data-toggle="dropdown" href="#">Apache Software Foundation <span class="caret"></span></a> |
| <ul class="dropdown-menu" role="menu" aria-labelledby="dLabel"> |
| <li><a href="http://www.apache.org">Apache Homepage</a></li> |
| <li><a href="http://www.apache.org/licenses/">License</a></li> |
| <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li> |
| <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li> |
| <li><a href="http://www.apache.org/security/">Security</a></li> |
| </ul> |
| </div> |
| </li> |
| |
| <li><a href="http://mesos.apache.org">Apache Mesos</a></li> |
| |
| |
| <li><a href="/documentation |
| /">Documentation |
| </a></li> |
| |
| |
| </ul><!-- /.breadcrumb --> |
| </div><!-- /.container --> |
| </div><!-- /.topnav --> |
| |
| <!-- navbar excitement --> |
| <div class="navbar navbar-default navbar-static-top" role="navigation"> |
| <div class="container"> |
| <div class="navbar-header"> |
| <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#mesos-menu" aria-expanded="false"> |
| <span class="sr-only">Toggle navigation</span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| <a class="navbar-brand" href="/"><img src="/assets/img/mesos_logo.png" alt="Apache Mesos logo"/></a> |
| </div><!-- /.navbar-header --> |
| |
| <div class="navbar-collapse collapse" id="mesos-menu"> |
| <ul class="nav navbar-nav navbar-right"> |
| <li><a href="/gettingstarted/">Getting Started</a></li> |
| <li><a href="/blog/">Blog</a></li> |
| <li><a href="/documentation/latest/">Documentation</a></li> |
| <li><a href="/downloads/">Downloads</a></li> |
| <li><a href="/community/">Community</a></li> |
| </ul> |
| </div><!-- /#mesos-menu --> |
| </div><!-- /.container --> |
| </div><!-- /.navbar --> |
| |
| <div class="content"> |
| <div class="container"> |
| <div class="row-fluid"> |
| <div class="col-md-4"> |
| <h4>If you're new to Mesos</h4> |
| <p>See the <a href="/gettingstarted/">getting started</a> page for more |
| information about downloading, building, and deploying Mesos.</p> |
| |
| <h4>If you'd like to get involved or you're looking for support</h4> |
| <p>See our <a href="/community/">community</a> page for more details.</p> |
| </div> |
| <div class="col-md-8"> |
| <h1>Quota</h1> |
| |
| <p>A problem when running multiple frameworks on Mesos is that the default |
| <a href="https://www.cs.berkeley.edu/~alig/papers/drf.pdf">wDRF allocator</a> may offer |
| resources to frameworks even though they are beyond their fair share (e.g., when |
| no other framework currently accepts these resources). There is no mechanism to |
| lay away resources for future consumption in an entire cluster. Even though |
| <a href="/documentation/latest/./reservation/">dynamic reservations</a> allow both operators and frameworks |
| to dynamically reserve resources on particular Mesos agents, it does not solve |
| the above problem because any individual agents might fail.</p> |
| |
| <p>Mesos 0.27 introduced support for <em>quotas</em>, which are a mechanism for |
| guaranteeing that a <a href="/documentation/latest/./roles/">role</a> will receive a certain minimum amount of |
| resources. A quota specifies a <em>minimum</em> amount of resources that the role is |
| guaranteed to receive (unless the total resources in the cluster are less than |
| the configured quota resources, which often indicates a misconfiguration). Of |
| course, the role might receive offers for more resources than its configured |
| quota, as dictated by the default Mesos DRF allocation scheme.</p> |
| |
| <p>Quotas are somewhat similar to <a href="/documentation/latest/./reservation/">reserved resources</a>, but they |
| have several differences. Most importantly, quota resources are not tied to a |
| particular agent—that is, a quota ensures that a role will receive a certain |
| amount of resources somewhere in the cluster, whereas reservations can be used |
| to assign specific resources at a particular agent to a given role. Quotas can |
| only be configured by operators, via the HTTP endpoint described below; dynamic |
| reservations can be made by frameworks, provided the framework’s principal is |
| <a href="/documentation/latest/./authorization/">authorized</a> to make reservations.</p> |
| |
| <p>Note that reserved resources are considered to satisfy a role’s quota. For |
| example, if a role has been assigned a quota of 4 CPUs and also has 2 reserved |
| resources at a particular agent, the role has a minimum guarantee of 4 CPUs, not |
| 6.</p> |
| |
| <h2>Terminology</h2> |
| |
| <p>For the purpose of this document, an “Operator” is a person, tool, or script |
| that manages the Mesos cluster.</p> |
| |
| <p>In computer science, a “quota” usually refers to one of the following:</p> |
| |
| <ul> |
| <li>A minimal guarantee.</li> |
| <li>A maximal limit.</li> |
| <li>A pair of both.</li> |
| </ul> |
| |
| |
| <p>In Mesos, a quota is a <strong>guaranteed</strong> resource allocation that a role may rely |
| on; in other words, a minimum share a role is entitled to receive.</p> |
| |
| <h2>Motivation and Limitations</h2> |
| |
| <p>Consider the following scenarios in a Mesos cluster to better understand which |
| use-cases are supported by quota, and which are not.</p> |
| |
| <h3>Scenario 1: Greedy Framework</h3> |
| |
| <p>There are two frameworks in a cluster, each running in a separate role with |
| equal weights: framework fA in role rA and framework fB in role rB. There is a |
| single resource available in the cluster: 100 CPUs. fA consumes 10 CPUs and is |
| idle (declines resource offers), while fB is greedy and accepts all offers it gets, |
| hogging the remaining 90 CPUs. Without quota, though fA’s fair share is 50 CPUs |
| it will not be able to make use of additional the 40 CPUs until some of fB’s |
| tasks terminate.</p> |
| |
| <h3>Scenario 2: Resources for a new Framework</h3> |
| |
| <p>A greedy framework fB in role rB is currently the only framework in the cluster |
| and it uses all available resources—100 CPUs. If a new framework fA in role rA |
| joins the cluster, it will not receive its fair share of the cluster resources |
| (50 CPUs) until some of fB’s tasks terminate.</p> |
| |
| <p>To deal with Scenario 2, quota by itself is not a sufficient solution as it |
| would be set after fB has started using all resources. Instead Scenario 2 |
| requires either always keeping a pool of resources which are not offered or |
| introducing preemption for running tasks.</p> |
| |
| <h2>Operator HTTP Endpoint</h2> |
| |
| <p>The master <a href="/documentation/latest/./endpoints/master/quota/">/quota</a> HTTP endpoint enables operators |
| to configure quotas. The endpoint currently offers a REST-like interface and |
| supports the following operations:</p> |
| |
| <ul> |
| <li><a href="#setRequest">Setting</a> a new quota with POST.</li> |
| <li><a href="#removeRequest">Removing</a> an existing quota with DELETE.</li> |
| <li><a href="#statusRequest">Querying</a> the currently set quota with GET.</li> |
| </ul> |
| |
| |
| <p>Currently it is not possible to update previously configured quotas. This means |
| in order to update a quota for a given role, the operator has to remove the |
| existing quota and then set a new one.</p> |
| |
| <p>The endpoint can optionally use authentication and authorization. See |
| <a href="/documentation/latest/./authentication/">authentication guide</a> for details.</p> |
| |
| <p><a name="setRequest"></a></p> |
| |
| <h3>Set</h3> |
| |
| <p>The operator can set a new quota by sending an HTTP POST request to the <code>/quota</code> |
| endpoint.</p> |
| |
| <p>An example request to the quota endpoint could look like this (using the JSON |
| file below):</p> |
| |
| <pre><code> $ curl -d @quota.json -X POST http://<master-ip>:<port>/quota |
| </code></pre> |
| |
| <p>For example to set a quota of 12 CPUs and 6144 MB of RAM for <code>role1</code> the operator |
| can use the following <code>quota.json</code>:</p> |
| |
| <pre><code> { |
| "role": "role1", |
| "guarantee": [ |
| { |
| "name": "cpus", |
| "type": "SCALAR", |
| "scalar": { "value": 12 } |
| }, |
| { |
| "name": "mem", |
| "type": "SCALAR", |
| "scalar": { "value": 6144 } |
| } |
| ] |
| } |
| </code></pre> |
| |
| <p>A set request is only valid for roles for which no quota is currently set. |
| However if the master is configured without an explicit |
| <a href="/documentation/latest/./roles/">role whitelist</a>, a set request can introduce new roles.</p> |
| |
| <p>In order to bypass the <a href="#capacityHeuristic">capacity heuristic</a> check the |
| operator should set an optional <code>force</code> field:</p> |
| |
| <pre><code> { |
| "force": true, |
| "role": "role1", |
| ... |
| } |
| </code></pre> |
| |
| <p>The operator will receive one of the following HTTP response codes:</p> |
| |
| <ul> |
| <li><code>200 OK</code>: Success (the set request was successful).</li> |
| <li><code>400 BadRequest</code>: Invalid arguments (e.g., quota for role exists, invalid |
| JSON, non-scalar resources included).</li> |
| <li><code>401 Unauthorized</code>: Unauthenticated request.</li> |
| <li><code>403 Forbidden</code>: Unauthorized request.</li> |
| <li><code>409 Conflict</code>: The capacity heuristic check failed due to insufficient |
| resources.</li> |
| </ul> |
| |
| |
| <p><a name="removeRequest"></a></p> |
| |
| <h3>Remove</h3> |
| |
| <p>The operator can remove a previously set quota by sending an HTTP DELETE request |
| to the <code>/quota/<role></code> endpoint. For example the following command removes |
| a previously set quota for <code>role1</code>:</p> |
| |
| <pre><code> $ curl -X DELETE http://<master-ip>:<port>/quota/role1 |
| </code></pre> |
| |
| <p>The operator will receive one of the following HTTP response codes:</p> |
| |
| <ul> |
| <li><code>200 OK</code>: Success (the remove request was successful).</li> |
| <li><code>400 BadRequest</code>: Invalid arguments (e.g., removing a quota for a role which |
| does not have any quota set).</li> |
| <li><code>401 Unauthorized</code>: Unauthenticated request.</li> |
| <li><code>403 Forbidden</code>: Unauthorized request.</li> |
| </ul> |
| |
| |
| <p><a name="statusRequest"></a></p> |
| |
| <h3>Status</h3> |
| |
| <p>The operator can query the configured quotas by sending an HTTP GET request |
| to the <code>/quota</code> endpoint.</p> |
| |
| <pre><code> $ curl -X GET http://<master-ip>:<port>/quota |
| </code></pre> |
| |
| <p>The response message body includes a JSON representation of the current quota |
| status for role(s) which principal is authorized to query quota status (if |
| authorization is enabled). For example:</p> |
| |
| <pre><code> { |
| "infos": [ |
| { |
| "role": "role1", |
| "guarantee": [ |
| { |
| "name": "cpus", |
| "role": "*", |
| "type": "SCALAR", |
| "scalar": { "value": 12 } |
| }, |
| { |
| "name": "mem", |
| "role": "*", |
| "type": "SCALAR", |
| "scalar": { "value": 6144 } |
| } |
| ] |
| } |
| ] |
| } |
| </code></pre> |
| |
| <p>The operator will receive one of the following HTTP response codes:</p> |
| |
| <ul> |
| <li><code>200 OK</code>: Success.</li> |
| <li><code>401 Unauthorized</code>: Unauthenticated request.</li> |
| </ul> |
| |
| |
| <p><strong>NOTE:</strong> If the principal is not authorized to query quota status for certain |
| role(s), the result will not include corresponding quota information.</p> |
| |
| <h2>How Does It Work?</h2> |
| |
| <p>There are several stages in the lifetime of a quota issued by operator. First |
| the <a href="#requestProcessing">quota set request is handled by the master</a>, after that |
| the <a href="#allocatorEnforcement">allocator enforces the quota</a>. |
| Quotas can be <a href="#removeProcessing">removed</a> by the operator.</p> |
| |
| <p>It is important to understand that the enforcement of quota depends on |
| the allocator being used. A custom allocator could choose to handle quota in its |
| own way or even to ignore quota. |
| <strong>The following section assumes the default |
| <a href="https://www.cs.berkeley.edu/~alig/papers/drf.pdf">wDRF</a> allocator is used.</strong></p> |
| |
| <p>Be aware that setting quota may affect other frameworks in the cluster, |
| because resources will be laid away and not offered to other frameworks. |
| Also note, that quota is only applicable for scalar resources (e.g., it is not |
| possible to set quota for port resources).</p> |
| |
| <p><a name="requestProcessing"></a></p> |
| |
| <h2>Quota Request Processing</h2> |
| |
| <p>When an operator submits a quota set request via the master <code>/quota</code> HTTP |
| endpoint, the following steps are triggered:</p> |
| |
| <ol> |
| <li><a href="/documentation/latest/./authentication/">Authenticate</a> the HTTP request.</li> |
| <li>Parse and validate the request. |
| See <a href="#setRequest">description of possible error codes</a>.</li> |
| <li><a href="/documentation/latest/./authentication/">Authorize</a> the HTTP request if authorization is enabled.</li> |
| <li>Run the <a href="#capacityHeuristic">capacity heuristic</a> if not disabled by |
| <a href="#setRequest">the <code>force</code> flag</a>.</li> |
| <li>Reliably store quota. See <a href="#failover">details on failover recovery</a>.</li> |
| <li><a href="#rescindOffers">Rescind outstanding offers</a>.</li> |
| </ol> |
| |
| |
| <p><a name="removeProcessing"></a> |
| The quota remove request processing is simpler and triggers the following steps:</p> |
| |
| <ol> |
| <li><a href="/documentation/latest/./authentication/">Authenticate</a> the HTTP request.</li> |
| <li>Validate the request. |
| See <a href="#removeRequest">description of potential error codes</a>.</li> |
| <li><a href="/documentation/latest/./authentication/">Authorize</a> the HTTP request if authorization is enabled.</li> |
| <li>Reliably remove quota.</li> |
| </ol> |
| |
| |
| <p><a name="capacityHeuristic"></a></p> |
| |
| <h3>Capacity Heuristic Check</h3> |
| |
| <p>Misconfigured quota can render a cluster into a state where no offers are made |
| to any frameworks. For example imagine an operator setting quota 1000 CPUs |
| for a role <code>prosuction</code> (note the typo) in a cluster with a total capacity of |
| 100 CPUs. In that case after the quota is accepted by the master, no offers will |
| be made to any framework in any actual role (including <code>production</code>).</p> |
| |
| <p>In order to prevent such extreme situations, the Mesos Master employs a capacity |
| heuristic check that ensures a quota set request can reasonably be satisfied |
| given the total cluster capacity. This heuristic tests whether the total quota, |
| including the new request, does not exceed the sum of total |
| non-statically-reserved cluster resources, i.e. the following inequality holds:</p> |
| |
| <pre><code> total resources - statically reserved >= total quota + quota request |
| </code></pre> |
| |
| <p>Please be advised that even if there are enough resources at the moment of |
| this check, agents may terminate at any time, rendering the cluster incapable |
| of satisfying the configured quotas.</p> |
| |
| <p>A <a href="#setRequest"><code>force</code> flag</a> can be set to bypass this check. For example, this |
| flag can be useful when the operator would like to configure a quota that |
| exceeds the current cluster capacity, but they know that additional cluster |
| resources will be added shortly.</p> |
| |
| <p><a name="rescindOffers"></a></p> |
| |
| <h3>Rescinding Outstanding Offers</h3> |
| |
| <p>When setting a new quota, the master rescinds outstanding offers. This avoids |
| situations where the quota request cannot be satisfied by the remaining |
| unoffered resources, but there are enough resources tied up in outstanding |
| offers to frameworks that have not accepted them yet. Hence, we rescind |
| outstanding offers with the following rules:</p> |
| |
| <ul> |
| <li>Rescind at least as many resources as there are in the quota request.</li> |
| <li>If at least one offer is to be rescinded from an agent, all offers from this |
| agent are rescinded. This is done in order to make the potential offer bigger, |
| which increases the chances that a quota'ed framework will be able to use the |
| offer.</li> |
| <li>Rescind offers from at least as many agents as there are frameworks in the |
| role for which quota is being set. This enables (but does not guarantee, due |
| to fair sharing) each framework in the role to receive an offer.</li> |
| </ul> |
| |
| |
| <p><a name="allocatorEnforcement"></a></p> |
| |
| <h2>Enforcement by wDRF Allocator</h2> |
| |
| <p>The wDRF allocator first allocates (or lays away if offers are declined) |
| resources to framework in roles with quota set. Once all quotas are |
| satisfied, it proceeds with the standard wDRF for all frameworks.</p> |
| |
| <p><strong>NOTE:</strong> A quota'ed role may not be allocated any unreserved non-revocable |
| resources beyond its quota guarantee. If frameworks in the quota'ed role have |
| not opted for revocable resources, they may stop getting offers once quota for |
| the role is satisfied. In this case setting quota to any value that is less than |
| the role’s fair share may reduce the amount of resources offered to this role.</p> |
| |
| <p><strong>NOTE:</strong> Currently quota guarantee also serves as quota limit, i.e. once quota |
| for the role is satisfied, frameworks in this role will not be offered any |
| resources except those reserved for the role. This behavior aims to mitigate the |
| absence of quota limit and will be changed in future releases.</p> |
| |
| <p>If there are multiple frameworks in a role with quota set, the standard wDRF |
| algorithm determines framework priority inside this role.</p> |
| |
| <p>The default wDRF allocator considers only non-revocable resources as applicable |
| towards quota.</p> |
| |
| <p><a name="failover"></a></p> |
| |
| <h2>Failover</h2> |
| |
| <p>If there is at least one role with quota set, the master failover recovery |
| changes significantly. The reason for this is that during the recovery there is |
| a period of time when not all agents have registered with the Master. Therefore |
| it is impossible to reason about whether all quotas are satisfied.</p> |
| |
| <p>To address this issue, if upon recovery any previously set quota are detected, |
| the allocator enters recovery mode, during which the allocator |
| <em>does not issue offers</em>. The recovery mode—and therefore offer |
| suspension—ends when either:</p> |
| |
| <ul> |
| <li>A certain amount of agents reregister (by default 80% of agents known before |
| the failover), or</li> |
| <li>a timeout expires (by default 10 minutes).</li> |
| </ul> |
| |
| |
| <h2>Current Limitations</h2> |
| |
| <ul> |
| <li>The quota set request does not allow specifying the granularity of the |
| requested resources (e.g., 10 CPUs on a single node).</li> |
| <li>The quota set request does not allow to specify constraints (e.g., 2*5 cpus |
| on disjoint nodes for an HA like setup).</li> |
| <li>Quota is not allowed for the default role ‘*’ (see |
| <a href="https://issues.apache.org/jira/browse/MESOS-3938">MESOS-3938</a>).</li> |
| <li>Currently it is not possible to update previously configured quotas. See |
| <a href="#setRequest">quota set request</a> for details.</li> |
| </ul> |
| |
| |
| </div> |
| </div> |
| |
| </div><!-- /.container --> |
| </div><!-- /.content --> |
| |
| <hr> |
| |
| |
| |
| <!-- footer --> |
| <div class="footer"> |
| <div class="container"> |
| <div class="col-md-4 social-blk"> |
| <span class="social"> |
| <a href="https://twitter.com/ApacheMesos" |
| class="twitter-follow-button" |
| data-show-count="false" data-size="large">Follow @ApacheMesos</a> |
| <script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script> |
| <a href="https://twitter.com/intent/tweet?button_hashtag=mesos" |
| class="twitter-hashtag-button" |
| data-size="large" |
| data-related="ApacheMesos">Tweet #mesos</a> |
| <script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script> |
| </span> |
| </div> |
| |
| <div class="col-md-8 trademark"> |
| <p>© 2012-2017 <a href="http://apache.org">The Apache Software Foundation</a>. |
| Apache Mesos, the Apache feather logo, and the Apache Mesos project logo are trademarks of The Apache Software Foundation. |
| <p> |
| </div> |
| </div><!-- /.container --> |
| </div><!-- /.footer --> |
| |
| <!-- JS --> |
| <script src="//code.jquery.com/jquery-1.11.0.min.js" type="text/javascript"></script> |
| <script src="//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js" type="text/javascript"></script> |
| </body> |
| </html> |