| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>Apache Aurora</title> |
| <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css"> |
| <link href="/assets/css/main.css" rel="stylesheet"> |
| <!-- Analytics --> |
| <script type="text/javascript"> |
| var _gaq = _gaq || []; |
| _gaq.push(['_setAccount', 'UA-45879646-1']); |
| _gaq.push(['_setDomainName', 'apache.org']); |
| _gaq.push(['_trackPageview']); |
| |
| (function() { |
| var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; |
| ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; |
| var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); |
| })(); |
| </script> |
| </head> |
| <body> |
| <div class="container-fluid section-header"> |
| <div class="container"> |
| <div class="nav nav-bar"> |
| <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> |
| <ul class="nav navbar-nav navbar-right"> |
| <li><a href="/documentation/latest/">Documentation</a></li> |
| <li><a href="/community/">Community</a></li> |
| <li><a href="/downloads/">Downloads</a></li> |
| <li><a href="/blog/">Blog</a></li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| |
| <div class="container-fluid"> |
| <div class="container content"> |
| <div class="col-md-12 documentation"> |
| <h5 class="page-header text-uppercase">Documentation |
| <select onChange="window.location.href='/documentation/' + this.value + '/reference/configuration-templating/'" |
| value="0.20.0"> |
| <option value="0.22.0" |
| > |
| 0.22.0 |
| (latest) |
| </option> |
| <option value="0.21.0" |
| > |
| 0.21.0 |
| </option> |
| <option value="0.20.0" |
| selected="selected"> |
| 0.20.0 |
| </option> |
| <option value="0.19.1" |
| > |
| 0.19.1 |
| </option> |
| <option value="0.19.0" |
| > |
| 0.19.0 |
| </option> |
| <option value="0.18.1" |
| > |
| 0.18.1 |
| </option> |
| <option value="0.18.0" |
| > |
| 0.18.0 |
| </option> |
| <option value="0.17.0" |
| > |
| 0.17.0 |
| </option> |
| <option value="0.16.0" |
| > |
| 0.16.0 |
| </option> |
| <option value="0.15.0" |
| > |
| 0.15.0 |
| </option> |
| <option value="0.14.0" |
| > |
| 0.14.0 |
| </option> |
| <option value="0.13.0" |
| > |
| 0.13.0 |
| </option> |
| <option value="0.12.0" |
| > |
| 0.12.0 |
| </option> |
| <option value="0.11.0" |
| > |
| 0.11.0 |
| </option> |
| <option value="0.10.0" |
| > |
| 0.10.0 |
| </option> |
| <option value="0.9.0" |
| > |
| 0.9.0 |
| </option> |
| <option value="0.8.0" |
| > |
| 0.8.0 |
| </option> |
| <option value="0.7.0-incubating" |
| > |
| 0.7.0-incubating |
| </option> |
| <option value="0.6.0-incubating" |
| > |
| 0.6.0-incubating |
| </option> |
| <option value="0.5.0-incubating" |
| > |
| 0.5.0-incubating |
| </option> |
| </select> |
| </h5> |
| <h1 id="aurora-configuration-templating">Aurora Configuration Templating</h1> |
| |
| <p>The <code>.aurora</code> file format is just Python. However, <code>Job</code>, <code>Task</code>, |
| <code>Process</code>, and other classes are defined by a templating library called |
| <em>Pystachio</em>, a powerful tool for configuration specification and reuse.</p> |
| |
| <p><a href="../configuration/">Aurora Configuration Reference</a> |
| has a full reference of all Aurora/Thermos defined Pystachio objects.</p> |
| |
| <p>When writing your <code>.aurora</code> file, you may use any Pystachio datatypes, as |
| well as any objects shown in the <em>Aurora+Thermos Configuration |
| Reference</em> without <code>import</code> statements - the Aurora config loader |
| injects them automatically. Other than that the <code>.aurora</code> format |
| works like any other Python script.</p> |
| |
| <h2 id="templating-1-binding-in-pystachio">Templating 1: Binding in Pystachio</h2> |
| |
| <p>Pystachio uses the visually distinctive {{}} to indicate template |
| variables. These are often called “mustache variables” after the |
| similarly appearing variables in the Mustache templating system and |
| because the curly braces resemble mustaches.</p> |
| |
| <p>If you are familiar with the Mustache system, templates in Pystachio |
| have significant differences. They have no nesting, joining, or |
| inheritance semantics. On the other hand, when evaluated, templates |
| are evaluated iteratively, so this affords some level of indirection.</p> |
| |
| <p>Let’s start with the simplest template; text with one |
| variable, in this case <code>name</code>;</p> |
| <pre class="highlight plaintext"><code>Hello {{name}} |
| </code></pre> |
| |
| <p>If we evaluate this as is, we’d get back:</p> |
| <pre class="highlight plaintext"><code>Hello |
| </code></pre> |
| |
| <p>If a template variable doesn’t have a value, when evaluated it’s |
| replaced with nothing. If we add a binding to give it a value:</p> |
| <pre class="highlight json"><code><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb"> </span><span style="color: #000080">"name"</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #d14">"Tom"</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb"> |
| </span></code></pre> |
| |
| <p>We’d get back:</p> |
| <pre class="highlight plaintext"><code>Hello Tom |
| </code></pre> |
| |
| <p>Every Pystachio object has an associated <code>.bind</code> method that can bind |
| values to {{}} variables. Bindings are not immediately evaluated. |
| Instead, they are evaluated only when the interpolated value of the |
| object is necessary, e.g. for performing equality or serializing a |
| message over the wire.</p> |
| |
| <p>Objects with and without mustache templated variables behave |
| differently:</p> |
| <pre class="highlight plaintext"><code>>>> Float(1.5) |
| Float(1.5) |
| |
| >>> Float('{{x}}.5') |
| Float({{x}}.5) |
| |
| >>> Float('{{x}}.5').bind(x = 1) |
| Float(1.5) |
| |
| >>> Float('{{x}}.5').bind(x = 1) == Float(1.5) |
| True |
| |
| >>> contextual_object = String('{{metavar{{number}}}}').bind( |
| ... metavar1 = "first", metavar2 = "second") |
| |
| >>> contextual_object |
| String({{metavar{{number}}}}) |
| |
| >>> contextual_object.bind(number = 1) |
| String(first) |
| |
| >>> contextual_object.bind(number = 2) |
| String(second) |
| </code></pre> |
| |
| <p>You usually bind simple key to value pairs, but you can also bind three |
| other objects: lists, dictionaries, and structurals. These will be |
| described in detail later.</p> |
| |
| <h3 id="structurals-in-pystachio-aurora">Structurals in Pystachio / Aurora</h3> |
| |
| <p>Most Aurora/Thermos users don’t ever (knowingly) interact with <code>String</code>, |
| <code>Float</code>, or <code>Integer</code> Pystashio objects directly. Instead they interact |
| with derived structural (<code>Struct</code>) objects that are collections of |
| fundamental and structural objects. The structural object components are |
| called <em>attributes</em>. Aurora’s most used structural objects are <code>Job</code>, |
| <code>Task</code>, and <code>Process</code>:</p> |
| <pre class="highlight plaintext"><code>class Process(Struct): |
| cmdline = Required(String) |
| name = Required(String) |
| max_failures = Default(Integer, 1) |
| daemon = Default(Boolean, False) |
| ephemeral = Default(Boolean, False) |
| min_duration = Default(Integer, 5) |
| final = Default(Boolean, False) |
| </code></pre> |
| |
| <p>Construct default objects by following the object’s type with (). If you |
| want an attribute to have a value different from its default, include |
| the attribute name and value inside the parentheses.</p> |
| <pre class="highlight plaintext"><code>>>> Process() |
| Process(daemon=False, max_failures=1, ephemeral=False, |
| min_duration=5, final=False) |
| </code></pre> |
| |
| <p>Attribute values can be template variables, which then receive specific |
| values when creating the object.</p> |
| <pre class="highlight plaintext"><code>>>> Process(cmdline = 'echo {{message}}') |
| Process(daemon=False, max_failures=1, ephemeral=False, min_duration=5, |
| cmdline=echo {{message}}, final=False) |
| |
| >>> Process(cmdline = 'echo {{message}}').bind(message = 'hello world') |
| Process(daemon=False, max_failures=1, ephemeral=False, min_duration=5, |
| cmdline=echo hello world, final=False) |
| </code></pre> |
| |
| <p>A powerful binding property is that all of an object’s children inherit its |
| bindings:</p> |
| <pre class="highlight plaintext"><code>>>> List(Process)([ |
| ... Process(name = '{{prefix}}_one'), |
| ... Process(name = '{{prefix}}_two') |
| ... ]).bind(prefix = 'hello') |
| ProcessList( |
| Process(daemon=False, name=hello_one, max_failures=1, ephemeral=False, min_duration=5, final=False), |
| Process(daemon=False, name=hello_two, max_failures=1, ephemeral=False, min_duration=5, final=False) |
| ) |
| </code></pre> |
| |
| <p>Remember that an Aurora Job contains Tasks which contain Processes. A |
| Job level binding is inherited by its Tasks and all their Processes. |
| Similarly a Task level binding is available to that Task and its |
| Processes but is <em>not</em> visible at the Job level (inheritance is a |
| one-way street.)</p> |
| |
| <h4 id="mustaches-within-structurals">Mustaches Within Structurals</h4> |
| |
| <p>When you define a <code>Struct</code> schema, one powerful, but confusing, feature |
| is that all of that structure’s attributes are Mustache variables within |
| the enclosing scope <em>once they have been populated</em>.</p> |
| |
| <p>For example, when <code>Process</code> is defined above, all its attributes such as |
| {{<code>name</code>}}, {{<code>cmdline</code>}}, {{<code>max_failures</code>}} etc., are all immediately |
| defined as Mustache variables, implicitly bound into the <code>Process</code>, and |
| inherit all child objects once they are defined.</p> |
| |
| <p>Thus, you can do the following:</p> |
| <pre class="highlight plaintext"><code>>>> Process(name = "installer", cmdline = "echo {{name}} is running") |
| Process(daemon=False, name=installer, max_failures=1, ephemeral=False, min_duration=5, |
| cmdline=echo installer is running, final=False) |
| </code></pre> |
| |
| <p>WARNING: This binding only takes place in one direction. For example, |
| the following does NOT work and does not set the <code>Process</code> <code>name</code> |
| attribute’s value.</p> |
| <pre class="highlight plaintext"><code>>>> Process().bind(name = "installer") |
| Process(daemon=False, max_failures=1, ephemeral=False, min_duration=5, final=False) |
| </code></pre> |
| |
| <p>The following is also not possible and results in an infinite loop that |
| attempts to resolve <code>Process.name</code>.</p> |
| <pre class="highlight plaintext"><code>>>> Process(name = '{{name}}').bind(name = 'installer') |
| </code></pre> |
| |
| <p>Do not confuse Structural attributes with bound Mustache variables. |
| Attributes are implicitly converted to Mustache variables but not vice |
| versa.</p> |
| |
| <h3 id="templating-2-structurals-are-factories">Templating 2: Structurals Are Factories</h3> |
| |
| <h4 id="a-second-way-of-templating">A Second Way of Templating</h4> |
| |
| <p>A second templating method is both as powerful as the aforementioned and |
| often confused with it. This method is due to automatic conversion of |
| Struct attributes to Mustache variables as described above.</p> |
| |
| <p>Suppose you create a Process object:</p> |
| <pre class="highlight plaintext"><code>>>> p = Process(name = "process_one", cmdline = "echo hello world") |
| |
| >>> p |
| Process(daemon=False, name=process_one, max_failures=1, ephemeral=False, min_duration=5, |
| cmdline=echo hello world, final=False) |
| </code></pre> |
| |
| <p>This <code>Process</code> object, “<code>p</code>”, can be used wherever a <code>Process</code> object is |
| needed. It can also be reused by changing the value(s) of its |
| attribute(s). Here we change its <code>name</code> attribute from <code>process_one</code> to |
| <code>process_two</code>.</p> |
| <pre class="highlight plaintext"><code>>>> p(name = "process_two") |
| Process(daemon=False, name=process_two, max_failures=1, ephemeral=False, min_duration=5, |
| cmdline=echo hello world, final=False) |
| </code></pre> |
| |
| <p>Template creation is a common use for this technique:</p> |
| <pre class="highlight plaintext"><code>>>> Daemon = Process(daemon = True) |
| >>> logrotate = Daemon(name = 'logrotate', cmdline = './logrotate conf/logrotate.conf') |
| >>> mysql = Daemon(name = 'mysql', cmdline = 'bin/mysqld --safe-mode') |
| </code></pre> |
| |
| <h3 id="advanced-binding">Advanced Binding</h3> |
| |
| <p>As described above, <code>.bind()</code> binds simple strings or numbers to |
| Mustache variables. In addition to Structural types formed by combining |
| atomic types, Pystachio has two container types; <code>List</code> and <code>Map</code> which |
| can also be bound via <code>.bind()</code>.</p> |
| |
| <h4 id="bind-syntax">Bind Syntax</h4> |
| |
| <p>The <code>bind()</code> function can take Python dictionaries or <code>kwargs</code> |
| interchangeably (when “<code>kwargs</code>” is in a function definition, <code>kwargs</code> |
| receives a Python dictionary containing all keyword arguments after the |
| formal parameter list).</p> |
| <pre class="highlight plaintext"><code>>>> String('{{foo}}').bind(foo = 'bar') == String('{{foo}}').bind({'foo': 'bar'}) |
| True |
| </code></pre> |
| |
| <p>Bindings done “closer” to the object in question take precedence:</p> |
| <pre class="highlight plaintext"><code>>>> p = Process(name = '{{context}}_process') |
| >>> t = Task().bind(context = 'global') |
| >>> t(processes = [p, p.bind(context = 'local')]) |
| Task(processes=ProcessList( |
| Process(daemon=False, name=global_process, max_failures=1, ephemeral=False, final=False, |
| min_duration=5), |
| Process(daemon=False, name=local_process, max_failures=1, ephemeral=False, final=False, |
| min_duration=5) |
| )) |
| </code></pre> |
| |
| <h4 id="binding-complex-objects">Binding Complex Objects</h4> |
| |
| <h5 id="lists">Lists</h5> |
| <pre class="highlight plaintext"><code>>>> fibonacci = List(Integer)([1, 1, 2, 3, 5, 8, 13]) |
| >>> String('{{fib[4]}}').bind(fib = fibonacci) |
| String(5) |
| </code></pre> |
| |
| <h5 id="maps">Maps</h5> |
| <pre class="highlight plaintext"><code>>>> first_names = Map(String, String)({'Kent': 'Clark', 'Wayne': 'Bruce', 'Prince': 'Diana'}) |
| >>> String('{{first[Kent]}}').bind(first = first_names) |
| String(Clark) |
| </code></pre> |
| |
| <h5 id="structurals">Structurals</h5> |
| <pre class="highlight plaintext"><code>>>> String('{{p.cmdline}}').bind(p = Process(cmdline = "echo hello world")) |
| String(echo hello world) |
| </code></pre> |
| |
| <h3 id="structural-binding">Structural Binding</h3> |
| |
| <p>Use structural templates when binding more than two or three individual |
| values at the Job or Task level. For fewer than two or three, standard |
| key to string binding is sufficient.</p> |
| |
| <p>Structural binding is a very powerful pattern and is most useful in |
| Aurora/Thermos for doing Structural configuration. For example, you can |
| define a job profile. The following profile uses <code>HDFS</code>, the Hadoop |
| Distributed File System, to designate a file’s location. <code>HDFS</code> does |
| not come with Aurora, so you’ll need to either install it separately |
| or change the way the dataset is designated.</p> |
| <pre class="highlight plaintext"><code>class Profile(Struct): |
| version = Required(String) |
| environment = Required(String) |
| dataset = Default(String, hdfs://home/aurora/data/{{environment}}') |
| |
| PRODUCTION = Profile(version = 'live', environment = 'prod') |
| DEVEL = Profile(version = 'latest', |
| environment = 'devel', |
| dataset = 'hdfs://home/aurora/data/test') |
| TEST = Profile(version = 'latest', environment = 'test') |
| |
| JOB_TEMPLATE = Job( |
| name = 'application', |
| role = 'myteam', |
| cluster = 'cluster1', |
| environment = '{{profile.environment}}', |
| task = SequentialTask( |
| name = 'task', |
| resources = Resources(cpu = 2, ram = 4*GB, disk = 8*GB), |
| processes = [ |
| Process(name = 'main', cmdline = 'java -jar application.jar -hdfsPath |
| {{profile.dataset}}') |
| ] |
| ) |
| ) |
| |
| jobs = [ |
| JOB_TEMPLATE(instances = 100).bind(profile = PRODUCTION), |
| JOB_TEMPLATE.bind(profile = DEVEL), |
| JOB_TEMPLATE.bind(profile = TEST), |
| ] |
| </code></pre> |
| |
| <p>In this case, a custom structural “Profile” is created to self-document |
| the configuration to some degree. This also allows some schema |
| “type-checking”, and for default self-substitution, e.g. in |
| <code>Profile.dataset</code> above.</p> |
| |
| <p>So rather than a <code>.bind()</code> with a half-dozen substituted variables, you |
| can bind a single object that has sensible defaults stored in a single |
| place.</p> |
| |
| </div> |
| |
| </div> |
| </div> |
| <div class="container-fluid section-footer buffer"> |
| <div class="container"> |
| <div class="row"> |
| <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> |
| <ul> |
| <li><a href="/downloads/">Downloads</a></li> |
| <li><a href="/community/">Mailing Lists</a></li> |
| <li><a href="http://issues.apache.org/jira/browse/AURORA">Issue Tracking</a></li> |
| <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> |
| </ul> |
| </div> |
| <div class="col-md-2"><h3>The ASF</h3> |
| <ul> |
| <li><a href="http://www.apache.org/licenses/">License</a></li> |
| <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li> |
| <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li> |
| <li><a href="http://www.apache.org/security/">Security</a></li> |
| </ul> |
| </div> |
| <div class="col-md-6"> |
| <p class="disclaimer">© 2014-2017 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/">Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/">Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> |
| </div> |
| </div> |
| </div> |
| |
| </body> |
| </html> |
