| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>Apache Aurora</title> |
| <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css"> |
| <link href="/assets/css/main.css" rel="stylesheet"> |
| <!-- Analytics --> |
| <script type="text/javascript"> |
| var _gaq = _gaq || []; |
| _gaq.push(['_setAccount', 'UA-45879646-1']); |
| _gaq.push(['_setDomainName', 'apache.org']); |
| _gaq.push(['_trackPageview']); |
| |
| (function() { |
| var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; |
| ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; |
| var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); |
| })(); |
| </script> |
| </head> |
| <body> |
| <div class="container-fluid section-header"> |
| <div class="container"> |
| <div class="nav nav-bar"> |
| <a href="/"><img src="/assets/img/aurora_logo_dkbkg.svg" width="300" alt="Transparent Apache Aurora logo with dark background"/></a> |
| <ul class="nav navbar-nav navbar-right"> |
| <li><a href="/documentation/latest/">Documentation</a></li> |
| <li><a href="/community/">Community</a></li> |
| <li><a href="/downloads/">Downloads</a></li> |
| <li><a href="/blog/">Blog</a></li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| |
| <div class="container-fluid"> |
| <div class="container content"> |
| <div class="col-md-12 documentation"> |
| <h5 class="page-header text-uppercase">Documentation |
| <select onChange="window.location.href='/documentation/' + this.value + '/development/thermos/'" |
| value="0.19.1"> |
| <option value="0.22.0" |
| > |
| 0.22.0 |
| (latest) |
| </option> |
| <option value="0.21.0" |
| > |
| 0.21.0 |
| </option> |
| <option value="0.20.0" |
| > |
| 0.20.0 |
| </option> |
| <option value="0.19.1" |
| selected="selected"> |
| 0.19.1 |
| </option> |
| <option value="0.19.0" |
| > |
| 0.19.0 |
| </option> |
| <option value="0.18.1" |
| > |
| 0.18.1 |
| </option> |
| <option value="0.18.0" |
| > |
| 0.18.0 |
| </option> |
| <option value="0.17.0" |
| > |
| 0.17.0 |
| </option> |
| <option value="0.16.0" |
| > |
| 0.16.0 |
| </option> |
| <option value="0.15.0" |
| > |
| 0.15.0 |
| </option> |
| <option value="0.14.0" |
| > |
| 0.14.0 |
| </option> |
| <option value="0.13.0" |
| > |
| 0.13.0 |
| </option> |
| <option value="0.12.0" |
| > |
| 0.12.0 |
| </option> |
| <option value="0.11.0" |
| > |
| 0.11.0 |
| </option> |
| <option value="0.10.0" |
| > |
| 0.10.0 |
| </option> |
| <option value="0.9.0" |
| > |
| 0.9.0 |
| </option> |
| <option value="0.8.0" |
| > |
| 0.8.0 |
| </option> |
| <option value="0.7.0-incubating" |
| > |
| 0.7.0-incubating |
| </option> |
| <option value="0.6.0-incubating" |
| > |
| 0.6.0-incubating |
| </option> |
| <option value="0.5.0-incubating" |
| > |
| 0.5.0-incubating |
| </option> |
| </select> |
| </h5> |
| <p>The Python components of Aurora are built using <a href="https://pantsbuild.github.io">Pants</a>.</p> |
| |
| <h1 id="python-build-conventions">Python Build Conventions</h1> |
| |
| <p>The Python code is laid out according to the following conventions:</p> |
| |
| <ol> |
| <li><p>1 <code>BUILD</code> per 3rd level directory. For a list of current top-level packages run:</p> |
| <pre class="highlight plaintext"><code>% find src/main/python -maxdepth 3 -mindepth 3 -type d |\ |
| while read dname; do echo $dname |\ |
| sed 's@src/main/python/\(.*\)/\(.*\)/\(.*\).*@\1.\2.\3@'; done |
| </code></pre></li> |
| <li><p>Each <code>BUILD</code> file exports 1 |
| <a href="https://pantsbuild.github.io/build_dictionary.html#bdict_python_library"><code>python_library</code></a> |
| that provides a |
| <a href="https://pantsbuild.github.io/build_dictionary.html#setup_py"><code>setup_py</code></a> |
| containing each |
| <a href="https://pantsbuild.github.io/build_dictionary.html#python_binary"><code>python_binary</code></a> |
| in the <code>BUILD</code> file, named the same as the directory it’s in so that it can be referenced |
| without a ’:’ character. The <code>sources</code> field in the <code>python_library</code> will almost always be |
| <code>rglobs('*.py')</code>.</p></li> |
| <li><p>Other BUILD files may only depend on this single public <code>python_library</code> |
| target. Any other target is considered a private implementation detail and |
| should be prefixed with an <code>_</code>.</p></li> |
| <li><p><code>python_binary</code> targets are always named the same as the exported console script.</p></li> |
| <li><p><code>python_binary</code> targets must have identical <code>dependencies</code> to the <code>python_library</code> exported |
| by the package and must use <code>entry_point</code>.</p> |
| |
| <p>The means a PEX file generated by pants will contain exactly the same files that will be |
| available on the <code>PYTHONPATH</code> in the case of <code>pip install</code> of the corresponding library |
| target. This will help our migration off of Pants in the future.</p></li> |
| </ol> |
| |
| <h2 id="annotated-example-apache-thermos-runner">Annotated example - apache.thermos.runner</h2> |
| <pre class="highlight plaintext"><code>% find src/main/python/apache/thermos/runner |
| src/main/python/apache/thermos/runner |
| src/main/python/apache/thermos/runner/__init__.py |
| src/main/python/apache/thermos/runner/thermos_runner.py |
| src/main/python/apache/thermos/runner/BUILD |
| % cat src/main/python/apache/thermos/runner/BUILD |
| # License boilerplate omitted |
| import os |
| |
| |
| # Private target so that a setup_py can exist without a circular dependency. Only targets within |
| # this file should depend on this. |
| python_library( |
| name = '_runner', |
| # The target covers every python file under this directory and subdirectories. |
| sources = rglobs('*.py'), |
| dependencies = [ |
| '3rdparty/python:twitter.common.app', |
| '3rdparty/python:twitter.common.log', |
| # Source dependencies are always referenced without a ':'. |
| 'src/main/python/apache/thermos/common', |
| 'src/main/python/apache/thermos/config', |
| 'src/main/python/apache/thermos/core', |
| ], |
| ) |
| |
| # Binary target for thermos_runner.pex. Nothing should depend on this - it's only used as an |
| # argument to ./pants binary. |
| python_binary( |
| name = 'thermos_runner', |
| # Use entry_point, not source so the files used here are the same ones tests see. |
| entry_point = 'apache.thermos.bin.thermos_runner', |
| dependencies = [ |
| # Notice that we depend only on the single private target from this BUILD file here. |
| ':_runner', |
| ], |
| ) |
| |
| # The public library that everyone importing the runner symbols uses. |
| # The test targets and any other dependent source code should depend on this. |
| python_library( |
| name = 'runner', |
| dependencies = [ |
| # Again, notice that we depend only on the single private target from this BUILD file here. |
| ':_runner', |
| ], |
| # We always provide a setup_py. This will cause any dependee libraries to automatically |
| # reference this library in their requirements.txt rather than copy the source files into their |
| # sdist. |
| provides = setup_py( |
| # Conventionally named and versioned. |
| name = 'apache.thermos.runner', |
| version = open(os.path.join(get_buildroot(), '.auroraversion')).read().strip().upper(), |
| ).with_binaries({ |
| # Every binary in this file should also be repeated here. |
| # Always use the dict-form of .with_binaries so that commands with dashes in their names are |
| # supported. |
| # The console script name is always the same as the PEX with .pex stripped. |
| 'thermos_runner': ':thermos_runner', |
| }), |
| ) |
| </code></pre> |
| |
| <h1 id="thermos-test-resources">Thermos Test resources</h1> |
| |
| <p>The Aurora source repository and distributions contain several |
| <a href="../../src/test/resources/org/apache/thermos/root/checkpoints">binary files</a> to |
| qualify the backwards-compatibility of thermos with checkpoint data. Since |
| thermos persists state to disk, to be read by the thermos observer), it is important that we have |
| tests that prevent regressions affecting the ability to parse previously-written data.</p> |
| |
| <p>The files included represent persisted checkpoints that exercise different |
| features of thermos. The existing files should not be modified unless |
| we are accepting backwards incompatibility, such as with a major release.</p> |
| |
| <p>It is not practical to write source code to generate these files on the fly, |
| as source would be vulnerable to drift (e.g. due to refactoring) in ways |
| that would undermine the goal of ensuring backwards compatibility.</p> |
| |
| <p>The most common reason to add a new checkpoint file would be to provide |
| coverage for new thermos features that alter the data format. This is |
| accomplished by writing and running a |
| <a href="../../reference/configuration/">job configuration</a> that exercises the feature, and |
| copying the checkpoint file from the sandbox directory, by default this is |
| <code>/var/run/thermos/checkpoints/<aurora task id></code>.</p> |
| |
| </div> |
| |
| </div> |
| </div> |
| <div class="container-fluid section-footer buffer"> |
| <div class="container"> |
| <div class="row"> |
| <div class="col-md-2 col-md-offset-1"><h3>Quick Links</h3> |
| <ul> |
| <li><a href="/downloads/">Downloads</a></li> |
| <li><a href="/community/">Mailing Lists</a></li> |
| <li><a href="http://issues.apache.org/jira/browse/AURORA">Issue Tracking</a></li> |
| <li><a href="/documentation/latest/contributing/">How To Contribute</a></li> |
| </ul> |
| </div> |
| <div class="col-md-2"><h3>The ASF</h3> |
| <ul> |
| <li><a href="http://www.apache.org/licenses/">License</a></li> |
| <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li> |
| <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li> |
| <li><a href="http://www.apache.org/security/">Security</a></li> |
| </ul> |
| </div> |
| <div class="col-md-6"> |
| <p class="disclaimer">© 2014-2017 <a href="http://www.apache.org/">Apache Software Foundation</a>. Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>. The <a href="https://www.flickr.com/photos/trondk/12706051375/">Aurora Borealis IX photo</a> displayed on the homepage is available under a <a href="https://creativecommons.org/licenses/by-nc-nd/2.0/">Creative Commons BY-NC-ND 2.0 license</a>. Apache, Apache Aurora, and the Apache feather logo are trademarks of The Apache Software Foundation.</p> |
| </div> |
| </div> |
| </div> |
| |
| </body> |
| </html> |
