| <!DOCTYPE html> |
| <html> |
| |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| |
| <link rel="stylesheet" href="/css/main.css"> |
| <link rel="stylesheet" href="/css/font-awesome.min.css"> |
| |
| <link rel="shortcut icon" href="/favicon.ico?1"> |
| <!-- Begin Jekyll SEO tag v2.3.0 --> |
| <title>Runtime Design | Nemo</title> |
| <meta property="og:title" content="Runtime Design" /> |
| <meta property="og:locale" content="en_US" /> |
| <meta name="description" content="Receiving a Job from the Nemo Compiler" /> |
| <meta property="og:description" content="Receiving a Job from the Nemo Compiler" /> |
| <link rel="canonical" href="http://nemo.apache.org//docs/runtime_design/" /> |
| <meta property="og:url" content="http://nemo.apache.org//docs/runtime_design/" /> |
| <meta property="og:site_name" content="Nemo" /> |
| <meta property="og:type" content="article" /> |
| <meta property="article:published_time" content="2018-11-12T23:54:18+09:00" /> |
| <script type="application/ld+json"> |
| {"publisher":null,"author":null,"description":"Receiving a Job from the Nemo Compiler","name":null,"@type":"WebPage","headline":"Runtime Design","dateModified":"2018-11-12T23:54:18+09:00","datePublished":"2018-11-12T23:54:18+09:00","url":"http://nemo.apache.org//docs/runtime_design/","sameAs":null,"mainEntityOfPage":null,"image":null,"@context":"http://schema.org"}</script> |
| <!-- End Jekyll SEO tag --> |
| |
| |
| <link rel="canonical" href="http://nemo.apache.org//docs/runtime_design/"> |
| <link rel="alternate" type="application/rss+xml" title="Nemo" href="http://nemo.apache.org//feed.xml" /> |
| </head> |
| |
| |
| <body> |
| |
| <nav class="navbar navbar-default navbar-fixed-top"> |
| <div class="container navbar-container"> |
| <div class="navbar-header"> |
| <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar"> |
| <span class="sr-only">Toggle navigation</span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| <a class="navbar-brand" href="/"> |
| <span><img src="/img/nemo-logo.png"></span> |
| </a> |
| </div> |
| <div id="navbar" class="collapse navbar-collapse"> |
| <ul class="nav navbar-nav"> |
| <li class="active" ><a href="/docs/home/">Docs</a></li> |
| <li ><a href="/apidocs">APIs</a></li> |
| <li ><a href="/pages/downloads">Downloads</a></li> |
| <li ><a href="/pages/talks">Talks</a></li> |
| <li ><a href="/pages/team">Team</a></li> |
| <li ><a href="/pages/license">License</a></li> |
| <li ><a href="/blog/2018/03/23/shuffle-on-nemo/">Blog</a></li> |
| </ul> |
| <div class="navbar-right"> |
| <form class="navbar-form navbar-left"> |
| <div class="form-group has-feedback"> |
| <input id="search-box" type="text" class="form-control" placeholder="Search..."> |
| <i class="fa fa-search form-control-feedback"></i> |
| </div> |
| </form> |
| <ul class="nav navbar-nav"> |
| <li><a href="https://github.com/apache/incubator-nemo-website"><i class="fa fa-github" aria-hidden="true"></i></a></li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| </nav> |
| |
| |
| <div class="page-content"> |
| <div class="wrapper"> |
| <div class="container"> |
| <div class="row"> |
| <div class="col-md-4"> |
| <div class="panel-group" id="accordion" role="tablist" aria-multiselectable="true"> |
| |
| <div class="panel panel-default"> |
| <div class="panel-heading"> |
| <h4 class="panel-title"> |
| <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapse-1" aria-expanded="false" aria-controls="collapse-1"> |
| Getting Started |
| </a> |
| </h4> |
| </div> |
| <div id="collapse-1" class="panel-collapse collapse" role="tabpanel" aria-label="Side Navigation"> |
| <div class="list-group"> |
| |
| |
| |
| <a class="list-group-item " href="/docs/home/">Overview</a> |
| |
| |
| |
| <a class="list-group-item " href="/docs/getting_started/">Getting Started</a> |
| |
| </div> |
| </div> |
| </div> |
| |
| <div class="panel panel-default"> |
| <div class="panel-heading"> |
| <h4 class="panel-title"> |
| <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapse-2" aria-expanded="false" aria-controls="collapse-2"> |
| Optimizations |
| </a> |
| </h4> |
| </div> |
| <div id="collapse-2" class="panel-collapse collapse" role="tabpanel" aria-label="Side Navigation"> |
| <div class="list-group"> |
| |
| |
| |
| <a class="list-group-item " href="/docs/ir/">Nemo Intermediate Representation (IR)</a> |
| |
| |
| |
| <a class="list-group-item " href="/docs/passes_and_policies/">Passes and Policies</a> |
| |
| </div> |
| </div> |
| </div> |
| |
| <div class="panel panel-default"> |
| <div class="panel-heading"> |
| <h4 class="panel-title"> |
| <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapse-3" aria-expanded="false" aria-controls="collapse-3"> |
| System Designs |
| </a> |
| </h4> |
| </div> |
| <div id="collapse-3" class="panel-collapse collapse" role="tabpanel" aria-label="Side Navigation"> |
| <div class="list-group"> |
| |
| |
| |
| <a class="list-group-item " href="/docs/compiler_design/">Compiler Design</a> |
| |
| |
| |
| <a class="list-group-item active" href="/docs/runtime_design/">Runtime Design</a> |
| |
| </div> |
| </div> |
| </div> |
| |
| </div> |
| |
| </div> |
| |
| <div class="col-md-8"> |
| <h1>Runtime Design</h1> |
| <div id="markdown-content-container"><h3 id="receiving-a-job-from-the-nemo-compiler">Receiving a Job from the Nemo Compiler</h3> |
| |
| <p>After the compiler goes through a set of passes for optimization, the optimized Nemo IR is translated into into a |
| physical form for the execution runtime to execute. This involves translations like expanding an operator annotated |
| with parallelism in Nemo IR to the desired number of tasks and connecting the tasks according to the data communication |
| patterns annotated on the IR edges. Physical execution plan is also in the form of a DAG, with the same values annotated |
| for execution properties as the given IR DAG if necessary. Nemo IR DAG and physical execution plan can be translated |
| from one another by sharing the identifiers.</p> |
| |
| <h3 id="runtime-architecture">Runtime Architecture</h3> |
| <p>The Nemo runtime consists of a <em>RuntimeMaster</em> and multiple <em>Executors</em>. |
| <em>RuntimeMaster</em> takes the submitted physical execution plan and schedules each <em>TaskGroup</em> to <em>Executor</em> for execution.</p> |
| |
| <p>The figure below shows the Nemo runtime’s overall architecture. |
| Our runtime’s components can be broken down into two parts, the processing backbone and the extensible modules.</p> |
| |
| <p>The processing backbone illustrated by the blue double stroked boxes in the figure below, |
| implements the inherent and basic code that must be executed for all Nemo jobs |
| (and potentially all data processing jobs). |
| The code includes references to the flexible and extensible data structures |
| representing our execution properties. |
| For example, “executor placement” is mainly controlled in ContainerManager as an extensible map.</p> |
| |
| <p>The extensible modules illustrated by the red dashed boxes in the figure below, |
| are interfaces which users can implement in order to get them to behave the way the user wants. |
| Each interface has been transparently integrated with other components of the runtime.</p> |
| |
| <p><img src="/assets/runtime_arch.png" alt="image" /></p> |
| |
| <h3 id="dictionary">Dictionary</h3> |
| <ul> |
| <li>Stage: A unit of execution the runtime uses for scheduling the job.</li> |
| <li>TaskGroup: A computation unit composed of one or more tasks that can be computed in a single executor.</li> |
| <li>Block: The unit of data output by a single task.</li> |
| <li>Partition: A block consists of one or more partitions, depending on the <em>Partitioner</em> choice.</li> |
| </ul> |
| |
| </div> |
| <p class="text-center"> |
| <br /> |
| <a target="_blank" href="https://github.com/apache/incubator-nemo-website/tree/asf-site/_docs/designs/runtime_design.md" class="btn btn-default btn-sm githubEditButton" role="button"> |
| <i class="fa fa-pencil"></i> Improve this page |
| </a> |
| </p> |
| <hr> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <ul class="pager"> |
| |
| |
| |
| <li class="previous"> |
| <a href="/docs/compiler_design/"> |
| <span aria-hidden="true">←</span> Previous |
| </a> |
| </li> |
| |
| |
| |
| <li class="next disabled"> |
| <a> |
| Next <span aria-hidden="true">→</span> |
| </a> |
| </li> |
| |
| </ul> |
| <div class="clear"></div> |
| |
| |
| </div> |
| |
| </div> |
| </div> |
| |
| </div> |
| </div> |
| |
| <footer class="footer"> |
| <div class="container"> |
| |
| <p class="text-center"> |
| Nemo 2018 | |
| Powered by <a href="https://github.com/aksakalli/jekyll-doc-theme">Jekyll Doc Theme</a> |
| </p> |
| <!-- <p class="text-muted">Place sticky footer content here.</p> --> |
| </div> |
| </footer> |
| |
| <script> |
| var baseurl = '' |
| </script> |
| <script src="//code.jquery.com/jquery-1.10.2.min.js"></script> |
| <script src="/js/bootstrap.min.js "></script> |
| <script src="/js/typeahead.bundle.min.js "></script> |
| |
| <script src="/js/main.js "></script> |
| |
| </body> |
| |
| </html> |