| <!DOCTYPE html> |
| <html lang="en-US"> |
| |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| |
| <link rel="stylesheet" href="/assets/css/custom.css"> |
| <link rel="stylesheet" href="/assets/css/main.css"> |
| <link rel="stylesheet" href="/assets/css/font-awesome.min.css"> |
| |
| <link rel="shortcut icon" href="/favicon.ico?1"> |
| <!-- Begin Jekyll SEO tag v2.8.0 --> |
| <title>Nemo Intermediate Representation (IR) | Nemo</title> |
| <meta name="generator" content="Jekyll v3.9.2" /> |
| <meta property="og:title" content="Nemo Intermediate Representation (IR)" /> |
| <meta property="og:locale" content="en_US" /> |
| <meta name="description" content="Overview" /> |
| <meta property="og:description" content="Overview" /> |
| <link rel="canonical" href="http://nemo.apache.org//docs/ir/" /> |
| <meta property="og:url" content="http://nemo.apache.org//docs/ir/" /> |
| <meta property="og:site_name" content="Nemo" /> |
| <meta property="og:type" content="article" /> |
| <meta property="article:published_time" content="2022-09-10T00:38:05+09:00" /> |
| <meta name="twitter:card" content="summary" /> |
| <meta property="twitter:title" content="Nemo Intermediate Representation (IR)" /> |
| <script type="application/ld+json"> |
| {"@context":"https://schema.org","@type":"WebPage","dateModified":"2022-09-10T00:38:05+09:00","datePublished":"2022-09-10T00:38:05+09:00","description":"Overview","headline":"Nemo Intermediate Representation (IR)","url":"http://nemo.apache.org//docs/ir/"}</script> |
| <!-- End Jekyll SEO tag --> |
| |
| |
| <link rel="canonical" href="http://nemo.apache.org//docs/ir/"> |
| <link rel="alternate" type="application/rss+xml" title="Nemo" href="http://nemo.apache.org//feed.xml" /> |
| </head> |
| |
| |
| <body> |
| |
| <nav class="navbar navbar-default navbar-fixed-top"> |
| <div class="container navbar-container"> |
| <div class="navbar-header"> |
| <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar"> |
| <span class="sr-only">Toggle navigation</span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| <a class="navbar-brand" href="/"> |
| <span><img src="/assets/img/nemo-logo.png" alt="Logo"></span> |
| </a> |
| </div> |
| <div id="navbar" class="collapse navbar-collapse"> |
| <ul class="nav navbar-nav"> |
| <li class="active" ><a href="/docs/home/">Docs</a></li> |
| <li ><a href="/apidocs">APIs</a></li> |
| <li ><a href="/pages/downloads">Downloads</a></li> |
| <li ><a href="/pages/talks">Talks</a></li> |
| <li ><a href="/pages/team">Team</a></li> |
| <li ><a href="/pages/license">License</a></li> |
| <li ><a href="/blog/2022/09/10/release-note-0.4/">Blog</a></li> |
| </ul> |
| <div class="navbar-right"> |
| <form class="navbar-form navbar-left"> |
| <div class="form-group has-feedback"> |
| <input id="search-box" type="search" class="form-control" placeholder="Search..."> |
| <i class="fa fa-search form-control-feedback"></i> |
| </div> |
| </form> |
| <ul class="nav navbar-nav"> |
| <li><a href="https://github.com/apache/incubator-nemo"><i class="fa fa-github" aria-hidden="true"></i></a></li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| </nav> |
| |
| |
| <div class="page-content"> |
| <div class="wrapper"> |
| <div class="container"> |
| <div class="row"> |
| <div class="col-md-4"> |
| <div class="panel-group" id="accordion" role="tablist" aria-multiselectable="true"> |
| |
| <div class="panel panel-default"> |
| <div class="panel-heading"> |
| <h4 class="panel-title"> |
| <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapse-1" aria-expanded="false" aria-controls="collapse-1"> |
| Getting Started |
| </a> |
| </h4> |
| </div> |
| |
| |
| <div class="panel-collapse collapse" id="collapse-1" role="tabpanel" aria-label="Side Navigation"> |
| <div class="list-group"> |
| |
| |
| |
| <a class="list-group-item " href="/docs/home/">Overview</a> |
| |
| |
| |
| <a class="list-group-item " href="/docs/getting_started/">Getting Started</a> |
| |
| </div> |
| </div> |
| </div> |
| |
| <div class="panel panel-default"> |
| <div class="panel-heading"> |
| <h4 class="panel-title"> |
| <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapse-2" aria-expanded="false" aria-controls="collapse-2"> |
| Optimizations |
| </a> |
| </h4> |
| </div> |
| |
| |
| <div class="panel-collapse" id="collapse-2" role="tabpanel" aria-label="Side Navigation"> |
| <div class="list-group"> |
| |
| |
| |
| <a class="list-group-item active" href="/docs/ir/">Nemo Intermediate Representation (IR)</a> |
| |
| |
| |
| <a class="list-group-item " href="/docs/passes_and_policies/">Passes and Policies</a> |
| |
| </div> |
| </div> |
| </div> |
| |
| <div class="panel panel-default"> |
| <div class="panel-heading"> |
| <h4 class="panel-title"> |
| <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapse-3" aria-expanded="false" aria-controls="collapse-3"> |
| System Designs |
| </a> |
| </h4> |
| </div> |
| |
| |
| <div class="panel-collapse collapse" id="collapse-3" role="tabpanel" aria-label="Side Navigation"> |
| <div class="list-group"> |
| |
| |
| |
| <a class="list-group-item " href="/docs/compiler_design/">Compiler Design</a> |
| |
| |
| |
| <a class="list-group-item " href="/docs/runtime_design/">Runtime Design</a> |
| |
| </div> |
| </div> |
| </div> |
| |
| <div class="panel panel-default"> |
| <div class="panel-heading"> |
| <h4 class="panel-title"> |
| <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapse-4" aria-expanded="false" aria-controls="collapse-4"> |
| Contribute |
| </a> |
| </h4> |
| </div> |
| |
| |
| <div class="panel-collapse collapse" id="collapse-4" role="tabpanel" aria-label="Side Navigation"> |
| <div class="list-group"> |
| |
| |
| |
| <a class="list-group-item " href="/docs/contribute/">Contribute</a> |
| |
| </div> |
| </div> |
| </div> |
| |
| <div class="panel panel-default"> |
| <div class="panel-heading"> |
| <h4 class="panel-title"> |
| <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapse-5" aria-expanded="false" aria-controls="collapse-5"> |
| Security |
| </a> |
| </h4> |
| </div> |
| |
| |
| <div class="panel-collapse collapse" id="collapse-5" role="tabpanel" aria-label="Side Navigation"> |
| <div class="list-group"> |
| |
| |
| |
| <a class="list-group-item " href="/docs/security/">Security Guide</a> |
| |
| </div> |
| </div> |
| </div> |
| |
| </div> |
| |
| </div> |
| |
| <div class="col-md-8"> |
| <h1>Nemo Intermediate Representation (IR)</h1> |
| <div id="markdown-content-container"><h3 id="overview">Overview</h3> |
| |
| <p>IR is an abstraction that we use to express the logical notion of data processing applications and the underlying execution runtime behaviors on separate layers. |
| It basically takes a form of directed acyclic graphs (DAGs), with which we can logically express dataflow programs. |
| To express various different execution properties to fully exploit different deployment characteristics, we enable flexible annotations to the IR on a separate layer. |
| On that layer, we can annotate specific execution properties related to the IR component.</p> |
| |
| <h3 id="ir-structure">IR structure</h3> |
| |
| <p>Nemo IR is composed of vertices, which each represent a data-parallel operator that transforms data, and edges between them, which each represents the dependency of data flow between the vertices. |
| Nemo IR supports four different types of IR vertices:</p> |
| |
| <ul> |
| <li><strong>UDF Vertex</strong>: Most commonly used vertex. Each UDF vertex contains a transform which determines the actions to take for the given input data. A transform can express any kind of data processing operation that high-level languages articulate.</li> |
| <li><strong>Source Vertex</strong>: This produces data by reading from an arbitrary source like disks and distributed filesystems.</li> |
| <li><strong>Metric Vertex</strong>: Metric vertex collects and emits metric data.</li> |
| <li><strong>Loop Vertex</strong>: Loop vertex is used to express iterative workflows, summarizing the part of the IR that occurs repetitively due to iterations. This comes very useful when expressing, controlling, and optimizing iterative workloads like MLR. It also effectively summarizes workloads.</li> |
| </ul> |
| |
| <h3 id="ir-annotation">IR annotation</h3> |
| |
| <p>Each IR vertex and edge can be annotated to be able to express the different <em>execution properties</em>. |
| For example, edges that the user wants to store intermediate data as local files can be annotated to use the ‘local file’ module for the ‘Data Store’ execution property. |
| <em>Execution properties</em> that can be configured for IR vertices include <em>parallelism, executor placement, stage number</em>, and <em>schedule group</em>, as they are related to the computation itself. |
| For IR edges, it includes <em>data store, data flow model, data communication pattern</em>, and <em>partitioning</em>, as they are used for expressing the behaviors regarding data transfer. |
| By having an IR for expressing workloads and the related execution properties, it enables the optimization phase to be decoupled, making it easier to implement and plug in different optimizations for different <em>deployment characteristics</em>.</p> |
| </div> |
| <div style="clear:both;"> |
| <p class="text-center"> |
| <br /> |
| <a target="_blank" href="https://github.com/apache/incubator-nemo-website/_docs/optimization/ir.md" class="btn btn-default" role="button"> |
| <i class="fa fa-pencil fa-lg"></i> Improve this page |
| </a> |
| </p> |
| </div> |
| <hr> |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| <ul class="pager"> |
| |
| |
| |
| <li class="previous"> |
| <a href="/docs/getting_started/"> |
| <span aria-hidden="true">←</span> Previous |
| </a> |
| </li> |
| |
| |
| |
| |
| |
| <li class="next"> |
| <a href="/docs/passes_and_policies/"> |
| Next <span aria-hidden="true">→</span> |
| </a> |
| </li> |
| |
| </ul> |
| <div class="clear"></div> |
| |
| |
| </div> |
| |
| </div> |
| </div> |
| |
| </div> |
| </div> |
| |
| <footer class="footer"> |
| <div class="container"> |
| |
| <p class="text-center"> |
| Nemo 2022 | |
| Powered by <a href="https://github.com/aksakalli/jekyll-doc-theme">Jekyll Doc Theme</a> |
| </p> |
| <!-- <p class="text-muted">Place sticky footer content here.</p> --> |
| </div> |
| </footer> |
| |
| <script> |
| var baseurl = '' |
| </script> |
| <script src="https://code.jquery.com/jquery-1.12.4.min.js"></script> |
| <script src="/assets/js/bootstrap.min.js "></script> |
| <script src="/assets/js/typeahead.bundle.min.js "></script> |
| |
| <script src="/assets/js/main.js "></script> |
| |
| </body> |
| |
| </html> |
