| <!DOCTYPE html> |
| <html> |
| |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| |
| <link rel="stylesheet" href="/css/main.css"> |
| <link rel="stylesheet" href="/css/font-awesome.min.css"> |
| |
| <link rel="shortcut icon" href="/favicon.ico?1"> |
| <!-- Begin Jekyll SEO tag v2.6.1 --> |
| <title>Overview | Nemo</title> |
| <meta name="generator" content="Jekyll v3.4.3" /> |
| <meta property="og:title" content="Overview" /> |
| <meta property="og:locale" content="en_US" /> |
| <meta name="description" content="Nemo aims to optimize data processing for better performance and datacenter efficiency, not only in general and common conditions, but also with various deployment characteristics. Such characteristics include processing data on specific resource environments, like transient resources, and running jobs with specific attributes, like skewed data." /> |
| <meta property="og:description" content="Nemo aims to optimize data processing for better performance and datacenter efficiency, not only in general and common conditions, but also with various deployment characteristics. Such characteristics include processing data on specific resource environments, like transient resources, and running jobs with specific attributes, like skewed data." /> |
| <link rel="canonical" href="http://nemo.apache.org//docs/home/" /> |
| <meta property="og:url" content="http://nemo.apache.org//docs/home/" /> |
| <meta property="og:site_name" content="Nemo" /> |
| <meta property="og:type" content="article" /> |
| <meta property="article:published_time" content="2019-06-28T03:51:21+09:00" /> |
| <script type="application/ld+json"> |
| {"url":"http://nemo.apache.org//docs/home/","headline":"Overview","dateModified":"2019-06-28T03:51:21+09:00","datePublished":"2019-06-28T03:51:21+09:00","description":"Nemo aims to optimize data processing for better performance and datacenter efficiency, not only in general and common conditions, but also with various deployment characteristics. Such characteristics include processing data on specific resource environments, like transient resources, and running jobs with specific attributes, like skewed data.","@type":"WebPage","@context":"https://schema.org"}</script> |
| <!-- End Jekyll SEO tag --> |
| |
| |
| <link rel="canonical" href="http://nemo.apache.org//docs/home/"> |
| <link rel="alternate" type="application/rss+xml" title="Nemo" href="http://nemo.apache.org//feed.xml" /> |
| </head> |
| |
| |
| <body> |
| |
| <nav class="navbar navbar-default navbar-fixed-top"> |
| <div class="container navbar-container"> |
| <div class="navbar-header"> |
| <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar"> |
| <span class="sr-only">Toggle navigation</span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| <a class="navbar-brand" href="/"> |
| <span><img src="/img/nemo-logo.png"></span> |
| </a> |
| </div> |
| <div id="navbar" class="collapse navbar-collapse"> |
| <ul class="nav navbar-nav"> |
| <li class="active" ><a href="/docs/home/">Docs</a></li> |
| <li ><a href="/apidocs">APIs</a></li> |
| <li ><a href="/pages/downloads">Downloads</a></li> |
| <li ><a href="/pages/talks">Talks</a></li> |
| <li ><a href="/pages/team">Team</a></li> |
| <li ><a href="/pages/license">License</a></li> |
| <li ><a href="/blog/2018/03/23/shuffle-on-nemo/">Blog</a></li> |
| </ul> |
| <div class="navbar-right"> |
| <form class="navbar-form navbar-left"> |
| <div class="form-group has-feedback"> |
| <input id="search-box" type="text" class="form-control" placeholder="Search..."> |
| <i class="fa fa-search form-control-feedback"></i> |
| </div> |
| </form> |
| <ul class="nav navbar-nav"> |
| <li><a href="https://github.com/apache/incubator-nemo"><i class="fa fa-github" aria-hidden="true"></i></a></li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| </nav> |
| |
| |
| <div class="page-content"> |
| <div class="wrapper"> |
| <div class="container"> |
| <div class="row"> |
| <div class="col-md-4"> |
| <div class="panel-group" id="accordion" role="tablist" aria-multiselectable="true"> |
| |
| <div class="panel panel-default"> |
| <div class="panel-heading"> |
| <h4 class="panel-title"> |
| <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapse-1" aria-expanded="false" aria-controls="collapse-1"> |
| Getting Started |
| </a> |
| </h4> |
| </div> |
| <div id="collapse-1" class="panel-collapse collapse" role="tabpanel" aria-label="Side Navigation"> |
| <div class="list-group"> |
| |
| |
| |
| <a class="list-group-item active" href="/docs/home/">Overview</a> |
| |
| |
| |
| <a class="list-group-item " href="/docs/getting_started/">Getting Started</a> |
| |
| </div> |
| </div> |
| </div> |
| |
| <div class="panel panel-default"> |
| <div class="panel-heading"> |
| <h4 class="panel-title"> |
| <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapse-2" aria-expanded="false" aria-controls="collapse-2"> |
| Optimizations |
| </a> |
| </h4> |
| </div> |
| <div id="collapse-2" class="panel-collapse collapse" role="tabpanel" aria-label="Side Navigation"> |
| <div class="list-group"> |
| |
| |
| |
| <a class="list-group-item " href="/docs/ir/">Nemo Intermediate Representation (IR)</a> |
| |
| |
| |
| <a class="list-group-item " href="/docs/passes_and_policies/">Passes and Policies</a> |
| |
| </div> |
| </div> |
| </div> |
| |
| <div class="panel panel-default"> |
| <div class="panel-heading"> |
| <h4 class="panel-title"> |
| <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapse-3" aria-expanded="false" aria-controls="collapse-3"> |
| System Designs |
| </a> |
| </h4> |
| </div> |
| <div id="collapse-3" class="panel-collapse collapse" role="tabpanel" aria-label="Side Navigation"> |
| <div class="list-group"> |
| |
| |
| |
| <a class="list-group-item " href="/docs/compiler_design/">Compiler Design</a> |
| |
| |
| |
| <a class="list-group-item " href="/docs/runtime_design/">Runtime Design</a> |
| |
| </div> |
| </div> |
| </div> |
| |
| <div class="panel panel-default"> |
| <div class="panel-heading"> |
| <h4 class="panel-title"> |
| <a role="button" data-toggle="collapse" data-parent="#accordion" href="#collapse-4" aria-expanded="false" aria-controls="collapse-4"> |
| Contribute |
| </a> |
| </h4> |
| </div> |
| <div id="collapse-4" class="panel-collapse collapse" role="tabpanel" aria-label="Side Navigation"> |
| <div class="list-group"> |
| |
| |
| |
| <a class="list-group-item " href="/docs/contribute/">Contribute</a> |
| |
| </div> |
| </div> |
| </div> |
| |
| </div> |
| |
| </div> |
| |
| <div class="col-md-8"> |
| <h1>Overview</h1> |
| <div id="markdown-content-container"><p>Nemo aims to optimize data processing for better performance and datacenter efficiency, not only in general and common conditions, but also with various <em>deployment characteristics</em>. |
| Such characteristics include processing data on <em>specific resource environments</em>, like transient resources, and running <em>jobs with specific attributes</em>, like skewed data.</p> |
| |
| <p>There exists many data processing systems with different designs to solve each of such problems it targets, but it fails to cover or adapt to unconsidered cases without substantial effort for modification. |
| The primary reason is because system runtime behaviors are hidden and planted inside the system core to hide the complexity of distributed computing. |
| This makes it very hard for a single system to support different <em>deployment characteristics</em> with different <em>runtime behaviors</em> without substantial effort.</p> |
| |
| <p>To solve this problem and easily modify <em>runtime behaviors</em> for different <em>deployment characteristics</em>, Nemo expresses workloads using the <a href="../ir">Nemo Intermediate Representation (IR)</a>, which represents the logical notion of data processing applications and its runtime behaviors on separate layers. |
| These layers can be easily modified through a set of high-level <a href="../passes_and_policies">graph pass</a> interfaces, exposed by the <a href="../compiler_design">Nemo Compiler</a>, enabling users to flexibly modify <em>runtime behaviors</em> at both compile-time and runtime. |
| Works represented this way can be executed by the <a href="../runtime_design">Nemo Execution Runtime</a> through its <a href="../extending_Nemo">modular and extensible</a> design.</p> |
| |
| <p>If you want to contribute, please refer to the <a href="../contribute">contribute</a> page. Otherwise, hit the button below to get started with Nemo!</p> |
| |
| <p><br /></p> |
| <div class="text-center"> |
| <a href="../getting_started/" class="btn btn-default btn-lg">Get Started!</a> |
| </div> |
| </div> |
| <p class="text-center"> |
| <br /> |
| <a target="_blank" href="https://github.com/apache/incubator-nemo-website/tree/asf-site/_docs/index.md" class="btn btn-default btn-sm githubEditButton" role="button"> |
| <i class="fa fa-pencil"></i> Improve this page |
| </a> |
| </p> |
| <hr> |
| |
| |
| |
| |
| |
| |
| |
| |
| <ul class="pager"> |
| |
| <li class="previous disabled"> |
| <a> |
| <span aria-hidden="true">←</span> Previous |
| </a> |
| </li> |
| |
| |
| |
| |
| |
| <li class="next"> |
| <a href="/docs/getting_started/"> |
| Next <span aria-hidden="true">→</span> |
| </a> |
| </li> |
| |
| </ul> |
| <div class="clear"></div> |
| |
| |
| </div> |
| |
| </div> |
| </div> |
| |
| </div> |
| </div> |
| |
| <footer class="footer"> |
| <div class="container"> |
| |
| <p class="text-center"> |
| Nemo 2019 | |
| Powered by <a href="https://github.com/aksakalli/jekyll-doc-theme">Jekyll Doc Theme</a> |
| </p> |
| <!-- <p class="text-muted">Place sticky footer content here.</p> --> |
| </div> |
| </footer> |
| |
| <script> |
| var baseurl = '' |
| </script> |
| <script src="//code.jquery.com/jquery-1.10.2.min.js"></script> |
| <script src="/js/bootstrap.min.js "></script> |
| <script src="/js/typeahead.bundle.min.js "></script> |
| |
| <script src="/js/main.js "></script> |
| |
| </body> |
| |
| </html> |
