| <!DOCTYPE html> |
| <html> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| |
| <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> |
| <link rel="icon" href="/favicon.ico" type="image/x-icon"> |
| |
| <title>NUMA Support</title> |
| |
| <!-- Bootstrap core CSS --> |
| <link href="/assets/css/bootstrap.min.css" rel="stylesheet"> |
| <!-- Bootstrap theme --> |
| <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet"> |
| |
| <!-- Custom styles for this template --> |
| <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css"> |
| <link href="/css/style.css" rel="stylesheet"> |
| <link href="/assets/css/owl.theme.css" rel="stylesheet"> |
| <link href="/assets/css/owl.carousel.css" rel="stylesheet"> |
| <script type="text/javascript" src="/assets/js/jquery.min.js"></script> |
| <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script> |
| <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script> |
| <script type="text/javascript" src="/assets/js/storm.js"></script> |
| <!-- Just for debugging purposes. Don't actually copy these 2 lines! --> |
| <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]--> |
| |
| <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> |
| <!--[if lt IE 9]> |
| <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script> |
| <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script> |
| <![endif]--> |
| </head> |
| |
| |
| <body> |
| <header> |
| <div class="container-fluid"> |
| <div class="row"> |
| <div class="col-md-5"> |
| <a href="/index.html"><img src="/images/logo.png" class="logo" /></a> |
| </div> |
| <div class="col-md-5"> |
| |
| <h1>Version: 2.3.0</h1> |
| |
| </div> |
| <div class="col-md-2"> |
| <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a> |
| </div> |
| </div> |
| </div> |
| </header> |
| <!--Header End--> |
| <!--Navigation Begin--> |
| <div class="navbar" role="banner"> |
| <div class="container-fluid"> |
| <div class="navbar-header"> |
| <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| </div> |
| <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> |
| <ul class="nav navbar-nav"> |
| <li><a href="/index.html" id="home">Home</a></li> |
| <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li> |
| <li><a href="/about/integrates.html" id="project-info">Project Information</a></li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| |
| |
| <li><a href="/releases/2.3.0/index.html">2.3.0</a></li> |
| |
| |
| |
| <li><a href="/releases/2.2.0/index.html">2.2.0</a></li> |
| |
| |
| |
| <li><a href="/releases/2.1.0/index.html">2.1.0</a></li> |
| |
| |
| |
| <li><a href="/releases/2.0.0/index.html">2.0.0</a></li> |
| |
| |
| |
| <li><a href="/releases/1.2.3/index.html">1.2.3</a></li> |
| |
| |
| </ul> |
| </li> |
| <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li> |
| <li><a href="/contribute/People.html">People</a></li> |
| <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> |
| </ul> |
| </li> |
| <li><a href="/2021/09/27/storm230-released.html" id="news">News</a></li> |
| </ul> |
| </nav> |
| </div> |
| </div> |
| |
| |
| |
| <div class="container-fluid"> |
| <h1 class="page-title">NUMA Support</h1> |
| <div class="row"> |
| <div class="col-md-12"> |
| <!-- Documentation --> |
| |
| <p class="post-meta"></p> |
| |
| <div class="documentation-content"><h1 id="table-of-contents">Table of Contents</h1> |
| |
| <ol> |
| <li><a href="#Introduction">Introduction</a></li> |
| <li><a href="#Architecture">Architecture</a></li> |
| <li><a href="#RII-interaction">Resource Isolation Interface interaction</a> |
| |
| <ol> |
| <li><a href="#Setting-Memory-Requirement">CgroupManager</a></li> |
| <li><a href="#Setting-Shared-Memory">DockerManager</a></li> |
| <li><a href="#Setting-CPU-Requirement">RuncLibManager</a></li> |
| </ol></li> |
| <li><a href="#Configuring-NUMA">Configuring NUMA</a></li> |
| </ol> |
| |
| <p><div id='Introduction'/></p> |
| |
| <h2 id="introduction">Introduction</h2> |
| |
| <p>Non Uniform Memory Access (<a href="https://www.cc.gatech.edu/%7Eechow/ipcc/hpc-course/HPC-numa.pdf">NUMA</a>) is a new system architecture |
| where the hosts resources are grouped by cores and memory in NUMA zones. Storm supports isolating/pinning worker prcesses to specific |
| NUMA zones via the supervisor to take advantage of this resource isolation and avoid the penalty of using cross zone bus transfers |
| <div id='Architecture'/></p> |
| |
| <h2 id="architecture">Architecture</h2> |
| |
| <p>Once Storm supervisors are configured for NUMA (see section below) they now heartbeat multiple heartbeats - one for each NUMA zone. |
| Each of these NUMA supervisors have a supervisor id with the same prefixed supervisor id with the NUMA id differentiating them. |
| Nimbus, and by extension the scheduler, see these heartbeats and views the supervisor as multiple supervisors - one per configured NUMA zone. |
| Nimbus schedules topologies and assignments accordingly. The supervisor reads all assignments with the prefixed assignments and then |
| pins each worker to the NUMA zone according to the numa id in the assignment. The pinning depends on the Resource Isolation Interface used |
| and is elaborated on in the following section.</p> |
| |
| <p><div id='RII-Interaction'/></p> |
| |
| <h3 id="resource-isolation-interface-interaction">Resource Isolation Interface interaction</h3> |
| |
| <p>Each implementation of the Resource Isolation Interface (RII) should now implement NUMA pinning. The following are the current/soon to be available |
| implementations </p> |
| |
| <h4 id="cgroupmanager">CgroupManager</h4> |
| |
| <p>The CgroupManager prefixes the worker launch command with the numactl command for Linux hosts-</p> |
| <div class="highlight"><pre><code class="language-" data-lang="">numactl --cpunodebind=<numaId>> --membind=<numaId> <worker launch command> |
| </code></pre></div> |
| <p>The worker is then bound to the NUMA zone's CPU cores and memory zone.</p> |
| |
| <h4 id="dockermanager">DockerManager</h4> |
| |
| <p>Will be updated upon adding Docker support</p> |
| |
| <h4 id="runclibmanager">RuncLibManager</h4> |
| |
| <p>Will be updated upon adding Runc support</p> |
| |
| <p><div id='Configuring NUMA'/></p> |
| |
| <h3 id="configuring-numa">Configuring NUMA</h3> |
| |
| <p>In the Supervisor Config the following settings need to be set</p> |
| <div class="highlight"><pre><code class="language-" data-lang="">supervisor.numa.meta: |
| "0": # Numa zone id |
| numa.cores: # Cores in NUMA zone (can be determined by using the numastat command) |
| - 0 |
| - 1 |
| - 2 |
| - 3 |
| - 4 |
| - 5 |
| - 12 |
| - 13 |
| - 14 |
| - 15 |
| - 16 |
| |
| numa.generic.resources.map: # Generic Resources in the zone to be used for generic resource scheduling (optional) |
| network.resource.units: 50.0 |
| |
| numa.memory.mb: 42461 # Size of memory zone |
| numa.ports: # Ports to be assigned to workers pinned to the NUMA zone (this may include ports not specified in SUPERVISOR_SLOTS_PORTS |
| - 6700 |
| - 6701 |
| - 6702 |
| - 6703 |
| - 6704 |
| - 6705 |
| - 6706 |
| - 6707 |
| - 6708 |
| - 6709 |
| - 6710 |
| - 6711 |
| </code></pre></div></div> |
| |
| |
| </div> |
| </div> |
| </div> |
| <footer> |
| <div class="container-fluid"> |
| <div class="row"> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>Meetups</h5> |
| <ul class="latest-news"> |
| |
| <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li> |
| |
| <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li> |
| |
| <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li> |
| |
| <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li> |
| |
| <li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li> |
| |
| <li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li> |
| |
| <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> --> |
| </ul> |
| </div> |
| </div> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>About Apache Storm</h5> |
| <p>Apache Storm integrates with any queueing system and any database system. Apache Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Apache Storm with database systems is easy.</p> |
| </div> |
| </div> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>First Look</h5> |
| <ul class="footer-list"> |
| <li><a href="/releases/current/Rationale.html">Rationale</a></li> |
| <li><a href="/releases/current/Tutorial.html">Tutorial</a></li> |
| <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li> |
| <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Apache Storm project</a></li> |
| </ul> |
| </div> |
| </div> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>Documentation</h5> |
| <ul class="footer-list"> |
| <li><a href="/releases/current/index.html">Index</a></li> |
| <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li> |
| <li><a href="/releases/current/FAQ.html">FAQ</a></li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| <hr/> |
| <div class="row"> |
| <div class="col-md-12"> |
| <p align="center">Copyright © 2019 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved. |
| <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation. |
| <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> |
| </div> |
| </div> |
| </div> |
| </footer> |
| <!--Footer End--> |
| <!-- Scroll to top --> |
| <span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span> |
| |
| </body> |
| |
| </html> |
| |
