| <!DOCTYPE html> |
| <!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> |
| <!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| |
| |
| <link rel="shortcut icon" href="../../img/favicon.ico"> |
| <title>Performance - Apache Gearpump(incubating)</title> |
| <link href='https://fonts.googleapis.com/css?family=Lato:400,700|Roboto+Slab:400,700|Inconsolata:400,700' rel='stylesheet' type='text/css'> |
| |
| <link rel="stylesheet" href="../../css/theme.css" type="text/css" /> |
| <link rel="stylesheet" href="../../css/theme_extra.css" type="text/css" /> |
| <link rel="stylesheet" href="../../css/highlight.css"> |
| |
| <script> |
| // Current page data |
| var mkdocs_page_name = "Performance"; |
| var mkdocs_page_input_path = "introduction/performance-report.md"; |
| var mkdocs_page_url = "/introduction/performance-report/index.html"; |
| </script> |
| |
| <script src="../../js/jquery-2.1.1.min.js"></script> |
| <script src="../../js/modernizr-2.8.3.min.js"></script> |
| <script type="text/javascript" src="../../js/highlight.pack.js"></script> |
| |
| </head> |
| |
| <body class="wy-body-for-nav" role="document"> |
| |
| <div class="wy-grid-for-nav"> |
| |
| |
| <nav data-toggle="wy-nav-shift" class="wy-nav-side stickynav"> |
| <div class="wy-side-nav-search"> |
| <a href="../../index.html" class="icon icon-home"> Apache Gearpump(incubating)</a> |
| <div role="search"> |
| <form id ="rtd-search-form" class="wy-form" action="../../search.html" method="get"> |
| <input type="text" name="q" placeholder="Search docs" /> |
| </form> |
| </div> |
| </div> |
| |
| <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> |
| <ul class="current"> |
| |
| |
| <li class="toctree-l1"> |
| |
| <a class="" href="../../index.html">Overview</a> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Introduction</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../submit-your-1st-application/index.html">Submit Your 1st Application</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../commandline/index.html">Client Command Line</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../basic-concepts/index.html">Basic Concepts</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../features/index.html">Technical Highlights</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../message-delivery/index.html">Reliable Message Delivery</a> |
| </li> |
| <li class=" current"> |
| |
| <a class="current" href="index.html">Performance</a> |
| <ul class="subnav"> |
| |
| <li class="toctree-l3"><a href="#performance-evaluation">Performance Evaluation</a></li> |
| |
| <ul> |
| |
| <li><a class="toctree-l4" href="#throughput">Throughput</a></li> |
| |
| <li><a class="toctree-l4" href="#latency">Latency</a></li> |
| |
| <li><a class="toctree-l4" href="#fault-recovery-time">Fault Recovery time</a></li> |
| |
| <li><a class="toctree-l4" href="#how-to-setup-the-benchmark-environment">How to setup the benchmark environment?</a></li> |
| |
| </ul> |
| |
| |
| </ul> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Deployment</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../deployment/deployment-local/index.html">Local Mode</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../deployment/deployment-standalone/index.html">Standalone Mode</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../deployment/deployment-yarn/index.html">YARN Mode</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../deployment/deployment-docker/index.html">Docker Mode</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../deployment/deployment-ui-authentication/index.html">UI Authentication</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../deployment/deployment-ha/index.html">High Availability</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../deployment/deployment-msg-delivery/index.html">Reliable Message Delivery</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../deployment/deployment-configuration/index.html">Configuration</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../deployment/deployment-resource-isolation/index.html">Resource Isolation</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../deployment/deployment-security/index.html">YARN Security Guide</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../deployment/get-gearpump-distribution/index.html">How to Get Your Gearpump Distribution</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../deployment/hardware-requirement/index.html">Hardware Requirement</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Programming Guide</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../dev/dev-write-1st-app/index.html">Write Your 1st App</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../dev/dev-custom-serializer/index.html">Customized Message Passing</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../dev/dev-connectors/index.html">Gearpump Connectors</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../dev/dev-storm/index.html">Storm Compatibility</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../dev/dev-ide-setup/index.html">IDE Setup</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../dev/dev-non-streaming-example/index.html">Non Streaming Examples</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../dev/dev-rest-api/index.html">REST API</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../api/scala/index.html">Scala API</a> |
| </li> |
| <li class=""> |
| |
| <a class="" href="../../api/java/index.html">Java API</a> |
| </li> |
| </ul> |
| </li> |
| |
| <li class="toctree-l1"> |
| |
| <span class="caption-text">Internals</span> |
| <ul class="subnav"> |
| <li class=""> |
| |
| <a class="" href="../../internals/gearpump-internals/index.html">Gearpump Internals</a> |
| </li> |
| </ul> |
| </li> |
| |
| </ul> |
| </div> |
| |
| </nav> |
| |
| <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> |
| |
| |
| <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> |
| <i data-toggle="wy-nav-top" class="fa fa-bars"></i> |
| <a href="../../index.html">Apache Gearpump(incubating)</a> |
| </nav> |
| |
| |
| <div class="wy-nav-content"> |
| <div class="rst-content"> |
| <div role="navigation" aria-label="breadcrumbs navigation"> |
| <ul class="wy-breadcrumbs"> |
| <li><a href="../../index.html">Docs</a> »</li> |
| |
| |
| |
| <li>Introduction »</li> |
| |
| |
| |
| <li>Performance</li> |
| <li class="wy-breadcrumbs-aside"> |
| |
| <a href="https://github.com/apache/incubator-gearpump/edit/master/docs/contents/introduction/performance-report.md" |
| class="icon icon-github"> Edit on GitHub</a> |
| |
| </li> |
| </ul> |
| <hr/> |
| </div> |
| <div role="main"> |
| <div class="section"> |
| |
| <h1 id="performance-evaluation">Performance Evaluation</h1> |
| <p>To illustrate the performance of Gearpump, we mainly focused on two aspects, throughput and latency, using a micro benchmark called SOL (an example in the Gearpump package) whose topology is quite simple. SOLStreamProducer delivers messages to SOLStreamProcessor constantly and SOLStreamProcessor does nothing. We set up a 4-nodes cluster with 10GbE network and each node's hardware is briefly shown as follows:</p> |
| <p>Processor: 32 core Intel(R) Xeon(R) CPU E5-2690 2.90GHz |
| Memory: 64GB</p> |
| <h2 id="throughput">Throughput</h2> |
| <p>We tried to explore the upper bound of the throughput, after launching 48 SOLStreamProducer and 48 SOLStreamProcessor the Figure below shows that the whole throughput of the cluster can reach about 18 million messages/second(100 bytes per message)</p> |
| <h2 id="latency">Latency</h2> |
| <p>When we transfer message at the max throughput above, the average latency between two tasks is 8ms.</p> |
| <h2 id="fault-recovery-time">Fault Recovery time</h2> |
| <p>When the corruption is detected, for example the Executor is down, Gearpump will reallocate the resource and restart the application. It takes about 10 seconds to recover the application.</p> |
| <p><img alt="Dashboard" src="../../img/dashboard.png" /></p> |
| <h2 id="how-to-setup-the-benchmark-environment">How to setup the benchmark environment?</h2> |
| <h3 id="prepare-the-env">Prepare the env</h3> |
| <p>1). Set up a 4-nodes Gearpump cluster with 10GbE network which have 4 Workers on each node. In our test environment, each node has 64GB memory and Intel(R) Xeon(R) 32-core processor E5-2690 2.90GHz. Make sure the metrics is enabled in Gearpump.</p> |
| <p>2). Submit a SOL application with 48 StreamProducers and 48 StreamProcessors:</p> |
| <pre class="codehilite"><code class="language-bash">bin/gear app -jar ./examples/sol-2.11-0.8.4-assembly.jar -streamProducer 48 -streamProcessor 48</code></pre> |
| |
| |
| <p>3). Launch Gearpump's dashboard and browser http://$HOST:8090/, switch to the Applications tab and you can see the detail information of your application. The HOST should be the node runs dashboard.</p> |
| |
| </div> |
| </div> |
| <footer> |
| |
| <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> |
| |
| <a href="../../deployment/deployment-local/index.html" class="btn btn-neutral float-right" title="Local Mode">Next <span class="icon icon-circle-arrow-right"></span></a> |
| |
| |
| <a href="../message-delivery/index.html" class="btn btn-neutral" title="Reliable Message Delivery"><span class="icon icon-circle-arrow-left"></span> Previous</a> |
| |
| </div> |
| |
| |
| <hr/> |
| |
| <div role="contentinfo"> |
| <!-- Copyright etc --> |
| |
| </div> |
| |
| Built with <a href="http://www.mkdocs.org">MkDocs</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. |
| </footer> |
| |
| </div> |
| </div> |
| |
| </section> |
| |
| </div> |
| |
| <div class="rst-versions" role="note" style="cursor: pointer"> |
| <span class="rst-current-version" data-toggle="rst-current-version"> |
| |
| <a href="https://github.com/apache/incubator-gearpump" class="fa fa-github" style="float: left; color: #fcfcfc"> GitHub</a> |
| |
| |
| <span><a href="../message-delivery/index.html" style="color: #fcfcfc;">« Previous</a></span> |
| |
| |
| <span style="margin-left: 15px"><a href="../../deployment/deployment-local/index.html" style="color: #fcfcfc">Next »</a></span> |
| |
| </span> |
| </div> |
| <script src="../../js/theme.js"></script> |
| |
| </body> |
| </html> |
