| <!DOCTYPE html> |
| <html> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| |
| <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> |
| <link rel="icon" href="/favicon.ico" type="image/x-icon"> |
| |
| <title>Metrics Reporting API v2</title> |
| |
| <!-- Bootstrap core CSS --> |
| <link href="/assets/css/bootstrap.min.css" rel="stylesheet"> |
| <!-- Bootstrap theme --> |
| <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet"> |
| |
| <!-- Custom styles for this template --> |
| <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css"> |
| <link href="/css/style.css" rel="stylesheet"> |
| <link href="/assets/css/owl.theme.css" rel="stylesheet"> |
| <link href="/assets/css/owl.carousel.css" rel="stylesheet"> |
| <script type="text/javascript" src="/assets/js/jquery.min.js"></script> |
| <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script> |
| <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script> |
| <script type="text/javascript" src="/assets/js/storm.js"></script> |
| <!-- Just for debugging purposes. Don't actually copy these 2 lines! --> |
| <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]--> |
| |
| <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> |
| <!--[if lt IE 9]> |
| <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script> |
| <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script> |
| <![endif]--> |
| </head> |
| |
| |
| <body> |
| <header> |
| <div class="container-fluid"> |
| <div class="row"> |
| <div class="col-md-5"> |
| <a href="/index.html"><img src="/images/logo.png" class="logo" /></a> |
| </div> |
| <div class="col-md-5"> |
| |
| <h1>Version: 2.3.0</h1> |
| |
| </div> |
| <div class="col-md-2"> |
| <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a> |
| </div> |
| </div> |
| </div> |
| </header> |
| <!--Header End--> |
| <!--Navigation Begin--> |
| <div class="navbar" role="banner"> |
| <div class="container-fluid"> |
| <div class="navbar-header"> |
| <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| </div> |
| <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> |
| <ul class="nav navbar-nav"> |
| <li><a href="/index.html" id="home">Home</a></li> |
| <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li> |
| <li><a href="/about/integrates.html" id="project-info">Project Information</a></li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| |
| |
| <li><a href="/releases/2.3.0/index.html">2.3.0</a></li> |
| |
| |
| |
| <li><a href="/releases/2.2.0/index.html">2.2.0</a></li> |
| |
| |
| |
| <li><a href="/releases/2.1.0/index.html">2.1.0</a></li> |
| |
| |
| |
| <li><a href="/releases/2.0.0/index.html">2.0.0</a></li> |
| |
| |
| |
| <li><a href="/releases/1.2.3/index.html">1.2.3</a></li> |
| |
| |
| </ul> |
| </li> |
| <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li> |
| <li><a href="/contribute/People.html">People</a></li> |
| <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> |
| </ul> |
| </li> |
| <li><a href="/2021/09/27/storm230-released.html" id="news">News</a></li> |
| </ul> |
| </nav> |
| </div> |
| </div> |
| |
| |
| |
| <div class="container-fluid"> |
| <h1 class="page-title">Metrics Reporting API v2</h1> |
| <div class="row"> |
| <div class="col-md-12"> |
| <!-- Documentation --> |
| |
| <p class="post-meta"></p> |
| |
| <div class="documentation-content"><p>Apache Storm version 1.2 introduced a new metrics system for reporting |
| internal statistics (e.g. acked, failed, emitted, transferred, queue metrics, etc.) as well as a |
| new API for user defined metrics.</p> |
| |
| <p>The new metrics system is based on <a href="http://metrics.dropwizard.io">Dropwizard Metrics</a>.</p> |
| |
| <h2 id="user-defined-metrics">User Defined Metrics</h2> |
| |
| <p>To allow users to define custom metrics, the following methods have been added to the <code>TopologyContext</code> |
| class, an instance of which is passed to spout's <code>open()</code> method and bolt's <code>prepare()</code> method:</p> |
| <div class="highlight"><pre><code class="language-" data-lang="">public Timer registerTimer(String name) |
| |
| public Histogram registerHistogram(String name) |
| |
| public Meter registerMeter(String name) |
| |
| public Counter registerCounter(String name) |
| |
| public Gauge registerGauge(String name, Gauge gauge) |
| </code></pre></div> |
| <p>API documentation: <a href="http://metrics.dropwizard.io/4.0.0/apidocs/com/codahale/metrics/Timer.html">Timer</a>, |
| <a href="http://metrics.dropwizard.io/4.0.0/apidocs/com/codahale/metrics/Histogram.html">Histogram</a>, |
| <a href="http://metrics.dropwizard.io/4.0.0/apidocs/com/codahale/metrics/Meter.html">Meter</a>, |
| <a href="http://metrics.dropwizard.io/4.0.0/apidocs/com/codahale/metrics/Counter.html">Counter</a>, |
| <a href="http://metrics.dropwizard.io/4.0.0/apidocs/com/codahale/metrics/Gauge.html">Gauge</a></p> |
| |
| <p>Each of these methods takes a <code>name</code> parameter that acts as an identifier. When metrics are |
| registered, Storm will add additional information such as hostname, port, topology ID, etc. to form a unique metric |
| identifier. For example, if we register a metric named <code>myCounter</code> as follows:</p> |
| <div class="highlight"><pre><code class="language-java" data-lang="java"> <span class="n">Counter</span> <span class="n">myCounter</span> <span class="o">=</span> <span class="n">topologyContext</span><span class="o">.</span><span class="na">registerCounter</span><span class="o">(</span><span class="s">"myCounter"</span><span class="o">);</span> |
| </code></pre></div> |
| <p>the resulting name sent to metrics reporters will expand to:</p> |
| <div class="highlight"><pre><code class="language-" data-lang=""> storm.topology.{topology ID}.{hostname}.{component ID}.{task ID}.{worker port}-myCounter |
| </code></pre></div> |
| <p>The additional information allows for the unique identification of metrics for component instances across the cluster.</p> |
| |
| <p><em>Important Note:</em> In order to ensure metric names can be reliably parsed, any <code>.</code> characters in name components will |
| be replaced with an underscore (<code>_</code>) character. For example, the hostname <code>storm.example.com</code> will appear as |
| <code>storm_example_com</code> in the metric name. This character substitution *is not applied to the user-supplied <code>name</code> parameter.</p> |
| |
| <h3 id="example-tuple-counter-bolt">Example: Tuple Counter Bolt</h3> |
| |
| <p>The following example is a simple bolt implementation that will report the running total up tuples received by a bolt:</p> |
| <div class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">TupleCountingBolt</span> <span class="kd">extends</span> <span class="n">BaseRichBolt</span> <span class="o">{</span> |
| <span class="kd">private</span> <span class="n">Counter</span> <span class="n">tupleCounter</span><span class="o">;</span> |
| <span class="nd">@Override</span> |
| <span class="kd">public</span> <span class="kt">void</span> <span class="nf">prepare</span><span class="o">(</span><span class="n">Map</span> <span class="n">stormConf</span><span class="o">,</span> <span class="n">TopologyContext</span> <span class="n">context</span><span class="o">,</span> <span class="n">OutputCollector</span> <span class="n">collector</span><span class="o">)</span> <span class="o">{</span> |
| <span class="k">this</span><span class="o">.</span><span class="na">tupleCounter</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="na">registerCounter</span><span class="o">(</span><span class="s">"tupleCount"</span><span class="o">);</span> |
| <span class="o">}</span> |
| |
| <span class="nd">@Override</span> |
| <span class="kd">public</span> <span class="kt">void</span> <span class="nf">execute</span><span class="o">(</span><span class="n">Tuple</span> <span class="n">input</span><span class="o">)</span> <span class="o">{</span> |
| <span class="k">this</span><span class="o">.</span><span class="na">tupleCounter</span><span class="o">.</span><span class="na">inc</span><span class="o">();</span> |
| <span class="o">}</span> |
| <span class="o">}</span> |
| </code></pre></div> |
| <h2 id="metric-reporter-configuration">Metric Reporter Configuration</h2> |
| |
| <p>For metrics to be useful they must be <em>reported</em>, in other words sent somewhere where they can be consumed and analyzed. |
| That can be as simple as writing them to a log file, sending them to a time series database, or exposing them via JMX.</p> |
| |
| <p>The following metric reporters are supported</p> |
| |
| <ul> |
| <li>Console Reporter (<code>org.apache.storm.metrics2.reporters.ConsoleStormReporter</code>): |
| Reports metrics to <code>System.out</code>.</li> |
| <li>CSV Reporter (<code>org.apache.storm.metrics2.reporters.CsvStormReporter</code>): |
| Reports metrics to a CSV file.</li> |
| <li>Graphite Reporter (<code>org.apache.storm.metrics2.reporters.GraphiteStormReporter</code>): |
| Reports metrics to a <a href="https://graphiteapp.org">Graphite</a> server.</li> |
| <li>JMX Reporter (<code>org.apache.storm.metrics2.reporters.JmxStormReporter</code>): |
| Exposes metrics via JMX.</li> |
| </ul> |
| |
| <p>Custom metrics reporters can be created by implementing <code>org.apache.storm.metrics2.reporters.StormReporter</code> interface |
| or extending <code>org.apache.storm.metrics2.reporters.ScheduledStormReporter</code> class.</p> |
| |
| <p>By default, Storm will collect metrics but not "report" or |
| send the collected metrics anywhere. To enable metrics reporting, add a <code>topology.metrics.reporters</code> section to <code>storm.yaml</code> |
| or in topology configuration and configure one or more reporters.</p> |
| |
| <p>The following example configuration sets up two reporters: a Graphite Reporter and a Console Reporter:</p> |
| <div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">topology.metrics.reporters</span><span class="pi">:</span> |
| <span class="c1"># Graphite Reporter</span> |
| <span class="pi">-</span> <span class="na">class</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.metrics2.reporters.GraphiteStormReporter"</span> |
| <span class="s">report.period</span><span class="pi">:</span> <span class="s">60</span> |
| <span class="s">report.period.units</span><span class="pi">:</span> <span class="s2">"</span><span class="s">SECONDS"</span> |
| <span class="s">graphite.host</span><span class="pi">:</span> <span class="s2">"</span><span class="s">localhost"</span> |
| <span class="s">graphite.port</span><span class="pi">:</span> <span class="s">2003</span> |
| |
| <span class="c1"># Console Reporter</span> |
| <span class="pi">-</span> <span class="na">class</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.metrics2.reporters.ConsoleStormReporter"</span> |
| <span class="s">report.period</span><span class="pi">:</span> <span class="s">10</span> |
| <span class="s">report.period.units</span><span class="pi">:</span> <span class="s2">"</span><span class="s">SECONDS"</span> |
| <span class="na">filter</span><span class="pi">:</span> |
| <span class="na">class</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.metrics2.filters.RegexFilter"</span> |
| <span class="na">expression</span><span class="pi">:</span> <span class="s2">"</span><span class="s">.*my_component.*emitted.*"</span> |
| </code></pre></div> |
| <p>Each reporter section begins with a <code>class</code> parameter representing the fully-qualified class name of the reporter |
| implementation. </p> |
| |
| <p>Many reporter implementations are <em>scheduled</em>, meaning they report metrics at regular intervals. The reporting interval |
| is determined by the <code>report.period</code> and <code>report.period.units</code> parameters.</p> |
| |
| <p>Reporters can also be configured with an optional filter that determines which metrics get reported. Storm includes the |
| <code>org.apache.storm.metrics2.filters.RegexFilter</code> filter which uses a regular expression to determine which metrics get |
| reported. Custom filters can be created by implementing the <code>org.apache.storm.metrics2.filters.StormMetricFilter</code> |
| interface:</p> |
| <div class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">interface</span> <span class="nc">StormMetricsFilter</span> <span class="kd">extends</span> <span class="n">MetricFilter</span> <span class="o">{</span> |
| |
| <span class="cm">/** |
| * Called after the filter is instantiated. |
| * @param config A map of the properties from the 'filter' section of the reporter configuration. |
| */</span> |
| <span class="kt">void</span> <span class="nf">prepare</span><span class="o">(</span><span class="n">Map</span><span class="o"><</span><span class="n">String</span><span class="o">,</span> <span class="n">Object</span><span class="o">></span> <span class="n">config</span><span class="o">);</span> |
| |
| <span class="cm">/** |
| * Returns true if the given metric should be reported. |
| */</span> |
| <span class="kt">boolean</span> <span class="nf">matches</span><span class="o">(</span><span class="n">String</span> <span class="n">name</span><span class="o">,</span> <span class="n">Metric</span> <span class="n">metric</span><span class="o">);</span> |
| |
| <span class="o">}</span> |
| </code></pre></div> |
| <p>V2 metrics can be reported with a long name (such as storm.topology.mytopologyname-17-1595349167.hostname.__system.-1.6700-memory.pools.Code-Cache.max) or with a short |
| name and dimensions (such as memory.pools.Code-Cache.max with dimensions task Id of -1 and component Id of __system) if reporters support this. Each reporter defaults |
| to using the long metric name, but can report the short name by configuring report.dimensions.enabled to true for the reporter.</p> |
| |
| <h2 id="backwards-compatibility-notes">Backwards Compatibility Notes</h2> |
| |
| <ol> |
| <li><p>V2 metrics can also be reported to the Metrics Consumers registered with <code>topology.metrics.consumer.register</code> by enabling the <code>topology.enable.v2.metrics.tick</code> configuration. |
| The rate that they will reported to Metric Consumers is controlled by <code>topology.v2.metrics.tick.interval.seconds</code>, defaulting to every 60 seconds.</p></li> |
| <li><p>Starting from storm 2.3, the config <code>storm.metrics.reporters</code> is deprecated in favor of <code>topology.metrics.reporters</code>.</p></li> |
| <li><p>Starting from storm 2.3, the <code>daemons</code> section is removed from <code>topology.metrics.reporters</code> (or <code>storm.metrics.reporters</code>). |
| Before storm 2.3, a <code>daemons</code> section is required in the reporter conf to determine which daemons the reporters will apply to. |
| However, the reporters configured with <code>topology.metrics.reporters</code> (or <code>storm.metrics.reporters</code>) actually only apply to workers. They are never really used in daemons like nimbus, supervisor and etc. |
| For daemon metrics, please refer to <a href="ClusterMetrics.html">Cluster Metrics</a>.</p></li> |
| <li><p><strong>Backwards Compatibility Breakage</strong>: starting from storm 2.3, the following configs no longer apply to <code>topology.metrics.reporters</code>: |
| <code>yaml |
| storm.daemon.metrics.reporter.plugin.locale |
| storm.daemon.metrics.reporter.plugin.rate.unit |
| storm.daemon.metrics.reporter.plugin.duration.unit |
| </code></p> |
| |
| <p>They only apply to daemon metric reporters configured via <code>storm.daemon.metrics.reporter.plugins</code> for storm daemons. |
| The corresponding configs for <code>topology.metrics.reporters</code> can be configured in reporter conf with <code>locale</code>, <code>rate.unit</code>, <code>duration.unit</code> respectively, for example, |
| <code>yaml |
| topology.metrics.reporters: |
| # Console Reporter |
| - class: "org.apache.storm.metrics2.reporters.ConsoleStormReporter" |
| report.period: 10 |
| report.period.units: "SECONDS" |
| locale: "en-US" |
| rate.unit: "SECONDS" |
| duration.unit: "SECONDS" |
| </code> |
| Default values will be used if they are not set or set to <code>null</code>.</p></li> |
| </ol> |
| </div> |
| |
| |
| </div> |
| </div> |
| </div> |
| <footer> |
| <div class="container-fluid"> |
| <div class="row"> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>Meetups</h5> |
| <ul class="latest-news"> |
| |
| <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li> |
| |
| <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li> |
| |
| <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li> |
| |
| <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li> |
| |
| <li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li> |
| |
| <li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li> |
| |
| <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> --> |
| </ul> |
| </div> |
| </div> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>About Apache Storm</h5> |
| <p>Apache Storm integrates with any queueing system and any database system. Apache Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Apache Storm with database systems is easy.</p> |
| </div> |
| </div> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>First Look</h5> |
| <ul class="footer-list"> |
| <li><a href="/releases/current/Rationale.html">Rationale</a></li> |
| <li><a href="/releases/current/Tutorial.html">Tutorial</a></li> |
| <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li> |
| <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Apache Storm project</a></li> |
| </ul> |
| </div> |
| </div> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>Documentation</h5> |
| <ul class="footer-list"> |
| <li><a href="/releases/current/index.html">Index</a></li> |
| <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li> |
| <li><a href="/releases/current/FAQ.html">FAQ</a></li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| <hr/> |
| <div class="row"> |
| <div class="col-md-12"> |
| <p align="center">Copyright © 2019 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved. |
| <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation. |
| <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> |
| </div> |
| </div> |
| </div> |
| </footer> |
| <!--Footer End--> |
| <!-- Scroll to top --> |
| <span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span> |
| |
| </body> |
| |
| </html> |
| |
