| <!DOCTYPE html> |
| <html> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| |
| <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> |
| <link rel="icon" href="/favicon.ico" type="image/x-icon"> |
| |
| <title>Storm Metricstore</title> |
| |
| <!-- Bootstrap core CSS --> |
| <link href="/assets/css/bootstrap.min.css" rel="stylesheet"> |
| <!-- Bootstrap theme --> |
| <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet"> |
| |
| <!-- Custom styles for this template --> |
| <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css"> |
| <link href="/css/style.css" rel="stylesheet"> |
| <link href="/assets/css/owl.theme.css" rel="stylesheet"> |
| <link href="/assets/css/owl.carousel.css" rel="stylesheet"> |
| <script type="text/javascript" src="/assets/js/jquery.min.js"></script> |
| <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script> |
| <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script> |
| <script type="text/javascript" src="/assets/js/storm.js"></script> |
| <!-- Just for debugging purposes. Don't actually copy these 2 lines! --> |
| <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]--> |
| |
| <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> |
| <!--[if lt IE 9]> |
| <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script> |
| <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script> |
| <![endif]--> |
| </head> |
| |
| |
| <body> |
| <header> |
| <div class="container-fluid"> |
| <div class="row"> |
| <div class="col-md-5"> |
| <a href="/index.html"><img src="/images/logo.png" class="logo" /></a> |
| </div> |
| <div class="col-md-5"> |
| |
| <h1>Version: 2.0.0</h1> |
| |
| </div> |
| <div class="col-md-2"> |
| <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a> |
| </div> |
| </div> |
| </div> |
| </header> |
| <!--Header End--> |
| <!--Navigation Begin--> |
| <div class="navbar" role="banner"> |
| <div class="container-fluid"> |
| <div class="navbar-header"> |
| <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| </div> |
| <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> |
| <ul class="nav navbar-nav"> |
| <li><a href="/index.html" id="home">Home</a></li> |
| <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li> |
| <li><a href="/about/integrates.html" id="project-info">Project Information</a></li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| |
| |
| <li><a href="/releases/2.3.0/index.html">2.3.0</a></li> |
| |
| |
| |
| <li><a href="/releases/2.2.0/index.html">2.2.0</a></li> |
| |
| |
| |
| <li><a href="/releases/2.1.0/index.html">2.1.0</a></li> |
| |
| |
| |
| <li><a href="/releases/2.0.0/index.html">2.0.0</a></li> |
| |
| |
| |
| <li><a href="/releases/1.2.3/index.html">1.2.3</a></li> |
| |
| |
| </ul> |
| </li> |
| <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li> |
| <li><a href="/contribute/People.html">People</a></li> |
| <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> |
| </ul> |
| </li> |
| <li><a href="/2021/09/27/storm230-released.html" id="news">News</a></li> |
| </ul> |
| </nav> |
| </div> |
| </div> |
| |
| |
| |
| <div class="container-fluid"> |
| <h1 class="page-title">Storm Metricstore</h1> |
| <div class="row"> |
| <div class="col-md-12"> |
| <!-- Documentation --> |
| |
| <p class="post-meta"></p> |
| |
| <div class="documentation-content"><p>A metric store (<a href="http://github.com/apache/storm/blob/v2.0.0/storm-server/src/main/java/org/apache/storm/metricstore/MetricStore.java"><code>MetricStore</code></a>) interface was added |
| to Nimbus to allow storing metric information (<a href="http://github.com/apache/storm/blob/v2.0.0/storm-server/src/main/java/org/apache/storm/metricstore/Metric.java"><code>Metric</code></a>) |
| to a database. The default implementation |
| (<a href="http://github.com/apache/storm/blob/v2.0.0/storm-server/src/main/java/org/apache/storm/metricstore/rocksdb/RocksDbStore.java"><code>RocksDbStore</code></a>) is using RocksDB, |
| a key-value store.</p> |
| |
| <p>As metrics are stored in RocksDB, their string values (for topology ID and executor ID, etc.) are converted to unique integer IDs, and these strings |
| are also stored to the database as metadata indexed by the integer ID. When a metric is stored, it is also aggregated with any existing metric |
| within the same 1, 10, and 60 minute timeframe. </p> |
| |
| <p>The <a href="http://github.com/apache/storm/blob/v2.0.0/storm-server/src/main/java/org/apache/storm/metricstore/FilterOptions.java"><code>FilterOptions</code></a> class provides an interface |
| to select which options can be used to scan the metrics.</p> |
| |
| <h3 id="configuration">Configuration</h3> |
| |
| <p>The following configuation options exist:</p> |
| <div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">storm.metricstore.class</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.metricstore.rocksdb.RocksDbStore"</span> |
| <span class="s">storm.metricprocessor.class</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.metricstore.NimbusMetricProcessor"</span> |
| <span class="s">storm.metricstore.rocksdb.location</span><span class="pi">:</span> <span class="s2">"</span><span class="s">storm_rocks"</span> |
| <span class="s">storm.metricstore.rocksdb.create_if_missing</span><span class="pi">:</span> <span class="no">true</span> |
| <span class="s">storm.metricstore.rocksdb.metadata_string_cache_capacity</span><span class="pi">:</span> <span class="s">4000</span> |
| <span class="s">storm.metricstore.rocksdb.retention_hours</span><span class="pi">:</span> <span class="s">240</span> |
| </code></pre></div> |
| <ul> |
| <li>storm.metricstore.class is the class that implements the |
| (<a href="http://github.com/apache/storm/blob/v2.0.0/storm-server/src/main/java/org/apache/storm/metricstore/MetricStore.java"><code>MetricStore</code></a>).</li> |
| <li>storm.metricprocessor.class is the class that implements the |
| (<a href="http://github.com/apache/storm/blob/v2.0.0/storm-server/src/main/java/org/apache/storm/metricstore/WorkerMetricsProcessor.java"><code>WorkerMetricsProcessor</code></a>).</li> |
| <li>storm.metricstore.rocksdb.location provides to location of the RocksDB database on Nimbus</li> |
| <li>storm.metricstore.rocksdb.create_if_missing permits creating a RocksDB database if missing</li> |
| <li>storm.metricstore.rocksdb.metadata_string_cache_capacity controls the number of metadata strings cached in memory.</li> |
| <li>storm.metricstore.rocksdb.retention_hours sets the length of time metrics will remain active.</li> |
| </ul> |
| |
| <h3 id="rocksdb-schema">RocksDB Schema</h3> |
| |
| <p>The RocksDB Key (represented by <a href="http://github.com/apache/storm/blob/v2.0.0/storm-server/src/main/java/org/apache/storm/metricstore/rocksdb/RocksDbKey.java"><code>RocksDbKey</code></a>) |
| fields are as follows:</p> |
| |
| <table><thead> |
| <tr> |
| <th>Field</th> |
| <th>Size</th> |
| <th>Offset</th> |
| <th>Description</th> |
| </tr> |
| </thead><tbody> |
| <tr> |
| <td>Type</td> |
| <td>1</td> |
| <td>0</td> |
| <td>The type maps to the KeyType enum, specifying a metric or various types of metadata strings</td> |
| </tr> |
| <tr> |
| <td>Aggregation Level</td> |
| <td>1</td> |
| <td>1</td> |
| <td>The aggregation level for a metric (see AggLevel enum). Set to 0 for metadata.</td> |
| </tr> |
| <tr> |
| <td>Topology Id</td> |
| <td>4</td> |
| <td>2</td> |
| <td>The metadata string Id representing a topologyId for a metric, or the unique string Id for a metadata string</td> |
| </tr> |
| <tr> |
| <td>Timestamp</td> |
| <td>8</td> |
| <td>6</td> |
| <td>The timestamp for a metric, unused for metadata</td> |
| </tr> |
| <tr> |
| <td>Metric Id</td> |
| <td>4</td> |
| <td>14</td> |
| <td>The metadata string Id for the metric name</td> |
| </tr> |
| <tr> |
| <td>Component Id</td> |
| <td>4</td> |
| <td>18</td> |
| <td>The metadata string Id for the component Id</td> |
| </tr> |
| <tr> |
| <td>Executor Id</td> |
| <td>4</td> |
| <td>22</td> |
| <td>The metadata string Id for the executor Id</td> |
| </tr> |
| <tr> |
| <td>Host Id</td> |
| <td>4</td> |
| <td>26</td> |
| <td>The metadata string Id for the host Id</td> |
| </tr> |
| <tr> |
| <td>Port</td> |
| <td>4</td> |
| <td>30</td> |
| <td>The port number</td> |
| </tr> |
| <tr> |
| <td>Stream Id</td> |
| <td>4</td> |
| <td>34</td> |
| <td>The metadata string Id for the stream Id</td> |
| </tr> |
| </tbody></table> |
| |
| <p>The RocksDB Value fields for metadata strings (represented by |
| <a href="http://github.com/apache/storm/blob/v2.0.0/storm-server/src/main/java/org/apache/storm/metricstore/rocksdb/RocksDbValue.java"><code>RocksDbValue</code></a>) are as follows:</p> |
| |
| <table><thead> |
| <tr> |
| <th>Field</th> |
| <th>Size</th> |
| <th>Offset</th> |
| <th>Description</th> |
| </tr> |
| </thead><tbody> |
| <tr> |
| <td>Version</td> |
| <td>1</td> |
| <td>0</td> |
| <td>The current metadata version - allows migrating if the format changes in the future</td> |
| </tr> |
| <tr> |
| <td>Timestamp</td> |
| <td>8</td> |
| <td>1</td> |
| <td>The time when the metadata was last used by a metric. Allows deleting of old metadata.</td> |
| </tr> |
| <tr> |
| <td>Metadata String</td> |
| <td>any</td> |
| <td>9</td> |
| <td>The metadata string</td> |
| </tr> |
| </tbody></table> |
| |
| <p>RocksDB Value fields for metric data are as follows:</p> |
| |
| <table><thead> |
| <tr> |
| <th>Field</th> |
| <th>Size</th> |
| <th>Offset</th> |
| <th>Description</th> |
| </tr> |
| </thead><tbody> |
| <tr> |
| <td>Version</td> |
| <td>1</td> |
| <td>0</td> |
| <td>The current metric version - allows migrating if the format changes in the future</td> |
| </tr> |
| <tr> |
| <td>Value</td> |
| <td>8</td> |
| <td>1</td> |
| <td>The metric value</td> |
| </tr> |
| <tr> |
| <td>Count</td> |
| <td>8</td> |
| <td>9</td> |
| <td>The metric count</td> |
| </tr> |
| <tr> |
| <td>Min</td> |
| <td>8</td> |
| <td>17</td> |
| <td>The minimum metric value</td> |
| </tr> |
| <tr> |
| <td>Max</td> |
| <td>8</td> |
| <td>25</td> |
| <td>The maximum metric value</td> |
| </tr> |
| <tr> |
| <td>Sum</td> |
| <td>8</td> |
| <td>33</td> |
| <td>The sum of the metric values</td> |
| </tr> |
| </tbody></table> |
| </div> |
| |
| |
| </div> |
| </div> |
| </div> |
| <footer> |
| <div class="container-fluid"> |
| <div class="row"> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>Meetups</h5> |
| <ul class="latest-news"> |
| |
| <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li> |
| |
| <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li> |
| |
| <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li> |
| |
| <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li> |
| |
| <li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li> |
| |
| <li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li> |
| |
| <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> --> |
| </ul> |
| </div> |
| </div> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>About Apache Storm</h5> |
| <p>Apache Storm integrates with any queueing system and any database system. Apache Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Apache Storm with database systems is easy.</p> |
| </div> |
| </div> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>First Look</h5> |
| <ul class="footer-list"> |
| <li><a href="/releases/current/Rationale.html">Rationale</a></li> |
| <li><a href="/releases/current/Tutorial.html">Tutorial</a></li> |
| <li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li> |
| <li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Apache Storm project</a></li> |
| </ul> |
| </div> |
| </div> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>Documentation</h5> |
| <ul class="footer-list"> |
| <li><a href="/releases/current/index.html">Index</a></li> |
| <li><a href="/releases/current/javadocs/index.html">Javadoc</a></li> |
| <li><a href="/releases/current/FAQ.html">FAQ</a></li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| <hr/> |
| <div class="row"> |
| <div class="col-md-12"> |
| <p align="center">Copyright © 2019 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved. |
| <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation. |
| <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> |
| </div> |
| </div> |
| </div> |
| </footer> |
| <!--Footer End--> |
| <!-- Scroll to top --> |
| <span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span> |
| |
| </body> |
| |
| </html> |
| |
