Google Git
Sign in
apache / storm-site / refs/heads/asf-site / . / content / releases / 2.3.0 / storm-opentsdb.html
blob: 938801e66e7eecb2f471c22b98048cb526808a15 [file] [log] [blame]
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
<link rel="icon" href="/favicon.ico" type="image/x-icon">
<title>Storm OpenTSDB Bolt and TridentState</title>
<!-- Bootstrap core CSS -->
<link href="/assets/css/bootstrap.min.css" rel="stylesheet">
<!-- Bootstrap theme -->
<link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet">
<!-- Custom styles for this template -->
<link href="/css/font-awesome.min.css" rel="stylesheet">
<link href="/css/style.css" rel="stylesheet">
<link href="/assets/css/owl.theme.css" rel="stylesheet">
<link href="/assets/css/owl.carousel.css" rel="stylesheet">
<script type="text/javascript" src="/assets/js/jquery.min.js"></script>
<script type="text/javascript" src="/assets/js/bootstrap.min.js"></script>
<script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script>
<script type="text/javascript" src="/assets/js/storm.js"></script>
</head>
<body>
<header>
<div class="container-fluid">
<div class="row">
<div class="col-md-5">
<a href="/index.html"><img src="/images/logo.png" class="logo"/></a>
</div>
<div class="col-md-5">
<h1>Version: 2.3.0</h1>
</div>
<div class="col-md-2">
<a href="/downloads.html" class="btn-std btn-block btn-download">Download</a>
</div>
</div>
</div>
</header>
<!--Header End-->
<!--Navigation Begin-->
<div class="navbar" role="banner">
<div class="container-fluid">
<div class="navbar-header">
<button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
</div>
<nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation">
<ul class="nav navbar-nav">
<li><a href="/index.html" id="home">Home</a></li>
<li><a href="/getting-help.html" id="getting-help">Getting Help</a></li>
<li><a href="/about/integrates.html" id="project-info">Project Information</a></li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b
class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="/releases/2.6.2/index.html">2.6.2</a></li>
<li><a href="/releases/2.6.1/index.html">2.6.1</a></li>
<li><a href="/releases/2.6.0/index.html">2.6.0</a></li>
<li><a href="/releases/2.5.0/index.html">2.5.0</a></li>
<li><a href="/releases/2.4.0/index.html">2.4.0</a></li>
<li><a href="/releases/2.3.0/index.html">2.3.0</a></li>
<li><a href="/releases/2.2.1/index.html">2.2.1</a></li>
<li><a href="/releases/2.2.0/index.html">2.2.0</a></li>
<li><a href="/releases/2.1.1/index.html">2.1.1</a></li>
<li><a href="/releases/2.1.0/index.html">2.1.0</a></li>
<li><a href="/releases/2.0.0/index.html">2.0.0</a></li>
<li><a href="/releases/1.2.4/index.html">1.2.4</a></li>
<li><a href="/releases/1.2.3/index.html">1.2.3</a></li>
</ul>
</li>
<li><a href="/talksAndVideos.html">Talks and Slideshows</a></li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b
class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li>
<li><a href="/contribute/People.html">People</a></li>
<li><a href="/contribute/BYLAWS.html">ByLaws</a></li>
<li><a href="/Powered-By.html">PoweredBy</a></li>
</ul>
</li>
<li><a href="/2024/04/05/storm262-released.html" id="news">News</a></li>
</ul>
</nav>
</div>
</div>
<div class="container-fluid">
<h1 class="page-title">Storm OpenTSDB Bolt and TridentState</h1>
<div class="row">
<div class="col-md-12">
<!-- Documentation -->
<p class="post-meta"></p>
<div class="documentation-content"><p>OpenTSDB offers a scalable and highly available storage for time series data. It consists of a
Time Series Daemon (TSD) servers along with command line utilities. Each TSD connects to the
configured HBase cluster to push/query the data.</p>
<p>Time series data point consists of:
- a metric name.
- a UNIX timestamp (seconds or milliseconds since Epoch).
- a value (64 bit integer or single-precision floating point value).
- a set of tags (key-value pairs) that describe the time series the point belongs to.</p>
<p>Storm bolt and trident state creates the above time series data from a tuple based on the given <code>TupleMetricPointMapper</code></p>
<p>This module provides core Storm and Trident bolt implementations for writing data to OpenTSDB. </p>
<p>Time series data points are written with at-least-once guarantee and duplicate data points should be handled as mentioned <a href="http://opentsdb.net/docs/build/html/user_guide/writing.html#duplicate-data-points">here</a> in OpenTSDB. </p>
<h2 id="examples">Examples</h2>
<h3 id="core-bolt">Core Bolt</h3>
<p>Below example describes the usage of core bolt which is <code>org.apache.storm.opentsdb.bolt.OpenTsdbBolt</code></p>
<div class="highlight"><pre><code class="language-java" data-lang="java">
<span class="nc">OpenTsdbClient</span><span class="o">.</span><span class="na">Builder</span> <span class="n">builder</span> <span class="o">=</span> <span class="nc">OpenTsdbClient</span><span class="o">.</span><span class="na">newBuilder</span><span class="o">(</span><span class="n">openTsdbUrl</span><span class="o">).</span><span class="na">sync</span><span class="o">(</span><span class="mi">30_000</span><span class="o">).</span><span class="na">returnDetails</span><span class="o">();</span>
<span class="kd">final</span> <span class="nc">OpenTsdbBolt</span> <span class="n">openTsdbBolt</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">OpenTsdbBolt</span><span class="o">(</span><span class="n">builder</span><span class="o">,</span> <span class="nc">Collections</span><span class="o">.</span><span class="na">singletonList</span><span class="o">(</span><span class="nc">TupleOpenTsdbDatapointMapper</span><span class="o">.</span><span class="na">DEFAULT_MAPPER</span><span class="o">));</span>
<span class="n">openTsdbBolt</span><span class="o">.</span><span class="na">withBatchSize</span><span class="o">(</span><span class="mi">10</span><span class="o">).</span><span class="na">withFlushInterval</span><span class="o">(</span><span class="mi">2000</span><span class="o">);</span>
<span class="n">topologyBuilder</span><span class="o">.</span><span class="na">setBolt</span><span class="o">(</span><span class="s">"opentsdb"</span><span class="o">,</span> <span class="n">openTsdbBolt</span><span class="o">).</span><span class="na">shuffleGrouping</span><span class="o">(</span><span class="s">"metric-gen"</span><span class="o">);</span>
</code></pre></div>
<h3 id="trident-state">Trident State</h3>
<div class="highlight"><pre><code class="language-java" data-lang="java">
<span class="kd">final</span> <span class="nc">OpenTsdbStateFactory</span> <span class="n">openTsdbStateFactory</span> <span class="o">=</span>
<span class="k">new</span> <span class="nf">OpenTsdbStateFactory</span><span class="o">(</span><span class="nc">OpenTsdbClient</span><span class="o">.</span><span class="na">newBuilder</span><span class="o">(</span><span class="n">tsdbUrl</span><span class="o">),</span>
<span class="nc">Collections</span><span class="o">.</span><span class="na">singletonList</span><span class="o">(</span><span class="nc">TupleOpenTsdbDatapointMapper</span><span class="o">.</span><span class="na">DEFAULT_MAPPER</span><span class="o">));</span>
<span class="nc">TridentTopology</span> <span class="n">tridentTopology</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">TridentTopology</span><span class="o">();</span>
<span class="kd">final</span> <span class="nc">Stream</span> <span class="n">stream</span> <span class="o">=</span> <span class="n">tridentTopology</span><span class="o">.</span><span class="na">newStream</span><span class="o">(</span><span class="s">"metric-tsdb-stream"</span><span class="o">,</span> <span class="k">new</span> <span class="nc">MetricGenSpout</span><span class="o">());</span>
<span class="n">stream</span><span class="o">.</span><span class="na">peek</span><span class="o">(</span><span class="k">new</span> <span class="nc">Consumer</span><span class="o">()</span> <span class="o">{</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">accept</span><span class="o">(</span><span class="nc">TridentTuple</span> <span class="n">input</span><span class="o">)</span> <span class="o">{</span>
<span class="no">LOG</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">"########### Received tuple: [{}]"</span><span class="o">,</span> <span class="n">input</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}).</span><span class="na">partitionPersist</span><span class="o">(</span><span class="n">openTsdbStateFactory</span><span class="o">,</span> <span class="nc">MetricGenSpout</span><span class="o">.</span><span class="na">DEFAULT_METRIC_FIELDS</span><span class="o">,</span> <span class="k">new</span> <span class="nc">OpenTsdbStateUpdater</span><span class="o">());</span>
</code></pre></div></div>
</div>
</div>
</div>
<footer>
<div class="container-fluid">
<div class="row">
<div class="col-md-2">
<div class="footer-widget">
<h5>Meetups</h5>
<div class="footer-widget">
<a class="acevent" data-format="wide" data-mode="dark"></a>
</div>
</div>
</div>
<div class="col-md-4">
<div class="footer-widget">
<h5>About Apache Storm</h5>
<p>Apache Storm integrates with any queueing system and any database system. Apache Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Apache Storm with database systems is easy.</p>
</div>
</div>
<div class="col-md-2">
<div class="footer-widget">
<h5>First Look</h5>
<ul class="footer-list">
<li><a href="/releases/current/Rationale.html">Rationale</a></li>
<li><a href="/releases/current/Tutorial.html">Tutorial</a></li>
<li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li>
<li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Apache Storm project</a></li>
</ul>
</div>
</div>
<div class="col-md-2">
<div class="footer-widget">
<h5>Documentation</h5>
<ul class="footer-list">
<li><a href="/releases/current/index.html">Index</a></li>
<li><a href="/releases/current/javadocs/index.html">Javadoc</a></li>
<li><a href="/releases/current/FAQ.html">FAQ</a></li>
</ul>
</div>
</div>
<div class="col-md-2">
<div class="footer-widget">
<h5>Misc</h5>
<ul class="footer-list">
<li><a href="https://www.apache.org/licenses/">Licenses</a></li>
<li><a href="https://www.apache.org/security/">Security</a></li>
<li><a href="https://www.apache.org/foundation/thanks.html">Sponsors</a></li>
<li><a href="https://privacy.apache.org/policies/privacy-policy-public.html">Privacy</a></li>
</ul>
</div>
</div>
</div>
<hr/>
<div class="row">
<div class="col-md-12">
<p align="center">Copyright © 2024 <a href="https://www.apache.org">Apache Software Foundation</a>
. All Rights Reserved.
<br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation.
<br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
</div>
</div>
</div>
</footer>
<!--Footer End-->
<!-- Matomo -->
<script>
var _paq = window._paq = window._paq || [];
/* tracker methods like "setCustomDimension" should be called before "trackPageView" */
/* We explicitly disable cookie tracking to avoid privacy issues */
_paq.push(['disableCookies']);
_paq.push(['trackPageView']);
_paq.push(['enableLinkTracking']);
(function() {
var u="//analytics.apache.org/";
_paq.push(['setTrackerUrl', u+'matomo.php']);
_paq.push(['setSiteId', '38']);
var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s);
})();
</script>
<!-- End Matomo Code -->
<script src="https://www.apachecon.com/event-images/snippet.js"></script>
<!-- Scroll to top -->
<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span>
</body>
</html>