Google Git
Sign in
apache / storm-site / b711e70434639e86ade5bfc8a72a132abcb3079a / . / content / releases / current / Eventlogging.html
blob: 15cafee76323c0815af6741169063844a66ff773 [file] [log] [blame]
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
<link rel="icon" href="/favicon.ico" type="image/x-icon">
<title>Topology event inspector</title>
<!-- Bootstrap core CSS -->
<link href="/assets/css/bootstrap.min.css" rel="stylesheet">
<!-- Bootstrap theme -->
<link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet">
<!-- Custom styles for this template -->
<link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css">
<link href="/css/style.css" rel="stylesheet">
<link href="/assets/css/owl.theme.css" rel="stylesheet">
<link href="/assets/css/owl.carousel.css" rel="stylesheet">
<script type="text/javascript" src="/assets/js/jquery.min.js"></script>
<script type="text/javascript" src="/assets/js/bootstrap.min.js"></script>
<script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script>
<script type="text/javascript" src="/assets/js/storm.js"></script>
<!-- Just for debugging purposes. Don't actually copy these 2 lines! -->
<!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]-->
<!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->
</head>
<body>
<header>
<div class="container-fluid">
<div class="row">
<div class="col-md-5">
<a href="/index.html"><img src="/images/logo.png" class="logo" /></a>
</div>
<div class="col-md-5">
<h1>Version: 2.3.0</h1>
</div>
<div class="col-md-2">
<a href="/downloads.html" class="btn-std btn-block btn-download">Download</a>
</div>
</div>
</div>
</header>
<!--Header End-->
<!--Navigation Begin-->
<div class="navbar" role="banner">
<div class="container-fluid">
<div class="navbar-header">
<button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
</div>
<nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation">
<ul class="nav navbar-nav">
<li><a href="/index.html" id="home">Home</a></li>
<li><a href="/getting-help.html" id="getting-help">Getting Help</a></li>
<li><a href="/about/integrates.html" id="project-info">Project Information</a></li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="/releases/2.3.0/index.html">2.3.0</a></li>
<li><a href="/releases/2.2.0/index.html">2.2.0</a></li>
<li><a href="/releases/2.1.0/index.html">2.1.0</a></li>
<li><a href="/releases/2.0.0/index.html">2.0.0</a></li>
<li><a href="/releases/1.2.3/index.html">1.2.3</a></li>
</ul>
</li>
<li><a href="/talksAndVideos.html">Talks and Slideshows</a></li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li>
<li><a href="/contribute/People.html">People</a></li>
<li><a href="/contribute/BYLAWS.html">ByLaws</a></li>
</ul>
</li>
<li><a href="/2021/09/27/storm230-released.html" id="news">News</a></li>
</ul>
</nav>
</div>
</div>
<div class="container-fluid">
<h1 class="page-title">Topology event inspector</h1>
<div class="row">
<div class="col-md-12">
<!-- Documentation -->
<p class="post-meta"></p>
<div class="documentation-content"><h1 id="introduction">Introduction</h1>
<p>Topology event inspector provides the ability to view the tuples as it flows through different stages in a storm topology.
This could be useful for inspecting the tuples emitted at a spout or a bolt in the topology pipeline while the topology is running, without stopping or redeploying the topology. The normal flow of tuples from the spouts to the bolts is not affected by turning on event logging.</p>
<h2 id="enabling-event-logging">Enabling event logging</h2>
<p>Note: Event logging needs to be enabled first by setting the storm config &quot;topology.eventlogger.executors&quot; to a non zero value. Please see
the <a href="#config">Configuration</a> section for more details.</p>
<p>Events can be logged by clicking the &quot;Debug&quot; button under the topology actions in the topology view. This logs the
tuples from all the spouts and bolts in a topology at the specified sampling percentage.</p>
<div align="center">
<img title="Enable Eventlogging" src="images/enable-event-logging-topology.png" style="max-width: 80rem"/>
<p>Figure 1: Enable event logging at topology level.</p>
</div>
<p>You could also enable event logging at a specific spout or bolt level by going to the corresponding component page and
clicking &quot;Debug&quot; under component actions.</p>
<div align="center">
<img title="Enable Eventlogging at component level" src="images/enable-event-logging-spout.png" style="max-width: 80rem"/>
<p>Figure 2: Enable event logging at component level.</p>
</div>
<h2 id="viewing-the-event-logs">Viewing the event logs</h2>
<p>The Storm &quot;logviewer&quot; should be running for viewing the logged tuples. If not already running log viewer can be started by running the &quot;bin/storm logviewer&quot; command from the storm installation directory. For viewing the tuples, go to the specific spout or bolt component page from storm UI and click on the &quot;events&quot; link under the component summary (as highlighted in Figure 2 above).</p>
<p>This would open up a view like below where you can navigate between different pages and view the logged tuples.</p>
<div align="center">
<img title="Viewing logged tuples" src="images/event-logs-view.png" style="max-width: 80rem"/>
<p>Figure 3: Viewing the logged events.</p>
</div>
<p>Each line in the event log contains an entry corresponding to a tuple emitted from a specific spout/bolt in a comma separated format.</p>
<p><code>Timestamp, Component name, Component task-id, MessageId (in case of anchoring), List of emitted values</code></p>
<h2 id="disabling-the-event-logs">Disabling the event logs</h2>
<p>Event logging can be disabled at a specific component or at the topology level by clicking the &quot;Stop Debug&quot; under the topology or component actions in the Storm UI.</p>
<div align="center">
<img title="Disable Eventlogging at topology level" src="images/disable-event-logging-topology.png" style="max-width: 80rem"/>
<p>Figure 4: Disable event logging at topology level.</p>
</div>
<h2 id="configuration"><a name="config"></a>Configuration</h2>
<p>Eventlogging works by sending the events (tuples) from each component to an internal eventlogger bolt. By default Storm does not start any event logger tasks, but this can be easily changed by setting the below parameter while running your topology (by setting it in storm.yaml or passing options via command line).</p>
<table><thead>
<tr>
<th>Parameter</th>
<th>Meaning</th>
</tr>
</thead><tbody>
<tr>
<td>&quot;topology.eventlogger.executors&quot;: 0</td>
<td>No event logger tasks are created (default).</td>
</tr>
<tr>
<td>&quot;topology.eventlogger.executors&quot;: 1</td>
<td>One event logger task for the topology.</td>
</tr>
<tr>
<td>&quot;topology.eventlogger.executors&quot;: nil</td>
<td>One event logger task per worker.</td>
</tr>
</tbody></table>
<h2 id="extending-eventlogging">Extending eventlogging</h2>
<p>Storm provides an <code>IEventLogger</code> interface which is used by the event logger bolt to log the events.</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="cm">/**
* EventLogger interface for logging the event info to a sink like log file or db
* for inspecting the events via UI for debugging.
*/</span>
<span class="kd">public</span> <span class="kd">interface</span> <span class="nc">IEventLogger</span> <span class="o">{</span>
<span class="cm">/**
* Invoked during eventlogger bolt prepare.
*/</span>
<span class="kt">void</span> <span class="nf">prepare</span><span class="o">(</span><span class="n">Map</span> <span class="n">stormConf</span><span class="o">,</span> <span class="n">Map</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Object</span><span class="o">&gt;</span> <span class="n">arguments</span><span class="o">,</span> <span class="n">TopologyContext</span> <span class="n">context</span><span class="o">);</span>
<span class="cm">/**
* Invoked when the {@link EventLoggerBolt} receives a tuple from the spouts or bolts that has event logging enabled.
*
* @param e the event
*/</span>
<span class="kt">void</span> <span class="nf">log</span><span class="o">(</span><span class="n">EventInfo</span> <span class="n">e</span><span class="o">);</span>
<span class="cm">/**
* Invoked when the event logger bolt is cleaned up
*/</span>
<span class="kt">void</span> <span class="nf">close</span><span class="o">();</span>
<span class="o">}</span>
</code></pre></div>
<p>The default implementation for this is a FileBasedEventLogger which logs the events to an events.log file ( <code>logs/workers-artifacts/&lt;topology-id&gt;/&lt;worker-port&gt;/events.log</code>).
Alternate implementations of the <code>IEventLogger</code> interface can be added to extend the event logging functionality (say build a search index or log the events in a database etc)</p>
<p>If you just want to use FileBasedEventLogger but with changing the log format, you can simply implement your own by extending FileBasedEventLogger and override <code>buildLogMessage(EventInfo)</code> to provide log line explicitly.</p>
<p>To register event logger to your topology, add to your topology&#39;s configuration like:</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">conf</span><span class="o">.</span><span class="na">registerEventLogger</span><span class="o">(</span><span class="n">org</span><span class="o">.</span><span class="na">apache</span><span class="o">.</span><span class="na">storm</span><span class="o">.</span><span class="na">metric</span><span class="o">.</span><span class="na">FileBasedEventLogger</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
</code></pre></div>
<p>You can refer <a href="javadocs/org/apache/storm/Config.html#registerEventLogger-java.lang.Class-">Config#registerEventLogger</a> and overloaded methods from javadoc.</p>
<p>Otherwise edit the storm.yaml config file:</p>
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="s">topology.event.logger.register</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">class</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.apache.storm.metric.FileBasedEventLogger"</span>
<span class="pi">-</span> <span class="na">class</span><span class="pi">:</span> <span class="s2">"</span><span class="s">org.mycompany.MyEventLogger"</span>
<span class="na">arguments</span><span class="pi">:</span>
<span class="na">endpoint</span><span class="pi">:</span> <span class="s2">"</span><span class="s">event-logger.mycompany.org"</span>
</code></pre></div>
<p>When you implement your own event logger, <code>arguments</code> is passed to Map<String, Object> when <a href="javadocs/org/apache/storm/metric/IEventLogger.html#prepare-java.util.Map-java.lang.Map-org.apache.storm.task.TopologyContext-">IEventLogger#prepare</a> is called.</p>
<p>Please keep in mind that EventLoggerBolt is just a kind of Bolt, so whole throughput of the topology will go down when registered event loggers cannot keep up handling incoming events, so you may want to take care of the Bolt like normal Bolt.
One of idea to avoid this is making your implementation of IEventLogger as <code>non-blocking</code> fashion.</p>
</div>
</div>
</div>
</div>
<footer>
<div class="container-fluid">
<div class="row">
<div class="col-md-3">
<div class="footer-widget">
<h5>Meetups</h5>
<ul class="latest-news">
<li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li>
<li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li>
<li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li>
<li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li>
<li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li>
<li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li>
<!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> -->
</ul>
</div>
</div>
<div class="col-md-3">
<div class="footer-widget">
<h5>About Apache Storm</h5>
<p>Apache Storm integrates with any queueing system and any database system. Apache Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Apache Storm with database systems is easy.</p>
</div>
</div>
<div class="col-md-3">
<div class="footer-widget">
<h5>First Look</h5>
<ul class="footer-list">
<li><a href="/releases/current/Rationale.html">Rationale</a></li>
<li><a href="/releases/current/Tutorial.html">Tutorial</a></li>
<li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li>
<li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Apache Storm project</a></li>
</ul>
</div>
</div>
<div class="col-md-3">
<div class="footer-widget">
<h5>Documentation</h5>
<ul class="footer-list">
<li><a href="/releases/current/index.html">Index</a></li>
<li><a href="/releases/current/javadocs/index.html">Javadoc</a></li>
<li><a href="/releases/current/FAQ.html">FAQ</a></li>
</ul>
</div>
</div>
</div>
<hr/>
<div class="row">
<div class="col-md-12">
<p align="center">Copyright © 2019 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved.
<br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation.
<br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
</div>
</div>
</div>
</footer>
<!--Footer End-->
<!-- Scroll to top -->
<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span>
</body>
</html>