blob: 34b5ec853c248e728848ad5aa6c85137247765b2 [file] [log] [blame]
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
<link rel="icon" href="/favicon.ico" type="image/x-icon">
<title>Apache Storm</title>
<!-- Bootstrap core CSS -->
<link href="/assets/css/bootstrap.min.css" rel="stylesheet">
<!-- Bootstrap theme -->
<link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet">
<!-- Custom styles for this template -->
<link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css">
<link href="/css/style.css" rel="stylesheet">
<link href="/assets/css/owl.theme.css" rel="stylesheet">
<link href="/assets/css/owl.carousel.css" rel="stylesheet">
<script type="text/javascript" src="/assets/js/jquery.min.js"></script>
<script type="text/javascript" src="/assets/js/bootstrap.min.js"></script>
<script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script>
<script type="text/javascript" src="/assets/js/storm.js"></script>
<!-- Just for debugging purposes. Don't actually copy these 2 lines! -->
<!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]-->
<!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->
</head>
<body>
<header>
<div class="container-fluid">
<div class="row">
<div class="col-md-5">
<a href="/index.html"><img src="/images/logo.png" class="logo" /></a>
</div>
<div class="col-md-5">
<h1>Version: 2.0.0</h1>
</div>
<div class="col-md-2">
<a href="/downloads.html" class="btn-std btn-block btn-download">Download</a>
</div>
</div>
</div>
</header>
<!--Header End-->
<!--Navigation Begin-->
<div class="navbar" role="banner">
<div class="container-fluid">
<div class="navbar-header">
<button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
</div>
<nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation">
<ul class="nav navbar-nav">
<li><a href="/index.html" id="home">Home</a></li>
<li><a href="/getting-help.html" id="getting-help">Getting Help</a></li>
<li><a href="/about/integrates.html" id="project-info">Project Information</a></li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" id="documentation">Documentation <b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="/releases/2.3.0/index.html">2.3.0</a></li>
<li><a href="/releases/2.2.0/index.html">2.2.0</a></li>
<li><a href="/releases/2.1.0/index.html">2.1.0</a></li>
<li><a href="/releases/2.0.0/index.html">2.0.0</a></li>
<li><a href="/releases/1.2.3/index.html">1.2.3</a></li>
</ul>
</li>
<li><a href="/talksAndVideos.html">Talks and Slideshows</a></li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a>
<ul class="dropdown-menu">
<li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li>
<li><a href="/contribute/People.html">People</a></li>
<li><a href="/contribute/BYLAWS.html">ByLaws</a></li>
</ul>
</li>
<li><a href="/2021/09/27/storm230-released.html" id="news">News</a></li>
</ul>
</nav>
</div>
</div>
<div class="container-fluid">
<h1 class="page-title"></h1>
<div class="row">
<div class="col-md-12">
<!-- Documentation -->
<p class="post-meta"></p>
<div class="documentation-content"><p>This page explains the multilang protocol for versions 0.7.0 and below. The protocol changed in version 0.7.1.</p>
<h1 id="storm-multi-language-protocol">Storm Multi-Language Protocol</h1>
<h2 id="the-shellbolt">The ShellBolt</h2>
<p>Support for multiple languages is implemented via the ShellBolt class. This
class implements the IBolt interfaces and implements the protocol for
executing a script or program via the shell using Java&#39;s ProcessBuilder class.</p>
<h2 id="output-fields">Output fields</h2>
<p>Output fields are part of the Thrift definition of the topology. This means that when you multilang in Java, you need to create a bolt that extends ShellBolt, implements IRichBolt, and declared the fields in <code>declareOutputFields</code>.
You can learn more about this on <a href="Concepts.html">Concepts</a></p>
<h2 id="protocol-preamble">Protocol Preamble</h2>
<p>A simple protocol is implemented via the STDIN and STDOUT of the executed
script or program. A mix of simple strings and JSON encoded data are exchanged
with the process making support possible for pretty much any language.</p>
<h1 id="packaging-your-stuff">Packaging Your Stuff</h1>
<p>To run a ShellBolt on a cluster, the scripts that are shelled out to must be
in the <code>resources/</code> directory within the jar submitted to the master.</p>
<p>However, During development or testing on a local machine, the resources
directory just needs to be on the classpath.</p>
<h2 id="the-protocol">The Protocol</h2>
<p>Notes:
* Both ends of this protocol use a line-reading mechanism, so be sure to
trim off newlines from the input and to append them to your output.
* All JSON inputs and outputs are terminated by a single line contained &quot;end&quot;.
* The bullet points below are written from the perspective of the script writer&#39;s
STDIN and STDOUT.</p>
<ul>
<li>Your script will be executed by the Bolt.</li>
<li>STDIN: A string representing a path. This is a PID directory.
Your script should create an empty file named with it&#39;s pid in this directory. e.g.
the PID is 1234, so an empty file named 1234 is created in the directory. This
file lets the supervisor know the PID so it can shutdown the process later on.</li>
<li>STDOUT: Your PID. This is not JSON encoded, just a string. ShellBolt will log the PID to its log.</li>
<li>STDIN: (JSON) The Storm configuration. Various settings and properties.</li>
<li>STDIN: (JSON) The Topology context</li>
<li>The rest happens in a while(true) loop</li>
<li>STDIN: A tuple! This is a JSON encoded structure like this:</li>
</ul>
<div class="highlight"><pre><code class="language-" data-lang="">{
// The tuple's id
"id": -6955786537413359385,
// The id of the component that created this tuple
"comp": 1,
// The id of the stream this tuple was emitted to
"stream": 1,
// The id of the task that created this tuple
"task": 9,
// All the values in this tuple
"tuple": ["snow white and the seven dwarfs", "field2", 3]
}
</code></pre></div>
<ul>
<li>STDOUT: The results of your bolt, JSON encoded. This can be a sequence of acks, fails, emits, and/or logs. Emits look like:</li>
</ul>
<div class="highlight"><pre><code class="language-" data-lang="">{
"command": "emit",
// The ids of the tuples this output tuples should be anchored to
"anchors": [1231231, -234234234],
// The id of the stream this tuple was emitted to. Leave this empty to emit to default stream.
"stream": 1,
// If doing an emit direct, indicate the task to sent the tuple to
"task": 9,
// All the values in this tuple
"tuple": ["field1", 2, 3]
}
</code></pre></div>
<p>An ack looks like:</p>
<div class="highlight"><pre><code class="language-" data-lang="">{
"command": "ack",
// the id of the tuple to ack
"id": 123123
}
</code></pre></div>
<p>A fail looks like:</p>
<div class="highlight"><pre><code class="language-" data-lang="">{
"command": "fail",
// the id of the tuple to fail
"id": 123123
}
</code></pre></div>
<p>A &quot;log&quot; will log a message in the worker log. It looks like:</p>
<div class="highlight"><pre><code class="language-" data-lang="">{
"command": "log",
// the message to log
"msg": "hello world!"
}
</code></pre></div>
<ul>
<li>STDOUT: emit &quot;sync&quot; as a single line by itself when the bolt has finished emitting/acking/failing and is ready for the next input</li>
</ul>
<h3 id="sync">sync</h3>
<p>Note: This command is not JSON encoded, it is sent as a simple string.</p>
<p>This lets the parent bolt know that the script has finished processing and is ready for another tuple.</p>
</div>
</div>
</div>
</div>
<footer>
<div class="container-fluid">
<div class="row">
<div class="col-md-3">
<div class="footer-widget">
<h5>Meetups</h5>
<ul class="latest-news">
<li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li>
<li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li>
<li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li>
<li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li>
<li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li>
<li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li>
<!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> -->
</ul>
</div>
</div>
<div class="col-md-3">
<div class="footer-widget">
<h5>About Apache Storm</h5>
<p>Apache Storm integrates with any queueing system and any database system. Apache Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Apache Storm with database systems is easy.</p>
</div>
</div>
<div class="col-md-3">
<div class="footer-widget">
<h5>First Look</h5>
<ul class="footer-list">
<li><a href="/releases/current/Rationale.html">Rationale</a></li>
<li><a href="/releases/current/Tutorial.html">Tutorial</a></li>
<li><a href="/releases/current/Setting-up-development-environment.html">Setting up development environment</a></li>
<li><a href="/releases/current/Creating-a-new-Storm-project.html">Creating a new Apache Storm project</a></li>
</ul>
</div>
</div>
<div class="col-md-3">
<div class="footer-widget">
<h5>Documentation</h5>
<ul class="footer-list">
<li><a href="/releases/current/index.html">Index</a></li>
<li><a href="/releases/current/javadocs/index.html">Javadoc</a></li>
<li><a href="/releases/current/FAQ.html">FAQ</a></li>
</ul>
</div>
</div>
</div>
<hr/>
<div class="row">
<div class="col-md-12">
<p align="center">Copyright © 2019 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved.
<br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation.
<br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p>
</div>
</div>
</div>
</footer>
<!--Footer End-->
<!-- Scroll to top -->
<span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span>
</body>
</html>