| <!DOCTYPE html> |
| <html> |
| <head> |
| <meta charset="utf-8"> |
| <meta http-equiv="X-UA-Compatible" content="IE=edge"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| |
| <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon"> |
| <link rel="icon" href="/favicon.ico" type="image/x-icon"> |
| |
| <title>Troubleshooting</title> |
| |
| <!-- Bootstrap core CSS --> |
| <link href="/assets/css/bootstrap.min.css" rel="stylesheet"> |
| <!-- Bootstrap theme --> |
| <link href="/assets/css/bootstrap-theme.min.css" rel="stylesheet"> |
| |
| <!-- Custom styles for this template --> |
| <link rel="stylesheet" href="http://fortawesome.github.io/Font-Awesome/assets/font-awesome/css/font-awesome.css"> |
| <link href="/css/style.css" rel="stylesheet"> |
| <link href="/assets/css/owl.theme.css" rel="stylesheet"> |
| <link href="/assets/css/owl.carousel.css" rel="stylesheet"> |
| <script type="text/javascript" src="/assets/js/jquery.min.js"></script> |
| <script type="text/javascript" src="/assets/js/bootstrap.min.js"></script> |
| <script type="text/javascript" src="/assets/js/owl.carousel.min.js"></script> |
| <script type="text/javascript" src="/assets/js/storm.js"></script> |
| <!-- Just for debugging purposes. Don't actually copy these 2 lines! --> |
| <!--[if lt IE 9]><script src="../../assets/js/ie8-responsive-file-warning.js"></script><![endif]--> |
| |
| <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries --> |
| <!--[if lt IE 9]> |
| <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script> |
| <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script> |
| <![endif]--> |
| </head> |
| |
| |
| <body> |
| <header> |
| <div class="container-fluid"> |
| <div class="row"> |
| <div class="col-md-10"> |
| <a href="/index.html"><img src="/images/logo.png" class="logo" /></a> |
| </div> |
| <div class="col-md-2"> |
| <a href="/downloads.html" class="btn-std btn-block btn-download">Download</a> |
| </div> |
| </div> |
| </div> |
| </header> |
| <!--Header End--> |
| <!--Navigation Begin--> |
| <div class="navbar" role="banner"> |
| <div class="container-fluid"> |
| <div class="navbar-header"> |
| <button class="navbar-toggle" type="button" data-toggle="collapse" data-target=".bs-navbar-collapse"> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| <span class="icon-bar"></span> |
| </button> |
| </div> |
| <nav class="collapse navbar-collapse bs-navbar-collapse" role="navigation"> |
| <ul class="nav navbar-nav"> |
| <li><a href="/index.html" id="home">Home</a></li> |
| <li><a href="/getting-help.html" id="getting-help">Getting Help</a></li> |
| <li><a href="/about/integrates.html" id="project-info">Project Information</a></li> |
| <li><a href="/documentation.html" id="documentation">Documentation</a></li> |
| <li><a href="/talksAndVideos.html">Talks and Slideshows</a></li> |
| <li class="dropdown"> |
| <a href="#" class="dropdown-toggle" data-toggle="dropdown" id="contribute">Community <b class="caret"></b></a> |
| <ul class="dropdown-menu"> |
| <li><a href="/contribute/Contributing-to-Storm.html">Contributing</a></li> |
| <li><a href="/contribute/People.html">People</a></li> |
| <li><a href="/contribute/BYLAWS.html">ByLaws</a></li> |
| </ul> |
| </li> |
| <li><a href="/2015/11/05/storm096-released.html" id="news">News</a></li> |
| </ul> |
| </nav> |
| </div> |
| </div> |
| |
| |
| |
| <div class="container-fluid"> |
| <h1 class="page-title">Troubleshooting</h1> |
| <div class="row"> |
| <div class="col-md-12"> |
| <!-- Documentation --> |
| |
| <p class="post-meta"></p> |
| |
| <p>This page lists issues people have run into when using Storm along with their solutions.</p> |
| |
| <h3 id="worker-processes-are-crashing-on-startup-with-no-stack-trace">Worker processes are crashing on startup with no stack trace</h3> |
| |
| <p>Possible symptoms:</p> |
| |
| <ul> |
| <li>Topologies work with one node, but workers crash with multiple nodes</li> |
| </ul> |
| |
| <p>Solutions:</p> |
| |
| <ul> |
| <li>You may have a misconfigured subnet, where nodes can't locate other nodes based on their hostname. ZeroMQ sometimes crashes the process when it can't resolve a host. There are two solutions: |
| |
| <ul> |
| <li>Make a mapping from hostname to IP address in /etc/hosts</li> |
| <li>Set up an internal DNS so that nodes can locate each other based on hostname.</li> |
| </ul></li> |
| </ul> |
| |
| <h3 id="nodes-are-unable-to-communicate-with-each-other">Nodes are unable to communicate with each other</h3> |
| |
| <p>Possible symptoms:</p> |
| |
| <ul> |
| <li>Every spout tuple is failing</li> |
| <li>Processing is not working</li> |
| </ul> |
| |
| <p>Solutions:</p> |
| |
| <ul> |
| <li>Storm doesn't work with ipv6. You can force ipv4 by adding <code>-Djava.net.preferIPv4Stack=true</code> to the supervisor child options and restarting the supervisor. </li> |
| <li>You may have a misconfigured subnet. See the solutions for <code>Worker processes are crashing on startup with no stack trace</code></li> |
| </ul> |
| |
| <h3 id="topology-stops-processing-tuples-after-awhile">Topology stops processing tuples after awhile</h3> |
| |
| <p>Symptoms:</p> |
| |
| <ul> |
| <li>Processing works fine for awhile, and then suddenly stops and spout tuples start failing en masse. </li> |
| </ul> |
| |
| <p>Solutions:</p> |
| |
| <ul> |
| <li>This is a known issue with ZeroMQ 2.1.10. Downgrade to ZeroMQ 2.1.7.</li> |
| </ul> |
| |
| <h3 id="not-all-supervisors-appear-in-storm-ui">Not all supervisors appear in Storm UI</h3> |
| |
| <p>Symptoms:</p> |
| |
| <ul> |
| <li>Some supervisor processes are missing from the Storm UI</li> |
| <li>List of supervisors in Storm UI changes on refreshes</li> |
| </ul> |
| |
| <p>Solutions:</p> |
| |
| <ul> |
| <li>Make sure the supervisor local dirs are independent (e.g., not sharing a local dir over NFS)</li> |
| <li>Try deleting the local dirs for the supervisors and restarting the daemons. Supervisors create a unique id for themselves and store it locally. When that id is copied to other nodes, Storm gets confused. </li> |
| </ul> |
| |
| <h3 id="quot-multiple-defaults-yaml-found-quot-error">"Multiple defaults.yaml found" error</h3> |
| |
| <p>Symptoms:</p> |
| |
| <ul> |
| <li>When deploying a topology with "storm jar", you get this error</li> |
| </ul> |
| |
| <p>Solution:</p> |
| |
| <ul> |
| <li>You're most likely including the Storm jars inside your topology jar. When packaging your topology jar, don't include the Storm jars as Storm will put those on the classpath for you.</li> |
| </ul> |
| |
| <h3 id="quot-nosuchmethoderror-quot-when-running-storm-jar">"NoSuchMethodError" when running storm jar</h3> |
| |
| <p>Symptoms:</p> |
| |
| <ul> |
| <li>When running storm jar, you get a cryptic "NoSuchMethodError"</li> |
| </ul> |
| |
| <p>Solution:</p> |
| |
| <ul> |
| <li>You're deploying your topology with a different version of Storm than you built your topology against. Make sure the storm client you use comes from the same version as the version you compiled your topology against.</li> |
| </ul> |
| |
| <h3 id="kryo-concurrentmodificationexception">Kryo ConcurrentModificationException</h3> |
| |
| <p>Symptoms:</p> |
| |
| <ul> |
| <li>At runtime, you get a stack trace like the following:</li> |
| </ul> |
| <div class="highlight"><pre><code class="language-" data-lang="">java.lang.RuntimeException: java.util.ConcurrentModificationException |
| at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84) |
| at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:55) |
| at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:56) |
| at backtype.storm.disruptor$consume_loop_STAR_$fn__1597.invoke(disruptor.clj:67) |
| at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377) |
| at clojure.lang.AFn.run(AFn.java:24) |
| at java.lang.Thread.run(Thread.java:679) |
| Caused by: java.util.ConcurrentModificationException |
| at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:390) |
| at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:409) |
| at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:408) |
| at java.util.HashMap.writeObject(HashMap.java:1016) |
| at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) |
| at java.lang.reflect.Method.invoke(Method.java:616) |
| at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:959) |
| at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1480) |
| at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416) |
| at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174) |
| at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346) |
| at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21) |
| at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554) |
| at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77) |
| at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18) |
| at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472) |
| at backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27) |
| </code></pre></div> |
| <p>Solution: </p> |
| |
| <ul> |
| <li>This means that you're emitting a mutable object as an output tuple. Everything you emit into the output collector must be immutable. What's happening is that your bolt is modifying the object while it is being serialized to be sent over the network.</li> |
| </ul> |
| |
| <h3 id="nullpointerexception-from-deep-inside-storm">NullPointerException from deep inside Storm</h3> |
| |
| <p>Symptoms:</p> |
| |
| <ul> |
| <li>You get a NullPointerException that looks something like:</li> |
| </ul> |
| <div class="highlight"><pre><code class="language-" data-lang="">java.lang.RuntimeException: java.lang.NullPointerException |
| at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84) |
| at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:55) |
| at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:56) |
| at backtype.storm.disruptor$consume_loop_STAR_$fn__1596.invoke(disruptor.clj:67) |
| at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377) |
| at clojure.lang.AFn.run(AFn.java:24) |
| at java.lang.Thread.run(Thread.java:662) |
| Caused by: java.lang.NullPointerException |
| at backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:24) |
| at backtype.storm.daemon.worker$mk_transfer_fn$fn__4126$fn__4130.invoke(worker.clj:99) |
| at backtype.storm.util$fast_list_map.invoke(util.clj:771) |
| at backtype.storm.daemon.worker$mk_transfer_fn$fn__4126.invoke(worker.clj:99) |
| at backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3904.invoke(executor.clj:205) |
| at backtype.storm.disruptor$clojure_handler$reify__1584.onEvent(disruptor.clj:43) |
| at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:81) |
| ... 6 more |
| </code></pre></div> |
| <p>or </p> |
| <div class="highlight"><pre><code class="language-" data-lang="">java.lang.RuntimeException: java.lang.NullPointerException |
| at |
| backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) |
| ~[storm-core-0.9.3.jar:0.9.3] |
| at |
| backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) |
| ~[storm-core-0.9.3.jar:0.9.3] |
| at |
| backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) |
| ~[storm-core-0.9.3.jar:0.9.3] |
| at |
| backtype.storm.disruptor$consume_loop_STAR_$fn__759.invoke(disruptor.clj:94) |
| ~[storm-core-0.9.3.jar:0.9.3] |
| at backtype.storm.util$async_loop$fn__458.invoke(util.clj:463) |
| ~[storm-core-0.9.3.jar:0.9.3] |
| at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] |
| at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] |
| Caused by: java.lang.NullPointerException: null |
| at clojure.lang.RT.intCast(RT.java:1087) ~[clojure-1.5.1.jar:na] |
| at |
| backtype.storm.daemon.worker$mk_transfer_fn$fn__3548.invoke(worker.clj:129) |
| ~[storm-core-0.9.3.jar:0.9.3] |
| at |
| backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3282.invoke(executor.clj:258) |
| ~[storm-core-0.9.3.jar:0.9.3] |
| at |
| backtype.storm.disruptor$clojure_handler$reify__746.onEvent(disruptor.clj:58) |
| ~[storm-core-0.9.3.jar:0.9.3] |
| at |
| backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) |
| ~[storm-core-0.9.3.jar:0.9.3] |
| ... 6 common frames omitted |
| </code></pre></div> |
| <p>Solution:</p> |
| |
| <ul> |
| <li>This is caused by having multiple threads issue methods on the <code>OutputCollector</code>. All emits, acks, and fails must happen on the same thread. One subtle way this can happen is if you make a <code>IBasicBolt</code> that emits on a separate thread. <code>IBasicBolt</code>'s automatically ack after execute is called, so this would cause multiple threads to use the <code>OutputCollector</code> leading to this exception. When using a basic bolt, all emits must happen in the same thread that runs <code>execute</code>.</li> |
| </ul> |
| |
| |
| |
| </div> |
| </div> |
| </div> |
| <footer> |
| <div class="container-fluid"> |
| <div class="row"> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>Meetups</h5> |
| <ul class="latest-news"> |
| |
| <li><a href="http://www.meetup.com/Apache-Storm-Apache-Kafka/">Apache Storm & Apache Kafka</a> <span class="small">(Sunnyvale, CA)</span></li> |
| |
| <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Apache Storm & Kafka Users</a> <span class="small">(Seattle, WA)</span></li> |
| |
| <li><a href="http://www.meetup.com/New-York-City-Storm-User-Group/">NYC Storm User Group</a> <span class="small">(New York, NY)</span></li> |
| |
| <li><a href="http://www.meetup.com/Bay-Area-Stream-Processing">Bay Area Stream Processing</a> <span class="small">(Emeryville, CA)</span></li> |
| |
| <li><a href="http://www.meetup.com/Boston-Storm-Users/">Boston Realtime Data</a> <span class="small">(Boston, MA)</span></li> |
| |
| <li><a href="http://www.meetup.com/storm-london">London Storm User Group</a> <span class="small">(London, UK)</span></li> |
| |
| <!-- <li><a href="http://www.meetup.com/Apache-Storm-Kafka-Users/">Seatle, WA</a> <span class="small">(27 Jun 2015)</span></li> --> |
| </ul> |
| </div> |
| </div> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>About Storm</h5> |
| <p>Storm integrates with any queueing system and any database system. Storm's spout abstraction makes it easy to integrate a new queuing system. Likewise, integrating Storm with database systems is easy.</p> |
| </div> |
| </div> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>First Look</h5> |
| <ul class="footer-list"> |
| <li><a href="/documentation/Rationale.html">Rationale</a></li> |
| <li><a href="/tutorial.html">Tutorial</a></li> |
| <li><a href="/documentation/Setting-up-development-environment.html">Setting up development environment</a></li> |
| <li><a href="/documentation/Creating-a-new-Storm-project.html">Creating a new Storm project</a></li> |
| </ul> |
| </div> |
| </div> |
| <div class="col-md-3"> |
| <div class="footer-widget"> |
| <h5>Documentation</h5> |
| <ul class="footer-list"> |
| <li><a href="/doc-index.html">Index</a></li> |
| <li><a href="/documentation.html">Manual</a></li> |
| <li><a href="https://storm.apache.org/javadoc/apidocs/index.html">Javadoc</a></li> |
| <li><a href="/documentation/FAQ.html">FAQ</a></li> |
| </ul> |
| </div> |
| </div> |
| </div> |
| <hr/> |
| <div class="row"> |
| <div class="col-md-12"> |
| <p align="center">Copyright © 2015 <a href="http://www.apache.org">Apache Software Foundation</a>. All Rights Reserved. |
| <br>Apache Storm, Apache, the Apache feather logo, and the Apache Storm project logos are trademarks of The Apache Software Foundation. |
| <br>All other marks mentioned may be trademarks or registered trademarks of their respective owners.</p> |
| </div> |
| </div> |
| </div> |
| </footer> |
| <!--Footer End--> |
| <!-- Scroll to top --> |
| <span class="totop"><a href="#"><i class="fa fa-angle-up"></i></a></span> |
| |
| </body> |
| |
| </html> |
| |
