This page lists issues people have run into when using Storm along with their solutions.
Possible symptoms:
Solutions:
Possible symptoms:
Solutions:
-Djava.net.preferIPv4Stack=true
to the supervisor child options and restarting the supervisor.Worker processes are crashing on startup with no stack trace
Symptoms:
Solutions:
Symptoms:
Solutions:
Symptoms:
Solution:
Symptoms:
Solution:
Symptoms:
java.lang.RuntimeException: java.util.ConcurrentModificationException at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84) at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:55) at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:56) at backtype.storm.disruptor$consume_loop_STAR_$fn__1597.invoke(disruptor.clj:67) at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377) at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:679) Caused by: java.util.ConcurrentModificationException at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:390) at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:409) at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:408) at java.util.HashMap.writeObject(HashMap.java:1016) at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:959) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1480) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346) at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554) at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77) at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18) at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472) at backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
Solution:
Symptoms:
java.lang.RuntimeException: java.lang.NullPointerException at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84) at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:55) at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:56) at backtype.storm.disruptor$consume_loop_STAR_$fn__1596.invoke(disruptor.clj:67) at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377) at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.NullPointerException at backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:24) at backtype.storm.daemon.worker$mk_transfer_fn$fn__4126$fn__4130.invoke(worker.clj:99) at backtype.storm.util$fast_list_map.invoke(util.clj:771) at backtype.storm.daemon.worker$mk_transfer_fn$fn__4126.invoke(worker.clj:99) at backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3904.invoke(executor.clj:205) at backtype.storm.disruptor$clojure_handler$reify__1584.onEvent(disruptor.clj:43) at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:81) ... 6 more
or
java.lang.RuntimeException: java.lang.NullPointerException at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) ~[storm-core-0.9.3.jar:0.9.3] at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) ~[storm-core-0.9.3.jar:0.9.3] at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) ~[storm-core-0.9.3.jar:0.9.3] at backtype.storm.disruptor$consume_loop_STAR_$fn__759.invoke(disruptor.clj:94) ~[storm-core-0.9.3.jar:0.9.3] at backtype.storm.util$async_loop$fn__458.invoke(util.clj:463) ~[storm-core-0.9.3.jar:0.9.3] at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] Caused by: java.lang.NullPointerException: null at clojure.lang.RT.intCast(RT.java:1087) ~[clojure-1.5.1.jar:na] at backtype.storm.daemon.worker$mk_transfer_fn$fn__3548.invoke(worker.clj:129) ~[storm-core-0.9.3.jar:0.9.3] at backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3282.invoke(executor.clj:258) ~[storm-core-0.9.3.jar:0.9.3] at backtype.storm.disruptor$clojure_handler$reify__746.onEvent(disruptor.clj:58) ~[storm-core-0.9.3.jar:0.9.3] at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) ~[storm-core-0.9.3.jar:0.9.3] ... 6 common frames omitted
Solution:
OutputCollector
. All emits, acks, and fails must happen on the same thread. One subtle way this can happen is if you make a IBasicBolt
that emits on a separate thread. IBasicBolt
's automatically ack after execute is called, so this would cause multiple threads to use the OutputCollector
leading to this exception. When using a basic bolt, all emits must happen in the same thread that runs execute
.