We are pleased to announce that Storm 0.9.0 has been released and is available from the downloads page. This release represents an important milestone in the evolution of Storm.
While a number of new features have been added, a key focus area for this release has been stability-related fixes. Though many users are successfully running work-in-progress builds for Storm 0.9.x in production, this release represents the most stable version to-date, and is highly recommended for everyone, especially users of 0.8.x versions.
The first hightlight of this release is the new Netty Transport contributed by Yahoo! Engineering. Storm's core network transport mechanism is now plugable, and Storm now comes with two implementations: The original 0MQ transport, and a new Netty-based implementation.
In earlier versions, Storm relied solely on 0MQ for transport. Since 0MQ is a native library, it was highly platform-dependent and, at times, challenging to install properly. In addition, stability between versions varied widely between versions and only a relatively old 0MQ version (2.1.7) was certified to work with Storm.
The Netty transport offers a pure Java alternative that eliminates Storm‘s dependency on native libraries. The Netty transport’s performance is up to twice as fast as 0MQ, and it will open the door for authorization and authentication between worker processes. For an in-depth performance comparison of the 0MQ and Netty transports, see this blog post by Storm contributor Bobby Evans.
To configure Storm to use the Netty transport simply add the following to your storm.yaml
configuration and adjust the values to best suit your use-case:
storm.messaging.transport: "backtype.storm.messaging.netty.Context" storm.messaging.netty.server_worker_threads: 1 storm.messaging.netty.client_worker_threads: 1 storm.messaging.netty.buffer_size: 5242880 storm.messaging.netty.max_retries: 100 storm.messaging.netty.max_wait_ms: 1000 storm.messaging.netty.min_wait_ms: 100
You can also write your own transport implementation by implementing the backtype.storm.messaging.IContext
interface.
Storm now includes a helpful new feature for debugging and monitoring topologies: The logviewer
daemon.
In earlier versions of Storm, viewing worker logs involved determining a worker‘s location (host/port), typically through Storm UI, then ssh
ing to that host and tail
ing the corresponding worker log file. With the new log viewer. You can now easily access a specific worker’s log in a web browser by clicking on a worker's port number right from Storm UI.
The logviewer
daemon runs as a separate process on Storm supervisor nodes. To enable the logviewer
run the following command (under supervision) on your cluster's supervisor nodes:
$ storm logviewer
In previous versions, running Storm on Microsoft Windows required installing third-party binaries (0MQ), modifying Storm's source, and adding Windows-specific scripts. With the addition of the platform-independent Netty transport, as well as numerous enhancements to make Storm more platform-independent, running Storm on Windows is easier than ever.
Security, Authentication, and Authorization have been and will continue to be important focus areas for upcoming features. Storm 0.9.0 introduces an API for pluggable tuple serialization and a blowfish encryption based implementation for encrypting tuple data for sensitive use cases.
For most Storm topology developers, upgrading to 0.9.0 is simply a matter of updating the dependency. Storm's core API has changed very little since the 0.8.2 release.
On the devops side, when upgrading to a new Storm release, it is safest to clear any existing state (Zookeeper, storm.local.dir
), prior to upgrading.
Another important change in 0.9.0 has to do with logging. Storm has largely switched over to the slf4j API (backed by a logback logger implementation). Some Storm dependencies rely on the log4j API, so Storm currently depends on log4j-over-slf4j.
These changes have implications for existing topologies and topology components that use the log4j API.
In general, and when possible, Storm topologies and topology components should use the slf4j API for logging.
Special thanks are due to all those who have contributed to Storm -- whether through direct code contributions, documentation, bug reports, or helping other users on the mailing lists. Your efforts are much appreciated.
storm jar
command to work properly when STORM_JAR_JVM_OPTS is not specified (thanks roadkill001)