A collection of examples illustrating various capabilities.
Checkout the projects source and perform a top level Maven build (i.e. from the
git clone https://github.com/apache/storm.git cd storm mvn install -DskipTests=true
This will create a shaded (i.e. “fat” or “uber”) jar in the
external/flux/flux-examples/target directory that can run/deployed with the
cd flux-examples storm jar ./target/flux-examples-*-SNAPSHOT.jar org.apache.storm.flux.Flux --local ./src/main/resources/simple_wordcount.yaml
The example YAML files are also packaged in the examples jar, so they can also be referenced with Flux's
--resource command line switch:
storm jar ./target/flux-examples-*.jar org.apache.storm.flux.Flux --local --resource /simple_wordcount.yaml
This is a very basic wordcount example using Java spouts and bolts. It simply logs the running count of each word received.
This example illustrates how to configure Storm's
storm-kafka spout using Flux YAML DSL
constructor arguments constructs.
This example demonstrates using Flux to setup a storm-hdfs bolt to write to an HDFS cluster. It also demonstrates Flux's variable substitution/filtering feature.
To run the
simple_hdfs.yaml example, copy the
hdfs_bolt.properties file to a convenient location and change, at least, the property
hdfs.url to point to a HDFS cluster. Then you can run the example something like:
storm jar ./target/flux-examples-*.jar org.apache.storm.flux.Flux --local ./src/main/resources/simple_hdfs.yaml --filter my_hdfs_bolt.properties
This example illustrates how to use Flux to setup a storm-hbase bolt to write to HBase.
To run the
simple_hbase.yaml example, copy the
hbase_bolt.properties file to a convenient location and change the properties
hbase.zookeeper.quorum. Then you can run the example something like:
storm jar ./target/flux-examples-*.jar org.apache.storm.flux.Flux --local ./src/main/resources/simple_hbase.yaml --filter my_hbase_bolt.properties
This example illustrates how to use Flux to set up a storm topology that contains windowing operations.
storm jar ./target/flux-examples-*.jar org.apache.storm.flux.Flux --local ./src/main/resources/simple_windowing.yaml
Flux also supports stateful bolts which is illustrated with this example. It is basically an extension of the basic wordcount example. The state is periodically saved (checkpointed) and restored when the topology is restarted.
storm jar ./target/flux-examples-*.jar org.apache.storm.flux.Flux --local ./src/main/resources/simple_stateful_wordcount.yaml
By default the state is stored in-memory only. As such you won't see a resumed state unless you configure to use Redis as the state backend. Ensure that you have Redis running at
localhost:6379 and that
storm-redis-*.jar is in the classpath.
STORM_EXT_CLASSPATH=../../storm-redis/target storm jar ./target/flux-examples-*.jar -c topology.state.provider=org.apache.storm.redis.state.RedisKeyValueStateProvider org.apache.storm.flux.Flux --local ./src/main/resources/simple_stateful_wordcount.yaml