| # Example Storm Topologies |
| |
| Learn to use Storm! |
| |
| --- |
| |
| Table of Contents |
| |
| * <a href="#getting-started">Getting started</a> |
| * <a href="#maven">Using storm-starter with Maven</a> |
| * <a href="#intellij-idea">Using storm-starter with IntelliJ IDEA</a> |
| |
| --- |
| |
| |
| <a name="getting-started"></a> |
| |
| # Getting started |
| |
| ## Prerequisites |
| |
| First, you need `java` and `git` installed and in your user's `PATH`. Also, two of the examples in storm-starter |
| require Python and Ruby. |
| |
| Next, make sure you have the storm-starter code available on your machine. Git/GitHub beginners may want to use the |
| following command to download the latest storm-starter code and change to the new directory that contains the downloaded |
| code. |
| |
| $ git clone git://github.com/apache/storm.git && cd storm/examples/storm-starter |
| |
| |
| ## storm-starter overview |
| |
| |
| storm-starter contains a variety of examples of using Storm. If this is your first time working with Storm, check out |
| these topologies first: |
| |
| 1. [ExclamationTopology](src/jvm/storm/starter/ExclamationTopology.java): Basic topology written in all Java |
| 2. [WordCountTopology](src/jvm/storm/starter/WordCountTopology.java): Basic topology that makes use of multilang by |
| implementing one bolt in Python |
| 3. [ReachTopology](src/jvm/storm/starter/ReachTopology.java): Example of complex DRPC on top of Storm |
| |
| After you have familiarized yourself with these topologies, take a look at the other topopologies in |
| [src/jvm/storm/starter/](src/jvm/storm/starter/) such as [RollingTopWords](src/jvm/storm/starter/RollingTopWords.java) |
| for more advanced implementations. |
| |
| If you want to learn more about how Storm works, please head over to the |
| [Storm project page](http://storm.apache.org). |
| |
| |
| <a name="maven"></a> |
| |
| # Using storm-starter with Maven |
| |
| ## Install Maven |
| |
| Install [Maven](http://maven.apache.org/) (preferably version 3.x) by following |
| the [Maven installation instructions](http://maven.apache.org/download.cgi). |
| |
| |
| ## Build and install Storm jars locally |
| |
| If you are using the latest development version of Storm, e.g. by having cloned the Storm git repository, |
| then you must first perform a local build of Storm itself. Otherwise you will run into Maven errors such as |
| "Could not resolve dependencies for project `org.apache.storm:storm-starter:<storm-version>-SNAPSHOT`". |
| |
| # Must be run from the top-level directory of the Storm code repository |
| $ mvn clean install -DskipTests=true |
| |
| This command will build Storm locally and install its jar files to your user's `$HOME/.m2/repository/`. When you run |
| the Maven command to build and run storm-starter (see below), Maven will then be able to find the corresponding version |
| of Storm in this local Maven repository at `$HOME/.m2/repository`. |
| |
| |
| ## Running topologies with Maven |
| |
| > Note: All following examples require that you run `cd examples/storm-starter` beforehand. |
| |
| storm-starter topologies can be run with the maven-exec-plugin. For example, to |
| compile and run `WordCountTopology` in local mode, use the command: |
| |
| $ mvn compile exec:java -Dstorm.topology=storm.starter.WordCountTopology |
| |
| You can also run clojure topologies with Maven: |
| |
| $ mvn compile exec:java -Dstorm.topology=storm.starter.clj.word_count |
| |
| In Windows parameter should be quoted, like this: |
| |
| $ mvn compile exec:java "-Dstorm.topology=storm.starter.clj.word_count" |
| |
| ## Packaging storm-starter for use on a Storm cluster |
| |
| You can package a jar suitable for submitting to a Storm cluster with the command: |
| |
| $ mvn package |
| |
| This will package your code and all the non-Storm dependencies into a single "uberjar" (or "fat jar") at the path |
| `target/storm-starter-{version}-jar-with-dependencies.jar`. |
| |
| Example filename of the uberjar: |
| |
| >>> target/storm-starter-0.9.3-incubating-SNAPSHOT-jar-with-dependencies.jar |
| |
| You can submit (run) a topology contained in this uberjar to Storm via the `storm` CLI tool: |
| |
| # Example 1: Run the RollingTopWords in local mode (LocalCluster) |
| $ storm jar storm-starter-*-jar-with-dependencies.jar storm.starter.RollingTopWords |
| |
| # Example 2: Run the RollingTopWords in remote/cluster mode, |
| # under the name "production-topology" |
| $ storm jar storm-starter-*-jar-with-dependencies.jar storm.starter.RollingTopWords production-topology remote |
| |
| _Submitting a topology in local vs. remote mode:_ |
| It depends on the actual code of a topology how you can or even must tell Storm whether to run the topology locally (in |
| an in-memory LocalCluster instance of Storm) or remotely (in a "real" Storm cluster). In the case of |
| [RollingTopWords](src/jvm/storm/starter/RollingTopWords.java), for instance, this can be done by passing command line |
| arguments. |
| Topologies other than `RollingTopWords` -- such as [ExclamationTopology](src/jvm/storm/starter/ExclamationTopology.java) |
| -- may behave differently, e.g. by always submitting to a remote cluster (i.e. hardcoded in a way that you, as a user, |
| cannot change without modifying the topology code), or by requiring a customized configuration file that the topology |
| code will parse prior submitting the topology to Storm. Similarly, further options such as the name of the topology may |
| be user-configurable or be hardcoded into the topology code. So make sure you understand how the topology of your |
| choice is set up and configured! |
| |
| |
| ## Running unit tests |
| |
| Use the following Maven command to run the unit tests that ship with storm-starter. Unfortunately `lein test` does not |
| yet run the included unit tests. |
| |
| $ mvn test |
| |
| |
| <a name="intellij-idea"></a> |
| |
| # Using storm-starter with IntelliJ IDEA |
| |
| ## Importing storm-starter as a project in IDEA |
| |
| The following instructions will import storm-starter as a new project in IntelliJ IDEA. |
| |
| |
| * Open _File > Import Project..._ and navigate to the storm-starter directory of your storm clone (e.g. |
| `~/git/storm/examples/storm-starter`). |
| * Select _Import project from external model_, select "Maven", and click _Next_. |
| * In the following screen, enable the checkbox _Import Maven projects automatically_. Leave all other values at their |
| defaults. Click _Next_. |
| * Click _Next_ on the following screen about selecting Maven projects to import. |
| * Select the JDK to be used by IDEA for storm-starter, then click _Next_. |
| * At the time of this writing you should use JDK 6. |
| * It is strongly recommended to use Sun/Oracle JDK 6 rather than OpenJDK 6. |
| * You may now optionally change the name of the project in IDEA. The default name suggested by IDEA is "storm-starter". |
| Click _Finish_ once you are done. |