# Topologies for measuring Storm performance

This module includes topologies designed for measuring Storm performance.

## Overview
There are two basic modes for running these topologies

- **Cluster mode:** Submits the topology to a storm cluster. This mode is useful for benchmarking. It calculates throughput and latency numbers every minute and prints them on the console.
- **In-process mode:** Uses LocalCluster to run topology. This mode helps identify bottlenecks using profilers like JProfiler from within a IDE. This mode does not print metrics.

In both the modes, a shutdown hook is setup to terminate the topology when the program that is submitting the topology is terminated.

The bundled topologies can be classified into two types.

- Topologies that measure purely the internal functioning of Storm. Such topologies do not interact with external systems like Kafka or Hdfs.
- Topologies that measure speed of I/O with external systems like Kafka and Hdfs.

Topologies that measure internal performance can be run in either in-proc or cluster modes.
Topologies that measure I/O with external systems are designed to run in cluster mode only.

## Topologies List

1. **ConstSpoutOnlyTopo:** Helps measure how fast spout can emit. This topology has a spout and is not connected to any bolts. Supports in-proc and cluster mode.
2. **ConstSpoutNullBoltTopo:** Helps measure how fast spout can send data to a bolt. Spout emits a stream of constant values to a DevNull bolt which discards the incoming tuples. Supports in-proc and cluster mode.
3. **ConstSpoutIdBoltNullBoltTopo:** Helps measure speed of messaging between spouts and bolts. Spout emits a stream of constant values to an ID bolt which clones the tuple and emits it downstream to a DevNull bolt. Supports in-proc and cluster mode.
4. **FileReadWordCount:** Measures speed of word counting. The spout loads a file into memory and emits these lines in an infinite loop. Supports in-proc and cluster mode.
5. **HdfsSpoutNullBolt:** Measures speed at which HdfsSpout can read from HDFS. Supports cluster mode only.
6. **StrGenSpoutHdfsBoltTopo:** Measures speed at which HdfsBolt can write to HDFS. Supports cluster mode only.
7. **KafkaSpoutNullBolt:** Measures speed at which KafkaSpout can read from Kafka. Supports cluster mode only.
8. **KafkaHdfsTopo:** Measures how fast Storm can read from Kafka and write to HDFS.


## How to run ?

### In-process mode:
This mode is intended for running the topology quickly and easily from within the IDE and does not expect any command line arguments.
Simply running the Topology's main() method without any arguments will get it running. The topology runs indefinitely till the program is terminated.


### Cluster mode:
When the topology is run with one or more than one cmd line arguments, the topology is submitted to the cluster.
The first argument indicates how long the topology should be run. Often the second argument refers to a yaml config
file which contains topology configuration settings. The conf/ directory in this module contains sample config files
with names matching the corresponding topology.

These topologies can be run using the standard storm jar command.

```
bin/storm jar  /path/storm-perf-1.1.0-jar-with-dependencies.jar org.apache.storm.perf.ConstSpoutNullBoltTopo  200  conf/ConstSpoutIdBoltNullBoltTopo.yaml
```