This framework is intended primarily to identify hotspots in S4 platform code easily and to evaluate the impact of refactorings or new features.
The numbers it produces are mainly useful in comparison with a baseline from other measurements from the same benchmark and do not represent absolute performance numbers. For that, one should use a full-fledged load injection framework or measure the performance of a live application.
That said, let's look at what the benchmarking framework does and how to use it.
The benchmarking framework consists of a multithreaded injector and an application. App nodes and injector are launched directly, there is no deployment step. This allows to skip the packaging and deployment steps and to easily add profiling parameters, but requires a source distribution and a shared file system.
2 simple applications are provided:
inputStream and inputStream2. You may inject on one or both of these independent streams. (A node will process more events overall if it gets them from more parallel sources, unless it reaches network or cpu boundaries).There is almost no processing involved in the PE themselves, other than delegating to the next processing element in the pipeline, if any.
The injector sends basic keyed messages to a given named stream. The outputstream of the injector uses a keyfinder to partition the events across the application nodes.
We get metrics from the probes across the codebase, in particular:
Metrics from the platform code are computed with weighted moving averages. It is recommended to let the application run for a few minutes and observe the metrics from the last minute.
Profiling options (e.g. YourKit) can easily be added to the injector or app nodes in order to identify hotspots.
We provide a script for that purpose: bench-cluster.sh
You can use arbitrary numbers of injectors and processing nodes, in order to vary the load and the number of concurrent connections.
Input parameters are:
bench-cluster.sh script is executed)Exmample configuration files are available in /config and you can configure :
s4.sender.maxRate and s4.sender.warmupPeriod.The total number of events sent from an injector is number of keys * number of test iterations * number of parallel injection threads. Make sure this is significant in order to be able to correctly interpret the messaging rates (1000 would be too little for instance!).
By default in this example the size of a message is 188 bytes.
Running 2 S4 nodes on the local machine: ./bench-cluster.sh "localhost localhost" pwd/config/injector.config pwd/config/node.config
For a distributed setup, you should modify the host names in the above command line, and specify the correct Zookeeper connection string in node.config.
Here is an example for driving the test on a cluster: incubator-s4/subprojects/s4-benchmarks/bench-cluster.sh "processingHost1 processingHost2 processingHost3" "injectorConfigStream1.cfg injectorConfigStream2.cfg" node.cfg 2 "injectorHost1 injectorHost2 injectorHost3 injectorHost4" (the 2 controls the number of injectors per stream per injector host)
When the benchmark finishes (and even during the execution), results are available in measurements/injectors for the injection rates and in measurements/node[0-n] for other statistics.
Results are also available from the console output for each of the nodes.
Most statistics files come from the probes of the platform and some of them use weighted moving averages. These are good for long running applications. For the benchmarks we also show instant rates, which are available in injection-rate.csv and simplePE1.csv files.
You may also check that all events have been processed:
total injected from all injectors >= total received in all nodes (minus events sent through internal streams in the app, if that applies)There are a lot of knobs for optimally configuring the stages, and the optimal settings will also depend upon: