This test tool benchmarks an OpenWhisk deployment for (warm) latency and throughput, with several new capabilities:
The general operation of a test is simple:
Final results are written to the standard output stream (so can be redirected to a file) as a single highly-detailed CSV record containing all the input settings and the output measurements (see below). There is additional control information that is written to the standard error stream and can be silenced in CLI. The control information also contains the CSV header, so it can be copied into a spreadsheet if needed.
It is possible to invoke the tool in “Master apart” mode, where the master client is invoking a diffrent activity than the workers, and at possibly a different (very likely, much slower) rate. In this mode, latency statsitics are computed based solely on the master‘s data, since the worker’s activity is used only as background to stress the OpenWhisk deployment. So one experiment can have the master client invoke rules and another one can have the master client invoke actions, while in both experiments the worker clients perform the same background activity.
The tool is highly customizable via CLI options. All the independent test variables are controlled via CLI. This includes number of workers, invocation pattern, OW client configuration, test action sleep time, etc.
Test setup and teardown can be independently skipped via CLI, and/or directly invoked from the external setup script (setup.sh
), so that setup can be shared between multiple tests. More advanced users can replace the test action with a custom action in the setup script to benchmark action invocation or event-respose throughput and latency of specific applications.
Clock skew: OpenWhisk is a distributed system, which means that clock skew is expected between the client machine computing invocation timestamps and the controllers or invokers that generate the timestamps in the activation records. However, this tool assumes that clock skew is bound at few msec range, due to having all machines clocks synchronized, typically using NTP. At such a scale, clock skew is quite small compared to the measured time periods. Some of the time periods are measured using the same clock (see below) and are therefore oblivious to clock skew issues.
The tool requires very little setup. You need to have node.js (v8+) and the wsk CLI client installed (on $PATH). Before the first run, execute npm install
in the tool folder to install the dependencies. Throttling: By default, OW performance is throttled according to some limits, such as maximum number of concurrent requests, or maximum invocations per minute. If your benchmark stresses OpenWhisk beyond the limit value, you might want to relax those limits. If it‘s an OpenWhisk deployment that you control, you can set the limits to 999999, thereby effectively cancelling the limits. If it’s a third-party service, you may want to consult the service documentation and/or support to see what limits can be relaxed and by how much.
To use the tool, run ./owperf.sh <options>
to perform a test. To see all the available options and defaults run ./owperf.sh -h
.
The default for ratio is 1. If using a different ratio, be sure to specify the same ratio value for all steps.
For example, let's perform a test of rule performance with 3 clients, using the default delta of 200 msec, for 100 iterations (counted at the master client, excluding the warmup), ratio of 4. Each client performs 5 iterations per second, each iteration firing a trigger that invokes 4 rules, yielding a total of 3x5x4=60 rule invocations per second. The command to run this test: ./owperf.sh -a rule -w 3 -i 100 -r 4
As explained above, the owperf tool collects both latency and throughput data at each experiment.
The following time-stamps are collected for each invocation, of either action, or rule (containing an action):
Based on these timestamps, the following measurements are taken:
For each measurement, the tool computes average (avg), standard deviation (std), and extremes (min and max).
The following chart depicts the relationship between the various measurements and the action invocation and rule invocation flows.
Throughput is measured w.r.t. several different counters. During post-processing of an experiment, each counter value is divided by the measurement time period to compute a respective throughput.
For each counter, the tool reports the total counter value (abs), total throughput per second (tp), througput of the worker clients without the master (tpw) and the master's percentage of throughput relative to workers (tpd). The last two values are important mostly for master apart mode.
Aside from that, the tool also counts errors. Failed invocations - of actions, of triggers, or of actions from triggers (via rules) are counted each as an error. The tool reports both absolute error count (abs) and percent out of requests (percent).
The owperf tool has been developed by IBM Research as part of the CLASS EU project. CLASS aims to integrate OpenWhisk as a foundation for latency-sensitive polyglot event-driven big-data analytics platform running on a compute continuum from the cloud to the edge. CLASS is funded by the European Union's Horizon 2020 Programme grant agreement No. 780622.