microbench.py

Runs the JMH micro-benchmarks in juneau-integration-tests through Maven and checks or reports the results.

When to use

  • You changed a hot path and want to confirm the observability no-op block still allocates nothing (observability).
  • You want ops/s and allocation numbers for serialization, parsing, class metadata and a mock REST round trip (runtime).

Usage

python3 scripts/microbench.py observability
python3 scripts/microbench.py observability --threshold 0
python3 scripts/microbench.py runtime --quick
python3 scripts/microbench.py runtime --include 'Json.*' --forks 1 --jfr
python3 scripts/microbench.py --verbose runtime

Options

OptionArgumentDefaultMeaning
--verbose, -vnoneoffStream the full Maven output instead of only its tail. Goes before the subcommand.
observabilitysubcommandrequired choiceRun ObservabilityNoopBenchmark and assert gc.alloc.rate.norm is within the threshold.
runtimesubcommandrequired choiceRun the runtime suite in org.apache.juneau.microbench.runtime.
--thresholdBYTES (float)8.0observability only. Maximum allowed gc.alloc.rate.norm in bytes per operation.
--includeREGEXthe whole suiteruntime only. JMH include regex. May be repeated.
--forksN (integer)per-class valueruntime only. Override the fork count.
--warmupN (integer)per-class valueruntime only. Override the warmup iteration count.
--iterationsN (integer)per-class valueruntime only. Override the measurement iteration count.
--timeSECONDS (integer)per-class valueruntime only. Override the warmup and measurement iteration time.
--quicknoneoffruntime only. Smoke run: 1 fork, 2 measurement iterations, 1 second iterations.
--jfrnoneoffruntime only. Also record a JFR file per benchmark (JMH -prof jfr).
--out-dirDIRjuneau-integration-tests/target/jmh-resultsruntime only. Where results are written.

What it does

  1. observability: runs mvn -f juneau-integration-tests/pom.xml -Pmicrobench test-compile exec:java with the benchmark class as exec.mainClass (run_benchmark, cmd_observability).
  2. Reads the newest juneau-integration-tests/jmh-results/observability-*.json, extracts the gc.alloc.rate.norm secondary metric, and compares it with --threshold.
  3. runtime: runs mvn -pl juneau-integration-tests -Pmicrobench-runtime process-test-classes -Dmicrobench.args=... from the reactor, so it measures this checkout's code and needs no earlier mvn install (cmd_runtime).
  4. The runtime results go to <out-dir>/runtime-<date>-<time>.json, and with --jfr the recordings go to <out-dir>/jfr-<date>-<time>/<benchmark>/.
  5. Prints ops/s (or time per operation) and gc.alloc.rate.norm per benchmark.

Output and exit codes

A pass or fail line for the allocation check, and the benchmark table for runtime. Without --verbose, only the tail of the Maven output is shown.

CodeMeaning
0The allocation rate was within the threshold, or the runtime suite ran.
1The rate exceeded the threshold, Maven failed, or no result file could be read.

Gotchas

  • --verbose is a top-level option, so it must come before observability or runtime.
  • The threshold default is 8 bytes per operation; the source notes it can be tightened to 0 once confirmed.
  • observability writes under juneau-integration-tests/jmh-results/, while runtime writes under target/, so runtime runs leave no untracked files.
  • --warmup also gives ColdStartBenchmark warmups, which defeats it. Exclude that benchmark with --include when overriding warmup.
  • To record a new baseline, run the observability check and update the [observability] section of perf-baseline.txt at the project root by hand.
  • A benchmark run can take many minutes. Use --quick for a smoke run.

See also

  • perf-required.py