blob: 6a5682feb2be1ee9b2dfe3dc891579895c84bee8 [file] [view]
# microbench.py
Runs the JMH micro-benchmarks in `juneau-integration-tests` through Maven and checks or reports the results.
## When to use
- You changed a hot path and want to confirm the observability no-op block still allocates nothing (`observability`).
- You want ops/s and allocation numbers for serialization, parsing, class metadata and a mock REST round trip (`runtime`).
## Usage
```bash
python3 scripts/microbench.py observability
python3 scripts/microbench.py observability --threshold 0
python3 scripts/microbench.py runtime --quick
python3 scripts/microbench.py runtime --include 'Json.*' --forks 1 --jfr
python3 scripts/microbench.py --verbose runtime
```
## Options
| Option | Argument | Default | Meaning |
|---|---|---|---|
| `--verbose`, `-v` | none | off | Stream the full Maven output instead of only its tail. Goes before the subcommand. |
| `observability` | subcommand | required choice | Run `ObservabilityNoopBenchmark` and assert `gc.alloc.rate.norm` is within the threshold. |
| `runtime` | subcommand | required choice | Run the runtime suite in `org.apache.juneau.microbench.runtime`. |
| `--threshold` | BYTES (float) | 8.0 | `observability` only. Maximum allowed `gc.alloc.rate.norm` in bytes per operation. |
| `--include` | REGEX | the whole suite | `runtime` only. JMH include regex. May be repeated. |
| `--forks` | N (integer) | per-class value | `runtime` only. Override the fork count. |
| `--warmup` | N (integer) | per-class value | `runtime` only. Override the warmup iteration count. |
| `--iterations` | N (integer) | per-class value | `runtime` only. Override the measurement iteration count. |
| `--time` | SECONDS (integer) | per-class value | `runtime` only. Override the warmup and measurement iteration time. |
| `--quick` | none | off | `runtime` only. Smoke run: 1 fork, 2 measurement iterations, 1 second iterations. |
| `--jfr` | none | off | `runtime` only. Also record a JFR file per benchmark (JMH `-prof jfr`). |
| `--out-dir` | DIR | `juneau-integration-tests/target/jmh-results` | `runtime` only. Where results are written. |
## What it does
1. `observability`: runs `mvn -f juneau-integration-tests/pom.xml -Pmicrobench test-compile exec:java` with the benchmark class as `exec.mainClass` (`run_benchmark`, `cmd_observability`).
2. Reads the newest `juneau-integration-tests/jmh-results/observability-*.json`, extracts the `gc.alloc.rate.norm` secondary metric, and compares it with `--threshold`.
3. `runtime`: runs `mvn -pl juneau-integration-tests -Pmicrobench-runtime process-test-classes -Dmicrobench.args=...` from the reactor, so it measures this checkout's code and needs no earlier `mvn install` (`cmd_runtime`).
4. The runtime results go to `<out-dir>/runtime-<date>-<time>.json`, and with `--jfr` the recordings go to `<out-dir>/jfr-<date>-<time>/<benchmark>/`.
5. Prints ops/s (or time per operation) and `gc.alloc.rate.norm` per benchmark.
## Output and exit codes
A pass or fail line for the allocation check, and the benchmark table for `runtime`. Without `--verbose`, only the tail of the Maven output is shown.
| Code | Meaning |
|---|---|
| 0 | The allocation rate was within the threshold, or the runtime suite ran. |
| 1 | The rate exceeded the threshold, Maven failed, or no result file could be read. |
## Gotchas
- `--verbose` is a top-level option, so it must come before `observability` or `runtime`.
- The threshold default is 8 bytes per operation; the source notes it can be tightened to 0 once confirmed.
- `observability` writes under `juneau-integration-tests/jmh-results/`, while `runtime` writes under `target/`, so runtime runs leave no untracked files.
- `--warmup` also gives `ColdStartBenchmark` warmups, which defeats it. Exclude that benchmark with `--include` when overriding warmup.
- To record a new baseline, run the `observability` check and update the `[observability]` section of `perf-baseline.txt` at the project root by hand.
- A benchmark run can take many minutes. Use `--quick` for a smoke run.
## See also
- `perf-required.py`