run.py installs the prerequisites, builds Comet in release mode, runs the micro benchmark suites in spark/src/test/scala/org/apache/spark/sql/benchmark, and collects the results into benchmarks/results/micro ready to be published in a pull request.
It is intended for a dedicated machine, typically an EC2 instance. See Micro Benchmarking on AWS EC2 for the full workflow.
The script has no dependencies beyond Python 3.9 and can be downloaded on its own, before the repository is cloned:
curl -sSLO https://raw.githubusercontent.com/apache/datafusion-comet/main/benchmarks/micro/run.py python3 run.py all
Subcommands:
| Command | Purpose |
|---|---|
setup | Install prerequisites, clone or update the repo, build release |
run | Run the suites, one JVM per suite |
collect | Copy results into benchmarks/results/micro with run metadata |
publish | Commit the results and optionally open a pull request |
all | setup + run + collect |
Run python3 run.py <command> --help for the options of each command.
The suites are discovered from the sources in spark/src/test/scala/org/apache/spark/sql/benchmark, so a new benchmark runs without any change here. EXCLUDED_SUITES lists the few that need something the runner cannot provide, with the reason, and run --list prints them. dev/ci/check-benchmark-runner.py checks in CI that those names still match the sources, and exercises the helpers in this script.