These utilities create instances and run tests in your AWS account
You must have the aws cli installed. If aws
is not on your path then you can try to install it with pip3 install awscli --upgrade --user
. See Amazon's aws cli installation instructions
You must also set your secret key for the CLI. You must set up a profile named geode-benchmarks
, so use the command aws configure --profile geode-benchmarks
to configure the CLI. You will need to specify the “AWS Access Key ID” and “AWS Secret Access Key,” which can be obtained from a team member. Set the “Default region name” to “us-west-2”. See Amazon's instructions
Alternately, if you already have AWS credentials and just need to create the geode-benchmarks
profile, you can first modify your existing AWS credentials file (found at ~/.aws/credentials
) and add the following lines:
[geode-benchmarks] aws_access_key_id = Your access ID key aws_secret_access_key = Your secret access key
Then modify the config
file found in the same directory and add the following:
[profile geode-benchmarks] region = us-west-2
If you need to build the image, you must have packer installed. The following scripts (launch, run, analyze, destroy) can be run without building the image. Build the image in the image directory using the build_image.sh
script.
In order to have the scripts know which configuration data to use, the data must be provided. One of the ways that can be done is via environment variables.
prompt> aws configure AWS Access Key ID [****************foo1]: AWS Secret Access Key [****************bar2]: Default region name [us-west-2]:
Export environment variables as follows.
export AWS_ACCESS_KEY_ID=myaccesskeyfoo1 export AWS_SECRET_ACCESS_KEY=mysecretaccesskeybar2 export AWS_REGION="us-west-2"
Creates an instance group in AWS based on an image created.
Usage:
./launch_cluster.sh -t [tag] -c [count] [--ci]
Options:
-t|--tag : Cluster tag to identify the cluster for use with other utilities -c|--count : Number of AWS instances to start (recommended: 4) --ci : (Optional) Set when the instances are being started for use in Continuous Integration -h|-?|--help : Help message
Runs benchmark tests against a single branch of geode
on the AWS instances with the specified tag.
Usage:
run_tests.sh -t [tag] [-v [version] | -b [branch]] <options...>
Options:
-t|--tag : Cluster tag -p|--br|--benchmark-repo : Benchmark repo (default: apache/geode-benchmarks) -e|--bb|--benchmark-branch : Benchmark branch (optional - defaults to develop) -o|--output : Output directory (optional - defaults to ./output-<date>-<tag>) -v|--version|--geode-version : Geode Version -r|--gr|--geode-repo : Geode repo (default: apache/geode) -b|--gb|--branch|--geode-branch : Geode Branch (default: develop) -m|--metadata : Test metadata to output to file, comma-delimited (optional) -- : All subsequent arguments are passed to the benchmark tast as arguments -h|-?|--help : Help message
Runs benchmark tests against two branches of geode for comparison purposes on the AWS instances with the specified tag.
Usage:
run_against_baseline.sh -t [tag] [-v [version] | -b [branch]] [-V [baseline version] | -B [baseline branch]] <options...>"
Options:
-t|--tag : Cluster tag -p|--br|--benchmark-repo : Benchmark repo (default: apache/geode-benchmarks) -e|--bb|--benchmark-branch : Benchmark branch (optional - defaults to develop) -o|--output : Output directory (optional - defaults to ./output-<date>-<tag>) -v|--version|--geode-version : Geode Version -r|--gr|--repo|--geode-repo : Geode repo (default: apache/geode) -b|--gb|--branch|--geode-branch : Geode Branch (default: develop) -R|--bgr|--baseline-repo|--baseline-geode-repo : Geode Baseline Repo (default: apache/geode) -V|--bgv|--baseline-version|--baseline-geode-version : Geode Baseline Version -B|--gbb|--baseline-branch|--baseline-geode-branch : Geode Baseline Branch (default: develop) -m|--metadata : Test metadata to output to file, comma-delimited (optional) --ci : Set when the instances are being started for use in Continuous Integration -- : All subsequent arguments are passed to the benchmark task as arguments -h|-?|--help : Help message e.g. ./run_against_baseline.sh -t test_environment -v <sha1 of target version> -V <sha1 of base version> -R <baseline repo e.g. user/geode> -B <baseline branch name> -b <target branch name> -r <target repo e.g. user/geode>
Compares the results of two benchmark runs and outputs analysis of their relative performance.
Usage:
./analyze_tests.sh [-o <output directory> | [--baselineDir <baseline directory> --branchDir <branch directory>]] [options ...] [-- arguments ...]
Options:
-o|--output|--outputDir : The directory containing benchmark results --baseline|--baselineDir : The directory containing baseline benchmark results --branch|--branchDir : The directory containing branch benchmark results --ci : Set if starting instances for Continuous Integration -- : All subsequent arguments are passed to the benchmark task as arguments. -h|--help : This help message
Destroys a cluster that you created.
Usage:
./destroy_cluster.sh -t [tag] [--ci]
Options:
-t|--tag : Cluster tag to identify the cluster for use with other utilities --ci : (Optional) Set when the instances are being started for use in Continuous Integration -h|-?|--help : Help message
Example 1 - Generating and comparing two benchmark runs using run_tests.sh and analyze_tests.sh.
The first command creates a new cluster with 4 instances and the tag “mycluster” using launch_cluster.sh.
The second command runs only the MyCustomBenchmark
benchmark test (by using the -- --tests=MyCustomBenchmark
argument) found on the myBenchmarkBranch
branch of the myGit/geode-benchmarks
repository. This benchmark runs against the develop branch of Geode, adds some metadata, and outputs the results to ~/benchmarking/baseline
using run_test.sh.
The third command runs the same benchmark against the myGeodeBranch
branch of the myGit/geode
repository and outputs the results to ~/benchmarking/branch
using run_tests.sh.
The fourth command compares the results found in ~/benchmarking/branch
to the results found in ~/benchmarking/baseline
and outputs analysis of the operations per second and latency of the branch benchmark relative to the baseline using analyze_tests.sh.
The fifth command destroys the cluster using destroy_cluster.sh.
./launch_cluster.sh --tag mycluster --count 4 ./run_tests.sh --tag mycluster --geode-branch develop --benchmark-repo myGit/geode-benchmarks --benchmark-branch myBenchmarkBranch --metadata "'name':'HelenaTestingCPUs','CPU':'256','geodeBranch':'CPUTest'" --output ~/benchmarking/baseline -- --tests=MyCustomBenchmark ./run_tests.sh --tag mycluster --geode-repo myGit/geode --geode-branch myGeodeBranch --benchmark-repo myGit/geode-benchmarks --benchmark-branch myBenchmarkBranch --metadata "'name':'HelenaTestingCPUs','CPU':'256','geodeBranch':'CPUTest'" --output ~/benchmarking/branch -- --tests=MyCustomBenchmark ./analyze_tests.sh --branch ~/benchmarking/branch --baseline ~/benchmarking/baseline ./destroy_cluster.sh --tag mycluster
Example 2 - run_against_baseline.sh:
./launch_cluster.sh --tag mycluster --count 4 ./run_against_baseline.sh --tag mycluster --geode-branch develop --benchmark-branch benchmarkBranch --metadata "'name':'HelenaTestingCPUs','CPU':'256','geodeBranch':'CPUTest'" ./destroy_cluster.sh --tag mycluster
Filters...
under the Settings
dropdownorg.apache
Ok
button to save the configurationorg.apache
namespace will not be profiled, so it must be unchecked to profile geode or geode-benchmarks/Users/<yourUsername>/.yjp/ui.ini
and add -Dyjp.zero.time.methods=false
. Restart the profiler for this change to take effect.Connection name
: some name for the configuration (the name of the member that is being connected is usually a good choice)Host or IP Address
: the public IP of the AWS VM hosting the member with which you want to connect (the launch cluster script prints these in the order that they were started: [locator-0, server-1, server-2, client-3])Advanced
and fill in the following:SSH User
: geode
SSH port
: 22
Authentication method
: private key
Private Key
: /Users/<yourUsername>/.geode-benchmarks/<clusterTag>-privkey.pem
Passphrase
: leave blankgeode-benchmarks/infrastructure/scripts/aws
./copy_to_cluster.sh -tag <clusterTag> -- <path to libyjpagent.so> <destination path>
libyjpagent.so
is probably /Applications/YourKit-Java-Profiler-2019.1.app/Contents/Resources/bin/linux-x86-64/libyjpagent.so
.
, which will end up putting the file at /home/geode/
run_tests.sh
script, with the additional CLI option -Pbenchmark.profiler.argument
:./run_tests.sh --tag <clusterTag> [other CLI options] -- -i -Pbenchmark.profiler.argument=-agentpath:/home/geode/libyjpagent.so=disablestacktelemetry,exceptions=disable,delay=60000,sessionname=JVM_ROLE-JVM_ID
./launch_cluster.sh --tag profiling --count 4 ./copy_to_cluster.sh --tag profiling -- /Applications/YourKit-Java-Profiler-2019.1.app/Contents/Resources/bin/linux-x86-64/libyjpagent.so . ./run_tests.sh --tag profiling --geode-branch develop -- -i -Pbenchmark.profiler.argument=-agentpath:/home/geode/libyjpagent.so=disablestacktelemetry,exceptions=disable,delay=60000,sessionname=JVM_ROLE-JVM_ID ./destroy_cluster.sh --tag profiling
To run benchmarks with SSL enabled, run the test using the run_tests.sh
script, with the additional CLI option -PwithSsl
:
./run_tests.sh --tag <clusterTag> [other CLI options] -- -PwithSsl