Apache Geode Benchmarks

Clone this repo:
  1. 36c9c19 Merge pull request #154 from jdeppe-pivotal/feature/reduce-file-copy-logging by Jens Deppe · 4 weeks ago develop
  2. 64d0950 Remove file copy logging by Jens Deppe · 6 weeks ago
  3. 4bb07d4 Add key range support to more benchmarks. by Jacob Barrett · 4 months ago
  4. 0641147 Rename Redis region. by Jacob Barrett · 4 months ago
  5. dfbe030 Add ManualRedisTopology test for hosts and ports (#151) by Sarah · 4 months ago

Build Status Total alerts License

Apache Geode Benchmarks

This project contains a benchmarking framework and benchmarks for Apache Geode. It is based on the yardstick framework, but with a java based test configuration and test execution framework.

These benchmarks are under development.

Running the benchmarks

The benchmarks require machines with passwordless ssh enabled in order to run, so ensure that the authentication key pair for SSH does not have a passphrase. If you have previously created a key pair with a passphrase, you can create a different key pair which is of a different type than the previously created one. Be sure to backup your existing key pair before creating a new one. The public key needs to be in PEM format, but some newer OpenSSH versions default to a new format. PEM format can be forced by using -m PEM:

ssh-keygen -m PEM -t rsa

While running a test on a single machine (i.e. localhost) add the generated key to authorized_keys to authorize the user:

cat <your_public_key_file> >> ~/.ssh/authorized_keys

Test if you can ssh to localhost:

ssh localhost

As long as that works, we are good to go.

Get your local hosts name:

hostname

Edit /etc/hosts and add the local host name with and without domain to localhost entries.

127.0.0.1 localhost mycomputer mycomputer.mydomain
::1       localhost mycomputer mycomputer.mydomain

To run all benchmarks, run the benchmark task and pass in a list of hosts.

For example:

./gradlew benchmark -Phosts=localhost,localhost,localhost,localhost

Options

The benchmarks can take configuration options. Some using Gradle's -P flag and other, which adjust benchmark behavior, via Java system properties using -D.

OptionDescription
-PhostsHosts used by benchmarks on the order of client,locator,server,server (-Phosts=localhost,localhost,localhost,localhost)
-PoutputDirResults output directory (-PoutputDir=/tmp/results)
-PtestJVMPath to an alternative JVM for running the client, locator, and servers. If not specified JAVA_HOME will be used. Note all compilation tasks will still use JAVA_HOME.
-Pbenchmark.XWhere X is a benchmark configuration, defined below.
--testsSpecific benchmarks to run (--tests=PartitionedPutBenchmark)
-dDebug
-iInfo

Benchmark Configuration

Common

These options may apply to all benchmarks.

OptionDescription
withGcSelect which GC to use. Valid values CMS (default), G1, Z, Shenandoah, Epsilon.
withHeapSpecify how large a heap the benchmark VMs should use, default “8g”. Accepts any -Xmx value, like “32g”.
withThreadsSpecify how many threads to use when executing the benchmark. Default varies by benchmark.
withWarmupSpecify how long to warm up the benchmark in seconds. Default is 60 seconds.
withDurationSpecify how long to run the benchmark in seconds. Default is 300 seconds.
withMinKeyThe minimum key value in the key range. Default is 0.
withMaxKeyThe maximum key value in the key range. Default varies by benchmark.
withLocatorCountNumber of locators a topology should use. Typically defaults to 1.
withServerCountNumber of servers a topology should use. Typically defaults to 2.
withClientCountNumber of clients a topology should use. Typically defaults to 1.
withReplicasNumber of region replicas.
withAsyncReplicationEnable asynch region replication.
withNettyThreadsNumber of threads Netty IO Services should have.
Geode Benchmarks

These options only apply to Geode benchmarks.

OptionDescription
withSslFlag to run geode with SSL. A self-signed certificate will be generated at runtime.
withSslProtocolsSpecifies enabled SSL protocols. See Geode property ssl-protocols
withSslCiphersSpecifies enabled SSL chipher suites. See Geode property ssl-ciphers
withSecurityManagerFlag to start Geode with the example implementation of SecurityManager
withSniProxyUse SNI proxy topology.
withSniProxyImageProvide an alternative Docker image coordinate for SNI proxy.
withRouterUse router with SNI proxy topology.
withRouterImageProvide an alternative Docker image coordinate for router.
Redis Benchmarks

These options only apply to Redis benchmarks.

OptionDescription
withRedisClientRedis client to use. May be ‘jedis’ (default) or ‘lettuce’.
withRedisClusterRedis cluster implementation. May be ‘geode’ (default), ‘redis’, ‘manual’.
withRedisServersA semicolon delimited list of Redis host:port pairs for manual cluster mode.
Debugging

These options should not be used when measuring benchmarks.

OptionDescription
withValidationEnable validation of operations. Default disabled.
withGcLoggingEnable GC logging. Default disabled.
withSafepointLoggingEnable Safepoint logging. Default disabled.
withStraceLaunch remote JVM via strace for tracing system calls. Default disabled.

Scripts for running in aws and analyzing results

This project includes some scripts to automate running benchmarks in AWS and analyzing the results produced (as well as the results produced from running locally). See the README.md in the infrastructure/aws directory.

Project structure

The project is divided into two modules

  • harness - Contains test framework code for running benchmarks. Does not depend on Apache Geode.
  • geode-benchmarks - Individual benchmarks of Apache Geode.

Sample benchmark

Benchmarks are defined in declarative configuration classes. Each configuration class is run as a junit test which calls the configure method and passes it to the TestRunner, which executes the test on the provided TEST_HOSTS.

Benchmarks are composed of before tasks, after tasks, and workload tasks. Each seperate before and after task is run once. Workload tasks are run repeatedly and their execution time is measured and reported by the yardstick framework.

/**
* Benchmark configuration class, which defines the topology of the test and
* the initialization tasks and workload tasks for the test.
*/
public class PartitionedPutBenchmark extends AbstractPerformanceTest {

  @Test
  public void run() throws Exception {
    TestRunners.defaultRunner().runTest(this::configure);
  }
  
  /**
  * Declare the configuration of the test by calling methods
  * on TestConfig.
  */
  public TestConfig configure() {
    TestConfig testConfig = new TestConfig();
    int locatorPort = 10334;

    //This test has three roles, a geode locator, server, and client
    config.role("locator", 1);
    config.role("server", 2);
    config.role("client", 1);
    
    //Define how the locator,server and client are initialized
    config.before(new StartLocator(locatorPort), "locator");
    config.before(new StartServer(locatorPort), "server");
    config.before(new StartClient(locatorPort), "client");
    config.before(new CreatePartitionedRegion(), "server");
    config.before(new CreateClientProxyRegion(), "client");
    //Define the benchmarked workload, which runs in a client
    config.workload(new PutTask());
    
    return config;
  }
}
/**
* Workload task, which extends the yardstick BenchmarkDriverAdapter
* 
* Workload tasks should execute a single unit of work, and will be run repeatedly
* for the duration of the test.
*/
public class PutTask extends BenchmarkDriverAdapter implements Serializable {
  private Region<Object, Object> region;
  
  @Override
  public void setUp(BenchmarkConfiguration cfg) throws Exception {
    super.setUp(cfg);
    ClientCache cache = ClientCacheFactory.getAnyInstance();
    region = cache.getRegion("region");
  }

  
  @Override
  public boolean test(Map<Object, Object> ctx) throws Exception {
    region.put(1,2);
    return true;
  }
}

SNI Proxy

You can run any benchmark on a topology that routes all client-server communication through an SNI proxy.

The withSniProxy property accepts:

  • HAProxy for HAProxy based SNI proxy (default).
  • Envoy for Envoy based SNI proxy.
  • Manual for providing your own SNI proxy and managing its lifecycle.

The withSniProxyImage property can be used to provide an alternative Docker image to one of the supported proxy implementations. The value should be set to a valid Docker image coordinate.

To run a test, e.g. PartitionedGetBenchmark, with default SNI Proxy:

./run_tests.sh -t anytagname -- -Pbenchmark.withSniProxy --tests=PartitionedGetBenchmark

Since SNI is a feature of TLS, running with the SNI topology incurs TLS overheads with implied -Pbenchmark.withSsl.

Router

An alternative topology uses a router sitting in front of the SNI proxy to simulate off network access to the cluster, enabled with -Pbenchmark.withRouter.

Enabling the router implies -Pbenchmark.withSniProxy.

The withRouter property accepts:

  • HAProxy for HAProxy based router (default).
  • Manual for providing your own router and managing its lifecycle.

Example:

./run_tests.sh -t anytagname -- -Pbenchmark.withRouter --tests=PartitionedGetBenchmark

Redis Benchmarking

You can run benchmarks utilizing the Redis protocol with various clients and backends. All Redis benchmarks take the pattern Redis*Benchmark. They expect 3 shards with 1 replica per shard, which when combined with 1 Geode locator and the benchmarking client needs a total of 8 hosts (./launch_cluster ... -c 8).

The withRedisClient property accepts:

  • Jedis for using the Jedis library (default).
  • Lettuce for using the Lettuce library.

The withRedisCluster property accepts:

  • Geode for using the Geode server backend (default). Builds a Geode cluster utilizing 7 hosts, 1 locator and 6 servers.
  • Redis for using the Redis server backend. Builds a Redis cluster utilizing 6 hosts and the Bitnami Redis image.
  • Manual for using a manually configured Redis server backend, like Elasticache. Use withRedisServers to specify the address(es) to the Redis server endpoints.

Examples:

  • Runs the RedisGetBenchmark against a Geode cluster using the Jedis client.
    ./run_tests.sh -t anytagname -- --tests=RedisGetBenchmark
    
  • Runs the RedisGetBenchmark against a Geode cluster using the Lettuce client.
    ./run_tests.sh -t anytagname -- -Pbenchmark.withRedisClient=lettuce --tests=RedisGetBenchmark
    
  • Runs the RedisGetBenchmark against a Redis cluster using the Jedis client.
    ./run_tests.sh -t anytagname -- -Pbenchmark.withRedisCluster=redis --tests=RedisGetBenchmark
    
  • Runs the RedisGetBenchmark against an Elasticache cluster using the Jedis client.
    ./run_tests.sh -t anytagname -- -Pbenchmark.withRedisCluster=manual -Pbenchmark.withRedisServers=my-cluster...clustercfg.usw2.cache.amazonaws.com:6379 --tests=RedisGetBenchmark