docs/_docs/perf-and-troubleshooting/yardstick-benchmarking.adoc - ignite - Git at Google

 // Licensed to the Apache Software Foundation (ASF) under one or more
 // contributor license agreements.  See the NOTICE file distributed with
 // this work for additional information regarding copyright ownership.
 // The ASF licenses this file to You under the Apache License, Version 2.0
 // (the "License"); you may not use this file except in compliance with
 // the License.  You may obtain a copy of the License at
 //
 // http://www.apache.org/licenses/LICENSE-2.0
 //
 // Unless required by applicable law or agreed to in writing, software
 // distributed under the License is distributed on an "AS IS" BASIS,
 // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 // See the License for the specific language governing permissions and
 // limitations under the License.
 = Benchmarking Ignite With Yardstick Framework

 == Yardstick Ignite Benchmarks

 Apache Ignite benchmarks are written on top of the Yardstick Framework, allowing you to measure the performance of
 various Apache Ignite components and modules. The documentation below describes how to execute and configure pre-assembled
 benchmarks. If you need to add new benchmarks
 or build existing one, then refer to the instructions from Ignite's `DEVNOTES.txt` file in the source directory.

 Visit the https://github.com/gridgain/yardstick[Yardstick Repository, window=_blank] for more details on the resulting graphs generation
 and how the framework works.

 == Running Ignite Benchmarks Locally

 The simplest way to start with benchmarking is to use one of the executable scripts available under the `benchmarks/bin` directory:

 [tabs]
 --
 tab:Shell[]
 [source, shell]
 ----
 ./bin/benchmark-run-all.sh config/benchmark-sample.properties
 ----
 --

 The command above will benchmark the cache `put` operations for a distributed atomic cache. The results of the benchmark
 will be added to an auto-generated `output/results-{DATE-TIME}+\` directory.

 If the `./bin/benchmark-run-all.sh` command is executed as-is without any parameters and modifications in the configuration
 file, then all the available benchmarks will be executed on a local machine using the `config/benchmark.properties`
 configuration. In case of any issues, refer to the logs that are added to an auto-generated `output/logs-{DATE-TIME}` directory.

 For more information about available benchmarks and configuration parameters, refer to the
 <<existing-benchmarks,Existing Benchmarks>> and <<properties-and-command-line-arguments,Properties And Command Line Arguments>>
 sections below.

 == Running Ignite Benchmarks Remotely

 To benchmark Apache Ignite across several remote hosts:

 . Go to `config/ignite-remote-config.xml` and replace `<value>127.0.0.1:47500..47509</value>` with actual IPs of all the remote
 hosts. Refer to the documentation section below if you prefer to use another kind of IP finder: link:clustering/clustering[Cluster Configuration]
 . Go to `config/benchmark-remote-sample.properties` and replace `localhost` with actual IPs of the remote hosts in the following places:
 `SERVERS=localhost,localhost`
 `DRIVERS=localhost,localhost`
  where the `DRIVER` is a host (usually an Ignite client node) that executes benchmarking logic. `SERVERS` are Ignite nodes
 that are benchmarked. Replace the `localhost` occurrences in the same places in the `config/benchmark-remote.properties`
 file if you plan to execute a full set of benchmarks available.
 . Upload Ignite Yardstick Benchmarks to one of your `DRIVERS` host in its own working directory.
 . Log in on the remote host that will be the `DRIVER`, and execute the following command:
 +
 [tabs]
 --
 tab:Shell[]
 [source, shell]
 ----
 ./bin/benchmark-run-all.sh config/benchmark-remote-sample.properties
 ----
 --

 By default, all the necessary files will be automatically uploaded from the host in which you run the command above to
 every other host to the same path. If you prefer to do it manually set the `AUTO_COPY` variable in property file to `false`.

 The command above will benchmark the cache put operation for a distributed atomic cache. The results of the benchmark will
 be added to an auto-generated `output/results-{DATE-TIME}` directory.

 If you want to execute all the available benchmarks across the remote hosts, then execute the following command on the `DRIVER` side:
 [tabs]
 --
 tab:Shell[]
 [source, shell]
 ----
 ./bin/benchmark-run-all.sh config/benchmark-remote.properties
 ----
 --

 == Existing Benchmarks

 The following benchmarks are provided by default:

 . `GetBenchmark` - benchmarks atomic distributed cache get operation.
 . `PutBenchmark` - benchmarks atomic distributed cache put operation.
 . `PutGetBenchmark` - benchmarks atomic distributed cache put and get operations together.
 . `PutTxBenchmark` - benchmarks transactional distributed cache put operation.
 . `PutGetTxBenchmark` - benchmarks transactional distributed cache put and get operations together.
 . `SqlQueryBenchmark` - benchmarks distributed SQL query over cached data.
 . `SqlQueryJoinBenchmark` - benchmarks distributed SQL query with a Join over cached data.
 . `SqlQueryPutBenchmark` - benchmarks distributed SQL query with simultaneous cache updates.
 . `AffinityCallBenchmark` - benchmarks affinity call operation.
 . `ApplyBenchmark` - benchmarks apply operation.
 . `BroadcastBenchmark` - benchmarks broadcast operations.
 . `ExecuteBenchmark` - benchmarks execute operations.
 . `RunBenchmark` - benchmarks running task operations.
 . `PutGetOffHeapBenchmark` - benchmarks atomic distributed cache put and get operations together off-heap.
 . `PutGetOffHeapValuesBenchmark` - benchmarks atomic distributed cache put value operations off-heap.
 . `PutOffHeapBenchmark` - benchmarks atomic distributed cache put operations off-heap.
 . `PutOffHeapValuesBenchmark` - benchmarks atomic distributed cache get value operations off-heap.
 . `PutTxOffHeapBenchmark` - benchmarks transactional distributed cache put operation off-heap.
 . `PutTxOffHeapValuesBenchmark` - benchmarks transactional distributed cache put value operation off-heap.
 . `SqlQueryOffHeapBenchmark` -benchmarks distributed SQL query over cached data off-heap.
 . `SqlQueryJoinOffHeapBenchmark` - benchmarks distributed SQL query with a Join over cached data off-heap.
 . `SqlQueryPutOffHeapBenchmark` - benchmarks distributed SQL query with simultaneous cache updates off-heap.
 . `PutAllBenchmark` - benchmarks atomic distributed cache batch put operation.
 . `PutAllTxBenchmark` - benchmarks transactional distributed cache batch put operation.

 == Properties And Command Line Arguments

 Note that this section only describes configuration parameters specific to Ignite benchmarks, and not for Yardstick framework.
 To run Ignite benchmarks and generate graphs, you will need to run them using the Yardstick framework scripts in the `bin` folder.

 Refer to the https://github.com/gridgain/yardstick/blob/master/README.md[Yardstick Documentation, window=_blank] for common Yardstick
 properties and command line arguments for running Yardstick scripts.

 The following Ignite benchmark properties can be defined in the benchmark configuration:

 * `-b <num>` or `--backups <num>` - Number of backups for every key.
 * `-cfg <path>` or `--Config <path>` - Path to Ignite configuration file.
 * `-cs` or `--cacheStore` - Enable or disable cache store readThrough, writeThrough.
 * `-cl` or `--client` - Client flag. Use this flag if you running more than one `DRIVER`, otherwise additional drivers would behave like a `servers`.
 * `-nc` or `--nearCache` - Near cache flag.
 * `-nn <num>` or `--nodeNumber <num>` - Number of nodes (automatically set in `benchmark.properties`); used to wait for the specified number of nodes to start.
 * `-sm <mode>` or `-syncMode <mode>` - Synchronization mode (defined in CacheWriteSynchronizationMode`).
 * `-r <num>` or `--range` - Range of keys that are randomly generated for cache operations.
 * `-rd or --restartdelay` - Restart delay in seconds.
 * `-rs or --restartsleep` - Restart sleep in seconds.
 * `-rth <host>` or `--restHost <host>` - REST TCP host.
 * `-rtp <num>` or `--restPort <num>` - REST TCP port, indicates that a Ignite node is ready to process Ignite Clients.
 * `-ss` or `--syncSend` - Flag indicating whether synchronous send is used in `TcpCommunicationSpi`.
 * `-txc <value>` or `--txConcurrency <value>` - Cache transaction concurrency control, either `OPTIMISTIC` or `PESSIMISTIC` (defined in `CacheTxConcurrency`).
 * `-txi <value>` or `--txIsolation <value>` - Cache transaction isolation (defined in `CacheTxIsolation`).
 * `-wb` or `--writeBehind` - Enable or disable writeBehind for cache store.

 For example, if you want to run 2 `IgniteNode` servers on localhost with the `PutBenchmark` benchmark, number of
 backups set to `1`, and synchronization mode set to `PRIMARY_SYNC`, then the following configuration should be specified
 in the `benchmark.properties` file:
 [tabs]
 --
 tab:Shell[]
 [source, shell]
 ----
 SERVER_HOSTS=localhost,localhost
 ...

 # Note that -dn and -sn, which stand for data node and server node,
 # are native Yardstick parameters and are documented in
 # Yardstick framework.
 CONFIGS="-b 1 -sm PRIMARY_SYNC -dn PutBenchmark`IgniteNode"
 ----
 --

 == Building From Sources

 Run `mvn clean package -Pyardstick -pl modules/yardstick -am -DskipTests` in the Apache Ignite root directory.

 This command will compile the project and also unpack the scripts from `yardstick-resources.zip` file to `modules/yardstick/target/assembly/bin` directory.

 Artifacts can be found in the `modules/yardstick/target/assembly` directory.

 == Custom Ignite Benchmarks

 All benchmarks extend the `AbstractBenchmark` class. A new benchmark should also extend this abstract class and
 implement the `test` method (this is the method that actually tests performance).
	// Licensed to the Apache Software Foundation (ASF) under one or more
	// contributor license agreements. See the NOTICE file distributed with
	// this work for additional information regarding copyright ownership.
	// The ASF licenses this file to You under the Apache License, Version 2.0
	// (the "License"); you may not use this file except in compliance with
	// the License. You may obtain a copy of the License at
	//
	// http://www.apache.org/licenses/LICENSE-2.0
	//
	// Unless required by applicable law or agreed to in writing, software
	// distributed under the License is distributed on an "AS IS" BASIS,
	// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	// See the License for the specific language governing permissions and
	// limitations under the License.
	= Benchmarking Ignite With Yardstick Framework

	== Yardstick Ignite Benchmarks

	Apache Ignite benchmarks are written on top of the Yardstick Framework, allowing you to measure the performance of
	various Apache Ignite components and modules. The documentation below describes how to execute and configure pre-assembled
	benchmarks. If you need to add new benchmarks
	or build existing one, then refer to the instructions from Ignite's `DEVNOTES.txt` file in the source directory.

	Visit the https://github.com/gridgain/yardstick[Yardstick Repository, window=_blank] for more details on the resulting graphs generation
	and how the framework works.

	== Running Ignite Benchmarks Locally

	The simplest way to start with benchmarking is to use one of the executable scripts available under the `benchmarks/bin` directory:

	[tabs]
	--
	tab:Shell[]
	[source, shell]
	----
	./bin/benchmark-run-all.sh config/benchmark-sample.properties
	----
	--

	The command above will benchmark the cache `put` operations for a distributed atomic cache. The results of the benchmark
	will be added to an auto-generated `output/results-{DATE-TIME}+\` directory.

	If the `./bin/benchmark-run-all.sh` command is executed as-is without any parameters and modifications in the configuration
	file, then all the available benchmarks will be executed on a local machine using the `config/benchmark.properties`
	configuration. In case of any issues, refer to the logs that are added to an auto-generated `output/logs-{DATE-TIME}` directory.

	For more information about available benchmarks and configuration parameters, refer to the
	<<existing-benchmarks,Existing Benchmarks>> and <<properties-and-command-line-arguments,Properties And Command Line Arguments>>
	sections below.

	== Running Ignite Benchmarks Remotely

	To benchmark Apache Ignite across several remote hosts:

	. Go to `config/ignite-remote-config.xml` and replace `<value>127.0.0.1:47500..47509</value>` with actual IPs of all the remote
	hosts. Refer to the documentation section below if you prefer to use another kind of IP finder: link:clustering/clustering[Cluster Configuration]
	. Go to `config/benchmark-remote-sample.properties` and replace `localhost` with actual IPs of the remote hosts in the following places:
	`SERVERS=localhost,localhost`
	`DRIVERS=localhost,localhost`
	where the `DRIVER` is a host (usually an Ignite client node) that executes benchmarking logic. `SERVERS` are Ignite nodes
	that are benchmarked. Replace the `localhost` occurrences in the same places in the `config/benchmark-remote.properties`
	file if you plan to execute a full set of benchmarks available.
	. Upload Ignite Yardstick Benchmarks to one of your `DRIVERS` host in its own working directory.
	. Log in on the remote host that will be the `DRIVER`, and execute the following command:
	+
	[tabs]
	--
	tab:Shell[]
	[source, shell]
	----
	./bin/benchmark-run-all.sh config/benchmark-remote-sample.properties
	----
	--

	By default, all the necessary files will be automatically uploaded from the host in which you run the command above to
	every other host to the same path. If you prefer to do it manually set the `AUTO_COPY` variable in property file to `false`.

	The command above will benchmark the cache put operation for a distributed atomic cache. The results of the benchmark will
	be added to an auto-generated `output/results-{DATE-TIME}` directory.

	If you want to execute all the available benchmarks across the remote hosts, then execute the following command on the `DRIVER` side:
	[tabs]
	--
	tab:Shell[]
	[source, shell]
	----
	./bin/benchmark-run-all.sh config/benchmark-remote.properties
	----
	--

	== Existing Benchmarks

	The following benchmarks are provided by default:

	. `GetBenchmark` - benchmarks atomic distributed cache get operation.
	. `PutBenchmark` - benchmarks atomic distributed cache put operation.
	. `PutGetBenchmark` - benchmarks atomic distributed cache put and get operations together.
	. `PutTxBenchmark` - benchmarks transactional distributed cache put operation.
	. `PutGetTxBenchmark` - benchmarks transactional distributed cache put and get operations together.
	. `SqlQueryBenchmark` - benchmarks distributed SQL query over cached data.
	. `SqlQueryJoinBenchmark` - benchmarks distributed SQL query with a Join over cached data.
	. `SqlQueryPutBenchmark` - benchmarks distributed SQL query with simultaneous cache updates.
	. `AffinityCallBenchmark` - benchmarks affinity call operation.
	. `ApplyBenchmark` - benchmarks apply operation.
	. `BroadcastBenchmark` - benchmarks broadcast operations.
	. `ExecuteBenchmark` - benchmarks execute operations.
	. `RunBenchmark` - benchmarks running task operations.
	. `PutGetOffHeapBenchmark` - benchmarks atomic distributed cache put and get operations together off-heap.
	. `PutGetOffHeapValuesBenchmark` - benchmarks atomic distributed cache put value operations off-heap.
	. `PutOffHeapBenchmark` - benchmarks atomic distributed cache put operations off-heap.
	. `PutOffHeapValuesBenchmark` - benchmarks atomic distributed cache get value operations off-heap.
	. `PutTxOffHeapBenchmark` - benchmarks transactional distributed cache put operation off-heap.
	. `PutTxOffHeapValuesBenchmark` - benchmarks transactional distributed cache put value operation off-heap.
	. `SqlQueryOffHeapBenchmark` -benchmarks distributed SQL query over cached data off-heap.
	. `SqlQueryJoinOffHeapBenchmark` - benchmarks distributed SQL query with a Join over cached data off-heap.
	. `SqlQueryPutOffHeapBenchmark` - benchmarks distributed SQL query with simultaneous cache updates off-heap.
	. `PutAllBenchmark` - benchmarks atomic distributed cache batch put operation.
	. `PutAllTxBenchmark` - benchmarks transactional distributed cache batch put operation.

	== Properties And Command Line Arguments

	Note that this section only describes configuration parameters specific to Ignite benchmarks, and not for Yardstick framework.
	To run Ignite benchmarks and generate graphs, you will need to run them using the Yardstick framework scripts in the `bin` folder.

	Refer to the https://github.com/gridgain/yardstick/blob/master/README.md[Yardstick Documentation, window=_blank] for common Yardstick
	properties and command line arguments for running Yardstick scripts.

	The following Ignite benchmark properties can be defined in the benchmark configuration:

	* `-b <num>` or `--backups <num>` - Number of backups for every key.
	* `-cfg <path>` or `--Config <path>` - Path to Ignite configuration file.
	* `-cs` or `--cacheStore` - Enable or disable cache store readThrough, writeThrough.
	* `-cl` or `--client` - Client flag. Use this flag if you running more than one `DRIVER`, otherwise additional drivers would behave like a `servers`.
	* `-nc` or `--nearCache` - Near cache flag.
	* `-nn <num>` or `--nodeNumber <num>` - Number of nodes (automatically set in `benchmark.properties`); used to wait for the specified number of nodes to start.
	* `-sm <mode>` or `-syncMode <mode>` - Synchronization mode (defined in CacheWriteSynchronizationMode`).
	* `-r <num>` or `--range` - Range of keys that are randomly generated for cache operations.
	* `-rd or --restartdelay` - Restart delay in seconds.
	* `-rs or --restartsleep` - Restart sleep in seconds.
	* `-rth <host>` or `--restHost <host>` - REST TCP host.
	* `-rtp <num>` or `--restPort <num>` - REST TCP port, indicates that a Ignite node is ready to process Ignite Clients.
	* `-ss` or `--syncSend` - Flag indicating whether synchronous send is used in `TcpCommunicationSpi`.
	* `-txc <value>` or `--txConcurrency <value>` - Cache transaction concurrency control, either `OPTIMISTIC` or `PESSIMISTIC` (defined in `CacheTxConcurrency`).
	* `-txi <value>` or `--txIsolation <value>` - Cache transaction isolation (defined in `CacheTxIsolation`).
	* `-wb` or `--writeBehind` - Enable or disable writeBehind for cache store.

	For example, if you want to run 2 `IgniteNode` servers on localhost with the `PutBenchmark` benchmark, number of
	backups set to `1`, and synchronization mode set to `PRIMARY_SYNC`, then the following configuration should be specified
	in the `benchmark.properties` file:
	[tabs]
	--
	tab:Shell[]
	[source, shell]
	----
	SERVER_HOSTS=localhost,localhost
	...

	# Note that -dn and -sn, which stand for data node and server node,
	# are native Yardstick parameters and are documented in
	# Yardstick framework.
	CONFIGS="-b 1 -sm PRIMARY_SYNC -dn PutBenchmark`IgniteNode"
	----
	--

	== Building From Sources

	Run `mvn clean package -Pyardstick -pl modules/yardstick -am -DskipTests` in the Apache Ignite root directory.

	This command will compile the project and also unpack the scripts from `yardstick-resources.zip` file to `modules/yardstick/target/assembly/bin` directory.

	Artifacts can be found in the `modules/yardstick/target/assembly` directory.

	== Custom Ignite Benchmarks

	All benchmarks extend the `AbstractBenchmark` class. A new benchmark should also extend this abstract class and
	implement the `test` method (this is the method that actually tests performance).