docs/_docs/monitoring-metrics/performance-statistics.adoc - ignite - Git at Google

 // Licensed to the Apache Software Foundation (ASF) under one or more
 // contributor license agreements.  See the NOTICE file distributed with
 // this work for additional information regarding copyright ownership.
 // The ASF licenses this file to You under the Apache License, Version 2.0
 // (the "License"); you may not use this file except in compliance with
 // the License.  You may obtain a copy of the License at
 //
 // http://www.apache.org/licenses/LICENSE-2.0
 //
 // Unless required by applicable law or agreed to in writing, software
 // distributed under the License is distributed on an "AS IS" BASIS,
 // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 // See the License for the specific language governing permissions and
 // limitations under the License.
 = Performance Statistics

 == Overview

 Ignite provides a built-in tool for cluster profiling.

 You can link:#collecting-statistics[collect] performance statistics from the cluster and then
 link:#building-the-report[build] the performance report.

 == What Is Collected

 Performance statistics includes the following records:

 [cols="1,3",opts="header"]
 |===
 |Category | Records
 |Cache operations | Cache API and cache store operations: `get`, `put`, `remove`, `invoke`, `lock`, bulk operations,
 conflict operations, and cache store load, write, and delete operations
 |Cache metadata | Cache start records with cache IDs and cache names
 |Transactions | Transaction commit and rollback records with affected cache IDs, start time, and duration
 |Queries | SQL, SCAN, and INDEX query records with query text or cache/index details, query ID, start time, duration,
 and success flag
 |Query details | Query logical and physical reads, processed rows, and custom query properties
 |Tasks and jobs | Compute task and job records with task names, session IDs, execution time, queue time,
 affinity partition, and timeout flag
 |Persistence | Checkpoint timing records and page write throttling records
 |System views | System view schemas and rows captured when statistics collection starts (some views are skipped, see
 the list below)
 |===

 The following system views are skipped because they are large or duplicate data that is already collected as
 performance records:

 * `baseline.node.attributes`
 * `node.attributes`
 * `metrics`
 * `caches`
 * `sql.queries`
 * `nodes`
 * `cacheGroupPageLists`
 * `dataRegionPageLists`
 * `partitionStates`
 * `statisticsPartitionData`
 * `metastorage`
 * `distributed.metastorage`

 == Collecting Statistics

 link:#jmx[JMX interface] and link:#control-script[Control Script] are used to manage statistics collecting.

 Each node collects performance statistics in binary files. These files are placed under
 the `Ignite_work_directory/perf_stat/` directory. Regular performance events are written to
 `node-{nodeId}.prf`. If a file with the same name already exists, Ignite adds an index to the file name:
 `node-{nodeId}-{index}.prf`.

 When statistics collection starts, each node also writes a snapshot of system views
 (link:monitoring-metrics/system-views[System Views]) to `node-{nodeId}-system-views.prf`.
 If a file with the same name already exists, Ignite adds an index:
 `node-{nodeId}-system-views-{index}.prf`. The `rotate` operation rotates only regular performance event files.

 [WARNING]
 ====
 [discrete]
 === Sensitive Information

 Performance statistics files can contain sensitive information, including query text, cache names, task names,
 node attributes, and values exposed by system views. Store and share these files according to your security policies.
 ====

 Performance statistics files are used to build the report offline.

 For regular performance events, nodes use off-heap cyclic buffer to temporarily store serialized statistics. The writer
 thread flushes buffer to the file when the flush size is reached. Some statistics are skipped if the buffer overflows
 due to a slow disk. See the link:#system-properties[properties] section for customization.

 Each statistics collection process creates new files on nodes. Each next file has the same name with the corresponding
 index. See the examples below:

 * `node-faedc6c9-3542-4610-ae10-4ff7e0600000.prf`
 * `node-faedc6c9-3542-4610-ae10-4ff7e0600000-1.prf`
 * `node-faedc6c9-3542-4610-ae10-4ff7e0600000-2.prf`
 * `node-faedc6c9-3542-4610-ae10-4ff7e0600000-system-views.prf`
 * `node-faedc6c9-3542-4610-ae10-4ff7e0600000-system-views-1.prf`

 == Building the Report

 Ignite provides a tool to generate the report from performance statistics files. The tool is published in the
 https://github.com/apache/ignite-extensions/tree/master/modules/performance-statistics-ext/[ignite-extensions,
 windows="_blank"] repository as link:extensions-and-integrations/performance-statistics[performance-statistics-ext]
 extension.

 Follow these steps to build the performance report:

 1. Stop collecting statistics and place files from all nodes under an empty directory. For example:

     /path_to_files/
         ├── node-162c7147-fef8-4ea2-bd25-8653c41fc7fa.prf
         ├── node-162c7147-fef8-4ea2-bd25-8653c41fc7fa-system-views.prf
         ├── node-7b8a7c5c-f3b7-46c3-90da-e66103c00001.prf
         ├── node-7b8a7c5c-f3b7-46c3-90da-e66103c00001-system-views.prf
         ├── node-faedc6c9-3542-4610-ae10-4ff7e0600000.prf
         └── node-faedc6c9-3542-4610-ae10-4ff7e0600000-system-views.prf

 2.  Run the script from the release package of the tool:

         performance-statistics-tool/build-report.sh path_to_files

 The performance report is created in the new directory under the performance statistics files
 path: `path_to_files/report_yyyy-MM-dd_HH-mm-ss/`. Open `report_yyyy-MM-dd_HH-mm-ss/index.html` in the browser to see
 the report.

 You can also print raw performance statistics records to the console or to a file by using `print-statistics.sh` from
 the `performance-statistics-ext` release package. For example, to print checkpoint timing and page write throttling
 records:

 [source,shell]
 ----
 performance-statistics-tool/print-statistics.sh path_to_files --ops CHECKPOINT,PAGES_WRITE_THROTTLE
 ----

 To write the output to a file, use the `--out` parameter:

 [source,shell]
 ----
 performance-statistics-tool/print-statistics.sh path_to_files --ops CHECKPOINT,PAGES_WRITE_THROTTLE --out checkpoints.json
 ----

 == Management

 The following section provides information on JMX, Control Script and system properties.

 === JMX

 You can manage the performance statistics collection via the `PerformanceStatisticsMBean` interface:

 [cols="1,2",opts="header"]
 |===
 |Method | Description
 |start() | Start collecting performance statistics in the cluster.
 |stop() | Stop collecting performance statistics in the cluster.
 |rotate() | Rotate collecting performance statistics in the cluster.
 |started() | True if performance statistics collection is started.
 |===


 === Control Script

 You can manage the performance statistics collection via the link:tools/control-script[Control Script].

 [tabs]
 --
 tab:Unix[]
 [source,shell]
 ----
 control.sh --performance-statistics [start|stop|rotate|status]
 ----
 tab:Window[]
 [source,shell]
 ----
 control.bat --performance-statistics [start|stop|rotate|status]
 ----
 --

 Parameters:

 [cols="1,3",opts="header"]
 |===
 | Parameter | Description
 | `start`| Start collecting performance statistics in the cluster.
 | `stop`| Stop collecting performance statistics in the cluster.
 | `rotate`| Rotate collecting performance statistics in the cluster.
 | `status`| Get status of collecting performance statistics in the cluster.
 |===


 === System properties

 [cols="2,1,1,3",opts="header"]
 |===
 |Property | Type | Default Value | Description
 |IGNITE_PERF_STAT_FILE_MAX_SIZE | Long | 32 Gb | Maximum performance statistics file size in bytes. Performance
 statistics collection is stopped when the file size is exceeded.
 |IGNITE_PERF_STAT_BUFFER_SIZE | Integer | 32 Mb | Performance statistics off heap buffer size in bytes.
 |IGNITE_PERF_STAT_FLUSH_SIZE | Integer | 8 Mb | Minimal performance statistics batch size to be flushed in bytes.
 |IGNITE_PERF_STAT_CACHED_STRINGS_THRESHOLD | Integer | 10240 | Maximum performance statistics cached strings threshold.
 String caching is stopped when the threshold is exceeded.
 |===
	// Licensed to the Apache Software Foundation (ASF) under one or more
	// contributor license agreements. See the NOTICE file distributed with
	// this work for additional information regarding copyright ownership.
	// The ASF licenses this file to You under the Apache License, Version 2.0
	// (the "License"); you may not use this file except in compliance with
	// the License. You may obtain a copy of the License at
	//
	// http://www.apache.org/licenses/LICENSE-2.0
	//
	// Unless required by applicable law or agreed to in writing, software
	// distributed under the License is distributed on an "AS IS" BASIS,
	// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
	// See the License for the specific language governing permissions and
	// limitations under the License.
	= Performance Statistics

	== Overview

	Ignite provides a built-in tool for cluster profiling.

	You can link:#collecting-statistics[collect] performance statistics from the cluster and then
	link:#building-the-report[build] the performance report.

	== What Is Collected

	Performance statistics includes the following records:

	[cols="1,3",opts="header"]
	\|===
	\|Category \| Records
	\|Cache operations \| Cache API and cache store operations: `get`, `put`, `remove`, `invoke`, `lock`, bulk operations,
	conflict operations, and cache store load, write, and delete operations
	\|Cache metadata \| Cache start records with cache IDs and cache names
	\|Transactions \| Transaction commit and rollback records with affected cache IDs, start time, and duration
	\|Queries \| SQL, SCAN, and INDEX query records with query text or cache/index details, query ID, start time, duration,
	and success flag
	\|Query details \| Query logical and physical reads, processed rows, and custom query properties
	\|Tasks and jobs \| Compute task and job records with task names, session IDs, execution time, queue time,
	affinity partition, and timeout flag
	\|Persistence \| Checkpoint timing records and page write throttling records
	\|System views \| System view schemas and rows captured when statistics collection starts (some views are skipped, see
	the list below)
	\|===

	The following system views are skipped because they are large or duplicate data that is already collected as
	performance records:

	* `baseline.node.attributes`
	* `node.attributes`
	* `metrics`
	* `caches`
	* `sql.queries`
	* `nodes`
	* `cacheGroupPageLists`
	* `dataRegionPageLists`
	* `partitionStates`
	* `statisticsPartitionData`
	* `metastorage`
	* `distributed.metastorage`

	== Collecting Statistics

	link:#jmx[JMX interface] and link:#control-script[Control Script] are used to manage statistics collecting.

	Each node collects performance statistics in binary files. These files are placed under
	the `Ignite_work_directory/perf_stat/` directory. Regular performance events are written to
	`node-{nodeId}.prf`. If a file with the same name already exists, Ignite adds an index to the file name:
	`node-{nodeId}-{index}.prf`.

	When statistics collection starts, each node also writes a snapshot of system views
	(link:monitoring-metrics/system-views[System Views]) to `node-{nodeId}-system-views.prf`.
	If a file with the same name already exists, Ignite adds an index:
	`node-{nodeId}-system-views-{index}.prf`. The `rotate` operation rotates only regular performance event files.

	[WARNING]
	====
	[discrete]
	=== Sensitive Information

	Performance statistics files can contain sensitive information, including query text, cache names, task names,
	node attributes, and values exposed by system views. Store and share these files according to your security policies.
	====

	Performance statistics files are used to build the report offline.

	For regular performance events, nodes use off-heap cyclic buffer to temporarily store serialized statistics. The writer
	thread flushes buffer to the file when the flush size is reached. Some statistics are skipped if the buffer overflows
	due to a slow disk. See the link:#system-properties[properties] section for customization.

	Each statistics collection process creates new files on nodes. Each next file has the same name with the corresponding
	index. See the examples below:

	* `node-faedc6c9-3542-4610-ae10-4ff7e0600000.prf`
	* `node-faedc6c9-3542-4610-ae10-4ff7e0600000-1.prf`
	* `node-faedc6c9-3542-4610-ae10-4ff7e0600000-2.prf`
	* `node-faedc6c9-3542-4610-ae10-4ff7e0600000-system-views.prf`
	* `node-faedc6c9-3542-4610-ae10-4ff7e0600000-system-views-1.prf`

	== Building the Report

	Ignite provides a tool to generate the report from performance statistics files. The tool is published in the
	https://github.com/apache/ignite-extensions/tree/master/modules/performance-statistics-ext/[ignite-extensions,
	windows="_blank"] repository as link:extensions-and-integrations/performance-statistics[performance-statistics-ext]
	extension.

	Follow these steps to build the performance report:

	1. Stop collecting statistics and place files from all nodes under an empty directory. For example:

	/path_to_files/
	├── node-162c7147-fef8-4ea2-bd25-8653c41fc7fa.prf
	├── node-162c7147-fef8-4ea2-bd25-8653c41fc7fa-system-views.prf
	├── node-7b8a7c5c-f3b7-46c3-90da-e66103c00001.prf
	├── node-7b8a7c5c-f3b7-46c3-90da-e66103c00001-system-views.prf
	├── node-faedc6c9-3542-4610-ae10-4ff7e0600000.prf
	└── node-faedc6c9-3542-4610-ae10-4ff7e0600000-system-views.prf

	2. Run the script from the release package of the tool:

	performance-statistics-tool/build-report.sh path_to_files

	The performance report is created in the new directory under the performance statistics files
	path: `path_to_files/report_yyyy-MM-dd_HH-mm-ss/`. Open `report_yyyy-MM-dd_HH-mm-ss/index.html` in the browser to see
	the report.

	You can also print raw performance statistics records to the console or to a file by using `print-statistics.sh` from
	the `performance-statistics-ext` release package. For example, to print checkpoint timing and page write throttling
	records:

	[source,shell]
	----
	performance-statistics-tool/print-statistics.sh path_to_files --ops CHECKPOINT,PAGES_WRITE_THROTTLE
	----

	To write the output to a file, use the `--out` parameter:

	[source,shell]
	----
	performance-statistics-tool/print-statistics.sh path_to_files --ops CHECKPOINT,PAGES_WRITE_THROTTLE --out checkpoints.json
	----

	== Management

	The following section provides information on JMX, Control Script and system properties.

	=== JMX

	You can manage the performance statistics collection via the `PerformanceStatisticsMBean` interface:

	[cols="1,2",opts="header"]
	\|===
	\|Method \| Description
	\|start() \| Start collecting performance statistics in the cluster.
	\|stop() \| Stop collecting performance statistics in the cluster.
	\|rotate() \| Rotate collecting performance statistics in the cluster.
	\|started() \| True if performance statistics collection is started.
	\|===


	=== Control Script

	You can manage the performance statistics collection via the link:tools/control-script[Control Script].

	[tabs]
	--
	tab:Unix[]
	[source,shell]
	----
	control.sh --performance-statistics [start\|stop\|rotate\|status]
	----
	tab:Window[]
	[source,shell]
	----
	control.bat --performance-statistics [start\|stop\|rotate\|status]
	----
	--

	Parameters:

	[cols="1,3",opts="header"]
	\|===
	\| Parameter \| Description
	\| `start`\| Start collecting performance statistics in the cluster.
	\| `stop`\| Stop collecting performance statistics in the cluster.
	\| `rotate`\| Rotate collecting performance statistics in the cluster.
	\| `status`\| Get status of collecting performance statistics in the cluster.
	\|===


	=== System properties

	[cols="2,1,1,3",opts="header"]
	\|===
	\|Property \| Type \| Default Value \| Description
	\|IGNITE_PERF_STAT_FILE_MAX_SIZE \| Long \| 32 Gb \| Maximum performance statistics file size in bytes. Performance
	statistics collection is stopped when the file size is exceeded.
	\|IGNITE_PERF_STAT_BUFFER_SIZE \| Integer \| 32 Mb \| Performance statistics off heap buffer size in bytes.
	\|IGNITE_PERF_STAT_FLUSH_SIZE \| Integer \| 8 Mb \| Minimal performance statistics batch size to be flushed in bytes.
	\|IGNITE_PERF_STAT_CACHED_STRINGS_THRESHOLD \| Integer \| 10240 \| Maximum performance statistics cached strings threshold.
	String caching is stopped when the threshold is exceeded.
	\|===