docs/_docs/extensions-and-integrations/performance-statistics.adoc

// Licensed to the Apache Software Foundation (ASF) under one or more
// contributor license agreements.  See the NOTICE file distributed with
// this work for additional information regarding copyright ownership.
// The ASF licenses this file to You under the Apache License, Version 2.0
// (the "License"); you may not use this file except in compliance with
// the License.  You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
= Performance Statistics Extension

== Overview

Ignite provides link:monitoring-metrics/performance-statistics[a built-in tool for cluster profiling.]

The extension provides utilities to link:#building-the-report[create the report] and
link:#print-statistics[print statistics] from the collected files.

== Building the Report

The extension provides a tool to generate an offline HTML report from performance statistics files.

Follow these steps to build the performance report:

1. Stop collecting statistics and place files from all nodes under an empty directory. For example:

    /path_to_files/
        ├── node-162c7147-fef8-4ea2-bd25-8653c41fc7fa.prf
        ├── node-162c7147-fef8-4ea2-bd25-8653c41fc7fa-system-views.prf
        ├── node-7b8a7c5c-f3b7-46c3-90da-e66103c00001.prf
        └── node-7b8a7c5c-f3b7-46c3-90da-e66103c00001-system-views.prf

+
`node-\*-system-views*.prf` files are stored next to regular performance statistics files. If the system view files are too large, you can omit them. The report will still be generated, but without `System views` data.

2.  Run the script from the release package of the tool:

[source,shell]
----
performance-statistics-tool/build-report.sh path_to_files
----

The performance report is created in the new directory under the performance statistics files:
`path_to_files/report_yyyy-MM-dd_HH-mm-ss/`.
Open `report_yyyy-MM-dd_HH-mm-ss/index.html` in the browser to see the report.

The report includes `Cluster info`, `Cache operations`, `Transactions`, `SQL queries`, `Scan queries`,
`Index queries`, `Tasks and jobs`, and `System views`. When `node-\*-system-views*.prf` files are available in the
input directory, the `System views` tab is populated with their data.

For more details run the help command:

[source,shell]
----
performance-statistics-tool/build-report.sh --help
----

=== HTML Report Tabs

:perf_stat_url: images/integrations/performance-statistics
:perf_stat_image_url: /docs/images/integrations/performance-statistics

==== Cluster info

Cluster topology and caches found in the input files: `Cluster nodes` and `Started caches` tables.

image::{perf_stat_url}/perf_stat_cluster_info.png[Cluster info report, link={perf_stat_image_url}/perf_stat_cluster_info.png]

==== Cache operations

Cache and cache store activity for the selected cache and node. Cache and node selectors filter all charts on the page.

A bar chart summarizes the distribution of cache operation counts by type:

image::{perf_stat_url}/perf_stat_cache_operations.png[Distribution of cache operations, link={perf_stat_image_url}/perf_stat_cache_operations.png]

For each operation type, a line chart shows the operation count over time:

image::{perf_stat_url}/perf_stat_cache_operations_1.png[Cache operation count over time, link={perf_stat_image_url}/perf_stat_cache_operations_1.png]

The `Cache store operations` section provides the same breakdowns for cache store calls. A distribution chart
summarizes cache store operation counts by type:

image::{perf_stat_url}/perf_stat_cache_operations_2.png[Distribution of cache store operations, link={perf_stat_image_url}/perf_stat_cache_operations_2.png]

Line charts show cache store operation counts over time:

image::{perf_stat_url}/perf_stat_cache_operations_3.png[Cache store operation count over time, link={perf_stat_image_url}/perf_stat_cache_operations_3.png]

==== Transactions

Transaction activity for the selected cache and node. A histogram shows the distribution of transaction durations:

image::{perf_stat_url}/perf_stat_transactions.png[Transactions histogram, link={perf_stat_image_url}/perf_stat_transactions.png]

Below it, line charts plot `commit` and `rollback` counts over time:

image::{perf_stat_url}/perf_stat_transactions_1.png[Transactions commit chart, link={perf_stat_image_url}/perf_stat_transactions_1.png]

==== SQL queries

Aggregated SQL query statistics. The `Overall statistics` table shows total executions, duration, and reads for each
query. Each row expands into a `Properties` subtable with map and reduce phase plans, and a `Rows` subtable with
rows fetched on mapper and reducer nodes:

image::{perf_stat_url}/perf_stat_sql_queries.png[SQL queries overall statistics, link={perf_stat_image_url}/perf_stat_sql_queries.png]

The `Top of slowest queries` table lists slow query executions with the same expandable details:

image::{perf_stat_url}/perf_stat_sql_queries_1.png[Top of slowest SQL queries, link={perf_stat_image_url}/perf_stat_sql_queries_1.png]

==== Scan queries

Aggregated scan query statistics by cache and the slowest scan queries: `Overall statistics` and `Top of slowest
queries` tables.

image::{perf_stat_url}/perf_stat_scan_queries.png[Scan queries report, link={perf_stat_image_url}/perf_stat_scan_queries.png]

==== Index queries

Aggregated index query statistics and the slowest index queries: `Overall statistics` and `Top of slowest queries`
tables.

image::{perf_stat_url}/perf_stat_index_queries.png[Index queries report, link={perf_stat_image_url}/perf_stat_index_queries.png]

==== Tasks and jobs

Aggregated task execution statistics. The `Overall statistics` table shows executions, total duration, and job
counts per task:

image::{perf_stat_url}/perf_stat_tasks_and_jobs.png[Tasks and jobs overall statistics, link={perf_stat_image_url}/perf_stat_tasks_and_jobs.png]

The `Top of slowest tasks` table lists slow task executions. Each row expands into a `Jobs` subtable with per-node
job durations:

image::{perf_stat_url}/perf_stat_tasks_and_jobs_1.png[Top of slowest tasks with jobs subtable, link={perf_stat_image_url}/perf_stat_tasks_and_jobs_1.png]

==== System views

Rows from collected system views. The tab provides a node selector, a system view selector, and a searchable
paginated table for the selected view.

image::{perf_stat_url}/perf_stat_system_views.png[System views report, link={perf_stat_image_url}/perf_stat_system_views.png]

== Print Statistics

The extension provides a tool to print raw performance statistics records to the console or to a file in JSON format.
Use this tool when you need access to individual records.

Run the script from the release package of the tool to print statistics:

[source,shell]
----
performance-statistics-tool/print-statistics.sh path_to_files
----

Note that `path_to_files` can point either to a single performance statistics file or to a directory with multiple
files.

The tool supports the following options:

[cols="1,3",opts="header"]
|===
|Option |Description

|`--out out_file` |Appends JSON output to the specified file.
|`--ops op_types` |Prints only the specified comma-separated operation types.
|`--from start_time_from` |Prints only records whose start time is greater than or equal to the specified value (Unix epoch milliseconds).
|`--to start_time_to` |Prints only records whose start time is less than or equal to the specified value (Unix epoch milliseconds).
|`--caches cache_names` |Prints only records related to the specified comma-separated cache names.
|===

For more details run the help command:

[source,shell]
----
performance-statistics-tool/print-statistics.sh --help
----

Examples:

[source,shell]
----
performance-statistics-tool/print-statistics.sh path_to_files --ops QUERY,QUERY_ROWS,QUERY_PROPERTY
performance-statistics-tool/print-statistics.sh path_to_files --ops SYSTEM_VIEW_ROW
performance-statistics-tool/print-statistics.sh path_to_files --ops CHECKPOINT,PAGES_WRITE_THROTTLE
performance-statistics-tool/print-statistics.sh path_to_files --from 1713700000000 --to 1713703600000
performance-statistics-tool/print-statistics.sh path_to_files --caches cache1,cache2 --out perfstat.json
----

See the output example below:

    {"op":"CACHE_GET","nodeId":"955130d1-5218-4e46-87f6-62755e92e9b4","cacheId":-1809642915,"startTime":1616837094237,"duration":64992213}
    {"op":"CACHE_PUT","nodeId":"955130d1-5218-4e46-87f6-62755e92e9b4","cacheId":-1809642915,"startTime":1616837094237,"duration":879869}
    {"op":"CACHE_GET_AND_PUT","nodeId":"955130d1-5218-4e46-87f6-62755e92e9b4","cacheId":1328364293,"startTime":1616837094248,"duration":17186240}
    {"op":"TX_COMMIT","nodeId":"955130d1-5218-4e46-87f6-62755e92e9b4","cacheIds":[-1809642915],"startTime":1616837094172,"duration":184887787}
    {"op":"QUERY","nodeId":"955130d1-5218-4e46-87f6-62755e92e9b4","type":"SQL_FIELDS","text":"create table Person (id int, val varchar, primary key (id))","id":0,"startTime":1616837094143,"duration":258741595,"success":true}