SkyWalking leverages ClickHouse built-in metrics data since v20.1.2.4. It leverages OpenTelemetry Collector to transfer the metrics to OpenTelemetry receiver and into the Meter System.
ClickHouse monitoring provides monitoring of the metrics 、events and asynchronous_metrics of the ClickHouse server. ClickHouse cluster is cataloged as a Layer: CLICKHOUSE Service in OAP. Each ClickHouse server is cataloged as an Instance in OAP.
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| CpuUsage | count | meter_clickhouse_instance_cpu_usage | CPU time spent seen by OS per second(according to ClickHouse.system.dashboard.CPU Usage (cores)). | ClickHouse |
| MemoryUsage | percentage | meter_clickhouse_instance_memory_usage | Total amount of memory (bytes) allocated by the server/ total amount of OS memory. | ClickHouse |
| MemoryAvailable | percentage | meter_clickhouse_instance_memory_available | Total amount of memory (bytes) available for program / total amount of OS memory. | ClickHouse |
| Uptime | sec | meter_clickhouse_instance_uptime | The server uptime in seconds. It includes the time spent for server initialization before accepting connections. | ClickHouse |
| Version | string | meter_clickhouse_instance_version | Version of the server in a single integer number in base-1000. | ClickHouse |
| FileOpen | count | meter_clickhouse_instance_file_open | Number of files opened. | ClickHouse |
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| TcpConnections | count | meter_clickhouse_instance_tcp_connections meter_clickhouse_tcp_connections | Number of connections to TCP server. | ClickHouse |
| MysqlConnections | count | meter_clickhouse_instance_mysql_connections meter_clickhouse_mysql_connections | Number of client connections using MySQL protocol. | ClickHouse |
| HttpConnections | count | meter_clickhouse_instance_http_connections meter_clickhouse_mysql_connections | Number of connections to HTTP server. | ClickHouse |
| InterserverConnections | count | meter_clickhouse_instance_interserver_connections meter_clickhouse_interserver_connections | Number of connections from other replicas to fetch parts. | ClickHouse |
| PostgresqlConnections | count | meter_clickhouse_instance_postgresql_connections meter_clickhouse_postgresql_connections | Number of client connections using PostgreSQL protocol. | ClickHouse |
| ReceiveBytes | bytes | meter_clickhouse_instance_network_receive_bytes meter_clickhouse_network_receive_bytes | Total number of bytes received from network. | ClickHouse |
| SendBytes | bytes | meter_clickhouse_instance_network_send_bytes meter_clickhouse_network_send_bytes | Total number of bytes send to network. | ClickHouse |
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| QueryCount | count | meter_clickhouse_instance_query meter_clickhouse_query | Number of executing queries. | ClickHouse |
| SelectQueryCount | count | meter_clickhouse_instance_query_select meter_clickhouse_query_select | Number of executing queries, but only for SELECT queries. | ClickHouse |
| InsertQueryCount | count | meter_clickhouse_instance_query_insert meter_clickhouse_query_insert | Number of executing queries, but only for INSERT queries. | ClickHouse |
| SelectQueryRate | count/sec | meter_clickhouse_instance_query_select_rate meter_clickhouse_query_select_rate | Number of SELECT queries per second. | ClickHouse |
| InsertQueryRate | count/sec | meter_clickhouse_instance_query_insert_rate meter_clickhouse_query_insert_rate | Number of INSERT queries per second. | ClickHouse |
| Querytime | microsec | meter_clickhouse_instance_querytime_microseconds meter_clickhouse_querytime_microseconds | Total time of all queries. | ClickHouse |
| SelectQuerytime | microsec | meter_clickhouse_instance_querytime_select_microseconds meter_clickhouse_querytime_select_microseconds | Total time of SELECT queries. | ClickHouse |
| InsertQuerytime | microsec | meter_clickhouse_instance_querytime_insert_microseconds meter_clickhouse_querytime_insert_microseconds | Total time of INSERT queries. | ClickHouse |
| OtherQuerytime | microsec | meter_clickhouse_instance_querytime_other_microseconds meter_clickhouse_querytime_other_microseconds | Total time of queries that are not SELECT or INSERT. | ClickHouse |
| QuerySlowCount | count | meter_clickhouse_instance_query_slow meter_clickhouse_query_slow | Number of reads from a file that were slow. | ClickHouse |
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| InsertQueryCount | count | meter_clickhouse_instance_query_insert meter_clickhouse_query_insert | Number of executing queries, but only for INSERT queries. | ClickHouse |
| InsertedRowCount | count | meter_clickhouse_instance_inserted_rows meter_clickhouse_inserted_rows | Number of rows INSERTed to all tables. | ClickHouse |
| InsertedBytes | bytes | meter_clickhouse_instance_inserted_bytes meter_clickhouse_inserted_bytes | Number of bytes INSERTed to all tables. | ClickHouse |
| DelayedInsertCount | count | meter_clickhouse_instance_delayed_insert meter_clickhouse_delayed_insert | Number of times the INSERT of a block to a MergeTree table was throttled due to high number of active data parts for partition. | ClickHouse |
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| ReplicatedChecks | count | meter_clickhouse_instance_replicated_checks meter_clickhouse_replicated_checks | Number of data parts checking for consistency. | ClickHouse |
| ReplicatedFetch | count | meter_clickhouse_instance_replicated_fetch meter_clickhouse_replicated_fetch | Number of data parts being fetched from replica. | ClickHouse |
| ReplicatedSend | count | meter_clickhouse_instance_replicated_send meter_clickhouse_replicated_send | Number of data parts being sent to replicas. | ClickHouse |
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| BackgroundMergeCount | count | meter_clickhouse_instance_background_merge meter_clickhouse_background_merge | Number of executing background merges. | ClickHouse |
| MergeRows | count | meter_clickhouse_instance_merge_rows meter_clickhouse_merge_rows | Rows read for background merges. This is the number of rows before merge. | ClickHouse |
| MergeUncompressedBytes | bytes | meter_clickhouse_instance_merge_uncompressed_bytes meter_clickhouse_merge_uncompressed_bytes | Uncompressed bytes (for columns as they stored in memory) that was read for background merges. This is the number before merge. | ClickHouse |
| MoveCount | count | meter_clickhouse_instance_move meter_clickhouse_move | Number of currently executing moves. | ClickHouse |
| PartsActive | Count | meter_clickhouse_instance_parts_active meter_clickhouse_parts_active | Active data part, used by current and upcoming SELECTs. | ClickHouse |
| MutationsCount | count | meter_clickhouse_instance_mutations meter_clickhouse_mutations | Number of mutations (ALTER DELETE/UPDATE). | ClickHouse |
When table engine works with Apache Kafka.
Kafka lets you:
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| KafkaMessagesRead | count | meter_clickhouse_instance_kafka_messages_read meter_clickhouse_kafka_messages_read | Number of Kafka messages already processed by ClickHouse. | ClickHouse |
| KafkaWrites | count | meter_clickhouse_instance_kafka_writes meter_clickhouse_kafka_writes | Number of writes (inserts) to Kafka tables. | ClickHouse |
| KafkaConsumers | count | meter_clickhouse_instance_kafka_consumers meter_clickhouse_kafka_consumers | Number of active Kafka consumers. | ClickHouse |
| KafkaProducers | count | meter_clickhouse_instance_kafka_producers meter_clickhouse_kafka_producers | Number of active Kafka producer created. | ClickHouse |
ClickHouse uses ZooKeeper for storing metadata of replicas when using replicated tables. If replicated tables are not used, this section of parameters can be omitted.
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| ZookeeperSession | count | meter_clickhouse_instance_zookeeper_session meter_clickhouse_zookeeper_session | Number of sessions (connections) to ZooKeeper. | ClickHouse |
| ZookeeperWatch | count | meter_clickhouse_instance_zookeeper_watch meter_clickhouse_zookeeper_watch | Number of watches (event subscriptions) in ZooKeeper. | ClickHouse |
| ZookeeperBytesSent | bytes | meter_clickhouse_instance_zookeeper_bytes_sent meter_clickhouse_zookeeper_bytes_sent | Number of bytes send over network while communicating with ZooKeeper. | ClickHouse |
| ZookeeperBytesReceive | bytes | meter_clickhouse_instance_zookeeper_bytes_received meter_clickhouse_zookeeper_bytes_received | Number of bytes send over network while communicating with ZooKeeper. | ClickHouse |
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| KeeperAliveConnections | count | meter_clickhouse_instance_keeper_connections_alive meter_clickhouse_keeper_connections_alive | Number of alive connections for embedded ClickHouse Keeper. | ClickHouse |
| KeeperOutstandingRequets | count | meter_clickhouse_instance_keeper_outstanding_requests meter_clickhouse_keeper_outstanding_requests | Number of outstanding requests for embedded ClickHouse Keeper. | ClickHouse |
You can customize your own metrics/expression/dashboard panel. The metrics definition and expression rules are found in /config/otel-rules/clickhouse. The ClickHouse dashboard panel configurations are found in /config/ui-initialized-templates/clickhouse.