Monitor Tool

Prometheus

The mapping from metric type to prometheus format

For metrics whose Metric Name is name and Tags are K1=V1, ..., Kn=Vn, the mapping is as follows, where value is a specific value

Metric Type	Mapping
Counter	name_total{cluster=“clusterName”, nodeType=“nodeType”, nodeId=“nodeId”, k1=“V1”, ..., Kn=“Vn”} value
AutoGauge、Gauge	name{cluster=“clusterName”, nodeType=“nodeType”, nodeId=“nodeId”, k1=“V1”, ..., Kn=“Vn”} value
Histogram	name_max{cluster=“clusterName”, nodeType=“nodeType”, nodeId=“nodeId”, k1=“V1”, ..., Kn=“Vn”} value name_sum{cluster=“clusterName”, nodeType=“nodeType”, nodeId=“nodeId”, k1=“V1”, ..., Kn=“Vn”} value name_count{cluster=“clusterName”, nodeType=“nodeType”, nodeId=“nodeId”, k1=“V1”, ..., Kn=“Vn”} value name{cluster=“clusterName”, nodeType=“nodeType”, nodeId=“nodeId”, k1=“V1”, ..., Kn=“Vn”, quantile=“0.5”} value name{cluster=“clusterName”, nodeType=“nodeType”, nodeId=“nodeId”, k1=“V1”, ..., Kn=“Vn”, quantile=“0.99”} value
Rate	name_total{cluster=“clusterName”, nodeType=“nodeType”, nodeId=“nodeId”, k1=“V1”, ..., Kn=“Vn”} value name_total{cluster=“clusterName”, nodeType=“nodeType”, nodeId=“nodeId”, k1=“V1”, ..., Kn=“Vn”, rate=“m1”} value name_total{cluster=“clusterName”, nodeType=“nodeType”, nodeId=“nodeId”, k1=“V1”, ..., Kn=“Vn”, rate=“m5”} value name_total{cluster=“clusterName”, nodeType=“nodeType”, nodeId=“nodeId”, k1=“V1”, ..., Kn=“Vn”, rate=“m15”} value name_total{cluster=“clusterName”, nodeType=“nodeType”, nodeId=“nodeId”, k1=“V1”, ..., Kn=“Vn”, rate=“mean”} value
Timer	name_seconds_max{cluster=“clusterName”, nodeType=“nodeType”, nodeId=“nodeId”, k1=“V1”, ..., Kn=“Vn”} value name_seconds_sum{cluster=“clusterName”, nodeType=“nodeType”, nodeId=“nodeId”, k1=“V1”, ..., Kn=“Vn”} value name_seconds_count{cluster=“clusterName”, nodeType=“nodeType”, nodeId=“nodeId”, k1=“V1”, ..., Kn=“Vn”} value name_seconds{cluster=“clusterName”, nodeType=“nodeType”, nodeId=“nodeId”, k1=“V1”, ..., Kn=“Vn”, quantile=“0.5”} value name_seconds{cluster=“clusterName”, nodeType=“nodeType”, nodeId=“nodeId”, k1=“V1”, ..., Kn=“Vn”, quantile=“0.99”} value

Config File

1. Taking DataNode as an example, modify the iotdb-datanode.properties configuration file as follows:

dn_metric_reporter_list=PROMETHEUS
dn_metric_level=CORE
dn_metric_prometheus_reporter_port=9091

Then you can get metrics data as follows

2. Start IoTDB DataNodes
3. Open a browser or use curl to visit http://servier_ip:9091/metrics, you can get the following metric data:

...
# HELP file_count
# TYPE file_count gauge
file_count{name="wal",} 0.0
file_count{name="unseq",} 0.0
file_count{name="seq",} 2.0
...

Prometheus + Grafana

As shown above, IoTDB exposes monitoring metrics data in the standard Prometheus format to the outside world. Prometheus can be used to collect and store monitoring indicators, and Grafana can be used to visualize monitoring indicators.

The following picture describes the relationships among IoTDB, Prometheus and Grafana

1. Along with running, IoTDB will collect its metrics continuously.
2. Prometheus scrapes metrics from IoTDB at a constant interval (can be configured).
3. Prometheus saves these metrics to its inner TSDB.
4. Grafana queries metrics from Prometheus at a constant interval (can be configured) and then presents them on the graph.

So, we need to do some additional works to configure and deploy Prometheus and Grafana.

For instance, you can config your Prometheus as follows to get metrics data from IoTDB:

job_name: pull-metrics
honor_labels: true
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
follow_redirects: true
static_configs:
  - targets:
      - localhost:9091

The following documents may help you have a good journey with Prometheus and Grafana.

Prometheus getting_started

Prometheus scrape metrics

Grafana getting_started

Grafana query metrics from Prometheus