Paimon Metrics

Paimon has built a metrics system to measure the behaviours of reading and writing, like how many manifest files it scanned in the last planning, how long it took in the last commit operation, how many files it deleted in the last compact operation.

In Paimon's metrics system, metrics are updated and reported at table granularity.

There are three types of metrics provided in the Paimon metric system, Gauge, Counter, Histogram.

Gauge: Provides a value of any type at a point in time.
Counter: Used to count values by incrementing and decrementing.
Histogram: Measure the statistical distribution of a set of values including the min, max, mean, standard deviation and percentile.

Paimon has supported built-in metrics to measure operations of commits, scans, writes and compactions, which can be bridged to any computing engine that supports, like Flink, Spark etc.

Metrics List

Below is lists of Paimon built-in metrics. They are summarized into types of scan metrics, commit metrics, write metrics, write buffer metrics and compaction metrics.

Scan Metrics

Commit Metrics

Write Buffer Metrics

Compaction Metrics

Bridging To Flink

Paimon has implemented bridging metrics to Flink's metrics system, which can be reported by Flink, and the lifecycle of metric groups are managed by Flink.

Please join the <scope>.<infix>.<metric_name> to get the complete metric identifier when using Flink to access Paimon, metric_name can be got from [Metric List]({{< ref “maintenance/metrics#metrics-list” >}}).

For example, the identifier of metric lastPartitionsWritten for table word_count in Flink job named insert_word_count is:

localhost.taskmanager.localhost:60340-775a20.insert_word_count.Global Committer : word_count.0.paimon.table.word_count.commit.lastPartitionsWritten.

From Flink Web-UI, go to the committer operator‘s metrics, it’s shown as:

0.Global_Committer___word_count.paimon.table.word_count.commit.lastPartitionsWritten.

Please refer to System Scope to understand Flink scope
Scan metrics are only supported by Flink versions >= 1.18 {{< /hint >}}

Flink Connector Standard Metrics

When using Flink to read and write, Paimon has implemented some key standard Flink connector metrics to measure the source latency and output of sink, see FLIP-33: Standardize Connector Metrics. Flink source / sink metrics implemented are listed here.

Source Metrics (Flink)

{{< hint info >}} Please note that if you specified consumer-id in your streaming query, the level of source metrics should turn into the reader operator, which is behind the Monitor operator. {{< /hint >}}