Pulsar exposes metrics in Prometheus format that can be collected and used for monitoring the health of the cluster.
The metrics exposed by Pulsar are in Prometheus format. The types of metrics are:
_bucket
suffix is the number of observations within a histogram bucket, configured with parameter {le="<upper inclusive bound>"}
. The _count
suffix is the number of observations, shown as a time series and behaves like a counter. The _sum
suffix is the sum of observed values, also shown as a time series and behaves like a counter. These suffixes are together denoted by _*
in this doc.The ZooKeeper metrics are exposed under “/metrics” at port 8000. You can use a different port by configuring the stats_server_port
system property.
Name | Type | Description |
---|---|---|
zookeeper_server_znode_count | Gauge | The number of z-nodes stored. |
zookeeper_server_data_size_bytes | Gauge | The total size of all of z-nodes stored. |
zookeeper_server_connections | Gauge | The number of currently opened connections. |
zookeeper_server_watches_count | Gauge | The number of watchers registered. |
zookeeper_server_ephemerals_count | Gauge | The number of ephemeral z-nodes. |
Name | Type | Description |
---|---|---|
zookeeper_server_requests | Counter | The total number of requests received by a particular server. |
zookeeper_server_requests_latency_ms | Summary | The requests latency calculated in milliseconds. Available labels: type (write, read). write: the requests that write data to ZooKeeper.read: the requests that read data from ZooKeeper. |
The BookKeeper metrics are exposed under “/metrics” at port 8000. You can change the port by updating prometheusStatsHttpPort
in bookkeeper.conf
configuration file.
Name | Type | Description |
---|---|---|
bookie_SERVER_STATUS | Gauge | The server status for bookie server. 1: the bookie is running in writable mode.0: the bookie is running in readonly mode. |
bookkeeper_server_ADD_ENTRY_count | Counter | The total number of ADD_ENTRY requests received at the bookie. The success label is used to distinguish successes and failures. |
bookkeeper_server_READ_ENTRY_count | Counter | The total number of READ_ENTRY requests received at the bookie. The success label is used to distinguish successes and failures. |
bookie_WRITE_BYTES | Counter | The total number of bytes written to the bookie. |
bookie_READ_BYTES | Counter | The total number of bytes read from the bookie. |
bookkeeper_server_ADD_ENTRY_REQUEST | Summary | The summary of request latency of ADD_ENTRY requests at the bookie. The success label is used to distinguish successes and failures. |
bookkeeper_server_READ_ENTRY_REQUEST | Summary | The summary of request latency of READ_ENTRY requests at the bookie. The success label is used to distinguish successes and failures. |
Name | Type | Description |
---|---|---|
bookie_journal_JOURNAL_SYNC_count | Counter | The total number of journal fsync operations happening at the bookie. The success label is used to distinguish successes and failures. |
bookie_journal_JOURNAL_QUEUE_SIZE | Gauge | The total number of requests pending in the journal queue. |
bookie_journal_JOURNAL_FORCE_WRITE_QUEUE_SIZE | Gauge | The total number of force write (fsync) requests pending in the force-write queue. |
bookie_journal_JOURNAL_CB_QUEUE_SIZE | Gauge | The total number of callbacks pending in the callback queue. |
bookie_journal_JOURNAL_ADD_ENTRY | Summary | The summary of request latency of adding entries to the journal. |
bookie_journal_JOURNAL_SYNC | Summary | The summary of fsync latency of syncing data to the journal disk. |
Name | Type | Description |
---|---|---|
bookie_ledgers_count | Gauge | The total number of ledgers stored in the bookie. |
bookie_entries_count | Gauge | The total number of entries stored in the bookie. |
bookie_write_cache_size | Gauge | The bookie write cache size (in bytes). |
bookie_read_cache_size | Gauge | The bookie read cache size (in bytes). |
bookie_DELETED_LEDGER_COUNT | Counter | The total number of ledgers deleted since the bookie has started. |
bookie_ledger_writable_dirs | Gauge | The number of writable directories in the bookie. |
The broker metrics are exposed under “/metrics/” at port 8080. You can change the port by updating webServicePort
to a different port in broker.conf
configuration file.
All the metrics exposed by a broker are labelled with cluster=${pulsar_cluster}
. The value of ${pulsar_cluster}
is the pulsar cluster name you configured in broker.conf
.
Broker has the following kinds of metrics:
Namespace metrics are only exposed when
exposeTopicLevelMetricsInPrometheus
is set tofalse
.
All the namespace metrics are labelled with the following labels:
cluster=${pulsar_cluster}
. ${pulsar_cluster}
is the cluster name that you configured in broker.conf
.namespace=${pulsar_namespace}
. ${pulsar_namespace}
is the namespace name.Name | Type | Description |
---|---|---|
pulsar_topics_count | Gauge | The number of Pulsar topics of the namespace owned by this broker. |
pulsar_subscriptions_count | Gauge | The number of Pulsar subscriptions of the namespace served by this broker. |
pulsar_producers_count | Gauge | The number of active producers of the namespace connected to this broker. |
pulsar_consumers_count | Gauge | The number of active consumers of the namespace connected to this broker. |
pulsar_rate_in | Gauge | The total message rate of the namespace coming into this broker (messages/second). |
pulsar_rate_out | Gauge | The total message rate of the namespace going out from this broker (messages/second). |
pulsar_throughput_in | Gauge | The total throughput of the namespace coming into this broker (bytes/second). |
pulsar_throughput_out | Gauge | The total throughput of the namespace going out from this broker (bytes/second). |
pulsar_storage_size | Gauge | The total storage size of the topics in this namespace owned by this broker (bytes). |
pulsar_storage_backlog_size | Gauge | The total backlog size of the topics of this namespace owned by this broker (messages). |
pulsar_storage_offloaded_size | Gauge | The total amount of the data in this namespace offloaded to the tiered storage (bytes). |
pulsar_storage_write_rate | Gauge | The total message batches (entries) written to the storage for this namespace (message batches / second). |
pulsar_storage_read_rate | Gauge | The total message batches (entries) read from the storage for this namespace (message batches / second). |
pulsar_subscription_delayed | Gauge | The total message batches (entries) are delayed for dispatching. |
pulsar_storage_write_latency_le_* | Histogram | The entry rate of a namespace that the storage write latency is smaller with a given threshold. Available thresholds: pulsar_storage_write_latency_le_0_5: <= 0.5ms pulsar_storage_write_latency_le_1: <= 1mspulsar_storage_write_latency_le_5: <= 5mspulsar_storage_write_latency_le_10: <= 10mspulsar_storage_write_latency_le_20: <= 20mspulsar_storage_write_latency_le_50: <= 50mspulsar_storage_write_latency_le_100: <= 100mspulsar_storage_write_latency_le_200: <= 200mspulsar_storage_write_latency_le_1000: <= 1spulsar_storage_write_latency_le_overflow: > 1s |
pulsar_entry_size_le_* | Histogram | The entry rate of a namespace that the entry size is smaller with a given threshold. Available thresholds: pulsar_entry_size_le_128: <= 128 bytes pulsar_entry_size_le_512: <= 512 bytespulsar_entry_size_le_1_kb: <= 1 KBpulsar_entry_size_le_2_kb: <= 2 KBpulsar_entry_size_le_4_kb: <= 4 KBpulsar_entry_size_le_16_kb: <= 16 KBpulsar_entry_size_le_100_kb: <= 100 KBpulsar_entry_size_le_1_mb: <= 1 MBpulsar_entry_size_le_overflow: > 1 MB |
If a namespace is configured to be replicated between multiple Pulsar clusters, the corresponding replication metrics will also be exposed when replicationMetricsEnabled
is enabled.
All the replication metrics will also be labelled with remoteCluster=${pulsar_remote_cluster}
.
Name | Type | Description |
---|---|---|
pulsar_replication_rate_in | Gauge | The total message rate of the namespace replicating from remote cluster (messages/second). |
pulsar_replication_rate_out | Gauge | The total message rate of the namespace replicating to remote cluster (messages/second). |
pulsar_replication_throughput_in | Gauge | The total throughput of the namespace replicating from remote cluster (bytes/second). |
pulsar_replication_throughput_out | Gauge | The total throughput of the namespace replicating to remote cluster (bytes/second). |
pulsar_replication_backlog | Gauge | The total backlog of the namespace replicating to remote cluster (messages). |
Topic metrics are only exposed when
exposeTopicLevelMetricsInPrometheus
is set to true.
All the topic metrics are labelled with the following labels:
cluster=${pulsar_cluster}
. ${pulsar_cluster}
is the cluster name that you configured in broker.conf
.namespace=${pulsar_namespace}
. ${pulsar_namespace}
is the namespace name.topic=${pulsar_topic}
. ${pulsar_topic}
is the topic name.Name | Type | Description |
---|---|---|
pulsar_subscriptions_count | Gauge | The number of Pulsar subscriptions of the topic served by this broker. |
pulsar_producers_count | Gauge | The number of active producers of the topic connected to this broker. |
pulsar_consumers_count | Gauge | The number of active consumers of the topic connected to this broker. |
pulsar_rate_in | Gauge | The total message rate of the topic coming into this broker (messages/second). |
pulsar_rate_out | Gauge | The total message rate of the topic going out from this broker (messages/second). |
pulsar_throughput_in | Gauge | The total throughput of the topic coming into this broker (bytes/second). |
pulsar_throughput_out | Gauge | The total throughput of the topic going out from this broker (bytes/second). |
pulsar_storage_size | Gauge | The total storage size of the topics in this topic owned by this broker (bytes). |
pulsar_storage_backlog_size | Gauge | The total backlog size of the topics of this topic owned by this broker (messages). |
pulsar_storage_offloaded_size | Gauge | The total amount of the data in this topic offloaded to the tiered storage (bytes). |
pulsar_storage_backlog_quota_limit | Gauge | The total amount of the data in this topic that limit the backlog quota (bytes). |
pulsar_storage_write_rate | Gauge | The total message batches (entries) written to the storage for this topic (message batches / second). |
pulsar_storage_read_rate | Gauge | The total message batches (entries) read from the storage for this topic (message batches / second). |
pulsar_subscription_delayed | Gauge | The total message batches (entries) are delayed for dispatching. |
pulsar_storage_write_latency_le_* | Histogram | The entry rate of a topic that the storage write latency is smaller with a given threshold. Available thresholds: pulsar_storage_write_latency_le_0_5: <= 0.5ms pulsar_storage_write_latency_le_1: <= 1mspulsar_storage_write_latency_le_5: <= 5mspulsar_storage_write_latency_le_10: <= 10mspulsar_storage_write_latency_le_20: <= 20mspulsar_storage_write_latency_le_50: <= 50mspulsar_storage_write_latency_le_100: <= 100mspulsar_storage_write_latency_le_200: <= 200mspulsar_storage_write_latency_le_1000: <= 1spulsar_storage_write_latency_le_overflow: > 1s |
pulsar_entry_size_le_* | Histogram | The entry rate of a topic that the entry size is smaller with a given threshold. Available thresholds: pulsar_entry_size_le_128: <= 128 bytes pulsar_entry_size_le_512: <= 512 bytespulsar_entry_size_le_1_kb: <= 1 KBpulsar_entry_size_le_2_kb: <= 2 KBpulsar_entry_size_le_4_kb: <= 4 KBpulsar_entry_size_le_16_kb: <= 16 KBpulsar_entry_size_le_100_kb: <= 100 KBpulsar_entry_size_le_1_mb: <= 1 MBpulsar_entry_size_le_overflow: > 1 MB |
pulsar_in_bytes_total | Counter | The total number of bytes received for this topic |
pulsar_producers_count | Counter | The total number of messages received for this topic |
If a namespace that a topic belongs to is configured to be replicated between multiple Pulsar clusters, the corresponding replication metrics will also be exposed when replicationMetricsEnabled
is enabled.
All the replication metrics will also be labelled with remoteCluster=${pulsar_remote_cluster}
.
Name | Type | Description |
---|---|---|
pulsar_replication_rate_in | Gauge | The total message rate of the topic replicating from remote cluster (messages/second). |
pulsar_replication_rate_out | Gauge | The total message rate of the topic replicating to remote cluster (messages/second). |
pulsar_replication_throughput_in | Gauge | The total throughput of the topic replicating from remote cluster (bytes/second). |
pulsar_replication_throughput_out | Gauge | The total throughput of the topic replicating to remote cluster (bytes/second). |
pulsar_replication_backlog | Gauge | The total backlog of the topic replicating to remote cluster (messages). |
Subscription metrics are only exposed when
exposeTopicLevelMetricsInPrometheus
is set to true.
All the subscription metrics are labelled with the following labels:
cluster=${pulsar_cluster}
. ${pulsar_cluster}
is the cluster name that you configured in broker.conf
.namespace=${pulsar_namespace}
. ${pulsar_namespace}
is the namespace name.topic=${pulsar_topic}
. ${pulsar_topic}
is the topic name.subscription=${subscription}
. ${subscription}
is the topic subscription name.Name | Type | Description |
---|---|---|
pulsar_subscription_back_log | Gauge | The total backlog of a subscription (messages). |
pulsar_subscription_delayed | Gauge | The total number of messages are delayed to be dispatched for a subscription (messages). |
pulsar_subscription_msg_rate_redeliver | Gauge | The total message rate for message being redelivered (messages/second). |
pulsar_subscription_unacked_messages | Gauge | The total number of unacknowledged messages of a subscription (messages). |
pulsar_subscription_blocked_on_unacked_messages | Gauge | Indicate whether a subscription is blocked on unacknowledged messages or not. 1 means the subscription is blocked on waiting unacknowledged messages to be acked.0 means the subscription is not blocked on waiting unacknowledged messages to be acked. |
pulsar_subscription_msg_rate_out | Gauge | The total message dispatch rate for a subscription (messages/second). |
pulsar_subscription_msg_throughput_out | Gauge | The total message dispatch throughput for a subscription (bytes/second). |
Consumer metrics are only exposed when both
exposeTopicLevelMetricsInPrometheus
andexposeConsumerLevelMetricsInPrometheus
are set to true.
All the consumer metrics are labelled with the following labels:
cluster=${pulsar_cluster}
. ${pulsar_cluster}
is the cluster name that you configured in broker.conf
.namespace=${pulsar_namespace}
. ${pulsar_namespace}
is the namespace name.topic=${pulsar_topic}
. ${pulsar_topic}
is the topic name.subscription=${subscription}
. ${subscription}
is the topic subscription name.consumer_name=${consumer_name}
. ${consumer_name}
is the topic consumer name.consumer_id=${consumer_id}
. ${consumer_id}
is the topic consumer id.Name | Type | Description |
---|---|---|
pulsar_consumer_msg_rate_redeliver | Gauge | The total message rate for message being redelivered (messages/second). |
pulsar_consumer_unacked_messages | Gauge | The total number of unacknowledged messages of a consumer (messages). |
pulsar_consumer_blocked_on_unacked_messages | Gauge | Indicate whether a consumer is blocked on unacknowledged messages or not. 1 means the consumer is blocked on waiting unacknowledged messages to be acked.0 means the consumer is not blocked on waiting unacknowledged messages to be acked. |
pulsar_consumer_msg_rate_out | Gauge | The total message dispatch rate for a consumer (messages/second). |
pulsar_consumer_msg_throughput_out | Gauge | The total message dispatch throughput for a consumer (bytes/second). |
pulsar_consumer_available_permits | Gauge | The available permits for for a consumer. |
For a functions-worker running separately from brokers, its Jetty metrics are only exposed when
includeStandardPrometheusMetrics
is set totrue
.
All the jetty metrics are labelled with the following labels:
cluster=${pulsar_cluster}
. ${pulsar_cluster}
is the cluster name that you have configured in the broker.conf
file.Name | Type | Description |
---|---|---|
jetty_requests_total | Counter | Number of requests. |
jetty_requests_active | Gauge | Number of requests currently active. |
jetty_requests_active_max | Gauge | Maximum number of requests that have been active at once. |
jetty_request_time_max_seconds | Gauge | Maximum time spent handling requests. |
jetty_request_time_seconds_total | Counter | Total time spent in all request handling. |
jetty_dispatched_total | Counter | Number of dispatches. |
jetty_dispatched_active | Gauge | Number of dispatches currently active. |
jetty_dispatched_active_max | Gauge | Maximum number of active dispatches being handled. |
jetty_dispatched_time_max | Gauge | Maximum time spent in dispatch handling. |
jetty_dispatched_time_seconds_total | Counter | Total time spent in dispatch handling. |
jetty_async_requests_total | Counter | Total number of async requests. |
jetty_async_requests_waiting | Gauge | Currently waiting async requests. |
jetty_async_requests_waiting_max | Gauge | Maximum number of waiting async requests. |
jetty_async_dispatches_total | Counter | Number of requested that have been asynchronously dispatched. |
jetty_expires_total | Counter | Number of async requests requests that have expired. |
jetty_responses_total | Counter | Number of responses, labeled by status code. The code label can be “1xx”, “2xx”, “3xx”, “4xx”, or “5xx”. |
jetty_stats_seconds | Gauge | Time in seconds stats have been collected for. |
jetty_responses_bytes_total | Counter | Total number of bytes across all responses. |
You can set up a Prometheus instance to collect all the metrics exposed at Pulsar components and set up Grafana dashboards to display the metrics and monitor your Pulsar cluster.
The following are some Grafana dashboards examples: