PIP-355: Enhancing Broker-Level Metrics for Pulsar

Background Knowledge

Pulsar provides broker-level, namespace-level, and topic-level metrics to monitor and analyze the behavior of the Pulsar service. These metrics are accessible through the Prometheus metrics endpoint. Detailed explanations of all metrics can be found on the Pulsar website: Pulsar Metrics Reference

Motivation

Within Pulsar's current metrics framework, the pulsar_out_bytes_total metric is utilized to expose the total bytes dispatched by the broker to consumers. However, there are notable limitations and challenges associated with this metric:

Inclusion of system subscriptions in the total bytes out, alongside user subscriptions, complicates accurate calculation of user-specific data.
The granularity of the metric (namespace-level vs. topic-subscription level) impacts the scalability and resource consumption when calculating cluster-level total out bytes.

Goals

This proposal aims to address the following objectives:

Simplify the process of calculating cluster-level total out bytes.
Enable the calculation of total out bytes dispatched to system subscriptions.

High-Level Design

To achieve the outlined goals, the proposal introduces two new broker-level metrics:

pulsar_broker_out_bytes_total{system_subscription="true|false"}: Represents the total out bytes dispatched by the broker to consumers. The label system_subscription="false" represents total traffic dispatched to user subscriptions, while system_subscription="true" represents total traffic dispatched to system cursors and cursor names added by additionalSystemCursorNames introduced in PIP-349.
pulsar_broker_in_bytes_total{system_topic="true|false"}: Tracks the total in bytes sent by producers to the broker. The label system_topic="false" represents total traffic from user topics, while system_topic="true" represents total traffic from system topics.

Detailed Design

The implementation involves the introduction of the following broker-level metrics:

pulsar_broker_out_bytes_total{system_subscription="true|false"}: Aggregates the total out bytes from all topics, presented as a broker-level metric.
pulsar_broker_in_bytes_total{system_topic="true|false"}: Calculation of total in bytes across all topics.

Metrics

The proposal includes the addition of two new broker-level metrics:

pulsar_broker_out_bytes_total{system_subscription="true|false"}
pulsar_broker_in_bytes_total{system_topic="true|false"}

Backward & Forward Compatibility

The proposed changes ensure full compatibility with existing systems and pave the way for seamless integration with future enhancements.