pip/pip-320.md - pulsar - Git at Google

 # PIP-320: OpenTelemetry Scaffolding

 # Background knowledge

 ## PIP-264 - parent PIP titled "Enhanced OTel-based metric system"
 [PIP-264](https://github.com/apache/pulsar/pull/21080), which can also be viewed [here](pip-264.md), describes in high
 level a plan to greatly enhance Pulsar metric system by replacing it with [OpenTelemetry](https://opentelemetry.io/).
 You can read in the PIP the numerous existing problems PIP-264 solves. Among them are:
 - Control which metrics to export per topic/group/namespace via the introduction of a metric filter configuration.
   This configuration is planned to be dynamic as outline in the [PIP-264](pip-264.md).
 - Reduce the immense metrics cardinality due to high topic count (One of Pulsar great features), by introducing
 the concept of Metric Group - a group of topics for metric purposes. Metric reporting will also be done to a
 group granularity. 100k topics can be downsized to 1k groups. The dynamic metric filter configuration would allow
 the user to control which metric group to un-filter.
 - Proper histogram exporting
 - Clean-up codebase clutter, by relying on a single industry standard API, SDK and metrics protocol (OTLP) instead of
 existing mix of home-brew libraries and hard coded Prometheus exporter.
 - any many more

 You can [here](pip-264.md#why-opentelemetry) why OpenTelemetry was chosen.

 ## OpenTelemetry
 Since OpenTelemetry (a.k.a. OTel) is an emerging industry standard, there are plenty of good articles, videos and
 documentation about it. In this very short paragraph I'll describe what you need to know about OTel from this PIP
 perspective.

 OpenTelemetry is a project aimed to standardize the way we instrument, collect and ship metrics from applications
 to telemetry backends, be it databases (e.g. Prometheus, Cortex, Thanos) or vendors (e.g. Datadog, Logz.io).
 It is divided into API, SDK and Collector:
 - API: interfaces to use to instrument: define a counter, record values to a histogram, etc.
 - SDK: a library, available in many languages, implementing the API, and other important features such as
 reading the metrics and exporting it out to a telemetry backend or OTel Collector.
 - Collector: a lightweight process (application) which can receive or retrieve telemetry, transform it (e.g.
 filter, drop, aggregate)  and export it (e.g. send it to various backends). The SDK supports out-of-the-box
 exporting metrics as Prometheus HTTP endpoint or sending them out using OTLP protocol. Many times companies choose to
 ship to the Collector and there ship to their preferred vendors, since each vendor already published their exporter
 plugin to OTel Collector. This makes the SDK exporters very light-weight as they don't need to support any
 vendor. It's also easier for the DevOps team as they can make OTel Collector their responsibility, and have
 application developers only focus on shipping metrics to that collector.

 Just to have some context: Pulsar codebase will use the OTel API to create counters / histograms and records values to
 them. So will the Pulsar plugins and Pulsar Function authors. Pulsar itself will be the one creating the SDK
 and using that to hand over an implementation of the API where ever needed in Pulsar. Collector is up to the choice
 of the user, as OTel provides a way to expose the metrics as `/metrics` endpoint on a configured port, so Prometheus
 compatible scrapers can grab it from it directly. They can also send it via OTLP to OTel collector.

 ## Telemetry layers
 PIP-264 clearly outlined there will be two layers of metrics, collected and exported, side by side: OpenTelemetry
 and the existing metric system - currently exporting in Prometheus. This PIP will explain in detail how it will work.
 The basic premise is that you will be able to enable or disable OTel metrics, alongside the existing Prometheus
 metric exporting.

 ## Why OTel in Pulsar will be marked experimental and not GA
 As specified in [PIP-264](pip-264.md), OpenTelemetry Java SDK has several fixes the Pulsar community must
 complete before it can be used in production. They are [documented](pip-264.md#what-we-need-to-fix-in-opentelemetry)
 in PIP-264. The most important one is reducing memory allocations to be negligible. OTel SDK is built upon immutability,
 hence allocated memory in O(`#topics`) which is a performance killer for low latency application like Pulsar.

 You can track the proposal and progress the Pulsar and OTel communities are making in
 [this issue](https://github.com/open-telemetry/opentelemetry-java/issues/5105).


 ## Metrics endpoint authentication
 Today Pulsar metrics endpoint `/metrics` has an option to be protected by the configured `AuthenticationProvider`.
 The configuration option is named `authenticateMetricsEndpoint` in the broker and
 `authenticateMetricsEndpoint` in the proxy.


 # Motivation

 Implementing PIP-264 consists of a long list of steps, which are detailed in
 [this issue](https://github.com/apache/pulsar/issues/21121). The first step is add all the bare-bones infrastructure
 to use OpenTelemetry in Pulsar, such that next PRs can use it to start translating existing metrics to their
 OTel form. It means the same metrics will co-exist in the codebase and also in runtime, if OTel was enabled.

 # Goals

 ## In Scope
 - Ability to add metrics using OpenTelemetry to Pulsar components: Broker, Function Worker and Proxy.
 - User can disable or enable OpenTelemetry metrics, which by default will be disabled
 - OpenTelemetry metrics will be configured via its native OTel Java SDK configuration options
 - All the necessary information to use OTel with Pulsar will be documented in Pulsar documentation site
 - OpenTelemetry metrics layer defined as experimental, and *not* GA


 ## Out of Scope
 - Ability to add metrics using OpenTelemetry as Pulsar Function author.
 - Only authenticated sessions can access OTel Prometheus endpoint, using Pulsar authentication
 - Metrics in Pulsar clients (as defined in [PIP-264](pip-264.md#out-of-scope)))

 # High Level Design

 ## Configuration
 OpenTelemetry, as any good telemetry library (e.g. log4j, logback), has its own configuration mechanisms:
 - System properties
 - Environment variables
 - Experimental file-based configuration

 Pulsar doesn't need to introduce any additional configuration. The user can decide, using OTel configuration
 things like:
 * How do I want to export the metrics? Prometheus? Which port prometheus will be exposed at
 * Change histogram buckets using Views
 * and more

 Pulsar will use `AutoConfiguredOpenTelemetrySdk` which uses all the above configuration mechanisms
 (documented [here](https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/autoconfigure)).
 This class builds an `OpenTelemetrySdk` based on configurations. This is the entry point to OpenTelemetry API, as it
 implements `OpenTelemetry` API class.

 ### Setting sensible defaults for Pulsar
 There are some configuration options we wish to change their default, but still allow the users to override it
 if they wish. We think those default values will make a much easier user experience.

 * `otel.experimental.metrics.cardinality.limit` - value: 10,000
 This property sets an upper bound on the amount of unique `Attributes` an instrument can have. Take Pulsar for example,
 an instrument like `pulsar.broker.messaging.topic.received.size`, the unique `Attributes` would be in the amount of
 active topics in the broker. Since Pulsar can handle up to 1M topics, it makes more sense to put the default value
 to 10k, which translates to 10k topics.

 `AutoConfiguredOpenTelemetrySdkBuilder` allows to add properties using the method `addPropertiesSupplier`. The
 System properties and environment variables override it. The file-based configuration still doesn't take
 those properties supplied into account, but it will.


 ## Opting in
 We would like to have the ability to toggle OpenTelemetry-based metrics, as they are still new.
 We won't need any special Pulsar configuration, as OpenTelemetry SDK comes with a configuration key to do that.
 Since OTel is still experimental, it will have to be opt-in, hence we will add the following property to be the default
 using the mechanism described [above](#setting-sensible-defaults-for-pulsar):

 * `otel.sdk.disabled` - value: true
   This property value disables OpenTelemetry.

 With OTel disabled, the user remains with the existing metrics system. OTel in a disabled state operates in a
 no-op mode. This means, instruments do get built, but the instrument builders return the same instance of a
 no-op instrument, which does nothing on record-values method (e.g. `add(number)`, `record(number)`). The no-op
 `MeterProvider` has no registered `MetricReader` hence no metric collection will be made. The memory impact
 is almost 0 and the same goes for CPU impact.

 The current metric system doesn't have a toggle which causes all existing data structures to stop collecting
 data. Inserting will need changing in so many places since we don't have a single place which through
 all metric instrument are created (one of the motivations for PIP-264).
 The current system do have a toggle: `exposeTopicLevelMetricsInPrometheus`. It enables toggling off
 topic-level metrics, which means the highest cardinality metrics will be namespace level.
 Once that toggle is `false`, the amount of data structures accounting memory would in the range of
 a few thousands which shouldn't post a burden memory wise. If the user refrain from calling
 `/metrics` it will also reduce the CPU and memory cost associated with collecting metrics.

 When the user enables OTel it means there will be a memory increase, but if the user disabled topic-level
 metrics in existing system, as specified above, the majority of the memory increase will be due to topic level
 metrics in OTel, at the expense of not having them in the existing metric system.


 ## Cluster attribute name
 A broker is part of a cluster. It is configured in the Pulsar configuration key `clusterName`. When the broker is part
 of a cluster, it means it shares the topics defined in that cluster (persisted in Metadata service: e.g. ZK)
 among the brokers of that cluster.

 Today, each unique time series emitted in Prometheus metrics contains the `cluster` label (almost all of them, as it
 is done manually). We wish the same with OTel - to have that attribute in each exported unique time series.

 OTel has the perfect location to place attributes which are shared across all time series: Resource. An application
 can have multiple Resource, with each having 1 or more attributes. You define it once, in OTel initialization or
 configuration. It can contain attributes like the hostname, AWS region, etc. The default contains the service name
 and some info on the SDK version.

 Attributes can be added dynamically, through `addResourceCustomizer()` in `AutoConfiguredOpenTelemetrySdkBuilder`.
 We will use that to inject the `cluster` attribute, taken from the configuration.

 In Prometheus, we submitted a [proposal](https://github.com/open-telemetry/opentelemetry-specification/pull/3761)
 to opentelemetry specifications, which was merged, to allow copying resource attributes into each exported
 unique time series in Prometheus exporter.
 We plan to contribute its implementation to OTel Java SDK.

 Resources in Prometheus exporter, are exported as `target_info{} 1` and the attributes are added to this
 time series. This will require making joins to get it, making it extremely difficult to use.
 The other alternative was to introduce our own `PulsarAttributesBuilder` class, on top of
 `AttributesBuilder` of OTel. Getting every contributor to know this class, use it, is hard. Getting this
 across Pulsar Functions or Plugins authors, will be immensely hard.  Also, when exporting as
 OTLP, it is very inefficient to repeat the attribute across all unique time series, instead of once using
 Resource. Hence, this needed to be solved in the Prometheus exporter as we did in the proposal.

 The attribute will be named `pulsar.cluster`, as both the proxy and the broker are part of this cluster.

 ## Naming and using OpenTelemetry

 ### Attributes
 * We shall prefix each attribute with `pulsar.`. Example: `pulsar.topic`, `pulsar.cluster`.

 ### Instruments
 We should have a clear hierarchy, hence use the following prefix
 * `pulsar.broker`
 * `pulsar.proxy`
 * `pulsar.function_worker`

 ### Meter
 It's customary to use reverse domain name for meter names. Hence, we'll use:
 * `org.apache.pulsar.broker`
 * `org.apache.pulsar.proxy`
 * `org.apache.pulsar.function_worker`

 OTel meter name is converted to the attribute name `otel_scope_name` and added to each unique time series
 attributes by Prometheus exporter.

 We won't specify a meter version, as it is used solely to signify the version of the instrumentation, and
 currently we are the first version, hence not use it.


 # Detailed Design

 ## Design & Implementation Details

 * `OpenTelemetryService` class
    * Parameters:
       * Cluster name
   * What it will do:
      - Override default max cardinality to 10k
      - Register a resource with cluster name
      - Place defaults setting to instruct Prometheus Exporter to copy resource attributes
      - In the future: place defaults for Memory Mode to be REUSABLE_DATA

 * `PulsarBrokerOpenTelemetry` class
   * Initialization
     * Construct an `OpenTelemetryService` using the cluster name taken from the broker configuration
     * Constructs a Meter for the broker metrics
   * Methods
     * `getMeter()` returns the `Meter` for the broker
   * Notes
     * This is the class that will be passed along to other Pulsar service classes that needs to define
       telemetry such as metrics (in the future: traces).

 * `PulsarProxyOpenTelemetry` class
   * Same as `PulsarBrokerOpenTelemetry` but for Pulsar Proxy
 * `PulsarWorkerOpenTelemetry` class
   * Same as `PulsarBrokerOpenTelemetry` but for Pulsar function worker


 ## Public-facing Changes

 ### Public API
 * OTel Prometheus Exporter adds `/metrics` endpoint on a user defined port, if user chose to use it

 ### Configuration
 * OTel configurations are used

 # Security Considerations
 * OTel currently does not support setting a custom Authenticator for Prometheus exporter.
 An issue has been raised [here](https://github.com/open-telemetry/opentelemetry-java/issues/6013).
    * Once it do we can secure the Prometheus exporter metrics endpoint using `AuthenticationProvider`
 * Any user can access metrics, and they are not protected per tenant. Like today's implementation

 # Links

 * Mailing List discussion thread: https://lists.apache.org/thread/xcn9rm551tyf4vxrpb0th0wj0kktnrr2
 * Mailing List voting thread: https://lists.apache.org/thread/zp6vl9z9dhwbvwbplm60no13t8fvlqs2
	# PIP-320: OpenTelemetry Scaffolding

	# Background knowledge

	## PIP-264 - parent PIP titled "Enhanced OTel-based metric system"
	[PIP-264](https://github.com/apache/pulsar/pull/21080), which can also be viewed [here](pip-264.md), describes in high
	level a plan to greatly enhance Pulsar metric system by replacing it with [OpenTelemetry](https://opentelemetry.io/).
	You can read in the PIP the numerous existing problems PIP-264 solves. Among them are:
	- Control which metrics to export per topic/group/namespace via the introduction of a metric filter configuration.
	This configuration is planned to be dynamic as outline in the [PIP-264](pip-264.md).
	- Reduce the immense metrics cardinality due to high topic count (One of Pulsar great features), by introducing
	the concept of Metric Group - a group of topics for metric purposes. Metric reporting will also be done to a
	group granularity. 100k topics can be downsized to 1k groups. The dynamic metric filter configuration would allow
	the user to control which metric group to un-filter.
	- Proper histogram exporting
	- Clean-up codebase clutter, by relying on a single industry standard API, SDK and metrics protocol (OTLP) instead of
	existing mix of home-brew libraries and hard coded Prometheus exporter.
	- any many more

	You can [here](pip-264.md#why-opentelemetry) why OpenTelemetry was chosen.

	## OpenTelemetry
	Since OpenTelemetry (a.k.a. OTel) is an emerging industry standard, there are plenty of good articles, videos and
	documentation about it. In this very short paragraph I'll describe what you need to know about OTel from this PIP
	perspective.

	OpenTelemetry is a project aimed to standardize the way we instrument, collect and ship metrics from applications
	to telemetry backends, be it databases (e.g. Prometheus, Cortex, Thanos) or vendors (e.g. Datadog, Logz.io).
	It is divided into API, SDK and Collector:
	- API: interfaces to use to instrument: define a counter, record values to a histogram, etc.
	- SDK: a library, available in many languages, implementing the API, and other important features such as
	reading the metrics and exporting it out to a telemetry backend or OTel Collector.
	- Collector: a lightweight process (application) which can receive or retrieve telemetry, transform it (e.g.
	filter, drop, aggregate) and export it (e.g. send it to various backends). The SDK supports out-of-the-box
	exporting metrics as Prometheus HTTP endpoint or sending them out using OTLP protocol. Many times companies choose to
	ship to the Collector and there ship to their preferred vendors, since each vendor already published their exporter
	plugin to OTel Collector. This makes the SDK exporters very light-weight as they don't need to support any
	vendor. It's also easier for the DevOps team as they can make OTel Collector their responsibility, and have
	application developers only focus on shipping metrics to that collector.

	Just to have some context: Pulsar codebase will use the OTel API to create counters / histograms and records values to
	them. So will the Pulsar plugins and Pulsar Function authors. Pulsar itself will be the one creating the SDK
	and using that to hand over an implementation of the API where ever needed in Pulsar. Collector is up to the choice
	of the user, as OTel provides a way to expose the metrics as `/metrics` endpoint on a configured port, so Prometheus
	compatible scrapers can grab it from it directly. They can also send it via OTLP to OTel collector.

	## Telemetry layers
	PIP-264 clearly outlined there will be two layers of metrics, collected and exported, side by side: OpenTelemetry
	and the existing metric system - currently exporting in Prometheus. This PIP will explain in detail how it will work.
	The basic premise is that you will be able to enable or disable OTel metrics, alongside the existing Prometheus
	metric exporting.

	## Why OTel in Pulsar will be marked experimental and not GA
	As specified in [PIP-264](pip-264.md), OpenTelemetry Java SDK has several fixes the Pulsar community must
	complete before it can be used in production. They are [documented](pip-264.md#what-we-need-to-fix-in-opentelemetry)
	in PIP-264. The most important one is reducing memory allocations to be negligible. OTel SDK is built upon immutability,
	hence allocated memory in O(`#topics`) which is a performance killer for low latency application like Pulsar.

	You can track the proposal and progress the Pulsar and OTel communities are making in
	[this issue](https://github.com/open-telemetry/opentelemetry-java/issues/5105).


	## Metrics endpoint authentication
	Today Pulsar metrics endpoint `/metrics` has an option to be protected by the configured `AuthenticationProvider`.
	The configuration option is named `authenticateMetricsEndpoint` in the broker and
	`authenticateMetricsEndpoint` in the proxy.


	# Motivation

	Implementing PIP-264 consists of a long list of steps, which are detailed in
	[this issue](https://github.com/apache/pulsar/issues/21121). The first step is add all the bare-bones infrastructure
	to use OpenTelemetry in Pulsar, such that next PRs can use it to start translating existing metrics to their
	OTel form. It means the same metrics will co-exist in the codebase and also in runtime, if OTel was enabled.

	# Goals

	## In Scope
	- Ability to add metrics using OpenTelemetry to Pulsar components: Broker, Function Worker and Proxy.
	- User can disable or enable OpenTelemetry metrics, which by default will be disabled
	- OpenTelemetry metrics will be configured via its native OTel Java SDK configuration options
	- All the necessary information to use OTel with Pulsar will be documented in Pulsar documentation site
	- OpenTelemetry metrics layer defined as experimental, and not GA


	## Out of Scope
	- Ability to add metrics using OpenTelemetry as Pulsar Function author.
	- Only authenticated sessions can access OTel Prometheus endpoint, using Pulsar authentication
	- Metrics in Pulsar clients (as defined in [PIP-264](pip-264.md#out-of-scope)))

	# High Level Design

	## Configuration
	OpenTelemetry, as any good telemetry library (e.g. log4j, logback), has its own configuration mechanisms:
	- System properties
	- Environment variables
	- Experimental file-based configuration

	Pulsar doesn't need to introduce any additional configuration. The user can decide, using OTel configuration
	things like:
	* How do I want to export the metrics? Prometheus? Which port prometheus will be exposed at
	* Change histogram buckets using Views
	* and more

	Pulsar will use `AutoConfiguredOpenTelemetrySdk` which uses all the above configuration mechanisms
	(documented [here](https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/autoconfigure)).
	This class builds an `OpenTelemetrySdk` based on configurations. This is the entry point to OpenTelemetry API, as it
	implements `OpenTelemetry` API class.

	### Setting sensible defaults for Pulsar
	There are some configuration options we wish to change their default, but still allow the users to override it
	if they wish. We think those default values will make a much easier user experience.

	* `otel.experimental.metrics.cardinality.limit` - value: 10,000
	This property sets an upper bound on the amount of unique `Attributes` an instrument can have. Take Pulsar for example,
	an instrument like `pulsar.broker.messaging.topic.received.size`, the unique `Attributes` would be in the amount of
	active topics in the broker. Since Pulsar can handle up to 1M topics, it makes more sense to put the default value
	to 10k, which translates to 10k topics.

	`AutoConfiguredOpenTelemetrySdkBuilder` allows to add properties using the method `addPropertiesSupplier`. The
	System properties and environment variables override it. The file-based configuration still doesn't take
	those properties supplied into account, but it will.


	## Opting in
	We would like to have the ability to toggle OpenTelemetry-based metrics, as they are still new.
	We won't need any special Pulsar configuration, as OpenTelemetry SDK comes with a configuration key to do that.
	Since OTel is still experimental, it will have to be opt-in, hence we will add the following property to be the default
	using the mechanism described [above](#setting-sensible-defaults-for-pulsar):

	* `otel.sdk.disabled` - value: true
	This property value disables OpenTelemetry.

	With OTel disabled, the user remains with the existing metrics system. OTel in a disabled state operates in a
	no-op mode. This means, instruments do get built, but the instrument builders return the same instance of a
	no-op instrument, which does nothing on record-values method (e.g. `add(number)`, `record(number)`). The no-op
	`MeterProvider` has no registered `MetricReader` hence no metric collection will be made. The memory impact
	is almost 0 and the same goes for CPU impact.

	The current metric system doesn't have a toggle which causes all existing data structures to stop collecting
	data. Inserting will need changing in so many places since we don't have a single place which through
	all metric instrument are created (one of the motivations for PIP-264).
	The current system do have a toggle: `exposeTopicLevelMetricsInPrometheus`. It enables toggling off
	topic-level metrics, which means the highest cardinality metrics will be namespace level.
	Once that toggle is `false`, the amount of data structures accounting memory would in the range of
	a few thousands which shouldn't post a burden memory wise. If the user refrain from calling
	`/metrics` it will also reduce the CPU and memory cost associated with collecting metrics.

	When the user enables OTel it means there will be a memory increase, but if the user disabled topic-level
	metrics in existing system, as specified above, the majority of the memory increase will be due to topic level
	metrics in OTel, at the expense of not having them in the existing metric system.



	## Cluster attribute name
	A broker is part of a cluster. It is configured in the Pulsar configuration key `clusterName`. When the broker is part
	of a cluster, it means it shares the topics defined in that cluster (persisted in Metadata service: e.g. ZK)
	among the brokers of that cluster.

	Today, each unique time series emitted in Prometheus metrics contains the `cluster` label (almost all of them, as it
	is done manually). We wish the same with OTel - to have that attribute in each exported unique time series.

	OTel has the perfect location to place attributes which are shared across all time series: Resource. An application
	can have multiple Resource, with each having 1 or more attributes. You define it once, in OTel initialization or
	configuration. It can contain attributes like the hostname, AWS region, etc. The default contains the service name
	and some info on the SDK version.

	Attributes can be added dynamically, through `addResourceCustomizer()` in `AutoConfiguredOpenTelemetrySdkBuilder`.
	We will use that to inject the `cluster` attribute, taken from the configuration.

	In Prometheus, we submitted a [proposal](https://github.com/open-telemetry/opentelemetry-specification/pull/3761)
	to opentelemetry specifications, which was merged, to allow copying resource attributes into each exported
	unique time series in Prometheus exporter.
	We plan to contribute its implementation to OTel Java SDK.

	Resources in Prometheus exporter, are exported as `target_info{} 1` and the attributes are added to this
	time series. This will require making joins to get it, making it extremely difficult to use.
	The other alternative was to introduce our own `PulsarAttributesBuilder` class, on top of
	`AttributesBuilder` of OTel. Getting every contributor to know this class, use it, is hard. Getting this
	across Pulsar Functions or Plugins authors, will be immensely hard. Also, when exporting as
	OTLP, it is very inefficient to repeat the attribute across all unique time series, instead of once using
	Resource. Hence, this needed to be solved in the Prometheus exporter as we did in the proposal.

	The attribute will be named `pulsar.cluster`, as both the proxy and the broker are part of this cluster.

	## Naming and using OpenTelemetry

	### Attributes
	* We shall prefix each attribute with `pulsar.`. Example: `pulsar.topic`, `pulsar.cluster`.

	### Instruments
	We should have a clear hierarchy, hence use the following prefix
	* `pulsar.broker`
	* `pulsar.proxy`
	* `pulsar.function_worker`

	### Meter
	It's customary to use reverse domain name for meter names. Hence, we'll use:
	* `org.apache.pulsar.broker`
	* `org.apache.pulsar.proxy`
	* `org.apache.pulsar.function_worker`

	OTel meter name is converted to the attribute name `otel_scope_name` and added to each unique time series
	attributes by Prometheus exporter.

	We won't specify a meter version, as it is used solely to signify the version of the instrumentation, and
	currently we are the first version, hence not use it.


	# Detailed Design

	## Design & Implementation Details

	* `OpenTelemetryService` class
	* Parameters:
	* Cluster name
	* What it will do:
	- Override default max cardinality to 10k
	- Register a resource with cluster name
	- Place defaults setting to instruct Prometheus Exporter to copy resource attributes
	- In the future: place defaults for Memory Mode to be REUSABLE_DATA

	* `PulsarBrokerOpenTelemetry` class
	* Initialization
	* Construct an `OpenTelemetryService` using the cluster name taken from the broker configuration
	* Constructs a Meter for the broker metrics
	* Methods
	* `getMeter()` returns the `Meter` for the broker
	* Notes
	* This is the class that will be passed along to other Pulsar service classes that needs to define
	telemetry such as metrics (in the future: traces).

	* `PulsarProxyOpenTelemetry` class
	* Same as `PulsarBrokerOpenTelemetry` but for Pulsar Proxy
	* `PulsarWorkerOpenTelemetry` class
	* Same as `PulsarBrokerOpenTelemetry` but for Pulsar function worker


	## Public-facing Changes

	### Public API
	* OTel Prometheus Exporter adds `/metrics` endpoint on a user defined port, if user chose to use it

	### Configuration
	* OTel configurations are used

	# Security Considerations
	* OTel currently does not support setting a custom Authenticator for Prometheus exporter.
	An issue has been raised [here](https://github.com/open-telemetry/opentelemetry-java/issues/6013).
	* Once it do we can secure the Prometheus exporter metrics endpoint using `AuthenticationProvider`
	* Any user can access metrics, and they are not protected per tenant. Like today's implementation

	# Links

	* Mailing List discussion thread: https://lists.apache.org/thread/xcn9rm551tyf4vxrpb0th0wj0kktnrr2
	* Mailing List voting thread: https://lists.apache.org/thread/zp6vl9z9dhwbvwbplm60no13t8fvlqs2