[Doc] Add monitoring guidelines for function and connector (#10133)
* [Doc] Add monitoring guidelines for function and connector
* update
* update
Co-authored-by: Anonymitaet <anonymitaet_hotmail.com>
diff --git a/site2/docs/deploy-monitoring.md b/site2/docs/deploy-monitoring.md
index 623c1ca..e1af186 100644
--- a/site2/docs/deploy-monitoring.md
+++ b/site2/docs/deploy-monitoring.md
@@ -69,6 +69,26 @@
Those metrics are added in the Prometheus interface, you can monitor and check the metrics stats in the Grafana.
+### Function and connector stats
+
+You can collect functions worker stats from `functions-worker` and export the metrics in JSON formats, which contain functions worker JVM metrics.
+
+```
+pulsar-admin functions-worker monitoring-metrics
+```
+
+You can collect functions and connectors metrics from `functions-worker` and export the metrics in JSON formats.
+
+```
+pulsar-admin functions-worker function-stats
+```
+
+The aggregated functions and connectors metrics can be exposed in Prometheus formats as below. You can get [`FUNCTIONS_WORKER_ADDRESS`](http://pulsar.apache.org/docs/en/next/functions-worker/) and `WORKER_PORT` from the `functions_worker.yml` file.
+
+```
+http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics:
+```
+
## Configure Prometheus
You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/).
@@ -103,4 +123,4 @@
- [apache-pulsar-grafana-dashboard](https://github.com/streamnative/apache-pulsar-grafana-dashboard): a collection of Grafana dashboard templates for different Pulsar components running on both Kubernetes and on-premise machines.
## Alerting rules
- You can set alerting rules according to your Pulsar environment. To configure alerting rules for Apache Pulsar, you can refer to [StreamNative platform](https://streamnative.io/docs/latest/configure/control-center/alertmanager) examples or [Alert Manager](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) alerting rules.
\ No newline at end of file
+ You can set alerting rules according to your Pulsar environment. To configure alerting rules for Apache Pulsar, you can refer to [StreamNative platform](https://streamnative.io/docs/latest/configure/control-center/alertmanager) examples or [Alert Manager](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) alerting rules.
diff --git a/site2/docs/functions-develop.md b/site2/docs/functions-develop.md
index d153c7f..a14e537 100644
--- a/site2/docs/functions-develop.md
+++ b/site2/docs/functions-develop.md
@@ -765,10 +765,27 @@
<!--END_DOCUSAURUS_CODE_TABS-->
## Metrics
-Pulsar Functions can publish arbitrary metrics to the metrics interface which can be queried.
+Pulsar Functions allows you to deploy and manage processing functions that consume messages from and publish messages to Pulsar topics easily. It is important to ensure that the running functions are healthy at any time. Pulsar Functions can publish arbitrary metrics to the metrics interface which can be queried.
+
+> **Note**
+>
> If a Pulsar Function uses the language-native interface for Java or Python, that function is not able to publish metrics and stats to Pulsar.
+You can monitor Pulsar Functions that have been deployed with the following methods:
+
+- Check the metrics provided by Pulsar.
+
+ Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide.
+
+ For the complete list of the function metrics, see [here](reference-metrics.md#pulsar-functions).
+
+- Set and check your customized metrics.
+
+ In addition to the metrics provided by Pulsar, Pulsar allows you to customize metrics for **Java and Python** functions. Function workers collect user-defined metrics to Prometheus automatically and you can check them in Grafana.
+
+Here are examples of how to customize metrics for Java and Python functions.
+
<!--DOCUSAURUS_CODE_TABS-->
<!--Java-->
You can record metrics using the [`Context`](#context) object on a per-key basis. For example, you can set a metric for the `process-count` key and a different metric for the `elevens-count` key every time the function processes a message.
@@ -793,8 +810,6 @@
}
```
-> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide.
-
<!--Python-->
You can record metrics using the [`Context`](#context) object on a per-key basis. For example, you can set a metric for the `process-count` key and a different metric for the `elevens-count` key every time the function processes a message. The following is an example.
@@ -813,9 +828,6 @@
<!--END_DOCUSAURUS_CODE_TABS-->
-### Access metrics
-To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring.md) in Pulsar.
-
## Security
If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings).
diff --git a/site2/docs/io-develop.md b/site2/docs/io-develop.md
index e318d55..5da0fc1 100644
--- a/site2/docs/io-develop.md
+++ b/site2/docs/io-develop.md
@@ -237,3 +237,42 @@
</executions>
</plugin>
```
+
+## Monitor
+
+Pulsar connectors enable you to move data in and out of Pulsar easily. It is important to ensure that the running connectors are healthy at any time. You can monitor Pulsar connectors that have been deployed with the following methods:
+
+- Check the metrics provided by Pulsar.
+
+ Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide.
+
+- Set and check your customized metrics.
+
+ In addition to the metrics provided by Pulsar, Pulsar allows you to customize metrics for **Java** connectors. Function workers collect user-defined metrics to Prometheus automatically and you can check them in Grafana.
+
+Here is an example of how to customize metrics for a Java connector.
+
+<!--DOCUSAURUS_CODE_TABS-->
+<!--Java-->
+
+```
+public class TestMetricSink implements Sink<String> {
+
+ @Override
+ public void open(Map<String, Object> config, SinkContext sinkContext) throws Exception {
+ sinkContext.recordMetric("foo", 1);
+ }
+
+ @Override
+ public void write(Record<String> record) throws Exception {
+
+ }
+
+ @Override
+ public void close() throws Exception {
+
+ }
+ }
+```
+
+<!--END_DOCUSAURUS_CODE_TABS-->
diff --git a/site2/docs/reference-metrics.md b/site2/docs/reference-metrics.md
index f0af55b..4496d21 100644
--- a/site2/docs/reference-metrics.md
+++ b/site2/docs/reference-metrics.md
@@ -405,17 +405,61 @@
| Name | Type | Description |
|---|---|---|
-| pulsar_function_processed_successfully_total | Counter | Total number of messages processed successfully. |
-| pulsar_function_processed_successfully_total_1min | Counter | Total number of messages processed successfully in the last 1 minute. |
-| pulsar_function_system_exceptions_total | Counter | Total number of system exceptions. |
-| pulsar_function_system_exceptions_total_1min | Counter | Total number of system exceptions in the last 1 minute. |
-| pulsar_function_user_exceptions_total | Counter | Total number of user exceptions. |
-| pulsar_function_user_exceptions_total_1min | Counter | Total number of user exceptions in the last 1 minute. |
-| pulsar_function_process_latency_ms | Summary | Process latency in milliseconds. |
-| pulsar_function_process_latency_ms_1min | Summary | Process latency in milliseconds in the last 1 minute. |
+| pulsar_function_processed_successfully_total | Counter | The total number of messages processed successfully. |
+| pulsar_function_processed_successfully_total_1min | Counter | The total number of messages processed successfully in the last 1 minute. |
+| pulsar_function_system_exceptions_total | Counter | The total number of system exceptions. |
+| pulsar_function_system_exceptions_total_1min | Counter | The total number of system exceptions in the last 1 minute. |
+| pulsar_function_user_exceptions_total | Counter | The total number of user exceptions. |
+| pulsar_function_user_exceptions_total_1min | Counter | The total number of user exceptions in the last 1 minute. |
+| pulsar_function_process_latency_ms | Summary | The process latency in milliseconds. |
+| pulsar_function_process_latency_ms_1min | Summary | The process latency in milliseconds in the last 1 minute. |
| pulsar_function_last_invocation | Gauge | The timestamp of the last invocation of the function. |
-| pulsar_function_received_total | Counter | Total number of messages received from source. |
-| pulsar_function_received_total_1min | Counter | Total number of messages received from source in the last 1 minute. |
+| pulsar_function_received_total | Counter | The total number of messages received from source. |
+| pulsar_function_received_total_1min | Counter | The total number of messages received from source in the last 1 minute. |
+pulsar_function_user_metric_ | Summary|The user-defined metrics.
+
+## Connectors
+
+All the Pulsar connector metrics are labelled with the following labels:
+
+- *cluster*: `cluster=${pulsar_cluster}`. `${pulsar_cluster}` is the cluster name that you have configured in the `broker.conf` file.
+- *namespace*: `namespace=${pulsar_namespace}`. `${pulsar_namespace}` is the namespace name.
+
+Connector metrics contain **source** metrics and **sink** metrics.
+
+- **Source** metrics
+
+ | Name | Type | Description |
+ |---|---|---|
+ pulsar_source_written_total|Counter|The total number of records written to a Pulsar topic.
+ pulsar_source_written_total_1min|Counter|The total number of records written to a Pulsar topic in the last 1 minute.
+ pulsar_source_received_total|Counter|The total number of records received from source.
+ pulsar_source_received_total_1min|Counter|The total number of records received from source in the last 1 minute.
+ pulsar_source_last_invocation|Gauge|The timestamp of the last invocation of the source.
+ pulsar_source_source_exception|Gauge|The exception from a source.
+ pulsar_source_source_exceptions_total|Counter|The total number of source exceptions.
+ pulsar_source_source_exceptions_total_1min |Counter|The total number of source exceptions in the last 1 minute.
+ pulsar_source_system_exception|Gauge|The exception from system code.
+ pulsar_source_system_exceptions_total|Counter|The total number of system exceptions.
+ pulsar_source_system_exceptions_total_1min|Counter|The total number of system exceptions in the last 1 minute.
+ pulsar_source_user_metric_ | Summary|The user-defined metrics.
+
+- **Sink** metrics
+
+ | Name | Type | Description |
+ |---|---|---|
+ pulsar_sink_written_total|Counter| The total number of records processed by a sink.
+ pulsar_sink_written_total_1min|Counter| The total number of records processed by a sink in the last 1 minute.
+ pulsar_sink_received_total_1min|Counter| The total number of messages that a sink has received from Pulsar topics in the last 1 minute.
+ pulsar_sink_received_total|Counter| The total number of records that a sink has received from Pulsar topics.
+ pulsar_sink_last_invocation|Gauge|The timestamp of the last invocation of the sink.
+ pulsar_sink_sink_exception|Gauge|The exception from a sink.
+ pulsar_sink_sink_exceptions_total|Counter|The total number of sink exceptions.
+ pulsar_sink_sink_exceptions_total_1min |Counter|The total number of sink exceptions in the last 1 minute.
+ pulsar_sink_system_exception|Gauge|The exception from system code.
+ pulsar_sink_system_exceptions_total|Counter|The total number of system exceptions.
+ pulsar_sink_system_exceptions_total_1min|Counter|The total number of system exceptions in the last 1 minute.
+ pulsar_sink_user_metric_ | Summary|The user-defined metrics.
## Proxy
