The OAP backend cluster itself is a distributed streaming process system. To assist the Ops team, we provide the telemetry for the OAP backend itself, also known as self-observability (so11y)
By default, the telemetry is disabled by setting selector
to none
, like this:
telemetry: selector: ${SW_TELEMETRY:none} none: prometheus: host: ${SW_TELEMETRY_PROMETHEUS_HOST:0.0.0.0} port: ${SW_TELEMETRY_PROMETHEUS_PORT:1234} sslEnabled: ${SW_TELEMETRY_PROMETHEUS_SSL_ENABLED:false} sslKeyPath: ${SW_TELEMETRY_PROMETHEUS_SSL_KEY_PATH:""} sslCertChainPath: ${SW_TELEMETRY_PROMETHEUS_SSL_CERT_CHAIN_PATH:""}
You may also set Prometheus
to enable them. For more information, refer to the details below.
SkyWalking supports exposing telemetry data representing OAP running status through Prometheus endpoint. Users could set up OpenTelemetry collector to scrap and forward telemetry data to OAP server for further analysis, eventually showing up UI or GraphQL API.
Add the following configuration to enable self-observability-related modules.
telemetry: selector: ${SW_TELEMETRY:prometheus} prometheus: host: 127.0.0.1 port: 1543
Refer to the E2E test case as an example.
For Kubernetes deployments, read the following section, otherwise you should be able to adjust the configurations below to fit your scenarios.
If you deploy an OAP server cluster on Kubernetes, the oap-server instance (pod) would not have a static IP or hostname. We can leverage OpenTelemetry Collector to discover the oap-server instance, and scrape & transfer the metrics to OAP OpenTelemetry receiver.
On how to install SkyWalking on k8s, you can refer to Apache SkyWalking Kubernetes.
Set this up following these steps:
Set the metrics port.
prometheus-port: 1234
Set environment variables.
SW_TELEMETRY=prometheus SW_OTEL_RECEIVER=default SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES=oap
Here is an example to install by Apache SkyWalking Kubernetes:
helm -n istio-system install skywalking skywalking \ --set elasticsearch.replicas=1 \ --set elasticsearch.minimumMasterNodes=1 \ --set elasticsearch.imageTag=7.5.1 \ --set oap.replicas=2 \ --set ui.image.repository=$HUB/skywalking-ui \ --set ui.image.tag=$TAG \ --set oap.image.tag=$TAG \ --set oap.image.repository=$HUB/skywalking-oap \ --set oap.storageType=elasticsearch \ --set oap.ports.prometheus-port=1234 \ # <<< Expose self observability metrics port --set oap.env.SW_TELEMETRY=prometheus \ --set oap.env.SW_OTEL_RECEIVER=default \ # <<< Enable Otel receiver --set oap.env.SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES=oap # <<< Add oap analyzer for Otel metrics
- job_name: 'skywalking-so11y' # make sure to use this in the so11y.yaml to filter only so11y metrics metrics_path: '/metrics' kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_container_name, __meta_kubernetes_pod_container_port_name] action: keep regex: oap;prometheus-port - source_labels: [] target_label: service replacement: oap-server - source_labels: [__meta_kubernetes_pod_name] target_label: host_name regex: (.+) replacement: $$1
For the full example for OpenTelemetry Collector configuration and recommended version, you can refer to showcase.
Users also could leverage the Prometheus endpoint for their own Prometheus and Grafana.
NOTE: Since Apr 21, 2021, the Grafana project has been relicensed to AGPL-v3, and is no longer licensed for Apache 2.0. Check the LICENSE details. The following Prometheus + Grafana solution is optional rather than recommended.
Prometheus is supported as a telemetry implementor, which collects metrics from SkyWalking's backend.
Set prometheus
to provider. The endpoint opens at http://0.0.0.0:1234/
and http://0.0.0.0:1234/metrics
.
telemetry: selector: ${SW_TELEMETRY:prometheus} prometheus:
Set host and port if needed.
telemetry: selector: ${SW_TELEMETRY:prometheus} prometheus: host: 127.0.0.1 port: 1543
Set relevant SSL settings to expose a secure endpoint. Note that the private key file and cert chain file could be uploaded once changes are applied to them.
telemetry: selector: ${SW_TELEMETRY:prometheus} prometheus: host: 127.0.0.1 port: 1543 sslEnabled: true sslKeyPath: /etc/ssl/key.pem sslCertChainPath: /etc/ssl/cert-chain.pem
Provide the Grafana dashboard settings. Check SkyWalking OAP Cluster Monitor Dashboard config and SkyWalking OAP Instance Monitor Dashboard config.