APISIX monitoring

APISIX performance from apisix prometheus plugin

SkyWalking leverages OpenTelemetry Collector to transfer the metrics to OpenTelemetry receiver and into the Meter System.

Data flow

1. APISIX Prometheus plugin collects metrics data from APSIX.
2. OpenTelemetry Collector fetches metrics from APISIX Prometheus plugin via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via the OpenCensus gRPC Exporter or OpenTelemetry gRPC exporter.
3. The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results.

Set up

1. Enable APISIX APISIX Prometheus plugin .
2. Set up OpenTelemetry Collector . For details on Prometheus Receiver in OpenTelemetry Collector, refer to here.
3. Config SkyWalking OpenTelemetry receiver.

APISIX Monitoring

APISIX prometheus plugin provide multiple dimensions metrics for APISIX server , upstream , route , etc. Accordingly, SkyWalking observes the status, payload, and latency of the APISIX server, which is cataloged as a LAYER: APISIX Service in the OAP. Meanwhile, the instances would be recognized as LAYER: APISIX instances. The route rules and nodes would be recognized as endpoints with route/ and upstream/ prefixes.

Specify SkyWalking Service name

SkyWalking expects OTEL Collector attribute skywalking_service to be the Service name.

Make sure skywalking_service attribute exists through static_configs of OTEL Prometheus scape config.

receivers:
  prometheus:
    config: 
     scrape_configs:
       - job_name: 'apisix-monitoring' 
         static_configs:
           - targets: ['apisix:9091']
             labels:
               skywalking_service: exmple_service_name  # Specify SkyWalking Service name              

You also could leverage OTEL Collector processor to add skywalking_service attribute , as following :

processors:
  resource/skywalking-service:
    attributes:
    - key: skywalking_service   
      value: exmple_service_name # Specify Skywalking Service name 
      action: insert                     

Notice , if you don't specify skywalking_service attribute, SkyWalking OAP would use APISIX as the default service name

Supported Metrics

Monitoring Panel	Unit	Metric Name	Catalog	Description	Data Source
HTTP status		meter_apisix_sv_http_status	Service	The increment rate of the status of HTTP requests	APISIX Prometheus plugin
HTTP latency		meter_apisix_sv_http_latency	Service	The increment rate of the latency of HTTP requests	APISIX Prometheus plugin
HTTP bandwidth	KB	meter_apisix_sv_bandwidth	Service	The increment rate of the bandwidth of HTTP requests	APISIX Prometheus plugin
HTTP status of non-matched requests		meter_apisix_sv_http_status	Service	The increment rate of the status of HTTP requests, which don't match any route	APISIX Prometheus plugin
HTTP latency non-matched requests		meter_apisix_sv_http_latency	Service	The increment rate of the latency of HTTP requests, which don't match any route	APISIX Prometheus plugin
HTTP bandwidth non-matched requests	KB	meter_apisix_sv_bandwidth	Service	The increment rate of the bandwidth of HTTP requests ,which don't match any route	APISIX Prometheus plugin
HTTP connection		meter_apisix_sv_http_connections	Service	The avg number of the connections	APISIX Prometheus plugin
HTTP Request Trend		meter_apisix_http_requests	Service	The increment rate of HTTP requests	APISIX Prometheus plugin
HTTP status		meter_apisix_instance_http_status	Instance	The increment rate of the status of HTTP requests	APISIX Prometheus plugin
HTTP latency		meter_apisix_instance_http_latency	Instance	The increment rate of the latency of HTTP requests	APISIX Prometheus plugin
HTTP bandwidth	KB	meter_apisix_instance_bandwidth	Instance	The increment rate of the bandwidth of HTTP requests	APISIX Prometheus plugin
HTTP status of non-matched requests		meter_apisix_instance_http_status	Instance	The increment rate of the status of HTTP requests, which don't match any route	APISIX Prometheus plugin
HTTP latency non-matched requests		meter_apisix_instance_http_latency	Instance	The increment rate of the latency of HTTP requests, which don't match any route	APISIX Prometheus plugin
HTTP bandwidth non-matched requests	KB	meter_apisix_instance_bandwidth	Instance	The increment rate of the bandwidth of HTTP requests ,which don't match any route	APISIX Prometheus plugin
HTTP connection		meter_apisix_instance_http_connections	Instance	The avg number of the connections	APISIX Prometheus plugin
HTTP Request Trend		meter_apisix_instance_http_requests	Instance	The increment rate of HTTP requests	APISIX Prometheus plugin
Shared dict capacity	MB	meter_apisix_instance_shared_dict_capacity_bytes	Instance	The avg capacity of shared dict capacity	APISIX Prometheus plugin
Shared free space	MB	meter_apisix_instance_shared_dict_free_space_bytes	Instance	The avg free space of shared dict capacity	APISIX Prometheus plugin
etcd index		meter_apisix_instance_sv_etcd_indexes	Instance	etcd modify index for APISIX keys	APISIX Prometheus plugin
etcd latest reachability		meter_apisix_instance_sv_etcd_reachable	Instance	etcd latest reachable , Refer to APISIX Prometheus plugin	APISIX Prometheus plugin
HTTP status		meter_apisix_endpoint_node_http_status	Endpoint	The increment rate of the status of HTTP requests	APISIX Prometheus plugin
HTTP latency		meter_apisix_endpoint_node_http_latency	Endpoint	The increment rate of the latency of HTTP requests	APISIX Prometheus plugin
HTTP bandwidth	KB	meter_apisix_endpoint_node_bandwidth	Endpoint	The increment rate of the bandwidth of HTTP requests	APISIX Prometheus plugin

Customizations

You can customize your own metrics/expression/dashboard panel. The metrics definition and expression rules are found in /config/otel-rules/apisix.yaml. The APISIX dashboard panel configurations are found in /config/ui-initialized-templates/apisix.