Nginx monitoring

Nginx performance from nginx-lua-prometheus

The nginx-lua-prometheus is a lua library that can be used with Nginx to collect metrics and expose them on a separate web page. To use this library, you will need Nginx with lua-nginx-module or directly OpenResty.

SkyWalking leverages OpenTelemetry Collector to transfer the metrics to OpenTelemetry receiver and into the Meter System.

Data flow

1. nginx-lua-prometheus collects metrics from Nginx and expose them to an endpoint.
2. OpenTelemetry Collector fetches metrics from the endpoint expose above via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter.
3. The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results.

Set up

1. Collect Nginx metrics and expose the following four metrics by nginx-lua-prometheus. For details on metrics definition, refer to here.

• histogram: nginx_http_latency
• gauge: nginx_http_connections
• counter: nginx_http_size_bytes
• counter: nginx_http_requests_total

2. Set up OpenTelemetry Collector . For details on Prometheus Receiver in OpenTelemetry Collector, refer to here.
3. Config SkyWalking OpenTelemetry receiver.

Nginx Monitoring

SkyWalking observes the status, payload, and latency of the Nginx server, which is cataloged as a LAYER: Nginx Service in the OAP and instances would be recognized as LAYER: Nginx instance.

About LAYER: Nginx endpoint, it depends on how precision you want to monitor the nginx. We do not recommend expose every request path metrics, because it will cause explosion of metrics endpoint data.

You can collect host metrics:

http {
  log_by_lua_block {
      metric_bytes:inc(tonumber(ngx.var.request_length), {"request", ngx.var.host})
      metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", ngx.var.host})
      metric_requests:inc(1, {ngx.var.status, ngx.var.host})
      metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.host})
  }
}

or grouped urls and upstream metrics:

upstream backend {
  server ip:port;
}

server {

  location /test {
    default_type application/json;
    return 200  '{"code": 200, "message": "success"}';
  
    log_by_lua_block {
      metric_bytes:inc(tonumber(ngx.var.request_length), {"request", "/test/**"})
      metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", "/test/**"})
      metric_requests:inc(1, {ngx.var.status, "/test/**"})
      metric_latency:observe(tonumber(ngx.var.request_time), {"/test/**"})
    }
  }
  
  location /test_upstream {
  
    proxy_pass http://backend;
  
    log_by_lua_block {
      metric_bytes:inc(tonumber(ngx.var.request_length), {"request", "upstream/backend"})
      metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", "upstream/backend"})
      metric_requests:inc(1, {ngx.var.status, "upstream/backend"})
      metric_latency:observe(tonumber(ngx.var.request_time), {"upstream/backend"})
    }
  }
}

Nginx Service Supported Metrics

Monitoring Panel	Unit	Metric Name	Catalog	Description	Data Source
HTTP Request Trend		meter_nginx_service_http_requests	Service	The increment rate of HTTP requests	nginx-lua-prometheus
HTTP Latency	ms	meter_nginx_service_http_latency	Service	The increment rate of the latency of HTTP requests	nginx-lua-prometheus
HTTP Bandwidth	KB	meter_nginx_service_bandwidth	Service	The increment rate of the bandwidth of HTTP requests	nginx-lua-prometheus
HTTP Connections		meter_nginx_service_http_connections	Service	The avg number of the connections	nginx-lua-prometheus
HTTP Status Trend		meter_nginx_service_http_status	Service	The increment rate of the status of HTTP requests	nginx-lua-prometheus
HTTP Status 4xx Percent	%	meter_nginx_service_http_4xx_requests_increment / meter_nginx_service_http_requests_increment	Service	The percentage of 4xx status of HTTP requests	nginx-lua-prometheus
HTTP Status 5xx Percent	%	meter_nginx_service_http_5xx_requests_increment / meter_nginx_service_http_requests_increment	Service	The percentage of 4xx status of HTTP requests	nginx-lua-prometheus

Nginx Instance Supported Metrics

Monitoring Panel	Unit	Metric Name	Catalog	Description	Data Source
HTTP Request Trend		meter_nginx_instance_http_requests	Instance	The increment rate of HTTP requests	nginx-lua-prometheus
HTTP Latency	ms	meter_nginx_instance_http_latency	Instance	The increment rate of the latency of HTTP requests	nginx-lua-prometheus
HTTP Bandwidth	KB	meter_nginx_instance_bandwidth	Instance	The increment rate of the bandwidth of HTTP requests	nginx-lua-prometheus
HTTP Connections		meter_nginx_instance_http_connections	Instance	The avg number of the connections	nginx-lua-prometheus
HTTP Status Trend		meter_nginx_instance_http_status	Instance	The increment rate of the status of HTTP requests	nginx-lua-prometheus
HTTP Status 4xx Percent	%	meter_nginx_instance_http_4xx_requests_increment / meter_nginx_instance_http_requests_increment	Instance	The percentage of 4xx status of HTTP requests	nginx-lua-prometheus
HTTP Status 5xx Percent	%	meter_nginx_instance_http_5xx_requests_increment / meter_nginx_instance_http_requests_increment	Instance	The percentage of 4xx status of HTTP requests	nginx-lua-prometheus

Nginx Endpoint Supported Metrics

Monitoring Panel	Unit	Metric Name	Catalog	Description	Data Source
HTTP Request Trend		meter_nginx_endpoint_http_requests	Endpoint	The increment rate of HTTP requests	nginx-lua-prometheus
HTTP Latency	ms	meter_nginx_endpoint_http_latency	Endpoint	The increment rate of the latency of HTTP requests	nginx-lua-prometheus
HTTP Bandwidth	KB	meter_nginx_endpoint_bandwidth	Endpoint	The increment rate of the bandwidth of HTTP requests	nginx-lua-prometheus
HTTP Status Trend		meter_nginx_endpoint_http_status	Endpoint	The increment rate of the status of HTTP requests	nginx-lua-prometheus
HTTP Status 4xx Percent	%	meter_nginx_endpoint_http_4xx_requests_increment / meter_nginx_endpoint_http_requests_increment	Endpoint	The percentage of 4xx status of HTTP requests	nginx-lua-prometheus
HTTP Status 5xx Percent	%	meter_nginx_endpoint_http_5xx_requests_increment / meter_nginx_endpoint_http_requests_increment	Endpoint	The percentage of 4xx status of HTTP requests	nginx-lua-prometheus

Customizations

You can customize your own metrics/expression/dashboard panel.

The metrics definition and expression rules are found in /config/otel-rules/nginx-service.yaml, /config/otel-rules/nginx-instance.yaml, /config/otel-rules/nginx-endpoint.yaml.

The Nginx dashboard panel configurations are found in /config/ui-initialized-templates/nginx.

Collect nginx access and error log

SkyWalking leverages fluentbit or other log agents for collecting access log and error log of Nginx.

Data flow

1. fluentbit agent collects access log and error log from Nginx.
2. fluentbit agent sends data to SkyWalking OAP Server using native meter APIs via HTTP.
3. The SkyWalking OAP Server parses the expression with LAL to parse/extract and store the results.

Set up

1. Install fluentbit.
2. Config fluent bit with fluent-bit.conf, refer to here.

Error Log Monitoring

Error Log monitoring provides monitoring of the error.log of the Nginx server.

Supported Metrics

Monitoring Panel	Metric Name	Catalog	Description	Data Source
Service Error Log Count	meter_nginx_service_error_log_count	Service	The count of log level of nginx error.log	fluent bit
Instance Error Log Count	meter_nginx_instance_error_log_count	Instance	The count of log level of nginx error.log	fluent bit

Customizations

You can customize your own metrics/expression/dashboard panel.

The log collect and analyse rules are found in /config/lal/nginx.yaml, /config/log-mal-rules/nginx.yaml.

The Nginx dashboard panel configurations are found in /config/ui-initialized-templates/nginx.