PromQL Service

PromQL(Prometheus Query Language) Service exposes Prometheus Querying HTTP APIs including the bundled PromQL expression system. Third-party systems or visualization platforms that already support PromQL (such as Grafana), could obtain metrics through PromQL Service.

As SkyWalking and Prometheus have fundamental differences in metrics classification, format, storage, etc. The PromQL Service supported will be a subset of the complete PromQL.

Details Of Supported Protocol

The following doc describes the details of the supported protocol and compared it to the PromQL official documentation. If not mentioned, it will not be supported by default.

Time series Selectors

Instant Vector Selectors

For example: select metric service_cpm which the service is $service and the layer is $layer.

service_cpm{service='$service', layer='$layer'}

Note: The label matching operators only support = instead of regular expressions.

Range Vector Selectors

For example: select metric service_cpm which the service is $service and the layer is $layer within the last 5 minutes.

service_cpm{service='$service', layer='$layer'}[5m]

Time Durations

UnitDefinitionSupport
msmillisecondsyes
ssecondsyes
mminutesyes
hhoursyes
ddaysyes
wweeksyes
yyearsno

Binary operators

Arithmetic binary operators

OperatorDefinitionSupport
+additionyes
-subtractionyes
*multiplicationyes
/divisionyes
%moduloyes
^power/exponentiationno
Between two scalars

For example:

1 + 2
Between an instant vector and a scalar

For example:

service_cpm{service='$service', layer='$layer'} / 100
Between two instant vectors

For example:

service_cpm{service='$service', layer='$layer'} + service_cpm{service='$service', layer='$layer'}

Note: The operations between vectors require the same metric and labels, and don't support Vector matching.

Comparison binary operators

OperatorDefinitionSupport
==equalyes
!=not-equalyes
>greater-thanyes
<less-thanyes
>=greater-or-equalyes
<=less-or-equal)yes
Between two scalars

For example:

1 > bool 2
Between an instant vector and a scalar

For example:

service_cpm{service='$service', layer='$layer'} > 1
Between two instant vectors

For example:

service_cpm{service='service_A', layer='$layer'} > service_cpm{service='service_B', layer='$layer'}

HTTP API

Expression queries

Instant queries
GET|POST /api/v1/query
ParameterDefinitionSupportOptional
queryprometheus expressionyesno
timeThe latest metrics value from current time to this time is returned. If time is empty, the default look-back time is 2 minutes.yesyes
timeoutevaluation timeoutnoignore

For example:

/api/v1/query?query=service_cpm{service='agent::songs', layer='GENERAL'}

Result:

{
    "status": "success",
    "data": {
        "resultType": "vector",
        "result": [
            {
                "metric": {
                    "__name__": "service_cpm",
                    "layer": "GENERAL",
                    "scope": "Service",
                    "service": "agent::songs"
                },
                "value": [
                    1677548400,
                    "6"
                ]
            }
        ]
    }
}
Range queries
GET|POST /api/v1/query_range
ParameterDefinitionSupportOptional
queryprometheus expressionyesno
startstart timestamp, secondsyesno
endend timestamp, secondsyesno
stepSkyWalking will automatically fit Step(DAY, HOUR, MINUTE) through start and end.noignore
timeoutevaluation timeoutnoignore

For example:

/api/v1/query_range?query=service_cpm{service='agent::songs', layer='GENERAL'}&start=1677479336&end=1677479636

Result:

{
    "status": "success",
    "data": {
        "resultType": "matrix",
        "result": [
            {
                "metric": {
                    "__name__": "service_cpm",
                    "layer": "GENERAL",
                    "scope": "Service",
                    "service": "agent::songs"
                },
                "values": [
                    [
                        1677479280,
                        "18"
                    ],
                    [
                        1677479340,
                        "18"
                    ],
                    [
                        1677479400,
                        "18"
                    ],
                    [
                        1677479460,
                        "18"
                    ],
                    [
                        1677479520,
                        "18"
                    ],
                    [
                        1677479580,
                        "18"
                    ]
                ]
            }
        ]
    }
}

Querying metadata

Finding series by label matchers
GET|POST /api/v1/series
ParameterDefinitionSupportOptional
match[]series selectoryesno
startstart timestamp, secondsyesno
endend timestamp, secondsyesno

For example:

/api/v1/series?match[]=service_traffic{layer='GENERAL'}&start=1677479336&end=1677479636

Result:

{
    "status": "success",
    "data": [
        {
            "__name__": "service_traffic",
            "service": "agent::songs",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::recommendation",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::app",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::gateway",
            "scope": "Service",
            "layer": "GENERAL"
        },
        {
            "__name__": "service_traffic",
            "service": "agent::frontend",
            "scope": "Service",
            "layer": "GENERAL"
        }
    ]
}

Note: SkyWalking's metadata exists in the following metrics(traffics):

  • service_traffic
  • instance_traffic
  • endpoint_traffic

Getting label names

GET|POST /api/v1/labels
ParameterDefinitionSupportOptional
match[]series selectoryesyes
startstart timestampnoyes
endend timestamp, if end time is not present, use current time as default end timeyesyes

For example:

/api/v1/labels?match[]=instance_jvm_cpu'

Result:

{
    "status": "success",
    "data": [
        "layer",
        "service",
        "top_n",
        "order",
        "service_instance",
        "parent_service"
    ]
}

Querying label values

GET /api/v1/label/<label_name>/values
ParameterDefinitionSupportOptional
match[]series selectoryesyes
startstart timestampnoyes
endend timestamp, if end time is not present, use current time as default end timeyesyes

For example:

/api/v1/label/__name__/values

Result:

{
  "status": "success",
  "data": [
    "meter_mysql_instance_qps",
    "service_cpm",
    "envoy_cluster_up_rq_active",
    "instance_jvm_class_loaded_class_count",
    "k8s_cluster_memory_requests",
    "meter_vm_memory_used",
    "meter_apisix_sv_bandwidth_unmatched",
    "meter_vm_memory_total",
    "instance_jvm_thread_live_count",
    "instance_jvm_thread_timed_waiting_state_thread_count",
    "browser_app_page_first_pack_percentile",
    "instance_clr_max_worker_threads",
    ...
  ]
}

Querying metric metadata

GET /api/v1/metadata
ParameterDefinitionSupportOptional
limitmaximum number of metrics to returnyesyes
metricmetric name, support regular expressionyesyes

For example:

/api/v1/metadata?limit=10

Result:

{
  "status": "success",
  "data": {
    "meter_mysql_instance_qps": [
      {
        "type": "gauge",
        "help": "",
        "unit": ""
      }
    ],
    "meter_apisix_sv_bandwidth_unmatched": [
      {
        "type": "gauge",
        "help": "",
        "unit": ""
      }
    ],
    "service_cpm": [
      {
        "type": "gauge",
        "help": "",
        "unit": ""
      }
    ],
    ...
  }
}

Metrics Type For Query

Supported Metrics Scope(Catalog)

Not all scopes are supported for now, please check the following table:

ScopeSupport
Serviceyes
ServiceInstanceyes
Endpointyes
ServiceRelationno
ServiceInstanceRelationno
Processno
ProcessRelationno

General labels

Each metric contains general labels: layer. Different metrics will have different labels depending on their Scope and metric value type.

Query LabelsScopeExpression Example
layer, serviceServiceservice_cpm{service=‘$service’, layer=‘$layer’}
layer, service, service_instanceServiceInstanceservice_instance_cpm{service=‘$service’, service_instance=‘$service_instance’, layer=‘$layer’}
layer, service, endpointEndpointendpoint_cpm{service=‘$service’, endpoint=‘$endpoint’, layer=‘$layer’}

Common Value Metrics

  • Query Labels:
{General labels}
  • Expression Example:
service_cpm{service='agent::songs', layer='GENERAL'}
  • Result (Instant Query):
{
    "status": "success",
    "data": {
        "resultType": "vector",
        "result": [
            {
                "metric": {
                    "__name__": "service_cpm",
                    "layer": "GENERAL",
                    "scope": "Service",
                    "service": "agent::songs"
                },
                "value": [
                    1677490740,
                    "3"
                ]
            }
        ]
    }
}

Labeled Value Metrics

  • Query Labels:
--{General labels}
--metric labels: Used to filter the value labels to be returned
  • Expression Example:
service_percentile{service='agent::songs', layer='GENERAL', p='50,75,90'}
  • Result (Instant Query):
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "service_percentile",
          "p": "50",
          "layer": "GENERAL",
          "scope": "Service",
          "service": "agent::songs"
        },
        "value": [
          1677493380,
          "0"
        ]
      },
      {
        "metric": {
          "__name__": "service_percentile",
          "p": "75",
          "layer": "GENERAL",
          "scope": "Service",
          "service": "agent::songs"
        },
        "value": [
          1677493380,
          "0"
        ]
      },
      {
        "metric": {
          "__name__": "service_percentile",
          "p": "90",
          "layer": "GENERAL",
          "scope": "Service",
          "service": "agent::songs"
        },
        "value": [
          1677493380,
          "0"
        ]
      }
    ]
  }
}

Sort Metrics

  • Query Labels:
--parent_service: <optional> Name of the parent service.
--top_n: The max number of the selected metric value
--order: ASC/DES
  • Expression Example:
service_instance_cpm{parent_service='agent::songs', layer='GENERAL',  top_n='10', order='DES'}
  • Result (Instant Query):
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "service_instance_cpm",
          "layer": "GENERAL",
          "scope": "ServiceInstance",
          "service_instance": "651db53c0e3843d8b9c4c53a90b4992a@10.4.0.28"
        },
        "value": [
          1677494280,
          "14"
        ]
      },
      {
        "metric": {
          "__name__": "service_instance_cpm",
          "layer": "GENERAL",
          "scope": "ServiceInstance",
          "service_instance": "4c04cf44d6bd408880556aa3c2cfb620@10.4.0.232"
        },
        "value": [
          1677494280,
          "6"
        ]
      },
      {
        "metric": {
          "__name__": "service_instance_cpm",
          "layer": "GENERAL",
          "scope": "ServiceInstance",
          "service_instance": "f5ac8ead31af4e6795cae761729a2742@10.4.0.236"
        },
        "value": [
          1677494280,
          "5"
        ]
      }
    ]
  }
}

Sampled Records

  • Query Labels:
--parent_service: Name of the parent service
--top_n: The max number of the selected records value
--order: ASC/DES
  • Expression Example:
top_n_database_statement{parent_service='localhost:-1', layer='VIRTUAL_DATABASE',  top_n='10', order='DES'}
  • Result (Instant Query):
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "top_n_database_statement",
          "layer": "VIRTUAL_DATABASE",
          "scope": "Service",
          "record": "select song0_.id as id1_0_, song0_.artist as artist2_0_, song0_.genre as genre3_0_, song0_.liked as liked4_0_, song0_.name as name5_0_ from song song0_ where song0_.liked>?"
        },
        "value": [
          1677501360,
          "1"
        ]
      },
      {
        "metric": {
          "__name__": "top_n_database_statement",
          "layer": "VIRTUAL_DATABASE",
          "scope": "Service",
          "record": "select song0_.id as id1_0_, song0_.artist as artist2_0_, song0_.genre as genre3_0_, song0_.liked as liked4_0_, song0_.name as name5_0_ from song song0_ where song0_.liked>?"
        },
        "value": [
          1677501360,
          "1"
        ]
      },
      {
        "metric": {
          "__name__": "top_n_database_statement",
          "layer": "VIRTUAL_DATABASE",
          "scope": "Service",
          "record": "select song0_.id as id1_0_, song0_.artist as artist2_0_, song0_.genre as genre3_0_, song0_.liked as liked4_0_, song0_.name as name5_0_ from song song0_ where song0_.liked>?"
        },
        "value": [
          1677501360,
          "1"
        ]
      }
    ]
  }
}