Apache SkyWalking CLI

The CLI (Command Line Interface) for Apache SkyWalking.

SkyWalking CLI is a command interaction tool for the SkyWalking user or OPS team, as an alternative besides using browser GUI. It is based on SkyWalking GraphQL query protocol, same as GUI.

Download

Go to the download page to download all available binaries, including MacOS, Linux, Windows. If you want to try the latest features, however, you can compile the latest codes yourself, as the guide below.

Install

As SkyWalking CLI is using Makefile, compiling the project is as easy as executing a command in the root directory of the project.

git clone https://github.com/apache/skywalking-cli
cd skywalking-cli
git submodule init
git submodule update
make

Then copy the ./bin/swctl-latest-(darwin|linux|windows)-amd64 to your PATH directory according to your OS, usually /usr/bin/ or /usr/local/bin, or you can copy it to any directory you like, and add that directory to PATH, we recommend you to rename the swctl-latest-(darwin|linux|windows)-amd64 to swctl.

Commands

Commands in SkyWalking CLI are organized into two levels, in the form of swctl --option <level1> --option <level2> --option, there're options in each level, which should follow right after the corresponding command, take the following command as example:

$ swctl --debug service list --start="2019-11-11" --end="2019-11-12"

where --debug is is an option of swctl, and since the swctl is a top-level command, --debug is also called global option, and --start is an option of the third level command list, there is no option for the second level command service.

Generally, the second level commands are entity related, there're entities like service, service instance, metrics in SkyWalking, and we have corresponding sub-command like service; the third level commands are operations on the entities, such as list command will list all the services, service instances, etc.

Common options

There're some common options that are shared by multiple commands, and they follow the same rules in different commands,

--start and --end specify a time range during which the query is preformed, they are both optional and their default values follow the rules below:

  • when start and end are both absent, start = now - 30 minutes and end = now, namely past 30 minutes;
  • when start and end are both present, they are aligned to the same precision by truncating the more precise one, e.g. if start = 2019-01-01 1234, end = 2019-01-01 18, then start is truncated (because it's more precise) to 2019-01-01 12, and end = 2019-01-01 18;
  • when start is absent and end is present, will determine the precision of end and then use the precision to calculate start (minus 30 units), e.g. end = 2019-11-09 1234, the precision is MINUTE, so start = end - 30 minutes = 2019-11-09 1204, and if end = 2019-11-09 12, the precision is HOUR, so start = end - 30HOUR = 2019-11-08 06;
  • when start is present and end is absent, will determine the precision of start and then use the precision to calculate end (plus 30 units), e.g. start = 2019-11-09 1204, the precision is MINUTE, so end = start + 30 minutes = 2019-11-09 1234, and if start = 2019-11-08 06, the precision is HOUR, so end = start + 30HOUR = 2019-11-09 12;

--timezone specifies the timezone where --start --end are based, in the form of +0800:

  • if --timezone is given in the command line option, then it's used directly;
  • else if the backend support the timezone API (since 6.5.0), CLI will try to get the timezone from backend, and use it;
  • otherwise, the CLI will use the current timezone in the current machine;

All available commands

This section covers all the available commands in SkyWalking CLI and their usages.

swctl

swctl is the top-level command, which has some options that will take effects globally.

optiondescriptiondefault
--configfrom where the default options values will be loaded~/.skywalking.yml, example can be found here
--debugenable debug mode, will print more detailed information at runtimefalse
--base-urlbase url of GraphQL backendhttp://127.0.0.1:12800/graphql
--grpcAddrThe address of gRPC endpoint127.0.0.1:11800
--usernameusername of Basic authorization``
--passwordpassword of Basic authorization``
--authorizationauthorization header, can be something like Basic base64<username:password> or Bearer jwt-token, if authorization is set, username and password are ignored``
--displaydisplay style when printing the query result, supported styles are: json, yaml, table, graphjson

Note that not all display styles (except for json and yaml) are supported in all commands due to data formats incompatibilities and the limits of Ascii Graph, like coloring in terminal, so please use json or yaml instead.

service

service list lists all the services in the time range of [start, end].

optiondescriptiondefault
--startSee Common optionsSee Common options
--endSee Common optionsSee Common options

instance

instance list lists all the instances in the time range of [start, end] and given --service-id or --service-name.

optiondescriptiondefault
--service-idQuery by service id (priority over --service-name)
--service-nameQuery by service name if --service-id is absent
--startSee Common optionsSee Common options
--endSee Common optionsSee Common options

instance search filter the instance in the time range of [start, end] and given --regex --service-id or --service-name.

optiondescriptiondefault
--regexQuery regex of instance name
--service-idQuery by service id (priority over --service-name)
--service-nameQuery by service name if service-id is absent
--startSee Common optionsSee Common options
--endSee Common optionsSee Common options

endpoint

endpoint list lists all the endpoints of the given service id in the time range of [start, end].

optiondescriptiondefault
--service-id whose endpoints are to be searched
--limitreturns at most endpoints (default: 100)100
--keyword of the endpoint name to search for, empty to search all""

metrics

metrics linear

optiondescriptiondefault
--nameMetrics name, defined in OAL.
--serviceThe name of the service.""
--instanceThe name of the service instance.""
--endpointThe name of the endpoint.""
--isNormalSet the service to normal or unnormal.true
--destServiceThe name of the destination service.""
--destInstanceThe name of the destination instance.""
--destEndpointThe name of the destination endpoint.""
--isDestNormalSet the destination service to normal or unnormal.true
--startSee Common optionsSee Common options
--endSee Common optionsSee Common options

metrics multiple-linear

optiondescriptiondefault
--nameMetrics name that ends with _percentile, defined in OAL, such as all_percentile, etc.
--serviceThe name of the service, when scope is All, no name is required.""
--labelsThe labels you need to query0,1,2,3,4
--instanceThe name of the service instance.""
--endpointThe name of the endpoint.""
--isNormalSet the service to normal or unnormal.true
--destServiceThe name of the destination service.""
--destInstanceThe name of the destination instance.""
--destEndpointThe name of the destination endpoint.""
--isDestNormalSet the destination service to normal or unnormal.true
--startSee Common optionsSee Common options
--endSee Common optionsSee Common options

metrics single

optiondescriptiondefault
--nameMetrics name, defined in OAL, such as service_sla, etc.
--serviceThe name of the service.""
--instanceThe name of the service instance.""
--endpointThe name of the endpoint.""
--isNormalSet the service to normal or unnormal.true
--destServiceThe name of the destination service.""
--destInstanceThe name of the destination instance.""
--destEndpointThe name of the destination endpoint.""
--isDestNormalSet the destination service to normal or unnormal.true
--startSee Common optionsSee Common options
--endSee Common optionsSee Common options

metrics top <n>

optiondescriptiondefault
argumentsThe first argument is the number of top entities5
--nameMetrics name, defined in OAL, such as service_sla, etc.
--serviceThe name of the parent service, could be null if query the global top N.""
--orderThe order of metrics, DES or ASC.DES
--scopeThe scope of the metrics entity, only accept Service/ServiceInstance/Endpoint, ignore others due to those are pointless.Service
--isNormalSet the service to normal or unnormal.true
--startSee Common optionsSee Common options
--endSee Common optionsSee Common options

metrics thermodynamic

optiondescriptiondefault
--nameMetrics name that ends with _heatmap, defined in OAL, such as all_heatmap, etc.all_heatmap
--destServiceThe name of the destination service.""
--destInstanceThe name of the destination instance.""
--destEndpointThe name of the destination endpoint.""
--isDestNormalSet the destination service to normal or unnormal.true
--startSee Common optionsSee Common options
--endSee Common optionsSee Common options

metrics list

optiondescriptiondefault
--regexFilter metrics with a regular expression""

trace

trace displays the spans of a given trace.

argumentdescriptiondefault
trace idThe trace id whose spans are to displayed

trace ls

argumentdescriptiondefault
--trace-idThe trace id whose spans are to displayed
--service-idThe service id whose trace are to displayed
--service-instance-idThe service instance id whose trace are to displayed
--tagsOnly tags defined in the core/default/searchableTagKeys are searchable. Check more details on the Configuration Vocabulary pageSee Configuration Vocabulary page
--startSee Common optionsSee Common options
--endSee Common optionsSee Common options

dashboard

dashboard global-metrics

dashboard global-metrics displays global metrics in the form of a dashboard.

argumentdescriptiondefault
--templateThe template file to customize how to display informationtemplates/Dashboard.Global.json
--startSee Common optionsSee Common options
--endSee Common optionsSee Common options

You can imitate the content of the default template file to customize the dashboard.

dashboard global

dashboard global displays global metrics, global response latency and global heat map in the form of a dashboard.

argumentdescriptiondefault
--templateThe template file to customize how to display informationtemplates/dashboard/global.yml
--refreshThe interval of auto-refresh (s). When start and end are both present, auto-refresh is disabled.6
--startSee Common optionsSee Common options
--endSee Common optionsSee Common options

You can imitate the content of the default template file to customize the dashboard.

checkHealth

argumentdescriptiondefault
--grpcEnable/Disable check gRPC endpointtrue
--grpcAddrThe address of gRPC endpoint127.0.0.1:11800
--grpcTLSEnable/Disable TLS to access gRPC endpointfalse

*Notice: Once enable gRPC TLS, checkHealth command would ignore server's cert.

install

manifest

argumentdescriptiondefault
--nameThe name of prefix of generated resourcesskywalking
--namespaceThe namespace where resource will be deployedskywalking-system
-fThe custom resource file describing custom resources defined by swck

event

report

event report reports an event to OAP server via gRPC.

argumentdescriptiondefault
uuidThe unique ID of the event.
serviceThe service of the event occurred on.
instanceThe service instance of the event occurred on.
endpointThe endpoint of the event occurred on.
nameThe name of the event. For example, ‘Reboot’ and ‘Upgrade’ etc.
typeThe type of the event, could be Normal or Error.Normal
messageThe detail of the event. This should be a one-line message that briefly describes why the event is reported.
startTimeThe start time (in milliseconds) of the event, measured between the current time and midnight, January 1, 1970 UTC.
endTimeThe end time (in milliseconds) of the event, measured between the current time and midnight, January 1, 1970 UTC.

Use Cases

# query the service named projectC
$ ./bin/swctl service ls projectC
[{"id":"4","name":"projectC"}]

If you have already got the id of the service:

$ ./bin/swctl instance ls --service-id=3
[{"id":"3","name":"projectD-pid:7909@skywalking-server-0001","attributes":[{"name":"os_name","value":"Linux"},{"name":"host_name","value":"skywalking-server-0001"},{"name":"process_no","value":"7909"},{"name":"ipv4s","value":"192.168.252.12"}],"language":"JAVA","instanceUUID":"ec8a79d7cb58447c978ee85846f6699a"}]

otherwise,

$ ./bin/swctl instance ls --service-name=projectC
[{"id":"3","name":"projectD-pid:7909@skywalking-server-0001","attributes":[{"name":"os_name","value":"Linux"},{"name":"host_name","value":"skywalking-server-0001"},{"name":"process_no","value":"7909"},{"name":"ipv4s","value":"192.168.252.12"}],"language":"JAVA","instanceUUID":"ec8a79d7cb58447c978ee85846f6699a"}]

If you have already got the id of the service:

$ ./bin/swctl endpoint ls --service-id=3

otherwise,

./bin/swctl service ls projectC | jq '.[].id' | xargs ./bin/swctl endpoint ls --service-id 
[{"id":"22","name":"/projectC/{value}"}]
$ ./bin/swctl --display=graph metrics linear --name=service_instance_resp_time --service "projectC.business-zone" --instance "5ca1e1be91064db6880abac4648667ff@192.168.252.13"

otherwise

$ ./bin/swctl instance ls --service-name=projectC | jq '.[] | select(.name == "projectC-pid:7895@skywalking-server-0001").id' | xargs ./bin/swctl --display=graph metrics linear --name=service_instance_resp_time --service-id
export SERVICE_NAME=projectC.business-zone
export ENDPOINT=/projectC/{value}
export METRICS_NAME=endpoint_cpm
./bin/swctl metrics single --name ${METRICS_NAME} --service ${SERVICE_NAME} --endpoint ${ENDPOINT}

Result:

23
export SERVICE_NAME=projectC.business-zone
export METRICS_NAME=endpoint_cpm
./bin/swctl endpoint ls --service-id=$(./bin/swctl service ls "$SERVICE_NAME" | jq -r '.[0].id') | jq -r '.[].name' | xargs ./bin/swctl metrics single --name "${METRICS_NAME}" --service "${SERVICE_NAME}" --endpoint

Result:

23
$ ./bin/swctl --display=graph --debug metrics multiple-linear --name all_percentile

$ ./bin/swctl metrics top 5 --name service_sla
[{"name":"load balancer1.system","id":"","value":"10000","refId":null},{"name":"load balancer2.system","id":"","value":"10000","refId":null},{"name":"projectB.business-zone","id":"","value":"10000","refId":null},{"name":"projectC.business-zone","id":"","value":"10000","refId":null},{"name":"projectD.business-zone","id":"","value":"10000","refId":null}]
$ ./bin/swctl metrics top 5 --name service_instance_sla     
[{"name":"load balancer1.system - load balancer1.system","id":"","value":"10000","refId":null},{"name":"load balancer2.system - load balancer2.system","id":"","value":"10000","refId":null},{"name":"projectA.business-zone - eb38c5efeb874734a7b17de780685c55@192.168.252.12","id":"","value":"10000","refId":null},{"name":"projectB.business-zone - 4e72bad0f2c14381a5657eaaca7f33ba@192.168.252.12","id":"","value":"10000","refId":null},{"name":"projectB.business-zone - 6e0e2e1cc63145859a21fc7bf7f18d2e@192.168.252.13","id":"","value":"10000","refId":null}]
$ ./bin/swctl metrics top 5 --name endpoint_sla  
[{"name":"load balancer1.system - /projectA/test","id":"","value":"10000","refId":null},{"name":"load balancer1.system - /","id":"","value":"10000","refId":null},{"name":"load balancer2.system - /projectA/test","id":"","value":"10000","refId":null},{"name":"load balancer2.system - /","id":"","value":"10000","refId":null},{"name":"projectA.business-zone - /projectA/{name}","id":"","value":"10000","refId":null}]
$ ./bin/swctl metrics thermodynamic
{"values":[{"id":"202008290939","values":[473,3,0,0,0,0,0,0,0,0,323,0,4,0,0,0,0,0,0,0,436]},{"id":"202008290940","values":[434,0,0,0,0,0,0,0,0,0,367,0,4,0,0,0,0,0,0,0,427]},{"id":"202008290941","values":[504,0,0,0,0,0,0,0,0,0,410,0,5,0,1,0,0,0,0,0,377]},{"id":"202008290942","values":[445,0,4,0,0,0,0,0,0,0,350,0,0,0,0,0,0,0,0,0,420]},{"id":"202008290943","values":[436,0,1,0,0,0,0,0,0,0,367,0,3,0,0,0,0,0,0,0,404]},{"id":"202008290944","values":[463,0,0,0,0,0,0,0,0,0,353,0,0,0,0,0,0,0,0,0,416]},{"id":"202008290945","values":[496,0,2,3,0,0,0,0,0,0,372,0,4,0,0,0,0,0,0,0,393]},{"id":"202008290946","values":[460,0,4,0,0,0,0,0,0,0,396,0,0,0,0,0,0,0,0,0,408]},{"id":"202008290947","values":[533,0,0,0,0,0,0,0,0,0,400,0,0,0,0,0,0,0,0,0,379]},{"id":"202008290948","values":[539,0,0,0,0,0,0,0,0,0,346,0,1,0,0,0,0,0,0,0,424]},{"id":"202008290949","values":[476,0,0,0,1,0,0,0,0,0,353,0,0,0,3,0,0,0,0,0,435]},{"id":"202008290950","values":[509,0,0,0,0,0,0,0,0,0,371,0,0,0,0,0,0,0,0,0,398]},{"id":"202008290951","values":[478,0,2,0,0,0,0,0,0,0,367,0,10,0,4,0,0,0,0,0,413]},{"id":"202008290952","values":[564,0,4,0,0,0,0,0,0,0,342,0,4,0,0,0,0,0,0,0,414]},{"id":"202008290953","values":[476,0,4,0,0,0,0,0,0,0,448,0,4,0,0,0,0,0,0,0,372]},{"id":"202008290954","values":[502,0,1,0,0,0,0,0,0,0,394,0,7,0,0,0,0,0,0,0,392]},{"id":"202008290955","values":[490,0,2,0,0,0,0,0,0,0,383,0,7,0,0,0,0,0,0,0,407]},{"id":"202008290956","values":[474,0,5,0,0,0,0,0,0,0,397,0,3,0,0,0,0,0,0,0,393]},{"id":"202008290957","values":[484,0,4,0,0,0,0,0,0,0,383,0,0,0,0,0,0,0,0,0,402]},{"id":"202008290958","values":[494,0,8,0,0,0,0,0,0,0,361,0,0,0,0,0,0,0,0,0,416]},{"id":"202008290959","values":[434,0,0,0,0,0,0,0,0,0,354,0,0,0,0,0,0,0,0,0,457]},{"id":"202008291000","values":[507,0,1,0,0,0,0,0,0,0,384,0,7,0,0,0,0,0,0,0,405]},{"id":"202008291001","values":[456,0,2,0,0,0,0,0,0,0,388,0,7,0,1,0,0,0,0,0,412]},{"id":"202008291002","values":[506,0,1,0,0,0,0,0,0,0,385,0,0,0,0,0,0,0,0,0,399]},{"id":"202008291003","values":[494,0,8,0,0,0,0,0,0,0,367,0,0,0,0,0,0,0,0,0,415]},{"id":"202008291004","values":[459,0,1,0,0,0,0,0,0,0,263,0,4,0,0,0,0,0,0,0,474]},{"id":"202008291005","values":[513,0,1,0,0,0,0,0,0,0,371,0,3,0,0,0,0,0,0,0,426]},{"id":"202008291006","values":[462,0,1,0,0,0,0,0,0,0,332,0,0,0,0,0,0,0,0,0,435]},{"id":"202008291007","values":[524,0,4,0,1,0,0,0,0,0,365,0,0,0,3,0,0,0,0,0,427]},{"id":"202008291008","values":[442,0,0,0,0,0,0,0,0,0,304,0,0,0,0,0,0,0,0,0,438]},{"id":"202008291009","values":[584,0,0,0,0,0,0,0,0,0,446,0,0,0,0,0,0,0,0,0,343]}],"buckets":[{"min":"0","max":"100"},{"min":"100","max":"200"},{"min":"200","max":"300"},{"min":"300","max":"400"},{"min":"400","max":"500"},{"min":"500","max":"600"},{"min":"600","max":"700"},{"min":"700","max":"800"},{"min":"800","max":"900"},{"min":"900","max":"1000"},{"min":"1000","max":"1100"},{"min":"1100","max":"1200"},{"min":"1200","max":"1300"},{"min":"1300","max":"1400"},{"min":"1400","max":"1500"},{"min":"1500","max":"1600"},{"min":"1600","max":"1700"},{"min":"1700","max":"1800"},{"min":"1800","max":"1900"},{"min":"1900","max":"2000"},{"min":"2000","max":"infinite+"}]}
$ ./bin/swctl --display=graph metrics thermodynamic

heatmap

$ ./bin/swctl --display graph trace 07841b21-c8ed-4ace-851f-5cf0a635dc9f

$ ./bin/swctl --display graph trace ls

$ ./bin/swctl --display graph db g

$ ./bin/swctl --display graph db g --template my-global-template.yml

dashboard-customize

if your backend nodes are deployed in docker and the timezone is UTC, you may not want to convert your timezone to UTC every time you type a command, --timezone comes to your rescue.

$ ./bin/swctl --debug --timezone="0" service ls

--timezone="+1200" and --timezone="-0900" are also valid usage.

if you want to check health status from GraphQL and the gRPC endpoint listening on 10.0.0.1:8843.

$ ./bin/swctl checkHealth --grpcAddr=10.0.0.1:8843

If you only want to query GraphQL.

$ ./bin/swctl checkHealth --grpc=false

Once the gRPC endpoint of OAP encrypts communication by TLS.

$ ./bin/swctl checkHealth --grpcTLS=true

Output manifest with default custom resource

./bin/swctl install manifest oap

Load overlay custom resource from flag

swctl install manifest ui -f oap-cr.yaml

Load overlay custom resource from stdin

cat ui-cr.yaml | ./bin/swctl install manifest oap -f=-

Apply directly to Kubernetes

./bin/swctl install manifest oap -f oap-cr.yaml | kubectl apply -f-

A custome resource file(oap-cr.yaml) to enable ALS analyzer and connect to elasticsearch cluster es1.foo:9200:

spec:
  config:
    - name: SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS
      value: k8s-mesh
    - name: SW_STORAGE
      value: elasticsearch
    - name: SW_STORAGE_ES_CLUSTER_NODES
      value: es1.foo:9200

Output manifest with default custom resource

./bin/swctl install manifest ui

Load overlay custom resource from flag

swctl install manifest ui -f ui-cr.yaml

Load overlay custom resource from stdin

cat ui-cr.yaml | ./bin/swctl install manifest ui -f=-

Apply directly to Kubernetes

./bin/swctl install manifest ui -f ui-cr.yaml | kubectl apply -f-

Some examples of custome resource overlay files(ui-cr.yaml).

  1. Set OAP server address to oap.test, use an ingress to expose UI
spec:
  OAPServerAddress: oap.test
  service:
    ingress:
      host: ui.skywalking.test
  1. Use a Loadbalancer to expose UI
spec:
  service:
    serviceSpec:
      type: LoadBalancer
      ports:
        - name: page
          port: 80
          targetPort: 8080

Contributing

For developers who want to contribute to this project, see Contribution Guide

License

Apache 2.0 License.