In the Apache SkyWalking ecosystem, the OAP obtains metrics, traces, logs, and event data through SkyWalking Agent, Envoy, or other data sources. Under the gRPC protocol, it transmits data by communicating with a single server node. Only when the connection is broken, the reconnecting policy would be used based on DNS round-robin mode. When new services are added at runtime or the OAP load is kept high due to increased traffic of observed services, the OAP cluster needs to scale out for increased traffic. The load of the new OAP node would be less due to all existing agents having connected to previous nodes. Even without scaling, the load of OAP nodes would be unbalanced, because the agent would keep the connection due to random policy at the booting stage. In these cases, it would become a challenge to keep up the health status of all nodes, and be able to scale out when needed.
In this article, we mainly discuss how to solve this challenge in SkyWalking.
SkyWalking mainly uses the gRPC protocol for data transmission, so this article mainly introduces load balancing in the gRPC protocol.
Based on the gRPC official Load Balancing blog, there are two approaches to load balancing:
From the perspective of observability system architecture:
Pros | Cons | |
---|---|---|
Client-side | High performance because of the elimination of extra hop | Complex client (cluster awareness, load balancing, health check, etc.)Ensure each data source to be connected provides complex client capabilities |
Proxy | Simple Client | Higher latency |
We choose Proxy mode for the following reasons:
In the proxy mode, we should determine the transmission path between downstream and upstream.
Different data protocols require different processing policies. There are two transmission policies:
The synchronization policy requires that the proxy send the message to the upstream server when receiving the client message, and synchronously return the response data to the downstream client. Usually, only a few protocols need to use the synchronization policy.
As shown below, after the client sends the request to the Proxy, the proxy would send the message to the server synchronously. When the proxy receives the result, it returns to the client.
The asynchronous batch policy means that the data is sent to the upstream server in batches asynchronously. This policy is more common because most protocols in SkyWalking are primarily based on data reporting. We think using the queue as a buffer could have a good effect. The asynchronous batch policy is executed according to the following steps:
The advantage of using queues is:
As shown below, after the proxy receives the message, the proxy would wrap the message as an event and push it to the queue. The message sender would take batch events from the queue and send them to the upstream OAP.
Routing algorithms are used to route messages to a single upstream server node.
The Round-Robin algorithm selects nodes in order from the list of upstream service nodes. The advantage of this algorithm is that the number of times each node is selected is average. When the size of the data is close to the same, each upstream node can handle the same quantity of data content.
With the Weight Round-Robin, each upstream server node has a corresponding routing weight ratio. The difference from Round-Robin is that each upstream node has more chances to be routed according to its weight. This algorithm is more suitable to use when the upstream server node machine configuration is not the same.
The Fixed algorithm is a hybrid algorithm. It can ensure that the same data is routed to the same upstream server node, and when the upstream server scales out, it still maintains routing to the same node; unless the upstream node does not exist, it will reroute. This algorithm is mainly used in the SkyWalking Meter protocol because this protocol needs to ensure that the metrics of the same service instance are sent to the same OAP node. The Routing steps are as follows:
The advantage of this algorithm is to bind the data with the upstream server node as much as possible, so the upstream server can better process continuous data. The disadvantage is that it takes up a certain amount of memory space to save the corresponding relationship.
As shown below, the image is divided into two parts:
We choose to use a combination of Round-Robin and Fixed algorithm for routing:
Proxy still needs to deal with the load balancing problem from client to itself, especially when deploying a Proxy cluster in a production environment.
There are three ways to solve this problem:
max_connection
config on the client-side to specify the maximum connection duration of each connection. For more information, please read the proposal.Connection management | Cluster awareness | Resource Limit+HPA | |
---|---|---|---|
Pros | Simple to use | Ensure that the number of connections in each proxy is relatively | Simple to use |
Cons | Each client needs to ensure that data is not lostThe client is required to accept GOWAY responses | May cause a sudden increase in traffic on some nodesEach client needs to ensure that data is not lost | Traffic will not be particularly balanced in each instance |
We choose Limit+HPA for these reasons:
We have implemented this Proxy in the SkyWalking-Satellite project. It’s used between Client and SkyWalking OAP, effectively solving the load balancing problem.
After the system is deployed, the Satellite would accept the traffic from the Client, and the Satellite will perceive all the nodes of the OAP through Kubernetes Label Selector or manual configuration, and load balance the traffic to the upstream OAP node.
As shown below, a single client still maintains a connection with a single Satellite, Satellite would establish the connection with each OAP, and load balance message to the OAP node.
When scaling Satellite, we need to deploy the SWCK adapter and configure the HPA in Kubernetes. SWCK is a platform for the SkyWalking users, provisions, upgrades, maintains SkyWalking relevant components, and makes them work natively on Kubernetes.
After deployment is finished, the following steps would be performed:
As shown below, use the dotted line to divide the two parts. HPA uses SWCK Adapter to read the metrics in the OAP. When the threshold is met, HPA would scale the Satellite deployment.
In this section, we will demonstrate two cases:
NOTE: All commands could be accessed through GitHub.
We will use the bookinfo application to demonstrate how to integrate Apache SkyWalking 8.9.1 with Apache SkyWalking-Satellite 0.5.0, and observe the service mesh through the Envoy ALS protocol.
Before starting, please make sure that you already have a Kubernetes environment.
Istio provides a very convenient way to configure the Envoy proxy and enable the access log service. The following step:
# install istioctl export ISTIO_VERSION=1.12.0 curl -L https://istio.io/downloadIstio | sh - sudo mv $PWD/istio-$ISTIO_VERSION/bin/istioctl /usr/local/bin/ # install istio istioctl install -y --set profile=demo \ --set meshConfig.enableEnvoyAccessLogService=true \ --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-system-satellite.skywalking-system:11800 # enbale envoy proxy in default namespace kubectl label namespace default istio-injection=enabled
SWCK provides convenience for users to deploy and upgrade SkyWalking related components based on Kubernetes. The automatic scale function of Satellite also mainly relies on SWCK. For more information, you could refer to the official documentation.
# Install cert-manager kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml # Deploy SWCK mkdir -p skywalking-swck && cd skywalking-swck wget https://dlcdn.apache.org/skywalking/swck/0.6.1/skywalking-swck-0.6.1-bin.tgz tar -zxvf skywalking-swck-0.6.1-bin.tgz cd config kubectl apply -f operator-bundle.yaml
We have provided a simple script to deploy the skywalking OAP, UI, and Satellite.
# Create the skywalking components namespace kubectl create namespace skywalking-system kubectl label namespace skywalking-system swck-injection=enabled # Deploy components kubectl apply -f https://raw.githubusercontent.com/mrproliu/sw-satellite-demo-scripts/5821a909b647f7c8f99c70378e197630836f45f7/resources/sw-components.yaml
export ISTIO_VERSION=1.12.0 kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml kubectl wait --for=condition=Ready pods --all --timeout=1200s kubectl port-forward service/productpage 9080
Next, please open your browser and visit http://localhost:9080
. You should be able to see the Bookinfo application. Refresh the webpage several times to generate enough access logs.
Then, you can see the topology and metrics of the Bookinfo application on SkyWalking WebUI. At this time, you can see that the Satellite is working!
We need to install OpenTelemetry Collector to collect metrics in OAPs and analyze them.
# Add OTEL collector kubectl apply -f https://raw.githubusercontent.com/mrproliu/sw-satellite-demo-scripts/5821a909b647f7c8f99c70378e197630836f45f7/resources/otel-collector-oap.yaml kubectl port-forward -n skywalking-system service/skywalking-system-ui 8080:80
Next, please open your browser and visit http://localhost:8080/
and create a new item on the dashboard. The SkyWalking Web UI pictured below shows how the data content is applied.
Scaling the number of OAPs by deployment.
kubectl scale --replicas=3 -n skywalking-system deployment/skywalking-system-oap
After a period of time, you will see that the number of OAPs becomes 3, and the ALS traffic is balanced to each OAP.
After we have completed the SkyWalking Scaling, we would carry out the Satellite Scaling demo.
SWCK provides an adapter to implement the Kubernetes external metrics to adapt the HPA through reading the metrics in SkyWalking OAP. We expose the metrics service in Satellite to OAP and configure HPA Resource to auto-scaling the Satellite.
Install the SWCK adapter into the Kubernetes environment:
kubectl apply -f skywalking-swck/config/adapter-bundle.yaml
Create the HPA resource, and limit each Satellite to handle a maximum of 10 connections:
kubectl apply -f https://raw.githubusercontent.com/mrproliu/sw-satellite-demo-scripts/5821a909b647f7c8f99c70378e197630836f45f7/resources/satellite-hpa.yaml
Then, you could see we have 9 connections in one satellite. One envoy proxy may establish multiple connections to the satellite.
$ kubectl get HorizontalPodAutoscaler -n skywalking-system NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE hpa-demo Deployment/skywalking-system-satellite 9/10 1 3 1 5m18s
The scaling application could establish more connections to the satellite, to verify whether the HPA is in effect.
kubectl scale --replicas=3 deployment/productpage-v1 deployment/details-v1
By default, Satellite will deploy a single instance and a single instance will only accept 11 connections. HPA resources limit one Satellite to handle 10 connections and use a stabilization window to make Satellite stable scaling up. In this case, we deploy the Bookinfo application in 10+ instances after scaling, which means that 10+ connections will be established to the Satellite.
So after HPA resources are running, the Satellite would be automatically scaled up to 2 instances. You can learn about the calculation algorithm of replicas through the official documentation. Run the following command to view the running status:
$ kubectl get HorizontalPodAutoscaler -n skywalking-system --watch NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE hpa-demo Deployment/skywalking-system-satellite 11/10 1 3 1 3m31s hpa-demo Deployment/skywalking-system-satellite 11/10 1 3 1 4m20s hpa-demo Deployment/skywalking-system-satellite 11/10 1 3 2 4m38s hpa-demo Deployment/skywalking-system-satellite 11/10 1 3 2 5m8s hpa-demo Deployment/skywalking-system-satellite 6/10 1 3 2 5m23s
By observing the “number of connections” metric, we would be able to see that when the number of connections of each gRPC exceeds 10 connections, then the satellite automatically scales through the HPA rule. As a result, the connection number is down to normal status (in this example, less than 10)
swctl metrics linear --name satellite_service_grpc_connect_count --service-name satellite::satellite-service