title: Kubernetes keywords:
The Kubernetes service discovery List-Watch real-time changes of Endpoints resources, then store theirs value into ngx.shared.DICT.
Discovery also provides a node query interface in accordance with the APISIX Discovery Specification.
Kubernetes service discovery both support single-cluster and multi-cluster modes, applicable to the case where the service is distributed in single or multiple Kubernetes clusters.
A detailed configuration for single-cluster mode Kubernetes service discovery is as follows:
discovery: kubernetes: service: # apiserver schema, options [http, https] schema: https #default https # apiserver host, options [ipv4, ipv6, domain, environment variable] host: ${KUBERNETES_SERVICE_HOST} #default ${KUBERNETES_SERVICE_HOST} # apiserver port, options [port number, environment variable] port: ${KUBERNETES_SERVICE_PORT} #default ${KUBERNETES_SERVICE_PORT} client: # serviceaccount token or token_file token_file: /var/run/secrets/kubernetes.io/serviceaccount/token #token: |- # eyJhbGciOiJSUzI1NiIsImtpZCI6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEif # 6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEifeyJhbGciOiJSUzI1NiIsImtpZCI default_weight: 50 # weight assigned to each discovered endpoint. default 50, minimum 0 # kubernetes discovery support namespace_selector # you can use one of [equal, not_equal, match, not_match] filter namespace namespace_selector: # only save endpoints with namespace equal default equal: default # only save endpoints with namespace not equal default #not_equal: default # only save endpoints with namespace match one of [default, ^my-[a-z]+$] #match: #- default #- ^my-[a-z]+$ # only save endpoints with namespace not match one of [default, ^my-[a-z]+$ ] #not_match: #- default #- ^my-[a-z]+$ # kubernetes discovery support label_selector # for the expression of label_selector, please refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels label_selector: |- first="a",second="b" # reserved lua shared memory size,1m memory can store about 1000 pieces of endpoint shared_size: 1m #default 1m # if watch_endpoint_slices setting true, watch apiserver with endpointslices instead of endpoints watch_endpoint_slices: false #default false
If the Kubernetes service discovery runs inside a pod, you can use minimal configuration:
discovery: kubernetes: { }
If the Kubernetes service discovery runs outside a pod, you need to create or select a specified ServiceAccount, then get its token value, and use following configuration:
discovery: kubernetes: service: schema: https host: # enter apiserver host value here port: # enter apiserver port value here client: token: # enter serviceaccount token value here #token_file: # enter file path here
The Kubernetes service discovery provides a query interface in accordance with the APISIX Discovery Specification.
function: nodes(service_name)
description: nodes() function attempts to look up the ngx.shared.DICT for nodes corresponding to service_name,
service_name should match pattern: [namespace]/[name]:[portName]
namespace: The namespace where the Kubernetes endpoints is located
name: The name of the Kubernetes endpoints
portName: The ports.name value in the Kubernetes endpoints, if there is no ports.name, use targetPort, port instead. If ports.name exists, then port number cannot be used.
return value: if the Kubernetes endpoints value is as follows:
apiVersion: v1 kind: Endpoints metadata: name: plat-dev namespace: default subsets: - addresses: - ip: "10.5.10.109" - ip: "10.5.10.110" ports: - port: 3306 name: port
a nodes(“default/plat-dev:port”) call will get follow result:
{
{
host="10.5.10.109",
port= 3306,
weight= 50,
},
{
host="10.5.10.110",
port= 3306,
weight= 50,
},
}
A detailed configuration for multi-cluster mode Kubernetes service discovery is as follows:
discovery: kubernetes: - id: release # a custom name refer to the cluster, pattern ^[a-z0-9]{1,8} service: # apiserver schema, options [http, https] schema: https #default https # apiserver host, options [ipv4, ipv6, domain, environment variable] host: "1.cluster.com" # apiserver port, options [port number, environment variable] port: "6443" client: # serviceaccount token or token_file token_file: /var/run/secrets/kubernetes.io/serviceaccount/token #token: |- # eyJhbGciOiJSUzI1NiIsImtpZCI6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEif # 6Ikx5ME1DNWdnbmhQNkZCNlZYMXBsT3pYU3BBS2swYzBPSkN3ZnBESGpkUEEifeyJhbGciOiJSUzI1NiIsImtpZCI default_weight: 50 # weight assigned to each discovered endpoint. default 50, minimum 0 # kubernetes discovery support namespace_selector # you can use one of [equal, not_equal, match, not_match] filter namespace namespace_selector: # only save endpoints with namespace equal default equal: default # only save endpoints with namespace not equal default #not_equal: default # only save endpoints with namespace match one of [default, ^my-[a-z]+$] #match: #- default #- ^my-[a-z]+$ # only save endpoints with namespace not match one of [default, ^my-[a-z]+$] #not_match: #- default #- ^my-[a-z]+$ # kubernetes discovery support label_selector # for the expression of label_selector, please refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels label_selector: |- first="a",second="b" # reserved lua shared memory size,1m memory can store about 1000 pieces of endpoint shared_size: 1m #default 1m # if watch_endpoint_slices setting true, watch apiserver with endpointslices instead of endpoints watch_endpoint_slices: false #default false
Multi-Kubernetes service discovery does not fill default values for service and client fields, you need to fill them according to the cluster configuration.
The Kubernetes service discovery provides a query interface in accordance with the APISIX Discovery Specification.
function: nodes(service_name)
description: nodes() function attempts to look up the ngx.shared.DICT for nodes corresponding to service_name,
service_name should match pattern: [id]/[namespace]/[name]:[portName]
id: value defined in service discovery configuration
namespace: The namespace where the Kubernetes endpoints is located
name: The name of the Kubernetes endpoints
portName: The ports.name value in the Kubernetes endpoints, if there is no ports.name, use targetPort, port instead. If ports.name exists, then port number cannot be used.
return value: if the Kubernetes endpoints value is as follows:
apiVersion: v1 kind: Endpoints metadata: name: plat-dev namespace: default subsets: - addresses: - ip: "10.5.10.109" - ip: "10.5.10.110" ports: - port: 3306 name: port
a nodes(“release/default/plat-dev:port”) call will get follow result:
{
{
host="10.5.10.109",
port= 3306,
weight= 50,
},
{
host="10.5.10.110",
port= 3306,
weight= 50,
},
}
Q: Why only support configuration token to access Kubernetes APIServer?
A: Usually, we will use three ways to complete the authentication of Kubernetes APIServer:
Because lua-resty-http does not currently support mTLS, and basic authentication is not recommended, so currently only the token authentication method is implemented.
Q: APISIX inherits Nginx's multiple process model, does it mean that each nginx worker process will List-Watch kubernetes endpoints resources?
A: The Kubernetes service discovery only uses privileged processes to List-Watch Kubernetes endpoints resources, then store theirs value into ngx.shared.DICT, worker processes get results by querying ngx.shared.DICT.
Q: What permissions do ServiceAccount require?
A: ServiceAccount requires the permissions of cluster-level [ get, list, watch ] endpoints resources, the declarative definition is as follows:
kind: ServiceAccount apiVersion: v1 metadata: name: apisix-test namespace: default --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: apisix-test rules: - apiGroups: [ "" ] resources: [ endpoints,endpointslices ] verbs: [ get,list,watch ] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: apisix-test roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: apisix-test subjects: - kind: ServiceAccount name: apisix-test namespace: default
Q: How to get ServiceAccount token value?
A: Assume your ServiceAccount located in namespace apisix and name is Kubernetes-discovery, you can use the following steps to get token value.
kubectl -n apisix get secrets | grep kubernetes-discovery
kubectl -n apisix get secret kubernetes-discovery-token-c64cv -o jsonpath={.data.token} | base64 -d
It also offers control api for debugging.
To query/list the nodes discoverd by kubernetes discovery, you can query the /v1/discovery/kubernetes/dump control API endpoint like so:
GET /v1/discovery/kubernetes/dump
Which will yield the following response:
{
"endpoints": [
{
"endpoints": [
{
"value": "{\"https\":[{\"host\":\"172.18.164.170\",\"port\":6443,\"weight\":50},{\"host\":\"172.18.164.171\",\"port\":6443,\"weight\":50},{\"host\":\"172.18.164.172\",\"port\":6443,\"weight\":50}]}",
"name": "default/kubernetes"
},
{
"value": "{\"metrics\":[{\"host\":\"172.18.164.170\",\"port\":2379,\"weight\":50},{\"host\":\"172.18.164.171\",\"port\":2379,\"weight\":50},{\"host\":\"172.18.164.172\",\"port\":2379,\"weight\":50}]}",
"name": "kube-system/etcd"
},
{
"value": "{\"http-85\":[{\"host\":\"172.64.89.2\",\"port\":85,\"weight\":50}]}",
"name": "test-ws/testing"
}
],
"id": "first"
}
],
"config": [
{
"default_weight": 50,
"id": "first",
"client": {
"token": "xxx"
},
"service": {
"host": "172.18.164.170",
"port": "6443",
"schema": "https"
},
"shared_size": "1m"
}
]
}