Configuration

Configure Operator

Spark Operator supports different ways to configure the behavior:

• spark-operator.properties provided when deploying the operator. In addition to the [property file](../build-tools/helm/spark-kubernetes-operator/conf/spark-operator. properties), it is also possible to override or append config properties in helm Values files.
• System Properties : when provided as system properties (e.g. via -D options to the operator JVM), it overrides the values provided in property file.
• Hot property loading : when enabled, a configmap would be created with the operator in the same namespace. Operator can monitor updates performed on the configmap. Hot properties reloading takes higher precedence comparing with default properties override.
  • An example use case: operator use hot properties to figure the list of namespace(s) to operate Spark applications. The hot properties config map can be updated and maintained by user or additional microservice to tune the operator behavior without rebooting it.
  • Please be advised that not all properties can be hot-loaded and honored at runtime. Refer the list of supported properties for more details.

To enable hot properties loading, update the helm chart values file with

operatorConfiguration:
  spark-operator.properties: |+
    spark.operator.dynamic.config.enabled=true
    # ... all other config overides...
  dynamicConfig:
    create: true

Metrics

Spark operator, following Apache Spark, has a configurable metrics system based on the Dropwizard Metrics Library. Note that Spark Operator does not have Spark UI, MetricsServlet and PrometheusServlet from org.apache.spark.metrics.sink package are not supported. If you are interested in Prometheus metrics exporting, please take a look at below section Forward Metrics to Prometheus

JVM Metrics

Spark Operator collects JVM metrics via Codahale JVM Metrics

• BufferPoolMetricSet
• FileDescriptorRatioGauge
• GarbageCollectorMetricSet
• MemoryUsageGaugeSet
• ThreadStatesGaugeSet

Kubernetes Client Metrics

Metrics Name	Type	Description
kubernetes.client.http.request	Meter	Tracking the rates of HTTP request sent to the Kubernetes API Server
kubernetes.client.http.response	Meter	Tracking the rates of HTTP response from the Kubernetes API Server
kubernetes.client.http.response.failed	Meter	Tracking the rates of HTTP requests which have no response from the Kubernetes API Server
kubernetes.client.http.response.latency.nanos	Histograms	Measures the statistical distribution of HTTP response latency from the Kubernetes API Server
kubernetes.client.http.response.	Meter	Tracking the rates of HTTP response based on response code from the Kubernetes API Server
kubernetes.client.http.request.	Meter	Tracking the rates of HTTP request based type of method to the Kubernetes API Server
kubernetes.client.http.response.1xx	Meter	Tracking the rates of HTTP Code 1xx responses (informational) received from the Kubernetes API Server per response code.
kubernetes.client.http.response.2xx	Meter	Tracking the rates of HTTP Code 2xx responses (success) received from the Kubernetes API Server per response code.
kubernetes.client.http.response.3xx	Meter	Tracking the rates of HTTP Code 3xx responses (redirection) received from the Kubernetes API Server per response code.
kubernetes.client.http.response.4xx	Meter	Tracking the rates of HTTP Code 4xx responses (client error) received from the Kubernetes API Server per response code.
kubernetes.client.http.response.5xx	Meter	Tracking the rates of HTTP Code 5xx responses (server error) received from the Kubernetes API Server per response code.
kubernetes.client..	Meter	Tracking the rates of HTTP request for a combination of one Kubernetes resource and one http method
kubernetes.client...	Meter	Tracking the rates of HTTP request for a combination of one namespace-scoped Kubernetes resource and one http method

Forward Metrics to Prometheus

In this section, we will show you how to forward Spark Operator metrics to Prometheus.

• Modify the metrics properties section in the file build-tools/helm/spark-kubernetes-operator/values.yaml:

metrics.properties:|+
  spark.metrics.conf.operator.sink.prometheus.class=org.apache.spark.kubernetes.operator.metrics.
sink.PrometheusPullModelSink

• Install Spark Operator

helm install spark-kubernetes-operator -f build-tools/helm/spark-kubernetes-operator/values.yaml build-tools/helm/spark-kubernetes-operator/

• Install Prometheus via Helm Chart

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/prometheus

• Find and Annotate Spark Operator Pods

kubectl get pods -l app.kubernetes.io/name=spark-kubernetes-operator
NAME                                         READY   STATUS    RESTARTS   AGE
spark-kubernetes-operator-598cb5d569-bvvd2   1/1     Running   0          24m

kubectl annotate pods spark-kubernetes-operator-598cb5d569-bvvd2 prometheus.io/scrape=true
kubectl annotate pods spark-kubernetes-operator-598cb5d569-bvvd2 prometheus.io/path=/prometheus
kubectl annotate pods spark-kubernetes-operator-598cb5d569-bvvd2 prometheus.io/port=19090

• Check Metrics via Prometheus UI

kubectl get pods | grep "prometheus-server"
prometheus-server-654bc74fc9-8hgkb                   2/2     Running   0          59m

kubectl port-forward --address 0.0.0.0 pod/prometheus-server-654bc74fc9-8hgkb  8080:9090

open your browser with address localhost:8080. Click on Status Targets tab, you should be able to find target as below.