docs/modules/ROOT/pages/troubleshooting/operating.adoc - camel-k - Git at Google

 [[operating]]
 = Operating

 NOTE: The following guide uses the terminology from the https://sre.google/sre-book/service-level-objectives/[Site Reliability Engineer] book.

 The Camel K operator exposes a monitoring endpoint, that publishes xref:observability/operator.adoc#metrics[metrics] indicating the _level of service_ provided to its users.
 These metrics materialize the Service Level Indicators (SLIs) for the Camel K operator.

 Service Level Objectives (SLOs) can be defined based on these SLIs.
 The xref:observability/operator.adoc#alerting[default alerts] created for the Camel K operator query the SLIs corresponding metrics, and match the SLOs for the Camel K operator, so that they fire up as soon as the _level of service_ is not met, and preemptive measures can be taken before beaching the Service Level Agreement (SLA) for the Camel K operator.

 [[operator-sops]]
 == Operator SOPs

 The following section lists the Standard Operating Procedures (SOPs), corresponding to the xref:observability/operator.adoc#alerting[default alerts], created for the Camel K operator.
 It assumes the operator has been installed according to the xref:observability/operator.adoc#installation[installation] section from the operator monitoring documentation.

 It documents the recommended troubleshooting actions to be performed when a particular alert fires.
 It is meant to be a living document, to be improved iteratively over time, as users face problematic situations, and actions to troubleshoot and solve them are perfected.

 NOTE: The commands in the following section rely on the `jq` tool, to process the output of the `kubectl` commands. You can refer to the https://stedolan.github.io/jq/download/[download] instructions from the tool Website.

 === CamelKReconciliationDuration

 ==== Description

 This alert has severity level of "warning".
 It's firing when more than 10% of the reconciliation requests have their duration above 0.5s.

 ==== Troubleshooting

 * Check the `rate(camel_k_reconciliation_duration_seconds_bucket{le="0.5"}[5m])` SLI, and identify the resource kinds for which the duration is longer than 0.5s.

 * Improve this SOP if there's anything missing, and contact engineering if there are any changes they could make to make this easier in the future.

 === CamelKReconciliationFailure

 ==== Description

 This alert has severity level of "warning".
 It's firing when some reconciliation requests have failed.

 ==== Troubleshooting

 * Check the `camel_k_reconciliation_duration_seconds_count{result="Errored"}` SLI, and identify the `kind` label(s) for which the value is not zero.

 * Search the operator logs for errors, e.g.:
 +
 [source,sh]
 ----
 $ kubectl logs deployment/camel-k-operator --since=1h \
 | jq -R 'fromjson?
 | select(.level == "error")'
 ----
 Check the `error`, `errorVerbose` and `stacktrace` fields.

 * Inspect the resources corresponding to the errors, e.g.:
 +
 [source,sh]
 ----
 $ kubectl logs deployment/camel-k-operator --since=1h \
 | jq -rR 'fromjson?
 | select(.level == "error")
 | [{namespace, name, controller}]
 | unique
 | .[]
 | "-n \(.namespace) \(.controller | rtrimstr("-controller"))/\(.name)"' \
 | xargs kubectl describe
 ----
 Check the resource specification and events.

 * Improve this SOP if there's anything missing, and contact engineering if there are any changes they could make to make this easier in the future.

 === CamelKSuccessBuildDuration2m

 ==== Description

 This alert has severity level of "warning".
 It's firing when more than 10% of the successful builds have their duration above 2 min.

 ==== Troubleshooting

 * Inspect the successful Builds whose duration is longer than 2 minutes, e.g.:
 +
 [source,sh]
 ----
 $ kubectl get builds.camel.apache.org -o json \
 | jq -r '.items[]
 | select(.status.phase == "Succeeded")
 | select(.status.duration
   | "01-Jan-1970 \(sub("(?<time>.*)\\..*"; "\(.time)s"))" | strptime("%d-%b-%Y %Mm%Ss")? // strptime("%d-%b-%Y %Ss")
   | mktime > 120)
 | "-n \(.metadata.namespace) builds.camel.apache.org/\(.metadata.name)"' \
 | xargs kubectl describe
 ----
 Check the resource specification and events.

 * Improve this SOP if there's anything missing, and contact engineering if there are any changes they could make to make this easier in the future.

 === CamelKSuccessBuildDuration5m

 === Description

 This alert has severity level of "critical".
 It's firing when more than 1% of the successful builds have their duration above 5 min.

 ==== Troubleshooting

 * Inspect the successful Builds whose duration is longer than 5 minutes, e.g.:
 +
 [source,sh]
 ----
 $ kubectl get builds.camel.apache.org -o json \
 | jq -r '.items[]
 | select(.status.phase == "Succeeded")
 | select(.status.duration
   | "01-Jan-1970 \(sub("(?<time>.*)\\..*"; "\(.time)s"))" | strptime("%d-%b-%Y %Mm%Ss")? // strptime("%d-%b-%Y %Ss")
   | mktime > 300)
 | "-n \(.metadata.namespace) builds.camel.apache.org/\(.metadata.name)"' \
 | xargs kubectl describe
 ----
 Check the resource specification and events.

 * Improve this SOP if there's anything missing, and contact engineering if there are any changes they could make to make this easier in the future.

 === CamelKBuildError

 === Description

 This alert has severity level of "critical".
 It's firing when more than 1% of the builds have errored over at least 10 min.

 ==== Troubleshooting

 * Inspect the errored Builds, e.g.:
 +
 [source,sh]
 ----
 $ kubectl get builds.camel.apache.org -o json \
 | jq -r '.items[]
 | select(.status.phase == "Error")
 | "-n \(.metadata.namespace) builds.camel.apache.org/\(.metadata.name)"' \
 | xargs kubectl get -o jsonpath='{.metadata.namespace}{"/"}{.metadata.name}{"\nError: "}{.status.error}{"\n"}'
 ----
 Check the resource specification and events.

 * Improve this SOP if there's anything missing, and contact engineering if there are any changes they could make to make this easier in the future.
	[[operating]]
	= Operating

	NOTE: The following guide uses the terminology from the https://sre.google/sre-book/service-level-objectives/[Site Reliability Engineer] book.

	The Camel K operator exposes a monitoring endpoint, that publishes xref:observability/operator.adoc#metrics[metrics] indicating the _level of service_ provided to its users.
	These metrics materialize the Service Level Indicators (SLIs) for the Camel K operator.

	Service Level Objectives (SLOs) can be defined based on these SLIs.
	The xref:observability/operator.adoc#alerting[default alerts] created for the Camel K operator query the SLIs corresponding metrics, and match the SLOs for the Camel K operator, so that they fire up as soon as the _level of service_ is not met, and preemptive measures can be taken before beaching the Service Level Agreement (SLA) for the Camel K operator.

	[[operator-sops]]
	== Operator SOPs

	The following section lists the Standard Operating Procedures (SOPs), corresponding to the xref:observability/operator.adoc#alerting[default alerts], created for the Camel K operator.
	It assumes the operator has been installed according to the xref:observability/operator.adoc#installation[installation] section from the operator monitoring documentation.

	It documents the recommended troubleshooting actions to be performed when a particular alert fires.
	It is meant to be a living document, to be improved iteratively over time, as users face problematic situations, and actions to troubleshoot and solve them are perfected.

	NOTE: The commands in the following section rely on the `jq` tool, to process the output of the `kubectl` commands. You can refer to the https://stedolan.github.io/jq/download/[download] instructions from the tool Website.

	=== CamelKReconciliationDuration

	==== Description

	This alert has severity level of "warning".
	It's firing when more than 10% of the reconciliation requests have their duration above 0.5s.

	==== Troubleshooting

	* Check the `rate(camel_k_reconciliation_duration_seconds_bucket{le="0.5"}[5m])` SLI, and identify the resource kinds for which the duration is longer than 0.5s.

	* Improve this SOP if there's anything missing, and contact engineering if there are any changes they could make to make this easier in the future.

	=== CamelKReconciliationFailure

	==== Description

	This alert has severity level of "warning".
	It's firing when some reconciliation requests have failed.

	==== Troubleshooting

	* Check the `camel_k_reconciliation_duration_seconds_count{result="Errored"}` SLI, and identify the `kind` label(s) for which the value is not zero.

	* Search the operator logs for errors, e.g.:
	+
	[source,sh]
	----
	$ kubectl logs deployment/camel-k-operator --since=1h \
	\| jq -R 'fromjson?
	\| select(.level == "error")'
	----
	Check the `error`, `errorVerbose` and `stacktrace` fields.

	* Inspect the resources corresponding to the errors, e.g.:
	+
	[source,sh]
	----
	$ kubectl logs deployment/camel-k-operator --since=1h \
	\| jq -rR 'fromjson?
	\| select(.level == "error")
	\| [{namespace, name, controller}]
	\| unique
	\| .[]
	\| "-n \(.namespace) \(.controller \| rtrimstr("-controller"))/\(.name)"' \
	\| xargs kubectl describe
	----
	Check the resource specification and events.

	* Improve this SOP if there's anything missing, and contact engineering if there are any changes they could make to make this easier in the future.

	=== CamelKSuccessBuildDuration2m

	==== Description

	This alert has severity level of "warning".
	It's firing when more than 10% of the successful builds have their duration above 2 min.

	==== Troubleshooting

	* Inspect the successful Builds whose duration is longer than 2 minutes, e.g.:
	+
	[source,sh]
	----
	$ kubectl get builds.camel.apache.org -o json \
	\| jq -r '.items[]
	\| select(.status.phase == "Succeeded")
	\| select(.status.duration
	\| "01-Jan-1970 \(sub("(?<time>.)\\.."; "\(.time)s"))" \| strptime("%d-%b-%Y %Mm%Ss")? // strptime("%d-%b-%Y %Ss")
	\| mktime > 120)
	\| "-n \(.metadata.namespace) builds.camel.apache.org/\(.metadata.name)"' \
	\| xargs kubectl describe
	----
	Check the resource specification and events.

	* Improve this SOP if there's anything missing, and contact engineering if there are any changes they could make to make this easier in the future.

	=== CamelKSuccessBuildDuration5m

	=== Description

	This alert has severity level of "critical".
	It's firing when more than 1% of the successful builds have their duration above 5 min.

	==== Troubleshooting

	* Inspect the successful Builds whose duration is longer than 5 minutes, e.g.:
	+
	[source,sh]
	----
	$ kubectl get builds.camel.apache.org -o json \
	\| jq -r '.items[]
	\| select(.status.phase == "Succeeded")
	\| select(.status.duration
	\| "01-Jan-1970 \(sub("(?<time>.)\\.."; "\(.time)s"))" \| strptime("%d-%b-%Y %Mm%Ss")? // strptime("%d-%b-%Y %Ss")
	\| mktime > 300)
	\| "-n \(.metadata.namespace) builds.camel.apache.org/\(.metadata.name)"' \
	\| xargs kubectl describe
	----
	Check the resource specification and events.

	* Improve this SOP if there's anything missing, and contact engineering if there are any changes they could make to make this easier in the future.

	=== CamelKBuildError

	=== Description

	This alert has severity level of "critical".
	It's firing when more than 1% of the builds have errored over at least 10 min.

	==== Troubleshooting

	* Inspect the errored Builds, e.g.:
	+
	[source,sh]
	----
	$ kubectl get builds.camel.apache.org -o json \
	\| jq -r '.items[]
	\| select(.status.phase == "Error")
	\| "-n \(.metadata.namespace) builds.camel.apache.org/\(.metadata.name)"' \
	\| xargs kubectl get -o jsonpath='{.metadata.namespace}{"/"}{.metadata.name}{"\nError: "}{.status.error}{"\n"}'
	----
	Check the resource specification and events.

	* Improve this SOP if there's anything missing, and contact engineering if there are any changes they could make to make this easier in the future.