solr/solr-ref-guide/src/solr-tracing.adoc

= Distributed Solr Tracing
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.

Solr includes a general tracing framework based on OpenTracing that can be used to trace lifecycle of a request for performance monitoring.

Tracing data can be configured to send to arbitrary backend like Jaeger, Zipkin, Datadog, etc.
At the moment, only Jaeger is supported out of the box.

A sampled distributed tracing query request on Jaeger looks like this:

.Tracing of a Solr query
image::images/solr-tracing/query-request-tracing.png[]

== Configuring Tracer

`TracerConfigurator` is a class used to provide an instance of `io.opentracing.Tracer` based on configuration in `solr.xml`.
The `JaegerTracerConfigurator` provides a JaegerTracer instance to a Solr node.

A setup of a `TracerConfigurator` in `solr.xml` looks like this:

[source,xml]
----
<solr>
  <tracerConfig name="tracerConfig" class="org.apache.solr.jaeger.JaegerTracerConfigurator">
    <str name="agentHost">localhost</str>
    <int name="agentPort">5775</int>
    <bool name="logSpans">true</bool>
    <int name="flushInterval">1000</int>
    <int name="maxQueueSize">10000</int>
  </tracerConfig>
</solr>
----

If `<tracerConfig>` is absent, `TracerConfigurator` will try to pick up the `Tracer` instance that was registered in `io.opentracing.util.GlobalTracer`.
By doing this some backends like DataDog are supported out of the box since
https://docs.datadoghq.com/tracing/setup/java/[datadog-java-agent] uses Javaagent to register a `Tracer` in
`io.opentracing.util.GlobalTracer`.

=== Configuring Sample Rate

By default only 0.1% of requests are sampled, this ensures that tracing activities do not affect performance of a node.

The rate can be changed without needing to restart Solr nodes by setting the new sample rate as a cluster property using the Collections API's <<cluster-node-management.adoc#clusterprop,CLUSTERPROP>> command.

For example, the following sets the `samplePercentage` property to 100%:

[source,text]
/admin/collections?action=CLUSTERPROP&name=samplePercentage&val=100

== Jaeger Tracer Configurator

The `contrib/jagertracer-configurator` provides a default implementation for setting up Jaeger Tracer.

Note that all libraries of `jaegertracer-configurator` must be included in the classpath of all nodes.
Then Jaeger tracer can be set up in `solr.xml` like this:

[source,xml]
----
<tracerConfig name="tracerConfig" class="org.apache.solr.jaeger.JaegerTracerConfigurator">
  <str name="agentHost">localhost</str>
  <int name="agentPort">5775</int>
  <bool name="logSpans">true</bool>
  <int name="flushInterval">1000</int>
  <int name="maxQueueSize">10000</int>
</tracerConfig>
----

The parameters for `JaegerTracerConfigurator` include:

`agentHost`::
+
[%autowidth,frame=none]
|===
s|Required |Default: none
|===
+
The Jaeger backend host.

`agentPort`::
+
[%autowidth,frame=none]
|===
s|Required |Default: none
|===
+
The Jaeger port.

`logSpans`::
+
[%autowidth,frame=none]
|===
|Optional |Default: `true`
|===
+
Whether the tracer should also log the spans.

`flushInterval`::
+
[%autowidth,frame=none]
|===
|Optional | Default: `5000` (ms)
|===
+
The tracer's flush interval.

`maxQueueSize`::
+
[%autowidth,frame=none]
|===
|Optional | Default: `10000`
|===
+
The tracer's maximum queue size.

Other parameters which are not listed above can be configured using system properties or environment variables.
The full list of parameters can be found in the https://github.com/jaegertracing/jaeger-client-java/blob/master/jaeger-core/README.md[Jaeger-README].