---
active_crumb: Metrics <span class="amp">&amp;</span> Tracing
layout: documentation
id: metrics
---

<!--
 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements.  See the NOTICE file distributed with
 this work for additional information regarding copyright ownership.
 The ASF licenses this file to You under the Apache License, Version 2.0
 (the "License"); you may not use this file except in compliance with
 the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
-->

<div id="metrics-and-tracing" class="col-md-8 second-column">
    <section id="overview">
        <h2 class="section-title">Overview</h2>
        <p>
            Both REST server and data probe come with <a target=_ href="https://opencensus.io">OpenCensus</a>
            instrumentation. NLPCraft bundles several built-in exporters for various OpenCensus trace and stats
            backends:
        </p>
        <table class="gradient-table checks">
            <thead>
            <tr>
                <th>Backend</th>
                <th>Built-In Exporters</th>
                <th>Stats or Trace?</th>
            </tr>
            </thead>
            <tbody>
            <tr>
                <td>
                    <a target=_ href="https://www.jaegertracing.io/">
                        <img alt="" style="width: 48px" src="/images/jaeger-logo-x.png">
                        <div class="exporter-name">Jaeger</div>
                    </a>
                </td>
                <td>
                    Server: <code>org.apache.nlpcraft.server.lifecycle.opencensus.NCJaegerExporter</code><br>
                    Probe:
                    <a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/opencensus/NCJaegerExporter.html">org.apache.nlpcraft.model.opencensus.NCJaegerExporter</a>
                </td>
                <td><center>Trace</center></td>
            </tr>
            <tr>
                <td>
                    <a target=_ href="https://zipkin.io/">
                        <img alt="" style="width: 48px" src="/images/zipkin-logo.jpg">
                        <div class="exporter-name">Zipkin</div>
                    </a>
                </td>
                <td>
                    Server: <code>org.apache.nlpcraft.server.lifecycle.opencensus.NCZipkinExporter</code><br>
                    Probe: <a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/opencensus/NCZipkinExporter.html">org.apache.nlpcraft.model.opencensus.NCZipkinExporter</a>
                </td>
                <td><center>Trace</center></td>
            </tr>
            <tr>
                <td>
                    <a target=_ href="https://prometheus.io/">
                        <img alt="" style="width: 48px" src="/images/prometheus-logo-x.png">
                        <div class="exporter-name">Prometheus</div>
                    </a>
                </td>
                <td>
                    Server: <code>org.apache.nlpcraft.server.lifecycle.opencensus.NCPrometheusExporter</code><br>
                    Probe: <a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/opencensus/NCPrometheusExporter.html">org.apache.nlpcraft.model.opencensus.NCPrometheusExporter</a>
                </td>
                <td><center>Stats</center></td>
            </tr>
            <tr>
                <td>
                    <a target=_ href="https://grafana.com/">
                        <img alt="" style="width: 48px" src="/images/grafana-logo-x.png">
                        <div class="exporter-name">Grafana<sup>*</sup></div>
                    </a>
                </td>
                <td>
                    Server: <code>org.apache.nlpcraft.server.lifecycle.opencensus.NCPrometheusExporter</code><br>
                    Probe: <a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/opencensus/NCPrometheusExporter.html">org.apache.nlpcraft.model.opencensus.NCPrometheusExporter</a>
                </td>
                <td><center>Stats</center></td>
            </tr>
            <tr>
                <td>
                    <a target=_ href="https://cloud.google.com/stackdriver/">
                        <img alt="" style="width: 48px" src="/images/stackdriver-stats-x.png">
                        <div class="exporter-name">Stackdriver</div>
                    </a>
                </td>
                <td>
                    Server: <code>org.apache.nlpcraft.server.lifecycle.opencensus.NCStackdriverStatsExporter</code><br>
                    Probe: <a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/opencensus/NCStackdriverStatsExporter.html">org.apache.nlpcraft.model.opencensus.NCStackdriverStatsExporter</a>
                </td>
                <td><center>Stats</center></td>
            </tr>
            <tr>
                <td>
                    <a target=_ href="https://cloud.google.com/stackdriver/">
                        <img alt="" style="width: 48px" src="/images/stackdriver-trace-x.png">
                        <div class="exporter-name">Stackdriver</div>
                    </a>
                </td>
                <td>
                    Server: <code>org.apache.nlpcraft.server.lifecycle.opencensus.NCStackdriverTraceExporter</code><br>
                    Probe: <a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/opencensus/NCStackdriverTraceExporter.html">org.apache.nlpcraft.model.opencensus.NCStackdriverTraceExporter</a>
                </td>
                <td><center>Trace</center></td>
            </tr>
            </tbody>
        </table>
        <p>
        <sup><b>NOTE:</b> (*) Grafana works through Prometheus integration.</sup>
        </p>
        <p>
            All built-in exporters are a simple adaptation of the standard OpenCensus <a target=_ href="https://opencensus.io/exporters/supported-exporters/java/">exporters</a>
            to life cycle components for the
            REST server or the data probe. Each exporter has just a few lines of code and you can easily create your own
            exporters for other backends. 
        </p>
    </section>
    <section id="server">
        <h2 class="section-title">Configuring Server</h2>
        <p>
            OpenCensus exporters for the REST server are lifecycle components. Server lifecycle components are the implementations
            of <code>NCServerLifecycle</code> trait. Lifecycle components get callbacks for various stages of the server
            lifecycle and thus can register and unregister "primordial" components such as OpenCensus exporters.
        </p>
        <p>
            REST server lifecycle components are specified in <code>nlpcraft.server.lifecycle</code>
            <a href="server-and-probe.html">configuration</a> property. For example, for file based configuration:
        </p>
        <pre class="brush: js, highlight: [17, 18]">
nlpcraft {
    server {
        ...
        lifecycle = [
            "org.apache.nlpcraft.server.lifecycle.opencensus.NCPrometheusExporter",
            "org.apache.nlpcraft.server.lifecycle.opencensus.NCJaegerExporter"
        ]
        ...
    }
}
        </pre>
        <p>
            Note that each server exporter has default configuration that can be overridden via
            <a href="server-and-probe.html">configuration</a> properties or environment variables. Note also that all
            built-in <b>trace exporters</b> use "always" sampling by default that is suitable
            only for demo or development purposes. For production usage you will need to modify the exporter
            to provide more effective <a target=_ href="https://opencensus.io/tracing/sampling/">sampling strategy</a>.
        </p>
    </section>
    <section id="probe">
        <h2 class="section-title">Configuring Probe</h2>
        <p>
            Just like on the REST server OpenCensus exporters for the data probe are lifecycle components. Probe lifecycle
            components are the implementations
            of <a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/NCProbeLifecycle.html">org.apache.nlpcraft.model.NCLifecycle</a>
            interface. Lifecycle components get callbacks for various stages of the data probe
            lifecycle and thus can register and unregister "primordial" components such as OpenCensus exporters. For example, using
            file based configuration:
        </p>
        <pre class="brush: js, highlight: [14, 15]">
nlpcraft {
    probe {
        ...
        lifecycle = [
            "org.apache.nlpcraft.model.opencensus.NCPrometheusExporter",
            "org.apache.nlpcraft.model.opencensus.NCJaegerExporter"
        ]
        ...
    }
}
        </pre>
        <p>
            Just like with server exporters, each probe exporter has default configuration that can be overridden via
            <a href="server-and-probe.html">configuration</a> properties or environment variables. Note also that all
            built-in <b>trace exporters</b> use "always" sampling by default that is suitable
            only for demo or development purposes. For production usage you will need to modify the exporter
            to provide more effective <a target=_ href="https://opencensus.io/tracing/sampling/">sampling strategy</a>.
            Please consult individual exporter classes:
        </p>
        <ul>
            <li><a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/opencensus/NCJaegerExporter.html">NCJaegerExporter</a></li>
            <li><a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/opencensus/NCZipkinExporter.html">NCZipkinExporter</a></li>
            <li><a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/opencensus/NCPrometheusExporter.html">NCPrometheusExporter</a></li>
            <li><a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/opencensus/NCStackdriverStatsExporter.html">NCStackdriverStatsExporter</a></li>
            <li><a target="javadoc" href="/apis/latest/org/apache/nlpcraft/model/opencensus/NCStackdriverTraceExporter.html">NCStackdriverTraceExporter</a></li>
        </ul>
    </section>
    <section id="traces">
        <h2 class="section-title">Traces</h2>
        <p>
            When traces are exporter, there are two service names used for NLPCraft traces that will appear on UI backends:
        </p>
        <ul>
            <li><code>nlpcraft-server</code> for traces from REST server.</li>
            <li><code>nlpcraft-probe</code> for traces from data probes.</li>
        </ul>
        <p>
            All traces are extensively tagged with various information. Specifically, watch out for <code>srvReqId</code>,
            <code>modelId</code> and <code>userId</code> tags that will help you navigate across traces pertaining to specific
            request, models or users.
        </p>
    </section>
    <section id="metrics">
        <h2 class="section-title">Metrics</h2>
        <p>
            When it comes to metrics each exporter has different set of configuration for either a namespace or
            name prefix on how these metrics will be available on the UI backends:
        </p>
        <ul>
            <li>
                <a target=_ href="https://prometheus.io/">Prometheus</a> and <a target=_ href="https://grafana.com">Grafana</a>
                stats exporters by default use <code>nlpcraft-server</code> and <code>nlpcraft-probe</code> namespaces.
            </li>
            <li>
                <a target=_ href="https://cloud.google.com/stackdriver/">Stackdriver</a> stats exporters use default
                <code>custom.googleapis.com/nlpcraft/server</code> and <code>custom.googleapis.com/nlpcraft/probe</code> metric
                name prefixes.
            </li>
        </ul>
        <p>
            <b>NOTE:</b> default namespaces and service prefixes can be changed via <a href="server-and-probe.html">configuration</a>
            properties or environment variables
        </p>
        <p>
            The following metrics are generated by the REST server:
        </p>
        <table class="gradient-table checks">
            <thead>
            <tr>
                <th>Name <sub>without prefix or namespace</sub></th>
                <th>Description</th>
            </tr>
            </thead>
            <tbody>
            {% for api in site.data.metrics.server %}
            <tr>
                <td>
                    <code>{{api.name}}_latdist_bucket</code><br>
                    <code>{{api.name}}_latdist_count</code><br>
                    <code>{{api.name}}_latdist_sum</code>
                </td>
                <td>Histogram, count and sum of values for the <b>{{api.description}}</b> latencies.</td>
            </tr>
            {% endfor %}
            <tr>
                <td>
                    <code>roundtrip_latdist_bucket</code><br>
                    <code>roundtrip_latdist_count</code><br>
                    <code>roundtrip_latdist_sum</code>
                </td>
                <td>Histogram, count and sum of values for the full server-probe-server round trip latencies.</td>
            </tr>
            </tbody>
        </table>
        <p>
            The following metrics are generated by the data probe:
        </p>
        <table class="gradient-table checks">
            <thead>
            <tr>
                <th>Name <sub>without prefix or namespace</sub></th>
                <th>Description</th>
            </tr>
            </thead>
            <tbody>
            {% for api in site.data.metrics.probe %}
            <tr>
                <td>
                    <code>{{api.name}}_latdist_bucket</code><br>
                    <code>{{api.name}}_latdist_count</code><br>
                    <code>{{api.name}}_latdist_sum</code>
                </td>
                <td>Histogram, count and sum of values for the <b>{{api.description}}</b> latencies.</td>
            </tr>
            {% endfor %}
            </tbody>
        </table>
    </section>
</div>
<div class="col-md-2 third-column">
    <ul class="side-nav">
        <li class="side-nav-title">On This Page</li>
        <li><a href="#overview">Overview</a></li>
        <li><a href="#server">Configuring Server</a></li>
        <li><a href="#probe">Configuring Probe</a></li>
        <li><a href="#traces">Traces</a></li>
        <li><a href="#metrics">Metrics</a></li>
        {% include quick-links.html %}
    </ul>
</div>