| // Licensed to the Apache Software Foundation (ASF) under one or more |
| // contributor license agreements. See the NOTICE file distributed with |
| // this work for additional information regarding copyright ownership. |
| // The ASF licenses this file to You under the Apache License, Version 2.0 |
| // (the "License"); you may not use this file except in compliance with |
| // the License. You may obtain a copy of the License at |
| // |
| // http://www.apache.org/licenses/LICENSE-2.0 |
| // |
| // Unless required by applicable law or agreed to in writing, software |
| // distributed under the License is distributed on an "AS IS" BASIS, |
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| // See the License for the specific language governing permissions and |
| // limitations under the License. |
| = Tracing |
| |
| :javaFile: {javaCodeDir}/Tracing.java |
| |
| WARNING: This feature is experimental. |
| |
| A number of APIs in Ignite are instrumented for tracing with OpenCensus. |
| You can collect distributed traces of various tasks executed in your cluster and use this information to diagnose latency problems. |
| |
| We suggest you get familiar with OpenCensus tracing documentation before reading this chapter: https://opencensus.io/tracing/[^]. |
| |
| The following Ignite APIs are instrumented for tracing: |
| |
| * Discovery |
| * Communication |
| * Exchange |
| * Transactions |
| |
| |
| To view traces, you must export them into an external system. |
| You can use one of the OpenCensus exporters or write your own, but in any case, you will have to write code that registers an exporter in Ignite. |
| Refer to <<Exporting Traces>> for details. |
| |
| |
| == Configuring Tracing |
| |
| Enable OpenCensus tracing in the node configuration. All nodes in the cluster must use the same tracing configuration. |
| |
| [tabs] |
| -- |
| tab:XML[] |
| [source, xml] |
| ---- |
| include::code-snippets/xml/tracing.xml[tags=ignite-config;!discovery, indent=0] |
| ---- |
| |
| tab:Java[] |
| [source, java] |
| ---- |
| include::{javaFile}[tags=config, indent=0] |
| ---- |
| tab:C#/.NET[] |
| |
| tab:C++[unsupported] |
| -- |
| |
| |
| == Enabling Trace Sampling |
| |
| When you start your cluster with the above configuration, Ignite does not collect traces. |
| You have to enable trace sampling for a specific API at runtime. |
| You can turn trace sampling on and off at will, for example, only for the period when you are troubleshooting a problem. |
| |
| You can do this in two ways: |
| |
| * via the control script from the command line |
| * programmatically |
| |
| Traces are collected at a given probabilistic sampling rate. |
| The rate is specified as a value between 0.0 and 1.0 inclusive: `0` means no sampling, `1` means always sampling. |
| |
| When the sampling rate is set to a value greater than 0, Ignite collects traces. |
| To disable trace collection, set the sampling rate to 0. |
| |
| The following sections describe the two ways of enabling trace sampling. |
| |
| === Using Control Script |
| |
| Go to the `{IGNITE_HOME}/bin` directory of your Ignite installation. |
| Enable experimental commands in the control script: |
| |
| [source, shell] |
| ---- |
| export IGNITE_ENABLE_EXPERIMENTAL_COMMAND=true |
| ---- |
| |
| Enable tracing for a specific API: |
| |
| [source, shell] |
| ---- |
| ./control.sh --tracing-configuration set --scope TX --sampling-rate 1 |
| ---- |
| |
| Refer to the link:control-script#tracing-configuration[Control Script] sections for the list of all parameters. |
| |
| === Programmatically |
| |
| Once you start the node, you can enable trace sampling as follows: |
| |
| [source, java] |
| ---- |
| include::{javaFile}[tags=enable-sampling, indent=0] |
| ---- |
| |
| |
| The `--scope` parameter specifies the API you want to trace. |
| The following APIs are instrumented for tracing: |
| |
| * `DISCOVERY` — discovery events |
| * `EXCHANGE` — exchange events |
| * `COMMUNICATION` — communication events |
| * `TX` — transactions |
| |
| The `--sampling-rate` is the probabilistic sampling rate, a number between `0` and `1`: |
| |
| * `0` means no sampling, |
| * `1` means always sampling. |
| |
| |
| == Exporting Traces |
| |
| To view traces, you need to export them to an external backend using one of the available exporters. |
| OpenCensus supports a number of exporters out-of-the-box, and you can write a custom one. |
| Refer to the link:https://opencensus.io/exporters/[OpenCensus Exporters^] for details. |
| |
| In this section, we will show how to export traces to link:https://zipkin.io[Zipkin^]. |
| |
| . Follow link:https://zipkin.io/pages/quickstart.html[this guide^] to launch Zipkin on your machine. |
| . Register `ZipkinTraceExporter` in the application where you start Ignite: |
| + |
| -- |
| [source, java] |
| ---- |
| include::{javaFile}[tags=export-to-zipkin, indent=0] |
| ---- |
| -- |
| |
| |
| . Open http://localhost:9411/zipkin[^] in your browser and click the search icon. |
| + |
| -- |
| This is what a trace of the transaction looks like: |
| |
| image::images/trace_in_zipkin.png[] |
| -- |
| |
| == Analyzing Trace Data |
| |
| A trace is recorded information about the execution of a specific event. |
| Each trace consists of a tree of _spans_. |
| A span is an individual unit of work performed by the system in order to process the event. |
| |
| Because of the distributed nature of Ignite, an operation usually involves multiple nodes. |
| Therefore, a trace can include spans from multiple nodes. |
| Each span always contains the information about the node where the corresponding operation was executed. |
| |
| In the image of the transaction trace presented above, you can see that the trace contains the spans associated with the following operations: |
| |
| * acquire locks (`transactions.colocated.lock.map`), |
| * get (`transactions.near.enlist.read`), |
| * put (`transactions.near.enlist.write`), |
| * commit (`transactions.commit`), and |
| * close (`transactions.close`). |
| |
| The commit operation, in turn, consists of two operations: prepare and finish. |
| |
| You can click on each span to view the annotations and tags attached to it. |
| |
| |
| image::images/span.png[Span] |
| |
| //// |
| TODO: describe annotations and tags |
| === Annotations |
| |
| === Tags |
| |
| The `node.id` and `node.consistentId` are the ID and consistent ID of the node where the root operation started. |
| //// |