This application provides an interface to opentracing compatible tracing systems.
Traces in OpenTracing are defined implicitly by their Spans. In particular, a Trace can be thought of as a directed acyclic graph (DAG) of Spans, where the edges between Spans are called References.
Each Span encapsulates the following state:
Every trace is identified by unique trace_id. Every trace includes zero or more tracing spans which are identified by a span id.
Jaeger is a distributed tracing system released as open source by Uber Technologies. It is one of implementations of open tracing specification. Jaeger supports Trace detail view where a single trace is represented as a tree of tracing span with detailed timing information about every span. In order to make this feature work all tracing spans should form a lineage from the same root span.
Every operation has unique identifier. Example identifiers are:
Tracing begins with a root span that can be filtered based on a set of configurable rules. When the root trace is created these rules are applied to see if the trace should be generated and logged. If a trace is disabled due to filtering then no trace data is generated.
The span lifecycle is controled by
ctrace:start_span
ctrace:finish_span
ctrace:with_span
The instrumentation can add tags and logs to a span.
Example of instrumentation:
ctrace:with_span('database.read', #{'db.name' => <<>>}, fun() -> ctrace:tag(#{ peer => Peer, 'http.method' => Method, nonce => Nonce, 'http.url' => Path, 'span.kind' => <<"server">>, component => <<"couchdb.chttpd">> }), ctrace:log(#{ field0 => "value0" }) handle_request(HttpReq) end),
As you can see the ctrace:with_span/3
function receives a function which wraps the operation we wanted to trace:
ctrace:tag/1
to add new tags to the spanctrace:log/1
add log event to the spanThere are some informative functions as well:
ctrace:refs/0
- returns all other spans we have references from the currentctrace:operation_name/0
- returns operation name for the current spanctrace:trace_id/0
- returns trace id for the current spanctrace:span_id/0
- returns span id for the current span<app>_httpd_handlers:handler_info/1
as needed to have operation ids. (We as community might need to work on naming conventions)Traces are configured using standard CouchDB ini file based configuration. There is a global toggle [tracing] enabled = true | false
that switches tracing on or off completely. The [tracing]
section also includes configuration for where to send trace data. There are two reporters which we support.
The thrift over udp reporter (this is the default) has following configuration options:
The thrift over http has following options
An example of [tracing]
section
[tracing] enabled = true thrift_format = compact ; compact | binary agent_host = 127.0.0.1 agent_port = 6831 app_name = couchdb ; Value to use for the `location.application` tag
In the [tracing.filters]
section we can define a set of rules for whether to include a trace. Keys are the operation name of the root span and values are a simple DSL for whether to include the given span based on its tags. See below for a more thorough description of the DSL. The all
key is special and is used when no other filter matches a given operation. If the all
key is not present then ctrace behaves as if it were defined as (#{}) -> false
. I.e., any trace that doesn't have a configuration entry is not generated and logged.
[tracing.filters] ; trace all events ; all = (#{}) -> true ; trace all events with X-B3-... headers ; all = (#{external := External}) when External == true -> true ; database-info.read = (#{'http.method' := Method}) when Method == 'GET' -> true ; view.build = (#{'view.name' := Name}) when Name == "foo" -> 0.25
<operation_name> = ( #{<[arguments]>} ) when <[conditions]> -> <[actions]>
Where:
<tag_or_field_name> := <variable_name>
report
<[condition]>
| <[condition]> <[operator]> <[condition]>
<variable_name> <[operator]> <value>
| <[guard_function]>(<[variable_name]>)
variable_name
- lowercase name without special charactersis_atom
is_float
is_integer
is_list
is_number
is_pid
is_port
is_reference
is_tuple
is_map
is_binary
is_function
element
- element(n, tuple)
abs
hd
- return head of the listlength
map_get
map_size
round
node
size
- returns size of the tuplebit_size
- returns number of bits in binarybyte_size
- returns number of bytes in binarytl
- return tail of a listtrunc
self
not
and
- evaluates both expressionsandalso
- evaluates second only when first is trueor
- evaluates both expressionsorelse
- evaluates second only when first is falsexor
+
-
*
div
rem
band
- bitwise ANDbor
- bitwise ORbxor
- bitwise XORbnot
- bitwise NOTbsl
- arithmetic bitshift leftbsr
- bitshift right>
>=
<
=<
=:=
==
=/=
/=
- not equalIn order to correlate spans across multiple systems the information about parent span can be passed via headers. Currently the chttpd application is responsible for extracting and parsing the header. The ctrace application provides following facilities to enable this use case:
{root, RootSpan}
option for ctrace:start_span/2
ctrace:external_span/3
to convert references to a root spanThe span references could be set either via b3
header of via individual headers. In case when individual headers are used the following set of headers is supported:
Alternatively a single b3
header could be used. It has to be in the following format:
b3={TraceId}-{SpanId}-{SamplingState}-{ParentSpanId}
Where SamplingState is either 0
or 1
. However we ignore the value.
Note: We only support 128 bit TraceId's.
Here we provide a list frequently used commands useful while working on this application.
make setup-eunit make && ERL_LIBS=`pwd`/src BUILDDIR=`pwd` mix test --trace src/chttpd/test/exunit/ src/ctrace/test/exunit/
make && ERL_LIBS=`pwd`/src BUILDDIR=`pwd` mix test --trace src/chttpd/test/exunit/ctrace_context_test.exs:59
make && ERL_LIBS=`pwd`/src BUILDDIR=`pwd` mix test --failed --trace src/chttpd/test/exunit/ src/ctrace/test/exunit/
docker run -d --net fdb-core --name jaeger.local -p 6831:6831/udp -p 16686:16686 jaegertracing/all-in-one:1.14
If Docker isn't your cup of tea, the Jaeger project also provides prebuilt binaries that can be downloaded. On macOS we can easily setup a development Jaeger instance by running the prebuilt jaeger-all-in-one
binary without any arguments.