blob: 2afe9915646168501f8323b6bf2422063570689c [file] [log] [blame]
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
See the License for the specific language governing permissions and
limitations under the License.
<title>Accumulo Distributed Tracing</title>
<link rel='stylesheet' type='text/css' href='documentation.css' media='screen'/>
<h1>Apache Accumulo Documentation : Distributed Tracing</h1>
<p>It can be difficult to determine why some operations are taking longer than expected. For example, you may be looking up items with
very low latency, but sometimes the lookups take much longer. Determining the cause of the delay is difficult because the system is
distributed, and the typical lookup is fast.</p>
<p>To provide insight into what accumulo is doing during your scan, you can turn on tracing before you do your operation:</p>
import org.apache.accumulo.core.trace.DistributedTrace;
import org.apache.htrace.Sampler;
import org.apache.htrace.Trace;
import org.apache.htrace.TraceScope;
DistributedTrace.enable(hostname, "myApplication");
TraceScope scanTrace = Trace.startSpan("client:scan", Sampler.ALWAYS);
BatchScanner scanner = conn.createBatchScanner(...);
// Configure your scanner
for (Entry<Key, Value> entry : scanner) {
<p>Accumulo has been instrumented to record the time that various operations take when tracing is turned on. The fact that tracing is
enabled follows all the requests made on behalf of the user throughout the distributed infrastructure of accumulo, and across all
threads of execution.</p>
<p>These time spans will be inserted into the trace accumulo table. You can browse recent traces from the accumulo monitor page.
You can also read the trace table directly.</p>
<p>Tracing is supported in the shell. For example:
root@test&gt; createtable test
root@test test&gt; insert a b c d
root@test test&gt; trace on
root@test test&gt; scan
a b:c [] d
root@test test&gt; trace off
Waiting for trace information
Waiting for trace information
Waiting for trace information
Trace started at 2011/03/16 09:20:31.387
Time Start Service@Location Name
3355+0 shell@host2 shell:root
1+1 shell@host2 client:listUsers
1+1434 tserver@host2 getUserAuthorizations
1+1434 shell@host2 client:getUserAuthorizations
10+1550 shell@host2 scan
9+1551 shell@host2 scan:location
7+1552 shell@host2 client:startScan
6+1553 tserver@host2 startScan
5+1553 tserver@host2 tablet read ahead 11
1+1559 shell@host2 client:closeScan
1+1561 shell@host2 client:listUsers
<p>Here we can see that the shell is getting the list of users (which is used for tab-completion) after every command. While
unexpected, it is a fast operation. In fact, all the requests are very fast, and most of the time is spent waiting for the user
to make a request while tracing is turned on.</p>
<p>Spans are added to the trace table asynchronously. The user may have to wait several seconds for all requests to complete before the
trace information is complete.</p>
<p>You can extract the trace data out of the trace table. Each span is a stored as a column in a row named for the trace id.
The following code will print out a trace:</p>
String table = AccumuloConfiguration.getSystemConfiguration().get(Property.TRACE_TABLE);
Scanner scanner = shellState.connector.createScanner(table, auths);
scanner.setRange(new Range(new Text(Long.toHexString(scanTrace.traceId()))));
TraceDump.printTrace(scanner, new Printer() {
void print(String line) {