<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<html>
<head>
<title>Accumulo Distributed Tracing</title>
<link rel='stylesheet' type='text/css' href='documentation.css' media='screen'/>
</head>
<body>

<h1>Apache Accumulo Documentation : Distributed Tracing</h1>

<p>It can be difficult to determine why some operations are taking longer than expected. For example, you may be looking up items with
very low latency, but sometimes the lookups take much longer. Determining the cause of the delay is difficult because the system is
distributed, and the typical lookup is fast.</p>

<p>To provide insight into what accumulo is doing during your scan, you can turn on tracing before you do your operation:</p>

<pre>
   import org.apache.accumulo.core.trace.DistributedTrace;
   import org.apache.htrace.Sampler;
   import org.apache.htrace.Trace;
   import org.apache.htrace.TraceScope;
   ...

   DistributedTrace.enable(hostname, "myApplication");
   TraceScope scanTrace = Trace.startSpan("client:scan", Sampler.ALWAYS);
   BatchScanner scanner = conn.createBatchScanner(...);
   // Configure your scanner
   for (Entry<Key, Value> entry : scanner) {
   }
   scanTrace.close();
   DistributedTrace.disableTracing();
</pre>


<p>Accumulo has been instrumented to record the time that various operations take when tracing is turned on. The fact that tracing is
enabled follows all the requests made on behalf of the user throughout the distributed infrastructure of accumulo, and across all
threads of execution.</p>

<p>These time spans will be inserted into the trace accumulo table. You can browse recent traces from the accumulo monitor page.
You can also read the trace table directly.</p>

<p>Tracing is supported in the shell. For example:

<pre>
root@test&gt; createtable test
root@test test&gt; insert a b c d
root@test test&gt; trace on
root@test test&gt; scan
a b:c []    d
root@test test&gt; trace off
Waiting for trace information
Waiting for trace information
Waiting for trace information
Trace started at 2011/03/16 09:20:31.387
Time  Start  Service@Location       Name
 3355+0      shell@host2 shell:root
    1+1        shell@host2 client:listUsers
    1+1434     tserver@host2 getUserAuthorizations
    1+1434     shell@host2 client:getUserAuthorizations
   10+1550     shell@host2 scan
    9+1551       shell@host2 scan:location
    7+1552         shell@host2 client:startScan
    6+1553         tserver@host2 startScan
    5+1553           tserver@host2 tablet read ahead 11
    1+1559         shell@host2 client:closeScan
    1+1561     shell@host2 client:listUsers
</pre>

<p>Here we can see that the shell is getting the list of users (which is used for tab-completion) after every command. While
unexpected, it is a fast operation. In fact, all the requests are very fast, and most of the time is spent waiting for the user
to make a request while tracing is turned on.</p>

<p>Spans are added to the trace table asynchronously. The user may have to wait several seconds for all requests to complete before the
trace information is complete.</p>

<p>You can extract the trace data out of the trace table. Each span is a stored as a column in a row named for the trace id.
The following code will print out a trace:</p>

<pre>
String table = AccumuloConfiguration.getSystemConfiguration().get(Property.TRACE_TABLE);
Scanner scanner = shellState.connector.createScanner(table, auths);
scanner.setRange(new Range(new Text(Long.toHexString(scanTrace.traceId()))));
TraceDump.printTrace(scanner, new Printer() {
    void print(String line) {
        System.out.println(line);
    }
});
</pre>

</body>
</html>