| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one or more |
| contributor license agreements. See the NOTICE file distributed with |
| this work for additional information regarding copyright ownership. |
| The ASF licenses this file to You under the Apache License, Version 2.0 |
| (the "License"); you may not use this file except in compliance with |
| the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, software |
| distributed under the License is distributed on an "AS IS" BASIS, |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| See the License for the specific language governing permissions and |
| limitations under the License. |
| --> |
| <html> |
| <head> |
| <title>Accumulo Distributed Tracing</title> |
| <link rel='stylesheet' type='text/css' href='documentation.css' media='screen'/> |
| </head> |
| <body> |
| |
| <h1>Apache Accumulo Documentation : Distributed Tracing</h1> |
| |
| <p>It can be difficult to determine why some operations are taking longer than expected. For example, you may be looking up items with |
| very low latency, but sometimes the lookups take much longer. Determining the cause of the delay is difficult because the system is |
| distributed, and the typical lookup is fast.</p> |
| |
| <p>To provide insight into what accumulo is doing during your scan, you can turn on tracing before you do your operation:</p> |
| |
| <pre> |
| import org.apache.accumulo.core.trace.DistributedTrace; |
| import org.apache.htrace.Sampler; |
| import org.apache.htrace.Trace; |
| import org.apache.htrace.TraceScope; |
| ... |
| |
| DistributedTrace.enable(hostname, "myApplication"); |
| TraceScope scanTrace = Trace.startSpan("client:scan", Sampler.ALWAYS); |
| BatchScanner scanner = conn.createBatchScanner(...); |
| // Configure your scanner |
| for (Entry<Key, Value> entry : scanner) { |
| } |
| scanTrace.close(); |
| DistributedTrace.disableTracing(); |
| </pre> |
| |
| |
| <p>Accumulo has been instrumented to record the time that various operations take when tracing is turned on. The fact that tracing is |
| enabled follows all the requests made on behalf of the user throughout the distributed infrastructure of accumulo, and across all |
| threads of execution.</p> |
| |
| <p>These time spans will be inserted into the trace accumulo table. You can browse recent traces from the accumulo monitor page. |
| You can also read the trace table directly.</p> |
| |
| <p>Tracing is supported in the shell. For example: |
| |
| <pre> |
| root@test> createtable test |
| root@test test> insert a b c d |
| root@test test> trace on |
| root@test test> scan |
| a b:c [] d |
| root@test test> trace off |
| Waiting for trace information |
| Waiting for trace information |
| Waiting for trace information |
| Trace started at 2011/03/16 09:20:31.387 |
| Time Start Service@Location Name |
| 3355+0 shell@host2 shell:root |
| 1+1 shell@host2 client:listUsers |
| 1+1434 tserver@host2 getUserAuthorizations |
| 1+1434 shell@host2 client:getUserAuthorizations |
| 10+1550 shell@host2 scan |
| 9+1551 shell@host2 scan:location |
| 7+1552 shell@host2 client:startScan |
| 6+1553 tserver@host2 startScan |
| 5+1553 tserver@host2 tablet read ahead 11 |
| 1+1559 shell@host2 client:closeScan |
| 1+1561 shell@host2 client:listUsers |
| </pre> |
| |
| <p>Here we can see that the shell is getting the list of users (which is used for tab-completion) after every command. While |
| unexpected, it is a fast operation. In fact, all the requests are very fast, and most of the time is spent waiting for the user |
| to make a request while tracing is turned on.</p> |
| |
| <p>Spans are added to the trace table asynchronously. The user may have to wait several seconds for all requests to complete before the |
| trace information is complete.</p> |
| |
| <p>You can extract the trace data out of the trace table. Each span is a stored as a column in a row named for the trace id. |
| The following code will print out a trace:</p> |
| |
| <pre> |
| String table = AccumuloConfiguration.getSystemConfiguration().get(Property.TRACE_TABLE); |
| Scanner scanner = shellState.connector.createScanner(table, auths); |
| scanner.setRange(new Range(new Text(Long.toHexString(scanTrace.traceId())))); |
| TraceDump.printTrace(scanner, new Printer() { |
| void print(String line) { |
| System.out.println(line); |
| } |
| }); |
| </pre> |
| |
| </body> |
| </html> |
