This document describe how to understand, develop and contribute plugin.
Span is an important and common concept in distributed tracing system. Learn Span from Google Dapper Paper and OpenTracing
SkyWalking supports OpenTracing and OpenTracing-Java API from 2017. Our Span concepts are similar with the paper and OpenTracing. Also we extend the Span.
There are three types of Span
1.1 EntrySpan EntrySpan represents a service provider, also the endpoint of server side. As an APM system, we are targeting the application servers. So almost all the services and MQ-consumer are EntrySpan(s).
1.2 LocalSpan LocalSpan represents a normal Java method, which does not relate to remote service, neither a MQ producer/consumer nor a service(e.g. HTTP service) provider/consumer.
1.3 ExitSpan ExitSpan represents a client of service or MQ-producer, as named as LeafSpan
at early age of SkyWalking. e.g. accessing DB by JDBC, reading Redis/Memcached are cataloged an ExitSpan.
In order to implement distributed tracing, the trace across process need to be bind, and the context should propagate across the process. That is ContextCarrier's duty.
Here are the steps about how to use ContextCarrier in a A->B
distributed call.
ContextCarrier
at client side.ContextManager#createExitSpan
or use ContextManager#inject
to init the ContextCarrier
.ContextCarrier
into heads(e.g. HTTP HEAD), attachments(e.g. Dubbo RPC framework) or messages(e.g. Kafka)ContextCarrier
propagates to server side by the service call.ContextManager#createEntrySpan
or use ContextManager#extract
to bind the client and server.Let's demonstrate the steps by Apache HTTPComponent client plugin and Tomcat 7 server plugin
span = ContextManager.createExitSpan("/span/operation/name", contextCarrier, "ip:port"); CarrierItem next = contextCarrier.items(); while (next.hasNext()) { next = next.next(); httpRequest.setHeader(next.getHeadKey(), next.getHeadValue()); }
ContextCarrier contextCarrier = new ContextCarrier(); CarrierItem next = contextCarrier.items(); while (next.hasNext()) { next = next.next(); next.setHeadValue(request.getHeader(next.getHeadKey())); } span = ContextManager.createEntrySpan(“/span/operation/name”, contextCarrier);
Besides across process, across thread but in a process need to be supported, because async process(In-memory MQ) and batch process are common in Java. Across process and across thread are similar, because they are both about propagating context. The only difference is that, don't need to serialize for across thread.
Here are the three steps about across thread propagation:
ContextManager#capture
to get the ContextSnapshot object.ContextManager#continued
in sub-thread.ContextManager provides all major and primary APIs.
public static AbstractSpan createEntrySpan(String endpointName, ContextCarrier carrier)
Create EntrySpan by operation name(e.g. service name, uri) and ContextCarrier.
public static AbstractSpan createLocalSpan(String endpointName)
Create LocalSpan by operation name(e.g. full method signature)
public static AbstractSpan createExitSpan(String endpointName, ContextCarrier carrier, String remotePeer)
Create ExitSpan by operation name(e.g. service name, uri) and new ContextCarrier and peer address (e.g. ip+port, hostname+port)
/** * Set the component id, which defines in {@link ComponentsDefine} * * @param component * @return the span for chaining. */ AbstractSpan setComponent(Component component); AbstractSpan setLayer(SpanLayer layer); /** * Set a key:value tag on the Span. * * @return this Span instance, for chaining */ AbstractSpan tag(String key, String value); /** * Record an exception event of the current walltime timestamp. * * @param t any subclass of {@link Throwable}, which occurs in this span. * @return the Span, for chaining */ AbstractSpan log(Throwable t); AbstractSpan errorOccurred(); /** * Record an event at a specific timestamp. * * @param timestamp The explicit timestamp for the log record. * @param event the events * @return the Span, for chaining */ AbstractSpan log(long timestamp, Map<String, ?> event); /** * Sets the string name for the logical operation this span represents. * * @return this Span instance, for chaining */ AbstractSpan setOperationName(String endpointName);
Besides set operation name, tags and logs, two attributes shoule be set, which are component and layer, especially for EntrySpan and ExitSpan
SpanLayer is the catalog of span. Here are 5 values:
Component IDs are defined and reserved by SkyWalking project. For component name/ID extension, please follow Component library definition and extension document.
There is a set of advanced APIs in Span, which work specific for async scenario. When tags, logs, attributes(including end time) of the span needs to set in another thread, you should use these APIs.
/** * The span finish at current tracing context, but the current span is still alive, until {@link #asyncFinish} * called. * * This method must be called<br/> * 1. In original thread(tracing context). * 2. Current span is active span. * * During alive, tags, logs and attributes of the span could be changed, in any thread. * * The execution times of {@link #prepareForAsync} and {@link #asyncFinish()} must match. * * @return the current span */ AbstractSpan prepareForAsync(); /** * Notify the span, it could be finished. * * The execution times of {@link #prepareForAsync} and {@link #asyncFinish()} must match. * * @return the current span */ AbstractSpan asyncFinish();
#prepareForAsync
in original context.ContextManager#stopSpan
in original context when your job in current thread is done.#asyncFinish
in any thread.#prepareForAsync
finished(Judged by count of API execution).The basic method to trace is intercepting a Java method, by using byte code manipulation tech and AOP concept. SkyWalking boxed the byte code manipulation tech and tracing context propagation, so you just need to define the intercept point(a.k.a. aspect pointcut in Spring)
SkyWalking provide two common defines to intercept Contructor, instance method and class method.
ClassInstanceMethodsEnhancePluginDefine
defines Contructor
intercept points and instance method
intercept points.ClassStaticMethodsEnhancePluginDefine
definec class method
intercept points.Of course, you can extend ClassEnhancePluginDefine
to set all intercept points. But it is unusual.
I will demonstrate about how to implement a plugin by extending ClassInstanceMethodsEnhancePluginDefine
protected abstract ClassMatch enhanceClass();
ClassMatch represents how to match the target classes, there are 4 ways:
.
+ class name)Attentions:
ThirdPartyClass.class
in the instrumentation definitions, such as takesArguments(ThirdPartyClass.class)
, or byName(ThirdPartyClass.class.getName())
, because of the fact that ThirdPartyClass
dose not necessarily exist in the target application and this will break the agent; we have import
checks to help on checking this in CI, but it doesn't cover all scenarios of this limitation, so never try to work around this limitation by something like using full-qualified-class-name (FQCN), i.e. takesArguments(full.qualified.ThirdPartyClass.class)
and byName(full.qualified.ThirdPartyClass.class.getName())
will pass the CI check, but are still invalid in the agent codes, Use Full Qualified Class Name String Literature Instead.*.class.getName()
to get the class String name. Recommend you to use literal String. This is for avoiding ClassLoader issues.by*AnnotationMatch
doesn't support the inherited annotations.byHierarchyMatch
, unless it is really necessary. Because using it may trigger intercepting many unexcepted methods, which causes performance issues and concerns.Example:
@Override protected ClassMatch enhanceClassName() { return byName("org.apache.catalina.core.StandardEngineValve"); }
public InstanceMethodsInterceptPoint[] getInstanceMethodsInterceptPoints(); public interface InstanceMethodsInterceptPoint { /** * class instance methods matcher. * * @return methods matcher */ ElementMatcher<MethodDescription> getMethodsMatcher(); /** * @return represents a class name, the class instance must instanceof InstanceMethodsAroundInterceptor. */ String getMethodsInterceptor(); boolean isOverrideArgs(); }
Also use Matcher
to set the target methods. Return true in isOverrideArgs
, if you want to change the argument ref in interceptor.
The following sections will tell you how to implement the interceptor.
tomcat-7.x/8.x=TomcatInstrumentation
As an interceptor for an instance method, the interceptor implements org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstanceMethodsAroundInterceptor
/** * A interceptor, which intercept method's invocation. The target methods will be defined in {@link * ClassEnhancePluginDefine}'s subclass, most likely in {@link ClassInstanceMethodsEnhancePluginDefine} */ public interface InstanceMethodsAroundInterceptor { /** * called before target method invocation. * * @param result change this result, if you want to truncate the method. * @throws Throwable */ void beforeMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes, MethodInterceptResult result) throws Throwable; /** * called after target method invocation. Even method's invocation triggers an exception. * * @param ret the method's original return value. * @return the method's actual return value. * @throws Throwable */ Object afterMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes, Object ret) throws Throwable; /** * called when occur exception. * * @param t the exception occur. */ void handleMethodException(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes, Throwable t); }
Use the core APIs in before, after and exception handle stages.
SkyWalking has packaged the bootstrap instrumentation in the agent core. It is easy to open by declaring it in the Instrumentation definition.
Override the public boolean isBootstrapInstrumentation()
and return true. Such as
public class URLInstrumentation extends ClassEnhancePluginDefine { private static String CLASS_NAME = "java.net.URL"; @Override protected ClassMatch enhanceClass() { return byName(CLASS_NAME); } @Override public ConstructorInterceptPoint[] getConstructorsInterceptPoints() { return new ConstructorInterceptPoint[] { new ConstructorInterceptPoint() { @Override public ElementMatcher<MethodDescription> getConstructorMatcher() { return any(); } @Override public String getConstructorInterceptor() { return "org.apache.skywalking.apm.plugin.jre.httpurlconnection.Interceptor2"; } } }; } @Override public InstanceMethodsInterceptPoint[] getInstanceMethodsInterceptPoints() { return new InstanceMethodsInterceptPoint[0]; } @Override public StaticMethodsInterceptPoint[] getStaticMethodsInterceptPoints() { return new StaticMethodsInterceptPoint[0]; } @Override public boolean isBootstrapInstrumentation() { return true; } }
NOTICE, doing bootstrap instrumentation should only happen in necessary, but mostly it effect the JRE core(rt.jar), and could make very unexpected result or side effect.
We are welcome everyone to contribute plugins.
Please follow there steps:
apm-sniffer/apm-sdk-plugin
or apm-sniffer/optional-plugins
, and the name should include supported library name and versionshow to write the plugin test case
from this doc