Setup java agent

  1. Agent is available for JDK 8 - 14 in 7.x releases. JDK 1.6 - JDK 12 are supported in all 6.x releases NOTICE¹
  2. Find agent folder in SkyWalking release package
  3. Set agent.service_name in config/agent.config. Could be any String in English.
  4. Set collector.backend_service in config/agent.config. Default point to 127.0.0.1:11800, only works for local backend.
  5. Add -javaagent:/path/to/skywalking-package/agent/skywalking-agent.jar to JVM argument. And make sure to add it before the -jar argument.

The agent release dist is included in Apache official release. New agent package looks like this.

+-- agent
    +-- activations
         apm-toolkit-log4j-1.x-activation.jar
         apm-toolkit-log4j-2.x-activation.jar
         apm-toolkit-logback-1.x-activation.jar
         ...
    +-- config
         agent.config  
    +-- plugins
         apm-dubbo-plugin.jar
         apm-feign-default-http-9.x.jar
         apm-httpClient-4.x-plugin.jar
         .....
    +-- optional-plugins
         apm-gson-2.x-plugin.jar
         .....
    +-- bootstrap-plugins
         jdk-http-plugin.jar
         .....
    +-- logs
    skywalking-agent.jar
  • Start your application.

Supported middleware, framework and library

SkyWalking agent has supported various middlewares, frameworks and libraries. Read supported list to get them and supported version. If the plugin is in Optional² catalog, go to optional plugins section to learn how to active it.

Advanced features

  • All plugins are in /plugins folder. The plugin jar is active when it is in there. Remove the plugin jar, it disabled.
  • The default logging output folder is /logs.

Install javaagent FAQs

  • Linux Tomcat 7, Tomcat 8, Tomcat 9
    Change the first line of tomcat/bin/catalina.sh.
CATALINA_OPTS="$CATALINA_OPTS -javaagent:/path/to/skywalking-agent/skywalking-agent.jar"; export CATALINA_OPTS
  • Windows Tomcat 7, Tomcat 8, Tomcat 9
    Change the first line of tomcat/bin/catalina.bat.
set "CATALINA_OPTS=-javaagent:/path/to/skywalking-agent/skywalking-agent.jar"
  • JAR file
    Add -javaagent argument to command line in which you start your app. eg:
java -javaagent:/path/to/skywalking-agent/skywalking-agent.jar -jar yourApp.jar
  • Jetty
    Modify jetty.sh, add -javaagent argument to command line in which you start your app. eg:
export JAVA_OPTIONS="${JAVA_OPTIONS} -javaagent:/path/to/skywalking-agent/skywalking-agent.jar"

Table of Agent Configuration Properties

This is the properties list supported in agent/config/agent.config.

property keyDescriptionDefault
agent.namespaceNamespace isolates headers in cross process propagation. The HEADER name will be HeaderName:Namespace.Not set
agent.service_nameThe service name to represent a logic group providing the same capabilities/logic. Suggestion: set a unique name for every logic service group, service instance nodes share the same code, Max length is 50(UTF-8 char). Optional, once service_name follows <group name>::<logic name> format, OAP server assigns the group name to the service metadata.Your_ApplicationName
agent.sample_n_per_3_secsNegative or zero means off, by default.SAMPLE_N_PER_3_SECS means sampling N TraceSegment in 3 seconds tops.Not set
agent.authenticationAuthentication active is based on backend setting, see application.yml for more details.For most scenarios, this needs backend extensions, only basic match auth provided in default implementation.Not set
agent.span_limit_per_segmentThe max number of spans in a single segment. Through this config item, SkyWalking keep your application memory cost estimated.300
agent.ignore_suffixIf the operation name of the first span is included in this set, this segment should be ignored.Not set
agent.is_open_debugging_classIf true, skywalking agent will save all instrumented classes files in /debugging folder. SkyWalking team may ask for these files in order to resolve compatible problem.Not set
agent.is_cache_enhanced_classIf true, SkyWalking agent will cache all instrumented classes files to memory or disk files (decided by class cache mode), allow another java agent to enhance those classes that enhanced by SkyWalking agent. To use some Java diagnostic tools (such as BTrace, Arthas) to diagnose applications or add a custom java agent to enhance classes, you need to enable this feature. Read this FAQ for more detailsfalse
agent.class_cache_modeThe instrumented classes cache mode: MEMORY or FILE. MEMORY: cache class bytes to memory, if instrumented classes is too many or too large, it may take up more memory. FILE: cache class bytes in /class-cache folder, automatically clean up cached class files when the application exits.MEMORY
agent.instance_nameInstance name is the identity of an instance, should be unique in the service. If empty, SkyWalking agent will generate an 32-bit uuid. Default, use UUID@hostname as the instance name. Max length is 50(UTF-8 char)""
agent.instance_properties[key]=valueAdd service instance custom properties.Not set
agent.cause_exception_depthHow depth the agent goes, when log all cause exceptions.5
agent.force_reconnection_period Force reconnection period of grpc, based on grpc_channel_check_interval.1
agent.operation_name_threshold The operationName max length, setting this value > 190 is not recommended.150
agent.keep_tracingKeep tracing even the backend is not available if this value is true.false
agent.force_tlsForce open TLS for gRPC channel if this value is true.false
osinfo.ipv4_list_sizeLimit the length of the ipv4 list size.10
collector.grpc_channel_check_intervalgrpc channel status check interval.30
collector.heartbeat_periodagent heartbeat report period. Unit, second.30
collector.properties_report_period_factorThe agent sends the instance properties to the backend every collector.heartbeat_period * collector.properties_report_period_factor seconds10
collector.backend_serviceCollector SkyWalking trace receiver service addresses.127.0.0.1:11800
collector.grpc_upstream_timeoutHow long grpc client will timeout in sending data to upstream. Unit is second.30 seconds
collector.get_profile_task_intervalSniffer get profile task list interval.20
collector.get_agent_dynamic_config_intervalSniffer get agent dynamic config interval20
collector.dns_period_resolve_activeIf true, skywalking agent will enable periodically resolving DNS to update receiver service addresses.false
logging.levelLog level: TRACE, DEBUG, INFO, WARN, ERROR, OFF. Default is info.INFO
logging.file_nameLog file name.skywalking-api.log
logging.outputLog output. Default is FILE. Use CONSOLE means output to stdout.FILE
logging.dirLog files directory. Default is blank string, means, use "{theSkywalkingAgentJarDir}/logs " to output logs. {theSkywalkingAgentJarDir} is the directory where the skywalking agent jar file is located""
logging.resolverLogger resolver: PATTERN or JSON. The default is PATTERN, which uses logging.pattern to print traditional text logs. JSON resolver prints logs in JSON format.PATTERN
logging.pattern Logging format. There are all conversion specifiers:
  * %level means log level.
  * %timestamp means now of time with format yyyy-MM-dd HH:mm:ss:SSS.
  * %thread means name of current thread.
  * %msg means some message which user logged.
  * %class means SimpleName of TargetClass.
  * %throwable means a throwable which user called.
  * %agent_name means agent.service_name. Only apply to the PatternLogger.
%level %timestamp %thread %class : %msg %throwable
logging.max_file_sizeThe max size of log file. If the size is bigger than this, archive the current file, and write into a new file.300 * 1024 * 1024
logging.max_history_filesThe max history log files. When rollover happened, if log files exceed this number,then the oldest file will be delete. Negative or zero means off, by default.-1
statuscheck.ignored_exceptionsListed exceptions would not be treated as an error. Because in some codes, the exception is being used as a way of controlling business flow.""
statuscheck.max_recursive_depthThe max recursive depth when checking the exception traced by the agent. Typically, we don't recommend setting this more than 10, which could cause a performance issue. Negative value and 0 would be ignored, which means all exceptions would make the span tagged in error status.1
correlation.element_max_numberMax element count in the correlation context.3
correlation.value_max_lengthMax value length of each element.128
correlation.auto_tag_keysTag the span by the key/value in the correlation context, when the keys listed here exist.""
jvm.buffer_sizeThe buffer size of collected JVM info.60 * 10
buffer.channel_sizeThe buffer channel size.5
buffer.buffer_sizeThe buffer size.300
profile.activeIf true, skywalking agent will enable profile when user create a new profile task. Otherwise disable profile.true
profile.max_parallelParallel monitor segment count5
profile.durationMax monitor segment time(minutes), if current segment monitor time out of limit, then stop it.10
profile.dump_max_stack_depthMax dump thread stack depth500
profile.snapshot_transport_buffer_sizeSnapshot transport to backend buffer size50
meter.activeIf true, the agent collects and reports metrics to the backend.true
meter.report_intervalReport meters interval. The unit is second20
meter.max_meter_sizeMax size of the meter pool500
plugin.mountMount the specific folders of the plugins. Plugins in mounted folders would work.plugins,activations
plugin.peer_max_length Peer maximum description limit.200
plugin.exclude_plugins Exclude some plugins define in plugins dir.Plugin names is defined in Agent plugin list""
plugin.mongodb.trace_paramIf true, trace all the parameters in MongoDB access, default is false. Only trace the operation, not include parameters.false
plugin.mongodb.filter_length_limitIf set to positive number, the WriteRequest.params would be truncated to this length, otherwise it would be completely saved, which may cause performance problem.256
plugin.elasticsearch.trace_dslIf true, trace all the DSL(Domain Specific Language) in ElasticSearch access, default is false.false
plugin.springmvc.use_qualified_name_as_endpoint_nameIf true, the fully qualified method name will be used as the endpoint name instead of the request URL, default is false.false
plugin.toolit.use_qualified_name_as_operation_nameIf true, the fully qualified method name will be used as the operation name instead of the given operation name, default is false.false
plugin.jdbc.trace_sql_parametersIf set to true, the parameters of the sql (typically java.sql.PreparedStatement) would be collected.false
plugin.jdbc.sql_parameters_max_lengthIf set to positive number, the db.sql.parameters would be truncated to this length, otherwise it would be completely saved, which may cause performance problem.512
plugin.jdbc.sql_body_max_lengthIf set to positive number, the db.statement would be truncated to this length, otherwise it would be completely saved, which may cause performance problem.2048
plugin.solrj.trace_statementIf true, trace all the query parameters(include deleteByIds and deleteByQuery) in Solr query request, default is false.false
plugin.solrj.trace_ops_paramsIf true, trace all the operation parameters in Solr request, default is false.false
plugin.light4j.trace_handler_chainIf true, trace all middleware/business handlers that are part of the Light4J handler chain for a request.false
plugin.opgroup.*Support operation name customize group rules in different plugins. Read Group rule supported pluginsNot set
plugin.springtransaction.simplify_transaction_definition_nameIf true, the transaction definition name will be simplified.false
plugin.jdkthreading.threading_class_prefixesThreading classes (java.lang.Runnable and java.util.concurrent.Callable) and their subclasses, including anonymous inner classes whose name match any one of the THREADING_CLASS_PREFIXES (splitted by ,) will be instrumented, make sure to only specify as narrow prefixes as what you're expecting to instrument, (java. and javax. will be ignored due to safety issues)Not set
plugin.tomcat.collect_http_paramsThis config item controls that whether the Tomcat plugin should collect the parameters of the request. Also, activate implicitly in the profiled trace.false
plugin.springmvc.collect_http_paramsThis config item controls that whether the SpringMVC plugin should collect the parameters of the request, when your Spring application is based on Tomcat, consider only setting either plugin.tomcat.collect_http_params or plugin.springmvc.collect_http_params. Also, activate implicitly in the profiled trace.false
plugin.httpclient.collect_http_paramsThis config item controls that whether the HttpClient plugin should collect the parameters of the requestfalse
plugin.http.http_params_length_thresholdWhen COLLECT_HTTP_PARAMS is enabled, how many characters to keep and send to the OAP backend, use negative values to keep and send the complete parameters, NB. this config item is added for the sake of performance.1024
plugin.http.http_headers_length_thresholdWhen include_http_headers declares header names, this threshold controls the length limitation of all header values. use negative values to keep and send the complete headers. Note. this config item is added for the sake of performance.2048
plugin.http.include_http_headersSet the header names, which should be collected by the plugin. Header name must follow javax.servlet.http definition. Multiple names should be split by comma.``(No header would be collected)
plugin.feign.collect_request_bodyThis config item controls that whether the Feign plugin should collect the http body of the request.false
plugin.feign.filter_length_limitWhen COLLECT_REQUEST_BODY is enabled, how many characters to keep and send to the OAP backend, use negative values to keep and send the complete body.1024
plugin.feign.supported_content_types_prefixWhen COLLECT_REQUEST_BODY is enabled and content-type start with SUPPORTED_CONTENT_TYPES_PREFIX, collect the body of the request , multiple paths should be separated by ,application/json,text/
plugin.influxdb.trace_influxqlIf true, trace all the influxql(query and write) in InfluxDB access, default is true.true
plugin.dubbo.collect_consumer_argumentsApache Dubbo consumer collect arguments in RPC call, use Object#toString to collect arguments.false
plugin.dubbo.consumer_arguments_length_thresholdWhen plugin.dubbo.collect_consumer_arguments is true, Arguments of length from the front will to the OAP backend256
plugin.dubbo.collect_provider_argumentsApache Dubbo provider collect arguments in RPC call, use Object#toString to collect arguments.false
plugin.dubbo.consumer_provider_length_thresholdWhen plugin.dubbo.provider_consumer_arguments is true, Arguments of length from the front will to the OAP backend256
plugin.kafka.bootstrap_serversA list of host/port pairs to use for establishing the initial connection to the Kafka cluster.localhost:9092
plugin.kafka.get_topic_timeoutTimeout period of reading topics from the Kafka server, the unit is second.10
plugin.kafka.consumer_configKafka producer configuration.
plugin.kafka.producer_configKafka producer configuration. Read producer configure to get more details. Check Kafka report doc for more details and examples.
plugin.kafka.topic_meterSpecify which Kafka topic name for Meter System data to report to.skywalking_meters
plugin.kafka.topic_metricsSpecify which Kafka topic name for JVM metrics data to report to.skywalking_metrics
plugin.kafka.topic_segmentSpecify which Kafka topic name for traces data to report to.skywalking_segments
plugin.kafka.topic_profilingsSpecify which Kafka topic name for Thread Profiling snapshot to report to.skywalking_profilings
plugin.kafka.topic_managementSpecify which Kafka topic name for the register or heartbeat data of Service Instance to report to.skywalking_managements
plugin.springannotation.classname_match_regexMatch spring beans with regular expression for the class name. Multiple expressions could be separated by a comma. This only works when Spring annotation plugin has been activated.All the spring beans tagged with @Bean,@Service,@Dao, or @Repository.
plugin.toolkit.log.transmit_formattedWhether or not to transmit logged data as formatted or un-formatted.true
plugin.toolkit.log.grpc.reporter.server_hostSpecify which grpc server's host for log data to report to.127.0.0.1
plugin.toolkit.log.grpc.reporter.server_portSpecify which grpc server's port for log data to report to.11800
plugin.toolkit.log.grpc.reporter.max_message_sizeSpecify the maximum size of log data for grpc client to report to.10485760
plugin.toolkit.log.grpc.reporter.upstream_timeoutHow long grpc client will timeout in sending data to upstream. Unit is second.30 seconds

Dynamic Configurations

All configurations above are static, if you need to change some agent settings at runtime, please read CDS - Configuration Discovery Service document for more details.

Optional Plugins

Java agent plugins are all pluggable. Optional plugins could be provided in optional-plugins folder under agent or 3rd party repositories. For using these plugins, you need to put the target plugin jar file into /plugins.

Now, we have the following known optional plugins.

  • Plugin of tracing Spring annotation beans
  • Plugin of tracing Oracle and Resin
  • Filter traces through specified endpoint name patterns
  • Plugin of Gson serialization lib in optional plugin folder.
  • Plugin of Zookeeper 3.4.x in optional plugin folder. The reason of being optional plugin is, many business irrelevant traces are generated, which cause extra payload to agents and backends. At the same time, those traces may be just heartbeat(s).
  • Customize enhance Trace methods based on description files, rather than write plugin or change source codes.
  • Plugin of Spring Cloud Gateway 2.1.x in optional plugin folder. Please only active this plugin when you install agent in Spring Gateway. spring-cloud-gateway-2.x-plugin and spring-webflux-5.x-plugin are both required.
  • Plugin of Spring Transaction in optional plugin folder. The reason of being optional plugin is, many local span are generated, which also spend more CPU, memory and network.
  • Plugin of Kotlin coroutine provides the tracing across coroutines automatically. As it will add local spans to all across routines scenarios, Please assess the performance impact.
  • Plugin of quartz-scheduler-2.x in the optional plugin folder. The reason for being an optional plugin is, many task scheduling systems are based on quartz-scheduler, this will cause duplicate tracing and link different sub-tasks as they share the same quartz level trigger, such as ElasticJob.
  • Plugin of spring-webflux-5.x in the optional plugin folder. Please only activate this plugin when you use webflux alone as a web container. If you are using SpringMVC 5 or Spring Gateway, you don't need this plugin.

Bootstrap class plugins

All bootstrap plugins are optional, due to unexpected risk. Bootstrap plugins are provided in bootstrap-plugins folder. For using these plugins, you need to put the target plugin jar file into /plugins.

Now, we have the following known bootstrap plugins.

  • Plugin of JDK HttpURLConnection. Agent is compatible with JDK 1.6+
  • Plugin of JDK Callable and Runnable. Agent is compatible with JDK 1.6+

The Logic Endpoint

In default, all the RPC server-side names as entry spans, such as RESTFul API path and gRPC service name, would be endpoints with metrics. At the same time, SkyWalking introduces the logic endpoint concept, which allows plugins and users to add new endpoints without adding new spans. The following logic endpoints are added automatically by plugins.

  1. GraphQL Query and Mutation are logic endpoints by using the names of them.
  2. Spring's ScheduledMethodRunnable jobs are logic endpoints. The name format is SpringScheduled/${className}/${methodName}.
  3. Apache ShardingSphere ElasticJob's jobs are logic endpoints. The name format is ElasticJob/${jobName}.
  4. XXLJob's jobs are logic endpoints. The name formats include xxl-job/MethodJob/${className}.${methodName}, xxl-job/ScriptJob/${GlueType}/id/${jobId}, and xxl-job/SimpleJob/${className}.
  5. Quartz(optional plugin)'s jobs are logic endpoints. the name format is quartz-scheduler/${className}.

User could use the SkyWalking's application toolkits to add the tag into the local span to label the span as a logic endpoint in the analysis stage. The tag is, key=x-le and value = {"logic-span":true}.

Advanced Features

Advanced Reporters

The advanced report provides an alternative way to submit the agent collected data to the backend. All of them are in the optional-reporter-plugins folder, move the one you needed into the reporter-plugins folder for the activation. Notice, don't try to activate multiple reporters, that could cause unexpected fatal errors.

  • Use Kafka to transport the traces, JVM metrics, instance properties, and profiled snapshots to the backend. Read the How to enable Kafka Reporter for more details.

Plugin Development Guide

SkyWalking java agent supports plugin to extend the supported list. Please follow our Plugin Development Guide.

Test

If you are interested in plugin compatible tests or agent performance, see the following reports.

Notice

¹ Due to gRPC didn't support JDK 1.6 since 2018, SkyWalking abandoned the JDK 6/7 supports in all 7.x releases. But, with gRPC back forward compatibility(at least for now), all SkyWalking 6.x agents could work with 7.x, including the agent and backend.