CLAUDE.md - Plugin Test Guide

This file guides AI assistants writing and changing plugin tests under test/plugin/scenarios/. For the full test-framework mechanics (scenario layout, configuration.yml, expectedData.yaml, containers, running locally) see the “Plugin Test Framework” section of apm-sniffer/apm-sdk-plugin/CLAUDE.md, and docs/.../Plugin-test.md.

Testing philosophy: plugin scenarios are the key test — not unit tests

For an agent plugin, the plugin test scenario is the primary and sufficient test. Do not over-engineer by adding many unit tests for a plugin.

A plugin's real risk is two things, and a scenario proves both at once by running the real framework:

  1. Cast / fetch-data logic — the interceptor casting the intercepted arguments to the framework types and reading request/response fields off them (headers, method, URI, status, params). Mocks can't validate that the real framework class actually exposes those methods the way you assumed.
  2. Version compatibility — the same interceptor must work across the supported version range.

So: when adding or changing a plugin, add or extend its scenario, not a pile of mock-based unit tests. Reserve unit tests for genuinely logic/namespace-critical shared code where a deterministic in-JVM assertion earns its keep (e.g. servlet-commons wrap() resolving javax vs jakarta with both APIs on the test classpath) — never for per-plugin interceptor mocking.

Validation path: assert the full span shape your plugin produces

A scenario proves a plugin by asserting the exact spans the real framework makes the agent produce. Pick the shape by what the framework actually does — and don't settle for a single entry span when the plugin also does propagation, or you silently skip the inject/extract path.

  • RPC / web frameworks that have both a server and a client side (Struts, Spring MVC, gRPC, Dubbo, HTTP servers …): drive the full round-trip — curl → server → its own client call → another service's endpoint. The trace then contains 2 entry spans + 1 exit span:

    1. the server entry span for the request curl hit (proves the server side: extract + entry),
    2. the client exit span for the outbound call (proves inject + exit),
    3. a second entry span on the other endpoint carrying a cross-process ref back to the first (proves propagation across the wire — the receiving/extract side).

    The ref itself must be asserted — for RPC the cross-process refs: block is the whole point, so expectedData.yaml must include it on that second entry span with refType: CrossProcess, parentEndpoint, parentSpanId (pointing at the exit span), parentTraceSegmentId: not null, parentService, and traceId: not null. A scenario that asserts spans but not the ref does not prove propagation.

    The struts2.7 scenario is exactly this: case.action (entry) → HttpClient (exit) → case1.action (entry + ref); jetty-12 uses the same shape via a JDK-HttpURLConnection self-call. Keep that nested self-call — it is the only thing that proves inject / extract / propagation; a lone entry span does not.

  • Client-only plugins (JDBC/PostgreSQL, Redis, MongoDB, an HTTP client, a message producer): the peer is a database/broker, not another instrumented SkyWalking service, so there is no second entry span. Assert just the client exit span (component, peer, and the db.* / http.* tags). You do NOT need to stand up another service endpoint.

Version coverage: one version per minor (latest patch)

support-version.list must cover the framework's supported range, but keep one version per minor version — the latest patch, not every patch. E.g. Jetty 12.0.36 + 12.1.10; Struts 7.0.3, 7.1.1, 7.2.1; Spring 6.0.4, 6.1.1, 6.2.19. Verify each version resolves on Maven Central. This is the compatibility proof; CI runs the scenario against each listed version.

How the test containers work (and where the JDK/Tomcat version really comes from)

A scenario runs inside one of two prebuilt container images, selected by type: in configuration.yml:

  • type: jvm → the agent-test-jvm container. Your app is a fat-jar started by bin/startup.sh with ${agent_opts} (the -javaagent: line — you MUST include it in startup.sh). Built FROM ${base_image_java}.
  • type: tomcat → the agent-test-tomcat container. Your app is a *.war dropped into /usr/local/tomcat/webapps/; the agent is wired in via the container's patched catalina.sh. Built FROM ${base_image_tomcat}.

The version pin is in the CI lane, not configuration.yml. configuration.yml only says type: tomcat — it does not choose a Tomcat version. The concrete JDK and Tomcat versions come from the base_image_java / base_image_tomcat inputs passed to ./.github/actions/build in the lane's Build job, and every scenario in that lane shares that one image. So:

  • type: tomcat on the plugins-jdk8-* lane → tomcat:8.5-jdk8 (javax); on plugins-jdk11-* → tomcat:9.0-jdk11 (javax); on plugins-jdk17-* → tomcat:10.1-jdk17 (jakarta).
  • That is why a jakarta/Servlet-6 framework (Struts 7, Jetty 12, Spring 6) MUST be registered on a JDK-17 lane — it needs the Tomcat-10.1 base — and a javax one on an older lane. Putting a jakarta WAR on a Tomcat-8.5 lane just won't deploy.

To run locally you pass the same two knobs explicitly (the defaults are the javax/JDK-8 pair): bash test/plugin/run.sh --base_image_java eclipse-temurin:17-jdk --base_image_tomcat tomcat:10.1-jdk17-temurin <scenario>. Some newer bases lack curl (the container health check needs it) — the JDK-25 lane bakes it in via .github/workflows/Dockerfile-tomcat-jdk25-withCurl.

How the GitHub Actions lanes work

Scenarios are load-balanced across .github/workflows/plugins-<jdk>-test.<group>.yaml files. Each file is one lane: its Build job pins base_image_java + base_image_tomcat, and its test job‘s matrix.case: list names the scenarios that run on that image. Lanes exist per JDK (jdk8/jdk11/jdk17/jdk21/jdk25); the trailing .0/.1/… is just a bucket to spread load. Register a new scenario by adding its directory name to the matrix.case: list of the lane whose base image matches the framework’s needs — use python3 tools/select-group.py to pick the least-loaded bucket in that JDK lane.

Cold-start and the 3s entry-service timeout (a real gotcha)

The container‘s run.sh runs under set -e and hits the entry service with curl -s --max-time 3 ${SCENARIO_ENTRY_SERVICE} — no warm-up first. So the very first (cold) entry request must finish within 3 seconds or curl exits 28 and the whole scenario fails. That first request pays a lot of one-time cost at once: the agent’s ByteBuddy enhancement of every class it touches on that path, JIT, and — for WARs — a cold Jasper JSP compile, all under the container's tight -Xmx256m heap and whatever CPU the runner gives, with jacoco simultaneously instrumenting the org.apache.skywalking.* classes for coverage.

Keep the entry path's cold cost low:

  • Don't render a JSP from the entry action if a lightweight result works (e.g. Struts httpheader, a plain-text/stream result). The view is irrelevant to what a plugin scenario verifies (entry/exit spans, tags, propagation), and a cold Jasper compile is expensive.
  • Be extra careful when the entry action makes a nested internal call (for cross-process propagation coverage): if that nested request also does cold work (a JSP compile) while the outer request is still cold-enhancing Struts/HttpClient, the two cold-start storms contend for the limited CPU and heap and blow up super-linearly (observed ~0.5s each in isolation but ~3–20s nested under throttle). This is a slow first request, not a plugin bug — verify with a CPU-throttled container (docker run --cpus=0.5 …) before assuming a hang.

Practical notes

  • One scenario per plugin (or per javax/jakarta era). Mirror an existing sibling scenario as a template and change only what differs (framework version, namespace, deployment model).
  • Match type: to the deployment: jvm (fat-jar started by bin/startup.sh, ${agent_opts} required) or tomcat (WAR on a container). Jakarta/newer frameworks go in a JDK-17 CI group.
  • Assert the entry span shape that exercises the plugin's data logic: componentId, spanType: Entry, spanLayer: Http, and the url / http.method / http.status_code (and http.params / http.headers where the plugin collects them) tags, plus cross-process refs.
  • Register the scenario in the right .github/workflows/plugins-*.yaml matrix (use python3 tools/select-group.py to pick the least-loaded group for its JDK lane).