Debugging a skill

This is step 6 in the learning progression. You wrote a skill in step 4, applied its safety patterns in step 5, and now the skill is running — but something is wrong. The output is not quite right, or it is right sometimes and not others. This page is the diagnostic path from “my skill did the wrong thing” to a fixed, verifiable skill.

Debugging an agentic skill is not the same as debugging normal code, because the output is probabilistic — the same input can produce slightly different results each time. The techniques here account for that.

Words used on this page

New to some of these words? Here is what they mean here. The landing page has a fuller list.

Audit log: the record the harness writes as the skill runs — the prompts sent, the tools called, and the model‘s responses. In a live agent session, this is the transcript shown in the session view; in tools/skill-evals/, it is the runner’s output.
Eval case (fixture): one example input, together with a description of what a good answer must contain or avoid. See Eval-driven development.
Flaky: a test or eval case that sometimes passes and sometimes fails with no change to the input or skill. Flakiness is normal in probabilistic systems; the goal is to understand why and reduce it, not to eliminate all variation.
Prompt problem: the issue is in what the skill says to the model — the wording, structure, or ordering of the instructions.
Tool problem: the issue is in how the skill calls an external system — a wrong argument, a missing pre-flight check, or an unexpected API response.
Model-capability problem: the task is at the edge of what the model can reliably do — the instructions are fine, but the model cannot execute them well enough.
Temperature: a setting that controls how much variation the model introduces. Higher temperature means more variation; lower means more consistent (but still not deterministic).

The diagnostic loop

When a skill produces wrong output, work through these questions in order. Each one narrows the problem to a smaller surface before you look at code.

Is this failure reproducible? Run the failing case several times with the same input. If it passes sometimes and fails others, you have a flaky failure. If it always fails, you have a deterministic bug.
Where in the skill did it go wrong? Read the audit log to find the step where the output first diverged from what you expected.
Is the problem in the prompt, the tool, or the model? Each has a different fix. See the three sections below.

Reading the audit log

The audit log is the most important debugging tool you have. It shows exactly what the model received and what it returned, at every step. You do not need to guess what happened — it is recorded.

In the eval harness (tools/skill-evals/), run the case with the --cli flag and --verbose. Without --verbose, --cli mode reports only pass/fail per case; adding it makes the runner print each prompt and the model's raw stdout, which is the audit log you want. (The default print mode, with no --cli, also prints the assembled prompts.)

In a live session (any interactive agent harness), the session view shows the model's reasoning and tool calls. Look for:

The exact text the model received at the failing step. Does it match what you intended to send?
The tool calls the model made. Were they correct? Did they return what you expected?
The model's response at the failing step. Is it in the right shape? Does it miss a required field?

If the prompt text the model received is not what you intended, the problem is in how the skill is structured — likely a prompt problem. If the prompt is correct but the tool call failed, it is a tool problem. If both are correct and the model's response is still wrong, it may be a model-capability problem.

Isolating the problem type

Prompt problems

A prompt problem is the most common. Signs:

The model does the right thing in the wrong order.
The model misses a field or skips a check you wrote into the step.
The model answers a different question than the one you asked.
Rephrasing the step in a test session changes the output.

How to fix: Read the step instructions as if you were the model, not the author. Would a careful reader who knew nothing else do what you intended? If not, rewrite for clarity. Common fixes:

Make the boundary explicit. If a step both reads an issue and classifies it, split it into two steps — reading, then classifying. (See Writing safe skills, Pattern 1.)
Make the output contract explicit. If the step should return a JSON object, say so: “Return a JSON object with fields label (string) and reason (one sentence).”
Add a negative example. If the model keeps confusing two cases, write one sentence describing what the wrong answer looks like and why it is wrong.

After the fix, write an eval case that would have caught the original bug and confirm it now passes.

Tool problems

A tool problem is in the interface between the skill and an external system. Signs:

The model's reasoning is correct but the tool call returns an error.
The tool call succeeds but returns data in a shape the model did not handle.
The skill works in a live session but fails in the eval harness (where the external system is mocked or absent).

How to fix: Check the tool call in the audit log. Verify:

The arguments match what the tool expects (look at the tool's own documentation or --help output).
The pre-flight step checked that the tool is available and authorised. If it did not, add a pre-flight check.
The skill handles the tool's error responses. If a gh issue view call returns a 404, what should the skill do? Write that into the step.

For eval fixtures, tool responses are usually mocked. If the mock does not match what the real tool returns, the fixture is wrong — update it.

Model-capability problems

A model-capability problem is harder to fix, because the solution is not a rewrite — it is a different approach. Signs:

Simplifying the prompt or splitting the step does not help.
The model reasons correctly about the task in isolation but fails when combined with the rest of the skill.
The failure rate stays high regardless of phrasing.

How to investigate:

Isolate the failing step: write a minimal prompt in a test session that contains only that step's input and instructions. Does it still fail?
If yes, the task is at the model's capability edge. Consider:
- Breaking the step into smaller sub-steps, each simpler.
- Using a more capable model for the failing step (see Choosing models).
- Providing a worked example in the step instructions (few-shot prompting).
If the isolated step passes but it fails in the full skill, the problem is context contamination — an earlier step's output is confusing this one. Check whether earlier steps leave ambiguous state in the conversation.

Narrowing a flaky failure

Flakiness — a case that passes sometimes and fails others — is expected in probabilistic systems. It becomes a problem only when the failure rate is high enough to matter, or when it hides a real defect. Here is how to tell the difference.

Step 1 — Measure the failure rate. Run the failing case at least five times. (Ten is better for a case you intend to keep.) Note the pass rate. A pass rate above 90 % is usually acceptable for a smoke check; a pass rate below 70 % is worth fixing regardless.

Step 2 — Check whether temperature is the cause. If the eval runner supports a temperature setting, lower it. If the failure rate drops significantly, the flakiness is model-variation, not a defect. In that case the fix is usually a tighter output contract (see the prompt problem section above).

Step 3 — Check whether the fixture is underspecified. If the output spec says “return a short summary” but does not define how short, reasonable answers vary widely. Tighten the spec: “return a summary of one to three sentences.” A more specific fixture is more stable.

Step 4 — Check whether the model is being asked to do too much at once. A step that reads, classifies, and summarises in a single call will be less consistent than three separate steps. Split it.

Step 5 — Tag stable cases as local-smoke. Once a case passes reliably on both a frontier and a local model at a fixed temperature, it is a good candidate for the local-smoke tag. See the case-meta.json format in tools/skill-evals/README.md.

The debug workflow from end to end

Here is the full workflow as a checklist.

Observe. Run the failing input several times and read the audit log. Note where the output first diverges from what you expected.
Classify. Is this a prompt problem, a tool problem, or a model-capability problem? Use the signs above to decide.
Isolate. Reproduce the problem in the smallest context you can — ideally a single eval case, or a one-step test session with no surrounding skill.
Fix. Apply the relevant fix from the section above.
Add a regression case. Write an eval fixture that would have caught the original bug. This stops it coming back and documents what the correct behaviour is.
Run the full suite. Confirm the new case passes and the existing cases still pass. Fix any regressions before continuing.

Check your understanding

A skill step is supposed to return a JSON object with two fields. In the audit log you see that the model returned a plain-text sentence instead. What type of problem is this, and what is the first thing you would fix?
A skill works correctly in a live session but always fails in the eval harness. The audit log shows the tool call returns a 404 error. What is the likely cause, and where would you look first?
An eval case passes seven times out of ten. The output spec says “return a label”. How would you approach fixing this flakiness?
After splitting a complex step into two simpler steps, the failure rate drops from 40 % to 5 %. What does this tell you about the original problem?

How this connects to the other guides

Writing safe skills is step 5, the page before this one. The injection-flag idiom and the draft-before-post pattern both appear in audit logs when they fire — knowing them makes the log easier to read.
Writing portable skills is step 7, the page after this one. Once a skill runs correctly, that page makes it work for any project and any model, not only the one you debugged it on.
Eval-driven development is step 8. That page covers how to design an eval suite; this page covers the debug loop you run when one fails. They pair: the evals surface the bug; this page fixes it.
Choosing models is step 3. When a failure turns out to be a model-capability problem, that page is where to look for guidance on which model tier to try next.
Agentic and autonomous work is step 9. When no one is watching every step, flakiness and silent tool failures become much harder to catch. The debugging habits here are the foundation for safe autonomy.
tools/skill-evals/README.md — the harness reference: runner flags, grading modes, the case-meta.json format, and the local-smoke tag.
Pattern catalogue — the patterns named in this page (injection-flag, draft-before-post, output-contract) are collected there as copy-ready blocks.

Licence

Everything in docs/education/ is under the Apache License 2.0 (PRINCIPLE 17). Pages written with help from AI carry a Generated-by: note in their commit message, following ASF Generative Tooling Guidance.