Google Git
Sign in
apache/magpie/HEAD/./tools/skill-evals/evals/security-issue-sync
tree: d19d041a6b27f40757390a808a0348fe94f8fb1b
  1. step-1f-process-step/
  2. step-2a-observed-state/
  3. step-2b-proposed-changes/
  4. step-2c-next-step/
  5. step-3-confirm/
  6. step-6-recap/
  7. step-bulk-orchestration/
  8. step-guardrails/
  9. README.md
tools/skill-evals/evals/security-issue-sync/README.md

security-issue-sync eval suite

Behavioral evals for the security-issue-sync skill. Eight steps are covered; steps 0 (pre-flight), 1a–1e (data gathering), 1g (cve.org API check), 4 (shell apply), and 5/5b/5c (CVE artifact regeneration) are skipped — all low-signal for structured-output evals.

Steps

StepNameCasesNotes
1fProcess step identification7Decision table covering steps 1-2 through 14
2aObserved state3Step 11 (PR merged), step 6 (CVE needed), step 2 (stale)
2bProposed changes3Label swap, milestone create, providers wave
2cNext-step recommendation4Steps 3, 6, 11, 13
GuardrailsGuardrail violation detection3CVSS propagation, ASF project naming, clean pass
3Confirm with user3apply-all, selective, cancel
6Recap2Structural assertions; with and without CVE/draft
Bulk orchestrationBucket-and-walk decision in bulk mode3All label-only (one bundled), mixed buckets (split), all CVE-affecting (all walked)

Hard rules exercised

  • CVSS not propagated: Reporter-supplied CVSS score in proposed body patch or status comment is flagged (guardrails case-1).
  • Other ASF project vulnerabilities not named: Naming a specific CVE from another ASF project is flagged (guardrails case-3).
  • Observed state is tight: 2a summary is a bullet list of facts only — no proposed actions embedded.
  • Milestone create needed: When the target milestone does not exist on the repo, milestone_create_needed: true must be set (2b case-2).
  • Providers milestone is never the PR milestone: For providers scope the milestone is the next wave date (2b case-3).
  • CVE allocate skill explicitly named and linked: Step-6 next-step must name and link security-cve-allocate (2c case-2).
  • No-action parking: Step-11 produces has_concrete_action: false (2c case-3).
  • Golden rule 2 — no bare issue numbers: has_bare_issue_numbers must always be false.
  • CVE-affecting trackers walked individually: in bulk mode, any tracker whose proposed_body_field_updates names a CVE-publication field (Title, Short public summary for publish, CWE, Severity, Affected versions, Reporter credited as, Remediation developer, PR with the fix, Public advisory URL) must end up in cve_affecting and the walk_order is ascending by tracker number (bulk-orchestration cases 2 + 3).
  • Non-CVE-affecting trackers bundled: trackers whose proposals are limited to label flips, milestone touches, assignee swaps, project-board moves, status-rollup entries, reporter Gmail drafts, or RM hand-off comments end up in non_cve_affecting (bulk-orchestration case 1 + the two label-only trackers in case 2).