The security audit pipeline ingests a codebase, assesses it, and triages vulnerabilities using LLMs, producing consolidated findings and GitHub issues. It began under ASF Security and scales with Tooling support, and relates to the broader Responsible AI Initiative.
This section is the pipeline‘s roadmap and reference: where it’s headed, how it's evaluated, and which security specifications it covers.
Where ASVS fits (web apps like ATR and Steve) and where it doesn't (libraries, backend services, infrastructure) — a chapter-by-chapter breakdown by project type, alternative standards for non-web projects, and how to frame the pipeline when offering it to ASF projects.
Test harness for measuring pipeline quality at scale: fixtures, metrics (recall, precision, false positive rate), LLM-as-judge for semantic comparison, auto-filed issues for novel errors, and operational dashboards. The ATR da901ba L1+L2 run is the regression baseline.
Implementation plan for making the pipeline spec-agnostic: the spec input parameter, per-spec data store schema, spec selection modes, and cross-spec deduplication. This is the prerequisite for every spec addition below.
The pipeline audits against a growing set of security specifications, with automatic spec selection based on project type.
| Status | Spec | Best for |
|---|---|---|
| In production | OWASP ASVS v5.0.0 | Web applications |
| Planned | CWE Top 25 | Libraries, any code |
| Planned | OWASP API Top 10 | API-heavy projects |
| Planned | ASF Security Baseline | All ASF projects |
| Planned | SLSA build levels | Publishing projects |
| Reference | WSTG | Web security testing guide |
After the multi-spec work, adding a spec requires no agent code changes — just requirements in the data store, an optional prompt template, and a discovery-agent mapping update.
How the portfolio — ASVS audit, GHA review, ASF Baseline, SLSA — fills gaps ATR can‘t cover, and how ATR covers the distribution layer the pipeline doesn’t: the three-layer model of source/CI security, build integrity, and release verification.