A proposed architecture to support the Responsible AI Initiative: a governed way for Apache projects to use language models, agents, and tools, with the catalog, metrics, and routing intelligence to keep it accountable.
See architecture.md for the full description, and Foundational AI Tooling for the ASF for a progress report on what Tooling has built toward this platform.
In brief:
agents, mcp, and the llm gateway that all model calls funnel through.
The proposal leaves open the formation of project committees which could include LLM gateway and model management, agent harnesses, and MCP server frameworks, which map directly onto the llm, agents, and mcp boxes above, and a cross-cutting governance layer which maps onto the advisor box.
Early work already maps onto some of the boxes above:
llm gateway — llmao / Hayward A thin litellm-proxy gateway fronted by asfquart: ASF identity, per-PMC budgets, a model catalog with governance metadata, and an OpenAI-compatible API. asfquart owns identity and per-PMC authorization; litellm owns the catalog, budgets, metering, and the API; the project is the seam between them plus a thin portal.agents — tooling-gofannon. A provider- and model-agnostic toolkit and web application for prototyping AI agents and the lightweight web UIs that wrap them, with model access through LiteLLM.