llmao is an Apache project, apache/tooling-llmao, and what follows is meant to go to it as pull requests, as its guide to pointing a harness at the gateway already did. Each proposal starts from what goes wrong without it, and from what the Foundation gets, not from a setting: the settings are the easy part, and the motivation is what a reviewer weighs. Where an open standard covers the need, the proposal names the standard, so the gateway‘s surface stays llmao’s own and LiteLLM stays an implementation behind it.
Written 11 October 2026, from building this reference implementation against the gateway and from llmao at 266122c. What the gateway answered that day, with a PAT:
| Route | Answer |
|---|---|
/v1/models | gemma4-26b, qwen3-8b, qwen3.8-27b; qwen3-8b is listed but often answers that it does not exist |
/v1beta/models/<model>:generateContent | 200: the Gemini route is deployed, and Gemini CLI and Antigravity run on it |
/v1beta/models | 404: the Gemini-shaped model list is not routed |
/v1/mcp/server | 200 with an empty list: the proxy's MCP registry is reachable, with no servers on it yet |
Each proposal ends with potential implementation details, naming the proxy‘s own setting where one exists. The setting names are from LiteLLM’s documentation and should be checked against the version the gateway runs, which llmao reads from the proxy's readiness endpoint.
What llmao already has, and these proposals build on: Apache identity on every key, keys per project, user and purpose, budgets per project, drop_params, and a health roll-up per model on its own pages.
Why. When an agent's run goes wrong, or costs more than it should, the question is always which task, which session, which model call, which tool call. Today the answer stops at the edge of each system: our launcher traces a session up to the request it sends, the gateway records spend per key, and nothing joins the two. Cost per task, the audit of what an agent decided, and debugging a failed run across our wrapper, the gateway and a tool server all need one trace. For a fleet of agents working around the clock on any task they can claim, this is how anyone finds out what happened.
Standard. W3C Trace Context (traceparent), W3C Baggage for the session and task, and OpenTelemetry's semantic conventions for generative AI for the attributes: model, tokens, finish reason, gen_ai.conversation.id.
Proposal. The gateway continues a trace it is handed and exports its own spans under its own service name, llm.apache.org, with a request, a model call and, once tool servers are on it, a tool call. Its collector keeps the proxy‘s internal spans out, so what a caller sees is the gateway’s contract. Each response carries the gateway‘s request id in a header, so a caller can join the two by id even without a shared trace backend. A shared backend where a PAT’s owner can query their own traces can come later.
Ours. The launcher‘s egress sends traceparent and baggage on every request and records the gateway’s request id on its span.
Potential implementation details.
litellm_settings: callbacks: ["otel"], configured by OTEL_EXPORTER, OTEL_ENDPOINT and OTEL_HEADERS, with OTEL_SERVICE_NAME set to llm.apache.org. It continues an incoming traceparent, so the gateway’s spans join the caller's trace without code in llmao.filter or transform processor that drops or renames the proxy‘s internal spans and strips prompt and response attributes, so what leaves is llmao’s contract.x-litellm-call-id, documented as llmao's request id; llmao can rename it at its vhost if the name should stay hidden.Why. A key is per project, user and purpose. A fleet of agents runs on one automation key, and to the gateway every session on it looks the same: one session cannot be limited, revoked or billed apart from the rest, and a runaway agent can spend the project's budget before anyone notices. The same gap blocks a hosted memory server, which must know which session is asking, since every read is checked against what that session has already read and the ceiling of where its model calls go.
Standard. Baggage for a session id now; OAuth 2.0 Token Exchange (RFC 8693) for a session-scoped token later.
Proposal. In two steps.
Ours. The egress already holds the PAT so that a harness never sees it; it would set the session id, strip any the harness sends, and later do the exchange.
Potential implementation details.
litellm_session_id in the request, which its spend logs and UI group by, and tags, or the x-litellm-tags header, for the task. A small pre-call hook in llmao maps W3C baggage onto them, so callers speak the standard and never the proxy’s names.max_budget with budget_duration, and tpm_limit, rpm_limit and max_parallel_requests. Automation keys would get them by default.duration, the PAT's team, the session in its metadata, and its own limits; revoking it is a key delete. No new store, and the proxy enforces it like any key.Why. A harness discovers models from the list and fails on the first call to one that does not answer. qwen3-8b has been listed and often unanswering since at least 5 October, and every caller pays a failed run to find out. Drift detection, which watches whether the model behind a name has changed, cannot tell a deliberate release from a silent swap while only aliases are listed.
Proposal. The API's model list follows the health roll-up llmao already shows on its pages. Each alias also names the version it points at, a versioned name or a field, so a change of version is visible. The Gemini-shaped list at /v1beta/models is routed like the rest. Logprobs pass through where vLLM returns them: they are the most sensitive drift signal there is, and nothing else shows a model changing before its answers do.
Potential implementation details.
/model/delete, and push it back with /model/new when it recovers, which llmao already uses. The proxy’s general_settings: background_health_checks with health_check_interval, and the router's allowed_fails and cooldown_time, keep a failing deployment out of routing in between, but not out of the list.model_info llmao sends with /model/new, such as version and base_model, which the proxy serves on /v1/model/info; or versioned model names with the alias kept as a second name.logprobs and top_logprobs are OpenAI parameters the proxy passes to hosted_vllm and vLLM honours; the check is that drop_params does not drop them for these deployments.GET /v1beta/models from /v1/models, in Gemini's shape.Why. The harnesses are written for their vendors' APIs, and four differences break them on the gateway‘s models. Our egress rewrites each one, as llmao’s harness guide records, and so does anyone else who gets it working, or they give up. Fixed in the gateway, every harness works by hand, and every Apache contributor's first hour on it is spent working rather than debugging.
| Difference | Harness | What breaks |
|---|---|---|
| a system message after the conversation | Claude Code | Qwen's chat template refuses the request |
| a Gemini tool result under the model's role | Gemini CLI, Antigravity | the proxy‘s Gemini translation drops it, and the model answers without its tool’s result |
| an output limit of 128k tokens | Muse Code | a 131k-context model refuses once the prompt is counted |
a reasoning effort of high | Claude Code | vLLM takes low, medium and xhigh; the model supports the parameter, so drop_params passes it on and the value is refused |
Proposal. The gateway normalises each, in a pre-call hook: fold the system message, move the tool result to the user‘s role, cap output at the model’s room, map effort to the nearest level the model takes. Each is a few lines with a test, and the upstream fix where there is one: the Gemini translation's issue was closed by a stale bot, unfixed.
Potential implementation details.
CustomLogger in llmao with an async_pre_call_hook, registered under litellm_settings: callbacks, that rewrites the request before it is routed: fold a late system message into the first, move a Gemini tool result to the user‘s role, clamp max_tokens to the model’s room, and map reasoning_effort to the nearest level the model takes. Each is a few lines with a test.litellm_settings: modify_params: true already fixes some message shapes for some providers; whether it covers these four on hosted_vllm is worth one test before the hook is written.Why. The proxy can manage MCP servers itself, so llm.apache.org can list the tool servers a key may reach, each running on its own host. Two things from building this make that more pressing.
The vendors' own tools do not cross the gateway. Codex‘s web_search runs inside OpenAI’s API and Claude Code‘s web search inside Anthropic’s. On the gateway's open models they silently do nothing: in the decision contexts our egress recorded up to 11 October, Codex offered web_search on all 56 requests to the gateway and not one search came back. A search server and a fetch server in the registry close that gap for every harness at once.
And a tool server is an injection surface. A tool's description is read by the model as instructions, and what a tool returns, a mail thread or a web page, is written by anyone.
Proposal. For each server in the registry:
Ours. The egress gets a route per registered server, /mcp/<name>, with the PAT injected so the harness never holds it. A tool call through it is then observed by the egress, logged with its policy decision, and labelled, where today a harness‘s tool calls reach the record only through its own transcript, a self-report. What a tool returns is data, never instructions: the memory server’s write scan applies before any of it is written to the record.
Potential implementation details.
mcp_servers configuration or through its API, each pointing at its own host.The memory server itself stays beside the harness, per session, until proposal 2's session token exists; before then, a hosted read-only search over public entries is the safe first step.
A reviewer at llmao will ask two things before any of the above: whether LiteLLM, the proxy behind the gateway, already offers memory the Foundation could use instead of this design, and what llmao itself has done since these proposals were written. Both, as of 12 October.
LiteLLM ships one feature it calls memory, and it is a key-value store, not a record.
/v1/memory stores a text value, with optional JSON metadata, under a globally unique key, in a Postgres table, scoped to the caller‘s user and team: create, read, update, delete, and list by key prefix (docs, PR #26218, merged 25 April 2026, in the open-source proxy since about v1.83). Nothing reads it while a model call runs: remembering and recalling are the client’s job. It was on a draft deprecation list in July and kept after the community objected (discussion #32090). The gateway runs LiteLLM 1.102.2 (llmao 4493935, 8 October), so the route exists; with an ordinary PAT it answers 401, “Only proxy admin can be used…”, so no user can reach it today.previous_response_id is given (docs); vector stores, whose data stays at the provider, the managed kind Enterprise-licensed (docs); prompt management, which is templates (docs); semantic caching, which LiteLLM’s own docs warn “goes badly wrong on agentic traffic” (docs); and Anthropic's memory tool, passed through for the client to execute. The A2A and MCP gateways keep no memory of their own.None of this is the record the paper describes. /v1/memory has no trust tiers, no labels or ceilings, no provenance, no search beyond a key prefix, no erasure with a receipt, and no domain to partition by. And the dropped gateway memory went the opposite way from this design: memory written and recalled out of sight of the harness is the poisoning surface the paper's gates exist for, where here every read and write is a tool call the harness makes and the policy decides, with its provenance kept. If that work comes back upstream, llmao should leave it switched off.
What LiteLLM does offer this design is its MCP registry, with an approval step and a cost per tool call: the place, under proposal 5, where the memory server and the taps would be registered as tool servers. Its Anthropic memory-tool pass-through is a second, smaller opening: the tool's commands could be served by the memory server, so a harness that speaks only that tool still writes to the record.
These proposals were written against llmao at 266122c (10 October). Since then:
| Proposal | In llmao |
|---|---|
| 1. One trace from a task to a model's answer | Nothing yet. 76a6899 (12 October) times each LiteLLM admin request and each page handler, to find why My Keys is slow: llmao's own pages, not model calls. Nothing in the code reads or sends traceparent. |
| 2. Sessions as a caller the gateway can tell apart | Nothing yet. |
| 3. A model list that tells the truth | On the human side only. The Models page was redesigned from issue #46: status per deployment rather than per model, privacy computed from every configured deployment (Private, External or Mixed, a down one included), and whether a model thinks by default (266122c, 689c4ae, 45da9c2, 957d607). The /v1/models list harnesses read is unchanged. |
| 4. Compatibility for harnesses | By documentation rather than in the gateway. The guide to pointing a harness at the gateway by hand (811d775, 9 October, contributed from this work) says what this launcher‘s egress fixes that a manual setup does not, the system-message fold among them; llmao’s help now has a page per harness, PR #51 (open) tightens the Claude Code setup, and issue #44 asks for configuration snippets to copy. The fixes themselves are not in the gateway. |
| 5. Tool servers | Issue #25, “MCP server registration through llmao” (open since 8 September, no replies), proposes LiteLLM's MCP registry with its review workflow, project access groups and per-tool cost: the place proposal 5 builds on. |
Beside them, issue #43 (open) considers metrics in Prometheus or OpenMetrics form for Infra‘s Datadog. The numbers it would publish per key and model, calls, tokens and time, are the ones this launcher’s egress now keeps per session from the other side (amem usage, amem cost), which is the argument for proposals 1 and 2: with a trace and a session on each call, the two sets of numbers join. Everything else in llmao since 10 October is the vLLM hosting fleet, the help pages and the navigation.