llmao is an Apache project, apache/tooling-llmao. The proposals below are to go to it as pull requests, in the same way as its guide to pointing a harness at the gateway. Its guide went to llmao in this way before. Each proposal starts with what goes wrong without it, and with what the Foundation gets. It does not start with a setting. The settings are the easy part. A reviewer weighs the motivation. Where an open standard covers the need, the proposal names the standard. Thus the gateway‘s surface stays llmao’s own, and LiteLLM stays an implementation behind it.
We wrote this on 11 October 2026. It comes from building this reference implementation against the gateway, and from llmao at 266122c. On that day, the gateway gave these answers to requests with a PAT:
| Route | Answer |
|---|---|
/v1/models | gemma4-26b, qwen3-8b, qwen3.8-27b. The list has qwen3-8b, but that model often answers that it does not exist |
/v1beta/models/<model>:generateContent | 200. The Gemini route is deployed. Gemini CLI and Antigravity run on it |
/v1beta/models | 404. The gateway does not route the Gemini-shaped model list |
/v1/mcp/server | 200 with an empty list. A caller can reach the MCP registry of the proxy, but the registry has no servers yet |
Each proposal ends with potential implementation details. Where the proxy has its own setting for a detail, the detail names that setting. The setting names come from LiteLLM's documentation. Check them against the version that the gateway runs. llmao reads that version from the readiness endpoint of the proxy.
These proposals build on what llmao already has:
drop_paramsWhy. An agent's session can go wrong, or cost more than it should. Then the question is always: which task, which session, which model call, which tool call? Today each system gives only its own part of the answer. Our launcher traces a session up to the request that it sends. The gateway records spend per key. Nothing joins the two. Three uses need one trace:
When a fleet of agents works at all hours, on any task that it can claim, one trace is how anyone finds out what happened.
Standard.
traceparent).gen_ai.conversation.id.Proposal. The gateway continues a trace that a caller gives it. It exports its own spans under its own service name, llm.apache.org. The spans are a request, a model call and, when tool servers are on the gateway, a tool call. The collector of the gateway keeps the internal spans of the proxy out. Thus what a caller sees is the gateway‘s contract. Each response has the gateway’s request id in a header. Thus a caller can join the two by id, also without a shared trace backend. A shared backend, where the owner of a PAT can query their own traces, can come later.
Ours. The launcher‘s egress sends traceparent and baggage on every request. It records the gateway’s request id on its span.
Potential implementation details.
litellm_settings: callbacks: ["otel"]. OTEL_EXPORTER, OTEL_ENDPOINT and OTEL_HEADERS configure it, with OTEL_SERVICE_NAME set to llm.apache.org. The callback continues an incoming traceparent. Thus the gateway’s spans join the caller's trace, with no code in llmao.filter or transform processor. The processor drops or renames the internal spans of the proxy. It also removes the prompt and response attributes. Thus what leaves is llmao's contract.x-litellm-call-id, documented as llmao's request id. If the name should stay hidden, llmao can rename it at its vhost.Why. A key is per project, user and purpose. A fleet of agents runs on one automation key. To the gateway, every session on that key looks the same. The gateway cannot limit, revoke or bill one session apart from the others. An agent that is out of control can spend the project's budget before anyone notices. The same problem blocks a hosted memory server. That server must know which session sends each request, because it checks every read against what that session has already read, and against the ceiling of the destinations of its model calls.
Standard. Now: Baggage for a session id. Later: OAuth 2.0 Token Exchange (RFC 8693) for a session-scoped token.
Proposal. In two steps.
Ours. The egress already holds the PAT, so that a harness never sees it. The egress would set the session id, and remove any session id that the harness sends. Later, the egress would do the exchange.
Potential implementation details.
The proxy's own fields carry the session id and the task. The session id goes in litellm_session_id in the request; the spend logs and UI of the proxy group by this field. The task goes in tags, or in the x-litellm-tags header. A small pre-call hook in llmao maps W3C baggage onto these fields. Thus callers use the standard, and never the names of the proxy.
Limits per key are parameters of the key API of the proxy. llmao already calls this API to issue PATs. The parameters are max_budget with budget_duration, and tpm_limit, rpm_limit and max_parallel_requests. Automation keys would get them by default.
The session token is llmao code over the same key API. On a token exchange, llmao checks the PAT and makes a virtual key with:
durationmetadataTo revoke the token, llmao deletes the key. llmao needs no new store, and the proxy enforces the token like any key.
Why. A harness finds models in the list. It fails on the first call to a model that does not answer. Since 5 October or earlier, the list has had qwen3-8b, and in that time the model often did not answer. Every caller finds this out at the cost of a failed session. Drift detection watches whether the model behind a name has changed. While the list has only aliases, drift detection cannot tell a deliberate release from a silent swap.
Proposal. The API's model list follows the health roll-up that llmao already shows on its pages. Each alias also names the version that it points at, as a versioned name or as a field. Thus a change of version is visible. The gateway routes the Gemini-shaped list at /v1beta/models like the rest. Logprobs pass through where vLLM returns them. They are the most sensitive drift signal there is. No other signal shows a change in a model before its answers change.
Potential implementation details.
/model/delete. When the deployment recovers, the loop can add it again with /model/new, which llmao already uses. Between these two events, other settings keep a failing deployment out of routing, but not out of the list. These settings are the proxy’s general_settings: background_health_checks with health_check_interval, and the router's allowed_fails and cooldown_time.model_info that llmao sends with /model/new, such as version and base_model. The proxy serves these on /v1/model/info. Or versioned model names, with the alias kept as a second name.logprobs and top_logprobs are OpenAI parameters. The proxy passes them to hosted_vllm, and vLLM honours them. Check that drop_params does not drop them for these deployments.GET /v1beta/models from /v1/models, in Gemini's shape.Why. Each harness is written for the API of its vendor. Four differences break the harnesses on the gateway‘s models. Our egress rewrites each one, as llmao’s harness guide records. Anyone else who gets a harness working does the same, or gives up. If the gateway fixes the differences, every harness works with a manual setup. Then every Apache contributor spends the first hour on the gateway on work, not on debugging.
| Difference | Harness | What breaks |
|---|---|---|
| a system message after the conversation | Claude Code | Qwen's chat template refuses the request |
| a Gemini tool result under the model's role | Gemini CLI, Antigravity | the proxy‘s Gemini translation drops it. The model answers without its tool’s result |
| an output limit of 128k tokens | Muse Code | a 131k-context model refuses when the count includes the prompt |
a reasoning effort of high | Claude Code | vLLM takes low, medium and xhigh. The model supports the parameter, so drop_params passes it on, and the value is refused |
Proposal. The gateway normalises each difference in a pre-call hook. The hook:
Each fix is a few lines with a test, and the upstream fix where there is one. A stale bot closed the issue for the Gemini translation, and the bug is not fixed.
Potential implementation details.
One CustomLogger in llmao with an async_pre_call_hook, registered under litellm_settings: callbacks. It rewrites the request before the proxy routes it:
max_tokens to the model's roomreasoning_effort to the nearest level that the model takes.Each is a few lines with a test.
litellm_settings: modify_params: true already fixes some message shapes for some providers. Before you write the hook, do one test: does it cover these four on hosted_vllm?
An upstream fix for the Gemini translation bug is also worth the work, because the hook only works around the bug.
Why. The proxy can manage MCP servers itself. Thus llm.apache.org can list the tool servers that a key may reach, each on its own host. Two things that we learned from building this make that more urgent.
The vendors' own tools do not go through the gateway. Codex‘s web_search runs inside OpenAI’s API. Claude Code‘s web search runs inside Anthropic’s API. On the gateway's open models, these tools do nothing, and they give no error. Our egress recorded decision contexts up to 11 October. In them, Codex offered web_search on all 56 requests to the gateway, and not one search came back. A search server and a fetch server in the registry fix this for every harness at the same time.
Also, a tool server is an injection surface. The model reads the description of a tool as instructions. Anyone can write what a tool returns, such as a mail thread or a web page.
Proposal. For each server in the registry:
Ours. The egress gets a route for each registered server, /mcp/<name>. The egress adds the PAT, so that the harness never holds it. The egress then observes each tool call through that route, logs it with its policy decision, and labels it. Today, a harness‘s tool calls reach the record only through its own transcript, which is a self-report. What a tool returns is data, never instructions. The memory server’s write scan applies before any of it goes into the record.
Potential implementation details.
mcp_servers configuration or through its API. Each server points at its own host.The memory server itself stays beside the harness, per session, until the session token of proposal 2 exists. Before then, the safe first step is a hosted read-only search over public entries.
A reviewer at llmao will ask two questions before any of the above. First: does LiteLLM, the proxy behind the gateway, already offer memory that the Foundation could use instead of this design? Second: what has llmao itself done since we wrote these proposals? This section answers both, as of 12 October.
LiteLLM ships one feature that it calls memory. That feature is a key-value store, not a record.
/v1/memory stores a text value, with optional JSON metadata, under a globally unique key, in a Postgres table. It is scoped to the caller‘s user and team. The route can create, read, update, delete, and list by key prefix (docs, PR #26218, merged 25 April 2026, in the open-source proxy since about v1.83). Nothing reads it while a model call runs. Remembering and recalling are the caller’s job. In July it was on a draft deprecation list. LiteLLM kept it after the community objected (discussion #32090). The gateway runs LiteLLM 1.102.2 (llmao 4493935, 8 October), so the route exists. With an ordinary PAT it answers 401, “Only proxy admin can be used…”. Thus no user can reach it today.
Memory the gateway would keep by itself was built and dropped. Two PRs by a BerriAI engineer made the gateway inject save, search and read tools into requests, and run hidden tool rounds. In those PRs, saving and recall were a setting for each user (#40844, closed 12 September; #40894, closed 4 October without comment). Nobody merged either PR, and nothing has replaced them.
Near it, and not memory:
previous_response_id (docs).The A2A and MCP gateways keep no memory of their own.
None of this is the record that the paper describes. /v1/memory has none of these:
Also, the dropped gateway memory went the opposite way from this design. Memory that is written and recalled where the harness cannot see it is the poisoning surface that the paper's gates exist for. In this design, every read and write is a tool call that the harness makes and the policy decides, and its provenance is kept. If that work comes back upstream, llmao should leave it switched off.
What LiteLLM does offer this design is its MCP registry, with an approval step and a cost per tool call. That registry is the place, under proposal 5, where the memory server and the taps would be registered as tool servers. Its pass-through of Anthropic's memory tool is a second, smaller opportunity. The memory server could serve the commands of that tool. Then a harness that uses only that tool still writes to the record.
We wrote these proposals against llmao at 266122c (10 October). Since then:
| Proposal | In llmao |
|---|---|
| 1. One trace from a task to a model's answer | Nothing yet. 76a6899 (12 October) times each LiteLLM admin request and each page handler, to find why My Keys is slow. This is for llmao's own pages, not for model calls. Nothing in the code reads or sends traceparent. |
| 2. Sessions as a caller the gateway can tell apart | Nothing yet. |
| 3. A model list that tells the truth | Only on the side that people see. llmao redesigned the Models page from issue #46. The page shows status per deployment, not per model. It computes privacy from every configured deployment (Private, External or Mixed, a down one included). It shows whether a model thinks by default (266122c, 689c4ae, 45da9c2, 957d607). The /v1/models list that harnesses read did not change. |
| 4. Compatibility for harnesses | In documentation, not in the gateway. This work contributed the guide to pointing a harness at the gateway by hand (811d775, 9 October). The guide says what this launcher‘s egress fixes and a manual setup does not. The system-message fold is one of these fixes. llmao’s help now has a page for each harness. PR #51 (open) tightens the Claude Code setup. Issue #44 asks for configuration snippets to copy. The fixes themselves are not in the gateway. |
| 5. Tool servers | Issue #25 is “MCP server registration through llmao” (open since 8 September, no replies). It proposes LiteLLM's MCP registry with its review workflow, project access groups and per-tool cost. Proposal 5 builds on that registry. |
Also, issue #43 (open) considers metrics in Prometheus or OpenMetrics form for Infra‘s Datadog. It would publish calls, tokens and time, per key and model. This launcher’s egress now keeps the same numbers per session, from the other side (amem usage, amem cost). That is the argument for proposals 1 and 2. With a trace and a session on each call, the two sets of numbers join. All other changes in llmao since 10 October are to the vLLM hosting fleet, the help pages and the navigation.