What we do with the prompt log, and for whom. docs/prompt-logging-design.md ends where the records exist: one JSON object per request, one file per UTC day, on the gateway host, and the structural figures folded into the demand values. This document takes it from there: the questions we ask of the records, who gets to see which answers, the pipeline that produces them, and the way an answer gets back to the people it concerns.
Everything here is R&D. It lives in analysis/ outside the llmao package, and nothing graduates into the package until it settles.
Two sources, and the line between them is the line between what the portal may show and what only an admin may read.
| Source | Holds | Who may read it | Where it is read |
|---|---|---|---|
The demand values (llmao:demand:) and LiteLLM's daily activity rows | shape: requests, tokens, tools, models, refusals, latency, prompt lengths, each project's share; keys and users per project | a project's members for their project, admins for all | the portal: /admin/capacity today, the project page and the keys page as planned below |
| The prompt log day files | content: the instructions, the messages, the tool, the key | admins, at a shell on the host | the scripts in analysis/ |
The portal never reads the day files, and the scripts never write to the portal. What crosses from the content side to the people is a finding an admin read and chose to pass on, in their own words.
One entry point, analysis/prompts, with a command per question. Two write the store, build and judge; the rest read it or the day files: report, clusters, show, verdicts, tiers.
build reads a day, writes its clusters and figures to the store, and marks the day as in. A rerun reads only days the store has not seen.build turns records into clusters and chooses an exemplar per cluster. Pure string work; no model.analysis_dir. It is the memory between runs: which days are in, what each cluster is, its figures per day, its exemplar, and the judges' verdicts on it.judge sends each cluster's exemplar to each configured judge, a self-hosted model, with the three questions and keeps the answers. It skips clusters a judge has already answered, so a run that stops at 200 of 1,000 picks up at 201, and ends by saying what it produced and what to run next.verdicts and tiers are what an admin reads: the judges' consensus per cluster, where they split, and the tier each cluster used against the tier the judges say would do.| Component | Where | State |
|---|---|---|
| The entry point: every command's arguments, and which module runs it | analysis/prompts | built; prompts --help is the map |
The shared module: config.yaml, the span of days, the fingerprint, the exemplar choice, the store and its views, the judge's call, the tier report, the table printer | analysis/_core.py | built |
report: the reports that need no content: by project, user, model, tool; prompt lengths; shares; failures; messages per request | analysis/_report.py | built; moved out of bin/llmao-prompt-log |
build, clusters, show: the store built from the day files, listed, one cluster in full | analysis/_clusters.py | built |
judge, verdicts, tiers: the questions, the call, the verdicts, the consensus, the tier report | analysis/_judge.py, analysis/judge-prompt.md | built; judge is a dry run by default, --judge sends |
| The settings | config.yaml: analysis_dir, analysis_retention_days, analysis_gateway, analysis_models, analysis_key, analysis_tiers | built; all required, rendered by p6 since 2026-11-06 |
| The judge's key | a service key whose purpose is llmao-analysis, under the rai project | created; the analysis recognises the judge's own requests by that purpose |
| The project page‘s panels: tools by model, refusals, latency, prompt lengths, a daily trend, from each project’s share of the demand values | pages.py, templates/project.ezt, llmao/planning.py | planned; the data has been in the demand values since fold version 3 |
| A note to a project: written by an admin, dated, shown on the project page until a PMC member acknowledges it | the notices mechanism, a prefix of its own | planned |
| A person‘s own shape on the keys page: model mix and tokens per key, from LiteLLM’s daily rows | pages.py, templates/keys.ezt | planned; no fold change needed |
| Tests | analysis/tests/ | the settings and the report so far; the rest when the functions stop moving |
Nothing in llmao/ or bin/ imports from analysis/. The scripts import llmao.prompt_log for the records and llmao.fmt for figures.
Four parts, hashed together with blake2b at 8 bytes, which is fast, stable across runs and needs no cryptographic strength: a clash would merge two clusters and nothing worse.
llmao:none. The prefix marks a label llmao made up, as llmao:master-key does for the person below; no ASF id has a colon, and one word splits cleanly.system parameter, else the opening user message. Not more of the text, because a prompt template carries its variable part early (“Extract an inventory from each file below. ## Domain: ...”) and a longer cut makes a cluster per file: the first 1,024 characters gave 13 clusters of four records where the opening gives one. Not the end, because agents append the working directory and the date there.The cut points and the 200 are set once: changing them changes every fingerprint and orphans every verdict. The trade-off accepted: two prompts that open the same way share a cluster. Two of Spark‘s reviewers both open “You are a security reviewer.” and are one cluster of 653 records; the judge’s activity label would be the same for both.
Health checks, listings and deployment-id calls are not demand, by the same rule the demand rollup uses, and are not clustered.
Chosen deterministically, so the same records always give the same exemplar, and the judge is not asked again for nothing.
The exemplar is chosen when the cluster is first seen and stored with a hash over the head and the messages. It is re-picked, from the records of the day being scanned, when the cluster‘s record count has at least doubled since the pick, or when the day it was picked from has been pruned, so an exemplar is always a record the store still holds. The doublings bound the judge’s extra work to a handful of calls over a cluster's first weeks: the count is the sum of the retained days, so a steady cluster doubles at days 1, 2, 4, 8, 16, 32 and 64, and once the retention caps the count, never again. The prune then gives each cluster a fresh exemplar, and a fresh verdict, once per retention. A rerun over the same days produces the same hash and no new judge call.
analysis_dir/prompts.db, SQLite from the standard library, in WAL mode, one writer.
| Table | Key | Holds |
|---|---|---|
days | day | that the day is in, when it was scanned, how many records |
clusters | fingerprint | project, person, tool, where the opening came from, the opening, first and last day seen, record count, the exemplar (call id, head, messages, hash, record count at the pick, the day of the pick) |
cluster_days | fingerprint, day | requests, failures, prompt and completion tokens, their medians, requests per model |
verdicts (planned) | fingerprint, judge prompt hash, model | exemplar hash, when, the verdict as JSON |
Why SQLite rather than a JSONL file of verdicts: the restart check is a primary key lookup rather than a scan, a crash leaves no half-written line, the day scan becomes incremental because the figures per day are rows, and the findings are a query. The cost is a schema that will churn while this is R&D, met two ways: the verdict is one JSON column, so a change in what the judge returns needs no migration; and --rebuild drops the days and clusters, which are cheap to re-derive, and never the verdicts, which cost model calls.
Every row older than analysis_retention_days is pruned at each run, clusters whose last day fell off included, with their verdicts. One value for all analysis output, set to the prompt log's retention, since the store holds text from the log and must not outlive it.
The store is derived, and the scan writes what the code of the day said. A change to the fingerprint‘s parts, the person’s name or the exemplar leaves an existing store stale until its days are scanned again: --rebuild over the span, after pulling such a change. There is no versioning of the store while it is R&D, unlike the demand values' fold version; the first stale listing on 2026-11-05 was the master-key naming, fixed by a rebuild. A verdict whose fingerprint no longer exists is dropped at the next prune.
A rebuild is cheap, about 12 seconds for a day of 4,800 records; it is string work and SQLite. Judging is the slow part, about 2.5 seconds per cluster through the gateway with one request in flight, so verdicts are what the store protects: kept across rebuilds, re-judged only when the judge prompt, the exemplar or the model changed. A change to the tables themselves has no migration while this is R&D: the store is deleted and rebuilt, and the judges run again. The views are recreated at every open.
What a day costs, in steady state. The scan touches every record every day; the judges answer only for clusters without a verdict from this judge prompt, this model and this exemplar. A fingerprint is the project, the person, the tool and the opening of the instructions, and for a bot or an agent none of those change from Monday to Friday: the Spark bot's fifteen prompts are the same fifteen clusters every day with larger figures. A new cluster is a new person, project or tool, a reworded system prompt, a tool with no system prompt and a different first message each session, or a cluster whose exemplar was re-picked, on a doubling or because the day it came from was pruned. So a day brings a handful of calls, often none, and a hundred calls in a day would itself be a finding. The full bill, clusters times judges, comes only after a judge prompt change, and at 2.5 seconds a call 100 clusters and five judges is about twenty minutes in the quiet hours.
One call per cluster, carrying the judge prompt and the exemplar: the head, the messages, and the cluster's figures as plain numbers (requests, mean prompt and completion tokens, the models used, and how many requests were the same request sent to more than one model), because the tier question needs the size of the work and the cut messages hide it. The answer is fixed in shape: the three questions, one answer each, and a sentence of reasoning, as JSON.
Two figures carry rules, learnt from three judges on one day. They agreed on Foundation work 23 of 23 times and on the activity 19 to 22 of 23, but on the tier only 8 to 13 of 23: gemma4-26b called 19 of 23 clusters low, the 8b called 14 medium, the int4 27b called five high. “Would a smaller model do” is a self-assessment, and a strong model finds the work easy. So the judge prompt now decides the tier from the completion length first, a few tokens out being a label and low, long completions over agent work being high, and in doubt the higher tier, since underestimating the work is the costly mistake (a colleague's point: lean high). And the paired count, the same request sent to more than one model, marks a person comparing models, which is Foundation work whatever the requests contain; the judge cannot see that from one exemplar, and without it a model-evaluation batch of German emails was judged non-Foundation work by all three judges.
qwen3-8b was, until it strayed on the tier. The model name is part of what makes a verdict stale, so switching re-judges everything once and never mixes two models' answers.https://llm.apache.org by default (analysis_gateway), not the proxy‘s local address: the judge is a client like any other, so it runs the same from a developer’s machine against a copy of the store as from the host, and its calls are attributed, logged and counted like anyone‘s. The key is a service key named llmao-analysis under the rai project. The scan never clusters that key’s own requests. Its calls land in the prompt log carrying other projects' exemplars; that is the same content under the same access, and accepted.--judge the script prints the clusters that lack a verdict and from which judges, the calls --judge would make and about how long, and the gateway and key they go to; nothing is sent. The synchronous httpx client, one request at a time.analysis_models lists them, each gives a verdict on every cluster, and the model is part of a verdict‘s key, so the verdicts sit side by side. The store’s consensus view aggregates them: Foundation work leans towards “no”, one “no” among the judges is a “no” for an admin to look at; the tier leans towards “high”, the highest any judge gave; the activity is the majority. The answers are stored as integers beside the judge‘s words so the view is a MIN, a MAX and a count, and the listing shows where the judges differed. On the first day’s 23 clusters, four judges agreed on Foundation work 23 of 23, on the activity 23 of 23 among the three larger models, on the tier 16 to 21 of 23 pairwise; the 8b was the one that strayed, and the int4 27b is the default judge.config.yaml also serves is refused at startup, because the gateway falls back to the provider when our boxes cannot answer and a judge‘s request carries other projects’ prompts. The mechanism that would let a request say so itself is LiteLLM‘s tag routing, which the pinned version has: tag the paid deployments paid when the portal adds them, switch on tag filtering in litellm.yaml, and the judge sends !paid. That is a change to the proxy and to p6, not yet made; until then the guard is the script’s.analysis_tiers saying which model serves which tier: the requests on a heavier model than the work needs are the saving to recommend, the requests on a lighter one the risk. Heaviest excess first.Six flat keys in config.yaml beside prompt_log_dir, all required, as it is: a missing one stops the script. p6 renders them (the key from eyaml), with these values in production:
| Key | Meaning | In production |
|---|---|---|
analysis_dir | where the store lives | /x1/llmao/analysis |
analysis_retention_days | how long every row of the store is kept | 90, the prompt log's |
analysis_gateway | the gateway the judge calls, through the front door | https://llm.apache.org |
analysis_models | the judges, a list; each gives a verdict on every cluster | [qwen3.8-27b-int4] |
analysis_key | the judge's gateway key; the only key without a default anywhere, so the judge stops before its first call without it | the llmao-analysis service key |
analysis_tiers | which models serve which tier, first to recommend, for the tier report | low: qwen3-8b, qwen3.5-9b; medium: qwen3.8-27b-int4; high: qwen3.8-27b, gemma4-26b. A guess to correct |
They were optional with coded defaults until p6 rendered them, on 2026-11-06; the defaults went with that.
A script stops with a message on a missing config.yaml, a missing required key, a day file it cannot read, a judge call that fails, or a model name that models.yaml does not define, whether the judge‘s, one to compare with, or any in the tier map. It can fail harder than the portal: a startup error there takes a service down, here it makes the admin fix their invocation. The same check in the portal, when it reads settings that name models, would be a warning. No script ever opens a connection to LiteLLM except the judge’s model calls, and none writes in the log directory.
Advice is about shape: which model, how much prompt, how long a completion, how a tool is configured. Never the content of a prompt, never a quotation from one. Each piece is a snapshot with a date, and nothing is built to track whether it landed.
The first channel is the simplest: an admin reads the findings and writes to the project. The second is a note on the project page, addressed to the project by an admin, dated, acknowledged by a PMC member when read, stored like the notices that /admin already shows. The script never writes to the portal; an admin always stands between a finding and a person.
Made 2026-11-04. Each is a choice, recorded so it is not re-derived.
analysis/, a top-level directory, not in bin/ (admin tools, stdlib only, no environment to install) and not in llmao/ (the product). The library holds only what the scripts read through: the records and the figures. Nothing in llmao/ or bin/ imports from analysis/.bin/llmao-prompt-log keeps only rollup, which the portal's runner spawns, and takes its directory on the command line; it never reads config.yaml. The reports are prompts report.config.yaml is read by the scripts, not by the library. Everything in llmao/ uses the config the portal loaded in main.py..db as the suffix.qwen3-8b by default, through a service key under rai; its calls landing in the prompt log is accepted.#!/usr/bin/env -S uv run --script, so ./analysis/prompts runs in the project's environment from the checkout. Pre-commit does not identify that shebang as Python, so the ruff hooks run over the tree and extend-include in pyproject.toml names the scripts.prompts declares every command‘s arguments in one place and the modules do the work; the two commands that write, build and judge, are named for it, and judge ends by saying what to run next. The modules import the core and nothing of each other’s.| Step | What | State |
|---|---|---|
| A1 | The split: analysis/ with _core.py, the reports, tests/; bin/llmao-prompt-log down to rollup | done 2026-11-04 (c8c4f46) |
| A2 | The clusters: fingerprint, cluster, exemplar, the store, prune, rebuild, list, show, JSON | done 2026-11-04 (43ed1de); the full day of 2026-11-04 ran in 12 s: 4,847 records, 66 clusters |
| A3 | The judge: judge-prompt.md, the verdicts table, the stale check, one verdict per transaction, the settings; then several judges and the consensus view | done 2026-11-04 and 05; four judges over the day's clusters, the tier rules rewritten from their disagreement |
| A4 | The reports from the verdicts: the consensus, where the judges split, the tier report | done 2026-11-05 as prompts verdicts and prompts tiers; what counts as a finding worth acting on is refined as verdicts accumulate |
| A4a | One entry point, prompts, a command per question, in place of three scripts | done 2026-11-05 |
| A5 | p6: the keys rendered, the directory created; then the keys become required | done 2026-11-06 (p6 PR #2924, then the defaults removed) |
| A6 | The project page‘s panels from each project’s share of the demand values; the sharper form of the “Usage Analysis page” the logging design parked | planned, in llmao/ |
| A7 | A note to a project, written by an admin, shown until acknowledged | planned, in llmao/ |
| A8 | A person‘s own shape on the keys page, from LiteLLM’s daily rows | planned, in llmao/ |
| A9 | The taxonomy refined against real verdicts, with colleagues; tests for _core.py once its functions stop moving | later |
The step numbers continue the logging design's: its N9, “which content analyses, and where they run”, is this document, and its N10, the parked Usage Analysis page, is A6.