Companion to vllm-fleet-design.md, which defines the control-plane contract: what a host is, how a box fetches its assignment, and what the JSON looks like. This document covers where fleet state lives, how it changes, and what happens when it is lost.
An earlier draft of this document proposed a runtime-owned YAML file for membership, on the grounds that a GPU box rebooting during a database outage could still fetch its assignment.
That argument does not hold. During a Postgres outage LiteLLM cannot authenticate any request — virtual keys live there — and with STORE_MODEL_IN_DB it cannot route either. The gateway is down regardless, so a box that fetches its config during that window comes up serving a model nothing can reach. The resilience buys nothing.
State lives in LiteLLM's database. There is no second store.
| field | fleet meaning |
|---|---|
model_name | the catalog model, and the routing key |
litellm_params.api_base | which host, which port |
litellm_params.api_key | the bearer token for that vLLM |
model_info | arbitrary dict — carries the recipe and provenance |
GET /vllm/config becomes: select routes whose api_base host matches the caller's IP, return their model_info.vllm blocks and ports.
STORE_MODEL_IN_DB is an environment variable, not a config key:
STORE_MODEL_IN_DB=True
Without it, /model/new returns HTTP 500 with Set 'STORE_MODEL_IN_DB='True'' in your env to enable this feature. The YAML model_list then becomes a bootstrap seed rather than the source of truth.
A provisioned instance can take fifteen minutes to load weights. A route whose backend is not yet serving will fail every request routed to it.
Rule: a route exists in LiteLLM if and only if its vLLM is serving.
add host -> record assignment, generate api_key, no route yet box boots -> GET /vllm/config health OK -> POST /model/new health DOWN -> POST /model/delete retire -> delete route, drop assignment
Uniform, with no special cases. An earlier draft proposed registering at add-time when the model_name already had healthy peers and deferring otherwise — that makes the same operation behave differently depending on the state of unrelated servers, which is fine when written and baffling later.
Verified against litellm 1.99.0 source. Two findings:
There is no per-deployment enable/disable. No enabled or is_disabled field, and nothing in the model-management endpoints. A route is registered or deleted; there is no third state.
Cooldown does not protect a booting backend. DEFAULT_COOLDOWN_TIME_SECONDS is 5, so a cooled deployment re-enters rotation almost immediately. And router_utils/cooldown_handlers.py deliberately exempts single-deployment model groups — SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD is 1000, with the comment “by default we should avoid cooldowns on single deployment model groups.”
So the first server for a new model would fail every request for the full boot window, uncooled. Health-gating is the only mechanism available.
cooldown_time and allowed_fails are settable per-deployment via model_info if different behaviour is wanted later.
Health-gating means an assignment must exist before its route does. That state is small — IP, port, model name, generated key, per pending host — but it is state.
Options, preferred first:
Config here has three distinct change costs, and conflating them causes most of the confusion:
| lifetime | what | to change it |
|---|---|---|
| baked | ASFQUART_URL, FLEET_KEY in the instance template | re-provision |
| boot | the vLLM assignment — model, port, launch args | box must re-fetch and restart vLLM |
| live | routes, keys, budgets in LiteLLM | immediate |
Changing max_model_len does nothing until that box restarts vLLM. The catalog says one thing and the server does another, silently.
This is a second kind of skew. check_config_skew compares llmao against LiteLLM. Nothing compares llmao's intended launch args against what a box is actually running.
/vllm/config responses carry a revision — a hash of the assignment payload. The box records what it applied and reports it back, so the UX can show applied rev 3, current rev 5 rather than letting a stale server pretend to be current.
If boxes re-fetch periodically rather than only at boot, a changed revision also becomes the trigger for a restart, making config changes eventually consistent.
Whether that restart should be automatic is open: an unattended restart drops in-flight requests. Probably detect automatically, apply on a button.
model_name is a contract with callers. Everything behind it must be interchangeable from their point of view.
model_info.vllm is a recipe for the box. Recipes may differ freely as long as the contract holds.
Same model_name, different recipes — fine:
gpu_memory_utilization, --kv-cache-memoryDifferent model_name required:
If two routes share a model_name, their caller-visible parameters must match. Otherwise identical requests behave differently depending on which backend they land on. Where a pool is uneven, advertise the minimum — a pool with a 40k and a 128k server advertises 40k, or it is not a pool.
Weights come from HF, from internal mirrors, from image registries, and in several quantizations. The recipe should say so explicitly rather than overloading one string:
model_info: vllm: model: Qwen/Qwen3-8B-FP8 # what vLLM is told to load source: kind: hf # hf | url | registry | local vram_gb: 8 # checkable before pulling 50GB disk_gb: 18 args: ["--reasoning-parser", "qwen3"]
Three reasons to separate source from model: the box can check fit before downloading; credentials differ by source kind; and a box that already holds the weights should not re-download because the catalog names an HF repo.
VRAM and disk are discoverable on the box. A hand-typed vram_gb: 80 is wrong the first time a provider supplies a different card than was ordered — which, with rented instances, is a matter of when.
The model's requirement goes in the recipe; the box decides fit against hardware it can see. --kv-cache-memory makes this exact: vLLM prints the byte count it wants, so the install step can start conservative, read the figure, and relaunch.
Deleting a route deletes the record, so “did this host ever exist” loses its answer. Either mark model_info.retired and remove the route from routing, or accept LiteLLM's own audit trail as the record.
Worth deciding rather than losing by default.
last_config_fetch is the liveness signal — surface it as a coloured age, green under an hour, red over a week.
Never auto-retire. A box down for maintenance is indistinguishable from one that is dead, and silently dropping it during an outage is the wrong default.
Retiring promptly is also hygiene: providers recycle IPs, so a stale entry may eventually match an instance rented by someone else. GET /vllm/config requires the fleet key with a constant-time compare, so the blast radius is bounded — but that key is a single shared secret baked into the box template, which makes “nobody else could hold it” an assumption.
model_name pools need a minimum-advertising rule (§4.1), which cannot be a static catalog value once a pool is uneven.notes field is worth considering. “Rented for the superset scan, kill after” is obvious for a week and unrecoverable after a month.| event | LiteLLM DB | in-memory health |
|---|---|---|
| Puppet run | untouched | untouched |
| app restart | untouched | rebuilt in ~45s from probes |
| Postgres outage | unavailable; auth and routing fail | untouched |
| host rebuild | lost | lost |
Health state is correctly ephemeral — a live measurement, not a record.
With routes in Postgres, a lost database means the proxy no longer knows what to proxy, not merely who may call it. Virtual keys are stored hashed and shown once at mint, so recovery also means re-minting every key and reconfiguring every consumer.
Partial mitigation for free: the GPU boxes keep running. They hold their assignment locally and continue serving; they simply cannot re-fetch.
sudo -u postgres pg_dump litellm | gzip > "$dest/litellm-$stamp.sql.gz"
One artifact now, rather than a database dump plus a state file.
$dest is unresolved and belongs to the p6 work. Three constraints:
bpc_client_asf pulls via rsync from a central BackupPC server, so the client has no local daemon and coverage cannot be confirmed from the box. The share list lives on the server — ask Infra which paths are pulled. A default share of /etc and /home would miss /var/backups.bpc_client_asf::excludelist.pg_dump is what makes it restorable.STORE_MODEL_IN_DB=True in the service environmentGET /vllm/config from routes matching the caller's IP/model/new on the healthy transition, /model/delete on down/vllm/config, reported back by install_sets.pymodel_info survive a /model/new round-trip intact? If LiteLLM normalises or drops unknown nested keys, recipes cannot live there. Testable against a local proxy with STORE_MODEL_IN_DB=True.Resolved: LiteLLM is the source of truth; no second datastore; registration is health-gated; there is no per-deployment enable/disable and cooldown does not cover the boot window; host capacity is discovered, not declared; model_name is a caller contract and model_info.vllm a box recipe.
End of design document.