The control-plane protocol

The channels a proxy uses to talk back to its control plane — usage, health reports, budgets and rate-limit reconciliation.

POST /usage: the reverse channel

--role proxy holds a usage::UsageReporter: a bounded queue plus a background flush task that batches events and posts them to /usage on its own schedule, authenticated with the same --proxy-token as /snapshot. Recording an event (UsageReporter::record) is a non-blocking try_send — a full queue means the control plane is not keeping up, and the event is dropped rather than applying backpressure to inference (the design's stated tradeoff: "dropping usage rather than blocking a request is deliberate — billing accuracy is not worth failing inference"). Dropped events are counted and exposed on /metrics as fastllm_usage_reports_dropped_total, so the loss is visible instead of silent. --role all loops this same request back to its own admin API over 127.0.0.1 rather than inventing a second, in-process delivery path.

On the control-plane side, POST /usage accepts a batch, persists it to usage_events, and folds each event's tokens into its principal's budgets.tokens_used, if that principal has a configured budget. A record naming a principal_id or model that no longer exists is dropped from the batch rather than failing the whole request — one stale id from one replica must not poison every other principal's usage in the same flush interval.

POST /health-report: the same channel, one more fact

--role proxy/all also post a health report every --health-report-interval (FASTLLM_HEALTH_REPORT_INTERVAL, default 10s) on that same --proxy-token channel, read back by GET /admin/fleet (see "Live backend health" above). Same posture as usage: a bounded queue — depth one, since only the newest report says anything true — and a failed delivery is dropped rather than retried or allowed to block anything. --role all loops it back over 127.0.0.1 exactly as usage does.

P3: usage accounting and budgets

The proxy never parses response bodies — that is its whole performance story — but usage lives in the body, so this is the one place that comes under tension. The resolution:

• Getting the numbers. A non-streaming response already carries a top-level usage object. A streaming one only does if the request set stream_options.include_usage, so src/proxy.rs's rewrite_model_if_needed injects that field into the upstream body, reusing the same body-rewrite path that already exists for model aliases. This happens for every attributable request — any request with a principal (principal_needs_usage). Only an unauthenticated request is exempt, because there is nobody to attribute the consumption to.

  This was once restricted to principals with a configured budget or a tokens_per_min limit, on the reasoning that nothing else read the number and the injection adds a usage chunk the client did not ask for. That made accounting a side effect of enforcement, and a deployment enforcing nothing recorded nothing: on the cluster where it was found, one principal of seven had a budget, usage_events held nineteen rows, and the newest was five days old while the gateway had served hundreds of requests. Consumption is worth knowing whether or not a cap is attached to it, and it cannot be recovered later — the counts live in a response body that is forwarded and gone.

• Reading them without becoming a parser. src/tail_buffer.rs's TailBuffer keeps a fixed-size (8 KiB) ring of the last bytes forwarded — TrackedBody::poll_frame in src/proxy.rs mirrors every frame into it with a memcpy, never a parse, alongside the pass-through that already exists. At clean end of stream, and only then, the tail is parsed once for a trailing usage object (streaming SSE or non-streaming JSON, tried in that order).

• A row per backend response, whether or not the counts were found. Finding nothing is an ordinary outcome, not an error, and the event is recorded either way with usage_reported = false. That flag is load-bearing in two directions. It is what makes request rate answerable from usage_events at all — a backend that answers 5xx carries no usage block, so recording only responses that had one dropped exactly the rows that describe failure. And it keeps the totals honest: zero tokens and unknown tokens are different facts, so an unreported row prices to NULL rather than to a confident $0.00 that would read as priced-and-free. A response larger than the buffer, or a stream that ends mid-frame, still costs exactly one bounded parse and never panics.

• Requests the gateway refuses are recorded separately. The usual row is written by the body that forwards the response, and a refused request has no such body, so proxy::record_refusal writes one directly for 403 (authorisation), 429 (rate_limit), 402 (budget) and the synthesised 502 for an unreachable chain (no_backend). The refusal column is NULL for everything a backend answered, which is what lets a chart separate the two: status alone cannot tell a 502 the proxy synthesised from a 502 an upstream returned. no_backend is the case that forced this — with every backend down nothing is forwarded, so before it a total outage wrote no rows and an error chart read a flat zero.

• Unauthenticated 401s and unknown-model 404s are counted, not rowed. They have no principal to attribute and no model to name, and 401 is the one refusal an anonymous caller can trigger at will — a row apiece would let unauthenticated traffic drive unbounded writes. They are aggregated per replica per minute in gateway_rejections, fed by the counters on the health report the proxies already send, and surfaced as refused_unattributed on GET /admin/timeseries. Excluded when filtering by model or principal, since they belong to neither. This means a caller-visible error total is answerable from Postgres alone — it is simply two tables, because the two kinds of failure are shaped differently. See docs/operations.md for the delta handling and the restart case.

• Enforcement is after the fact. Principal.budget is pre-resolved into the snapshot (control::build::roll_over_and_load_budgets), so the request path's check is one integer comparison, no I/O — the same shape as the rate limiter just above it. A request that pushes a principal over budget still completes; the next one is refused, with 402 Payment Required, not 429: rate limiting (429) is a pacing problem where waiting a few seconds fixes it, and Retry-After says how long; a budget is a spending problem where no amount of waiting helps until the window rolls over or an operator raises the limit, which is what 402's stated meaning — access denied pending an accounting action — actually matches.

• Budgets roll over. daily/weekly/monthly (fixed-length — 1/7/30 days, not calendar-month arithmetic) checked on every snapshot rebuild; an elapsed window resets tokens_used to zero and persists the new window_start, advancing exactly one window forward even if several were missed while nobody looked.

Rate limits

Limits attach to a principal, not a key: requests_per_min and tokens_per_min, either or both, set via PUT /admin/principals/{id}/limits (or, in File mode, a limits: block under an auth.keys entry — see "Per-key RBAC in File mode" below). A principal with no configured limit is unlimited, not limited to zero.

Enforcement is a local token bucket per principal per replica (src/limiter.rs): one hash lookup and a short, synchronous decrement, no I/O, no allocation once the principal's bucket exists. tokens_per_min is charged against an estimate — the same prompt-size estimate the P1 routing rules use, plus the client's requested max_tokens — since actual usage is not known until the response completes. Exceeding either dimension answers 429 with Retry-After (whole seconds).

A single replica enforcing the full configured limit locally would, with N replicas, admit N× the intended traffic. Accuracy comes from periodic reconciliation instead of a shared counter: every --rate-limit-reconcile-interval seconds (default 5) each --role proxy reports its locally observed request/token counts to POST /limits/reconcile, which aggregates across every replica that has reported recently and returns each one's share — proportional to how much of a principal's traffic it is actually handling — of that principal's configured limit for the next window. The reporting client (src/reconcile.rs) reuses the same pooled Upstream client and bearer-token pattern as POST /usage, but it is not fire-and-forget like that route: the whole point is the allowance in the response body. Before a replica's first successful reconciliation (at startup, or after a control-plane outage) it enforces the full configured limit locally, which is the design's accepted cost: a limit can be exceeded by up to one reconciliation window's worth of traffic during a sharp spike, in exchange for never putting a network round trip on the request path.

--role all never spawns the reconciliation client at all — one process's local counters already are the global counters, so there is nothing to reconcile, and the machinery is inert rather than merely harmless: no background task, no timer, no socket.