The lowest-overhead
LLM router.
One OpenAI-compatible endpoint in front of everything you serve. Written in Rust for teams putting real traffic through more than one model or node — 0.76 µs of work per request, and no I/O on the request path at all.
What it is
RBAC, per-model grants, rate limits and budgets are integer comparisons against a snapshot already flattened in memory. A test in the repo fails the build if anything I/O-shaped lands there.
An upstream's frames reach your client exactly as they arrived — never deserialised, never re-encoded, never buffered. Cost does not grow with how much your users read.
A shared prefix goes back to the node already holding its KV cache, unless that node is meaningfully hotter than the least-loaded one. Round-robin makes every request pay full prefill.
A proxy that loses its control plane keeps serving from its last-known-good snapshot. Health is per replica, never merged, and SIGHUP swaps the routing table without touching in-flight generations.
The numbers, and their conditions
Measured against LiteLLM on the same cluster, same backends, interleaved A/B runs — with the GPU removed, so the gateway is the only thing being measured.
With real GPUs, aggregate throughput is a wash — both gateways saturate the same hardware. What survives contact with real silicon is steadiness: p99 TTFT of 766 ms against 2921 ms at 32 concurrent streams, and inter-token jitter 15–25% lower at every concurrency level.
Every number, its conditions, and what was not measured →
What you get
The full list, with its measured trade-offs and honest limits →
Routing you can inspect before you trust it
Dry-run answers which rule would decide and what the chain resolves to, without dispatching anything — because a routing table you cannot interrogate is a routing table you find out about in production.
History, not just a live view
Requests stacked as served / upstream errors / refusals-by-kind — because a caller stopped by a budget and a backend that fell over need different people to do different things. A gap in the latency line is a bucket with nothing to measure, never zero.
Already running LiteLLM?
fastllm-proxy import --config litellm_config.yaml --database-url postgres://...
Models, backends, keys and each key's per-model grants come across. Idempotent — re-importing an edited file converges rather than duplicating, and grants removed from the file are revoked. Your existing keys keep working against the same models they already had.
Start here
Deploying to Kubernetes: the Helm chart, or the worked manifests for one real cluster. Everything else — troubleshooting, security, the CLI, the API, the changelog — is in the sidebar.