Command-line reference

Every flag and every subcommand. fastllm-proxy --help prints the same thing from the binary you are actually running, which is the version to trust if this page and it ever disagree.

Most flags also read an environment variable — FASTLLM_ plus the flag in upper snake case — and the flag wins where both are given. That is what makes a container configurable without an entrypoint script. The exceptions are the tuning knobs that a deployment sets once and a container never overrides: --policy, --admin-port, --snapshot-cache, --workers, --max-retries, --max-body-mb, --pool-max-idle, --upstream-timeout, --health-interval and --health-timeout are flags only. --help on your own binary is the authority: it prints [env: …] beside every flag that has one.

Subcommands

Run with no subcommand, it is the gateway. The five subcommands are operator tools, and each takes its own --database-url so none of them has to be issued alongside --role.

import

fastllm-proxy import --config litellm_config.yaml --database-url postgres://...

The migration path off a file-driven deployment. Idempotent: run it once per environment, or again after editing the file — grants dropped from the file are revoked rather than left behind, so re-importing converges instead of accumulating.

It seeds models, model_backends and the auth: block: a service_account principal per key, the key as a SHA-256 hash, and its model grants as a role named import:<name>. models: ['*'] becomes model:invoke on model/*; a named list becomes one grant per model.

It never prints a key back. The config file remains the only copy of any plaintext.

set-password

fastllm-proxy set-password --name you --password 'change-me' \
  --database-url postgres://...

The one gap the admin API cannot close on its own. PUT /admin/principals/{id}/password is gated behind a session cookie, and a freshly migrated database has no session anyone can obtain — so this is how the first login gets one, run by whoever already holds cluster access.

Creates the principal if the name is new, and grants it admin unless it already holds a role granting config:write. Checking for the permission rather than for "any role at all" is deliberate: the seeded bootstrap principal already holds inference, and a laxer check would turn it into an account that can log in and administer nothing.

Safe to run again to reset a forgotten password. --password also reads FASTLLM_BOOTSTRAP_PASSWORD, which is the form to use if your shell history is somewhere you would rather a password was not.

sync-prices

fastllm-proxy sync-prices --database-url postgres://... --dry-run

Only touches models whose price is unset unless --overwrite: an operator who entered a negotiated rate should not have it replaced by a list price on the next run.

This is the fallback source. Where a provider reports what it actually charged — OpenRouter returns usage.cost unasked — that figure wins and nothing here competes with it.

reencrypt-backends

fastllm-proxy reencrypt-backends --database-url postgres://...

One-shot migration for model_backends.upstream_api_key rows still holding pre-encryption plaintext. Safe to run more than once; an already-encrypted row is left alone. Needed once, by deployments that predate migrations/0004_encrypted_upstream_api_key.sql.

classify-bench

kubectl exec deploy/fastllm-proxy -- fastllm-proxy classify-bench --concurrency 8

It ships inside the image on purpose. The refined-tier cost in the classifier chapter was first measured on a laptop and the deployed container turned out to be more than an order of magnitude slower. Guessing at why — thread counts, core quotas, token windows — is what this exists to stop: it measures the pod's real CPU quota rather than a developer's machine.

Flags

Roles and planes

Listener

Routing and upstreams

Cache

Both matter, and neither alone is enough: a thousand embedding responses is nothing and a thousand completions is hundreds of megabytes, so a single ceiling leaves the other dimension unbounded. Only models that turn caching on ever reach them.

Observability

Classifier

Shutdown

Secrets that are not flags

Three values are environment-only, because a flag ends up in a process listing and these should not:

Where next