Deploy a production LiteLLM multi-provider gateway

Why this step matters

Virtual keys, spend records and policy objects are durable security and financial state. Mixing migrations with ordinary replica startup creates races and makes rollback ambiguous; process-local rate-limit state makes limits inaccurate behind a load balancer.

What to understand

Keep PostgreSQL backups encrypted with an independently recoverable key. Inventory database extensions, roles, grants, schema revision and application version in every backup manifest.

Do not grant the runtime role ownership, database creation, role management or broad public-schema privileges. The migration identity should be unavailable to the running gateway.

Decide whether Redis loss rejects requests, disables only cache, or applies conservative local limits. Document the choice per feature; never silently disable authentication or hard budgets.

Set connection pool sizes from a total database connection budget across all replicas, migrations, workers and administrative sessions. Include headroom for recovery.

System changes

• Creates durable control-plane state, least-privilege identities and shared limiter/cache connectivity; it does not create provider keys or public aliases.

Syntax explained

sslmode=verify-full

Encrypts the PostgreSQL connection and verifies both certificate chain and server name.

migration role

Temporary release identity allowed to apply reviewed schema changes but not used by gateway replicas.

shared limiter

Coordinates counters across replicas so an aggregate tenant limit is not multiplied by replica count.

Why this step matters

A signed image and a valid YAML fragment do not create an operating service. This step gives readers a directly inspectable runtime boundary and separates image identity, secret injection, private reachability, process health, and later public routing into independently testable decisions.

What to understand

Write `LITELLM_IMAGE=docker.litellm.ai/berriai/litellm@sha256:<approved-digest>` to `/etc/litellm/image.env`; never deploy a tag-only value. Keep `/etc/litellm/runtime.env` mode 0600, root-owned, outside the repository, and render it from the secret manager immediately before deployment.

The runtime env file contains secret references or injected values for `LITELLM_MASTER_KEY`, `DATABASE_URL`, provider credentials and Redis authentication. Do not print `docker compose config` without `--images`, because a fully interpolated configuration can disclose environment values.

Loopback publishing prevents direct remote access but does not constrain outbound provider traffic. Enforce an approved provider allowlist with the host, network, or egress proxy and test denied destinations separately.

The health check proves process liveness only. Authentication, alias policy, database schema, provider reachability, limits, privacy and quality are accepted by later negative and behavioral fixtures.

The `ops/*.mjs` commands elsewhere in this guide name version-controlled operator test harnesses, not built-in LiteLLM commands. Implement their stated contracts in your deployment repository or run equivalent explicit curl, database and observability checks; never assume an undefined helper has enforced a control.

System changes

• Creates and starts one private LiteLLM gateway container, reads the root-owned runtime environment, connects to private state and controlled egress, and publishes port 4000 only on host loopback.

Syntax explained

image@sha256

Runs the exact signed and reviewed bytes recorded by the release manifest rather than resolving a mutable tag at restart time.

127.0.0.1:4000:4000

Makes the backend reachable only from the host until the authenticated TLS edge is configured.

read_only + tmpfs

Keeps the container root immutable while providing bounded ephemeral space required by the Python runtime.

config --images

Shows the resolved image identity without rendering secret-bearing environment values into terminal history or deployment evidence.

Why this step matters

Aliases let applications remain stable while the platform changes deployment details, but they can hide material provider differences. External policy metadata and behavioral fixtures make the abstraction reviewable.

What to understand

Use provider-side project or account credentials dedicated to this gateway and environment. Never reuse personal keys or a production key in development.

Set provider timeouts and gateway deadlines coherently. Retries are configured later as a routing policy so nested SDK, LiteLLM and provider retries do not multiply attempts.

Resolve secret references before serving traffic and report only the secret version or fingerprint. A missing provider secret must make only that deployment unhealthy.

Keep passthrough routes disabled unless separately inventoried, authenticated and tested; a passthrough endpoint can bypass alias policy and expose a broader provider API surface.

System changes

• Creates two reviewed aliases and provider deployment references; provider credentials remain in the secret manager and no public access is enabled.

Syntax explained

model_name

Stable alias requested by approved applications and governed by the platform policy.

model_info.id

Unique deployment identity used for health, audit and routing evidence.

max_retries: 0

Avoids hidden per-deployment retries so the explicit router attempt budget remains authoritative.

Why this step matters

The gateway's broad proxy API and administrative surfaces should not become an accidental public control plane. Route allowlisting and network separation reduce exposure before application-level policy runs.

What to understand

Generate or preserve a request ID at the trusted edge and propagate it to the gateway and provider metadata where supported. Never encode tenant, email, prompt or key material in the ID.

Choose streaming-compatible timeouts while retaining an absolute request deadline. A connection that remains open forever can consume concurrency even after provider work is lost.

Restrict outbound network access from LiteLLM to approved provider endpoints, state services, secret manager and telemetry destinations. DNS and proxy policy should be observable.

Keep health probes shallow and non-secret. Provider checks use dedicated test deployments and tiny synthetic requests under strict cost limits.

System changes

• Publishes the inference endpoint behind TLS and hides unapproved routes; it does not yet issue client keys or send production traffic.

Syntax explained

exact location

Publishes only the intended inference path rather than proxying the complete application surface.

client_max_body_size

Rejects unexpectedly large bodies before they consume parsing, logging, memory or provider cost.

absolute deadline

Bounds the full request including queue, attempts, fallback and streaming.

Why this step matters

A shared provider key makes tenant attribution, revocation and least privilege impossible. Virtual keys create a controllable boundary, but only when every route enforces them and the master key is isolated.

What to understand

Prefer automated issuance to an authenticated workload identity. If a key must be delivered manually, show it once over an approved secret channel and require confirmation that it was stored securely.

Use short expiry and rotation with overlap. Revocation must propagate across replicas and caches within a tested objective.

Do not expose raw virtual keys in administrative list responses, logs or database exports used by analysts. Use stable hashed identifiers for correlation.

Key metadata is policy input, not a user assertion. Only trusted administrators or identity automation may set tenant, owner, environment and data-class claims.

System changes

• Creates scoped, expiring workload credentials and negative authorization tests; the master key remains outside caller paths.

Syntax explained

allowed_models

Restricts a workload key to approved public aliases and prevents arbitrary provider/model access.

hard_budget_usd

Defines a blocking spend boundary in addition to alerts and provider-side limits.

max_parallel_requests

Caps concurrent work so one client cannot consume every gateway or provider slot.

Why this step matters

Budgets protect the organization only if they block consistently under concurrency and map to provider reality. A dashboard warning after spend occurs is not an enforcement control.

What to understand

Use maximum input/output tokens and request-size limits to bound the cost of a single request. A monthly budget alone permits one malformed call to consume disproportionate spend.

Define who receives alerts, who may grant a temporary increase and how it expires. Record reason, approver, amount and time window.

Provider-side projects and budgets are defense in depth when available. Keep separate credentials per environment so a staging leak cannot consume production credit.

Reconcile by provider request ID, timestamp, deployment and token dimensions. Never store prompt content merely to explain a financial variance.

System changes

• Creates layered budget policy, alerts, threshold concurrency tests and daily reconciliation; it may reject requests when cost cannot be determined safely.

Syntax explained

unknown_price_action: deny

Prevents an unpriced model from bypassing a hard financial boundary.

incident_reserve

Separately approved capacity for incident response rather than hidden permanent budget inflation.

maximum variance

Triggers investigation when gateway estimates diverge materially from provider billing.

Why this step matters

Provider limits alone protect the provider account, not fair sharing among internal tenants. Multiple independent dimensions prevent a small number of long streams or huge prompts from defeating an RPM-only limit.

What to understand

Count the original request and provider attempts separately. Retry and fallback must consume an explicit attempt budget and provider capacity even when the caller sees one request.

Use a bounded queue or no queue. An unbounded gateway queue converts provider overload into stale, expensive work and memory pressure.

Apply limits before reading or logging a complete oversized body where possible. Validate declared and actual content length.

Design a documented degraded mode for shared-counter failure: conservative local quotas, read-only cached responses where suitable, or explicit unavailability—not unlimited traffic.

System changes

• Adds shared multi-dimensional limits and overload responses across replicas; aggressive clients may now receive deterministic 429 or deadline errors.

Syntax explained

RPM

Bounds request bursts but does not by itself represent token cost or long-lived streams.

TPM

Bounds input and output token throughput; both dimensions should be controlled independently.

attempt budget

Caps the total primary, retry and fallback provider calls made for one client request.

Why this step matters

Reliability logic can become a privacy, quality and cost policy engine. A generic 'try another model' rule hides material changes from applications and customers and may multiply an outage.

What to understand

Do not retry authentication, authorization or structurally invalid requests; retries cannot repair them and can trigger provider defenses or duplicate cost.

For non-idempotent tool or batch operations, verify provider semantics and idempotency keys before any retry. This guide focuses on model inference, not arbitrary provider passthrough.

Use circuit breaking and health signals carefully. A health endpoint can be green while a specific model, region or capability is impaired.

Expose the resolved public alias, deployment ID and fallback occurrence in privacy-safe response metadata or audit records so operators can explain behavior without exposing credentials.

System changes

• Enables bounded retries and compatible failover; when no approved substitute exists, availability intentionally yields to privacy and correctness with a 503 response.

Syntax explained

least-busy

Routes among policy-equivalent deployments using observed in-flight work rather than random choice alone.

never_retry_on

Prevents amplification of permanent authentication, permission and request errors.

unsafe_fallback_action: fail

Makes an incompatible provider/model unavailable rather than silently crossing an approval boundary.

Why this step matters

A gateway observes every prompt and response, so default debugging can create a second ungoverned copy of sensitive content. Redaction must be tested end to end, including errors and third-party callbacks.

What to understand

Use allowlisted telemetry fields rather than attempting to enumerate every possible secret pattern. Redaction remains defense in depth for unexpected values.

Do not put team names, user emails, document titles or prompt fragments in metric labels; labels are widely replicated and high cardinality harms the monitoring system.

Retain request-to-provider correlation through opaque IDs. Operators should diagnose latency, rate limit and spend without reading customer content.

Review each provider's data-control and abuse-monitoring behavior separately. Gateway log minimization does not change what an upstream provider receives or retains.

System changes

• Reduces observability to approved metadata, configures retention and adds canary-based privacy tests across success, error, stream and fallback paths.

Syntax explained

allowlisted fields

Stores only explicitly approved operational metadata instead of filtering an open-ended event body.

key fingerprint

Stable non-secret identifier used for correlation without exposing the virtual key.

raw_provider_errors: false

Prevents provider responses from copying prompt fragments or credentials into gateway logs.

Why this step matters

A gateway outage, overspend and privacy violation are different incidents. Separate signals and runbooks let operators respond without reaching for raw prompt content.

What to understand

Record configuration revision and image digest on every replica and deployment event. Mixed revisions should be visible immediately.

Audit administrative actions with actor, object, before/after policy hashes, timestamp, approval and request ID. Never record the generated secret value.

Use provider request IDs for reconciliation where contractual policy permits, but do not expose them as public identifiers or metric labels.

Test alerts in staging and production canaries. An alert that has never fired under a controlled fixture is only an assumption.

System changes

• Adds dashboards, alerts and administrative audit evidence using metadata-only telemetry; it does not enable prompt inspection.

Syntax explained

SLO burn

Alerts when error or latency budget is being consumed at a rate that threatens the objective.

bounded labels

Prevents user-controlled identifiers and high-cardinality values from destabilizing metrics storage.

policy hash

Links an administrative change or request to the exact reviewed policy revision.