Run a production vLLM server safely
Deploy a pinned vLLM release behind a trusted TLS and authentication proxy, size GPU and KV-cache capacity from real workloads, isolate artifacts and listeners, verify quality and overload behavior, monitor the engine, and rehearse complete-bundle rollback.
Provide an OpenAI-compatible internal inference endpoint whose exact model and runtime provenance are reproducible, whose direct backend is unreachable, whose capacity and failure contracts are measured, and whose previous release can be restored without fetching unreviewed artifacts.
- Linux GPU host systemd, NVIDIA CUDA-capable GPU
- vLLM pinned and compatibility-tested release
- NGINX supported stable release
- Approved model release An immutable local model, tokenizer, template and generation-configuration bundle with checksums, license review, data-classification decision and a named owner.
sha256sum -c {{modelDir}}/SHA256SUMS - Compatible GPU host Dedicated or scheduled GPU capacity with a documented host driver, CUDA/vLLM compatibility set, sufficient VRAM/RAM/storage and access to equivalent staging hardware.
nvidia-smi --query-gpu=name,uuid,memory.total,driver_version --format=csv - Trusted proxy and identity A maintained TLS certificate, authentication/authorization endpoint, private backend network and owners for proxy, monitoring and incident response.
- Representative evaluation Frozen synthetic or approved-redacted API, task-quality, prompt-length, concurrency, overload and negative fixtures with explicit acceptance thresholds.
- Rollback bundle The previous image or Python lock, model release, proxy configuration, dashboard rules and client-contract fixtures are retained locally and have a tested restore procedure.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- An immutable, checksum-backed vLLM release whose model, tokenizer, template, generation defaults, runtime, GPU layout and public alias can be identified from one manifest.
- A loopback-only inference service running as a dedicated identity with narrow filesystem and GPU access, plus a trusted NGINX boundary for TLS, authentication, limits and access evidence.
- A measured admission envelope based on representative prompt lengths and concurrency, with private Prometheus metrics, overload behavior, canary promotion and a rehearsed rollback.
- Operators can distinguish artifact identity, compatibility, capacity, API-contract, semantic-quality, proxy and hardware failures instead of treating every symptom as a model problem.
- Callers cannot bypass the trusted proxy, while rejected requests fail predictably before consuming unbounded GPU resources.
- A release decision includes provenance, quality, latency, throughput, isolation, observability and rollback evidence rather than a single successful completion.
Architecture
How the parts fit together
A controlled import path produces an immutable local model release. A dedicated vLLM process reads it and uses only assigned GPUs while listening on loopback. NGINX is the sole ingress, terminating TLS and identity policy before forwarding admitted requests. Prometheus and GPU telemetry observe private endpoints. A parallel canary and retained former bundle enable gradual promotion and fast rollback.
- An operator imports an approved immutable model bundle, verifies checksums and records model/runtime/proxy revisions before any listener starts.
- The dedicated service loads the bundle on assigned GPUs and accepts only loopback requests with explicit capacity and generation controls.
- Authenticated clients connect to the public TLS hostname; NGINX rejects invalid or excess work and forwards only admitted requests.
- vLLM batches work within the measured envelope, while private metrics and GPU signals feed alerts that reference the release and rollback runbook.
- A candidate receives frozen and then bounded traffic; threshold breach removes it from routing and restores the complete previous bundle.
Assumptions
- The organization already has an approved artifact import path, model-license review, TLS/identity platform, secret manager, monitoring system and incident owner.
- The guide uses a single-node systemd example because it makes trust boundaries explicit; containers or orchestration may implement the same controls when pinned and tested.
- The exact model and GPU are intentionally variables. Their memory and quality behavior must be measured; this guide does not promise that a named parameter count fits a particular card.
- No model is allowed to execute tools or authorize consequential actions merely because the serving layer is operationally hardened.
Key concepts
- Weights versus KV cache
- Weights are mostly fixed model memory, while KV cache grows with active sequence lengths and concurrency. Both plus runtime overhead determine usable capacity.
- Continuous batching
- The engine schedules tokens from multiple requests together to improve utilization. It changes the latency-throughput tradeoff and must be tested under realistic offered load.
- Tensor and pipeline parallelism
- Tensor parallelism splits operations across GPUs; pipeline parallelism splits layers or stages. Topology, communication and model shape determine whether either helps.
- TTFT and TPOT
- Time to first token captures queue and prefill responsiveness; time per output token captures decode pace. End-to-end latency depends on both plus requested output length.
- Readiness
- A composite release state: correct immutable artifacts, accepted API and quality fixtures, sufficient capacity, enforced security boundary, working observability and rehearsed rollback.
Before you copy
Values used in this guide
{{modelDir}}Absolute, immutable local directory containing the approved model release and SHA256SUMS.
Example: /srv/vllm/models/support-assistant-r7{{modelName}}Stable public API alias mapped to the immutable artifact in the release manifest.
Example: support-assistant{{listenPort}}Loopback-only vLLM HTTP port.
Example: 8000{{maxModelLen}}Evaluated maximum context accepted by this release and capacity plan.
Example: 8192{{tensorParallel}}Reviewed tensor-parallel GPU count matching assigned devices and topology.
Example: 2{{gpuMemoryUtilization}}Measured vLLM utilization ceiling that preserves operational headroom.
Example: 0.82{{publicHost}}TLS hostname owned by the trusted proxy, never the direct vLLM listener.
Example: llm-api.example.net{{previousRelease}}Complete known-good release identifier retained locally.
Example: prod-r17{{candidateRelease}}Immutable candidate bundle under evaluation.
Example: prod-r18Security and production boundaries
- Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
- Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
- Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
- Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
- A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.
Stop before continuing if
- Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
- Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
- Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
- Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.
decision
Freeze the serving contract and model provenance
Write a release record before touching a GPU. Name the business use, permitted data classification, API shape, latency and throughput objectives, model license, immutable source revision, checksums, tokenizer and chat-template revisions, generation defaults, accountable owner, maintenance window, and the previous release that will be restored. The public model alias must point to this record rather than to a mutable repository name.
Why this step matters
Most serving incidents that look like inference bugs are identity failures: a mutable model revision, tokenizer, template, or default changed without appearing in the deployment name. A checksum-backed manifest makes later benchmark, incident and rollback evidence refer to the same artifact.
What to understand
Record license and use restrictions separately from technical compatibility; a model that loads can still be impermissible for the intended data or audience.
Keep the original imported artifacts read-only. Promote by copying or mounting an immutable release directory, not by editing files inside a shared cache.
Describe what is intentionally outside the service contract, including unsupported tools, modalities, message roles, context sizes and consequential uses.
System changes
- Creates a change-controlled release manifest and evidence directory but does not start or expose inference.
Syntax explained
sha256sum -c- Verifies each file against the reviewed local manifest; it does not prove who published the manifest, so protect the manifest through the organization change-control or signing process.
immutable revision- A content-addressed commit, digest or archived release identifier that cannot silently move like a branch, tag or latest label.
Values stay on this page and are never sent or saved.
sha256sum -c {{modelDir}}/SHA256SUMS && sed -n '1,80p' {{modelDir}}/RELEASE.txtmodel.safetensors.index.json: OK tokenizer.json: OK config.json: OK generation_config.json: OK release=vllm-prod-2026-07-28-r1 model_revision=8f3b1c9e
Checkpoint: Checkpoint: Freeze the serving contract and model provenance
Continue whenEvery required file reports OK, the release record contains an immutable revision and license decision, and an operator can name the exact previous release without consulting a mutable registry.
Stop whenAny file is missing or mismatched, the license or data classification is unresolved, the tokenizer/template is not recorded, or rollback depends on downloading an artifact during the incident.
If this step fails
The downloaded model directory exists, but its checksum or recorded revision differs from the approved release manifest.
Likely causeA mutable branch or tag was downloaded, an interrupted transfer left partial files, a cache entry was reused across revisions, or the manifest was produced from a different tokenizer or generation configuration.
Compare the approved source, revision, license record and every expected filename without starting the server.Run sha256sum -c against the signed or change-controlled local manifest and retain the exact mismatch.Inspect whether symlinks or cache paths resolve outside the reviewed immutable release directory.
ResolutionQuarantine the artifact, clear only the isolated failed staging directory, fetch the approved immutable revision through the controlled import path, regenerate checksums from the reviewed source, and restart validation from provenance rather than accepting a similar model name.
Security notes
- Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
- Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
- Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
- Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
- A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.
Alternatives
- Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.
Stop conditions
- Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
- Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
- Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
- Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.
instruction
Size weights, KV cache, concurrency, and operating headroom
Build a capacity worksheet from measured artifacts and a representative workload. Include weight memory, runtime and graph overhead, KV cache at the accepted context and concurrency distribution, temporary initialization peaks, tensor or pipeline replication, non-model GPU users, host RAM, local storage, network bandwidth, and at least fifteen percent operational headroom. Nameplate VRAM and parameter count are screening inputs, not a production sizing result.
Why this step matters
vLLM uses remaining GPU memory for KV cache and scheduling. A model that barely loads can collapse under real prompt lengths and concurrency. Capacity must therefore be accepted from load tests that preserve both latency and error objectives, not from a successful startup.
What to understand
Use redacted prompt-length and output-length distributions rather than a single average; long-tail requests usually dominate memory and queue behavior.
Measure cold initialization separately from steady state and include the largest transient allocation observed during graph capture or worker startup.
Choose single GPU when the evaluated model fits with headroom; add tensor or pipeline parallelism only for a documented capacity or throughput reason.
System changes
- Creates the capacity envelope that will become proxy admission limits and monitoring thresholds; it does not change GPU allocation yet.
Syntax explained
memory.total,memory.free- Shows physical capacity and current availability; neither value alone predicts KV-cache demand under the target request mix.
GPU UUID- Provides a stable device identity for inventory and assignment, preferable to an index that may change after maintenance.
nvidia-smi --query-gpu=index,name,uuid,memory.total,memory.free,driver_version --format=csv,noheader0, NVIDIA H100 PCIe, GPU-2f83d9b0-7c74-4ac4-a523-98e12e45b191, 81559 MiB, 80944 MiB, 575.57.08 1, NVIDIA H100 PCIe, GPU-75b2d600-9204-47e4-b1b3-4a10d73f2bc1, 81559 MiB, 80950 MiB, 575.57.08
Checkpoint: Checkpoint: Size weights, KV cache, concurrency, and operating headroom
Continue whenThe worksheet identifies a tested model length, concurrency, parallel layout, utilization ceiling, p95 TTFT and end-to-end targets, and a rejection point below GPU exhaustion.
Stop whenSizing relies only on parameter count, free memory includes unrelated processes, representative prompt distributions are unavailable, or the plan has no headroom for monitoring and recovery.
If this step fails
The engine reports CUDA out of memory while loading the model before it accepts requests.
Likely causeWeights, temporary initialization allocations, graph capture, runtime overhead, or the requested tensor-parallel layout exceed usable memory, and nameplate VRAM was mistaken for allocatable model capacity.
Record free and total memory with nvidia-smi while no unrelated workload is present.Confirm model dtype, quantization, tensor-parallel size, maximum model length and gpu-memory-utilization.Compare the failure point with a clean-host measurement and the capacity worksheet.
ResolutionReduce one reviewed capacity dimension, select an evaluated smaller or quantized artifact, increase the planned parallel layout, or move to larger GPUs; never hide the failure by raising utilization to the point that the host has no operating headroom.
Security notes
- Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
- Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
- Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
- Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
- A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.
Alternatives
- Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.
Stop conditions
- Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
- Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
- Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
- Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.
verification
Prove the driver, runtime, GPU, and vLLM compatibility set
Validate the exact host driver and GPU architecture against the pinned vLLM runtime before model loading. Capture the image digest or Python lock, CUDA runtime visible to the process, PyTorch CUDA build, vLLM version, GPU UUIDs, and a minimal allocation test. Repeat on every node in a distributed replica. This check is intentionally separate from nvidia-smi because the host utility can work while user-space libraries remain incompatible.
Why this step matters
Driver and runtime compatibility is a release property, not an installation detail. Recording it before model load distinguishes environmental failure from model capacity or serving configuration and makes a working rollback combination reproducible.
What to understand
For containers, record the immutable image digest and the host driver; containers share the host kernel driver even when they package CUDA user-space libraries.
For multi-node serving, compare the full captured environment rather than assuming nodes from the same pool are identical.
Do not upgrade the host driver, runtime, PyTorch and vLLM simultaneously unless the complete combination has already passed a canary.
System changes
- Allocates a minimal CUDA context for verification but does not load model weights or expose a listener.
Syntax explained
torch.cuda.is_available()- Checks whether this reviewed Python environment can initialize CUDA; it is stronger than checking only the host utility but still not a model-capacity test.
vllm.__version__- Captures the serving package version for the release manifest and later metric or CLI compatibility review.
python -c "import torch,vllm; print('vllm',vllm.__version__); print('torch_cuda',torch.version.cuda); print('available',torch.cuda.is_available()); print('devices',torch.cuda.device_count()); print('name',torch.cuda.get_device_name(0))"vllm 0.22.0 torch_cuda 12.8 available True devices 2 name NVIDIA H100 PCIe
Checkpoint: Checkpoint: Prove the driver, runtime, GPU, and vLLM compatibility set
Continue whenEvery node reports the approved vLLM and CUDA combination, the expected number and identity of GPUs, and a successful CUDA initialization with no unexpected device users.
Stop whenVersions differ between nodes, CUDA is unavailable, the GPU architecture is unsupported, or resolving the issue would require an unreviewed production driver change.
If this step fails
vLLM exits during initialization with an unsupported CUDA, driver, compute capability, or shared-library error.
Likely causeThe host driver, CUDA libraries packaged with vLLM, PyTorch build, GPU architecture, and container or virtual-environment revision were selected independently instead of as one tested compatibility set.
Capture nvidia-smi, the GPU model and compute capability, the installed vLLM version, and the exact image or Python lock revision.Compare host driver capability with the CUDA runtime required by the reviewed vLLM release.Run the approved minimal CUDA smoke test before loading model weights.
ResolutionReturn to the last tested driver/runtime/vLLM combination or build a new canary compatibility set; do not replace random libraries on the production host until the matrix passes on an equivalent staging GPU.
Security notes
- Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
- Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
- Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
- Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
- A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.
Alternatives
- Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.
Stop conditions
- Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
- Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
- Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
- Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.
config
Create an isolated service identity and immutable runtime
Run vLLM as a dedicated unprivileged identity with no interactive login, no home-directory secrets, a read-only model release, a writable cache and temporary directory scoped to the service, and only the selected GPU devices. Bind the future listener to loopback. If containers are used, pin the image by digest, drop unnecessary Linux capabilities, deny privilege escalation, use a read-only root filesystem where supported, and mount only explicit paths.
Why this step matters
Inference processes parse large, externally sourced artifacts and accept untrusted request content. A dedicated identity and narrow filesystem/network boundary reduce the blast radius of a parser, runtime, model or application compromise and make service-owned state distinguishable during recovery.
What to understand
Do not grant the inference process write access to the immutable model release; stage a new revision in a separate directory and switch only through the release procedure.
Offline mode prevents an accidental model download during startup from silently changing the reviewed artifact, but the import pipeline must populate all required files beforehand.
Systemd sandbox options vary with GPU access and distribution; apply incrementally on an equivalent canary and document any exception rather than disabling the entire boundary.
System changes
- Creates or modifies a systemd drop-in, service-owned state directories and operating-system permissions. It does not start vLLM.
Syntax explained
ProtectSystem=strict- Makes most of the filesystem read-only for the service, with explicit writable paths added separately.
HF_HUB_OFFLINE=1- Prevents online Hub lookups in this service environment so startup depends on the pre-reviewed local artifact.
UMask=0077- Makes newly created service files private by default; it does not replace correct ownership and mount policy.
/etc/systemd/system/vllm.service.d/10-security.confValues stay on this page and are never sent or saved.
[Service]
User=vllm
Group=vllm
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadOnlyPaths={{modelDir}}
ReadWritePaths=/var/lib/vllm /var/cache/vllm
UMask=0077
Environment=HF_HUB_OFFLINE=1● vllm.service - vLLM inference service
Loaded: loaded (/etc/systemd/system/vllm.service; enabled)
Drop-In: /etc/systemd/system/vllm.service.d/10-security.conf
Active: inactive (dead)
Security: service identity and paths staged; listener not startedCheckpoint: Checkpoint: Create an isolated service identity and immutable runtime
Continue whenThe service identity cannot log in, cannot modify the model release, can write only its approved state paths, sees only assigned GPUs, and has no listener before the serving command is reviewed.
Stop whenThe runtime requires root, broad host mounts, privileged mode, unrestricted device access, online model downloads, or write access to the release artifact without an approved exception.
If this step fails
The hardened service cannot initialize CUDA or read a required model file.
Likely causeA device, supplementary group, directory traversal permission, cache path, or systemd sandbox rule is narrower than the documented runtime requirement.
Inspect journal errors and effective unit properties without disabling all hardening.Verify ownership and each parent-directory execute permission for approved paths.Compare assigned GPU device access with a minimal service-identity CUDA check.
ResolutionAdd the smallest documented path, device or group permission, retest the canary, and record the exception; never solve it with root, privileged mode or a writable host root.
Security notes
- Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
- Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
- Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
- Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
- A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.
Alternatives
- Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.
Stop conditions
- Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
- Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
- Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
- Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.
verification
Stage and inspect the immutable model release offline
Import the approved revision into a new staging directory through the organization artifact path, verify checksums, then inspect configuration, tokenizer, chat template and generation defaults before promotion. Look for remote-code requirements, unexpected executable files, symbolic links outside the directory, mismatched architecture, excessive context defaults, and generation_config values that would silently change sampling. Promote by making the directory read-only and recording its resolved path.
Why this step matters
A model repository is a release bundle rather than one weight file. Tokenizer, template and generation configuration affect safety, correctness and API behavior, while remote code or unexpected links change the execution boundary. Offline inspection prevents startup from becoming the import and trust decision.
What to understand
If a model requires remote code, treat that Python code as a software dependency requiring independent review, pinning and sandboxing; do not enable it merely to make startup succeed.
Explicitly decide whether model-provided generation configuration is accepted or whether vLLM defaults are enforced, then freeze the choice in tests.
Keep staging and active directories separate so a failed import cannot alter the release currently serving users.
System changes
- Populates a new immutable model-release directory and records its metadata; the active service and proxy remain unchanged.
Syntax explained
find -xdev -type l- Lists symbolic links without crossing filesystems so operators can verify that the release does not escape into mutable or unauthorized paths.
jq selected fields- Surfaces architecture, declared context and dtype for review without dumping large or potentially sensitive metadata into routine logs.
Values stay on this page and are never sent or saved.
find {{modelDir}} -xdev -type l -printf '%p -> %l\n'; sha256sum -c {{modelDir}}/SHA256SUMS; jq '{model_type,max_position_embeddings,torch_dtype}' {{modelDir}}/config.jsonmodel.safetensors.index.json: OK
tokenizer.json: OK
config.json: OK
generation_config.json: OK
{
"model_type": "llama",
"max_position_embeddings": 32768,
"torch_dtype": "bfloat16"
}Checkpoint: Checkpoint: Stage and inspect the immutable model release offline
Continue whenAll checksums pass, no link escapes the release, configuration matches the approved architecture and capacity plan, and template/default decisions are recorded before service startup.
Stop whenRemote code is unreviewed, checksums differ, files resolve outside the release, model configuration contradicts the capacity plan, or the tokenizer and template provenance cannot be established.
If this step fails
The downloaded model directory exists, but its checksum or recorded revision differs from the approved release manifest.
Likely causeA mutable branch or tag was downloaded, an interrupted transfer left partial files, a cache entry was reused across revisions, or the manifest was produced from a different tokenizer or generation configuration.
Compare the approved source, revision, license record and every expected filename without starting the server.Run sha256sum -c against the signed or change-controlled local manifest and retain the exact mismatch.Inspect whether symlinks or cache paths resolve outside the reviewed immutable release directory.
ResolutionQuarantine the artifact, clear only the isolated failed staging directory, fetch the approved immutable revision through the controlled import path, regenerate checksums from the reviewed source, and restart validation from provenance rather than accepting a similar model name.
Security notes
- Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
- Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
- Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
- Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
- A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.
Alternatives
- Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.
Stop conditions
- Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
- Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
- Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
- Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.
config
Start vLLM on loopback with explicit capacity controls
Create the serving command from the release manifest rather than accepting defaults. Use the immutable local model path, a stable public alias, loopback host, selected port, reviewed dtype, explicit maximum model length, tensor-parallel size, GPU-memory utilization ceiling, and vLLM generation defaults unless model-provided defaults were separately approved. Start only on the canary host and keep the proxy route disabled.
Why this step matters
Explicit controls turn sizing decisions into enforceable runtime behavior and make process arguments auditable. Loopback binding prevents a partially validated service from becoming reachable while quality, capacity and proxy controls are still incomplete.
What to understand
gpu-memory-utilization is a ceiling for model executor use, not a promise that every concurrency and prompt-length combination will fit.
The served model name is an API alias; keep its mapping to the immutable artifact in the release manifest and verify it after every restart.
Generation defaults influence output quality and reproducibility. Using vLLM defaults avoids silently importing model-repository sampling values unless those values were explicitly evaluated.
System changes
- Installs and starts the vLLM systemd service on loopback, allocates GPU memory, and creates local service state; no public route is enabled.
Syntax explained
--max-model-len- Caps the accepted model context used for cache planning; select it from real requirements and measured capacity, not the largest model-advertised value.
--tensor-parallel-size- Shards model computation across the reviewed number of GPUs; it must match device assignment and topology.
--gpu-memory-utilization- Controls the fraction available to the model executor and KV cache while preserving explicit host headroom.
--generation-config vllm- Uses vLLM generation defaults instead of automatically applying repository generation_config.json values.
/etc/systemd/system/vllm.serviceValues stay on this page and are never sent or saved.
[Unit]
Description=vLLM inference service
After=network.target
[Service]
ExecStart=/opt/vllm/bin/vllm serve {{modelDir}} --served-model-name {{modelName}} --host 127.0.0.1 --port {{listenPort}} --dtype auto --max-model-len {{maxModelLen}} --tensor-parallel-size {{tensorParallel}} --gpu-memory-utilization {{gpuMemoryUtilization}} --generation-config vllm
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.targetJul 28 14:02:11 inference-01 vllm[21442]: INFO platform cuda detected Jul 28 14:02:19 inference-01 vllm[21442]: INFO model loading took 7.62 GiB and 8.31 seconds Jul 28 14:02:22 inference-01 vllm[21442]: INFO Available routes include /v1/models and /metrics Jul 28 14:02:22 inference-01 vllm[21442]: INFO Uvicorn running on http://127.0.0.1:8000
Checkpoint: Checkpoint: Start vLLM on loopback with explicit capacity controls
Continue whenThe process loads the approved artifact, binds only to 127.0.0.1, reports the selected model alias, reserves memory below the accepted ceiling, and remains absent from every public route.
Stop whenThe process binds to all interfaces, resolves a remote model, loads an unrecorded template or default, exceeds the memory envelope, or requires removing the service isolation.
If this step fails
The engine reports CUDA out of memory while loading the model before it accepts requests.
Likely causeWeights, temporary initialization allocations, graph capture, runtime overhead, or the requested tensor-parallel layout exceed usable memory, and nameplate VRAM was mistaken for allocatable model capacity.
Record free and total memory with nvidia-smi while no unrelated workload is present.Confirm model dtype, quantization, tensor-parallel size, maximum model length and gpu-memory-utilization.Compare the failure point with a clean-host measurement and the capacity worksheet.
ResolutionReduce one reviewed capacity dimension, select an evaluated smaller or quantized artifact, increase the planned parallel layout, or move to larger GPUs; never hide the failure by raising utilization to the point that the host has no operating headroom.
Security notes
- Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
- Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
- Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
- Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
- A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.
Alternatives
- For a model that does not fit one GPU, evaluate documented tensor or pipeline parallelism on identical GPUs; keep the smallest topology that meets accepted capacity.
- For an initial rehearsal, use a smaller approved model with the same API and template shape before allocating the production weights.
Stop conditions
- Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
- Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
- Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
- Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.
config
Put TLS, authentication, and admission control at the trusted proxy
Configure an independently maintained NGINX proxy as the only network path to loopback vLLM. Terminate modern TLS, authenticate each caller through the organization identity or gateway, reject oversized requests, limit request and connection rates, preserve streaming without broad buffering, set finite timeouts, generate a synthetic request ID, strip spoofable forwarding headers, and keep the backend port unreachable from every user segment. The example is a skeleton: integrate the approved authentication module rather than inventing a shared static password.
Why this step matters
vLLM is an inference server, not the complete public security boundary. A trusted proxy can centralize identity, certificates, rate and size limits, access evidence, and emergency denial while the inference process remains isolated and replaceable.
What to understand
A body-size limit is only a byte limit; also enforce token and context limits at the application or admission layer before expensive GPU work begins.
Authentication must fail closed when its upstream is unavailable, and authorization should distinguish models, tenants and permitted data classes where required.
Do not log Authorization headers, full request bodies or generated text. Retain request IDs, caller identity pseudonyms, status, duration and size according to privacy policy.
System changes
- Adds a TLS reverse-proxy route, authentication subrequest, connection behavior and admission limits; public traffic must remain disabled until all tests pass.
Syntax explained
auth_request- Delegates authentication or authorization to a reviewed internal endpoint; that endpoint and failure behavior must be tested independently.
client_max_body_size- Rejects oversized HTTP bodies before forwarding, reducing memory and abuse exposure; it does not count model tokens.
proxy_buffering off- Allows incremental streaming to clients; validate capacity and disconnect handling because it changes proxy resource use.
X-Forwarded-For $remote_addr- Replaces a caller-supplied forwarding chain with the direct peer address instead of trusting spoofable input.
/etc/nginx/conf.d/{{publicHost}}.confValues stay on this page and are never sent or saved.
limit_req_zone $binary_remote_addr zone=vllm_api:10m rate=5r/s;
upstream vllm_backend { server 127.0.0.1:{{listenPort}}; keepalive 32; }
server {
listen 443 ssl http2;
server_name {{publicHost}};
client_max_body_size 1m;
location /v1/ {
auth_request /_auth;
limit_req zone=vllm_api burst=10 nodelay;
proxy_buffering off;
proxy_read_timeout 300s;
proxy_set_header X-Request-ID $request_id;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
proxy_pass http://vllm_backend;
}
}nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful unauthenticated_status=401 authenticated_models_status=200 backend_from_unapproved_segment=filtered
Checkpoint: Checkpoint: Put TLS, authentication, and admission control at the trusted proxy
Continue whenNGINX validates, unauthenticated requests fail, authenticated synthetic requests work, limits return predictable errors, streaming is incremental, and only the proxy can reach the backend port.
Stop whenAuthentication can fail open, TLS ownership is unclear, request/token limits are absent, the proxy logs secrets or content, or any unapproved segment reaches vLLM directly.
If this step fails
The public hostname returns 401, 403, 413, 429, 502, or 504 while the loopback vLLM endpoint responds.
Likely causeThe trusted proxy has the wrong authentication rule, body limit, rate limit, upstream address, timeout, streaming configuration, or forwarded-header policy.
Test the loopback readiness endpoint locally without sending production content.Correlate one synthetic request ID across proxy access/error logs and the vLLM request log.Inspect the effective proxy configuration and confirm the backend still binds only to loopback.
ResolutionFix the smallest failing proxy control, reload only after configuration validation, and retest authentication, request limits and streaming separately; do not expose the backend as a workaround for a proxy error.
Security notes
- Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
- Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
- Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
- Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
- A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.
Alternatives
- Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.
Stop conditions
- Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
- Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
- Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
- Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.
verification
Verify API identity, chat behavior, and safe failure contracts
Exercise the loopback endpoint first, then the authenticated proxy, with frozen synthetic fixtures. Verify model listing, a deterministic low-temperature completion, chat-template roles, streaming, context rejection, malformed JSON, oversized request, unauthenticated request, rate limit, client cancellation, and an unsupported field. Record status, headers, timing, release alias and a short non-sensitive result. Do not call the service production-ready because one friendly prompt looks plausible.
Why this step matters
Serving correctness includes predictable rejection and cancellation, not only successful generation. Contract fixtures catch wrong aliases, unsupported fields, chat-template drift and proxy behavior before callers build dependencies on an accidental interface.
What to understand
Keep fixture prompts synthetic and stable. Store expected structural properties and task answers, not brittle full free-form text when multiple valid responses exist.
Separate API compatibility from semantic quality: HTTP 200 and valid JSON can still contain an unacceptable answer.
Test the same public hostname and authentication path that clients will use; a loopback-only result cannot validate TLS, identity or limits.
System changes
- Sends synthetic inference and negative requests, consuming a bounded amount of GPU capacity without changing model or proxy configuration.
Syntax explained
--fail-with-body- Makes curl fail on HTTP errors while preserving the short response body needed to diagnose the contract.
jq selected fields- Records only the public alias and structural evidence instead of persisting full responses unnecessarily.
Values stay on this page and are never sent or saved.
curl --fail-with-body -sS http://127.0.0.1:{{listenPort}}/v1/models | jq '{models:[.data[].id]}'{
"models": [
"{{modelName}}"
]
}
contract_fixture=chat-minimal-007 status=200 finish_reason=stop
negative_fixture=unauthenticated status=401
negative_fixture=oversized status=413Checkpoint: Checkpoint: Verify API identity, chat behavior, and safe failure contracts
Continue whenThe intended alias is returned, approved chat roles work, streaming and cancellation release resources, and every malformed, oversized, unauthenticated or over-limit request fails with the documented status.
Stop whenThe wrong alias serves, a negative fixture succeeds, cancellation leaks running work, the template changes fixture meaning, or validation would require production data.
If this step fails
The health endpoint succeeds, but the wrong model, tokenizer, alias, or generation defaults are serving.
Likely causeReadiness checks only process liveness, a mutable cache path was reused, generation_config.json changed defaults, or the public model alias was not tied to an immutable release manifest.
Query the model-list endpoint and compare the served alias with the release record.Run deterministic tokenizer and golden-prompt fixtures that distinguish the approved release.Inspect the process arguments, resolved model directory and generation configuration.
ResolutionRemove the instance from service, start it from the approved immutable directory with explicit defaults, repeat quality and API-contract checks, and restore traffic only when the release identity is independently proven.
Security notes
- Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
- Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
- Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
- Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
- A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.
Alternatives
- Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.
Stop conditions
- Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
- Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
- Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
- Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.
verification
Benchmark batching and admission under representative load
Replay a versioned, redacted workload that preserves prompt lengths, output limits, streaming ratio and concurrency. Warm the model, then increase admitted concurrency in controlled stages. Record request success, queue time, TTFT, inter-token latency, end-to-end latency, input/output token throughput, GPU utilization, memory and KV-cache pressure. Select the highest operating point that meets SLOs with headroom, then configure the proxy or scheduler to reject above that point.
Why this step matters
Continuous batching trades latency, throughput and cache capacity. Only a workload-shaped benchmark reveals where queueing and memory pressure violate service objectives; a maximum-throughput number without failures, percentiles and workload shape is operationally misleading.
What to understand
Random data is safe for infrastructure calibration but must be complemented by frozen task fixtures for semantic quality and by production-shaped length distributions.
Change one parameter at a time and retain the baseline; otherwise a faster result cannot be attributed or rolled back confidently.
Include overload behavior. The accepted operating point is below the first error or SLO breach, not exactly at the GPU saturation cliff.
System changes
- Generates bounded synthetic load and may temporarily consume all canary capacity; it does not change runtime settings by itself.
Syntax explained
--num-prompts 200- Runs enough requests to observe queue and latency behavior; increase only when the canary window and statistics plan justify it.
--request-rate 4- Controls offered load independently from concurrency so operators can identify the point where waiting work grows.
--random-input-len / --random-output-len- Creates synthetic lengths for infrastructure tests; use redacted distribution fixtures for final production capacity.
Values stay on this page and are never sent or saved.
vllm bench serve --backend vllm --base-url http://127.0.0.1:{{listenPort}} --model {{modelName}} --dataset-name random --num-prompts 200 --request-rate 4 --random-input-len 1024 --random-output-len 256============ Serving Benchmark Result ============ Successful requests: 200 Failed requests: 0 Request throughput (req/s): 3.96 Output token throughput (tok/s): 982.41 Mean TTFT (ms): 122.68 P95 TTFT (ms): 189.44 P95 TPOT (ms): 37.91 P99 E2EL (ms): 10483.20
Checkpoint: Checkpoint: Benchmark batching and admission under representative load
Continue whenThe accepted rate and concurrency meet success and latency SLOs, KV-cache use stays below the rollback threshold, and predictable proxy admission prevents overload beyond the envelope.
Stop whenThe workload contains sensitive content, failures or SLO breaches begin below required demand, KV cache approaches exhaustion, or benchmark and production length distributions are materially different.
If this step fails
Time to first token or queue time grows sharply while GPU utilization appears high.
Likely causeThe replica is saturated, prompts are longer than planned, batching favors throughput over latency, cache pressure causes preemption, or a single large request monopolizes admission.
Compare queue, running-request, KV-cache, prompt-token, TTFT and end-to-end latency metrics over the same interval.Replay the benchmark workload and one long-request stress case separately.Check whether proxy concurrency and token limits match the accepted capacity envelope.
ResolutionApply the previously measured admission limit, scale replicas or hardware, or tune one batching/cache parameter in a canary; preserve before-and-after workload distributions so an apparent latency gain is not merely rejected work.
Security notes
- Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
- Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
- Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
- Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
- A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.
Alternatives
- Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.
Stop conditions
- Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
- Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
- Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
- Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.
config
Monitor capacity, latency, errors, and release identity
Scrape the private vLLM metrics endpoint from an authorized monitoring network and build versioned recording rules, dashboards and alerts. At minimum watch running and waiting requests, KV-cache utilization, prompt and generation tokens, request success by finish reason, queue time, TTFT, inter-token latency, end-to-end latency, prefill/decode time, process restarts, GPU memory/temperature/errors, proxy authentication failures, rate limits and backend errors. Attach release and host identity without putting user or prompt values in metric labels.
Why this step matters
Engine metrics explain whether latency comes from queueing, prefill, decode or cache pressure, while proxy and GPU signals show admission and hardware context. Versioned queries are necessary because metric names and availability may change with vLLM engine or release revisions.
What to understand
Alert on sustained symptoms and error-budget impact rather than one noisy sample; include a runbook link and the current release manifest in every actionable alert.
Keep the metrics endpoint private. Prometheus access does not justify exposing process, model or capacity details publicly.
Test dashboards and alerts during canary load so a green dashboard proves the queries work rather than merely the absence of data.
System changes
- Adds private scraping, retained time series, dashboards and alerts; this may expose operational metadata to the monitoring system and must follow its access and retention policy.
Syntax explained
vllm:kv_cache_usage_perc- Shows the fraction of cache blocks used and helps correlate prompt/concurrency demand with preemption or OOM risk.
TTFT, TPOT and E2E- Separates initial queue/prefill responsiveness, per-token decode pace, and total user-visible latency.
bounded labels- Use model alias, release and host; never use request IDs, callers, prompts or generated text as high-cardinality metric labels.
Values stay on this page and are never sent or saved.
curl --fail -sS http://127.0.0.1:{{listenPort}}/metrics | grep -E '^vllm:(num_requests_running|num_requests_waiting|kv_cache_usage_perc|time_to_first_token_seconds)' | headvllm:num_requests_running{model_name="{{modelName}}"} 0.0
vllm:num_requests_waiting{model_name="{{modelName}}"} 0.0
vllm:kv_cache_usage_perc{model_name="{{modelName}}"} 0.03125
vllm:time_to_first_token_seconds_bucket{le="0.25",model_name="{{modelName}}"} 194.0Checkpoint: Checkpoint: Monitor capacity, latency, errors, and release identity
Continue whenPrometheus targets are healthy, dashboards show canary load, alerts fire and recover in rehearsal, release identity is visible, and no request content or credential appears in labels or logs.
Stop whenMetrics require public exposure, queries silently return no data, labels contain user-controlled text, alerts lack owners, or dashboards are not versioned with the serving release.
If this step fails
Prometheus scrapes no vLLM metrics, or dashboards show missing and renamed series after an upgrade.
Likely causeThe scrape path or network policy is wrong, the endpoint is protected differently than planned, metric names changed between releases, or the dashboard assumes a different engine mode.
Curl the private metrics endpoint locally and record its HTTP status and first vllm-prefixed series.Inspect Prometheus target health and scrape error without opening the endpoint publicly.Compare the new release metrics inventory with alert and dashboard queries in staging.
ResolutionRestore private scrape connectivity or update versioned dashboard and alert rules alongside the serving release; keep old rules available for rollback and never expose the metrics endpoint to the internet to make scraping convenient.
Security notes
- Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
- Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
- Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
- Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
- A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.
Alternatives
- Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.
Stop conditions
- Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
- Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
- Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
- Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.
decision
Canary upgrades as complete release bundles
Build every upgrade as a new immutable bundle: driver compatibility record, image or Python lock, vLLM version, model/tokenizer/template revision, explicit arguments, proxy policy, client contract, dashboards, fixtures and rollback target. Start it beside the current release, keep it off the main route, run provenance, API, quality, security, capacity and failure tests, then send a small synthetic and approved production-shadow sample. Promote gradually while comparing old and new by workload segment.
Why this step matters
A serving release spans more than the vLLM package. Canarying the complete bundle makes behavior changes observable before broad traffic and preserves the old instance as an immediate rollback target instead of reconstructing it under pressure.
What to understand
Never use a mutable latest image or model branch for the candidate. Resolve and record the digest before testing.
Keep quality and capacity gates independent: an upgrade can improve throughput while degrading task accuracy, refusals or template behavior.
Define automatic stop thresholds for errors, TTFT, queue, GPU faults and quality fixtures, plus a human owner who can drain the candidate.
System changes
- Creates a parallel candidate instance and may route a bounded traffic fraction after all offline gates pass; the former release remains warm and reversible.
Syntax explained
diff release manifests- Shows every intended material change; an unexplained difference is a release blocker rather than acceptable drift.
gradual promotion- Moves traffic in measured stages with explicit observation windows and rollback thresholds instead of replacing all replicas at once.
Values stay on this page and are never sent or saved.
diff -u /srv/vllm/releases/{{previousRelease}}/RELEASE.txt /srv/vllm/releases/{{candidateRelease}}/RELEASE.txt || true--- /srv/vllm/releases/prod-r17/RELEASE.txt +++ /srv/vllm/releases/prod-r18/RELEASE.txt -vllm=0.21.1 +vllm=0.22.0 -model_revision=8f3b1c9e +model_revision=8f3b1c9e generation_defaults=vllm max_model_len=8192
Checkpoint: Checkpoint: Canary upgrades as complete release bundles
Continue whenThe candidate passes all frozen fixtures and load gates, no unexplained manifest drift exists, metrics are comparable, and traffic can return to the still-running previous release in one documented action.
Stop whenMultiple unexplained variables changed, dashboards are not comparable, the old release was removed, quality evidence is missing, or any rollback threshold is crossed.
If this step fails
A canary upgrade passes smoke tests but changes quality, latency, memory use, sampling defaults, or metrics under representative load.
Likely causeOnly liveness was tested, the model or generation defaults drifted, engine behavior changed, compiled kernels differ, or the canary workload missed important prompt and concurrency segments.
Diff model, tokenizer, generation config, container or environment lock, vLLM arguments and proxy policy.Compare frozen quality fixtures and load-test percentiles between old and canary revisions.Check GPU memory, queue, TTFT, token throughput, error and refusal metrics for each workload segment.
ResolutionDrain the canary, restore the previous immutable release, preserve evidence, classify the regression, and expand the evaluation fixture before attempting another single-variable upgrade.
Security notes
- Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
- Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
- Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
- Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
- A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.
Alternatives
- Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.
Stop conditions
- Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
- Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
- Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
- Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.
warning
Rehearse rollback, quarantine, and post-incident evidence
Before general availability, simulate a bad model response, rising queue, proxy authentication failure and startup regression. Remove the candidate from routing, drain or cancel bounded work, restore the previous proxy upstream and complete release, verify model identity and golden fixtures, then preserve the failed bundle, logs, metrics and decision timeline under restricted access. Rotate any exposed credential and invalidate compromised artifacts rather than merely restarting them.
Why this step matters
Rollback that exists only in prose is not an availability control. A timed rehearsal discovers hidden dependencies such as mutable caches, incompatible proxy contracts, missing dashboards or an old artifact that cannot start before a real incident forces improvisation.
What to understand
Drain first when safe, but prioritize containment when the release exposes data, authorization or network boundaries; the incident commander decides which objective wins.
Verify the restored release semantically and operationally. A running old process is insufficient if the proxy, alias, cache or dashboard still references the candidate.
Preserve only necessary redacted evidence under retention and access controls; do not turn incident collection into a new repository of prompts or generated personal data.
System changes
- Stops the candidate, changes active proxy routing, restores the former release, and quarantines failed artifacts and evidence. In-flight candidate requests may be interrupted.
Syntax explained
systemctl stop candidate- Prevents new local work on the failed instance after routing is removed; confirm no other service name or replica remains active.
nginx -t before reload- Validates syntax before changing live routing; semantic upstream and authentication tests are still required afterward.
post-rollback model query- Confirms the public route returns the expected alias, followed by golden fixtures that identify the exact restored release.
Values stay on this page and are never sent or saved.
sudo systemctl stop vllm-candidate.service && sudo nginx -t && sudo systemctl reload nginx && curl --fail -sS https://{{publicHost}}/v1/models | jq -r '.data[].id'nginx: configuration file /etc/nginx/nginx.conf test is successful
{{modelName}}
rollback_release={{previousRelease}}
synthetic_contract=PASS
candidate_state=quarantinedCheckpoint: Checkpoint: Rehearse rollback, quarantine, and post-incident evidence
Continue whenOperators restore the previous full bundle within the recovery objective, the candidate is unreachable and quarantined, quality and API fixtures pass, and evidence supports a root-cause review without leaking user content.
Stop whenThe command targets are ambiguous, the previous release is not independently verified, stopping the candidate would destroy the only evidence, or proxy reload would broaden exposure.
If this step fails
Rollback starts the former binary but the model cache, proxy route, dashboard, or client contract still points at the failed release.
Likely causeRollback was defined as reinstalling one package rather than restoring the complete release bundle and traffic state.
Compare all running process arguments, artifact checksums, proxy upstreams, dashboards and aliases with the previous release manifest.Run the old golden prompt and API-contract fixtures.Verify new requests reach only the restored instance while the failed release remains quarantined.
ResolutionRestore every component from the previous manifest, restart or reload in dependency order, verify release identity and quality, then re-enable traffic gradually; document any state that could not be reversed automatically.
Security notes
- Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
- Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
- Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
- Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
- A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.
Alternatives
- During a non-security performance regression, first set candidate traffic weight to zero and allow bounded requests to drain before stopping the process.
- If both releases are unsafe, fail closed with a documented maintenance response rather than routing to an unverified model.
Stop conditions
- Stop before issuing service or proxy commands unless host, unit, release paths, traffic weights and the incident decision are explicitly confirmed.
- Stop general availability until the rollback rehearsal meets the recovery objective and the restored public route passes identity, contract and representative quality checks.
Finish line
Verification checklist
ss -ltnp | grep ':{{listenPort}} 'The vLLM listener is bound only to 127.0.0.1 or the explicitly approved private service address, and an unapproved segment cannot connect.curl --fail -sS https://{{publicHost}}/v1/modelsAuthenticated callers receive only the approved alias; unauthenticated, oversized and over-limit fixtures fail with documented statuses.run-vllm-release-gates {{candidateRelease}}Frozen task fixtures meet accepted quality, refusal and API thresholds while p95 TTFT, queue, error, KV-cache and GPU metrics remain inside the capacity envelope.test -r /srv/vllm/releases/{{previousRelease}}/RELEASE.txtThe complete prior release is locally readable, its checksums pass, routing instructions are current, and the last rehearsal met the recovery objective.Recovery guidance
Common problems and safe checks
The downloaded model directory exists, but its checksum or recorded revision differs from the approved release manifest.
Likely causeA mutable branch or tag was downloaded, an interrupted transfer left partial files, a cache entry was reused across revisions, or the manifest was produced from a different tokenizer or generation configuration.
Compare the approved source, revision, license record and every expected filename without starting the server.Run sha256sum -c against the signed or change-controlled local manifest and retain the exact mismatch.Inspect whether symlinks or cache paths resolve outside the reviewed immutable release directory.
ResolutionQuarantine the artifact, clear only the isolated failed staging directory, fetch the approved immutable revision through the controlled import path, regenerate checksums from the reviewed source, and restart validation from provenance rather than accepting a similar model name.
vLLM exits during initialization with an unsupported CUDA, driver, compute capability, or shared-library error.
Likely causeThe host driver, CUDA libraries packaged with vLLM, PyTorch build, GPU architecture, and container or virtual-environment revision were selected independently instead of as one tested compatibility set.
Capture nvidia-smi, the GPU model and compute capability, the installed vLLM version, and the exact image or Python lock revision.Compare host driver capability with the CUDA runtime required by the reviewed vLLM release.Run the approved minimal CUDA smoke test before loading model weights.
ResolutionReturn to the last tested driver/runtime/vLLM combination or build a new canary compatibility set; do not replace random libraries on the production host until the matrix passes on an equivalent staging GPU.
The engine reports CUDA out of memory while loading the model before it accepts requests.
Likely causeWeights, temporary initialization allocations, graph capture, runtime overhead, or the requested tensor-parallel layout exceed usable memory, and nameplate VRAM was mistaken for allocatable model capacity.
Record free and total memory with nvidia-smi while no unrelated workload is present.Confirm model dtype, quantization, tensor-parallel size, maximum model length and gpu-memory-utilization.Compare the failure point with a clean-host measurement and the capacity worksheet.
ResolutionReduce one reviewed capacity dimension, select an evaluated smaller or quantized artifact, increase the planned parallel layout, or move to larger GPUs; never hide the failure by raising utilization to the point that the host has no operating headroom.
The model loads, but requests fail with out-of-memory errors only when concurrency or prompt length rises.
Likely causeKV-cache demand and in-flight activation memory were omitted from sizing, request admission is unbounded, or the production length and concurrency distribution differs from the benchmark.
Inspect vllm:kv_cache_usage_perc, waiting and running request counts, prompt lengths, and the first failing concurrency level.Replay a redacted frozen workload at increasing concurrency while holding model and configuration constant.Confirm the proxy enforces the reviewed request-size, token and concurrency policy.
ResolutionLower admitted concurrency or maximum length, add capacity, or select a tested cache configuration; make the proxy reject excess work predictably instead of allowing GPU OOM to become the admission controller.
A multi-GPU replica fails with NCCL, peer-access, topology, or worker startup errors.
Likely causeTensor parallelism was chosen without verifying GPU interconnect topology, devices are inconsistently visible, processes use different environments, or the requested parallel size does not match the assigned GPUs.
List GPU UUIDs, topology and peer access on every participating host.Verify identical vLLM, driver, CUDA, model-path and environment revisions for all workers.Reduce to a one-GPU canary and then add one reviewed parallel dimension at a time.
ResolutionCorrect device assignment and topology, use pipeline parallelism when the documented topology calls for it, or return to the last known single-node layout; never bypass NCCL errors by exposing unreviewed host networking or privileged containers.
The public hostname returns 401, 403, 413, 429, 502, or 504 while the loopback vLLM endpoint responds.
Likely causeThe trusted proxy has the wrong authentication rule, body limit, rate limit, upstream address, timeout, streaming configuration, or forwarded-header policy.
Test the loopback readiness endpoint locally without sending production content.Correlate one synthetic request ID across proxy access/error logs and the vLLM request log.Inspect the effective proxy configuration and confirm the backend still binds only to loopback.
ResolutionFix the smallest failing proxy control, reload only after configuration validation, and retest authentication, request limits and streaming separately; do not expose the backend as a workaround for a proxy error.
A network scan reaches the vLLM port directly from a user, office, or internet segment.
Likely causeThe server bound to all interfaces, a container published the port, a firewall rule is broader than intended, or a load balancer targets an unprotected backend address.
Inspect listening addresses and container port mappings on the inference host.Test reachability from both an approved proxy host and an explicitly unapproved segment.Review host, cloud and network firewall rules against the documented service flow.
ResolutionRemove the public mapping, bind to loopback or the private service interface, restrict firewall sources to the proxy, rotate any backend credential that may have been exposed, and review access logs before restoring traffic.
The health endpoint succeeds, but the wrong model, tokenizer, alias, or generation defaults are serving.
Likely causeReadiness checks only process liveness, a mutable cache path was reused, generation_config.json changed defaults, or the public model alias was not tied to an immutable release manifest.
Query the model-list endpoint and compare the served alias with the release record.Run deterministic tokenizer and golden-prompt fixtures that distinguish the approved release.Inspect the process arguments, resolved model directory and generation configuration.
ResolutionRemove the instance from service, start it from the approved immutable directory with explicit defaults, repeat quality and API-contract checks, and restore traffic only when the release identity is independently proven.
Time to first token or queue time grows sharply while GPU utilization appears high.
Likely causeThe replica is saturated, prompts are longer than planned, batching favors throughput over latency, cache pressure causes preemption, or a single large request monopolizes admission.
Compare queue, running-request, KV-cache, prompt-token, TTFT and end-to-end latency metrics over the same interval.Replay the benchmark workload and one long-request stress case separately.Check whether proxy concurrency and token limits match the accepted capacity envelope.
ResolutionApply the previously measured admission limit, scale replicas or hardware, or tune one batching/cache parameter in a canary; preserve before-and-after workload distributions so an apparent latency gain is not merely rejected work.
Prometheus scrapes no vLLM metrics, or dashboards show missing and renamed series after an upgrade.
Likely causeThe scrape path or network policy is wrong, the endpoint is protected differently than planned, metric names changed between releases, or the dashboard assumes a different engine mode.
Curl the private metrics endpoint locally and record its HTTP status and first vllm-prefixed series.Inspect Prometheus target health and scrape error without opening the endpoint publicly.Compare the new release metrics inventory with alert and dashboard queries in staging.
ResolutionRestore private scrape connectivity or update versioned dashboard and alert rules alongside the serving release; keep old rules available for rollback and never expose the metrics endpoint to the internet to make scraping convenient.
Chat requests fail or produce malformed turns while plain completions still work.
Likely causeThe model lacks a compatible chat template, a template revision differs from the tokenizer, message roles are unsupported, or the proxy/client sends fields that this vLLM release does not implement.
Inspect the exact model/tokenizer/template release without including user conversation text.Run the frozen minimal system-user-assistant fixture against the loopback endpoint.Compare requested fields with the documented API support for the pinned vLLM version.
ResolutionPin an approved chat template with the model release, adjust the client contract, or expose only the completion endpoint for a base model; do not invent a production template without representative quality and safety evaluation.
Streaming responses stop early, arrive buffered, or leave requests running after clients disconnect.
Likely causeThe proxy buffers upstream data, has an incompatible timeout, does not propagate disconnects, or application and serving versions disagree on streaming framing.
Test a short synthetic streaming request directly on loopback and through the proxy.Inspect proxy buffering and timeout configuration plus active request metrics after cancelling the client.Correlate request IDs without logging generated content.
ResolutionUse the reviewed streaming proxy settings, align timeouts with explicit maximum generation limits, verify cancellation releases capacity, and roll back the proxy or server revision if the accepted contract cannot be restored.
A canary upgrade passes smoke tests but changes quality, latency, memory use, sampling defaults, or metrics under representative load.
Likely causeOnly liveness was tested, the model or generation defaults drifted, engine behavior changed, compiled kernels differ, or the canary workload missed important prompt and concurrency segments.
Diff model, tokenizer, generation config, container or environment lock, vLLM arguments and proxy policy.Compare frozen quality fixtures and load-test percentiles between old and canary revisions.Check GPU memory, queue, TTFT, token throughput, error and refusal metrics for each workload segment.
ResolutionDrain the canary, restore the previous immutable release, preserve evidence, classify the regression, and expand the evaluation fixture before attempting another single-variable upgrade.
Rollback starts the former binary but the model cache, proxy route, dashboard, or client contract still points at the failed release.
Likely causeRollback was defined as reinstalling one package rather than restoring the complete release bundle and traffic state.
Compare all running process arguments, artifact checksums, proxy upstreams, dashboards and aliases with the previous release manifest.Run the old golden prompt and API-contract fixtures.Verify new requests reach only the restored instance while the failed release remains quarantined.
ResolutionRestore every component from the previous manifest, restart or reload in dependency order, verify release identity and quality, then re-enable traffic gradually; document any state that could not be reversed automatically.
Reference
Frequently asked questions
Is the vLLM API key enough for an internet-facing production service?
No. Keep vLLM private and use a maintained trusted boundary for TLS, identity, authorization, admission, request-size controls, audit evidence and emergency denial. A static backend key does not provide a complete tenant or network security model.
Should gpu-memory-utilization be set as high as possible?
No. Select it from measured initialization, cache, concurrency and failure behavior while preserving operating headroom. A value that maximizes a benchmark may make the service fragile during long prompts, monitoring, graph capture or transient allocations.
Does HTTP 200 mean the new model release is correct?
It proves only that a request produced a syntactically successful response. Verify immutable identity, tokenizer/template, API negatives, representative task quality, safety behavior, latency, overload, monitoring and rollback separately.
When should tensor parallelism be used?
Use a single GPU when the model fits with accepted headroom. Add tensor parallelism when the model or measured throughput requires it and the GPU topology supports efficient communication; evaluate pipeline parallelism for unsuitable or uneven topologies.
Can prompts and outputs be logged for debugging?
Not by default. Prefer request IDs, structural metadata, status, sizes, timing and synthetic reproduction. Any content capture needs explicit purpose, minimization, redaction, access, retention and user/data-owner approval.
Why retain the old dashboard and alert rules during rollback?
Metrics and names can change across vLLM releases. Restoring only the binary while observing it with candidate-only queries can create false health or hide the regression.
Recovery
Rollback
Remove the candidate from routing, drain or stop it according to incident severity, restore the previous immutable model/runtime/proxy/monitoring bundle, verify public identity plus golden fixtures, and quarantine the failed release. Rollback does not undo caller-visible text already returned, so handle harmful output or data exposure through the incident process.
- Declare the rollback target, reason, owners and traffic state; preserve a redacted request ID, manifest diff, metrics window and logs before volatile evidence disappears.
- Set candidate traffic weight to zero. Drain bounded requests for an ordinary performance regression; terminate promptly for credential, authorization, model-integrity or data-exposure incidents.
- Restore the previous proxy upstream and authentication policy from version control, validate syntax, reload, and prove the backend is still unreachable from unapproved networks.
- Start or confirm the previous vLLM runtime from its immutable local release, verify checksums, process arguments, served alias, tokenizer/template and generation defaults.
- Run API negative fixtures, representative quality gates and a short capacity canary through the public hostname before increasing traffic.
- Quarantine the candidate artifact and mutable cache, rotate potentially exposed credentials, retain restricted evidence, and open corrective work before another promotion.
Evidence