AI/ML / System Administration / Security

Choose, verify, and serve GGUF models with llama.cpp

Select a licensed, checksum-backed GGUF and measured quantization, build llama.cpp from a pinned commit, size CPU/GPU resources, validate metadata and quality, isolate llama-server behind a trusted proxy, benchmark real workloads, and roll back the complete binary/model bundle.

240 min11 stepsChanges system stateRevision 1

Save or explore

#llama-cpp #gguf #quantization #local-llm #benchmark #gpu-offload #inference #privacy

0 of 11 steps completed

Goal

Operate a local llama.cpp endpoint whose model and quantization provenance, resource envelope, task quality, API surface, network isolation, metrics, upgrade evidence and former release can be independently verified.

Supported environments

llama.cpp pinned source commit
GGUF checksum-backed evaluated release
Linux or macOS supported compiler and backend

Prerequisites

Approved model provenance Upstream model/license review, immutable source and converter revisions, GGUF checksum, quantization details, tokenizer/template metadata and an accountable quality owner.sha256sum -c {{modelFile}}.sha256
Pinned build toolchain Official llama.cpp source at an immutable commit plus reviewed compiler, CMake and CPU/GPU backend dependencies.git -C {{sourceDir}} rev-parse HEAD
Representative evaluation Frozen task, language, structured-output, refusal, long-context, benchmark and negative API fixtures using synthetic or approved-redacted data.
Trusted ingress and rollback A private loopback service boundary, maintained TLS/authentication proxy, monitoring, former binary/model bundle and owners for incident response.

Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System

A license- and checksum-backed GGUF release that records upstream, converter, quantization, tokenizer/template, evaluation and hardware provenance.
A reproducible llama.cpp binary from a pinned commit and toolchain, with CPU/GPU resource sizing, measured offload, task-quality and performance evidence.
An unprivileged loopback llama-server exposing only the approved /v1 contract through a trusted proxy, with private metrics, overload limits, canary upgrades and complete-bundle rollback.

Observable outcome

Operators choose quantization from explicit memory, speed and task-quality tradeoffs instead of filename fashion or file size alone.
A healthy listener cannot be mistaken for the correct model: checksum, metadata, template, backend and golden fixtures identify the active release.
Local inference remains a network service with authentication, admission and incident obligations, even when model weights never leave the host.

Architecture

How the parts fit together

A controlled import/build path produces an immutable GGUF and llama.cpp executable bundle. Offline metadata, perplexity, task and benchmark gates establish compatibility, quality and capacity. A dedicated llama-server process maps the model and listens on loopback with explicit context, slots and offload. A trusted proxy publishes only authenticated /v1 routes. Private metrics and host telemetry support capacity and rollback decisions.

GGUF release manifestBind upstream license/revision, converter, quantization, checksums, template and quality evidence to exact model bytes.

Pinned llama.cpp buildProvide reproducible native CLI, benchmark, perplexity and server binaries with an explicit CPU/GPU backend.

Offline evaluation harnessCompare quantizations against a higher-quality reference using fixed perplexity, task, language, format and refusal cases.

Dedicated llama-serverLoad the approved model with measured context, threads, offload and slots while binding API, health and metrics to loopback.

Trusted proxy and monitoringExpose only authenticated caller routes, deny management endpoints, enforce admission and observe private throughput, queue and resource signals.

Provenance review selects immutable source/converter/model revisions and verifies GGUF bytes before parsing.
A pinned llama.cpp build inspects metadata and benchmarks reference plus candidate quantizations on target hardware.
Task and quality gates choose the smallest acceptable model/quantization with resource headroom.
llama-server loads the bundle on loopback; the trusted proxy admits authenticated /v1 requests while metrics remain private.
A parallel immutable candidate passes all gates, receives bounded traffic, and can be replaced by the retained former binary/model bundle.

Assumptions

The organization has an approved source-import process and model-license review. This guide deliberately avoids server-side model downloads in production.
Linux systemd and NGINX are the worked serving example. macOS can use the same provenance, evaluation and private-listener controls with its supported launch and Metal backend tooling.
The target may be CPU-only or GPU-accelerated. Resource and quality evidence must come from the exact production backend and hardware.
Local execution does not make generated text trusted or automatically private; callers, logs, proxy, backups and model license still require policy.

Key concepts

GGUF: A model and metadata file format used by llama.cpp. It carries runtime information but does not by itself authenticate publisher, license or conversion quality.
Quantization: Representing weights at reduced precision to lower storage and memory and often improve speed, with a task-dependent quality cost that must be measured.
Importance matrix: Calibration information that can guide which tensors need more precision during quantization. Its dataset and conversion provenance affect the result.
Prompt versus generation throughput: Prompt processing measures ingestion/prefill tokens per second; generation measures sequential decode. Different hardware and settings can optimize each differently.
GPU layer offload: Moving selected model layers to an accelerator. More layers are not always better when VRAM, transfer, cache and backend overhead are considered.

Before you copy

Values used in this guide

{{modelFile}}

Absolute path to the immutable approved GGUF model file.

Example: /srv/llama/models/assistant-q5_k_m.gguf

{{modelAlias}}

Stable public API alias mapped to the exact GGUF checksum.

Example: local-assistant

{{llamaCommit}}

Immutable upstream llama.cpp commit reviewed for this release.

Example: a17c92f1f0d6f9f881a47ec6d2a37e1330e7b117

{{sourceDir}}

Controlled local checkout of official llama.cpp source.

Example: /srv/llama/src/llama.cpp

{{buildDir}}

Candidate out-of-tree CMake build directory.

Example: /srv/llama/build/a17c92f-cuda

{{buildJobs}}

Build parallelism selected for host capacity without disrupting active services.

Example: 12

{{ctxSize}}

Evaluated operational context size, not automatically the model maximum.

Example: 8192

{{threads}}

Measured CPU thread count for the selected hardware and workload.

Example: 12

{{gpuLayers}}

Measured number of layers offloaded while preserving VRAM headroom.

Example: 28

{{parallelSlots}}

Accepted concurrent server slots within memory and latency SLOs.

Example: 2

{{listenPort}}

Loopback llama-server port.

Example: 8080

{{publicHost}}

TLS hostname of the trusted proxy.

Example: local-llm.example.net

{{candidateRelease}}

Immutable candidate binary/model release directory.

Example: llama-r12

{{previousRelease}}

Retained known-good full release directory.

Example: llama-r11

Security and production boundaries

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Stop before continuing if

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

decision

Select a licensed model and immutable GGUF provenance

read-only

Define the permitted task, languages, data classification and quality floor before selecting a file. Record the upstream model and license, original revision, converter and llama.cpp revision, quantization method and parameters, importance-matrix provenance if used, GGUF filename and SHA-256, tokenizer/template metadata, publisher identity, intended hardware and accountable owner. Treat a community filename such as Q4_K_M as a claim to verify, not provenance.

Why this step matters

GGUF packages model tensors and metadata, but the format does not authenticate who created the file or whether the license and conversion are acceptable. A release manifest ties the operational artifact to its upstream and evaluation evidence.

What to understand

Prefer an artifact built by the organization or a publisher with verifiable checksums and conversion details. Preserve original higher-precision input provenance for future re-quantization.

Do not re-quantize an already quantized artifact unless the documented method and quality evaluation explicitly accept compounded loss.

Store the model read-only in a release directory; use a separate staging directory for imports and candidate files.

System changes

Creates a model release and license record but does not build llama.cpp, load the GGUF or expose a listener.

Syntax explained

sha256sum -c: Verifies bytes against the protected release manifest; it does not establish publisher identity by itself.
quantization label: Describes a tensor precision strategy. The same label does not guarantee identical conversion settings, importance matrix or task quality.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Model File{{modelFile}}Absolute path to the immutable approved GGUF model file.

sha256sum -c {{modelFile}}.sha256 && sed -n '1,100p' {{modelFile}}.release.txt

Example output / evidence

/srv/llama/models/assistant-q5_k_m.gguf: OK
upstream_revision=3c8a5e91
converter_commit={{llamaCommit}}
quantization=Q5_K_M
license_review=approved
chat_template_revision=template-r4

Checkpoint: Checkpoint: Select a licensed model and immutable GGUF provenance

Continue whenThe exact GGUF bytes, upstream/converter revisions, license, tokenizer/template and quality owner are recorded, and the former accepted model remains locally available.

Stop whenThe source, license, checksum, converter revision, quantization details or tokenizer/template cannot be verified, or the artifact was produced from an unknown quantized parent.

If this step fails

The GGUF file downloads successfully, but its checksum or recorded source revision does not match the approved manifest.

Likely causeA mutable model tag moved, the wrong quantization variant was selected, a transfer is incomplete, a shared cache reused another file, or the manifest was copied from a different release.

Safe checks

Compare filename, byte size, upstream model and conversion revision, quantization label, license and expected SHA-256.
Run sha256sum -c from the isolated staging directory.
Inspect symlinks and ensure the active service cannot replace the file.

ResolutionQuarantine the file, fetch the exact approved artifact through the controlled import path, verify its manifest and license, and repeat metadata and quality evaluation before activation.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

decision

Choose quantization from measured quality and hardware limits

read-only

Shortlist at least one higher-quality reference and candidate quantizations that fit the target. Compare file size, RAM/VRAM after load, prompt and generation speed, perplexity or distribution divergence when meaningful, and frozen task-level accuracy, structure, language and refusal behavior. Select the smallest candidate that still meets every task threshold with margin; do not choose the smallest file first and invent acceptance afterward.

Why this step matters

Quantization reduces memory and may improve speed, but loss is model-, tensor-, task- and language-dependent. Perplexity can reveal broad degradation yet cannot replace task evaluation, especially for structured output, coding, safety and rare-language behavior.

What to understand

Use the same llama.cpp commit, template, seed and generation settings for reference and candidate comparisons.

Inspect failures, not only aggregate scores; one critical formatting or refusal regression may outweigh a small average gain.

If no candidate fits while meeting quality, select a smaller upstream model, larger hardware or lower concurrency rather than weakening hidden thresholds.

System changes

Runs offline evaluation and reads model files; it may consume substantial CPU/GPU time but does not alter the active service.

Syntax explained

llama-perplexity: Measures next-token prediction performance on a fixed text corpus; useful for relative comparison but not a direct measure of every production task.
-c {{ctxSize}}: Uses the reviewed evaluation context; keep it identical across candidates and within measured memory.

Command

Fill variables0/3 ready

Values stay on this page and are never sent or saved.

Model File{{modelFile}}Absolute path to the immutable approved GGUF model file.Build Dir{{buildDir}}Candidate out-of-tree CMake build directory.Ctx Size{{ctxSize}}Evaluated operational context size, not automatically the model maximum.

ls -lh {{modelFile}}; ./{{buildDir}}/bin/llama-perplexity -m {{modelFile}} -f /srv/llama/evals/perplexity.txt -c {{ctxSize}}

Example output / evidence

-r--r----- 1 llama llama 9.2G Jul 28 10:12 /srv/llama/models/assistant-q5_k_m.gguf
perplexity: 6.4821 +/- 0.0387
reference_perplexity: 6.3914
relative_delta: 1.42%
task_suite_passed: 238/240

Checkpoint: Checkpoint: Choose quantization from measured quality and hardware limits

Continue whenThe selected quantization meets every named task and quality threshold, fits with operating headroom, and retains a higher-quality reference for future regression analysis.

Stop whenOnly file size or one synthetic prompt was compared, evaluation settings differ, a critical task regresses, or the candidate barely fits without context/concurrency headroom.

If this step fails

A smaller quantization is faster or fits, but task accuracy, formatting, factuality, or refusal behavior regresses.

Likely causeQuantization loss affects important tensors or tasks, a quantized-from-quantized artifact compounded error, the chat template changed, or the evaluation set is too narrow.

Safe checks

Compare candidate and reference with identical runtime, prompt/template and deterministic evaluation settings.
Run perplexity or distribution metrics where appropriate plus task-specific frozen cases.
Review failures by task and language rather than relying only on one aggregate score.

ResolutionReject the quantization, select a higher-quality evaluated type or a smaller base model, and retain the failing cases as regression fixtures; do not trade required quality for an unlabeled size improvement.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

instruction

Calculate RAM, VRAM, context, slots, and storage headroom

read-only

Measure GGUF bytes and build a resource worksheet for model mapping, KV cache at the accepted context and parallel slots, compute buffers, GPU offload, host page cache, binary/build artifacts and rollback copies. Reserve at least fifteen percent headroom and prevent sustained swap. Test CPU-only first, then add GPU layers while observing both RAM and VRAM. The maximum model-advertised context is not automatically the operational context.

Why this step matters

GGUF size approximates weight storage, not total inference memory. Context, slots, cache types and offload determine runtime buffers, while mmap and page cache change apparent process RSS. Explicit sizing avoids OOM and swap becoming the request scheduler.

What to understand

Keep active and previous models plus build artifacts locally if rollback depends on them; storage capacity is part of availability.

Measure cold and warm load, then representative longest prompts and concurrent slots. Do not infer peak memory from an idle health check.

On multi-socket CPU systems, benchmark NUMA and thread placement rather than assuming every core improves throughput.

System changes

Creates a capacity worksheet and reads host state; optional canary tests allocate memory but do not change active routing.

Syntax explained

MemAvailable: A better estimate of memory available without swapping than free alone, still requiring observed runtime headroom.
parallel slots: Server concurrency units that may multiply context/cache demand; admit fewer callers than the first SLO or memory failure.
GPU layers: Layers offloaded to supported accelerators. Increase from a measured baseline while preserving VRAM for cache and buffers.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Model File{{modelFile}}Absolute path to the immutable approved GGUF model file.

stat --printf='model_bytes=%s\n' {{modelFile}}; free -h; df -h /srv/llama; nvidia-smi --query-gpu=uuid,memory.total,memory.free --format=csv,noheader

Example output / evidence

model_bytes=9876543210
               total        used        free      shared  buff/cache   available
Mem:           125Gi        11Gi        96Gi       1.2Gi        18Gi       111Gi
/dev/nvme0n1p3  1.8T       612G       1.1T  36% /srv/llama
GPU-2f83d9b0-7c74-4ac4-a523-98e12e45b191, 24564 MiB, 23980 MiB

Checkpoint: Checkpoint: Calculate RAM, VRAM, context, slots, and storage headroom

Continue whenThe accepted model/context/slot/offload combination meets RAM, VRAM and storage headroom, avoids sustained swap, and has a measured rejection point below failure.

Stop whenSizing counts only GGUF bytes, the host has unrelated load, rollback artifacts do not fit, or the required workload exceeds memory before accepted concurrency.

If this step fails

The model process is killed, swaps heavily, or fails allocation on CPU startup.

Likely causeModel mapping, KV cache at the requested context, temporary buffers, parallel slots and operating-system headroom exceed usable RAM or address space.

Safe checks

Record model byte size, MemAvailable, swap activity, context size, parallel slots and cache types.
Run a one-slot, shorter-context canary while holding the model constant.
Check the kernel journal for OOM-killer evidence.

ResolutionChoose an evaluated smaller quantization or model, reduce accepted context/parallelism, add RAM, or isolate the host; do not depend on sustained swap as production capacity.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

command

Build a pinned llama.cpp commit with explicit backend

caution

Clone or import the official llama.cpp source through the controlled software path, check out the reviewed immutable commit, inspect submodules and build instructions, then compile a Release build with the intended CPU or GPU backend. Keep the source and build directory read-only after acceptance, record compiler/CMake/backend versions, hash installed binaries, and never run a binary fetched from an unverified release mirror.

Why this step matters

llama.cpp changes quickly, and build backends affect performance, compatibility and available devices. A commit-pinned reproducible build makes runtime behavior attributable and rollback independent from whatever the repository contains later.

What to understand

For CPU-only hosts omit the CUDA option and record the selected BLAS/CPU features. Do not ship a binary built with unsupported instructions to older fleet nodes.

Build as an unprivileged user without production secrets and install by copying reviewed binaries into a versioned release, not by running from a mutable source tree.

Treat compiler, CMake and CUDA toolkit as release inputs; a rebuild at the same commit may differ if those tools change.

System changes

Checks out source and produces native executables in a candidate build directory; it can execute compiler toolchains but does not replace active binaries.

Syntax explained

checkout --detach {{llamaCommit}}: Builds the exact reviewed commit without following a mutable branch.
-DGGML_CUDA=ON: Enables the CUDA backend; omit it for CPU-only releases or select another officially supported backend after equivalent testing.
-DCMAKE_BUILD_TYPE=Release: Requests optimized release compilation rather than a debug build with different performance characteristics.

Command

Fill variables0/4 ready

Values stay on this page and are never sent or saved.

Source Dir{{sourceDir}}Controlled local checkout of official llama.cpp source.Llama Commit{{llamaCommit}}Immutable upstream llama.cpp commit reviewed for this release.Build Dir{{buildDir}}Candidate out-of-tree CMake build directory.Build Jobs{{buildJobs}}Build parallelism selected for host capacity without disrupting active services.

git -C {{sourceDir}} checkout --detach {{llamaCommit}} && cmake -S {{sourceDir}} -B {{buildDir}} -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON && cmake --build {{buildDir}} --config Release -j {{buildJobs}}

Example output / evidence

HEAD is now at a17c92f server: tighten model loading validation
-- The C compiler identification is GNU 14.2.0
-- Found CUDAToolkit: /usr/local/cuda (found version "12.8")
-- Including CUDA backend
[100%] Built target llama-server
[100%] Built target llama-bench
[100%] Built target llama-perplexity

Checkpoint: Checkpoint: Build a pinned llama.cpp commit with explicit backend

Continue whenThe source HEAD matches the manifest, build output includes the intended backend and required tools, binary hashes are recorded, and no active path changed.

Stop whenThe commit is mutable or unsigned according to policy, dependencies resolve from unreviewed sources, the backend is absent, compilation warnings indicate unsupported architecture, or a privileged build is required.

If this step fails

llama.cpp reports an unsupported model architecture, tensor type, GGUF version, or missing metadata key.

Likely causeThe GGUF was created by a newer or incompatible converter, the pinned llama.cpp commit predates model support, the file is corrupt, or it is an auxiliary projector rather than the language model.

Safe checks

Capture the pinned llama.cpp commit, build features and the first metadata/error lines.
Verify checksum, model architecture and whether a separate multimodal projector is required.
Test the file with the exact converter/runtime compatibility set used to approve it.

ResolutionUse a GGUF produced by a compatible reviewed converter or stage a newer pinned llama.cpp canary; never patch metadata or enable arbitrary overrides merely to force an unknown model to load.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

verification

Verify GGUF metadata and runtime compatibility offline

read-only

Use the candidate llama-cli to load the GGUF without generating meaningful output. Capture architecture, quantization/tensor summary, context metadata, tokenizer and chat-template information, backend devices and load memory. Compare them with the release manifest and upstream model expectations. Reject unknown metadata overrides, missing tokenizer pieces, an unexpected projector requirement or a runtime that silently falls back from the planned backend.

Why this step matters

Checksum proves bytes, while metadata inspection proves that the pinned runtime interprets those bytes as the intended architecture, tokenizer and context. It catches mislabeled files and compatibility problems before a network service or evaluation result obscures the cause.

What to understand

Record only non-sensitive model metadata. Do not dump embedded templates or proprietary names into broadly visible logs when access policy restricts them.

For multimodal models verify the exact mmproj artifact and its checksum separately; the language GGUF alone does not validate vision or audio components.

Avoid --override-kv unless a documented, evaluated release decision requires it; overriding metadata can conceal an incompatible artifact.

System changes

Maps and parses the candidate GGUF, consuming temporary RAM/VRAM, but performs no service or active-model change.

Syntax explained

-n 0: Loads the model without requesting generated tokens, isolating compatibility and metadata from generation behavior.
--no-warmup: Avoids the normal warmup generation during this metadata-focused check; later benchmarks must include a controlled warmup.

Command

Fill variables0/3 ready

Values stay on this page and are never sent or saved.

Build Dir{{buildDir}}Candidate out-of-tree CMake build directory.Model File{{modelFile}}Absolute path to the immutable approved GGUF model file.Ctx Size{{ctxSize}}Evaluated operational context size, not automatically the model maximum.

{{buildDir}}/bin/llama-cli -m {{modelFile}} -n 0 -c {{ctxSize}} --no-display-prompt --no-warmup

Example output / evidence

llama_model_loader: loaded meta data with 32 key-value pairs and 291 tensors
llama_model_loader: model type = Q5_K - Medium
llm_load_print_meta: model type = 8B
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: model size = 9.20 GiB
load_tensors: offloading 0 repeating layers to GPU
main: llama threadpool initialized, n_threads = 12

Checkpoint: Checkpoint: Verify GGUF metadata and runtime compatibility offline

Continue whenArchitecture, quantization, model size, training context, tokenizer/template and backend behavior match the approved release, with no checksum or parser errors.

Stop whenThe runtime reports an unsupported architecture, unexpected quantization, missing template/tokenizer, silent backend fallback, or metadata different from the reviewed manifest.

If this step fails

llama.cpp reports an unsupported model architecture, tensor type, GGUF version, or missing metadata key.

Safe checks

Capture the pinned llama.cpp commit, build features and the first metadata/error lines.
Verify checksum, model architecture and whether a separate multimodal projector is required.
Test the file with the exact converter/runtime compatibility set used to approve it.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

verification

Benchmark prompt, generation, threads, and GPU offload reproducibly

read-only

Run llama-bench on an idle, thermally stable host with the candidate binary and exact GGUF. Measure prompt processing and generation separately across a small reviewed matrix of threads, batch sizes, context and GPU-layer offload. Use warmup and multiple repetitions, emit JSON or Markdown evidence, and record host power/temperature, binary commit and model checksum. Choose stable median performance with low variance, not the single fastest sample.

Why this step matters

Prompt ingestion and token generation stress different hardware paths. Repeated, versioned measurements expose whether a thread or offload choice is stable and provide a baseline for later binary, model or driver regressions.

What to understand

Benchmark one variable at a time and include a CPU-only or former-release control so an improvement has a meaningful reference.

Observe RAM, VRAM, temperature and throttling; tokens per second obtained at an unstable thermal or memory state are not a production baseline.

Performance acceptance follows quality acceptance. A faster quantization that fails task fixtures cannot be promoted.

System changes

Generates sustained local CPU/GPU load and writes benchmark evidence; it does not alter service configuration.

Syntax explained

-p 512 and -n 128: Measure prompt processing and 128-token generation as separate benchmark cases; add workload-shaped sizes for final capacity.
-r 5: Repeats each configuration to expose variance after warmup rather than reporting one noisy observation.
-ngl {{gpuLayers}}: Offloads the reviewed number of layers; increase only while measuring VRAM headroom and end-to-end performance.

Command

Fill variables0/4 ready

Values stay on this page and are never sent or saved.

Build Dir{{buildDir}}Candidate out-of-tree CMake build directory.Model File{{modelFile}}Absolute path to the immutable approved GGUF model file.Threads{{threads}}Measured CPU thread count for the selected hardware and workload.Gpu Layers{{gpuLayers}}Measured number of layers offloaded while preserving VRAM headroom.

{{buildDir}}/bin/llama-bench -m {{modelFile}} -p 512 -n 128 -t {{threads}} -ngl {{gpuLayers}} -r 5 -o json

Example output / evidence

[{"model_filename":"assistant-q5_k_m.gguf","model_size":9876543210,"n_threads":12,"n_gpu_layers":28,"n_prompt":512,"n_gen":0,"avg_ts":842.36,"stddev_ts":9.41},{"model_filename":"assistant-q5_k_m.gguf","n_threads":12,"n_gpu_layers":28,"n_prompt":0,"n_gen":128,"avg_ts":71.82,"stddev_ts":1.27}]

Checkpoint: Checkpoint: Benchmark prompt, generation, threads, and GPU offload reproducibly

Continue whenBenchmark variance is acceptable, prompt and generation baselines meet requirements, RAM/VRAM stay within headroom, and the selected thread/offload settings are recorded with exact binary/model identity.

Stop whenThe host is shared or throttling, variance is unexplained, memory exceeds the envelope, results omit model/binary identity, or a quality-failing candidate is being optimized.

If this step fails

llama-bench results vary widely between runs or are much slower in production.

Likely causeWarmup, CPU frequency, thermal throttling, background load, NUMA placement, thread count, memory pressure, GPU offload or prompt/output sizes differ between measurements.

Safe checks

Record hardware, power/thermal state, binary commit, model checksum and every benchmark argument.
Run repeated warm measurements on an otherwise idle host and inspect variance.
Compare prompt processing and generation separately.

ResolutionStabilize the host and benchmark protocol, set measured thread/NUMA/offload choices, report median plus variance, and reject conclusions from incomparable one-off runs.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

verification

Evaluate task quality, template behavior, and quantization loss

read-only

Run a frozen suite covering each intended task, language, structured format, long-context requirement, refusal boundary and adversarial input. Compare the candidate GGUF with the higher-quality reference under identical template, seed and generation settings. Track exact-match or rubric outcomes, schema validity, unsupported claims, refusals, latency and human-reviewed critical cases. Keep evaluation data synthetic or explicitly approved and redacted.

Why this step matters

A quantized model can preserve average language modeling performance while regressing a narrow production task, formatting rule or safety boundary. Frozen task evidence and critical-case review are the only defensible basis for accepting that tradeoff.

What to understand

Define critical cases that require perfect pass independently from aggregate thresholds; a single unsafe tool recommendation or broken machine-readable response can block release.

When free-form answers have multiple valid forms, use evidence-backed rubrics and blinded review rather than brittle string matching or uncalibrated model grading.

Retain failed cases and reviewer decisions with model, template, binary and settings revisions so the result can be reproduced after upgrade.

System changes

Runs bounded offline inference and writes a redacted quality report; no listener or active model changes.

Syntax explained

--seed 42: Reduces sampling variance where the runtime/model path is deterministic; still record hardware and settings because perfect determinism is not universal.
reference delta: Compares candidate with the accepted higher-quality baseline, while absolute task thresholds continue to govern promotion.

Command

Fill variables0/3 ready

Values stay on this page and are never sent or saved.

Build Dir{{buildDir}}Candidate out-of-tree CMake build directory.Model File{{modelFile}}Absolute path to the immutable approved GGUF model file.Candidate Release{{candidateRelease}}Immutable candidate binary/model release directory.

run-llama-quality-suite --binary {{buildDir}}/bin/llama-cli --model {{modelFile}} --suite /srv/llama/evals/production-v3.jsonl --seed 42 --report /srv/llama/evidence/{{candidateRelease}}-quality.json

Example output / evidence

suite=production-v3 cases=240 passed=238 failed=2
critical_failures=0
schema_valid=60/60
refusal_boundary=40/40
language_en=78/80
reference_delta=-0.6pp
release_gate=PASS_WITH_REVIEW

Checkpoint: Checkpoint: Evaluate task quality, template behavior, and quantization loss

Continue whenAll critical cases pass, aggregate and segment thresholds meet margin, template behavior is correct, and every accepted regression has an explicit owner and rationale.

Stop whenCritical safety or format cases fail, evaluation contains unauthorized data, reference/candidate settings differ, or no reviewer can explain the quality tradeoff.

If this step fails

A smaller quantization is faster or fits, but task accuracy, formatting, factuality, or refusal behavior regresses.

Likely causeQuantization loss affects important tensors or tasks, a quantized-from-quantized artifact compounded error, the chat template changed, or the evaluation set is too narrow.

Safe checks

Compare candidate and reference with identical runtime, prompt/template and deterministic evaluation settings.
Run perplexity or distribution metrics where appropriate plus task-specific frozen cases.
Review failures by task and language rather than relying only on one aggregate score.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

config

Run llama-server on loopback with explicit limits and metrics

caution

Install the candidate binary and GGUF into immutable release paths, run as a dedicated unprivileged service identity, and bind only to loopback. Set the exact model, alias, context, threads, GPU layers, batch and parallel slots from measured evidence. Enable metrics privately. Keep the built-in web UI and dynamic model download/router behavior outside the production contract unless separately needed, authenticated and evaluated.

Why this step matters

Explicit service arguments make the evaluated context, concurrency, offload and API alias enforceable, while loopback and operating-system isolation keep a partially validated native model parser outside the network trust boundary.

What to understand

The server health endpoint is intentionally unauthenticated according to upstream documentation; network isolation and the trusted proxy must prevent it from becoming an information leak.

Built-in API keys can provide defense in depth, but keep secrets in a root-readable key file or secret mechanism and do not replace centralized TLS and identity controls.

Dynamic model download and router endpoints broaden network, disk and lifecycle authority. Disable or isolate them unless the production design explicitly requires and tests them.

System changes

Installs and starts a native inference server, maps the GGUF into memory, may allocate GPU resources, and exposes API/health/metrics only on loopback.

Syntax explained

--ctx-size {{ctxSize}}: Sets the accepted context used for memory planning rather than automatically selecting a maximum from metadata.
--parallel {{parallelSlots}}: Creates concurrent server slots; each affects cache and admission capacity and must remain within the load-test envelope.
--metrics: Enables the Prometheus-compatible endpoint; protect it through loopback/private monitoring access.
--alias {{modelAlias}}: Exposes a stable API name that the release manifest maps to the exact GGUF checksum.

File /etc/systemd/system/llama-server.service

Configuration

Fill variables0/8 ready

Values stay on this page and are never sent or saved.

Candidate Release{{candidateRelease}}Immutable candidate binary/model release directory.Model File{{modelFile}}Absolute path to the immutable approved GGUF model file.Model Alias{{modelAlias}}Stable public API alias mapped to the exact GGUF checksum.Listen Port{{listenPort}}Loopback llama-server port.Ctx Size{{ctxSize}}Evaluated operational context size, not automatically the model maximum.Threads{{threads}}Measured CPU thread count for the selected hardware and workload.Gpu Layers{{gpuLayers}}Measured number of layers offloaded while preserving VRAM headroom.Parallel Slots{{parallelSlots}}Accepted concurrent server slots within memory and latency SLOs.

[Unit]
Description=llama.cpp inference server
After=network.target

[Service]
User=llama
Group=llama
ExecStart=/srv/llama/releases/{{candidateRelease}}/bin/llama-server --model {{modelFile}} --alias {{modelAlias}} --host 127.0.0.1 --port {{listenPort}} --ctx-size {{ctxSize}} --threads {{threads}} --n-gpu-layers {{gpuLayers}} --parallel {{parallelSlots}} --metrics
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadOnlyPaths=/srv/llama/releases /srv/llama/models
Restart=on-failure

[Install]
WantedBy=multi-user.target

Example output / evidence

llama_model_loader: loaded meta data with 32 key-value pairs and 291 tensors
llama_context: n_ctx = 8192
srv  load_model: prompt caching enabled
main: model loaded
main: chat template, built-in: chatml
main: server is listening on http://127.0.0.1:8080

Checkpoint: Checkpoint: Run llama-server on loopback with explicit limits and metrics

Continue whenThe approved binary and GGUF load, the process runs unprivileged, listener is loopback-only, /health returns ready, /props matches the manifest, and /metrics exposes llama.cpp series privately.

Stop whenThe listener is public, active paths are mutable, dynamic downloads are enabled unintentionally, process requires root, context/slots differ from sizing, or the wrong template/backend loads.

If this step fails

GET /health returns 200, but the wrong GGUF, quantization, context, template or runtime commit is active.

Likely causeHealth means the model loaded, not that release identity and semantic behavior match the approved bundle.

Safe checks

Query /props and the model list, then compare paths and defaults with the release manifest.
Verify the active file checksum and process executable checksum.
Run quantization-sensitive golden task fixtures.

ResolutionRemove the instance from routing, start the intended immutable model and binary, repeat quality/capacity/API checks, and only then interpret health as one readiness signal.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

config

Expose only through a trusted proxy and enforce admission

caution

Use the same maintained proxy boundary as other internal inference: TLS, organization authentication and model authorization, byte and token limits, request and connection rate limits, finite timeouts, streaming support, privacy-preserving request IDs and logs, and a backend firewall restricted to the proxy. Keep /health, /props, /metrics, slot-management, model-management, UI and any download endpoints private unless an explicit administrative policy requires them.

Why this step matters

llama-server provides rich local operational and management endpoints, some intentionally public to its listener. The trusted proxy must expose only the caller contract and keep administration, model identity, metrics and cache controls on the private side.

What to understand

Do not proxy the built-in /tools endpoint to downstream users; upstream documents it as an internal Web UI interface subject to change.

Reject large or over-context work before inference where possible. A byte limit alone does not bound tokenization cost or model context.

Log authentication identity, status, duration and sizes under retention policy, but omit API keys, prompts, generated text and full model paths.

System changes

Adds a TLS authenticated public route for /v1 while explicitly denying operational and management endpoints; public traffic remains disabled until contract tests pass.

Syntax explained

location /v1/: Defines the supported external API surface; downstream clients should not depend on internal completion, slot or model-management endpoints.
deny operational paths: Prevents unauthenticated health, model properties, metrics and management endpoints on the public hostname even when the backend implements them.
proxy_buffering off: Supports streaming responses; verify disconnect cleanup and capacity under the accepted maximum duration.

File /etc/nginx/conf.d/{{publicHost}}-llama.conf

Configuration

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

Listen Port{{listenPort}}Loopback llama-server port.Public Host{{publicHost}}TLS hostname of the trusted proxy.

upstream llama_backend { server 127.0.0.1:{{listenPort}}; keepalive 16; }
server {
  listen 443 ssl http2;
  server_name {{publicHost}};
  client_max_body_size 1m;
  location /v1/ {
    auth_request /_auth;
    proxy_buffering off;
    proxy_read_timeout 300s;
    proxy_set_header X-Request-ID $request_id;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_pass http://llama_backend;
  }
  location ~ ^/(health|props|metrics|slots|models|tools) { return 404; }
}

Example output / evidence

nginx: configuration file /etc/nginx/nginx.conf test is successful
public_v1_models=200
public_metrics=404
public_props=404
unauthenticated_chat=401
backend_unapproved_network=filtered

Checkpoint: Checkpoint: Expose only through a trusted proxy and enforce admission

Continue whenAuthenticated /v1 fixtures work, unauthenticated and excess requests fail predictably, management/metrics paths are absent publicly, and only the proxy reaches loopback llama-server.

Stop whenAny model-management, slot, metrics, props, tools or download function is public, identity fails open, or an unapproved network reaches the backend.

If this step fails

The server works locally but is reachable directly from an unapproved network or returns data without authentication.

Likely causeThe listener was set to 0.0.0.0, a container published the port, firewall policy is broad, or operators relied on optional server API keys instead of the planned trusted proxy boundary.

Safe checks

Inspect listening addresses, process arguments and container port mappings.
Test reachability from the trusted proxy and an unapproved segment.
Review proxy TLS, identity, body/token/rate limits and backend firewall rules.

ResolutionBind to loopback or a private service address, remove broad publication, make the trusted proxy the only backend source, rotate exposed keys, and review access evidence before restoring traffic.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

verification

Load-test slots, context, cancellation, and monitoring

read-only

Replay the accepted prompt/output length and concurrency distribution through the proxy. Include streaming, cancellation, longest permitted context, invalid JSON, oversized body, unauthenticated, over-limit and slot-saturation cases. Correlate latency and errors with llamacpp prompt/prediction token rates, requests_processing, requests_deferred, context high-water mark, RAM, swap, GPU memory and temperature. Set proxy admission below the first SLO or memory failure.

Why this step matters

Server slots and context consume finite cache and compute. Monitoring plus overload fixtures establish whether the configured parallelism serves the target or merely accumulates deferred requests until latency and memory fail.

What to understand

Test cancellation and client disconnects, then prove requests_processing returns to baseline; leaked slots can create a delayed outage after apparently successful traffic.

Keep health and metrics private but include their behavior in rollback. Metric names and endpoints may change across commits.

Use synthetic or approved-redacted workloads and never place prompt text in metric labels or routine access logs.

System changes

Generates bounded inference load through the public proxy and may saturate the canary; no configuration changes occur unless admission is updated after review.

Syntax explained

requests_deferred: Shows work waiting for slots; sustained growth indicates admission exceeds current server capacity.
n_tokens_max: High-water mark of observed context usage that helps compare actual demand with the reviewed context envelope.
processing versus deferred: Separates active slots from queued work and supports overload alarms before users experience unbounded delay.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Listen Port{{listenPort}}Loopback llama-server port.

curl --fail -sS http://127.0.0.1:{{listenPort}}/metrics | grep -E '^llamacpp:(requests_processing|requests_deferred|prompt_tokens_seconds|predicted_tokens_seconds|n_tokens_max)'

Example output / evidence

llamacpp:requests_processing 2
llamacpp:requests_deferred 0
llamacpp:prompt_tokens_seconds 811.42
llamacpp:predicted_tokens_seconds 70.91
llamacpp:n_tokens_max 6144
workload_p95_ttft_ms=184
workload_errors=0
cancelled_slot_released=true

Checkpoint: Checkpoint: Load-test slots, context, cancellation, and monitoring

Continue whenAccepted load meets latency and quality SLOs, no swap or memory cliff appears, deferred work clears, cancellations release slots, and negative requests fail before unbounded compute.

Stop whenSlots leak, swap grows, context exceeds the plan, metrics are absent, public management endpoints appear, or required demand fails below the admission target.

If this step fails

Requests queue indefinitely, return 503, or latency rises when parallel users connect.

Likely causeThe number of server slots, context allocation, batch sizes, CPU threads, GPU offload and admitted concurrency do not match measured capacity.

Safe checks

Inspect llamacpp requests_processing, requests_deferred, token rates and context high-water mark.
Replay the frozen concurrency/length workload and identify the first SLO breach.
Confirm proxy admission limits do not exceed server slots and memory capacity.

ResolutionSet admission below the accepted slot/context envelope, tune one measured parameter at a time, add capacity, or choose a smaller evaluated model; reject predictably instead of allowing unbounded queueing.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

warning

Canary and roll back binary, GGUF, template, and proxy together

danger

Create each upgrade as an immutable release directory containing binary hashes and build manifest, GGUF/checksum/release record, optional projector, service arguments, proxy policy, quality report, benchmark, metric queries and previous target. Start beside the active release, run all offline and public contract gates, then shift bounded traffic. Rehearse rollback by routing to the former service and verifying exact binary/model identity plus quantization-sensitive golden cases.

Why this step matters

Binary, build backend, GGUF, template and server behavior are tightly coupled. A complete parallel bundle avoids rebuilding during an incident and catches the common false rollback where an old executable still loads a candidate model through a mutable path.

What to understand

Keep the previous process warm when capacity permits. For ordinary performance regression, remove traffic and drain; for artifact integrity or data exposure, contain immediately.

Resolve every symlink and verify hashes after rollback. The alias alone cannot prove the quantization or binary commit.

Preserve redacted benchmark, metrics and failure evidence under restricted access, then clear only candidate-scoped caches after evidence capture.

System changes

Stops the candidate, changes proxy routing, activates the former immutable bundle and quarantines candidate artifacts; in-flight candidate requests may be interrupted.

Syntax explained

complete SHA256SUMS: Verifies both executable and model artifacts from the former release, preventing a hybrid rollback.
quantization-sensitive golden cases: Semantically distinguish the approved former model/template from a candidate that may share the same public alias.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Previous Release{{previousRelease}}Retained known-good full release directory.

sudo systemctl stop llama-server-candidate.service && sudo nginx -t && sudo systemctl reload nginx && sha256sum -c /srv/llama/releases/{{previousRelease}}/SHA256SUMS

Example output / evidence

nginx: configuration file /etc/nginx/nginx.conf test is successful
bin/llama-server: OK
models/assistant-q5_k_m.gguf: OK
rollback_health=200
rollback_quality_critical=PASS
candidate_state=quarantined

Checkpoint: Checkpoint: Canary and roll back binary, GGUF, template, and proxy together

Continue whenThe public route serves the exact former binary, GGUF, template and limits within the recovery objective, all critical fixtures pass, and the candidate is unreachable and preserved for review.

Stop whenThe former release requires a network download or rebuild, active paths are ambiguous, proxy reload broadens access, or the restored artifact cannot pass checksum and quality identity checks.

If this step fails

Rollback starts the old binary but still reads the candidate GGUF or retained cache and produces candidate behavior.

Likely causeThe model path, working-directory symlink, service environment, router cache, template or proxy upstream was not restored with the executable.

Safe checks

Resolve every active path and checksum against the previous manifest.
Inspect process arguments, /props, model alias and proxy upstream.
Run the previous quantization-sensitive golden fixtures.

ResolutionRestore all binary, model, template, service, proxy and monitoring artifacts from the previous release directory, clear only candidate-scoped cache, restart, and verify semantic identity before traffic.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

For a proxy-only regression, restore only the former proxy configuration while keeping the already accepted binary/model release.
If both releases fail critical quality or security gates, fail closed with a maintenance response instead of serving an unverified local model.

Stop conditions

Stop before service or routing changes unless candidate and previous unit names, immutable directories, checksums, traffic state and incident authority are explicit.
Stop fleet promotion on any unexplained quality, API, memory, device, isolation, metric or rollback regression.

Finish line

Verification checklist

Artifact and binary identitysha256sum -c /srv/llama/releases/{{candidateRelease}}/SHA256SUMSThe executable, GGUF, optional projector and release metadata exactly match the approved immutable candidate.

Private readinesscurl --fail http://127.0.0.1:{{listenPort}}/healthllama-server reports status ok on loopback, while /props and process arguments match the release manifest.

Public contract and isolationcurl --fail https://{{publicHost}}/v1/modelsAuthenticated /v1 requests use the approved alias; operational, model-management and metrics endpoints are not public; unapproved networks cannot reach the backend.

Quality, capacity, and rollbackrun-llama-release-gates {{candidateRelease}}Critical quality cases pass, benchmark and load SLOs retain memory headroom, overload fails predictably, and the former full bundle has a timed successful restore.

Recovery guidance

Common problems and safe checks

The GGUF file downloads successfully, but its checksum or recorded source revision does not match the approved manifest.

Safe checks

Compare filename, byte size, upstream model and conversion revision, quantization label, license and expected SHA-256.
Run sha256sum -c from the isolated staging directory.
Inspect symlinks and ensure the active service cannot replace the file.

ResolutionQuarantine the file, fetch the exact approved artifact through the controlled import path, verify its manifest and license, and repeat metadata and quality evaluation before activation.

llama.cpp reports an unsupported model architecture, tensor type, GGUF version, or missing metadata key.

Safe checks

Capture the pinned llama.cpp commit, build features and the first metadata/error lines.
Verify checksum, model architecture and whether a separate multimodal projector is required.
Test the file with the exact converter/runtime compatibility set used to approve it.

The model process is killed, swaps heavily, or fails allocation on CPU startup.

Likely causeModel mapping, KV cache at the requested context, temporary buffers, parallel slots and operating-system headroom exceed usable RAM or address space.

Safe checks

Record model byte size, MemAvailable, swap activity, context size, parallel slots and cache types.
Run a one-slot, shorter-context canary while holding the model constant.
Check the kernel journal for OOM-killer evidence.

ResolutionChoose an evaluated smaller quantization or model, reduce accepted context/parallelism, add RAM, or isolate the host; do not depend on sustained swap as production capacity.

GPU offload fails, uses no GPU, or runs out of VRAM after increasing --n-gpu-layers.

Likely causeThe binary lacks the required backend, driver/runtime is incompatible, selected layers plus KV cache exceed VRAM, or a device-selection assumption is wrong.

Safe checks

Run llama-bench --list-devices and capture build/backend information.
Observe GPU memory while increasing offload from a known CPU baseline.
Compare the exact binary commit and build options with the release manifest.

ResolutionReturn to the last measured gpu-layers value, rebuild the pinned commit with the intended backend, or use CPU inference; preserve VRAM headroom and never select maximal offload only because startup once succeeds.

A smaller quantization is faster or fits, but task accuracy, formatting, factuality, or refusal behavior regresses.

Likely causeQuantization loss affects important tensors or tasks, a quantized-from-quantized artifact compounded error, the chat template changed, or the evaluation set is too narrow.

Safe checks

Compare candidate and reference with identical runtime, prompt/template and deterministic evaluation settings.
Run perplexity or distribution metrics where appropriate plus task-specific frozen cases.
Review failures by task and language rather than relying only on one aggregate score.

llama-bench results vary widely between runs or are much slower in production.

Likely causeWarmup, CPU frequency, thermal throttling, background load, NUMA placement, thread count, memory pressure, GPU offload or prompt/output sizes differ between measurements.

Safe checks

Record hardware, power/thermal state, binary commit, model checksum and every benchmark argument.
Run repeated warm measurements on an otherwise idle host and inspect variance.
Compare prompt processing and generation separately.

ResolutionStabilize the host and benchmark protocol, set measured thread/NUMA/offload choices, report median plus variance, and reject conclusions from incomparable one-off runs.

Chat completions contain role markers, ignore system content, or fail although plain completion works.

Likely causeThe GGUF lacks the expected tokenizer chat template, the runtime applied a different template, the model is a base rather than instruction model, or client roles do not match the template.

Safe checks

Inspect /props and GGUF template metadata without logging production conversations.
Run the frozen minimal chat fixture and a plain-completion control.
Compare the template and tokenizer with the approved release manifest.

ResolutionPin and evaluate the correct model/template pair, restrict the API to supported message roles, or serve completion only; do not improvise a template in production without quality tests.

The server works locally but is reachable directly from an unapproved network or returns data without authentication.

Safe checks

Inspect listening addresses, process arguments and container port mappings.
Test reachability from the trusted proxy and an unapproved segment.
Review proxy TLS, identity, body/token/rate limits and backend firewall rules.

GET /health returns 200, but the wrong GGUF, quantization, context, template or runtime commit is active.

Likely causeHealth means the model loaded, not that release identity and semantic behavior match the approved bundle.

Safe checks

Query /props and the model list, then compare paths and defaults with the release manifest.
Verify the active file checksum and process executable checksum.
Run quantization-sensitive golden task fixtures.

ResolutionRemove the instance from routing, start the intended immutable model and binary, repeat quality/capacity/API checks, and only then interpret health as one readiness signal.

Requests queue indefinitely, return 503, or latency rises when parallel users connect.

Likely causeThe number of server slots, context allocation, batch sizes, CPU threads, GPU offload and admitted concurrency do not match measured capacity.

Safe checks

Inspect llamacpp requests_processing, requests_deferred, token rates and context high-water mark.
Replay the frozen concurrency/length workload and identify the first SLO breach.
Confirm proxy admission limits do not exceed server slots and memory capacity.

The /metrics endpoint returns 501 or monitoring sees no llama.cpp series.

Likely causeThe server was started without --metrics, the private scrape path is blocked, router mode requires a model parameter, or dashboard queries target another release.

Safe checks

Inspect process arguments and curl /metrics locally.
Check Prometheus target status and whether single-model or router mode is used.
Compare available llamacpp-prefixed metrics with versioned rules.

ResolutionEnable metrics in the reviewed service definition, preserve private access, add any required model query in router mode, and validate dashboards with synthetic load before launch.

A llama.cpp or GGUF upgrade improves benchmark speed but changes API behavior or quality.

Likely causeBinary commit, build backend, quantization, template, context defaults or server API changed together and the release gate measured only throughput.

Safe checks

Diff binary commit/build flags, GGUF checksum/metadata, service arguments and proxy contract.
Run task quality, perplexity where appropriate, API negative fixtures and workload-shaped benchmarks against old and candidate.
Compare metric and health behavior before traffic promotion.

ResolutionDrain and quarantine the candidate, restore the previous complete binary/model/configuration bundle, classify the change, and attempt another single-variable canary with expanded fixtures.

Rollback starts the old binary but still reads the candidate GGUF or retained cache and produces candidate behavior.

Likely causeThe model path, working-directory symlink, service environment, router cache, template or proxy upstream was not restored with the executable.

Safe checks

Resolve every active path and checksum against the previous manifest.
Inspect process arguments, /props, model alias and proxy upstream.
Run the previous quantization-sensitive golden fixtures.

After the procedure

Alternatives and next steps

Consider these alternatives

Use CPU-only llama.cpp when latency and throughput permit; it removes GPU driver integration but still requires provenance, memory sizing, network isolation and quality evidence.
Use a higher-quality quantization or smaller upstream model when the smallest candidate fails tasks; do not obscure the regression behind a generic local-model label.
Use the llama.cpp built-in API-key file only as defense in depth while retaining a trusted proxy for TLS, centralized identity, admission and access policy.
Use an offline CLI rather than a long-running server for single-user sensitive workflows when remote or concurrent access is unnecessary.

Operate it safely

Automate source/build/model manifests and artifact checksums, then sign or protect the release decision through existing supply-chain controls.
Expand quality suites by task, language, context and failure boundary whenever users report a model or quantization-specific regression.
Schedule quarterly binary/model rollback, overload, client-cancellation and public management-endpoint exposure tests.
Track llama.cpp server, build and GGUF/quantization documentation changes before upgrading the pinned commit.

Reference

Frequently asked questions

Is Q4_K_M always the best default?

No. It is a common tradeoff label, not a universal quality guarantee. Compare several quantizations and a higher-quality reference on the actual tasks, languages, template, hardware and context.

Does a valid GGUF checksum prove the model is trustworthy?

It proves the file matches a particular manifest. Trust also requires protected manifest provenance, upstream and converter identity, license review, metadata compatibility and task-quality evaluation.

Can the maximum context from model metadata be enabled immediately?

No. Operational context consumes cache and affects latency and concurrency. Enable the smallest requirement that passes quality and capacity tests with headroom.

Why build from a pinned commit instead of installing the newest binary?

llama.cpp changes quickly. A pinned source/toolchain/backend build makes behavior, benchmarks and rollback reproducible and prevents a future download from silently changing the release.

Is llama-server safe to expose directly because it supports API keys and TLS?

Keep it private. A trusted proxy provides maintained organization identity, authorization, request/token/rate limits, controlled public paths and audit integration. Several operational endpoints have different authentication behavior.

Does a faster tokens-per-second result justify promotion?

Only after the exact candidate meets task quality, critical refusal/format, API, memory, latency, isolation, metrics and rollback gates. Throughput is one dimension of a complete release decision.

Recovery

Rollback

Remove the candidate from routing, drain or stop it according to incident severity, activate the previous immutable binary/GGUF/template/service/proxy/monitoring bundle, verify checksums plus semantic golden cases, and quarantine candidate files and caches. Returned model text cannot be retracted, so handle unsafe output or data exposure through the incident process.

Record candidate and former release identifiers, resolved paths, checksums, traffic state, metrics and a redacted failing fixture before volatile evidence disappears.
Set candidate traffic to zero; drain ordinary performance failures or stop immediately for artifact integrity, authorization or data-exposure incidents.
Restore the former proxy upstream and policy, validate and reload, then prove private management endpoints remain hidden publicly.
Start the former llama-server from its immutable binary and GGUF paths; verify both checksums, /props, model alias, context, slots, template and backend.
Run critical task/format/refusal cases, API negative fixtures and a short workload-shaped capacity test through the public hostname.
Quarantine the candidate release and scoped cache, rotate any exposed secret, preserve restricted evidence, and add the regression before another canary.

Evidence

Sources and review

Verified 2026-07-24Review due 2026-10-22

llama.cpp official repository and model format overviewofficial llama.cpp official build documentationofficial llama.cpp server documentationofficial llama.cpp quantization documentationofficial llama.cpp benchmark documentationofficial NGINX HTTP proxy moduleofficial NGINX auth request moduleofficial