AI/ML / Containers & Kubernetes / Security

Install and operate NVIDIA Container Toolkit safely

Integrate NVIDIA GPUs with Docker from a pinned driver/toolkit/CUDA matrix, assign devices by UUID or CDI, harden non-root workloads, monitor hardware with DCGM, and rehearse daemon-aware upgrades and rollback.

180 min10 stepsChanges system stateRevision 1

Save or explore

#nvidia #container-toolkit #docker #cuda #gpu #cdi #dcgm #isolation #observability

0 of 10 steps completed

Goal

Give approved containers access to exactly their assigned GPUs and minimum driver capabilities while preserving host-driver compatibility, private telemetry, deterministic maintenance and a complete rollback path.

Supported environments

Linux GPU host supported NVIDIA driver
Docker Engine CDI-capable version when CDI is used
NVIDIA Container Toolkit pinned compatibility-tested release

Prerequisites

Supported host and driver An inventoried NVIDIA GPU host on a supported Linux distribution with a working, package-managed driver and recovery access.nvidia-smi
Drained maintenance window Approval to restart Docker and recreate every affected GPU workload, plus owners for unrelated containers on the same daemon.
Pinned compatibility bundle Exact driver, toolkit packages, Docker version, CUDA/application image digests, device assignments and former rollback bundle.
Private monitoring Prometheus or equivalent private scraping and an incident owner for GPU health, isolation and workload regressions.

Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System

A versioned compatibility record joining host driver, toolkit packages, Docker engine, CDI behavior, immutable CUDA/application images and stable GPU assignments.
A Docker integration in which non-root containers receive only explicitly assigned GPU UUIDs or CDI devices and only the driver capabilities their workloads need.
A private DCGM monitoring path plus a drained-canary upgrade and rollback process that restores packages, daemon configuration, CDI state and recreated workloads together.

Observable outcome

Operators can diagnose host-driver, CUDA compatibility, toolkit, Docker, CDI, permission, workload and hardware failures in the correct layer.
A successful GPU workload does not imply broad all-GPU access; both positive assignment and negative isolation are tested continuously.
Driver or toolkit maintenance cannot silently change device identity, capability scope or monitoring without failing release gates.

Architecture

How the parts fit together

The Linux host owns the kernel driver and physical GPU inventory. NVIDIA Container Toolkit configures Docker and generates CDI device descriptions. Docker creates hardened application containers with explicit GPU UUIDs and minimum driver capabilities. DCGM Exporter observes approved devices on a private endpoint. Package, daemon, CDI, image and assignment manifests move through a drained canary before fleet rollout.

Host driver and GPU inventoryProvide kernel-level GPU access, stable UUIDs, hardware health and the compatibility ceiling for container CUDA runtimes.

NVIDIA Container Toolkit and nvidia-ctkIntegrate device and driver capability injection with Docker and generate or inspect CDI specifications.

Docker EngineCreate the container boundary, apply explicit device requests and ordinary least-privilege controls, and recreate workloads after runtime changes.

Application and CUDA imagesSupply pinned user-space libraries and workload binaries that remain compatible with the host driver.

DCGM and release evidenceObserve device health and capacity by UUID and retain package/configuration/image baselines for canary and rollback decisions.

Inventory establishes current UUID/MIG state and driver health before toolkit changes.
A reviewed compatibility matrix selects exact toolkit packages, Docker/CDI support and immutable CUDA/application images.
nvidia-ctk modifies Docker configuration; a drained restart and pinned smoke prove one explicit GPU while a negative test proves no default access.
Hardened workloads request only their UUID/CDI devices and required compute/utility capabilities.
DCGM observes private hardware signals; a canary gate and complete-bundle rollback control upgrades.

Assumptions

The NVIDIA driver is installed through a supported distribution or NVIDIA package workflow before this guide; this guide does not use toolkit installation as a driver repair method.
Docker is the worked example. Containerd, CRI-O, Podman, Kubernetes and rootless modes require their current official integration path and equivalent isolation tests.
The organization can drain the host and restart Docker. Installing runtime integration underneath active workloads is outside the procedure.
Mutually untrusted high-risk workloads may require separate hosts or a stronger isolation product because containers share the host kernel and GPU driver.

Key concepts

Host driver versus container CUDA: The host supplies the kernel driver; the image supplies CUDA user-space libraries. Their supported compatibility relationship must be tested as a pair.
GPU UUID: A device identifier used for stable assignment and telemetry. Unlike numeric index, it normally survives reboot but changes when hardware or MIG topology changes.
Container Device Interface: A standard device-description mechanism consumed by compatible runtimes. NVIDIA tooling generates CDI entries from current GPU and MIG inventory.
Driver capabilities: Feature groups such as compute, utility, graphics, video and display that control which driver libraries and functions enter a container.
Positive and negative isolation test: A positive test proves the assigned device works; a negative test proves unassigned devices are absent. Both are required to validate least privilege.

Before you copy

Values used in this guide

{{toolkitVersion}}

Exact package version string for every NVIDIA Container Toolkit package-family member.

Example: 1.19.1-1

{{previousToolkitVersion}}

Locally retained exact former package version used for rollback.

Example: 1.18.1-1

{{gpuUuid}}

Approved physical GPU or MIG UUID assigned to the test and workload.

Example: GPU-2f83d9b0-7c74-4ac4-a523-98e12e45b191

{{cudaImage}}

Reviewed NVIDIA CUDA or application image reference pinned by immutable digest.

Example: nvidia/cuda:12.8.1-base-ubuntu24.04@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef

{{metricsPort}}

Loopback or private DCGM Exporter metrics port.

Example: 9400

Security and production boundaries

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Stop before continuing if

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

decision

Inventory GPUs, workloads, ownership, and recovery access

read-only

Before installing packages, identify each GPU by UUID, model, memory, PCI address and MIG state; record the host kernel, distribution, driver, Docker version, cgroup mode and current workloads. Map each workload to an explicit device assignment, required CUDA level and driver capabilities. Name owners for the host, container platform, GPU consumers and monitoring, then schedule a drain and console-capable recovery window.

Why this step matters

Indexes can change, while UUIDs provide a stable assignment reference until hardware or MIG topology changes. Inventory plus workload ownership prevents installation from silently granting every container every GPU and gives incident responders a known pre-change state.

What to understand

Record current GPU users and drain them rather than testing runtime configuration underneath active inference or training jobs.

Separate base GPU UUIDs from MIG instance identifiers; reconfiguration changes the device inventory and can invalidate CDI specifications and scheduler assignments.

Document console or out-of-band access because driver and container-daemon maintenance can interrupt remote workloads or require a reboot.

System changes

Creates a versioned inventory and maintenance decision; no driver, toolkit, Docker or device configuration changes occur.

Syntax explained

uuid: Stable device identity used for explicit assignment and monitoring, preferable to a reboot-sensitive numeric index.
mig.mode.current: Shows whether the physical GPU is partitioned; changing MIG requires its own workload drain, inventory refresh and CDI regeneration.

Command

nvidia-smi --query-gpu=index,uuid,name,pci.bus_id,memory.total,driver_version,mig.mode.current --format=csv

Example output / evidence

index, uuid, name, pci.bus_id, memory.total [MiB], driver_version, mig.mode.current
0, GPU-2f83d9b0-7c74-4ac4-a523-98e12e45b191, NVIDIA H100 PCIe, 00000000:31:00.0, 81559 MiB, 575.57.08, Disabled

Checkpoint: Checkpoint: Inventory GPUs, workloads, ownership, and recovery access

Continue whenEvery GPU and workload has an owner, stable identifier, required capabilities and rollback dependency; no unexplained process uses the devices and a maintenance path is approved.

Stop whenGPU users cannot be drained, UUID/MIG inventory is ambiguous, recovery access is missing, or any workload expects unrestricted all-GPU access without review.

If this step fails

The host has an NVIDIA GPU, but nvidia-smi reports that it cannot communicate with the driver.

Likely causeThe kernel module is missing or failed to load, Secure Boot policy blocked it, the installed driver does not match the running kernel, the GPU is unhealthy, or the host requires a reboot after maintenance.

Safe checks

Record lspci, uname -r, loaded NVIDIA modules and the relevant kernel journal without installing another package.
Check the platform driver-installation record and whether a reboot is pending.
Inspect hardware health through the approved out-of-band or vendor diagnostics path.

ResolutionRepair the host driver through the supported distribution or NVIDIA package workflow, preserve the previous package versions, reboot only in an approved window, and require a clean host nvidia-smi result before troubleshooting containers.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

verification

Validate the host driver before container integration

read-only

Prove the host driver works independently of Docker. Capture kernel modules, driver version, GPU health and any pending reboot. Run a minimal read-only CUDA or vendor diagnostic through the already approved host tooling if available. Container Toolkit exposes host driver capabilities to containers; it cannot repair a missing, blocked or incompatible kernel driver.

Why this step matters

Troubleshooting must establish the lowest working layer first. A clean host check prevents Docker, toolkit or application configuration from being blamed for a kernel driver or hardware problem and captures the compatibility anchor for container CUDA selection.

What to understand

A host nvidia-smi result is necessary but not sufficient; application CUDA compatibility is tested later with the exact container image.

Uncorrected errors or unusual temperature are operational blockers even if the command exits zero.

Use the distribution or NVIDIA supported driver packaging path and retain the former package set; avoid mixing runfile and package-manager installations.

System changes

Reads kernel and GPU state only; any driver repair or reboot is a separate approved host-maintenance change.

Syntax explained

lsmod: Shows NVIDIA kernel modules loaded into the running kernel; it does not prove user-space CUDA compatibility.
uncorrected volatile ECC: Surfaces recent uncorrected memory errors on supported GPUs and should be interpreted with the hardware operations policy.

Command

uname -r; lsmod | grep '^nvidia'; nvidia-smi --query-gpu=uuid,temperature.gpu,ecc.errors.uncorrected.volatile --format=csv,noheader

Example output / evidence

6.8.0-71-generic
nvidia_uvm           2056192  0
nvidia_drm            143360  4
nvidia_modeset       1548288  3 nvidia_drm
nvidia               97943552  44 nvidia_uvm,nvidia_modeset
GPU-2f83d9b0-7c74-4ac4-a523-98e12e45b191, 34, 0

Checkpoint: Checkpoint: Validate the host driver before container integration

Continue whenThe approved driver is loaded for the running kernel, every assigned GPU appears by UUID, health is acceptable, and no reboot or unresolved hardware alarm remains.

Stop whenThe driver cannot communicate, modules do not match the running kernel, hardware health is suspect, or the remediation would mix unsupported installation methods.

If this step fails

The host has an NVIDIA GPU, but nvidia-smi reports that it cannot communicate with the driver.

Safe checks

Record lspci, uname -r, loaded NVIDIA modules and the relevant kernel journal without installing another package.
Check the platform driver-installation record and whether a reboot is pending.
Inspect hardware health through the approved out-of-band or vendor diagnostics path.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

decision

Freeze the driver, toolkit, Docker, and CUDA compatibility matrix

read-only

Select an immutable combination rather than installing whatever is newest: host driver, NVIDIA Container Toolkit packages, Docker engine, CDI schema support, CUDA base image by digest, application image by digest and target GPU architecture. Compare CUDA minor-version requirements with the host driver and check Container Toolkit release notes for known runtime and CDI issues. Record the previous combination beside the candidate.

Why this step matters

GPU containers combine the host kernel driver with container user-space libraries and a runtime injection layer. Each may be individually supported yet incompatible as a set. Freezing the matrix makes a successful smoke and workload test reusable evidence.

What to understand

Pin images by digest in production. A readable CUDA tag communicates intent but can move or be republished independently from your change record.

Check Docker or container runtime support for the CDI schema emitted by the selected toolkit release.

Do not upgrade driver, Docker, toolkit, application image and MIG topology in one change unless the entire bundle was rehearsed and rollback covers every component.

System changes

Creates a candidate and rollback compatibility manifest; no packages or daemon files change.

Syntax explained

apt-cache policy: Shows installed and candidate package versions from configured repositories so installation can be pinned explicitly.
image digest: Content identity used to reproduce a tested CUDA or application image instead of trusting a mutable tag.

Command

docker version --format 'docker={{.Server.Version}}'; nvidia-smi --query-gpu=driver_version --format=csv,noheader; apt-cache policy nvidia-container-toolkit | sed -n '1,12p'

Example output / evidence

docker=28.3.2
575.57.08
nvidia-container-toolkit:
  Installed: (none)
  Candidate: 1.19.1-1
  Version table:
     1.19.1-1 500

Checkpoint: Checkpoint: Freeze the driver, toolkit, Docker, and CUDA compatibility matrix

Continue whenThe candidate matrix is supported by current NVIDIA and Docker documentation, has no accepted known-issue blocker, and every former package/configuration/image artifact is locally recoverable.

Stop whenThe CUDA image needs a newer driver, the runtime cannot consume the emitted CDI schema, image digests are unresolved, or former package versions and daemon configuration are unavailable.

If this step fails

The container starts, but CUDA reports that the driver version is insufficient for the CUDA runtime.

Likely causeThe container CUDA user-space runtime requires a newer driver than the host provides, or an unsupported forward-compatibility assumption was made.

Safe checks

Capture the host driver version and the exact immutable container image digest and CUDA tag.
Compare the pair with NVIDIA CUDA compatibility documentation.
Run the last known-good CUDA image on the same host to separate driver and candidate-image changes.

ResolutionReturn to the last tested image or upgrade the driver through a staged host-maintenance release; do not copy arbitrary compatibility libraries into a production container.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

command

Install pinned NVIDIA Container Toolkit packages

caution

Configure the official NVIDIA package repository and keyring through configuration management, verify the repository definition, then install the exact reviewed package version for nvidia-container-toolkit, nvidia-container-toolkit-base, libnvidia-container-tools and libnvidia-container1. Prevent an unattended partial upgrade by pinning the bundle according to the host package policy. Save package metadata and checksums with the release record.

Why this step matters

Explicitly pinning the complete package family avoids a mixed runtime stack. Repository trust and version selection must happen before installation so a production host does not become the place where package provenance is first decided.

What to understand

Use the exact version syntax reported by the official repository for the distribution; the example value is a variable rather than a promise that one version is universally correct.

Capture dpkg status and package cache availability before and after installation so rollback can restore all four related packages.

Installation can create or update CDI refresh units. Inspect them before assuming that device specifications are current.

System changes

Installs privileged host runtime libraries, command-line tools and systemd units that can affect device injection for future containers.

Syntax explained

package={{toolkitVersion}}: Requests the exact reviewed package revision instead of accepting a newer repository candidate.
four-package bundle: Keeps the toolkit, base, CLI tools and runtime library on one release rather than mixing independently resolved versions.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Toolkit Version{{toolkitVersion}}Exact package version string for every NVIDIA Container Toolkit package-family member.

sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit={{toolkitVersion}} nvidia-container-toolkit-base={{toolkitVersion}} libnvidia-container-tools={{toolkitVersion}} libnvidia-container1={{toolkitVersion}}

Example output / evidence

The following NEW packages will be installed:
  libnvidia-container-tools libnvidia-container1
  nvidia-container-toolkit nvidia-container-toolkit-base
Setting up libnvidia-container1 (1.19.1-1) ...
Setting up nvidia-container-toolkit (1.19.1-1) ...

Checkpoint: Checkpoint: Install pinned NVIDIA Container Toolkit packages

Continue whenAll toolkit packages report the approved version, nvidia-ctk is available, package provenance matches the official repository, and no active workload or Docker configuration has changed.

Stop whenRepository definitions conflict, package signatures fail, apt proposes unrelated removals or a driver replacement, versions differ across the bundle, or rollback packages were not retained.

If this step fails

Package installation fails with conflicting Signed-By entries or duplicate NVIDIA repository definitions.

Likely causeAn older repository configuration, keyring path, distribution-specific setup, or repeated automation created incompatible apt source entries.

Safe checks

List only NVIDIA-related files in /etc/apt/sources.list.d and their Signed-By values.
Compare them with the current official installation guide and the configuration-management source.
Back up the files before removing a duplicate.

ResolutionConsolidate to the one reviewed official repository and keyring definition managed by configuration control, update package metadata, and verify the selected versions before installation.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

config

Configure Docker through nvidia-ctk and preserve daemon rollback

caution

Back up /etc/docker/daemon.json with ownership and mode, validate existing JSON, then let nvidia-ctk add the NVIDIA runtime configuration for Docker. Review the resulting diff rather than blindly restarting. If rootless Docker is the approved design, follow the separate documented rootless path and configuration file; do not apply the rootful command to a different daemon.

Why this step matters

nvidia-ctk generates the supported runtime stanza and reduces manual JSON drift, but it modifies a privileged shared daemon configuration. Backing up and reviewing the exact diff protects unrelated Docker settings and provides a clean rollback artifact.

What to understand

Do not set the NVIDIA runtime as a permissive excuse to expose GPUs by default; each workload still needs explicit device assignment.

Validate JSON before and after modification. Syntax success does not prove storage, logging, network or authorization semantics were preserved.

A Docker restart affects every container. Coordinate drain, restart policy and client expectations before the next step.

System changes

Modifies /etc/docker/daemon.json and stages a restricted rollback copy; Docker has not yet reloaded the change.

Syntax explained

nvidia-ctk runtime configure: Writes the NVIDIA runtime configuration for the chosen container engine using the toolkit-supported format.
install -m 0600: Creates a root-readable backup with explicit restrictive mode instead of a loosely permissioned copy.

Configuration

sudo install -m 0600 /etc/docker/daemon.json /var/backups/daemon.json.pre-nvidia && sudo nvidia-ctk runtime configure --runtime=docker && python -m json.tool /etc/docker/daemon.json >/dev/null

Example output / evidence

INFO[0000] Loading config from /etc/docker/daemon.json
INFO[0000] Wrote updated config to /etc/docker/daemon.json
INFO[0000] It is recommended that the docker daemon be restarted.

Checkpoint: Checkpoint: Configure Docker through nvidia-ctk and preserve daemon rollback

Continue whenThe JSON validates, the diff contains only the intended NVIDIA runtime stanza, unrelated daemon policy is unchanged, and the original file is recoverable.

Stop whenThe original file is invalid, the generated diff changes unrelated policy, the wrong Docker daemon is targeted, or restarting all containers is not approved.

If this step fails

Docker reports that it could not select the device driver with capabilities [[gpu]] or does not recognize the nvidia runtime.

Likely causeThe toolkit is not installed, Docker configuration was not generated, daemon.json is invalid, Docker was not restarted, or a rootless daemon uses a different configuration path.

Safe checks

Check nvidia-ctk and package versions, then inspect the effective Docker info output.
Validate /etc/docker/daemon.json as JSON and compare it with the saved pre-change file.
Inspect Docker journal messages after the last restart.

ResolutionRun the reviewed nvidia-ctk runtime configuration for the correct Docker mode, validate and restart during the maintenance window, then repeat a pinned minimal smoke test.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

verification

Restart Docker and run a pinned one-GPU smoke test

caution

During the approved drained window, restart Docker, verify daemon health and existing non-GPU workloads, then run a minimal pinned CUDA image against exactly one GPU UUID with only compute and utility driver capabilities. Compare inside-container UUID, driver and CUDA information with the compatibility manifest. Also run a negative container with no GPU request and prove that it cannot access devices.

Why this step matters

The smoke test validates Docker, runtime injection, host driver compatibility, immutable image and explicit device selection as one path. The negative test is equally important because a working GPU container may still violate isolation by exposing devices by default.

What to understand

Use UUID rather than index so the assignment remains tied to inventory. Quote Docker --gpus syntax exactly as documented and verify the created container configuration.

A CUDA base-image smoke is not an application benchmark; it proves device and library integration before the application adds its own variables.

Check existing containers after the daemon restart, especially restart policies and network/storage dependencies unrelated to GPUs.

System changes

Restarts the Docker daemon, affecting all containers, and runs short-lived read-only test containers on one GPU.

Syntax explained

--gpus device={{gpuUuid}}: Requests one explicit inventoried GPU instead of the broader all selector.
NVIDIA_DRIVER_CAPABILITIES=compute,utility: Mounts CUDA compute and NVML utility capabilities while omitting graphics, display, video and compatibility features not required by this smoke test.
{{cudaImage}}: A reviewed NVIDIA CUDA image pinned by digest and compatible with the host driver; never substitute latest during maintenance.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

Gpu Uuid{{gpuUuid}}Approved physical GPU or MIG UUID assigned to the test and workload.Cuda Image{{cudaImage}}Reviewed NVIDIA CUDA or application image reference pinned by immutable digest.

sudo systemctl restart docker && docker run --rm --gpus '"device={{gpuUuid}}"' -e NVIDIA_DRIVER_CAPABILITIES=compute,utility {{cudaImage}} nvidia-smi --query-gpu=uuid,name,driver_version --format=csv,noheader

Example output / evidence

GPU-2f83d9b0-7c74-4ac4-a523-98e12e45b191, NVIDIA H100 PCIe, 575.57.08
positive_gpu_smoke=PASS
negative_no_gpu_device_count=0

Checkpoint: Checkpoint: Restart Docker and run a pinned one-GPU smoke test

Continue whenDocker is healthy, the positive container sees only the selected UUID and expected driver, the negative container sees no GPU, and unrelated workloads recover as planned.

Stop whenThe container sees all GPUs, CUDA compatibility fails, Docker or unrelated workloads do not recover, or the command would use an unpinned image.

If this step fails

A container requested one GPU but can see every GPU on the host.

Likely causeThe image defaulted NVIDIA_VISIBLE_DEVICES to all, the Docker --gpus request omitted a device selector, the runtime was configured as a permissive default, or an orchestration layer broadened assignment.

Safe checks

Compare the host GPU UUID inventory with nvidia-smi output inside the container.
Inspect the exact Docker create arguments and NVIDIA_VISIBLE_DEVICES value.
Repeat with an explicit GPU UUID and compute,utility driver capabilities.

ResolutionStop the over-broad container, restart with the approved UUID or CDI device and minimum driver capabilities, audit affected workloads for cross-tenant exposure, and encode explicit assignment in deployment policy.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

config

Adopt explicit CDI or UUID assignment and least privilege

caution

Choose one reviewed device-injection path for each platform. Where Docker and the toolkit support it, use CDI names generated from current inventory; otherwise use explicit GPU UUID requests. Never rely on all. Run the application as a non-root user, drop capabilities, deny privilege escalation, mount a read-only root filesystem when compatible, limit writable paths, omit host sockets, and grant only compute/utility driver features. Recreate, rather than mutate, workloads after assignment changes.

Why this step matters

GPU access crosses the container boundary into a privileged host driver. Explicit device and capability assignment plus ordinary container hardening reduce accidental cross-workload visibility and limit unrelated host access even though they cannot turn the shared kernel driver into a perfect security boundary.

What to understand

CDI specifications refresh automatically in current toolkit releases for common driver and reboot events, but driver removal and MIG reconfiguration require explicit attention.

Keep high-risk tenants or mutually untrusted workloads on separate hosts or stronger isolation domains when the shared driver boundary is not acceptable.

Test non-root access without global chmod. A cosmetic missing group-name warning differs from a permission-denied device open.

System changes

Changes future workload device injection, user identity and container privileges; workloads must be recreated to receive the reviewed configuration.

Syntax explained

--device nvidia.com/gpu=<UUID>: Requests the explicit CDI device identity generated from current inventory on compatible Docker versions.
--cap-drop ALL: Removes ambient Linux capabilities; add back only a documented application requirement after testing.
--read-only: Makes the container root filesystem immutable; provide explicit temporary or state mounts only where the application requires them.

Configuration

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

Gpu Uuid{{gpuUuid}}Approved physical GPU or MIG UUID assigned to the test and workload.Cuda Image{{cudaImage}}Reviewed NVIDIA CUDA or application image reference pinned by immutable digest.

nvidia-ctk cdi list && docker run --rm --device nvidia.com/gpu={{gpuUuid}} --user 65532:65532 --cap-drop ALL --security-opt no-new-privileges --read-only {{cudaImage}} nvidia-smi -L

Example output / evidence

INFO[0000] Found 2 CDI devices
nvidia.com/gpu=GPU-2f83d9b0-7c74-4ac4-a523-98e12e45b191
nvidia.com/gpu=all
GPU 0: NVIDIA H100 PCIe (UUID: GPU-2f83d9b0-7c74-4ac4-a523-98e12e45b191)

Checkpoint: Checkpoint: Adopt explicit CDI or UUID assignment and least privilege

Continue whenThe workload runs non-root with no unnecessary Linux or NVIDIA capabilities, sees only its approved UUID, and no device appears in a container that omitted an assignment.

Stop whenThe application requires privileged mode, host Docker socket, all devices, global device chmod, an unreviewed capability, or CDI inventory is stale after driver/MIG maintenance.

If this step fails

A rootless or non-root container receives permission denied for NVIDIA or render device nodes.

Likely causeUser namespaces, cgroup device policy, supplementary group injection, CDI support, host device ownership, or container-engine configuration does not match the chosen rootless design.

Safe checks

Inspect device major/minor numbers and ownership without changing permissions.
List generated CDI devices and the rootless engine version and configuration.
Distinguish a cosmetic missing group-name warning from an actual failed device open.

ResolutionUse the documented CDI and rootless configuration for the supported engine, preserve narrow device ownership, and add only required groups or mappings; do not chmod device nodes globally or run the application privileged.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

verification

Validate CDI refresh and MIG lifecycle behavior

read-only

Inspect nvidia-cdi-refresh path and service status, generated device names, schema compatibility and current GPU/MIG inventory. Reboot or driver upgrades should refresh supported current releases automatically, but driver removal and MIG reconfiguration need explicit regeneration and workload recreation. Rehearse the exact maintenance event your fleet performs, then verify that old identifiers fail closed rather than resolving to another device.

Why this step matters

A generated device name is trustworthy only while it matches current physical and MIG inventory. Maintenance can invalidate identifiers or schema compatibility, so refresh behavior and fail-closed assignment are part of the workload lifecycle, not one-time installation.

What to understand

A one-shot refresh service may be inactive after successful completion; inspect exit status, journal and generated specification rather than treating inactive as failure.

After MIG reconfiguration, drain workloads, regenerate CDI data and update the assignment registry before recreation.

Validate the container engine version against toolkit release-note CDI schema requirements before upgrading either side.

System changes

Reads CDI and systemd state; a rehearsal may regenerate /var/run/cdi/nvidia.yaml and requires drained workloads if device topology changes.

Syntax explained

nvidia-ctk cdi list: Lists device names resolved from generated CDI specifications; compare them with host UUID/MIG inventory.
nvidia-cdi-refresh: Systemd path/service that generates current specifications in recent toolkit versions; known lifecycle limitations still require an operator runbook.

Command

systemctl is-enabled nvidia-cdi-refresh.path; systemctl status --no-pager nvidia-cdi-refresh.service | sed -n '1,12p'; nvidia-ctk cdi list

Example output / evidence

enabled
● nvidia-cdi-refresh.service - Generate NVIDIA CDI specifications
     Loaded: loaded
     Active: inactive (dead)
    Process: 1842 ExecStart=/usr/bin/nvidia-ctk cdi generate --output=/var/run/cdi/nvidia.yaml
nvidia.com/gpu=GPU-2f83d9b0-7c74-4ac4-a523-98e12e45b191

Checkpoint: Checkpoint: Validate CDI refresh and MIG lifecycle behavior

Continue whenGenerated CDI devices exactly match the approved UUID/MIG inventory, the engine supports the schema, refresh succeeds after tested lifecycle events, and stale identifiers do not grant another GPU.

Stop whenThe specification is stale or conflicting, engine schema support is uncertain, MIG changed without a drain, or an old identifier could broaden access.

If this step fails

nvidia-ctk cdi list is empty, stale, or reports a parse/schema error.

Likely causeThe driver inventory was unavailable at generation time, refresh failed, MIG changed, the runtime cannot consume the generated CDI schema version, or multiple generated specifications conflict.

Safe checks

Inspect nvidia-cdi-refresh service and path status plus recent journal output.
Validate /var/run/cdi/nvidia.yaml and list the container engine version.
Compare the required CDI schema support with the toolkit release notes.

ResolutionCorrect driver inventory, remove only the stale generated file, regenerate through the official service or command, or use the documented compatibility feature flag for the runtime; verify explicit device resolution before starting workloads.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

config

Add private DCGM telemetry and assignment-aware alerts

caution

Deploy a pinned NVIDIA DCGM Exporter or approved DCGM integration with explicit device scope and private network exposure. Scrape temperature, power, utilization, memory, ECC, XID and health signals where supported; label by stable UUID and approved workload mapping without tenant secrets. Alert on uncorrected errors, repeated XID events, thermal/power anomalies, exporter absence and assignment mismatches. Validate every query with a bounded synthetic workload.

Why this step matters

Application latency alone cannot distinguish saturation from hardware or driver faults. DCGM telemetry supplies GPU-specific evidence, while stable UUID mapping lets responders find the affected workload without depending on mutable device indexes.

What to understand

Metric availability varies by GPU, driver and DCGM release. Version dashboards with the tested exporter rather than assuming every field exists.

Keep the endpoint private and avoid serial numbers, tenant names or request content in broadly visible labels.

Monitoring containers need device visibility for telemetry but should not inherit application secrets, writable model storage or the Docker socket.

System changes

Starts a GPU telemetry process, exposes a private metrics endpoint and adds time-series retention plus alerts under monitoring access policy.

Syntax explained

DCGM_FI_DEV_XID_ERRORS: Reports the latest XID error code; interpret through NVIDIA guidance and correlate with journal and workload symptoms.
UUID label: Connects time series to the stable device inventory and assignment register instead of a mutable GPU index.

Configuration

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Metrics Port{{metricsPort}}Loopback or private DCGM Exporter metrics port.

curl --fail -sS http://127.0.0.1:{{metricsPort}}/metrics | grep -E '^DCGM_FI_DEV_(GPU_UTIL|FB_USED|XID_ERRORS|ECC_DBE_VOL_TOTAL)' | head

Example output / evidence

DCGM_FI_DEV_GPU_UTIL{gpu="0",UUID="GPU-2f83d9b0-7c74-4ac4-a523-98e12e45b191"} 14
DCGM_FI_DEV_FB_USED{gpu="0",UUID="GPU-2f83d9b0-7c74-4ac4-a523-98e12e45b191"} 8421
DCGM_FI_DEV_XID_ERRORS{gpu="0",UUID="GPU-2f83d9b0-7c74-4ac4-a523-98e12e45b191"} 0

Checkpoint: Checkpoint: Add private DCGM telemetry and assignment-aware alerts

Continue whenMetrics show only approved devices by UUID, synthetic load changes utilization, alert rehearsals fire and resolve, and the endpoint is unreachable from public or tenant networks.

Stop whenTelemetry requires privileged host access beyond the reviewed design, labels expose sensitive data, device mapping is wrong, or alerts cannot distinguish missing data from healthy zero.

If this step fails

DCGM Exporter starts but Prometheus receives no useful GPU metrics or labels the wrong devices.

Likely causeThe exporter lacks required device visibility, the host engine is absent or incompatible, its endpoint is unreachable, a MIG inventory changed, or scrape relabeling dropped series.

Safe checks

Curl the exporter metrics endpoint privately and inspect a small DCGM_FI_DEV sample.
Compare exporter-visible UUID/MIG inventory with nvidia-smi and assignment records.
Inspect Prometheus target health and the pinned exporter image or package revision.

ResolutionRestore explicit monitoring-device access and private scrape connectivity, regenerate inventory after MIG changes, update versioned queries, and verify alerts with synthetic load before declaring observability healthy.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

Rehearse the same decision with an offline fixture and a non-production endpoint before changing the serving host.

Stop conditions

Stop if the exact model or binary revision, license, source, checksum, intended data classification, accountable owner, or rollback artifact is unknown.
Stop if the service can be reached outside its approved private boundary before TLS, authentication, rate limits, request-size limits, and audit logging are independently verified.
Stop if a benchmark uses production secrets or personal data, if a quality result cannot be reproduced from a frozen fixture, or if a proposed optimization changes more than one material variable at a time.
Stop if the previous release, configuration, model files, container image, driver packages, or proxy policy cannot be restored without downloading an unreviewed replacement during the incident.

warning

Upgrade and roll back the complete GPU container stack

danger

Upgrade only on a drained canary host. Preserve package files, repository state, daemon.json, CDI specifications, driver record, Docker version, image digests, device assignments, metrics rules and representative workload baselines. Apply one reviewed bundle, restart, repeat host, positive/negative isolation, CDI, application and telemetry tests, then return capacity gradually. Rehearse rollback by restoring packages and configuration, restarting Docker, recreating workloads and proving exact UUID isolation.

Why this step matters

Toolkit state is distributed across packages, Docker configuration, generated CDI data and recreated container specifications. Restoring only one package can leave the host in a hybrid candidate state, so rollback must be rehearsed as a complete compatibility bundle.

What to understand

A driver rollback may require reboot and has greater host risk; treat it as a distinct, console-backed maintenance phase when the toolkit-only restore cannot reestablish the prior matrix.

Recreate GPU workloads after runtime or CDI rollback so candidate device requests are not retained by existing container specifications.

Keep canary evidence by UUID and release. A successful host nvidia-smi does not replace the application and isolation tests.

System changes

Downgrades privileged host packages, restores Docker daemon configuration, restarts the daemon and requires recreation of GPU workloads. All host containers may be interrupted.

Syntax explained

{{previousToolkitVersion}}: Exact locally retained package revision from the former accepted compatibility matrix.
restore daemon.json: Returns Docker runtime configuration to the known-good file, preserving ownership/mode and requiring semantic review before restart.
positive and negative tests: Prove that the assigned GPU works and every unassigned GPU remains inaccessible after rollback.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Previous Toolkit Version{{previousToolkitVersion}}Locally retained exact former package version used for rollback.

sudo cp -a /var/backups/daemon.json.pre-nvidia /etc/docker/daemon.json && sudo apt-get install -y nvidia-container-toolkit={{previousToolkitVersion}} nvidia-container-toolkit-base={{previousToolkitVersion}} libnvidia-container-tools={{previousToolkitVersion}} libnvidia-container1={{previousToolkitVersion}} && sudo systemctl restart docker

Example output / evidence

Setting up libnvidia-container1 (1.18.1-1) ...
Setting up nvidia-container-toolkit (1.18.1-1) ...
● docker.service - Docker Application Container Engine
     Active: active (running)
rollback_gpu_positive=PASS
rollback_gpu_isolation_negative=PASS

Checkpoint: Checkpoint: Upgrade and roll back the complete GPU container stack

Continue whenThe canary returns to the former package/configuration/CDI/image matrix, Docker and non-GPU workloads recover, applications meet their baseline, and containers see only assigned UUIDs.

Stop whenFormer packages are unavailable, daemon backup is unverified, host driver compatibility changed, restart impact is unapproved, or workload device assignments cannot be recreated exactly.

If this step fails

Rollback restores package versions but Docker still fails or workloads retain the candidate device configuration.

Likely causeThe former daemon.json, CDI specification, daemon lifecycle and workload recreation were not included in rollback.

Safe checks

Compare package versions, effective Docker configuration and CDI inventory with the previous release record.
Check whether candidate containers remain running or were automatically restarted.
Repeat the pinned one-GPU smoke test and device-isolation negative test.

ResolutionRestore the complete recorded bundle, restart Docker in the approved window, recreate workloads with explicit UUIDs, and verify both positive GPU access and negative isolation before returning capacity.

Security notes

Treat model weights, tokenizer files, chat templates, generation configuration, adapters, container images, native binaries, GPU drivers, and proxy configuration as a single versioned release. A model name alone does not identify what is running.
Keep the inference listener on loopback or a private service network. Terminate TLS, authenticate callers, enforce request-size and concurrency limits, and produce access logs at a separately maintained trusted proxy; do not publish a development listener directly.
Never place registry credentials, API keys, private model URLs, user prompts, generated text, or complete request bodies in commands, unit files, metrics labels, screenshots, or routine logs. Use a secret manager and redact before retention.
Generated text is untrusted data. It must not directly authorize commands, writes, purchases, permission changes, tool calls, or network requests, even when the model claims certainty or emits schema-valid JSON.
A successful health endpoint proves only that a process responds. Production readiness also requires the intended immutable artifact, tokenizer and template, representative quality checks, latency and capacity evidence, isolation, and a rehearsed rollback.

Alternatives

If only one candidate application image fails, restore that image digest without changing the already accepted toolkit and driver.
If the host is unhealthy or its driver cannot be restored safely, keep it drained and move workload to an equivalent verified spare rather than improvising on production capacity.

Stop conditions

Stop before package or daemon changes unless the canary is drained, console recovery is available, former packages/configuration are local, and the exact affected containers are known.
Stop fleet rollout after any GPU visibility broadening, UUID mismatch, driver error, application regression, missing telemetry or rollback rehearsal failure.

Finish line

Verification checklist

Host and runtime healthnvidia-smi && nvidia-ctk --version && docker infoThe host driver, pinned toolkit and Docker daemon are healthy and match the accepted compatibility manifest.

Explicit GPU assignmentdocker run --rm --gpus '"device={{gpuUuid}}"' {{cudaImage}} nvidia-smi -LThe container sees exactly the assigned UUID; a container with no GPU request sees no NVIDIA devices.

CDI lifecyclenvidia-ctk cdi listCDI names match current base GPU and MIG inventory, the runtime supports the emitted schema, and stale identifiers fail closed.

Telemetry and rollbackcurl --fail http://127.0.0.1:{{metricsPort}}/metricsPrivate DCGM metrics identify expected UUIDs, alert rehearsal passes, and the complete former toolkit/configuration/image bundle has a timed restore result.

Recovery guidance

Common problems and safe checks

The host has an NVIDIA GPU, but nvidia-smi reports that it cannot communicate with the driver.

Safe checks

Record lspci, uname -r, loaded NVIDIA modules and the relevant kernel journal without installing another package.
Check the platform driver-installation record and whether a reboot is pending.
Inspect hardware health through the approved out-of-band or vendor diagnostics path.

The container starts, but CUDA reports that the driver version is insufficient for the CUDA runtime.

Likely causeThe container CUDA user-space runtime requires a newer driver than the host provides, or an unsupported forward-compatibility assumption was made.

Safe checks

Capture the host driver version and the exact immutable container image digest and CUDA tag.
Compare the pair with NVIDIA CUDA compatibility documentation.
Run the last known-good CUDA image on the same host to separate driver and candidate-image changes.

ResolutionReturn to the last tested image or upgrade the driver through a staged host-maintenance release; do not copy arbitrary compatibility libraries into a production container.

Package installation fails with conflicting Signed-By entries or duplicate NVIDIA repository definitions.

Likely causeAn older repository configuration, keyring path, distribution-specific setup, or repeated automation created incompatible apt source entries.

Safe checks

List only NVIDIA-related files in /etc/apt/sources.list.d and their Signed-By values.
Compare them with the current official installation guide and the configuration-management source.
Back up the files before removing a duplicate.

Docker reports that it could not select the device driver with capabilities [[gpu]] or does not recognize the nvidia runtime.

Likely causeThe toolkit is not installed, Docker configuration was not generated, daemon.json is invalid, Docker was not restarted, or a rootless daemon uses a different configuration path.

Safe checks

Check nvidia-ctk and package versions, then inspect the effective Docker info output.
Validate /etc/docker/daemon.json as JSON and compare it with the saved pre-change file.
Inspect Docker journal messages after the last restart.

ResolutionRun the reviewed nvidia-ctk runtime configuration for the correct Docker mode, validate and restart during the maintenance window, then repeat a pinned minimal smoke test.

A container requested one GPU but can see every GPU on the host.

Safe checks

Compare the host GPU UUID inventory with nvidia-smi output inside the container.
Inspect the exact Docker create arguments and NVIDIA_VISIBLE_DEVICES value.
Repeat with an explicit GPU UUID and compute,utility driver capabilities.

A requested GPU UUID works before maintenance but disappears or maps incorrectly afterward.

Likely causeThe GPU was replaced, MIG was reconfigured, device indexes changed, or the CDI specification was not regenerated after a change not covered by automatic refresh.

Safe checks

Compare current host UUID and MIG inventory with the assignment registry.
Run nvidia-ctk cdi list and inspect the cdi-refresh service status.
Check whether driver removal or MIG reconfiguration occurred during maintenance.

ResolutionKeep workloads stopped, update the approved inventory, regenerate and validate CDI specifications when required, then assign the new stable UUIDs explicitly rather than falling back to all GPUs.

A rootless or non-root container receives permission denied for NVIDIA or render device nodes.

Likely causeUser namespaces, cgroup device policy, supplementary group injection, CDI support, host device ownership, or container-engine configuration does not match the chosen rootless design.

Safe checks

Inspect device major/minor numbers and ownership without changing permissions.
List generated CDI devices and the rootless engine version and configuration.
Distinguish a cosmetic missing group-name warning from an actual failed device open.

GPU access is lost after systemctl daemon-reload and the application reports Failed to initialize NVML: Unknown Error.

Likely causeA documented interaction between the systemd cgroup driver and device injection caused the running container to lose requested GPU device access.

Safe checks

Correlate the first NVML error with daemon-reload and container lifecycle timestamps.
Inspect whether CDI was used and whether required device nodes remain in the container specification.
Confirm the host GPU and driver still work outside the container.

ResolutionFollow the NVIDIA-documented mitigation for the affected runtime, prefer CDI where supported, recreate the workload in a controlled window, and add daemon-reload to the regression and maintenance test plan.

nvidia-ctk cdi list is empty, stale, or reports a parse/schema error.

Safe checks

Inspect nvidia-cdi-refresh service and path status plus recent journal output.
Validate /var/run/cdi/nvidia.yaml and list the container engine version.
Compare the required CDI schema support with the toolkit release notes.

A container can perform graphics, display, video, or compatibility operations that its compute workload does not need.

Likely causeNVIDIA_DRIVER_CAPABILITIES was set to all or inherited a broad image default instead of the minimum compute and utility capabilities.

Safe checks

Inspect the exact environment and runtime device request for the container.
Compare required application functions with NVIDIA capability definitions.
Run functional tests with only compute,utility in a canary.

ResolutionRedeploy with the minimum documented capability set, verify the workload and monitoring, and add a policy check that rejects all unless a reviewed use case requires it.

DCGM Exporter starts but Prometheus receives no useful GPU metrics or labels the wrong devices.

Likely causeThe exporter lacks required device visibility, the host engine is absent or incompatible, its endpoint is unreachable, a MIG inventory changed, or scrape relabeling dropped series.

Safe checks

Curl the exporter metrics endpoint privately and inspect a small DCGM_FI_DEV sample.
Compare exporter-visible UUID/MIG inventory with nvidia-smi and assignment records.
Inspect Prometheus target health and the pinned exporter image or package revision.

A toolkit or driver upgrade passes nvidia-smi but causes workload crashes, latency regression, missing GPUs, or changed CDI behavior.

Likely causeThe maintenance test checked only host visibility, while runtime hooks, CDI schema, engine compatibility, application CUDA libraries, isolation and representative workload behavior changed.

Safe checks

Diff driver, toolkit packages, Docker configuration, CDI specification, image digest and workload assignment.
Run pinned minimal CUDA plus representative application benchmarks on a drained canary.
Compare GPU error, memory, utilization and latency metrics with the former release.

ResolutionDrain the canary, restore previous toolkit packages, Docker configuration, CDI specification and if necessary the prior tested driver, then expand the compatibility suite before another staged attempt.

Rollback restores package versions but Docker still fails or workloads retain the candidate device configuration.

Likely causeThe former daemon.json, CDI specification, daemon lifecycle and workload recreation were not included in rollback.

Safe checks

Compare package versions, effective Docker configuration and CDI inventory with the previous release record.
Check whether candidate containers remain running or were automatically restarted.
Repeat the pinned one-GPU smoke test and device-isolation negative test.

After the procedure

Alternatives and next steps

Consider these alternatives

Use CDI exclusively on a supported engine to simplify rootless and explicit device injection, but retain current inventory, schema compatibility and negative isolation tests.
Use NVIDIA GPU Operator for Kubernetes when lifecycle ownership belongs to the cluster platform, following its official compatibility and upgrade procedures rather than mixing manual host state.
Use physically separate GPU hosts for tenants or data classes that cannot accept a shared kernel/driver security boundary.
Keep a host CPU-only when GPU capacity is not required; removing privileged device integration is safer than installing it speculatively.

Operate it safely

Automate inventory drift, explicit UUID assignment and forbidden all-capability checks in deployment policy.
Add a quarterly daemon-reload, CDI regeneration, GPU error and complete rollback game day on an equivalent canary.
Track toolkit, driver, Docker and CDI release notes as one platform dependency and schedule compatibility testing before support windows close.
Connect DCGM UUIDs to workload ownership and runbooks without exposing tenant names or request data in broad metric labels.

Reference

Frequently asked questions

Should Docker be configured to give every container all GPUs by default?

No. Request explicit UUID or CDI devices per workload and test that containers without a request see none. Broad defaults make accidental cross-workload access difficult to detect.

Does the container include its own NVIDIA kernel driver?

Normally it uses the host kernel driver while packaging compatible CUDA user-space libraries. That is why host driver, image CUDA and toolkit must be selected and tested together.

Is an inactive nvidia-cdi-refresh service broken?

Not necessarily. It is commonly a one-shot unit triggered by a path/service lifecycle. Inspect its last exit status, journal and generated CDI file, then compare listed devices with current inventory.

Can chmod 666 on GPU device nodes fix non-root access?

It may hide the symptom by granting broad host access and is not an acceptable design. Use supported CDI/rootless configuration, precise group or namespace mapping and explicit devices.

Why test a container with no GPU request?

The positive smoke only proves the assigned path works. The negative test detects permissive defaults or runtime configuration that exposes GPUs to workloads that were never authorized.

Can toolkit packages be upgraded without restarting Docker?

Treat restart and workload recreation requirements as version-specific and test them. A safe production procedure drains a canary, preserves daemon/CDI state and proves the new complete path rather than assuming a live package replacement is effective.

Recovery

Rollback

Drain the canary, stop and recreate GPU workloads, restore the former toolkit package bundle and Docker daemon configuration, regenerate or restore compatible CDI state, restart Docker, verify host and container GPU paths plus negative isolation, and return capacity gradually. Driver rollback is a separate reboot-capable host change when required.

Record the failure, current UUID assignments, candidate package/configuration/image revisions and last healthy metrics, then keep the host out of scheduling.
Stop candidate GPU workloads and preserve their exact create specifications; restore the former /etc/docker/daemon.json with original owner and mode.
Install the retained previous versions of all NVIDIA Container Toolkit package-family members; if the driver changed, follow the separate supported driver rollback and reboot procedure.
Restart Docker during the approved window, confirm unrelated containers, regenerate CDI data for current physical/MIG inventory, and inspect every explicit device name.
Recreate one pinned CUDA smoke container and then each application with exact UUID or CDI assignment and minimum capabilities.
Run negative no-GPU tests, application baselines and DCGM alert checks before gradually returning the host to service.

Evidence

Sources and review

Verified 2026-07-24Review due 2026-10-22

NVIDIA Container Toolkit installation guideofficial NVIDIA Docker GPU enumeration and driver capabilitiesofficial NVIDIA Container Device Interface supportofficial NVIDIA Container Toolkit troubleshootingofficial NVIDIA Container Toolkit release notesofficial NVIDIA CUDA compatibilityofficial NVIDIA DCGM Exporterofficial Docker container runtime securityofficial