OneLinersCommand workbench
Guides
Containers & Kubernetes / Incident Response & Troubleshooting

Fix Docker No space left on device without deleting needed data

Attribute ENOSPC to host blocks, inodes, Docker layers, build cache, writable containers, volumes, or logs before selecting any cleanup.

40 min8 stepsHigh-impact changeRevision 1
Save or explore
Save to collectionCreate a collection in the sidebar first.
0 of 8 steps completed
Goal

Recover enough Docker storage safely, preserve named-volume data, and add controls that prevent the same unbounded growth.

Supported environments
  • Docker Engine 27.x, 28.x
Prerequisites
  • Recovery space Reserve an alternate filesystem or emergency capacity before modifying production Docker data.
  • Workload owner Identify containers and named volumes that hold durable or customer data.
  • Backup evidence Verify recoverability before any prune or volume removal.
Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System
  • A repeatable Docker storage incident record that starts with the literal Docker `No space left on device` symptom and preserves the first useful failure instead of hiding it with an early restart or broad permission change.
  • A layer-by-layer decision path from client context through configuration and runtime evidence to one narrowly scoped remediation, followed by positive and negative verification.
  • A reusable evidence bundle containing commands, concrete example output, timestamps, identities, effective configuration, and stop conditions that another operator can review without access to the original terminal.
Observable outcome
  • You can identify which boundary failed, explain why competing hypotheses were rejected, and state what changed before declaring the incident resolved.
  • The repaired path succeeds under the intended identity while an unauthorized or incorrect path still fails, proving that the fix did not simply remove a security control.
  • The final evidence distinguishes a transient recovery from a durable repair by checking logs, counters, configuration provenance, and the original user-visible operation.

Architecture

How the parts fit together

Docker storage troubleshooting is treated as an evidence pipeline rather than a list of guesses. The operator captures the symptom, resolves client and target context, inspects the first authoritative server-side failure, tests the smallest cause boundary, applies one reviewed change, and repeats the same observation path. This keeps a secondary error from replacing the primary cause and makes rollback possible.

Client contextProves the executable, identity, target, selected configuration, environment variables, and exact request that produced the visible error.
Transport and targetConfirms that the request reached the intended host, socket, service, repository, or upstream instead of a stale context or similarly named environment.
Authoritative evidenceUses effective configuration, service state, event records, and logs from the component that made the decision rather than relying on a client summary.
Controlled remediationChanges one demonstrated cause, retains a backup or previous reference, and avoids global permission, trust, timeout, or force settings.
Verification boundaryRepeats the original operation, exercises a negative case, and watches for recurrence long enough to distinguish a repair from a restart-only recovery.
  1. Copy the literal error and the command that produced it before retrying, restarting, pruning, resetting, or editing anything.
  2. Resolve the selected identity, configuration, target, and current runtime state so later commands inspect the same path the user exercised.
  3. Read the nearest authoritative log or effective configuration and align timestamps across the client and server evidence.
  4. Test one hypothesis with a read-only command; if evidence disagrees, return to the previous layer instead of stacking speculative changes.
  5. Back up the affected configuration or reference, apply the narrow repair, validate syntax, and reload only the component that owns the decision.
  6. Repeat the original action plus a negative control, record the outcome, and keep rollback material until the observation window is complete.

Assumptions

  • DockerRootDir is identified from the affected daemon.
  • Overlay and volume files are not edited directly under /var/lib/docker.
  • Cleanup candidates are reviewed by workload owners.
  • Clocks are close enough that client and server timestamps can be correlated. If they are not, record the offset before comparing logs.
  • The operator has a recovery path that does not depend on the component being changed, such as console access, a second session, or a preserved remote reference.
  • Commands are first run in the affected environment with placeholders reviewed. OneLiners never executes them and does not know local policy, tenancy, or maintenance constraints.

Key concepts

ENOSPC
An allocation failure caused by exhausted blocks, inodes, quotas, or backing storage, not necessarily a large image list.
DockerRootDir
The daemon-owned storage root containing image, layer, metadata, and local volume state.
Reclaimable
Docker's estimate of unreferenced storage; it is not a guarantee that removal has no operational consequence.
Writable layer
Per-container changes above the image that disappear with the container and can grow unexpectedly.
Named volume
Daemon-managed persistent data that can outlive containers and must be backed up before removal.
Set guide values0/1 ready

Fill these once. Every matching command and configuration block updates immediately; values stay in this page only.

Security and production boundaries

  • Storage evidence can expose image names, customer paths, and application logs.
  • Emergency cleanup must not bypass backup retention or incident-evidence requirements.
  • Do not paste private keys, tokens, complete environment dumps, authorization headers, or unredacted customer data into tickets or external analysis tools.
  • A successful operation after disabling authentication, trust, host verification, sandboxing, or least privilege is a security regression, not a valid repair.
  • Prefer effective configuration and narrow identity tests over recursive ownership changes, mode 777, global trust bypasses, or force options copied from unrelated incidents.

Stop before continuing if

  • Stop on I/O errors or a read-only remount.
  • Stop before deleting any named volume without a tested restore.
  • Stop if the target, account, environment, repository, or service instance cannot be identified unambiguously.
  • Stop before a destructive cleanup, force update, broad permission change, or production reload when backup and recovery evidence is missing.
  • Stop if the proposed action would conceal the first error or remove logs, failed objects, repository references, or configuration needed for diagnosis.
01

instruction

Freeze the symptom and define the incident boundary

read-only

Before changing Docker storage, preserve the literal Docker `No space left on device` message, the exact action that produced it, the affected identity, target, UTC time, and expected result. Decide which production boundary is in scope and name the independent recovery path you will keep available.

Why this step matters

A retry, restart, cleanup, permission change, or configuration edit can replace the primary failure with a secondary one. A fixed starting record lets every later check answer a specific hypothesis.

What to understand

Record whether the problem affects one user or workload, one target, one host, or every comparable path; this determines whether to begin at the client, transport, or service boundary.

Write down the expected successful behavior in observable terms such as an exit status, HTTP response, remote object ID, authenticated principal, or stable service state.

Keep console access, a second session, an earlier configuration, or a recoverable reference independent of the component being diagnosed.

System changes

  • No persistent change. This step reads current configuration, identity, runtime state, or logs and records evidence for the incident timeline.
Example output / evidence
Incident scope
Tool: Docker storage
Observed: Docker `No space left on device`
Affected target: production-app-01
Observed at: 2026-08-17T14:02:11Z
Expected: the reviewed operation completes without weakening an access or safety control
Recovery path: console session and last known-good configuration retained

Security notes

  • Redact secrets and customer data, but retain error text, timestamps, object IDs, modes, owners, and target names needed to reproduce the decision.

Alternatives

  • When direct production inspection is restricted, reproduce the same version and configuration boundary in an isolated environment and mark which evidence is illustrative.

Stop conditions

  • Stop before any mutation if the target, identity, affected environment, expected result, or recovery path is still ambiguous.
02

command

Locate the affected storage boundary

read-only

Record DockerRootDir, filesystem blocks, inodes, mounts, and quotas before inspecting Docker objects.

Why this step matters

Block, inode, quota, and mount failures need different repairs, and DockerRootDir may not be the conventional path.

What to understand

Use the path reported by the live daemon; a second daemon or rootless context can use another root.

A full root filesystem can affect journals, package databases, and applications even when Docker is only one contributor.

Record the command, UTC timestamp, exit status, and the exact output before moving to the next layer. A later successful command must not erase evidence of the original failure.

Interpret this result together with the previous checkpoint. One matching line is evidence for a hypothesis, not permission to apply every commonly suggested fix.

System changes

  • No persistent change. This step reads current configuration, identity, runtime state, or logs and records evidence for the incident timeline.

Syntax explained

findmnt -T
Resolves the filesystem and mount options that contain the exact Docker path.
df -hT / df -i
Separately reports byte blocks and inode capacity.
Command
docker info --format 'root={{.DockerRootDir}} driver={{.Driver}}' && findmnt -T /var/lib/docker && df -hT /var/lib/docker && df -i /var/lib/docker
Example output / evidence
root=/var/lib/docker driver=overlay2
TARGET SOURCE    FSTYPE OPTIONS
/      /dev/vda2 ext4   rw,relatime
Filesystem Size Used Avail Use% Mounted on
/dev/vda2   80G  79G  0.6G  100% /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vda2   5.2M 1.1M  4.1M   22% /

Checkpoint: Checkpoint: Locate the affected storage boundary

docker info --format 'root={{.DockerRootDir}} driver={{.Driver}}' && findmnt -T /var/lib/docker && df -hT /var/lib/docker && df -i /var/lib/docker

Continue whenThe exact full boundary and whether blocks or inodes are exhausted are recorded.

Stop whenThe filesystem is read-only, reports I/O errors, or DockerRootDir belongs to another daemon.

If this step fails

df shows free space but Docker still reports ENOSPC.

Likely causeInodes, quota, thin-pool metadata, or an underlying snapshot boundary is exhausted.

Safe checks
  • df -i /var/lib/docker
  • quota -s 2>/dev/null || true
  • dmesg -T | tail -n 100

ResolutionInvestigate the exhausted allocator; do not delete arbitrary image files.

Security notes

  • Do not publish image names, volume paths, or tenant mount details without review.

Alternatives

  • Attach temporary capacity or stop noncritical writers when the filesystem has no working margin.

Stop conditions

  • The filesystem is read-only, reports I/O errors, or DockerRootDir belongs to another daemon.
03

command

Attribute images, containers, volumes, and build cache

read-only

Use Docker's object graph and per-object accounting before filesystem-level deletion.

Why this step matters

Object-aware accounting preserves dependency information that raw du output cannot distinguish safely.

What to understand

Reclaimable images and cache are usually safer first candidates than named volumes.

A large writable layer often means logs, uploads, caches, or application state are being written outside a volume.

Record the command, UTC timestamp, exit status, and the exact output before moving to the next layer. A later successful command must not erase evidence of the original failure.

Interpret this result together with the previous checkpoint. One matching line is evidence for a hypothesis, not permission to apply every commonly suggested fix.

System changes

  • No persistent change. This step reads current configuration, identity, runtime state, or logs and records evidence for the incident timeline.

Syntax explained

docker system df -v
Shows detailed daemon object usage and estimated reclaimable space.
docker ps --size
Adds writable-layer and virtual image size to running-container inventory.
docker buildx du
Shows BuildKit cache records when Buildx is installed.
Command
docker system df -v && docker ps --size --format 'table {{.Names}}\t{{.Image}}\t{{.Size}}' && docker buildx du 2>/dev/null || true
Example output / evidence
TYPE            TOTAL ACTIVE SIZE    RECLAIMABLE
Images          37    6      28.4GB  19.2GB (67%)
Containers      9     6      6.1GB   212MB (3%)
Local Volumes   18    8      31.7GB  4.2GB (13%)
Build Cache     92    0      14.8GB  14.8GB

Checkpoint: Checkpoint: Attribute images, containers, volumes, and build cache

docker system df -v && docker ps --size --format 'table {{.Names}}\t{{.Image}}\t{{.Size}}' && docker buildx du 2>/dev/null || true

Continue whenThe dominant object class and its active owners are identified without deleting anything.

Stop whenUsage cannot be attributed to an owner or the daemon metadata is unhealthy.

If this step fails

Docker accounting hangs or returns daemon errors.

Likely causeThe daemon is unhealthy because metadata or the backing filesystem cannot allocate working space.

Safe checks
  • systemctl status docker --no-pager
  • journalctl -u docker --since '-15 minutes' --no-pager

ResolutionCreate emergency capacity outside Docker metadata or stop writers, then recover the daemon before pruning.

Security notes

  • Object names may expose internal projects; redact them in shared evidence.

Alternatives

  • Use read-only filesystem accounting on the reported DockerRootDir when the daemon cannot answer.

Stop conditions

  • Usage cannot be attributed to an owner or the daemon metadata is unhealthy.
04

command

Inspect container logs and named volumes

read-only

Measure daemon-managed JSON logs, volume mount ownership, and top directories without following data outside the storage boundary.

Why this step matters

Unrotated JSON logs and durable volumes are common large consumers but have different retention, backup, and ownership rules.

What to understand

Truncating an active JSON log manually can race the daemon; configure rotation and use the owning service's retention path.

A volume with no running mount can still be required for rollback, a stopped workload, or backup restoration.

Record the command, UTC timestamp, exit status, and the exact output before moving to the next layer. A later successful command must not erase evidence of the original failure.

Interpret this result together with the previous checkpoint. One matching line is evidence for a hypothesis, not permission to apply every commonly suggested fix.

System changes

  • No persistent change. This step reads current configuration, identity, runtime state, or logs and records evidence for the incident timeline.

Syntax explained

find -printf '%s %p'
Reports exact file size and path without modifying log content.
volume inspect
Maps stable volume names to daemon-owned mountpoints and metadata.
Command
sudo find /var/lib/docker/containers -name '*-json.log' -printf '%s %p\n' | sort -nr | head -20 && docker volume ls -q | while read v; do docker volume inspect --format '{{.Name}} {{.Mountpoint}}' "$v"; done
Example output / evidence
12884901888 /var/lib/docker/containers/7a1.../7a1...-json.log
orders /var/lib/docker/volumes/orders/_data
metrics /var/lib/docker/volumes/metrics/_data

Checkpoint: Checkpoint: Inspect container logs and named volumes

sudo find /var/lib/docker/containers -name '*-json.log' -printf '%s %p\n' | sort -nr | head -20 && docker volume ls -q | while read v; do docker volume inspect --format '{{.Name}} {{.Mountpoint}}' "$v"; done

Continue whenLarge logs and volumes are assigned to a workload owner and retention decision.

Stop whenA candidate volume lacks a verified owner or backup, or log removal would violate retention policy.

If this step fails

The largest usage is under overlay2 but no container appears large.

Likely causeDeleted-open files, layer accounting differences, or an abandoned daemon namespace consumes blocks.

Safe checks
  • sudo lsof +L1 /var/lib/docker
  • sudo du -x -d 2 /var/lib/docker | sort -n | tail

ResolutionMap open files to processes and restart only the owning service after evidence is retained.

Security notes

  • Container logs and volume paths may contain customer data and secrets.

Alternatives

  • Export volume backups to an independent filesystem before any lifecycle cleanup.

Stop conditions

  • A candidate volume lacks a verified owner or backup, or log removal would violate retention policy.
05

command

Preview the smallest safe cleanup set

read-only

List dangling images, stopped containers, unused networks, and cache records; inspect age and ownership before invoking a prune.

Why this step matters

Docker prune commands do not provide a reliable universal dry run, so the candidate set must be built explicitly and reviewed first.

What to understand

An exited migration container or dangling volume can still be rollback evidence.

Do not use `system prune -a --volumes` as an emergency default; it combines object classes with different data-loss risks.

Record the command, UTC timestamp, exit status, and the exact output before moving to the next layer. A later successful command must not erase evidence of the original failure.

Interpret this result together with the previous checkpoint. One matching line is evidence for a hypothesis, not permission to apply every commonly suggested fix.

System changes

  • No persistent change. This step reads current configuration, identity, runtime state, or logs and records evidence for the incident timeline.

Syntax explained

--filter dangling=true
Restricts inventory to untagged or currently unreferenced objects; review is still required.
container ls -a --filter status=exited
Lists stopped containers that retain writable-layer and investigation evidence.
Command
docker image ls --filter dangling=true --format 'image {{.ID}} created={{.CreatedSince}} size={{.Size}}' && docker container ls -a --filter status=exited --format 'container {{.Names}} image={{.Image}} status={{.Status}}' && docker volume ls --filter dangling=true
Example output / evidence
image sha256:ad31... created=6 weeks ago size=1.4GB
container api-migrate-20260701 image=api:2026.07 status=Exited (0) 6 weeks ago
DRIVER VOLUME NAME
local  previous_release_db

Checkpoint: Checkpoint: Preview the smallest safe cleanup set

docker image ls --filter dangling=true --format 'image {{.ID}} created={{.CreatedSince}} size={{.Size}}' && docker container ls -a --filter status=exited --format 'container {{.Names}} image={{.Image}} status={{.Status}}' && docker volume ls --filter dangling=true

Continue whenEvery removal candidate has an owner, age, rollback decision, and expected reclaimed size.

Stop whenAny candidate is recent, unexplained, needed for rollback, or contains unverified volume data.

If this step fails

No safe candidates explain the shortage.

Likely causeActive workload data, logs, another directory, inode exhaustion, or deleted-open files own the space.

Safe checks
  • sudo du -x -d 1 / | sort -n | tail
  • sudo lsof +L1

ResolutionRecover capacity from the demonstrated owner or expand storage; do not force Docker cleanup.

Security notes

  • Retain stopped-container evidence needed for incident response before removal.

Alternatives

  • Expand the filesystem or move a documented DockerRootDir during maintenance when all usage is legitimate.

Stop conditions

  • Any candidate is recent, unexplained, needed for rollback, or contains unverified volume data.
06

command

Remove one reviewed object class

danger

After approval, remove only the selected dangling images first; capture Docker's confirmation and remeasure blocks plus inodes.

Why this step matters

One bounded cleanup preserves auditability and allows the operator to stop as soon as the filesystem regains a safe working margin.

What to understand

The interactive confirmation is part of the safety boundary; automation should use an independently reviewed allowlist and retention rule.

Recheck both bytes and inodes because reclaimed image layers may not address the actual allocator.

Record the command, UTC timestamp, exit status, and the exact output before moving to the next layer. A later successful command must not erase evidence of the original failure.

Interpret this result together with the previous checkpoint. One matching line is evidence for a hypothesis, not permission to apply every commonly suggested fix.

System changes

  • Deletes only reviewed dangling image objects older than seven days and releases their unshared layer data.

Syntax explained

image prune
Removes dangling images rather than all images unused by current containers.
--filter until=168h
Excludes newer candidates from this first cleanup wave.
Command
docker image prune --filter 'until=168h' && docker system df && df -h /var/lib/docker && df -i /var/lib/docker
Example output / evidence
Deleted Images:
deleted: sha256:ad31...
Total reclaimed space: 4.6GB
Filesystem Size Used Avail Use% Mounted on
/dev/vda2 80G 74G 5.3G 94% /

Checkpoint: Checkpoint: Remove one reviewed object class

docker image prune --filter 'until=168h' && docker system df && df -h /var/lib/docker && df -i /var/lib/docker

Continue whenThe named images are removed, reclaimed space is reported, and filesystem headroom exceeds the incident threshold.

Stop whenThe candidate list changes unexpectedly, confirmation names an active image, or backups and rollback references are incomplete.

If this step fails

Prune reports reclaimed space but df barely changes.

Likely causeOpen file descriptors, shared layers, snapshots, thin provisioning, or another full boundary retains blocks.

Safe checks
  • sudo lsof +L1 /var/lib/docker
  • findmnt -T /var/lib/docker

ResolutionFollow the retaining layer or process; do not repeat broader prunes blindly.

Security notes

  • Do not add `-a` or `--volumes` unless those separate candidate sets were reviewed and recoverability proven.

Alternatives

  • Delete an explicitly named object ID after inspection instead of using a class-wide prune.

Stop conditions

  • The candidate list changes unexpectedly, confirmation names an active image, or backups and rollback references are incomplete.
07

decision

Verify headroom and prevent repeat growth

caution

Confirm workloads, configure bounded Docker log rotation in a staged daemon configuration, validate it, and monitor object growth.

Why this step matters

Capacity recovery is temporary unless the owning growth path receives retention, quota, monitoring, and a tested lifecycle policy.

What to understand

Docker's json-file logging driver supports max-size and max-file for newly created containers after daemon configuration.

Monitor bytes, inodes, build cache, volume growth, and log size separately so alerts identify the owner before ENOSPC.

Record the command, UTC timestamp, exit status, and the exact output before moving to the next layer. A later successful command must not erase evidence of the original failure.

Interpret this result together with the previous checkpoint. One matching line is evidence for a hypothesis, not permission to apply every commonly suggested fix.

System changes

  • May stage daemon logging defaults; applying them requires a separately approved daemon reload/restart and container recreation plan.

Syntax explained

dockerd --validate
Checks daemon JSON syntax and supported options without starting a second daemon.
docker system df
Creates the post-change object-accounting baseline.
Command
sudo dockerd --validate --config-file=/etc/docker/daemon.json && docker ps --format 'table {{.Names}}\t{{.Status}}' && docker system df && df -h /var/lib/docker
Example output / evidence
configuration OK
NAMES STATUS
api   Up 18 minutes (healthy)
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 18 6 14.1GB 2.1GB (14%)

Checkpoint: Checkpoint: Verify headroom and prevent repeat growth

sudo dockerd --validate --config-file=/etc/docker/daemon.json && docker ps --format 'table {{.Names}}\t{{.Status}}' && docker system df && df -h /var/lib/docker

Continue whenConfiguration validates, workloads are healthy, and both filesystem and Docker accounting show stable headroom.

Stop whenValidation fails, a restart would affect unapproved workloads, or growth ownership is still unknown.

If this step fails

Existing container logs ignore the new rotation default.

Likely causeLogging defaults apply when containers are created, not retroactively to every existing container.

Safe checks
  • docker inspect --format '{{json .HostConfig.LogConfig}}' {{container}}

ResolutionRoll containers through a controlled deployment with explicit logging options and verify retention.

Security notes

  • Preserve required audit logs in a managed external destination before reducing local retention.

Alternatives

  • Use a logging driver with central retention and backpressure controls rather than large local JSON logs.

Stop conditions

  • Validation fails, a restart would affect unapproved workloads, or growth ownership is still unknown.
08

verification

Observe for recurrence and close with evidence

read-only

After the original Docker storage operation succeeds, repeat the same read-only observation path used at the start. Watch the relevant logs, counters, identities, and target state for a bounded period, then record the proven cause, exact change, verification result, rollback point, and remaining uncertainty.

Why this step matters

A single successful retry may be a transient recovery. Repeating the original checks and retaining the rollback point distinguishes a durable repair from a restart-only improvement.

What to understand

Use the same client identity, target, path, and configuration selection as the original failure so the positive result is comparable.

Include one negative or unauthorized control where safe; this proves the repair did not work by removing authentication, trust, isolation, or branch protection.

Keep the evidence concise enough for another operator to reproduce, but include software versions and exact object or configuration references.

System changes

  • No persistent change. This step reads current configuration, identity, runtime state, or logs and records evidence for the incident timeline.
Example output / evidence
Observation window: 15 minutes
Original operation: PASS
Original signature recurrences: 0
Negative control: PASS
Rollback point retained: yes
Cause and repair recorded: yes
Remaining uncertainty: none observed within the stated boundary

Checkpoint: Checkpoint: the repair remains valid

sudo dockerd --validate --config-file=/etc/docker/daemon.json && docker ps --format 'table {{.Names}}\t{{.Status}}' && docker system df && df -h /var/lib/docker

Continue whenThe intended operation stays healthy, the original signature does not recur, and the negative control still enforces the expected boundary.

Stop whenThe result depends on an unexplained retry, disabled control, different target, or unrecorded manual state.

If this step fails

The error returns during the observation window.

Likely causeThe change treated a symptom, another instance still has the old state, or an automated process reverted or recreated the failing condition.

Safe checks
  • docker info --format 'root={{.DockerRootDir}} driver={{.Driver}}' && findmnt -T /var/lib/docker && df -hT /var/lib/docker && df -i /var/lib/docker
  • sudo dockerd --validate --config-file=/etc/docker/daemon.json && docker ps --format 'table {{.Names}}\t{{.Status}}' && docker system df && df -h /var/lib/docker

ResolutionPreserve the recurrence, compare it with the first evidence set, restore the known-good state if necessary, and reopen the unresolved layer instead of stacking another repair.

Security notes

  • Store only redacted operational evidence and remove temporary debug output according to local retention policy after review.

Alternatives

  • Use the service's approved monitoring or audit trail when an interactive observation window is not practical.

Stop conditions

  • Do not close the incident while the result is intermittent, the rollback point is missing, or a safety control remains weakened.

Finish line

Verification checklist

Original operation succeedssudo dockerd --validate --config-file=/etc/docker/daemon.json && docker ps --format 'table {{.Names}}\t{{.Status}}' && docker system df && df -h /var/lib/dockerThe operation completes with exit status 0 and without the original error. The output identifies the intended host, service, repository, or endpoint.
Failure evidence stays absentdocker info --format 'root={{.DockerRootDir}} driver={{.Driver}}' && findmnt -T /var/lib/docker && df -hT /var/lib/docker && df -i /var/lib/dockerA fresh diagnostic capture shows the healthy path and no recurrence of the original signature. Logs and counters remain stable during the observation window.

Recovery guidance

Common problems and safe checks

The filesystem remounts read-only.

Likely causeStorage or filesystem integrity failure, not ordinary capacity exhaustion.

Safe checks
  • findmnt -T /var/lib/docker
  • dmesg -T | tail -n 100

ResolutionStop writes and follow storage-recovery procedure; do not prune on an unhealthy filesystem.

Inodes are at 100% with modest byte usage.

Likely causeA very large number of small files in layers, volumes, or logs.

Safe checks
  • df -i /var/lib/docker
  • sudo find /var/lib/docker -xdev -printf '%h\n' | sort | uniq -c | sort -n | tail

ResolutionAttribute the high-count directory and use its owning application's retention path.

A named volume is dangling but contains production data.

Likely causeThe workload was removed or renamed while its persistent volume intentionally remained.

Safe checks
  • docker volume inspect VOLUME
  • sudo du -sh /var/lib/docker/volumes/VOLUME/_data

ResolutionBack up, label, and reconnect or archive the volume; do not remove it because of the dangling filter alone.

After the procedure

Alternatives and next steps

Consider these alternatives

  • Expand the filesystem when all large objects are legitimate and retention is already bounded.
  • Move builds to a dedicated builder with independent cache lifecycle.

Operate it safely

  • Add alerts for block and inode headroom plus Docker object growth.
  • Set per-workload log, cache, artifact, and volume retention owners.
  • Turn the verified checks into a read-only health probe or alert using stable fields rather than matching a whole human-formatted line.
  • Record the cause, exact repair, rollback point, software version, and follow-up owner in the incident note so the same failure becomes cheaper to diagnose.
  • Review adjacent environments for the demonstrated cause, but apply changes only where the same evidence is present.

Reference

Frequently asked questions

Is docker system prune -a --volumes safe?

No universal prune is safe. It combines images, containers, networks, build cache, and optionally volumes whose owners and rollback value differ.

Why does Docker say no space when df has free bytes?

The failed allocator may be inodes, quota, thin-pool metadata, a different mount, a rootless storage path, or a deleted-open file.

Can I delete files inside /var/lib/docker?

Do not edit daemon storage directly. Use Docker object lifecycle commands or a documented storage migration after backup.

Why did removing a container not free its volume?

Named volumes have an independent lifecycle so data can outlive containers. That behavior protects persistent state.

Recovery

Rollback

Deleted image and cache objects cannot be restored locally unless they remain in a registry or backup; prevention/configuration changes can be reverted.

  1. Re-pull immutable image digests or restore volume data from a verified backup.
  2. Restore the previous daemon configuration and validate it before a controlled restart.
  3. Recreate workloads from versioned Compose, Swarm, or orchestration definitions.

Evidence

Sources and review

Verified 2026-08-17Review due 2027-02-13
Docker system df referenceofficialDocker prune unused objectsofficialDocker logging driver configurationofficial