Resolve disk-space, inode, and deleted-open-file exhaustion on Linux
Distinguish block, inode, reservation, mount, journal, container, and deleted-open-file pressure; recover a small safe operating margin; identify the owning process or retention policy; and verify that space remains available after normal workload resumes.
Restore service without deleting unknown data, restarting arbitrary processes, crossing filesystem boundaries, or mistaking an empty-looking directory for the filesystem that is actually full, then install a durable retention and monitoring control.
- Ubuntu Server 24.04 LTS
- GNU coreutils 9.x
- systemd 255
- Console or second administrative session A full root filesystem can break SSH, sudo, logging, package operations, and service restarts. Keep an existing root shell or provider console while diagnosing.
- Known data and retention owners Do not delete database files, mail queues, container layers, backups, audit logs, journals, or application state merely because they are large. Identify the owner and required retention first.
- A recovery destination or expansion path Have a separate filesystem, approved backup destination, or documented volume-expansion option if safe cleanup cannot create enough margin.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A layered capacity diagnosis that identifies the exact mounted filesystem and classifies block exhaustion, inode exhaustion, hidden mount behavior, deleted-open allocation, journald retention, ordinary logs, container storage, and real large files without crossing into unrelated filesystems.
- A bounded emergency response that first reclaims an approved amount of archived journal data or releases one proven deleted log through its owning service, rather than recursively deleting whichever directory happens to be large.
- A durable prevention control consisting of an owned logrotate policy, block and inode thresholds, deleted-open-file detection, journal budget, container ownership, and a post-change evidence set after workload writes resume.
- The operator can reconcile df, du, lsof, and inode accounting: df identifies filesystem allocation, du attributes linked namespace entries, and lsof reveals unlinked blocks still charged to open descriptors.
- The affected filesystem regains a documented operating margin in both blocks and inodes, required services resume successfully, and no large unexplained deleted-open file remains.
- The actual growth owner has a retention, rotation, archival, expansion, or application fix with a named owner and alert threshold, so the incident does not recur at the next log burst or small-file storm.
Architecture
How the parts fit together
Linux capacity is accounted at multiple layers. A mount maps a filesystem source to a path. df asks that filesystem about allocated blocks and inodes, while du walks linked directory entries and sums their apparent or allocated usage. An unlinked file disappears from the directory tree but remains allocated until every process closes its descriptor, which is why df can stay high while du looks small. Journald, application loggers, containers, package caches, databases, mail queues, and user data each have separate retention owners. Recovery therefore moves from filesystem identity to attribution, then applies the smallest owner-aware action before durable policy and alerting.
- df and findmnt identify the actual full filesystem, its type, source, options, byte margin, and inode margin.
- du, find, journalctl, Docker accounting, and inode counts attribute visible consumption while staying inside the selected filesystem.
- lsof finds blocks held by unlinked paths, reconciling a large df-versus-du gap with the process that still owns the descriptor.
- The operator reclaims a small approved emergency margin, then uses the data owner's supported restart, rotation, archival, cleanup, or expansion process.
- The same measurements run after normal workload resumes, and durable retention plus monitoring are accepted only when free blocks and inodes remain stable.
Assumptions
- The examples target Ubuntu 24.04 LTS with GNU coreutils, systemd-journald, lsof, and optional Docker. Filesystem-specific reserved-space, project-quota, copy-on-write, snapshot, and thin-provisioning behavior may add another layer.
- The target mountpoint is known or can be derived from the failing application path. Commands use `-x` or `-xdev` to avoid traversing separate local, network, container, or bind mounts.
- The operator has a console or existing privileged session because a full root filesystem can prevent new logins, temporary files, sudo authentication, package changes, and service state updates.
- Retention and regulatory owners can approve deletion or archival. Large data is not assumed disposable because its path contains log, cache, tmp, backup, queue, or old.
- A filesystem or logical-volume expansion path exists when safe cleanup cannot create sustainable margin; emergency deletion is not used to postpone a known capacity deficit indefinitely.
Key concepts
- Allocated blocks versus directory usage
- df reports filesystem allocation, while du walks linked names. Reserved blocks, metadata, snapshots, sparse-file accounting, hidden mounts, and deleted-open files can make the values differ.
- Inode exhaustion
- Many filesystems allocate a finite number of inodes. Creating a file needs an inode even when data blocks remain, so millions of tiny files can produce ENOSPC with apparent byte capacity.
- Deleted-open file
- unlink removes a directory name, but an open descriptor keeps the inode and blocks alive. Space returns only when the owning process closes that descriptor or exits.
- Mount shadowing
- Mounting a filesystem over a nonempty directory hides the underlying files without deleting them. du on the mounted view cannot see hidden parent-filesystem usage.
- Retention owner
- The application or policy responsible for data lifecycle. Correct cleanup uses its supported rotation, archival, compaction, vacuum, expiry, or backup process.
- Emergency margin
- A small amount of safely reclaimed capacity that lets logging, service control, and recovery tools operate. It is a stabilization step, not the root-cause fix.
Before you copy
Values used in this guide
{{mountpoint}}A path on the affected filesystem; use `/` for root or the exact application mount rather than scanning every filesystem.
Example: /{{largeFileMiB}}Minimum regular-file size for the bounded initial search, expressed in MiB without the suffix.
Example: 512{{journalVacuumSize}}Approved total journal budget passed to `--vacuum-size`; choose a value that preserves required incident and audit history.
Example: 1G{{serviceName}}Exact systemd unit proven by lsof to own the deleted log and able to reopen it safely.
Example: api.service{{logrotateName}}Safe filename for the dedicated policy under /etc/logrotate.d, normally matching the application.
Example: api{{logFile}}Absolute active log path owned by the application; verify it is not a symlink to another policy domain.
Example: /var/log/api/api.log{{logOwner}}Operating-system account that should own each newly created active log.
Example: api{{logGroup}}Group allowed to read the application log according to least privilege.
Example: admSecurity and production boundaries
- Logs, journals, queues, core dumps, and temporary files can contain credentials, personal data, request bodies, tokens, or incident evidence. Do not move them to an unprotected filesystem or public paste during cleanup.
- Recursive deletion as root is irreversible and vulnerable to path, mount, symlink, and ownership mistakes. This guide diagnoses first and uses owner-specific retention instead of broad `rm -rf`, `find -delete`, or wildcards.
- Restarting a service may interrupt traffic, replay queues, lose in-memory state, or break a forensic timeline. Verify the PID-to-unit mapping and application consistency before using restart to release a descriptor.
- Journal vacuum and log rotation alter available forensic history. Record the reason, before/after size, retention approval, and any archived copy.
Stop before continuing if
- Stop deletion if the data owner, retention requirement, backup state, filesystem identity, or mount boundary is unknown.
- Stop and preserve evidence when the full filesystem contains audit, security, mail queue, database, or incident data that may be legally or operationally required.
- Do not restart an unknown PID, kill a database, truncate an open file, or write through `/proc/<pid>/fd` merely to reclaim blocks.
- Stop cleanup when block or inode usage immediately returns; cap writes or stop the generating workload and fix the root cause before it consumes the emergency margin.
- Move to controlled expansion or migration when approved cleanup cannot restore the documented headroom without deleting required data.
verification
Classify block and inode exhaustion on the exact filesystem
Use df for allocated filesystem accounting, not directory size. Compare human-readable block use, byte-stable output, and inode use for the selected mountpoint.
Why this step matters
The string ENOSPC does not identify which resource is exhausted. Measuring both allocated blocks and inodes on the exact path prevents byte cleanup from being applied to an inode problem or another mount.
What to understand
df resolves the supplied path to its containing filesystem. Use the same path the failing service writes rather than assuming root is the only constrained mount.
Human output is useful for operators; byte output is stable for comparisons and incident records. The percentage can round to 100 while some bytes remain.
Available blocks may be lower than free blocks because filesystem policy reserves space or quota excludes the current user.
System changes
- No changes; captures initial block and inode accounting for the affected filesystem.
Syntax explained
df -hT- Shows human-readable capacity and filesystem type for rapid classification.
df -B1 --output=...- Uses exact one-byte units and selected fields for before/after comparison.
df -i- Reports inode totals and availability instead of data-block usage.
Values stay on this page and are never sent or saved.
df -hT {{mountpoint}} && df -B1 --output=source,fstype,size,used,avail,pcent,target {{mountpoint}} && df -i {{mountpoint}}Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg-root ext4 40G 39G 312M 100% / Filesystem Type 1B-blocks Used Available Use% Mounted on /dev/mapper/vg-root ext4 42949672960 42078208000 327155712 100% / Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/vg-root 2621440 2617312 4128 100% /
Checkpoint: The exhausted resource and filesystem source are known
df -B1 {{mountpoint}} && df -i {{mountpoint}}Continue whenThe incident records exact source, type, mount, free bytes, free inodes, and which threshold failed.
Stop whenThe path resolves to a different mount than expected or the filesystem is already read-only.
If this step fails
df itself fails or cannot create temporary output.
Likely causeThe environment or shell startup needs temporary space, or the filesystem has broader metadata errors.
/bin/df -kP {{mountpoint}}findmnt -T {{mountpoint}}
ResolutionUse the existing console shell with absolute command paths, minimize writes, and investigate filesystem state before cleanup.
Security notes
- Filesystem names and capacity can expose infrastructure layout; share only the needed incident summary.
Alternatives
- Use filesystem-native accounting in addition to df for ZFS, Btrfs, thin pools, quotas, or snapshots.
Stop conditions
- Do not delete bytes until block versus inode exhaustion is explicitly classified.
verification
Confirm mount identity, options, and filesystem state
Resolve the target to its actual source and parent topology, confirm it is mounted read-write, and capture filesystem block statistics. This prevents scanning a hidden parent directory or cleaning the wrong layer.
Why this step matters
Path-based cleanup is safe only when the mounted source and its boundaries are understood. A hidden underlying directory, read-only remount, thin device, or unexpected bind mount can make correct-looking commands affect the wrong storage.
What to understand
findmnt -T resolves the deepest mount containing the path and shows source, type, options, and capacity in one record.
lsblk connects device-mapper, partition, RAID, and filesystem layers; expansion must target the owner, not merely the mounted leaf.
stat -f exposes block and inode figures from the filesystem API and helps corroborate df when aliases are confusing.
System changes
- No changes; records mount source, options, block topology, and filesystem-level statistics.
Syntax explained
findmnt -T- Finds the filesystem containing the supplied path rather than matching mount text approximately.
lsblk -f- Shows filesystem labels, UUIDs, types, mounts, and parent/child block relationships.
stat -f- Reads filesystem metadata and counts for the selected path.
Values stay on this page and are never sent or saved.
findmnt -T {{mountpoint}} -o TARGET,SOURCE,FSTYPE,OPTIONS,SIZE,USED,AVAIL,USE% && lsblk -f && stat -f {{mountpoint}}TARGET SOURCE FSTYPE OPTIONS SIZE USED AVAIL USE%
/ /dev/mapper/vg-root ext4 rw,relatime 40G 39.2G 312M 99%
File: "/"
ID: 9e7cc7464a57d380 Namelen: 255 Type: ext2/ext3
Block size: 4096 Fundamental block size: 4096
Blocks: Total: 10485760 Free: 79872 Available: 79872
Inodes: Total: 2621440 Free: 4128Checkpoint: The affected mount and storage owner are identified
findmnt -T {{mountpoint}} -o TARGET,SOURCE,FSTYPE,OPTIONSContinue whenOne reviewed mount source and type, expected read-write options, and a known expansion or recovery owner.
Stop whenThe mount is read-only, source is unexpected, or underlying storage health is degraded.
If this step fails
The application path is on a bind mount and usage points elsewhere.
Likely causeA bind mount presents another directory tree while accounting remains on the source filesystem.
findmnt -T {{mountpoint}} -o TARGET,SOURCE,FSTYPE,OPTIONSfindmnt -R {{mountpoint}}
ResolutionTrace the bind source to its containing filesystem and perform attribution there, preserving the application's path ownership.
Security notes
- Do not remount read-write automatically; a filesystem may be read-only because the kernel detected errors.
Alternatives
- Use `lvs`, `zfs list`, `btrfs filesystem usage`, or storage-array tooling when the mount is only the top of a layered capacity problem.
Stop conditions
- Stop cleanup when kernel or filesystem health, not ordinary capacity, caused a read-only remount.
verification
Measure top-level usage without crossing filesystems
Use du on the affected filesystem only and sort direct children. Excluding other mounts keeps network storage, containers, and bind mounts from distorting the result or causing a slow fleet-wide crawl.
Why this step matters
A bounded top-level walk locates the broad owner without generating an expensive all-files listing. Staying on one filesystem prevents unrelated network or nested mounts from inflating totals and consuming incident time.
What to understand
`du -x` prevents crossing into other filesystems even when their mountpoints are nested below the selected path.
Byte units support exact sorting; `--max-depth=1` reports direct children and the root total before drilling into one candidate.
Capture permission and race errors separately. Files disappearing during a live scan is common, but widespread read errors may indicate another incident.
System changes
- Creates a small diagnostic error file under /var/tmp; no target data changes.
Syntax explained
-x- Restricts traversal to the same filesystem device as the starting path.
-B1- Reports allocated sizes in exact bytes for numeric sorting.
--max-depth=1- Limits aggregation to one level so the next drill-down is deliberate.
Values stay on this page and are never sent or saved.
sudo du -x -B1 --max-depth=1 {{mountpoint}} 2>/var/tmp/du-errors.log | sort -n | tail -20 && sudo sed -n '1,40p' /var/tmp/du-errors.log14548992 /opt 38404096 /etc 804257792 /home 3652194304 /usr 9311354880 /var 42074112000 / du: cannot read directory '/proc/2841/fd/5': No such file or directory
Checkpoint: One or more top-level owners explain visible allocation
sudo du -x -B1 --max-depth=1 {{mountpoint}} | sort -n | tailContinue whenThe largest same-filesystem directories are ranked and their total is compared with df.
Stop whenThe scan causes unacceptable I/O, returns filesystem errors, or its total is far below df without deleted-open checks.
If this step fails
du takes hours and increases storage latency.
Likely causeThe filesystem contains millions of entries, slow media, or busy metadata and requires a narrower starting path.
findmnt -T {{mountpoint}}iostat -xz 2 3 2>/dev/null || true
ResolutionStop the broad scan and target known top-level owners or use filesystem/application-native accounting.
Security notes
- Directory names in output may identify customers or projects; keep the raw ranking restricted.
Alternatives
- Use `ncdu -x` interactively only on an approved host and package source; it does not solve deleted-open or snapshot accounting.
Stop conditions
- Do not cross network, backup, pseudo, or unrelated container mounts during an emergency scan.
verification
Find large regular files on the affected filesystem
Search only the target filesystem for regular files above the reviewed threshold, print byte size and path, and sort numerically. Inspect age, owner, links, and application purpose before moving or deleting anything.
Why this step matters
Large-file ranking identifies candidates for owner review but deliberately avoids deletion. Size, age, owner, and path together distinguish an abandoned archive from an active database segment of the same size.
What to understand
`-xdev` retains the same mount boundary as the prior du step, and `-type f` excludes device nodes, sockets, and directories.
The threshold is a triage tool. Lower it only inside an identified large directory rather than generating millions of rows from the root.
A database or mail-store file must be managed through its application even when it is the largest object. Removing it below the service can corrupt the dataset.
System changes
- Creates only a diagnostic error log; no candidate file is moved, truncated, or deleted.
Syntax explained
-xdev- Keeps find on the filesystem containing the start path.
-size +{{largeFileMiB}}M- Selects regular files larger than the reviewed MiB threshold.
-printf- Emits exact size, modification time, owner, group, and path for accountable review.
Values stay on this page and are never sent or saved.
sudo find {{mountpoint}} -xdev -type f -size +{{largeFileMiB}}M -printf '%s\t%TY-%Tm-%Td %TH:%TM\t%u:%g\t%p\n' 2>/var/tmp/find-errors.log | sort -n | tail -301073741824 2026-07-14 02:00 backup:backup /var/backups/app-weekly.tar 4294967296 2026-07-28 01:16 syslog:adm /var/log/api/api.log 8589934592 2026-07-21 04:00 postgres:postgres /var/lib/postgresql/16/main/base/16384/2619
Checkpoint: Each large candidate has a retention owner
Continue whenThe incident record identifies active, archived, regenerable, backed-up, and unknown files without deleting any unknown item.
Stop whenA candidate belongs to a database, queue, audit trail, backup, or unknown application.
If this step fails
A single sparse file has a huge apparent size but modest disk allocation.
Likely causeLogical length contains holes; the printed `%s` is apparent bytes, while allocated blocks are lower.
ls -lh {{logFile}}du -h {{logFile}}stat {{logFile}}
ResolutionCompare apparent and allocated size before blaming the file; use the owning application's policy if allocation really needs reduction.
Security notes
- Never upload or sample large files until their data classification is known.
Alternatives
- Use application-native size views for databases, object stores, queues, and mail systems.
Stop conditions
- A large path is evidence for ownership review, never automatic deletion authority.
verification
Find deleted files that processes still hold open
Compare df and du, then list files whose directory link is gone but whose blocks remain charged because a process retains an open descriptor. Sum field output to estimate recoverable bytes without parsing human text.
Why this step matters
Unlinked open files are invisible to normal directory walks but remain fully allocated. Mapping them to a process often explains the exact df-versus-du gap and enables recovery through a safe reopen.
What to understand
`+L1` selects files whose link count is below one, the conventional unlinked-open case. Supplying the mountpoint scopes the scan.
Field mode with NUL records avoids parsing aligned human columns and allows retained bytes to be summed reproducibly.
The process may be a logger, database, web server, container runtime, or deleted executable mapping. Its unit and application semantics determine the release method.
System changes
- No changes; reads process file-descriptor tables and calculates retained bytes.
Syntax explained
+L1- Selects open files with link count less than one, meaning no directory name remains.
-Fn0- Uses machine-readable fields with NUL record separators for safer aggregation.
lsof {{mountpoint}}- Scopes the search to the affected filesystem rather than all processes and mounts.
Values stay on this page and are never sent or saved.
sudo lsof +L1 {{mountpoint}} && sudo lsof -Fn0 +L1 {{mountpoint}} | awk -v RS='\0' '/^s[0-9]+$/ {sum+=substr($0,2)} END {printf "deleted_open_bytes=%d\n",sum}'COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME api 2841 api 4w REG 253,0 4294967296 0 19421 /var/log/api/api.log.1 (deleted) deleted_open_bytes=4294967296
Checkpoint: The df-versus-du gap is reconciled with retained descriptors
sudo lsof +L1 {{mountpoint}}Continue whenEvery large unlinked file has a PID, service, retained byte count, and approved release action.
Stop whenThe process is unknown, critical, unmanaged, or preserving incident evidence.
If this step fails
lsof shows the PID but systemctl cannot find its unit.
Likely causeThe process belongs to a container, user session, supervisor, cron job, or manually launched daemon.
ps -o pid,ppid,user,lstart,cmd -p 2841cat /proc/2841/cgroup
ResolutionIdentify the real supervisor and workload owner; do not kill the PID until consistency and restart behavior are understood.
Security notes
- Process arguments and file names can contain secrets; redact displayed evidence carefully.
Alternatives
- Use `/proc/<pid>/fd` only for read-only correlation; avoid truncating descriptors as an emergency shortcut.
Stop conditions
- Do not restart or kill a process solely because lsof shows a large deleted file.
verification
Locate inode-heavy directories and small-file storms
When free inodes are scarce, rank directories by inode count rather than bytes. A million tiny cache, session, queue, or spool files can stop new file creation while gigabytes remain free.
Why this step matters
Inode incidents require counting names, not chasing large bytes. Ranking directory inode use reveals cache, session, queue, spool, or extraction storms that can prevent every service from creating files.
What to understand
`du --inodes` counts directory entries within the same filesystem and makes tiny-file owners visible.
A three-level view is a starting point; narrow further inside the dominant owner to reduce traversal cost.
Deletion must use the application's expiration or queue semantics. Removing mail, job, package, or session files behind its back can violate consistency.
System changes
- Creates only a diagnostic error log; no inode or file is removed.
Syntax explained
--inodes- Reports inode counts instead of allocated bytes.
--max-depth=3- Balances localization with bounded output; adjust only after identifying the owner.
-x- Prevents counting separate mounted filesystems.
Values stay on this page and are never sent or saved.
sudo du -x --inodes --max-depth=3 {{mountpoint}} 2>/var/tmp/du-inode-errors.log | sort -n | tail -3018240 /var/lib/systemd 43782 /var/log 220944 /var/spool/postfix 1840132 /var/cache/app/sessions 2617312 /
Checkpoint: The small-file owner and lifecycle are identified
sudo du -x --inodes --max-depth=3 {{mountpoint}} | sort -n | tailContinue whenOne or more directories explain inode use and have an application-approved expiry or migration action.
Stop whenThe directory is an active queue, spool, package database, or unknown state store.
If this step fails
Inode counts remain high after visible files are expired.
Likely causeAnother directory is dominant, deleted-open files remain, snapshots retain metadata, or the cleanup process recreated entries.
df -i {{mountpoint}}du -x --inodes --max-depth=2 {{mountpoint}} | sort -n | tail
ResolutionRe-rank after cleanup, stop the generating workload if growth continues, and inspect filesystem-specific metadata or snapshots.
Security notes
- Session, queue, and spool filenames may encode user or message identifiers.
Alternatives
- Partition or allocate a separate filesystem for unbounded small-file workloads so they cannot exhaust root inodes.
Stop conditions
- Do not use broad `find -delete` against an application-owned directory.
verification
Measure journal and log ownership before cleanup
Read journald's own disk accounting, list archived journal files, inspect log directory size, and identify logrotate policy. Journal retention and plain-text application logs are separate owners.
Why this step matters
journald and plain logs have different retention mechanisms. Measuring both and validating logrotate before cleanup prevents duplicate ownership and preserves required history.
What to understand
`journalctl --disk-usage` reports active plus archived journal consumption according to journald's own files.
du identifies plain-text or application log directories; a large path may have no working logrotate rule or the service may not reopen it.
`logrotate --debug` parses and simulates decisions without rotating files. Warnings and duplicate patterns must be resolved before force testing.
System changes
- Creates no rotations; reads journal usage, log directory allocation, and logrotate configuration.
Syntax explained
journalctl --disk-usage- Reports space used by all active and archived journal files visible to journald.
logrotate --debug- Parses configuration and explains decisions without modifying logs.
du -x -h --max-depth=2 /var/log- Ranks visible log directories on the local filesystem at human-readable scale.
sudo journalctl --disk-usage && sudo du -x -h --max-depth=2 /var/log | sort -h | tail -20 && sudo logrotate --debug /etc/logrotate.conf 2>&1 | sed -n '1,100p'Archived and active journals take up 3.8G in the file system. 220M /var/log/apt 612M /var/log/journal 4.1G /var/log/api 5.2G /var/log reading config file /etc/logrotate.conf including /etc/logrotate.d reading config file api rotating pattern: /var/log/api/api.log after 1 days (14 rotations)
Checkpoint: Journal and application-log ownership are separated
Continue whenThe incident records journal bytes, largest log directories, matching rotation policies, and retention approval.
Stop whenAudit or incident history would be removed without authorization, or logrotate reports syntax conflicts.
If this step fails
The application log has rotated names but the active process still writes an old deleted file.
Likely causeThe postrotate signal is missing, wrong, or unsupported by the service.
lsof +L1 /var/loglogrotate --debug /etc/logrotate.conf
ResolutionUse the service's documented reopen signal or controlled restart, then verify the new active path receives writes.
Security notes
- Journal and audit history may be security evidence; document every vacuum or retention change.
Alternatives
- Forward logs to a protected remote system before shortening local retention, with delivery and query verification.
Stop conditions
- Do not delete journal directories or current log files manually.
verification
Measure container storage without pruning
If Docker is installed, use its accounting and detailed views to distinguish images, build cache, writable layers, and volumes. A large Docker root is not authorization to prune volumes or images used for rollback.
Why this step matters
Container storage can dominate a host while its overlay and volume layout makes ordinary directory deletion unsafe. Docker's accounting identifies reclaim classes without changing them.
What to understand
Images can share layers, so summed apparent image sizes overstate unique reclaimable space. The verbose view separates shared and unique values.
Volumes often contain databases and durable application state. A zero-container link count is not sufficient retention approval.
Build cache and dangling images may be regenerable, but removal can slow recovery or eliminate a known rollback artifact.
System changes
- No Docker objects change; the command reads image, container, volume, and build-cache accounting.
Syntax explained
docker system df -v- Displays detailed shared, unique, active, and reclaimable estimates without pruning.
command -v docker- Avoids treating an absent optional container runtime as a command failure.
if command -v docker >/dev/null 2>&1; then sudo docker system df -v; else printf 'Docker is not installed on this host.\n'; fiImages space usage: REPOSITORY TAG IMAGE ID CREATED SIZE SHARED SIZE UNIQUE SIZE CONTAINERS api 2026.07 1a2b3c4d5e6f 6 days ago 812MB 610MB 202MB 2 <none> <none> 7f6e5d4c3b2a 31 days ago 790MB 590MB 200MB 0 Local Volumes space usage: VOLUME NAME LINKS SIZE api_postgres 1 18.4GB Build cache usage: 6.7GB
Checkpoint: Every large container object has an owner and retention decision
Continue whenActive volumes are protected, rollback images are identified, and only explicitly approved unused objects become cleanup candidates.
Stop whenA volume, stopped container, image, or cache has unknown application or recovery value.
If this step fails
Docker accounting is modest but /var/lib/docker is much larger.
Likely causeContainer logs, orphaned overlay data, plugin state, or a runtime inconsistency is outside the summarized classes.
docker info --format '{{json .DockerRootDir}}'du -x -h --max-depth=2 /var/lib/docker | sort -h | tail
ResolutionIdentify the runtime owner and inspect daemon logs; do not delete overlay or metadata directories manually.
Security notes
- Volume names and image metadata can reveal application topology; protect the detailed inventory.
Alternatives
- Use the corresponding CRI, containerd, Podman, or Kubernetes accounting and garbage-collection policy when Docker is not the owner.
Stop conditions
- Never add `--volumes` or broad prune flags during diagnosis without separate owner approval.
command
Recover a bounded emergency margin from archived journals
When journald is a confirmed major owner and retention policy permits it, vacuum only archived journals to a reviewed total size. Capture before and after accounting; active journal files remain.
Why this step matters
A bounded journal vacuum can restore enough capacity for service control and logging without making arbitrary filesystem-wide deletions. It is appropriate only after journald is proven to be a material owner.
What to understand
`--vacuum-size` removes oldest archived journals until archived plus active storage approaches the requested bound; active journal files are not removed by the vacuum.
The actual freed amount is printed and must be compared with df on the affected filesystem. A small result means another owner is responsible.
Choose the budget from incident, audit, and compliance retention. Export required history before vacuuming when local copies are the only evidence.
System changes
- Deletes older archived journal files until journal storage is within `{{journalVacuumSize}}`, permanently removing that local history.
Syntax explained
--vacuum-size={{journalVacuumSize}}- Bounds total retained journal storage by expiring archived files from oldest to newest.
--disk-usage- Provides before and after journal accounting from journald itself.
Values stay on this page and are never sent or saved.
sudo journalctl --disk-usage && sudo journalctl --vacuum-size={{journalVacuumSize}} && sudo journalctl --disk-usage && df -hT {{mountpoint}}Archived and active journals take up 3.8G in the file system. Deleted archived journal /var/log/journal/7a.../system@00063a.journal (128.0M). Vacuuming done, freed 2.9G of archived journals. Archived and active journals take up 896.0M in the file system. /dev/mapper/vg-root ext4 40G 36G 3.2G 92% /
Checkpoint: Emergency margin is restored and retained history is acceptable
journalctl --disk-usage && df -B1 {{mountpoint}}Continue whenJournal storage is within the approved budget and the filesystem has enough bytes for controlled remediation.
Stop whenRequired security history is not archived, journald is not the major owner, or the freed space disappears immediately.
If this step fails
Vacuum reports zero bytes freed.
Likely causeMost space belongs to active journal files, the requested target exceeds current use, or another path owns the filesystem.
journalctl --disk-usagedu -x -h /var/log/journal/*/* | sort -h | tail
ResolutionDo not delete active journal files. Rotate journald through supported controls if appropriate and continue attribution for other owners.
Security notes
- Vacuum permanently removes local history; retain approval and an external evidence copy when required.
Alternatives
- Move or expand storage when retention cannot be reduced safely.
Stop conditions
- Do not vacuum below the incident and audit retention floor.
command
Release a deleted file through its owning service
When lsof proves that one named service holds a large deleted log, validate its unit and restart or reload it through the supported lifecycle. Recheck lsof and df immediately; never kill an arbitrary PID solely to reclaim blocks.
Why this step matters
The service that owns a deleted descriptor is the only safe place to reopen or close it with application semantics. An approved restart releases the old inode and restores normal logging to a named path.
What to understand
Check the unit, current health, dependencies, and maintenance impact before restart. Some services support a no-downtime reopen signal instead.
After restart, lsof should no longer show the old unlinked descriptor and df should reclaim roughly its allocated size.
Confirm the service writes to the current active log and does not immediately recreate runaway growth.
System changes
- Restarts `{{serviceName}}`, closes its old file descriptors, and may interrupt in-memory workload state or traffic.
Syntax explained
systemctl restart- Performs the unit's configured stop and start lifecycle, closing inherited descriptors.
systemctl is-active- Confirms supervisor state but must be supplemented with application health.
lsof +L1- Verifies the specific retained allocation disappeared after lifecycle action.
Values stay on this page and are never sent or saved.
sudo systemctl status {{serviceName}} --no-pager && sudo systemctl restart {{serviceName}} && sudo systemctl is-active {{serviceName}} && sudo lsof +L1 {{mountpoint}} && df -hT {{mountpoint}}● api.service - Example API
Loaded: loaded (/etc/systemd/system/api.service; enabled)
Active: active (running) since Tue 2026-07-28 03:42:01 UTC
active
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg-root ext4 40G 32G 6.7G 83% /Checkpoint: Deleted allocation is released and the service is healthy
systemctl is-active {{serviceName}} && lsof +L1 {{mountpoint}} && df -hT {{mountpoint}}Continue whenService and application health pass, the target descriptor is gone, and expected space is available.
Stop whenThe service has no safe restart path, stores critical in-memory state, or is not the proven owner.
If this step fails
The service restarts but the same deleted path reappears.
Likely causeRotation policy deletes the active file again or the application continues writing an old path through another worker.
lsof +L1 {{mountpoint}}systemctl status {{serviceName}} --no-pagerlogrotate --debug /etc/logrotate.conf
ResolutionCorrect rotation and ensure every worker receives the supported reopen signal before another production rotation.
Security notes
- Preserve application and security evidence before restart changes process lifetime and logs.
Alternatives
- Use the documented HUP, USR1, reopen command, or graceful reload when it reliably closes logs without a full restart.
Stop conditions
- Do not kill arbitrary PIDs or truncate `/proc/<pid>/fd/*` to force reclamation.
config
Install and validate an owned log rotation policy
Create a dedicated policy for the confirmed application log, with bounded daily retention and a post-rotate signal that makes the process reopen its path. Validate in debug mode before forcing a controlled test.
Why this step matters
A correct rotation policy fixes the mechanism that produced the deleted-open or runaway log, while bounded retention prevents root capacity from becoming an accidental archive.
What to understand
The dedicated file avoids editing global policy and makes ownership review clear. Daily plus size creates a time and volume trigger.
`create` must match the application's required owner, group, and mode; otherwise the process may fail to reopen or expose sensitive logs.
`sharedscripts` runs postrotate once for the pattern, and the HUP must be explicitly supported by the service. Debug first, then force one controlled test.
Compression is delayed so a process that briefly retains the rotated file does not conflict with immediate compression.
System changes
- Creates `/etc/logrotate.d/{{logrotateName}}`, rotates `{{logFile}}` in the controlled test, compresses older generations, and signals `{{serviceName}}`.
Syntax explained
rotate 14 / daily / size 100M- Combines bounded generation count, daily consideration, and a reviewed size threshold.
create 0640 {{logOwner}} {{logGroup}}- Creates the replacement active log with explicit least-privilege ownership and mode.
postrotate ... HUP- Requests the named service to reopen logs; it is valid only when the service documents HUP semantics.
logrotate --debug / --force- First validates without changes, then performs one deliberate rotation to prove the full lifecycle.
/etc/logrotate.d/{{logrotateName}}Values stay on this page and are never sent or saved.
{{logFile}} {
daily
rotate 14
size 100M
missingok
notifempty
compress
delaycompress
create 0640 {{logOwner}} {{logGroup}}
sharedscripts
postrotate
/usr/bin/systemctl kill -s HUP {{serviceName}} 2>/dev/null || true
endscript
}Values stay on this page and are never sent or saved.
sudo logrotate --debug /etc/logrotate.d/{{logrotateName}} && sudo logrotate --force /etc/logrotate.d/{{logrotateName}} && sudo systemctl is-active {{serviceName}} && sudo lsof {{logFile}}reading config file /etc/logrotate.d/api rotating pattern: /var/log/api/api.log forced from command line (14 rotations) renaming /var/log/api/api.log to /var/log/api/api.log.1 creating new /var/log/api/api.log mode = 0640 uid = 991 gid = 4 running postrotate script active api 2922 api 4w REG 253,0 0 20294 /var/log/api/api.log
Checkpoint: Rotation parses, rotates, reopens, and preserves service health
sudo logrotate --debug /etc/logrotate.d/{{logrotateName}} && systemctl is-active {{serviceName}} && lsof {{logFile}}Continue whenNo syntax conflict, correct active file ownership, service holding the current path, and bounded archives present.
Stop whenHUP is unsupported, ownership is wrong, policy duplicates another rule, or required retention exceeds the configured generations.
If this step fails
logrotate reports duplicate log entry.
Likely causeAnother package or local policy already owns the same path.
grep -R -F '{{logFile}}' /etc/logrotate.conf /etc/logrotate.dlogrotate --debug /etc/logrotate.conf
ResolutionMaintain exactly one authoritative policy, merge the reviewed settings there, and retest without forcing competing configurations.
Security notes
- Use restrictive modes and ensure compressed archives remain protected; rotation must not broaden access.
Alternatives
- Configure native structured logging, remote archival, or the application's internal rotation when it provides stronger delivery and reopen guarantees.
Stop conditions
- Do not force rotation until debug output and service reopen semantics are approved.
verification
Verify sustained capacity, inode recovery, and service health
Repeat the same block, inode, deleted-file, journal, and workload checks after normal writes resume. Record the reclaimed owner and install alerts below emergency thresholds.
Why this step matters
Capacity incidents often recur immediately because the producer remains active or the wrong resource was cleaned. Repeating every original measurement after workload resume proves sustained recovery rather than a momentary dip.
What to understand
Compare exact free bytes and inodes with the before-state and with operational alert thresholds, not only the rounded percentage.
A clean lsof result proves no large deleted descriptor remains; journal accounting and du identify whether retention now holds.
Observe at least one representative write cycle or peak period when feasible, and calculate growth rate and time-to-full.
System changes
- No additional changes; records post-remediation capacity, inode, descriptor, journal, directory, and service evidence.
Syntax explained
df -hT / df -i- Rechecks both capacity resources on the original filesystem.
lsof +L1- Confirms rotation no longer leaves large unlinked allocations.
du -x --max-depth=1- Compares visible owner allocation after remediation without crossing mounts.
Values stay on this page and are never sent or saved.
df -hT {{mountpoint}} && df -i {{mountpoint}} && sudo lsof +L1 {{mountpoint}} && sudo journalctl --disk-usage && systemctl is-active {{serviceName}} && sudo du -x -B1 --max-depth=1 {{mountpoint}} | sort -n | tail -10Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg-root ext4 40G 32G 6.7G 83% / Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/vg-root 2621440 612408 2009032 24% / Archived and active journals take up 896.0M in the file system. active 42074112000 /
Checkpoint: Recovery remains stable under normal writes
df -B1 {{mountpoint}} && df -i {{mountpoint}} && systemctl is-active {{serviceName}}Continue whenFree bytes and inodes exceed alert and emergency floors, the service is healthy, and measured growth stays within forecast.
Stop whenAny resource approaches the floor, deleted files recur, or the service emits new storage errors.
If this step fails
Inodes recover but bytes remain full, or vice versa.
Likely causeMultiple simultaneous owners exist and the first remediation addressed only one resource class.
df -B1 {{mountpoint}}df -i {{mountpoint}}lsof +L1 {{mountpoint}}
ResolutionContinue owner attribution for the remaining constrained resource rather than repeating the first cleanup.
Security notes
- Retain the detailed incident bundle in a restricted system and expose only aggregate capacity metrics broadly.
Alternatives
- Declare controlled expansion as the durable fix when valid data growth leaves insufficient forecast headroom.
Stop conditions
- Do not close the incident until alerting and an owner for the actual producer are in place.
Finish line
Verification checklist
df -hT {{mountpoint}} && df -i {{mountpoint}}The correct filesystem has a documented operating margin in both blocks and inodes after the workload resumes, not merely a temporary one-block improvement.sudo lsof +L1 {{mountpoint}}No large unlinked file remains, or each remaining entry has a named owner, purpose, size, and planned release time.sudo logrotate --debug /etc/logrotate.d/{{logrotateName}} && systemctl is-active {{serviceName}} && sudo journalctl --disk-usageThe application policy parses, the service is healthy and writes the current log, and journald remains within its approved budget.Recovery guidance
Common problems and safe checks
df reports a full filesystem but du accounts for far less space.
Likely causeA process holds deleted files, data is hidden below a mount, filesystem-reserved or snapshot space exists, or du crossed a different view than df.
df -B1 {{mountpoint}}du -x -B1 --max-depth=1 {{mountpoint}}lsof +L1 {{mountpoint}}findmnt -R {{mountpoint}}
ResolutionReconcile the exact mount first, release deleted files through their service, inspect hidden underlying directories during a controlled unmount, or use filesystem-specific snapshot/reservation tools.
Applications return No space left on device while df shows free gigabytes.
Likely causeInodes are exhausted, a project/user quota is reached, the thin pool or metadata device is full, or the application writes to another mount.
df -i {{mountpoint}}findmnt -T {{mountpoint}}quota -s 2>/dev/null || truelvs -a -o lv_name,lv_size,data_percent,metadata_percent 2>/dev/null
ResolutionAddress the exhausted resource: expire owner-approved small files, raise a reviewed quota, or expand the correct thin/data/metadata layer. Do not delete large files when inodes are the constraint.
A deleted log still consumes space after rotation.
Likely causeThe process never reopened its log, logrotate used the wrong signal, or the file was manually deleted while still open.
lsof +L1 {{mountpoint}}systemctl status {{serviceName}} --no-pagerlogrotate --debug /etc/logrotate.d/{{logrotateName}}
ResolutionCorrect the owner-specific postrotate signal or perform an approved service restart, then verify the old descriptor disappeared and new writes reach the active path.
Space is reclaimed but usage climbs back within minutes.
Likely causeAn application loop, debug logging, queue backlog, runaway container, failed rotation, or attack continues generating data.
du -x -B1 --max-depth=2 {{mountpoint}} | sort -n | tailjournalctl --since '-10 minutes' --no-pager | tail -200docker system df -v 2>/dev/null || true
ResolutionThrottle or stop the producer, preserve diagnostic samples, correct logging/queue/application behavior, and keep monitoring growth rate until stable.
Reference
Frequently asked questions
Why can df and du disagree by gigabytes?
df reports filesystem allocation; du sees linked directory entries in the walked view. Deleted-open files, hidden mount contents, metadata, reserved blocks, snapshots, and sparse or copy-on-write accounting create legitimate differences.
Can I truncate a deleted file through /proc/PID/fd?
It may reclaim space but can violate application semantics, logging guarantees, or forensic needs. Prefer the service's supported reopen, reload, or restart after confirming the owner and impact.
Is `docker system prune -a --volumes` a safe emergency command?
No. It can remove images needed for rollback, build cache, stopped-container state, and volumes. Measure with `docker system df -v`, identify owners, and prune only reviewed unused objects.
Should I always vacuum journals when root is full?
No. Vacuum only when archived journals are a material confirmed owner and retention policy permits it. Preserve required security and incident history, and fix the producer or persistent budget.
Recovery
Rollback
Deleted journal archives and discarded files cannot be reconstructed by rollback; recover them from the approved log archive or backup if required. Service restarts can be reversed only by restoring the prior service state, while the logrotate policy can be restored from its recorded previous revision.
- If the new logrotate policy is wrong, restore the previous `/etc/logrotate.d/{{logrotateName}}` from configuration management, run `logrotate --debug`, and signal the service only after validating the corrected path and ownership.
- If the service restart causes an application failure, keep the filesystem margin, restore the previous application configuration or release, and start the unit while preserving its journal. Do not recreate the deleted old log as a sparse placeholder.
- If vacuumed journal history is required, restore the archived journal directory from the authorized backup into an isolated analysis path; do not merge arbitrary files into the active machine-id journal directory.
- If cleanup cannot provide safe headroom, stop nonessential writes and expand or migrate the filesystem through its storage owner's documented procedure rather than continuing deletions.
Evidence