Fix Too many open files on Linux: process limits, leaks and systemd
“Too many open files” can mean a process hit its descriptor limit, the host exhausted its file table, or a library reported another resource problem. Identify the failing PID and errno, compare live limits with descriptor use, and repair the correct launch boundary without hiding a leak behind unlimited settings.
Restore the affected Linux workload by distinguishing EMFILE from ENFILE, identifying descriptor growth or a legitimate capacity requirement, and proving a reviewed application or service-limit change under representative load.
- Ubuntu Server 24.04 LTS
- Linux and systemd Supported distribution kernel and packaged systemd
- Ownership and impact Identify the affected service, its owner, recovery access and whether draining or restarting an instance would drop requests.
- Process visibility Use a session allowed to inspect the target PID and namespace. Permission-denied output is missing evidence, not a zero descriptor count.
- Preservation Retain the exact error, timestamps, deployment revision and existing unit/limit configuration in an access-controlled incident record.
- Tools Use the distribution's procfs, ps, find, systemctl and journalctl. Do not install an arbitrary troubleshooting bundle on a stressed production host.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A process-specific explanation of the error, with an evidence trail to the launch configuration.
- A finite, reviewable repair and an application-level recovery check.
- Distinguish EMFILE from ENFILE and avoid changing unrelated budgets.
- Detect descriptor growth and preserve evidence before mitigation.
- Apply and verify limits where the workload actually starts.
Architecture
How the parts fit together
Follow the failure from the application to its process, launch manager and host before changing policy.
- Retain the original errno and affected PID.
- Read live limits and bounded descriptor samples.
- Choose an application repair or justified finite policy change.
- Verify the replacement process and original operation.
Assumptions
- Examples are illustrative; this Guide does not claim that production exhaustion was reproduced.
- Commands target Linux procfs and systemd. Windows and macOS use different inspection and launch mechanisms.
- A service restart is performed only with the service owner's approved impact and recovery procedure.
Key concepts
- EMFILE
- The operation exhausted a process-related descriptor boundary; inspect the actual process limit and lifecycle.
- ENFILE
- A system-wide open-file object limit was reached; correlate with host evidence.
- Soft and hard limits
- The soft value is enforced; the hard value constrains an unprivileged increase.
- Leak versus legitimate load
- A leak retains resources beyond their intended lifecycle; legitimate concurrency should be bounded and return toward a measured baseline.
Fill these once. Every matching command and configuration block updates immediately; values stay in this page only.
Security and production boundaries
- Process metadata and descriptor targets may contain sensitive paths or tenant information; keep samples private.
- Do not disable service isolation or use a privileged account as a limit workaround.
- Avoid unlimited settings and host-wide changes without a measured capacity and memory-impact review.
Stop before continuing if
- Stop when the PID, namespace or process lifetime cannot be established.
- Stop before an unapproved restart of a stateful or single-instance service.
- Stop when increasing the limit would expose an incompatible select implementation or worsen host pressure.
command
Identify the failing operation and service process
Capture the exact Too many open files error and its numeric or symbolic errno from the application. EMFILE normally concerns a process descriptor boundary; ENFILE concerns a system-wide file-table boundary. A paraphrased dashboard alert is not enough to choose which limit to change.
Why this step matters
The manager's healthy state and the application's ability to accept work are different signals.
What to understand
Record the process start time and worker role, not only a PID that can be reused. A multi-process service may exhaust one child while its main supervisor has few descriptors; map the PID in the error to the unit's actual process tree.
A file watcher may report inotify instance/watch exhaustion with a different errno or wording. Disk space, memory mappings and open descriptors are separate resource budgets. Avoid changing fs.inotify.max_user_watches simply because a log mentions files.
System changes
- Reads service state and retained logs; no limits or processes are changed.
Syntax explained
systemctl show -p- Reads selected manager properties without restarting the service.
--since and tail- Bounds the log interval and displayed evidence; select the real incident window.
Values stay on this page and are never sent or saved.
date --iso-8601=seconds; systemctl show '{{serviceUnit}}' -p MainPID -p ControlGroup -p ActiveState -p SubState; journalctl -u '{{serviceUnit}}' --since '-15 minutes' --no-pager | tail -n 80Illustrative incident, not a live service test: MainPID=4242 ActiveState=active app[4242]: accept: Too many open files (EMFILE) A worker can fail while the unit still reports active.
Checkpoint: Correct failure boundary identified
Continue whenThe original error, failing PID, start time, workload and errno are known.
Stop whenThe log belongs to another process, tenant, container or time interval, or access is insufficient to inspect it.
command
Read the target process limits, not your shell's defaults
Set processId to the current affected worker. Inspect its live limits and identity. The shell running this command is not necessarily the parent that launched the application, so ulimit in an SSH session cannot establish the service's effective configuration.
Why this step matters
Only the live process proves what was inherited and what the application may have adjusted after startup.
What to understand
RLIMIT_NOFILE limits descriptor numbers, not bytes of file content. The hard value constrains how far an unprivileged process can raise its soft value; it is not automatic extra capacity when the soft boundary is reached.
If the process disappears or its start time changes, stop comparing it with earlier samples and identify the replacement. For containers, inspect the PID in the correct namespace and the runtime's configured ulimits. Do not infer container limits solely from the host's login shell.
System changes
- Reads process and unit metadata; it does not modify the running process.
Syntax explained
/proc/PID/limits- Reports the current soft and hard resource limits of that process.
FragmentPath and DropInPaths- Identify the systemd configuration sources that must be reviewed before an edit.
Values stay on this page and are never sent or saved.
ps -p '{{processId}}' -o pid=,ppid=,lstart=,user=,comm=; sudo cat '/proc/{{processId}}/limits'; systemctl show '{{serviceUnit}}' -p LimitNOFILE -p LimitNOFILESoft -p FragmentPath -p DropInPathsIllustrative comparison: Max open files 1024 65536 files LimitNOFILE=65536 LimitNOFILESoft=1024 The process enforces the soft limit, not the larger hard ceiling.
command
Measure descriptor count, highest number and short-term growth
Take two small samples of the same verified PID. Count descriptor directory entries and inspect the highest descriptor number. These are snapshots of a changing process, so small differences between commands are expected and do not by themselves prove a leak.
Why this step matters
A trend tied to workload is more useful than a single number taken after an automatic restart.
What to understand
A low count does not guarantee that the next allocation is safe: descriptors can be sparse, and individual operations can need several new descriptors. Inspect the limit and highest number together, then retain the exact failing syscall from application evidence.
Run at most a small agreed sample rate. Walking a huge descriptor table repeatedly can itself add overhead. If find prints access errors or the PID is gone, discard the count rather than interpreting an empty pipeline as zero usage.
System changes
- Performs bounded directory reads and a five-second interval; it does not open the target's files or close its descriptors.
Syntax explained
-mindepth 1 -maxdepth 1- Lists only the descriptor entries, without following target files or recursively inspecting data.
sort -n | tail -n 1- Reports the highest numeric descriptor entry as a separate observation.
Values stay on this page and are never sent or saved.
for sample in 1 2; do date --iso-8601=seconds; ps -p '{{processId}}' -o pid=,lstart=,comm=; sudo find '/proc/{{processId}}/fd' -mindepth 1 -maxdepth 1 -printf '%f\n' | wc -l; sudo find '/proc/{{processId}}/fd' -mindepth 1 -maxdepth 1 -printf '%f\n' | sort -n | tail -n 1; if [ "$sample" = 1 ]; then sleep 5; fi; doneIllustrative samples: 05:00:00 count=1009 highest_fd=1023 05:00:05 count=1015 highest_fd=1023 A soft limit of 1024 is relevant, but sustained load and descriptor ownership still need investigation.
Checkpoint: Pressure is supported by evidence
Continue whenSamples belong to one process lifetime and relate to its effective limits and workload.
Stop whenSampling causes material overhead, produces permission errors or crosses a process restart.
decision
Separate legitimate concurrency from an application leak
Use the application's pool metrics and a small, access-controlled descriptor sample to classify sockets, ordinary files, pipes and event objects. Compare growth with incoming requests, active jobs, timeouts and the latest deployment. A busy server and a leaking idle server need different repairs.
Why this step matters
Extra capacity can buy recovery time but cannot make an unbounded lifecycle safe.
What to understand
Descriptor targets can reveal internal paths, peer details or tenant identifiers. Keep only the minimum required private sample; do not paste a complete lsof dump into a public issue. Prefer existing application telemetry before attaching an intrusive tracer.
Look for missing close calls on error and cancellation paths, retries opening a new connection without releasing the previous one, unbounded connection pools and watchers that survive project reloads. Reproduce the suspected lifecycle in a disposable environment with the same code revision.
A controlled drain or restart can be an emergency mitigation when the owner accepts the impact. Preserve evidence first and label the action as temporary. Do not repeatedly restart a database or critical service merely to make the descriptor graph fall.
System changes
- This classification step makes no configuration change; any drain, restart or tracing requires a separate reviewed action.
Decision example: Workload returns to baseline after the queue drains; socket count remains high and keeps growing. Next owner action: review connection cleanup, cancellation and pool bounds, not repeated global limit increases.
command
Check system-wide pressure only when the error points there
Inspect the kernel file-table counters and relevant retained messages, especially for ENFILE or simultaneous failures in unrelated services. Do not assume that raising a per-service limit fixes host-wide exhaustion, and do not sum per-process counts as if shared file objects were independent.
Why this step matters
Process descriptor limits and the kernel's open-file object accounting are related but not interchangeable.
What to understand
The file-nr fields describe allocated, unused and maximum file handles; modern Linux commonly reports zero for the unused field. Interpret them using the running kernel documentation. nr_open is a ceiling on the per-process resource-limit setting, not a target that every daemon should receive.
If many unrelated services fail, treat it as a host incident. Investigate memory pressure, a runaway workload and the recent change before a global sysctl increase. A larger allowance can increase the host's potential memory consumption and worsen contention.
System changes
- Reads host counters and logs without changing kernel settings.
Syntax explained
file-nr- Reads the kernel's file-handle accounting, not a process descriptor count.
sysctl without assignment- Reads the current settings; no -w or persistent file is used.
cat /proc/sys/fs/file-nr; sysctl fs.file-max fs.nr_open; journalctl -k --since '-15 minutes' --no-pager | tail -n 80Illustrative host observation: file-nr: 18432 0 9223372036854775807 No file-max exhaustion message in the retained interval. This does not replace the affected process's EMFILE evidence.
config
Apply an owner-approved limit at the actual launch boundary
Use this systemd drop-in only when evidence supports a finite increase and the application is known to support the resulting descriptor range. Back up the exact existing drop-in first. Set both values deliberately; the examples are placeholders for a measured budget, not recommended limits for every service.
Why this step matters
Persistence must be configured where the workload is actually launched, not in an unrelated operator shell.
What to understand
Before raising the soft limit above 1024, confirm that the application and libraries do not rely on the traditional select descriptor-set boundary. A higher limit can expose a latent select bug rather than improve capacity; fix or upgrade that code path first.
For interactive sessions, PAM limits apply only when the relevant login stack uses pam_limits and a new session is established. Existing sessions and system services do not automatically inherit a limits.conf edit. For containers, use the orchestrator/runtime's documented ulimit configuration and verify the actual worker afterward.
Keep the change per-service and finite. Include normal concurrency, burst requirements and operational headroom, while retaining alerts well before exhaustion. Do not use a global DefaultLimitNOFILE or fs.file-max change to conceal one leaking process.
System changes
- Creates or edits one approved systemd drop-in. It does not change a running process until the service is relaunched.
Syntax explained
LimitNOFILE=soft:hard- Sets the soft boundary and its hard ceiling for processes launched by this unit.
Dedicated drop-in file- Keeps the limit change separate from unrelated security hardening and makes rollback precise.
/etc/systemd/system/{{serviceUnit}}.d/70-file-limits.confValues stay on this page and are never sent or saved.
[Service]
LimitNOFILE={{softFdLimit}}:{{hardFdLimit}}Illustrative reviewed policy: [Service] LimitNOFILE=8192:65536 The running PID is unchanged until the approved activation step.
Checkpoint: Change is justified and reversible
Continue whenMeasured capacity, application compatibility, owner approval and the exact previous file state are recorded.
Stop whenThe application uses an incompatible select path, has an unexplained leak, or cannot be safely drained/restarted.
command
Activate on one drained instance and identify its new PID
Schedule or approve the impact first. Reload the manager's configuration, then restart only the chosen service instance through its established rollout procedure. A daemon-reload alone does not replace a process or change the limits it inherited earlier.
Why this step matters
The manager's intended settings must be checked against the newly launched application, not an old PID.
What to understand
For replicated services, drain and verify one instance before continuing. For a single-instance stateful service, use its maintenance and recovery procedure; the restart command is not permission to interrupt it unexpectedly.
A process may deliberately lower its own limits at startup or launch child workers with different values. Repeat the live-process inspection for the actual failing role. If activation fails, preserve the journal and restore only the reviewed change rather than removing all service drop-ins.
System changes
- Reloads systemd unit metadata and restarts the explicitly selected service.
Syntax explained
daemon-reload- Reloads unit definitions in the manager; it does not restart existing services.
restart one unit- Relaunches that workload and can interrupt traffic; use only after the impact is approved.
Values stay on this page and are never sent or saved.
sudo systemctl daemon-reload && sudo systemctl restart '{{serviceUnit}}' && systemctl show '{{serviceUnit}}' -p MainPID -p ActiveState -p SubState -p LimitNOFILE -p LimitNOFILESoftIllustrative activation: MainPID=5278 ActiveState=active LimitNOFILE=65536 LimitNOFILESoft=8192 Update processId to the affected new worker before repeating procfs checks.
verification
Prove recovery and bounded resource use under representative load
Repeat the original failing application operation after updating the PID. Confirm the live soft/hard values, absence of new EMFILE/ENFILE errors and descriptor behavior over an agreed workload cycle. Successful service startup alone does not demonstrate that the incident is resolved.
Why this step matters
A valid repair explains both the failure and the observed recovery, including what will warn the owner before recurrence.
What to understand
Record the workload, duration and bounds of the check. Two quiet samples cannot establish a leak is fixed. Monitor the process lifetime, active work and descriptor trend across expected bursts, failure paths and cancellation.
If usage grows without returning, continue the application investigation even when the new limit postpones errors. If restoring a previous release stops growth, retain the regression evidence and use the approved rollout process. Do not declare the whole host healthy from one service's result.
System changes
- Reads the replacement process and logs; workload tests are separately authorized and bounded.
Syntax explained
ps and /proc/PID/limits- Checks the exact replacement process, its lifetime and the live limits it currently enforces.
--since '-15 minutes' and tail -n 80- Limits the displayed recent service-log evidence; it does not prove that no earlier or rotated errors occurred.
Values stay on this page and are never sent or saved.
ps -p '{{processId}}' -o pid=,lstart=,comm=; sudo cat '/proc/{{processId}}/limits'; journalctl -u '{{serviceUnit}}' --since '-15 minutes' --no-pager | tail -n 80Acceptance record example: Original operation succeeds on the replacement worker. Live limit matches the approved policy. Descriptor use returns to the observed baseline after work completes. No new EMFILE/ENFILE during the agreed representative test.
Checkpoint: Owner accepts the repair
Continue whenThe original operation works, the policy matches the process, usage remains bounded and monitoring has an owner.
Stop whenErrors recur, the PID differs from the tested instance, descriptor growth remains unexplained or other services degrade.
Finish line
Verification checklist
sudo cat '/proc/{{processId}}/limits'The replacement worker has the approved finite soft/hard values, not just a matching unit file.journalctl -u '{{serviceUnit}}' --since '-15 minutes' --no-pager | tail -n 80The original operation succeeds and no new relevant errors appear during the explicitly recorded workload test.Recovery guidance
Common problems and safe checks
Too many open files continues after changing ulimit in SSH
Likely causeThe service was launched by another parent or manager.
Read the failing PID's /proc/PID/limits.Inspect the unit or runtime configuration and process start time.
ResolutionApply the reviewed policy at the real launch boundary and verify a newly launched worker.
The service is active but requests still fail
Likely causeA child worker, rather than MainPID, reached the limit or leaked resources.
Match the PID in the error to the process tree.Inspect worker-specific limits and descriptor trends.
ResolutionRepair or roll out the affected worker path; do not use the unit's active flag as the success criterion.
Errors return later after every limit increase
Likely causeGrowth is unbounded or capacity planning ignores a recurring workload.
Correlate descriptor trends with jobs, connections, cancellation and deployment revisions.Compare post-drain use with a measured baseline.
ResolutionFix the lifecycle or bound the pool/workload; label any restart or temporary limit increase as mitigation.
Reference
Frequently asked questions
Does Too many open files mean the disk is full?
No. It refers to a descriptor or file-table resource boundary. Inspect the exact errno and process; disk capacity is a separate check.
Why is ulimit -n different from the application's limit?
It describes the current shell. A systemd service, container or earlier session can inherit different limits and may adjust them after startup.
Can I set LimitNOFILE to unlimited?
Do not use unlimited as a generic fix. Establish legitimate demand, application compatibility and host impact, then choose a finite budget and alerts.
Is a low descriptor count proof there is no problem?
No. Descriptor numbers can be sparse, snapshots race with activity, and allocation can require several descriptors. Compare the highest number, soft limit and actual failing operation.
Should I restart the process first?
Only as an approved mitigation after preserving evidence and considering traffic or data impact. A restart can hide a leak temporarily without fixing it.
Recovery
Rollback
Restore only the reviewed limit change or application revision; a restart may still be required and descriptor leaks need their own code fix.
- Retain the failing and replacement process evidence before rollback.
- Restore the previous contents of 70-file-limits.conf, or remove only that newly created file through the approved configuration workflow. Do not use systemctl revert to erase unrelated hardening.
- Reload systemd and perform the same controlled single-instance restart or rollout.
- Identify the new PID, verify its limits and repeat the original operation. If the original limit was insufficient, use a separately approved mitigation rather than silently reopening the incident.
Evidence