OneLinersCommand workbench
Verified diagnostic signature

Kubernetes OOMKilled: separate the termination reason from exit code and current metrics

OOMKilled is a reported container termination reason associated with an out-of-memory kill. It does not identify the exact allocation, prove a leak, or tell you whether container-level limits or wider node pressure were responsible. Preserve which state or lastState recorded it and when. Exit code 137 without OOMKilled is not the same evidence, and a low metric from a later restarted container cannot disprove the earlier memory incident. Match this output from kubernetes, then continue with sourced, read-only checks.

Confidence
high
Platforms
linux · macos · windows
Verified
2026-10-01
Recognized output

Signature matched locally

Kubernetes OOMKilled: separate the termination reason from exit code and current metrics

OneLiners matches this pattern in your browser. Your pasted output is not stored or indexed.

Likely causes

What usually produces this signal

  1. The affected container's actual admitted memory budget is unsuitable for its demonstrated workload peak, concurrency or cache behavior. Review the recorded limit separately from the request: requests inform placement, not a guaranteed memory cap. A current controller template or a unit typo is not evidence of the historical effective limit. Increasing every workload budget without capacity and quota review can move the failure elsewhere.
  2. The application retains memory unexpectedly, or its workload uses memory-backed storage alongside normal allocations. Correlate owner-retained process/cgroup measurements and release/load changes with the termination timestamp. A current kubectl top snapshot has neither the historical peak nor the identity of an earlier process. Do not induce memory exhaustion or mount new diagnostic agents to manufacture the missing evidence.
  3. Wider node pressure or a different termination boundary is being confused with this container's limit. Have the node/runtime owner correlate scoped incident evidence; ordinary SIGKILL-like exit values and Pod eviction are not interchangeable with an OOMKilled reason. Keep the assigned node, UID and container together, and distinguish init/sidecar/application usage instead of adding unrelated replica samples.
Safe next checks

Collect evidence before changing the system

01

Retain the exact current or previous OOM termination and timestamp

# Replace approved-context, approved-namespace and pod-name with owner-approved identities.
# Read-only, named Pod only; keep private names/results local and do not broaden RBAC.
kubectl --context "approved-context" --namespace "approved-namespace" get pod "pod-name" --request-timeout=10s -o=jsonpath="{.metadata.uid}{'\n'}{.spec.nodeName}{'\n'}{.status.phase}{'\n'}{range .status.initContainerStatuses[*]}{.name}{'\t'}{.ready}{'\t'}{.restartCount}{'\t'}{.state.waiting.reason}{'\t'}{.state.terminated.reason}{'\t'}{.state.terminated.exitCode}{'\t'}{.lastState.terminated.reason}{'\t'}{.lastState.terminated.exitCode}{'\t'}{.lastState.terminated.finishedAt}{'\n'}{end}{range .status.containerStatuses[*]}{.name}{'\t'}{.ready}{'\t'}{.restartCount}{'\t'}{.state.waiting.reason}{'\t'}{.state.terminated.reason}{'\t'}{.state.terminated.exitCode}{'\t'}{.lastState.terminated.reason}{'\t'}{.lastState.terminated.exitCode}{'\t'}{.lastState.terminated.finishedAt}{'\n'}{end}"

Expected signal: The affected status row reports OOMKilled in state.terminated or lastState.terminated with its exit code and finish time. A current waiting/running row does not erase the prior cause. If only exit 137 is available, keep the termination mechanism unresolved until the owner supplies the associated reason and actor evidence.

Do not infer cgroup versus whole-node memory pressure from this one reason. The projection is read-only and omits potentially private termination messages. Correlate the assigned node and incident interval through existing owner telemetry; do not run stress tools, signal processes, read unrestricted kernel logs or increase privileges to reproduce a kill.

02

Compare admitted per-container memory request and limit with incident-time evidence

# Replace approved-context, approved-namespace and pod-name with owner-approved identities.
# Read-only, named Pod only; keep private names/results local and do not broaden RBAC.
kubectl --context "approved-context" --namespace "approved-namespace" get pod "pod-name" --request-timeout=10s -o=jsonpath="{.metadata.uid}{'\n'}{.spec.nodeName}{'\n'}{range .spec.initContainers[*]}{.name}{'\t'}{.resources.requests.memory}{'\t'}{.resources.limits.memory}{'\n'}{end}{range .spec.containers[*]}{.name}{'\t'}{.resources.requests.memory}{'\t'}{.resources.limits.memory}{'\n'}{end}"

Expected signal: The actual Pod lists request/limit fields for its init and application containers. Match the affected name and retain Kubernetes quantity units rather than interpreting 128Mi as 128 bytes or treating the request as a cap. A blank container limit requires owner review of effective Pod/node/runtime constraints; it does not prove unlimited safe allocation.

The projection does not read environment, arguments or Secret values, and it does not resize the Pod. Current spec values can differ from earlier in-place resize or deployment history. Use appropriately timestamped metrics for the incident actor; a later low sample is not a historical maximum. Any limit change needs workload, node capacity, quota, availability and rollback review.

Reproducible example

Check the input against the result

Input and result

api-0 terminated: OOMKilled exitCode=137
# Constructed Kubernetes-shaped termination; no memory was exhausted.
# Executed bounded owned HTTP/JSON controls, NOT a Kubernetes cluster:
owned_termination: reason=OOMKilled exit=137
same_exit_control: reason=Error exit=137 OOMKilled=false
constructed_resources: request=64Mi limit=128Mi
later_metric=20Mi incident_peak_proven=false
memory_exhaustion_signal_kernel_or_cgroup=not_executed

Actual owned HTTP/JSON exchanges compared two partial termination records with the same exit code and different reasons, then projected distinct request/limit and a later metric. All quantities, times and termination reasons are constructed. No allocation load, signal, native OOM killer, cgroup enforcement, metrics server or cluster recovery was tested. The comparison only establishes what these supplied records can and cannot support.

Evidence-led recovery

Correct only the demonstrated cause

The application/platform owner corrects the demonstrated retention, workload or effective-budget problem in a reviewed change, with capacity/quota checks and rollback. Preserve the causal interval and approved release. Do not remove limits, add swap blindly, increase every container's memory or suppress OOM evidence as a universal repair.

Verify: Run the intended normal workload through the former peak interval using approved existing telemetry; the correct container remains ready without new OOM terminations/restarts, and node/neighboring workload pressure remains acceptable. A higher configured limit or a single idle low-memory sample is not sufficient verification.

Stop if: Only an exit code or later metric is available, the original UID/node/effective budget is unknown, or remediation requires disruptive resource changes, unbounded load tests, elevated kernel access or unauthorized edits to workload/cluster limits.

Related signals

Other output patterns from this error family

Kubernetes CrashLoopBackOff: diagnose the last termination before changing retriesCrashLoopBackOff describes a container waiting between repeated restart attempts. It is not a Pod phase, a diagnosis of application code, or proof of exhausted memory. Establish the exact Pod UID and init/application container, then inspect the most recent termination and the relevant policy. A Pod can have phase Running while this container is waiting and not ready. Do not treat a displayed retry count or age as the precise next restart time.Kubernetes Back-off restarting failed container: correlate the event with current stateThis BackOff event reports a delayed container restart, not the error that originally stopped execution. Its repeated occurrence count is not a count of newly created containers, and an old event can remain visible after the current instance starts. Bind the event's involvedObject UID to the actual Pod and affected container before interpreting a restart rate or root cause. BackOff for restarting a container is a different operation from an image-pull retry.Kubernetes failed to start container: locate the runtime failure before expecting app logsA Kubernetes-qualified start event reports that the runtime could not start the container. Preserve the concrete inner reason and stage before assuming an application crash, registry password problem or liveness timeout. The process may not have started, so its application log may be absent. The reviewed kubelet source records a runtime StartContainer failure separately from successful start and subsequent lifecycle-hook handling; not every similarly worded error is a Kubernetes event.Kubernetes liveness probe failed: separate health-check failure from readiness and restartThe kubelet's liveness check returned a failure for a container. One failed event is not proof that a restart already happened, that a browser cannot reach the Service, or that the application needs more memory. Correlate the affected container's admitted probe, failure tolerance and recent termination/status. Liveness, readiness and startup checks have different roles; a readiness failure must not be treated as a container restart instruction.