Containers & Kubernetes / Incident Response & Troubleshooting

Troubleshoot Kubernetes CrashLoopBackOff, ImagePullBackOff, Pending, and OOMKilled Pods

Diagnose common Kubernetes Pod failures from owner and scheduler through image pull, process exit, probes, memory, and Service traffic, then apply a controlled declarative fix.

55 min12 stepsChanges system stateRevision 2

Save or explore

#kubernetes #kubectl #crashloopbackoff #imagepullbackoff #pending #oomkilled #pods

0 of 12 steps completed

Goal

Identify the exact failing lifecycle boundary, preserve evidence, remediate the owning workload without exposing secrets, and prove stable service recovery.

Supported environments

Kubernetes 1.35, 1.36
kubectl 1.35, 1.36

Prerequisites

Verified context Prove the active cluster, user, and namespace before collecting evidence.kubectl config current-context && kubectl auth whoami
Read access Obtain namespace workload, Pod, event, Service, EndpointSlice, PVC, node summary, and metrics read permissions without granting Secret data access.kubectl auth can-i get pods -n {{namespace}}
Incident identity Record the affected service, expected release, incident start, last known-good revision, and error-budget impact.
Declarative owner Identify GitOps, CI/CD, Helm, or another source that owns the Pod template before any remediation.

Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System

A repeatable evidence-first triage path that classifies a failing Pod by lifecycle stage before an operator changes the workload.
A diagnostic record joining Pod UID, owner, rollout revision, container state, previous termination, scheduler and kubelet events, current and previous logs, resources, probes, image identity, storage, and node health.
A controlled remediation and rollback path for CrashLoopBackOff, ImagePullBackOff, Pending, OOMKilled, and readiness failures without treating Pod deletion as a repair.

Observable outcome

The incident has a supported root-cause hypothesis backed by Kubernetes status, events, logs, resource evidence, and controller history rather than by a single STATUS column.
Any change is applied to the owning declarative workload, observed through rollout checkpoints, and reversible to the last known-good revision.
Service endpoints, readiness, restart stability, resource headroom, and external request behavior all pass after remediation.

Architecture

How the parts fit together

Kubernetes surfaces one failure through several control loops. The scheduler decides placement, the kubelet and container runtime pull and start images, probes influence readiness and restarts, controllers replace Pods from templates, Services publish only ready endpoints, and resource controls govern both scheduling and runtime memory. Diagnosis therefore follows the Pod from declaration to owner, scheduling, image acquisition, initialization, process execution, probes, and traffic instead of guessing from the user-facing reason string.

Workload controllerDeployment, StatefulSet, DaemonSet, Job, or another owner that holds the durable Pod template and rollout history. A recreated Pod inherits its decisions.

SchedulerSelects an eligible node using requests, taints, tolerations, affinity, topology, volumes, and policy. Pending Pods frequently stop at this boundary.

Kubelet and container runtimePull images, mount data, start containers, run probes, report termination details, and enforce resource limits on the selected node.

Application containerProvides current and previous logs, exit codes, listeners, and process behavior. Its failure is evidence, not always the root cause.

Service and EndpointSliceRoute traffic only to selected, ready backends. They prove whether recovery reached the data plane rather than merely reaching Running.

Observability systemsRetain historical logs and metrics after short-lived Pods disappear, allowing comparison before and after the failing rollout.

Capture namespace, Pod UID, node, owner references, images, states, restart counts, and recent events before the controller replaces evidence.
Map the Pod to its owning controller and compare its template and revision with the last known-good rollout.
Classify the failure boundary as scheduling, image retrieval, initialization, process exit, OOM termination, probe failure, or traffic publication.
Use the narrow evidence path for that boundary while preserving Secrets and production data.
Change the owning declaration, watch the rollout, and verify both control-plane status and real service traffic.

Assumptions

The operator has read access to Pods, events, workloads, Services, EndpointSlices, nodes, and relevant metrics, plus separately approved write access only if a remediation is authorized.
The cluster is Kubernetes 1.35 or 1.36 and `kubectl` is close enough to the API server version to expose current fields and debugging behavior.
Application logs are available from the API or a central log platform, and the incident timeline uses synchronized clocks.
The owning manifest is managed in Git, an approved deployment system, or another declarative source that will not immediately overwrite an emergency patch.
Ephemeral debug images are allowlisted, contain no embedded credentials, and may be used only under the cluster's admission and audit policy.
Stateful workloads have an application-specific backup and rollback procedure; this guide does not infer data safety from Pod health.

Key concepts

Pod phase versus container reason: Pod phase is a coarse lifecycle summary. CrashLoopBackOff and ImagePullBackOff are waiting reasons reported for an individual container; OOMKilled is usually a previous termination reason. Inspect every container status, including init containers.
Back-off: Kubernetes delays repeated restart or image-pull attempts after failures. Deleting a Pod resets some local history but does not correct the template, dependency, credential, or image.
Previous container: After a restart, `kubectl logs --previous` reads the immediately preceding container instance in the same Pod. It is often the shortest path to the crash message, but it disappears when the Pod itself is replaced.
Resource request: A scheduling reservation used to place the Pod. Requests that cannot fit leave a Pod Pending; requests set too low can overcommit a node and make runtime behavior unstable.
Resource limit: A runtime ceiling. Exceeding a memory limit can terminate a container with OOMKilled and exit code 137; increasing it without understanding consumption can shift the failure to node pressure.
Owner reference: Metadata that identifies the controller responsible for recreating the Pod. Durable remediation belongs in that controller's Pod template, not in an individual disposable Pod.
Readiness: A condition controlling whether a Pod is eligible for Service traffic. Running is not equivalent to Ready, and a process can survive while failing every request.
Ephemeral debug container: A temporary diagnostic container added to a running Pod when its application image lacks tools. It is powerful evidence tooling, not a mechanism for silently changing production state.

Before you copy

Values used in this guide

{{namespace}}

Namespace containing the failing Pod and its owner. Namespace boundaries also scope Secrets, Services, and image pull credentials.

Example: payments

{{pod}}

Exact current Pod name captured together with its UID so replacement is not confused with recovery.

Example: api-7c8d9f54d8-k9q2m

{{container}}

Failing application or init-container name selected from the Pod specification.

Example: api

{{workloadKind}}

Lowercase kubectl resource kind of the durable owner.

Example: deployment

{{workloadName}}

Name of the controller that owns the Pod template.

Example: api

{{labelSelector}}

Narrow label selector that identifies only the workload's Pods.

Example: app.kubernetes.io/name=payments-api

{{service}}

Service expected to publish ready endpoints for the recovered Pods.

Example: payments-api

{{debugImage}}

Approved diagnostic image pinned by digest and compatible with the Pod architecture.

Example: registry.k8s.io/e2e-test-images/busybox:1.29-4

Security and production boundaries

Pod descriptions, environment sources, logs, command lines, and debug sessions can expose customer identifiers, tokens, connection strings, internal hostnames, and Secret names. Store evidence in the incident system with the same access controls as production logs.
Never run `kubectl get secret -o yaml`, echo decoded registry credentials, or copy a production Secret into another namespace for convenience. Verify metadata and repair the approved secret-management source.
Do not use privileged debug containers, hostPID, hostNetwork, hostPath, or broad capabilities unless a separate node-level incident procedure explicitly authorizes them.
An emergency `kubectl edit` or `set image` can drift from GitOps and be reverted automatically. Know the source of truth and record exactly which controller will reconcile the object.
Repeated Pod deletion destroys short-lived evidence and can amplify load on registries, dependencies, queues, and databases. Preserve evidence and rate-limit remediation.

Stop before continuing if

Stop if the kubeconfig context, cluster, namespace, Pod UID, or owner cannot be proven; a correct command against the wrong cluster is an incident.
Stop before viewing logs or starting a debug container if the workload handles regulated or highly sensitive data and the incident evidence policy is not established.
Stop if the failure affects many unrelated workloads or control-plane components; escalate to a cluster or node incident instead of applying application-specific changes.
Stop before changing requests, limits, probes, image credentials, scheduling constraints, or storage when the owning team and source of truth are unknown.
Stop after a new revision worsens availability, error rate, data integrity, or dependency load. Roll back the controller and preserve the failed revision.

verification

Scope the failure and freeze the Pod identity

read-only

List the selected Pods with node, IP, readiness, status, restart count, age, and UID, then capture the exact failing Pod before a controller replaces it.

Why this step matters

Status labels are lossy and Pod names are reusable. The UID, node, creation time, selected peer Pods, and restart pattern establish which concrete object failed and whether the incident is isolated, rollout-wide, or node-wide before the controller replaces evidence.

What to understand

Use the workload's narrow selector rather than listing the entire namespace. Compare replica count, age, node distribution, and restart acceleration; one old failing Pod among healthy new replicas tells a different story from every replica failing simultaneously.

Write the observed UID and incident timestamp into the ticket. If the Pod name later points to a different UID, treat it as a new object and correlate the old evidence through rollout history and centralized logs.

STATUS may display a container waiting reason while Pod phase remains Running. The next step extracts individual states instead of interpreting this table as the root cause.

System changes

No cluster state changes; the command reads selected Pod metadata and status through the API.

Syntax explained

-l '{{labelSelector}}': Restricts the result to the workload's reviewed label set so unrelated Pods do not distort the incident scope.
-o custom-columns=...: Projects stable identity and lifecycle fields into an auditable table without exposing environment or Secret data.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

Namespace{{namespace}}Namespace containing the failing Pod and its owner. Namespace boundaries also scope Secrets, Services, and image pull credentials.Label Selector{{labelSelector}}Narrow label selector that identifies only the workload's Pods.

kubectl -n {{namespace}} get pods -l '{{labelSelector}}' -o custom-columns=NAME:.metadata.name,UID:.metadata.uid,READY:.status.containerStatuses[*].ready,STATUS:.status.phase,NODE:.spec.nodeName,POD_IP:.status.podIP,RESTARTS:.status.containerStatuses[*].restartCount,AGE:.metadata.creationTimestamp

Example output / evidence

NAME                       UID                                    READY   STATUS    NODE       POD_IP      RESTARTS   AGE
api-7c8d9f54d8-k9q2m       8707fb30-18ef-4a78-aac2-2f6016411d74   false   Running   worker-2   10.42.2.17  6          2026-07-28T08:42:16Z

Checkpoint: Affected object and blast radius are unambiguous

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.metadata.uid}{" "}{.spec.nodeName}{" "}{.metadata.creationTimestamp}{"\n"}'

Continue whenReturns the exact UID, assigned node or an empty node for Pending, and creation timestamp recorded in the incident.

Stop whenThe Pod is absent, its UID changed during collection, the selector includes another service, or failures span unrelated workloads or control-plane namespaces.

If this step fails

The Pod alternates between Running and CrashLoopBackOff, and the restart count keeps increasing.

Likely causeThe application process exits after startup, a required configuration or Secret is absent, a startup dependency is unavailable, or a liveness probe kills a process that would otherwise recover.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o wide
kubectl -n {{namespace}} logs {{pod}} -c {{container}} --previous --timestamps
kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.status.containerStatuses[?(@.name=="{{container}}")]}'

ResolutionCorrelate the previous termination reason, exit code, application log, probe events, and rollout revision. Correct the owning workload declaratively; do not repeatedly delete the Pod because its controller will recreate the same failure.

Security notes

Metadata can still disclose internal service and node names. Keep the captured table inside the incident workspace.

Alternatives

When the Pod has already vanished, query the controller's ReplicaSets plus central logs and metrics using the last known UID and revision.

Stop conditions

Do not continue until cluster context, namespace, selector, Pod name, and current UID have been independently verified.

verification

Read every init and application container state

read-only

Extract current and previous states, reasons, exit codes, restart counts, images, and image IDs so one sidecar does not hide the failing container.

Why this step matters

A multi-container Pod can show one summary reason while an init container, sidecar, or application has the actual failure. Current and previous state together separate waiting from termination and reveal whether Kubernetes ever started the process.

What to understand

Read initContainerStatuses and containerStatuses separately. An application container cannot start until every required init container completes, while a failing sidecar can keep the Pod unready even when the primary process is healthy.

The previous termination reason and exit code survive a container restart inside the same Pod. Exit 1 points toward application failure, 137 with OOMKilled toward memory, and an absent imageID toward a failure before process execution.

Compare the declared image with imageID. A mutable tag can resolve to a different digest on different nodes; the digest is the runtime artifact that actually executed.

System changes

No cluster state changes; jsonpath reads status and image identity from the Pod object.

Syntax explained

.status.initContainerStatuses[*]: Includes initialization gates that can keep all application containers waiting.
.lastState.terminated: Reads the immediately preceding container instance's reason and exit code after kubelet restarts it.
.imageID: Shows the resolved runtime image digest rather than only the mutable reference in the template.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{range .status.initContainerStatuses[*]}init/{.name}{" current="}{.state.waiting.reason}{.state.terminated.reason}{" last="}{.lastState.terminated.reason}{" exit="}{.lastState.terminated.exitCode}{" restarts="}{.restartCount}{"\n"}{end}{range .status.containerStatuses[*]}app/{.name}{" current="}{.state.waiting.reason}{.state.terminated.reason}{" last="}{.lastState.terminated.reason}{" exit="}{.lastState.terminated.exitCode}{" restarts="}{.restartCount}{" image="}{.image}{" imageID="}{.imageID}{"\n"}{end}'

Example output / evidence

app/api current=CrashLoopBackOff last=Error exit=1 restarts=6 image=registry.example/payments/api:2026.07.28 imageID=registry.example/payments/api@sha256:7db9f4b6
app/metrics current= last= exit= restarts=0 image=registry.example/observability/exporter:2.4 imageID=registry.example/observability/exporter@sha256:01f2c9aa

Checkpoint: The failing container and lifecycle boundary are classified

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{range .status.containerStatuses[*]}{.name}{"|"}{.state.waiting.reason}{"|"}{.lastState.terminated.reason}{"|"}{.lastState.terminated.exitCode}{"\n"}{end}'

Continue whenEach container has a separate row and the incident record identifies one primary waiting or termination reason.

Stop whenContainer status is absent, the Pod UID changed, or several containers fail for unrelated reasons that require separate investigation branches.

If this step fails

The previous log command returns `previous terminated container not found` even though the Pod is unhealthy.

Likely causeThe selected container has not restarted, the wrong container name was used, or the Pod was replaced and its earlier container history belongs to a different Pod UID.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.metadata.uid}{"\n"}{.spec.containers[*].name}{"\n"}'
kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{"\n"}{end}'

ResolutionSelect the container whose restart count is nonzero. If the workload replaced the Pod, inspect the rollout events and centralized logs keyed by the previous Pod UID instead of assuming the current Pod contains old history.

Security notes

Image paths may reveal private registry structure and release names; redact them in external reports while preserving the digest internally.

Alternatives

Use `kubectl get pod -o yaml` in a protected evidence file when jsonpath omits a newly introduced status field.

Stop conditions

Do not infer CrashLoopBackOff, OOMKilled, or image failure from the table alone; require a matching container state and supporting event or log evidence.

verification

Map the Pod to its durable owner and rollout revision

read-only

Follow owner references through ReplicaSet to the workload, then compare the current template, rollout history, and change annotations with the last known-good revision.

Why this step matters

Pods are disposable execution records. The ReplicaSet, Deployment, StatefulSet, DaemonSet, or Job is the durable declaration that recreates them, so diagnosis and repair must be tied to its revision and reconciliation owner.

What to understand

A Deployment usually owns a ReplicaSet which owns the Pod. The immediate owner shown on the Pod may therefore need one more lookup before the durable workload and rollout history are known.

Compare change cause, image digest, ConfigMap or Secret references, resource settings, probes, and scheduling fields between the failing and last known-good revisions. A timestamp correlation is a clue, not proof.

Determine whether Helm, Argo CD, Flux, another operator, or a custom controller will revert direct API edits. The incident plan must use or deliberately pause the actual source of truth.

System changes

No cluster state changes; owner references and rollout history are read from workload metadata.

Syntax explained

.metadata.ownerReferences[0]: Identifies the immediate controller responsible for this Pod; follow a ReplicaSet owner to its Deployment where applicable.
rollout history: Lists controller revisions and recorded change causes without altering rollout state.

Command

Fill variables0/4 ready

Values stay on this page and are never sent or saved.

Namespace{{namespace}}Namespace containing the failing Pod and its owner. Namespace boundaries also scope Secrets, Services, and image pull credentials.Pod{{pod}}Exact current Pod name captured together with its UID so replacement is not confused with recovery.Workload Kind{{workloadKind}}Lowercase kubectl resource kind of the durable owner.Workload Name{{workloadName}}Name of the controller that owns the Pod template.

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.metadata.ownerReferences[0].kind}{"/"}{.metadata.ownerReferences[0].name}{" revision="}{.metadata.annotations.deployment\.kubernetes\.io/revision}{"\n"}' && kubectl -n {{namespace}} rollout history {{workloadKind}}/{{workloadName}}

Example output / evidence

ReplicaSet/api-7c8d9f54d8 revision=
deployment.apps/api
REVISION  CHANGE-CAUSE
41        release 2026.07.21
42        release 2026.07.28

Checkpoint: Durable owner, current revision, and last known-good revision are recorded

kubectl -n {{namespace}} get {{workloadKind}} {{workloadName}} -o jsonpath='{.metadata.uid}{" generation="}{.metadata.generation}{" observed="}{.status.observedGeneration}{"\n"}'

Continue whenReturns the owner UID and matching generation values or documents an in-progress reconciliation.

Stop whenNo owner is found, a custom operator owns the resource, generation is not observed, or the deployment source cannot be identified.

If this step fails

The rollout creates healthy new Pods but the old unhealthy revision remains or rollout status times out.

Likely causeA PodDisruptionBudget, unavailable capacity, maxUnavailable/maxSurge policy, failing readiness check, or finalizer prevents the controller from completing replacement.

Safe checks

kubectl -n {{namespace}} rollout status {{workloadKind}}/{{workloadName}} --timeout=60s
kubectl -n {{namespace}} get pods -l '{{labelSelector}}' -o wide
kubectl -n {{namespace}} get poddisruptionbudget

ResolutionIdentify the specific rollout constraint and restore capacity or application health. If the new revision is unsafe, use the controller's reviewed rollback mechanism; do not force-delete Pods or bypass a disruption budget without proving application redundancy.

Security notes

Do not copy full workload YAML into a public ticket because environment references, annotations, and registry paths may be sensitive.

Alternatives

For an unmanaged standalone Pod, create a reviewed controller manifest rather than repeatedly editing the ephemeral object.

Stop conditions

Do not perform a direct change until the reconciliation owner and rollback revision are known.

verification

Correlate Pod, scheduler, volume, image, probe, and node events

read-only

Describe the Pod and read recent namespace events in time order, distinguishing repeated symptoms from the earliest causal event.

Why this step matters

Events connect scheduler, volume, kubelet, image, and probe decisions to the object timeline. The earliest warning often describes the boundary that failed, while later BackOff messages merely describe retry behavior.

What to understand

Read the full describe output once because it includes conditions, mounts, requests, probe definitions, and the event list. Then use the sorted event query to preserve chronology and repetition counts.

Event retention is finite and aggregation can collapse repeated messages. Capture timestamps, count, reason, reporting component, and Pod UID promptly; supplement with provider or central event storage when available.

Classify the emitting component: default-scheduler suggests placement, kubelet suggests node runtime, attach/detach controllers suggest storage, and FailedToRetrieveImagePullSecret points to namespace credential configuration.

System changes

No cluster state changes; Pod details and recent namespace events are read through the API.

Syntax explained

--sort-by=.lastTimestamp: Orders current event objects by their last observed timestamp to reconstruct the recent sequence.
--field-selector involvedObject.name={{pod}}: Narrows the query to events attached to the named object; verify UID in describe when names may be reused.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

kubectl -n {{namespace}} describe pod {{pod}} && kubectl -n {{namespace}} get events --sort-by=.lastTimestamp --field-selector involvedObject.name={{pod}}

Example output / evidence

Events:
Type     Reason     Age   From               Message
Normal   Scheduled  9m    default-scheduler  Successfully assigned payments/api-7c8d9f54d8-k9q2m to worker-2
Normal   Pulled     9m    kubelet            Container image already present on machine
Warning  BackOff    42s   kubelet            Back-off restarting failed container api in pod api-7c8d9f54d8-k9q2m_payments(8707fb30-18ef-4a78-aac2-2f6016411d74)

Checkpoint: A causal event chain is preserved

kubectl -n {{namespace}} get events --field-selector involvedObject.name={{pod}} -o custom-columns=TIME:.lastTimestamp,TYPE:.type,REASON:.reason,FROM:.reportingComponent,MESSAGE:.message

Continue whenShows a coherent sequence from scheduling through the first failure and subsequent retry or back-off.

Stop whenEvents refer to a different UID, retention has removed the incident window, or the earliest warning points to cluster-wide storage, network, or node failure.

If this step fails

The Pod reports ImagePullBackOff and events show `manifest unknown` or `not found`.

Likely causeThe image repository, tag, architecture-specific manifest, or digest in the Pod template does not exist in the target registry.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.spec.containers[*].image}{"\n"}'
kubectl -n {{namespace}} describe pod {{pod}}
kubectl -n {{namespace}} get {{workloadKind}} {{workloadName}} -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}'

ResolutionCompare the declared image reference with the release artifact produced by CI. Update the owning workload to a verified immutable digest or existing tag, then observe a new rollout; never create an unrelated image under the misspelled tag merely to clear the event.

ImagePullBackOff events contain `unauthorized`, `denied`, or `FailedToRetrieveImagePullSecret`.

Likely causeThe image pull Secret is missing from the Pod namespace, its name is misspelled, the registry credential expired, or the ServiceAccount does not reference the intended Secret.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.spec.imagePullSecrets[*].name}{"\n"}'
kubectl -n {{namespace}} get secret
kubectl -n {{namespace}} get serviceaccount -o custom-columns=NAME:.metadata.name,PULL_SECRETS:.imagePullSecrets[*].name

ResolutionRepair the namespace-local credential through the approved secret-management path and reference the correct Secret from the Pod template or ServiceAccount. Do not print `.dockerconfigjson`, copy credentials between tenants, or paste registry tokens into terminal history.

The Pod remains Pending and the scheduler reports `Insufficient cpu` or `Insufficient memory`.

Likely causeNo eligible node has allocatable capacity for the sum of container requests, init-container requirements, Pod overhead, and already reserved resources.

Safe checks

kubectl -n {{namespace}} describe pod {{pod}}
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory
kubectl top nodes

ResolutionConfirm that requests reflect measured steady-state and startup demand. Restore capacity, scale the node group, or deliberately revise requests after evidence; do not erase requests simply to force placement and create node pressure.

Security notes

Event messages can contain internal image names, volume identifiers, IP addresses, and admission-policy details.

Alternatives

Query the cluster event exporter or cloud audit platform when API event retention no longer covers the incident.

Stop conditions

Escalate rather than patch the workload if events show a shared provisioner, registry, admission controller, node, or control-plane outage.

verification

Capture current and previous logs without leaking secrets

read-only

Read timestamped logs from the selected container and its immediately previous instance, storing only redacted incident evidence.

Why this step matters

Current logs describe the live instance, but CrashLoopBackOff frequently leaves the useful error only in the immediately previous container. Timestamped, bounded collection preserves sequence without dumping an uncontrolled log volume.

What to understand

Specify the container explicitly in multi-container Pods. `--previous` addresses only the prior instance of that same container in the current Pod and returns an error when no previous instance exists.

Correlate application timestamps with the kubelet event and termination time. A dependency timeout after startup is different from an immediate argument or configuration validation failure.

Redact secrets, authentication headers, personal data, and customer payloads before attaching output. Preserve hashes or stable identifiers only when necessary for correlation.

System changes

No cluster state changes; bounded current and previous stdout/stderr are read from the kubelet log endpoint.

Syntax explained

-c {{container}}: Selects the failing container rather than relying on a default that may point to a sidecar.
--previous: Reads the immediately preceding terminated instance in the same Pod, often containing the crash message.
--timestamps --tail=200: Adds correlation timestamps and bounds collection to the newest 200 lines.

Command

Fill variables0/3 ready

Values stay on this page and are never sent or saved.

Namespace{{namespace}}Namespace containing the failing Pod and its owner. Namespace boundaries also scope Secrets, Services, and image pull credentials.Pod{{pod}}Exact current Pod name captured together with its UID so replacement is not confused with recovery.Container{{container}}Failing application or init-container name selected from the Pod specification.

kubectl -n {{namespace}} logs {{pod}} -c {{container}} --timestamps --tail=200 && kubectl -n {{namespace}} logs {{pod}} -c {{container}} --previous --timestamps --tail=200

Example output / evidence

2026-07-28T08:49:27.188Z INFO starting payments-api revision=42
2026-07-28T08:49:27.442Z ERROR configuration validation failed: required key DATABASE_HOST is empty
2026-07-28T08:49:27.443Z INFO exiting code=1

Checkpoint: The process-level failure is tied to a timestamp and exit

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.status.containerStatuses[?(@.name=="{{container}}")].lastState.terminated.finishedAt}{" exit="}{.status.containerStatuses[?(@.name=="{{container}}")].lastState.terminated.exitCode}{"\n"}'

Continue whenReturns a termination timestamp and exit code that align with the captured previous log and event timeline.

Stop whenLogs expose secrets or regulated data, the selected container never restarted, or the Pod has been replaced and the evidence belongs to another UID.

If this step fails

The previous log command returns `previous terminated container not found` even though the Pod is unhealthy.

Likely causeThe selected container has not restarted, the wrong container name was used, or the Pod was replaced and its earlier container history belongs to a different Pod UID.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.metadata.uid}{"\n"}{.spec.containers[*].name}{"\n"}'
kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{"\n"}{end}'

Security notes

Treat application logs as production data. Never paste unreviewed output into public chat, community submissions, or external issue trackers.

Alternatives

Use centralized logs keyed by Pod UID and container when the current Pod no longer retains the previous instance.

Stop conditions

Pause collection and involve the data owner if logs contain credentials, payment data, health data, or another restricted class.

decision

Separate process crashes, probe restarts, and missing dependencies

read-only

Inspect the declared command, arguments, environment references, mounts, probes, and termination message alongside the previous exit code; verify referenced object names without reading Secret values.

Why this step matters

A restart loop can come from the application process or from kubelet probe policy. Reading declarations and referenced object names separates an invalid command, missing configuration, failed dependency, mount problem, and overly aggressive probe without exposing Secret values.

What to understand

Check command and args against the image entrypoint contract. A shell-form assumption, missing executable, wrong working directory, or malformed flag can exit before the application emits useful logs.

Compare startup, readiness, and liveness purposes. A startup probe protects slow initialization; readiness removes traffic without restarting; liveness should detect an unrecoverable process, not a temporary external dependency outage.

Confirm that every ConfigMap, Secret, volume, Service, and DNS name referenced by the Pod exists in the correct namespace. Existence does not prove content correctness, but absence is actionable evidence.

System changes

No cluster state changes; the command reads Pod declaration and names of namespace-scoped configuration objects.

Syntax explained

.spec.containers[?(@.name=="{{container}}")]: Selects only the failing container's command, environment references, mounts, and probes.
get configmap,secret -o name: Confirms object names without reading Secret payloads or dumping ConfigMap content.

Command

Fill variables0/3 ready

Values stay on this page and are never sent or saved.

Namespace{{namespace}}Namespace containing the failing Pod and its owner. Namespace boundaries also scope Secrets, Services, and image pull credentials.Pod{{pod}}Exact current Pod name captured together with its UID so replacement is not confused with recovery.Container{{container}}Failing application or init-container name selected from the Pod specification.

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{range .spec.containers[?(@.name=="{{container}}")]}command={.command}{" args="}{.args}{"\n"}envFrom={.envFrom}{"\n"}volumeMounts={.volumeMounts}{"\n"}startup={.startupProbe}{"\n"}readiness={.readinessProbe}{"\n"}liveness={.livenessProbe}{"\n"}{end}' && kubectl -n {{namespace}} get configmap,secret -o name

Example output / evidence

command=[/app/payments-api] args=[serve]
envFrom=[{"configMapRef":{"name":"api-runtime"}},{"secretRef":{"name":"api-database"}}]
startup={"failureThreshold":30,"httpGet":{"path":"/startup","port":8080},"periodSeconds":2}
readiness={"httpGet":{"path":"/ready","port":8080},"periodSeconds":5}
configmap/api-runtime
secret/api-database

Checkpoint: Crash hypothesis identifies one declaration or dependency boundary

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.status.containerStatuses[?(@.name=="{{container}}")].lastState.terminated.reason}{" exit="}{.status.containerStatuses[?(@.name=="{{container}}")].lastState.terminated.exitCode}{" message="}{.status.containerStatuses[?(@.name=="{{container}}")].lastState.terminated.message}{"\n"}'

Continue whenTermination evidence and the inspected declaration support a specific process, probe, configuration, mount, or dependency hypothesis.

Stop whenThe proposed fix requires printing Secret data, weakening a security context, disabling probes globally, or changing an unknown external dependency.

If this step fails

The Pod alternates between Running and CrashLoopBackOff, and the restart count keeps increasing.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o wide
kubectl -n {{namespace}} logs {{pod}} -c {{container}} --previous --timestamps
kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.status.containerStatuses[?(@.name=="{{container}}")]}'

The Pod shows Running but never becomes Ready and receives no Service traffic.

Likely causeThe readiness probe fails, the application binds the wrong address or port, a dependency is unavailable, or the Service selector and target port do not match the Pod.

Safe checks

kubectl -n {{namespace}} describe pod {{pod}}
kubectl -n {{namespace}} get service {{service}} -o yaml
kubectl -n {{namespace}} get endpointslice -l kubernetes.io/service-name={{service}} -o wide

ResolutionTreat readiness separately from process survival. Repair the probe, listener, dependency, or Service mapping in declarative configuration, and verify EndpointSlice readiness before reopening traffic.

Security notes

List Secret metadata only. Validate Secret content through the owning secret manager and application-specific procedure.

Alternatives

Use a copied debug Pod with a changed command when the application exits before interactive inspection can begin.

Stop conditions

Do not disable liveness, remove securityContext, or add a shell to the production image as an unreviewed incident workaround.

decision

Diagnose image reference, registry authentication, and node-specific pulls

read-only

For ImagePullBackOff, compare the Pod and owner image, list namespace-local pull Secret names and ServiceAccount references, then use events to distinguish missing artifacts, authorization, TLS, DNS, architecture, and node storage.

Why this step matters

ImagePullBackOff is a retry state rather than a single cause. The image reference, pull policy, Secret metadata, ServiceAccount, event text, and node distribution distinguish artifact absence, authorization, trust, DNS, architecture, rate limit, and node storage problems.

What to understand

A private-registry Secret must exist in the same namespace as the Pod. The Pod or its ServiceAccount must reference the correct name, and the credential must match the registry host in the image reference.

Use events to distinguish `not found` from `unauthorized`, certificate errors, timeout, no space, or incompatible manifest. Test the artifact through the release pipeline or registry API without exposing a token in shell history.

If the same immutable digest pulls on other nodes, investigate the affected node's runtime, DNS, proxy, trust store, content cache, and disk. A workload edit may hide a node incident.

System changes

No cluster state changes; image references and credential object names are read without secret payloads.

Syntax explained

.spec.containers[?(@.name=="{{container}}")].image: Reads the exact repository, tag, or digest declared for the failing container.
.spec.imagePullSecrets[*].name: Lists Pod-level credential references which must resolve inside the same namespace.
.spec.serviceAccountName: Identifies the ServiceAccount that may contribute additional image pull Secret references.

Command

Fill variables0/3 ready

Values stay on this page and are never sent or saved.

Namespace{{namespace}}Namespace containing the failing Pod and its owner. Namespace boundaries also scope Secrets, Services, and image pull credentials.Pod{{pod}}Exact current Pod name captured together with its UID so replacement is not confused with recovery.Container{{container}}Failing application or init-container name selected from the Pod specification.

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='image={.spec.containers[?(@.name=="{{container}}")].image}{"\n"}pullPolicy={.spec.containers[?(@.name=="{{container}}")].imagePullPolicy}{"\n"}pullSecrets={.spec.imagePullSecrets[*].name}{"\n"}serviceAccount={.spec.serviceAccountName}{"\n"}' && kubectl -n {{namespace}} get serviceaccount,secret -o name

Example output / evidence

image=registry.example/payments/api@sha256:7db9f4b6a2c4
pullPolicy=IfNotPresent
pullSecrets=registry-payments
serviceAccount=api
serviceaccount/api
secret/registry-payments

Checkpoint: Image pull cause is classified without exposing credentials

kubectl -n {{namespace}} get events --field-selector involvedObject.name={{pod}},reason=Failed -o custom-columns=REASON:.reason,MESSAGE:.message

Continue whenThe latest pull failure message identifies artifact, authentication, transport, architecture, storage, or node runtime as the next boundary.

Stop whenResolving the failure would require disabling registry TLS, using a personal credential, copying a Secret across tenants, or changing an unsigned image tag.

If this step fails

The Pod reports ImagePullBackOff and events show `manifest unknown` or `not found`.

Likely causeThe image repository, tag, architecture-specific manifest, or digest in the Pod template does not exist in the target registry.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.spec.containers[*].image}{"\n"}'
kubectl -n {{namespace}} describe pod {{pod}}
kubectl -n {{namespace}} get {{workloadKind}} {{workloadName}} -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}'

ImagePullBackOff events contain `unauthorized`, `denied`, or `FailedToRetrieveImagePullSecret`.

Likely causeThe image pull Secret is missing from the Pod namespace, its name is misspelled, the registry credential expired, or the ServiceAccount does not reference the intended Secret.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.spec.imagePullSecrets[*].name}{"\n"}'
kubectl -n {{namespace}} get secret
kubectl -n {{namespace}} get serviceaccount -o custom-columns=NAME:.metadata.name,PULL_SECRETS:.imagePullSecrets[*].name

Only Pods scheduled to one node fail to pull an image while the same image starts elsewhere.

Likely causeThe affected node has DNS, proxy, certificate trust, disk, container-runtime, or registry reachability problems rather than a workload-level image reference problem.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o wide
kubectl describe node $(kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.spec.nodeName}')
kubectl get events -A --field-selector involvedObject.kind=Node --sort-by=.lastTimestamp

ResolutionCordon the node if failures are spreading, preserve kubelet and runtime evidence, and repair node connectivity or storage through the node runbook. Do not weaken registry TLS verification or add credentials to the node as a shortcut.

Security notes

Never decode `.dockerconfigjson` in terminal output. Rotate credentials through the approved secret controller and revoke any token accidentally exposed.

Alternatives

Promote a verified image digest through the normal release pipeline instead of retagging a local image during the incident.

Stop conditions

Escalate to a registry or node incident if multiple workloads or only one node fail to pull unrelated known-good digests.

decision

Diagnose Pending scheduling and storage constraints

read-only

Read FailedScheduling events, requests, node selection, affinity, tolerations, topology spread, PVCs, quotas, and eligible node capacity before considering any constraint change.

Why this step matters

Pending means the Pod has not reached a runnable node, but the constraint may be compute, taints, affinity, topology, quota, PVC binding, admission, or a missing scheduler. Reading all constraints before changing one prevents accidental policy bypass.

What to understand

Scheduler capacity uses requests, not current usage alone. Sum normal containers, consider init-container effective requests and Pod overhead, then compare with allocatable capacity and reservations on eligible nodes.

Read the exact FailedScheduling message and count filtered nodes by reason. Multiple predicates can be true simultaneously; adding capacity will not solve a missing toleration or unbound claim.

PVC binding can interact with topology and WaitForFirstConsumer. Inspect claim and StorageClass events before forcing nodeName, because manual placement can make the volume impossible to attach.

System changes

No cluster state changes; scheduling constraints, claims, quotas, and node allocatable resources are read from the API.

Syntax explained

.resources.requests: Displays the CPU, memory, and extended resources reserved for scheduling each container.
.spec.affinity and .spec.tolerations: Expose placement requirements and taint exceptions that can filter otherwise capable nodes.
get pvc,resourcequota: Checks namespace storage binding and quota gates which can leave a Pod Pending before startup.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='requests={range .spec.containers[*]}{.name}:{.resources.requests}{" "}{end}{"\n"}nodeSelector={.spec.nodeSelector}{"\n"}tolerations={.spec.tolerations}{"\n"}affinity={.spec.affinity}{"\n"}claims={.spec.volumes[*].persistentVolumeClaim.claimName}{"\n"}' && kubectl -n {{namespace}} get pvc,resourcequota && kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory

Example output / evidence

requests=api:{"cpu":"750m","memory":"768Mi"} metrics:{"cpu":"50m","memory":"64Mi"}
nodeSelector={"workload-tier":"general"}
claims=api-cache
NAME       TAINTS   CPU   MEMORY
worker-1   <none>   4     7875140Ki
worker-2   <none>   4     7875140Ki

Checkpoint: Every filtered node reason has an evidence-backed disposition

kubectl -n {{namespace}} get events --field-selector involvedObject.name={{pod}},reason=FailedScheduling -o custom-columns=TIME:.lastTimestamp,MESSAGE:.message

Continue whenThe newest scheduler message is recorded and each listed capacity, taint, affinity, topology, quota, or storage cause has an owner.

Stop whenThe only proposed workaround is removing requests, adding broad tolerations, setting nodeName, deleting a claim, or bypassing admission without understanding policy.

If this step fails

The Pod remains Pending and the scheduler reports `Insufficient cpu` or `Insufficient memory`.

Likely causeNo eligible node has allocatable capacity for the sum of container requests, init-container requirements, Pod overhead, and already reserved resources.

Safe checks

kubectl -n {{namespace}} describe pod {{pod}}
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory
kubectl top nodes

The scheduler reports untolerated taints, node affinity conflicts, or no nodes matching the selector.

Likely causeThe Pod template's scheduling constraints no longer match the labels and taints on available nodes, or a topology policy is stricter than the current failure-domain capacity.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.affinity}{"\n"}{.spec.tolerations}{"\n"}'
kubectl get nodes --show-labels
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

ResolutionReconcile the workload's documented placement intent with current node labels and taints. Change the controller template or node policy through review; never add a broad Exists toleration that permits sensitive workloads onto control-plane or dedicated nodes accidentally.

The Pod is Pending with an unbound PersistentVolumeClaim.

Likely causeNo compatible StorageClass, volume, topology, capacity, or provisioner can satisfy the claim; the Pod cannot be scheduled until storage binding succeeds.

Safe checks

kubectl -n {{namespace}} get pvc
kubectl -n {{namespace}} describe pvc
kubectl get storageclass -o wide

ResolutionDiagnose the claim and CSI provisioner before changing the Pod. Preserve the requested access mode and data-retention policy; do not delete a bound claim or change reclaim behavior during an application incident without a storage recovery plan.

Security notes

Node labels and taints can encode tenant, compliance, accelerator, and isolation boundaries. Treat them as policy, not merely placement hints.

Alternatives

Scale an eligible node group or restore a failed node when the request and placement policy are correct.

Stop conditions

Do not change placement or storage policy until the application owner and cluster capacity owner approve the resulting isolation and cost impact.

decision

Distinguish container OOMKilled from node memory pressure

read-only

Correlate last termination, requests and limits, current and historical usage, QoS, node conditions, eviction events, and application workload before changing memory policy.

Why this step matters

OOMKilled identifies a termination but not whether the container hit its limit or the node ran out of memory. Combining cgroup limits, usage history, QoS, node conditions, events, and workload behavior prevents a blind increase that moves risk elsewhere.

What to understand

Exit code 137 means SIGKILL; require lastState reason OOMKilled or node evidence before concluding memory. A rollout, manual deletion, or runtime failure can also terminate with SIGKILL.

Current `kubectl top` is a snapshot and may miss the peak immediately before termination. Use time-series metrics for working set, RSS, cache, allocation rate, request concurrency, and node available memory over the incident window.

Requests influence scheduling and QoS; limits enforce a ceiling. Correct application leaks, concurrency, batch size, cache bounds, JVM heap/native overhead, and sidecar consumption before selecting revised values.

System changes

No cluster state changes; container resources, last termination, metrics, and node conditions are read.

Syntax explained

.lastState.terminated.reason and exitCode: Distinguish an OOM termination from a generic restart and retain the process exit signal.
top pod --containers: Reports current per-container CPU and memory from the resource metrics API when installed.
describe node: Shows MemoryPressure, allocatable resources, requests, limits, and recent node-level warnings for the assigned node.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{range .status.containerStatuses[*]}{.name}{" lastReason="}{.lastState.terminated.reason}{" exit="}{.lastState.terminated.exitCode}{"\n"}{end}{range .spec.containers[*]}{.name}{" requests="}{.resources.requests}{" limits="}{.resources.limits}{"\n"}{end}{"qos="}{.status.qosClass}{"\n"}' && kubectl -n {{namespace}} top pod {{pod}} --containers && kubectl describe node $(kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.spec.nodeName}')

Example output / evidence

api lastReason=OOMKilled exit=137
api requests={"cpu":"750m","memory":"768Mi"} limits={"cpu":"2","memory":"1Gi"}
qos=Burstable
POD                      NAME   CPU(cores)   MEMORY(bytes)
api-7c8d9f54d8-k9q2m    api    612m         1008Mi
MemoryPressure   False

Checkpoint: Memory failure is classified as container-limit or node-pressure

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.status.qosClass}{" "}{range .status.containerStatuses[*]}{.name}:{.lastState.terminated.reason}/{.lastState.terminated.exitCode}{" "}{end}{"\n"}'

Continue whenShows the QoS class and a specific OOMKilled/137 container, with node conditions and metrics recorded separately.

Stop whenHistorical metrics are unavailable, node MemoryPressure affects multiple workloads, or increasing the limit would exceed documented node headroom.

If this step fails

The container last state is OOMKilled with exit code 137.

Likely causeThe process exceeded its cgroup memory limit, experienced an unbounded workload spike or leak, or shared node pressure led to eviction or a node-level OOM event.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{range .status.containerStatuses[*]}{.name}{" reason="}{.lastState.terminated.reason}{" exit="}{.lastState.terminated.exitCode}{"\n"}{end}'
kubectl -n {{namespace}} top pod {{pod}} --containers
kubectl describe node $(kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.spec.nodeName}')

ResolutionSeparate container-limit exhaustion from node pressure. Use historical metrics and workload behavior to correct the leak, concurrency, cache, JVM or runtime sizing, request, and limit; a blind limit increase may only postpone the next failure and consume node safety margin.

Security notes

Memory dumps can contain keys, tokens, customer records, and credentials; do not capture or export one without a separate secure forensic process.

Alternatives

Reduce workload concurrency or roll back the leaking release while a properly sized and tested application fix is prepared.

Stop conditions

Do not increase memory blindly when the application has unbounded growth or the node lacks capacity for the revised request and limit.

command

Use a constrained ephemeral or copied debug container

caution

When the application image lacks tools, attach an approved non-privileged debug image to a running Pod or create a disposable copy for a fast-crashing container, then remove the copy after evidence is captured.

Why this step matters

Distroless and minimal images intentionally omit shells and diagnostic tools. An approved ephemeral or copied debug container provides observation without modifying the application image, but it expands access and must remain constrained and auditable.

What to understand

Use `--target` to request the target container's namespaces when the runtime supports it. Confirm that the debug image architecture matches the node and that admission policy allows the selected profile.

For a fast-crashing process, `kubectl debug --copy-to` creates a separate Pod whose command can be changed. Never redirect production Service traffic to that copy or mount write-capable production volumes without an application-specific plan.

Record every command run inside the debug session. Inspect listeners, DNS, files, process state, and connectivity read-only; avoid installing packages or editing mounted configuration.

System changes

Adds an ephemeral container specification to the current Pod; this is auditable and cannot be removed from the Pod specification until the Pod is replaced.

Syntax explained

--image={{debugImage}}: Uses the organization's approved diagnostic image; pinning by digest prevents an unexpected toolset.
--target={{container}}: Requests attachment to the target container's process namespace when supported by the container runtime.
--profile=general: Uses a non-privileged general debug profile rather than escalating to node or sysadmin capabilities.

Command

Fill variables0/4 ready

Values stay on this page and are never sent or saved.

Namespace{{namespace}}Namespace containing the failing Pod and its owner. Namespace boundaries also scope Secrets, Services, and image pull credentials.Pod{{pod}}Exact current Pod name captured together with its UID so replacement is not confused with recovery.Debug Image{{debugImage}}Approved diagnostic image pinned by digest and compatible with the Pod architecture.Container{{container}}Failing application or init-container name selected from the Pod specification.

kubectl -n {{namespace}} debug {{pod}} -it --image={{debugImage}} --target={{container}} --profile=general -- sh

Example output / evidence

Targeting container "api". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-x7k2p.
/ #

Checkpoint: Debug access is scoped and evidence is captured

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{range .status.ephemeralContainerStatuses[*]}{.name}{" image="}{.image}{" state="}{.state.running.startedAt}{"\n"}{end}'

Continue whenShows only the approved debug image and a running or completed status tied to the incident.

Stop whenThe image is unapproved, the command requests privilege or host access, the Pod contains highly sensitive data without authorization, or the runtime cannot isolate the intended target.

If this step fails

A debug container starts but cannot see the target process, filesystem path, or network namespace expected.

Likely causeThe ephemeral container did not request process sharing, targets the wrong container runtime namespace, or the original container terminated before attachment.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.spec.shareProcessNamespace}{"\n"}'
kubectl -n {{namespace}} describe pod {{pod}}

ResolutionUse a copied debug Pod with `--copy-to` for a repeatedly crashing container, or target the correct running container with an approved ephemeral image. Do not add privileged capabilities or host mounts merely to make inspection easier.

Security notes

An ephemeral container can inspect shared network and volume data. Apply least privilege, audit the session, and remove temporary RBAC after the incident.

Alternatives

Use `kubectl debug --copy-to` with a non-production copy when the original container is not running long enough for attachment.

Stop conditions

Do not add privileged mode, host namespaces, hostPath, or broad Linux capabilities merely to bypass a failed diagnostic attempt.

warning

Change the owning declaration and watch one controlled rollout

caution

Commit or apply the smallest reviewed change to the workload source of truth, record the prior revision, and stop automatically if rollout health or service error signals degrade.

Why this step matters

The correction must live in the durable declaration and be observed as a controlled rollout. Recording both revisions and stopping on adverse signals protects against turning one failing Pod into a service-wide outage.

What to understand

Prefer a reviewed commit in the normal deployment source. If emergency API mutation is explicitly permitted, immediately reconcile that exact patch back into the source of truth and document drift.

Change one causal variable when possible: image digest, object reference, probe threshold, resource setting, or placement rule. Bundled changes make rollback and evidence interpretation harder.

Watch desired, updated, available, and unavailable replicas together with application error rate, dependency load, queue depth, and latency. Kubernetes rollout success alone does not prove business recovery.

System changes

The actual manifest change occurs in the deployment source outside this read-only command; the command observes the resulting controller rollout and replacement Pods.

Syntax explained

rollout history: Records the revision being replaced and the explicit rollback candidate before the new rollout proceeds.
rollout status --timeout=10m: Waits for controller completion but exits after a bounded interval so a stalled rollout becomes a decision point.
get pods -l ... -o wide: Shows whether replacement replicas are Ready, restarting, or concentrated on one node.

Command

Fill variables0/4 ready

Values stay on this page and are never sent or saved.

Namespace{{namespace}}Namespace containing the failing Pod and its owner. Namespace boundaries also scope Secrets, Services, and image pull credentials.Workload Kind{{workloadKind}}Lowercase kubectl resource kind of the durable owner.Workload Name{{workloadName}}Name of the controller that owns the Pod template.Label Selector{{labelSelector}}Narrow label selector that identifies only the workload's Pods.

kubectl -n {{namespace}} rollout history {{workloadKind}}/{{workloadName}} && kubectl -n {{namespace}} rollout status {{workloadKind}}/{{workloadName}} --timeout=10m && kubectl -n {{namespace}} get pods -l '{{labelSelector}}' -o wide

Example output / evidence

deployment.apps/api
REVISION  CHANGE-CAUSE
41        release 2026.07.21
42        release 2026.07.28
43        fix DATABASE_HOST reference
deployment "api" successfully rolled out
NAME                   READY   STATUS    RESTARTS   AGE   IP          NODE
api-645bcdd97f-2w4qj   2/2     Running   0          4m    10.42.1.9  worker-1

Checkpoint: One reviewed revision replaces the failing Pods safely

kubectl -n {{namespace}} get {{workloadKind}} {{workloadName}} -o jsonpath='desired={.spec.replicas} updated={.status.updatedReplicas} available={.status.availableReplicas} unavailable={.status.unavailableReplicas}{"\n"}'

Continue whenUpdated and available equal desired, unavailable is zero, and external health signals have not degraded.

Stop whenUnavailable replicas increase beyond policy, the rollout stalls, new Pods show a different failure, data integrity is uncertain, or application error and dependency load rise.

If this step fails

The rollout creates healthy new Pods but the old unhealthy revision remains or rollout status times out.

Likely causeA PodDisruptionBudget, unavailable capacity, maxUnavailable/maxSurge policy, failing readiness check, or finalizer prevents the controller from completing replacement.

Safe checks

kubectl -n {{namespace}} rollout status {{workloadKind}}/{{workloadName}} --timeout=60s
kubectl -n {{namespace}} get pods -l '{{labelSelector}}' -o wide
kubectl -n {{namespace}} get poddisruptionbudget

Security notes

A direct API patch can bypass repository review and be overwritten by GitOps. Use the approved break-glass path and preserve audit identity.

Alternatives

Roll back to the recorded known-good revision before preparing a forward fix when the incident is active and the earlier artifact remains supported.

Stop conditions

Abort and roll back immediately if the change reduces ready capacity or worsens customer and dependency signals.

verification

Verify restart stability, endpoints, and real service behavior

read-only

Observe the workload beyond its previous failure interval, prove ready EndpointSlices, query health through the intended network path, and compare error rate and resource use with the baseline.

Why this step matters

A Pod can reach Running briefly and still restart, remain outside the Service, exceed its memory envelope, or fail real requests. Acceptance must outlast the previous failure interval and cover the control plane, data plane, and application signal.

What to understand

Observe restart counts over at least the former back-off, probe, or workload cycle. A zero restart count immediately after creation is necessary but not sufficient.

EndpointSlices prove that Service selection and readiness publish the backend. Compare expected replica count, addresses, ports, zones, and readiness conditions.

Run a synthetic request through the same DNS, load balancer, TLS, Service, and application route used by consumers. Pair it with application-level integrity and dependency checks defined by the service owner.

System changes

No cluster state changes; Pod status, EndpointSlices, and resource metrics are read for acceptance evidence.

Syntax explained

custom-columns=...STARTED: Makes container start time and restart counters visible for stability comparison across replicas.
-l kubernetes.io/service-name={{service}}: Selects EndpointSlices owned by the intended Service rather than inspecting unrelated endpoints.
top pod -l ... --containers: Confirms per-container usage after recovery instead of hiding a leaking sidecar in Pod totals.

Command

Fill variables0/3 ready

Values stay on this page and are never sent or saved.

kubectl -n {{namespace}} get pods -l '{{labelSelector}}' -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[*].ready,RESTARTS:.status.containerStatuses[*].restartCount,STARTED:.status.containerStatuses[*].state.running.startedAt && kubectl -n {{namespace}} get endpointslice -l kubernetes.io/service-name={{service}} -o wide && kubectl -n {{namespace}} top pod -l '{{labelSelector}}' --containers

Example output / evidence

NAME                   READY        RESTARTS   STARTED
api-645bcdd97f-2w4qj   true,true    0,0        2026-07-28T09:18:12Z
NAME                     ADDRESSTYPE   PORTS   ENDPOINTS   AGE
payments-api-k9g6s       IPv4          8080    10.42.1.9   6m
POD                      NAME   CPU(cores)   MEMORY(bytes)
api-645bcdd97f-2w4qj     api    184m         612Mi

Checkpoint: Recovered service remains stable through the acceptance window

kubectl -n {{namespace}} get pods -l '{{labelSelector}}' -o jsonpath='{range .items[*]}{.metadata.name}{" ready="}{range .status.conditions[?(@.type=="Ready")]}{.status}{end}{" restarts="}{range .status.containerStatuses[*]}{.restartCount}{","}{end}{"\n"}{end}'

Continue whenEvery desired Pod remains Ready with unchanged restart counts, and the Service plus external synthetic checks pass.

Stop whenAny restart counter increments, endpoints disappear, resource use approaches the revised limit, or real requests fail despite healthy-looking Kubernetes status.

If this step fails

The Pod shows Running but never becomes Ready and receives no Service traffic.

Likely causeThe readiness probe fails, the application binds the wrong address or port, a dependency is unavailable, or the Service selector and target port do not match the Pod.

Safe checks

kubectl -n {{namespace}} describe pod {{pod}}
kubectl -n {{namespace}} get service {{service}} -o yaml
kubectl -n {{namespace}} get endpointslice -l kubernetes.io/service-name={{service}} -o wide

Security notes

Synthetic verification must use a test identity and non-sensitive payload; do not put production credentials into copied commands.

Alternatives

Use the service's existing canary, smoke-test, or transaction monitor when it exercises the complete consumer path.

Stop conditions

Do not close the incident on rollout success alone; require stable restarts, ready endpoints, representative traffic, and resource headroom.

Finish line

Verification checklist

Stable containers

kubectl -n {{namespace}} get pods -l '{{labelSelector}}' -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[*].ready,RESTARTS:.status.containerStatuses[*].restartCount

Every desired Pod is ready and restart counts remain unchanged beyond the former crash or probe interval.

Controller rolloutkubectl -n {{namespace}} rollout status {{workloadKind}}/{{workloadName}} --timeout=60sThe owning controller reports a completed rollout at the reviewed revision.

Service endpointskubectl -n {{namespace}} get endpointslice -l kubernetes.io/service-name={{service}} -o wideThe expected number of ready endpoint addresses is published on the intended port.

No repeating warningkubectl -n {{namespace}} get events --sort-by=.lastTimestamp --field-selector involvedObject.kind=Pod | tail -n 30No new BackOff, Failed, Unhealthy, FailedScheduling, FailedMount, or OOM warning repeats for the recovered revision.

Resource headroomkubectl -n {{namespace}} top pod -l '{{labelSelector}}' --containersObserved CPU and memory remain within the reviewed operating envelope during representative traffic.

Recovery guidance

Common problems and safe checks

The Pod alternates between Running and CrashLoopBackOff, and the restart count keeps increasing.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o wide
kubectl -n {{namespace}} logs {{pod}} -c {{container}} --previous --timestamps
kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.status.containerStatuses[?(@.name=="{{container}}")]}'

The previous log command returns `previous terminated container not found` even though the Pod is unhealthy.

Likely causeThe selected container has not restarted, the wrong container name was used, or the Pod was replaced and its earlier container history belongs to a different Pod UID.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.metadata.uid}{"\n"}{.spec.containers[*].name}{"\n"}'
kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{"\n"}{end}'

The Pod reports ImagePullBackOff and events show `manifest unknown` or `not found`.

Likely causeThe image repository, tag, architecture-specific manifest, or digest in the Pod template does not exist in the target registry.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.spec.containers[*].image}{"\n"}'
kubectl -n {{namespace}} describe pod {{pod}}
kubectl -n {{namespace}} get {{workloadKind}} {{workloadName}} -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}'

ImagePullBackOff events contain `unauthorized`, `denied`, or `FailedToRetrieveImagePullSecret`.

Likely causeThe image pull Secret is missing from the Pod namespace, its name is misspelled, the registry credential expired, or the ServiceAccount does not reference the intended Secret.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.spec.imagePullSecrets[*].name}{"\n"}'
kubectl -n {{namespace}} get secret
kubectl -n {{namespace}} get serviceaccount -o custom-columns=NAME:.metadata.name,PULL_SECRETS:.imagePullSecrets[*].name

Only Pods scheduled to one node fail to pull an image while the same image starts elsewhere.

Likely causeThe affected node has DNS, proxy, certificate trust, disk, container-runtime, or registry reachability problems rather than a workload-level image reference problem.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o wide
kubectl describe node $(kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.spec.nodeName}')
kubectl get events -A --field-selector involvedObject.kind=Node --sort-by=.lastTimestamp

The Pod remains Pending and the scheduler reports `Insufficient cpu` or `Insufficient memory`.

Likely causeNo eligible node has allocatable capacity for the sum of container requests, init-container requirements, Pod overhead, and already reserved resources.

Safe checks

kubectl -n {{namespace}} describe pod {{pod}}
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory
kubectl top nodes

The scheduler reports untolerated taints, node affinity conflicts, or no nodes matching the selector.

Likely causeThe Pod template's scheduling constraints no longer match the labels and taints on available nodes, or a topology policy is stricter than the current failure-domain capacity.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.affinity}{"\n"}{.spec.tolerations}{"\n"}'
kubectl get nodes --show-labels
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

The Pod is Pending with an unbound PersistentVolumeClaim.

Likely causeNo compatible StorageClass, volume, topology, capacity, or provisioner can satisfy the claim; the Pod cannot be scheduled until storage binding succeeds.

Safe checks

kubectl -n {{namespace}} get pvc
kubectl -n {{namespace}} describe pvc
kubectl get storageclass -o wide

The container last state is OOMKilled with exit code 137.

Likely causeThe process exceeded its cgroup memory limit, experienced an unbounded workload spike or leak, or shared node pressure led to eviction or a node-level OOM event.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{range .status.containerStatuses[*]}{.name}{" reason="}{.lastState.terminated.reason}{" exit="}{.lastState.terminated.exitCode}{"\n"}{end}'
kubectl -n {{namespace}} top pod {{pod}} --containers
kubectl describe node $(kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.spec.nodeName}')

The Pod shows Running but never becomes Ready and receives no Service traffic.

Likely causeThe readiness probe fails, the application binds the wrong address or port, a dependency is unavailable, or the Service selector and target port do not match the Pod.

Safe checks

kubectl -n {{namespace}} describe pod {{pod}}
kubectl -n {{namespace}} get service {{service}} -o yaml
kubectl -n {{namespace}} get endpointslice -l kubernetes.io/service-name={{service}} -o wide

A debug container starts but cannot see the target process, filesystem path, or network namespace expected.

Likely causeThe ephemeral container did not request process sharing, targets the wrong container runtime namespace, or the original container terminated before attachment.

Safe checks

kubectl -n {{namespace}} get pod {{pod}} -o jsonpath='{.spec.shareProcessNamespace}{"\n"}'
kubectl -n {{namespace}} describe pod {{pod}}

The rollout creates healthy new Pods but the old unhealthy revision remains or rollout status times out.

Likely causeA PodDisruptionBudget, unavailable capacity, maxUnavailable/maxSurge policy, failing readiness check, or finalizer prevents the controller from completing replacement.

Safe checks

kubectl -n {{namespace}} rollout status {{workloadKind}}/{{workloadName}} --timeout=60s
kubectl -n {{namespace}} get pods -l '{{labelSelector}}' -o wide
kubectl -n {{namespace}} get poddisruptionbudget

After the procedure

Alternatives and next steps

Consider these alternatives

Use the organization's log and metrics platform for history that has already aged out of the Kubernetes API, correlating by namespace, Pod UID, container, node, and rollout revision.
Use `kubectl debug --copy-to` when a container crashes too quickly for an ephemeral container, preserving the original Pod and changing the command only in the disposable copy.
For managed Kubernetes, combine this workflow with provider control-plane and node diagnostics when API events point outside the workload.
For a broad node outage, switch to the node-pressure or runtime incident runbook rather than patching every affected workload separately.

Operate it safely

Create alerts on restart acceleration, image pull failures, unschedulable duration, OOMKilled terminations, readiness loss, and unavailable replicas using labels that identify the owning service.
Add startup, readiness, and liveness probes with distinct purposes, realistic failure thresholds, and documented dependency behavior.
Pin deployable images by immutable digest, automate registry-credential rotation, and verify architecture manifests before promotion.
Measure resource usage over representative traffic and set requests and limits from evidence rather than from copied defaults.
Retain rollout annotations, application version, Git revision, and change ticket in workload metadata so incident evidence maps directly to a release.

Reference

Frequently asked questions

Should I delete a CrashLoopBackOff Pod first?

No. Capture status, previous logs, events, UID, owner, and revision first. A controller recreates the same template, while deletion can erase the most useful evidence and briefly increase dependency load.

Is exit code 137 always a container memory-limit failure?

No. It indicates SIGKILL. OOMKilled in the last termination state strongly supports a memory event, but you must distinguish cgroup limit exhaustion from node pressure and deliberate termination using Pod state, events, node conditions, and historical metrics.

Can I fix Pending by lowering requests?

Only after measured demand and capacity prove the existing request is incorrect. Lowering it blindly may schedule the Pod while creating contention or eviction. Capacity, placement constraints, quotas, PVC binding, and topology may be the real cause.

Why inspect EndpointSlices after the Pod becomes Ready?

They show whether the Service controller actually published the recovered backend. This catches selector, port, readiness, and topology mistakes that Pod status alone cannot prove.

Recovery

Rollback

Return the owning controller to its last known-good declarative revision if the remediation increases failures, removes endpoints, or creates unsafe resource or dependency behavior.

Stop further rollout automation and record the current and previous controller revisions, Pod UIDs, images, and incident timestamps.
Revert the GitOps or deployment source to the last known-good manifest; when that system is unavailable and policy permits, use the controller-specific rollout undo to the explicitly recorded revision.
Watch the rollback with `kubectl rollout status`, preserving events and logs from both failed and restored Pods.
Verify ready EndpointSlices, external traffic, data integrity, dependency health, restart stability, and resource usage before closing the incident.
Remove disposable debug Pods and ephemeral access grants, but retain redacted evidence and the failed manifest for root-cause analysis.

Evidence

Sources and review

Verified 2026-07-24Review due 2026-10-22

Kubernetes Debug Running Podsofficial Kubernetes Pull an Image from a Private Registryofficial Kubernetes Assign Memory Resources to Containers and Podsofficial Kubernetes Resource Management for Pods and Containersofficial Kubernetes Pod Lifecycleofficial