Containers & Kubernetes / Packages & Updates

Upgrade a kubeadm Kubernetes cluster from 1.35 to 1.36 with controlled drains and recovery

Upgrade a kubeadm-managed Kubernetes 1.35 cluster to an exact 1.36 patch one control plane and worker at a time, preserving quorum, application availability, backups, and recovery evidence.

240 min12 stepsHigh-impact changeRevision 2

Save or explore

#kubernetes #kubeadm #upgrade #drain #etcd #version-skew #rollback

0 of 12 steps completed

Goal

Complete the supported no-skipped-minor kubeadm sequence while every advancement gate proves API, etcd, node, add-on, networking, storage, and workload health.

Supported environments

Kubernetes 1.35.x, 1.36.x
kubeadm 1.35.x, 1.36.x
Ubuntu Server 24.04 LTS

Prerequisites

Supported start Prove every control plane starts on a healthy Kubernetes 1.35 patch and that no minor version will be skipped.kubectl version && kubectl get nodes -o wide
Quorum and recovery Document etcd topology, quorum, snapshot and restore procedure, control-plane endpoint, console path, and responsible recovery operators.
Application protection Verify application backups, spare capacity, PodDisruptionBudgets, topology, data semantics, synthetic checks, and an owner for every critical workload.
Compatibility review Approve 1.36 support for the container runtime, cgroups v2, CNI, CSI, ingress controller, operators, webhooks, observability, and security agents.
Release review Read Kubernetes 1.36 release notes, version skew policy, kubeadm upgrade guide, and relevant add-on upgrade documents before the window.

Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System

A staged upgrade plan for a kubeadm-managed, highly available Kubernetes cluster from the latest 1.35 patch to one exact 1.36 patch without skipping a minor release.
A control-plane and application recovery package containing etcd snapshot evidence, static manifest backups, application backups, version inventory, CNI compatibility, drain constraints, and a tested administrative path.
A node-by-node execution and acceptance record that keeps capacity, disruption budgets, DNS, networking, storage, API health, and external service behavior visible at every gate.

Observable outcome

The first control plane, every additional control plane, and each worker complete the supported kubeadm sequence with matching target packages and healthy kubelets.
Every drained node returns Ready and schedulable only after system DaemonSets, workload traffic, DNS, Service routing, CNI behavior, and storage have passed.
The cluster reports Kubernetes 1.36 on all intended nodes, no skipped minor, no unexplained failed workloads, and a retained recovery package for the acceptance window.

Architecture

How the parts fit together

A kubeadm cluster upgrades in layers. The first control-plane node changes cluster-wide control-plane configuration, while additional control planes run the node phase. Kubelet upgrades require a prior drain. Add-ons and CNI have separate compatibility and sequencing. Etcd quorum and the API load balancer must survive one control-plane instance being restarted. Worker capacity and PodDisruptionBudgets must survive one node being drained. The process advances only when the previous layer is accepted.

Primary control-plane nodeRuns `kubeadm upgrade plan` and `kubeadm upgrade apply` with admin.conf, initiating the cluster-wide 1.36 transition.

Additional control-plane nodesRun `kubeadm upgrade node` sequentially and preserve etcd and API quorum while their static Pods restart.

Worker nodesAre drained, upgraded, restarted, verified, and uncordoned in bounded batches that preserve application capacity.

EtcdStores cluster state. Snapshot, member health, quorum, version, and restore rehearsal define the disaster-recovery boundary.

CNI and kube-proxyProvide Pod and Service networking and require their own release compatibility and health checks.

Control-plane endpointLoad balances API clients across healthy instances and must remove a restarting API server quickly enough to preserve availability.

Workload protectionPodDisruptionBudgets, replica topology, capacity, application backups, and synthetic checks prevent maintenance from becoming customer downtime.

Verify that the current cluster is a healthy kubeadm 1.35 cluster and select the latest exact 1.36 patch from the official package stream.
Review release notes, deprecated APIs, runtime and cgroup requirements, CNI and CSI support, and application disruption budgets.
Create and verify application backups, an etcd snapshot, configuration evidence, and independent access before touching packages.
Upgrade kubeadm on one primary control plane, run the plan, apply the target, and accept API plus etcd health.
Upgrade CNI as required, then process additional control planes one at a time.
Drain, upgrade kubeadm, kubelet, and kubectl, restart, verify, and uncordon each worker or bounded batch.
Run cluster-wide and application acceptance, retain backups, and remove temporary maintenance artifacts only after the rollback window.

Assumptions

The cluster was created and is currently managed with kubeadm, uses static control-plane and local or external etcd as documented by kubeadm, and starts at Kubernetes 1.35.x.
The target is the latest approved Kubernetes 1.36 patch available from `pkgs.k8s.io`; exact package versions are discovered at execution time and recorded before installation.
Ubuntu 24.04 LTS nodes use systemd, containerd or another supported CRI runtime, cgroups v2, and package holds for kubeadm, kubelet, and kubectl.
The control plane has enough healthy instances and etcd quorum to upgrade one instance at a time, or the maintenance plan explicitly accepts single-control-plane downtime.
Applications have PodDisruptionBudgets, spare capacity, data backups, and owner-approved synthetic verification; critical state is not protected solely by etcd.
CNI, CSI, ingress, admission webhooks, observability agents, and operators have documented compatibility with Kubernetes 1.36 before the first apply.
Console or out-of-band access exists for every control plane, and the API endpoint remains reachable through healthy instances during maintenance.

Key concepts

Version skew: The supported difference between Kubernetes component versions. Kubeadm upgrades one minor at a time; skipping 1.35 directly to a later minor is unsupported.
Exact patch: A complete package version such as `1.36.x-*` selected from the repository. Every operator records the resolved patch rather than installing an ambiguous newest package mid-window.
Drain: Marks a node unschedulable and requests eviction of ordinary workloads while respecting PodDisruptionBudgets. It does not magically protect local data or unmanaged Pods.
Cordon: Prevents new scheduling without evicting existing workloads. A failed or unverified upgraded node remains cordoned.
Static Pod: A control-plane component manifest managed directly by kubelet under `/etc/kubernetes/manifests`. Kubeadm rewrites these files during upgrade.
Etcd snapshot: A point-in-time backup of Kubernetes cluster state. It does not include application data in external databases or persistent volumes and must have a rehearsed restore path.
Idempotent upgrade: Kubeadm upgrade can be rerun after a corrected failure to converge to the declared target. This is safer than inventing an unsupported package downgrade.
Acceptance gate: Observable health criteria that must pass before the next control plane or worker is touched, preserving a known stopping point.

Before you copy

Values used in this guide

{{targetKubernetesVersion}}

Exact 1.36 package version returned by the approved package repository, including package suffix.

Example: 1.36.1-1.1

{{targetKubernetesRelease}}

Exact semantic Kubernetes release reported by kubeadm upgrade plan for the approved 1.36 package.

Example: v1.36.1

{{primaryControlPlane}}

First control-plane node containing `/etc/kubernetes/admin.conf` and selected for `kubeadm upgrade apply`.

Example: cp-01

{{node}}

Current additional control-plane or worker node processed alone or within the approved bounded batch.

Example: worker-03

{{controlPlaneEndpoint}}

Stable API load balancer name used by nodes and administrators.

Example: api.cluster.example.com

{{etcdSnapshot}}

Encrypted, off-node path for the verified pre-upgrade etcd snapshot.

Example: /var/backups/kubernetes/2026-07-28/etcd-pre-1.36.db

{{maintenanceBatch}}

Maximum number of workers upgraded together after capacity and disruption analysis.

Example: 1

Security and production boundaries

Files under `/etc/kubernetes`, admin.conf, etcd snapshots, certificates, keys, and kubeconfig contain cluster-admin or equivalent authority. Encrypt backups, restrict permissions, and transfer them only through the approved recovery channel.
Package repository keys and sources are part of the software supply chain. Use the official pkgs.k8s.io per-minor repository, verify signed metadata, select an exact patch, and preserve package policy evidence.
Do not use `--force`, `--disable-eviction`, broad PDB removal, or control-plane downtime as routine shortcuts. Each bypass can remove the safety property the gate is intended to prove.
Review deprecated APIs and admission webhooks before upgrade. A webhook that becomes unreachable or incompatible can block all workload changes after the API moves forward.
Kubeadm backs up manifests and local etcd during upgrade, but those automatic directories are not a complete disaster-recovery design and do not replace an independent verified snapshot.

Stop before continuing if

Stop if the starting cluster is not healthy Kubernetes 1.35, if any node version is unknown, or if an operator proposes skipping directly from an older minor.
Stop if etcd quorum, snapshot restore evidence, application backup, console access, control-plane endpoint health, or package supply-chain verification is missing.
Stop if any CNI, CSI, ingress, runtime, operator, or admission component lacks confirmed Kubernetes 1.36 support.
Stop if draining violates a PodDisruptionBudget, exposes unmanaged local data, or removes more healthy capacity than the application can tolerate.
Stop after any control-plane, DNS, network, storage, node Ready, or external application acceptance gate fails. Keep the current node isolated and do not advance the batch.

verification

Inventory versions, topology, and control-plane ownership

read-only

Capture server and client versions, every node's kubelet and runtime, labels, taints, control-plane instances, etcd members, kube-system workloads, and active package candidates.

Why this step matters

A supported transition starts from known component versions and topology. Mixed minors, hidden control planes, unexpected runtimes, or an already unschedulable node change the sequence and recovery design before any package can be selected safely.

What to understand

Capture both API and kubectl versions, every node's kubelet, runtime, operating system, roles, labels, taints, Ready condition, and schedulability. Confirm that all control planes are 1.35 before planning 1.36.

Identify whether etcd is stacked on control-plane nodes or external, how many voting members exist, and which node will run the first apply. Record the stable API endpoint and load-balancer health behavior.

Inventory system add-ons and images now so later restarts or version changes are attributable to the maintenance window rather than assumed from memory.

System changes

No cluster state changes; version, topology, and kube-system workload status are read from the API.

Syntax explained

custom-columns=...KUBELET,RUNTIME,OS: Projects the node software layers that must satisfy target skew and compatibility.
-n kube-system get pods -o wide: Shows system workload placement, readiness, restarts, IPs, and nodes before disruption.

Command

kubectl version && kubectl get nodes -o custom-columns=NAME:.metadata.name,ROLES:.metadata.labels.node-role\.kubernetes\.io/control-plane,KUBELET:.status.nodeInfo.kubeletVersion,RUNTIME:.status.nodeInfo.containerRuntimeVersion,OS:.status.nodeInfo.osImage,READY:.status.conditions[-1].type && kubectl -n kube-system get pods -o wide

Example output / evidence

Client Version: v1.35.7
Server Version: v1.35.7
NAME       ROLES   KUBELET   RUNTIME              OS                    READY
cp-01      true    v1.35.7   containerd://2.1.4   Ubuntu 24.04.3 LTS    Ready
cp-02      true    v1.35.7   containerd://2.1.4   Ubuntu 24.04.3 LTS    Ready
worker-01  <none>  v1.35.7   containerd://2.1.4   Ubuntu 24.04.3 LTS    Ready

Checkpoint: Supported 1.35 starting topology is complete

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.nodeInfo.kubeletVersion}{" ready="}{range .status.conditions[?(@.type=="Ready")]}{.status}{end}{"\n"}{end}'

Continue whenEvery node is documented, Ready, and on an approved 1.35 patch, with control planes and etcd topology identified.

Stop whenAny node is NotReady, uses an unsupported minor, has unknown ownership, or the cluster was not created and managed by kubeadm.

If this step fails

`kubeadm upgrade plan` refuses the target and reports that the version is too far ahead.

Likely causeThe cluster is older than 1.35, the local kubeadm package is not 1.36, or an operator attempted to skip one or more minor versions.

Safe checks

kubectl version
kubeadm version
kubectl get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion

ResolutionUpgrade one minor release at a time using the archived official guide for each transition. Bring every control plane and worker to the supported starting minor before planning 1.35 to 1.36; never use `--force` to bypass version-skew policy.

Security notes

Do not include kubeconfig content or certificate material in the inventory report.

Alternatives

Use a provider-specific upgrade workflow if the control plane is managed and kubeadm is not its supported interface.

Stop conditions

Never proceed from 1.34 or older directly to 1.36; complete and accept each minor transition.

verification

Prove API, etcd, workloads, disruption budgets, and spare capacity

read-only

Require verbose API readiness, Ready nodes, healthy system workloads, no unexplained failures, satisfiable PodDisruptionBudgets, and capacity to drain one node before maintenance.

Why this step matters

An upgrade should change a healthy cluster, not conceal an existing failure. API readiness, etcd, nodes, system Pods, application state, PDBs, and spare capacity establish whether one component or node can restart without violating availability.

What to understand

Use verbose readyz because a successful TCP or version request can coexist with a failed post-start hook, etcd check, or informer synchronization.

Review every PDB's currentHealthy, desiredHealthy, disruptionsAllowed, and unhealthyPodEvictionPolicy. Confirm replacement replicas can schedule outside the node selected for maintenance.

List non-running Pods and compare with an approved exception list. Existing Pending, CrashLoopBackOff, storage, or DNS failures must be resolved before their signals are mixed with upgrade effects.

System changes

No cluster state changes; API, node, system workload, PDB, and Pod health are read.

Syntax explained

get --raw='/readyz?verbose': Queries each API server readiness check instead of relying on a shallow endpoint response.
--field-selector=status.phase!=Running,status.phase!=Succeeded: Surfaces Pods outside normal active or completed phases for explicit disposition.

Command

kubectl get --raw='/readyz?verbose' && kubectl get nodes && kubectl -n kube-system get pods && kubectl get poddisruptionbudget -A && kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded

Example output / evidence

[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/rbac/bootstrap-roles ok
readyz check passed
NAME       STATUS   ROLES           AGE   VERSION
cp-01      Ready    control-plane   420d  v1.35.7
cp-02      Ready    control-plane   420d  v1.35.7
worker-01  Ready    <none>          420d  v1.35.7

Checkpoint: The cluster can lose one planned node safely

kubectl get poddisruptionbudget -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ALLOWED:.status.disruptionsAllowed,CURRENT:.status.currentHealthy,DESIRED:.status.desiredHealthy

Continue whenCritical workloads have healthy replicas and at least the disruption allowance required for the selected node and batch.

Stop whenAPI readiness fails, etcd is unhealthy, a node is NotReady, a critical PDB allows no planned eviction, or spare capacity is unproven.

If this step fails

The upgrade plan reports an unhealthy control plane or unreachable API server.

Likely causeA static control-plane Pod, etcd member, load balancer path, certificate, DNS record, or node is already unhealthy before the maintenance starts.

Safe checks

kubectl get --raw='/readyz?verbose'
kubectl -n kube-system get pods -o wide
kubectl get nodes

ResolutionCancel the upgrade, restore the current 1.35 control plane to a fully healthy state, and repeat backup plus preflight. An upgrade is not a repair mechanism for an unexplained control-plane incident.

Draining a node waits indefinitely because eviction is blocked by a PodDisruptionBudget.

Likely causeThe protected workload lacks enough healthy replicas, capacity, or topology to satisfy its disruption budget while one node is unavailable.

Safe checks

kubectl get poddisruptionbudget -A
kubectl get pods -A --field-selector spec.nodeName={{node}} -o wide
kubectl describe node {{node}}

ResolutionRestore workload health or capacity and let the eviction API honor the budget. Do not add `--disable-eviction` or delete the budget unless the application owner has a tested, documented availability exception.

Security notes

Cluster-wide read output exposes workload and infrastructure names; keep the preflight in the restricted change record.

Alternatives

Add temporary approved capacity and restore unhealthy replicas before opening the maintenance window.

Stop conditions

Do not treat `--force` or a PDB bypass as a substitute for preflight health and capacity.

decision

Review release, API, runtime, cgroup, CNI, CSI, and webhook compatibility

read-only

Record component versions and deprecated API use, confirm cgroups v2 on every node, and link each add-on to an approved Kubernetes 1.36 compatibility decision.

Why this step matters

Kubernetes upgrades an ecosystem, not only four core binaries. A runtime, CNI, CSI, ingress controller, operator, webhook, or API client that does not support 1.36 can make a successful kubeadm apply an application outage.

What to understand

Create a component matrix with installed image digest, current version, target-compatible version, official compatibility source, owner, change order, and rollback. Include admission and conversion webhooks because they can block all API writes.

Kubernetes 1.36 sets FailCgroupV1 true by default for kubelet. Confirm each node uses cgroups v2 and that the runtime plus monitoring stack understands it.

Use API request metrics and manifest scanners to find deprecated APIs, then exercise operators and CI clients against a staging 1.36 API. Absence of a current metric does not prove an offline manifest is compatible.

System changes

No cluster state changes; component images, webhook registrations, and deprecation metrics are read.

Syntax explained

get daemonset,deployment -A: Enumerates long-running cluster agents and controllers whose images need explicit 1.36 compatibility.
apiserver_requested_deprecated_apis: Surfaces deprecated API requests observed by the API server for migration planning.

Command

kubectl get daemonset,deployment -A -o custom-columns=NS:.metadata.namespace,KIND:.kind,NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image && kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations && kubectl get --raw /metrics | grep 'apiserver_requested_deprecated_apis.* 1$' || true

Example output / evidence

NAMESPACE     KIND         NAME              IMAGE
kube-system   DaemonSet    cni-node          registry.example/cni/node:v3.30.2
kube-system   Deployment   cni-controllers   registry.example/cni/controller:v3.30.2
No resources found matching deprecated API metric value 1

Checkpoint: Every critical integration has an approved 1.36 disposition

kubectl get nodes -o name | xargs -n1 -I{} sh -c 'echo {}; kubectl get {} -o jsonpath="{.status.nodeInfo.containerRuntimeVersion}{\"\\n\"}"'

Continue whenRuntime, cgroup mode, CNI, CSI, ingress, webhooks, operators, and observability all have documented supported target versions and upgrade order.

Stop whenAny critical component lacks 1.36 support, uses a removed API, or has no rollback and test evidence.

If this step fails

Pods on an upgraded node cannot reach Services or external destinations.

Likely causeThe CNI, kube-proxy, kernel networking, nftables or iptables mode, MTU, or network policy is incompatible or was not upgraded in the required sequence.

Safe checks

kubectl -n kube-system get pods -o wide --field-selector spec.nodeName={{node}}
kubectl describe node {{node}}
kubectl get networkpolicy -A

ResolutionCordon the node again, preserve CNI and kube-proxy logs, follow the official CNI upgrade procedure, and validate cross-node, Service, DNS, and egress traffic before returning capacity.

The upgraded cluster reports API deprecation or rejected manifest errors.

Likely causeWorkloads, admission webhooks, operators, or stored objects still use APIs removed or changed in 1.36.

Safe checks

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
kubectl get apiservices
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations

ResolutionPause node rollout, migrate the affected API consumers or webhooks using release-note guidance, test again on a staging cluster, and resume only when required controllers reconcile cleanly.

Security notes

Admission and security agents are enforcement boundaries; never disable them during upgrade merely to allow incompatible resources.

Alternatives

Migrate incompatible workloads to a parallel 1.36 staging or blue-green cluster before changing production.

Stop conditions

Pause the cluster upgrade until every control-plane admission and networking dependency supports the target.

warning

Create verified etcd, configuration, and application recovery evidence

caution

Take an encrypted off-node etcd snapshot with the cluster's supported client method, copy kubeadm configuration and manifests securely, verify snapshot status, and confirm separate application restores.

Why this step matters

Kubeadm creates some local backups during upgrade, but an independently verified etcd snapshot, configuration copy, application backups, and restore rehearsal define recovery when a node or the entire control plane cannot converge.

What to understand

Use the etcd client and certificate paths appropriate to the installed stacked or external etcd topology. Snapshot one healthy member, verify status immediately, encrypt it, and copy it off the node.

Record member list and endpoint health, Kubernetes manifests, kubeadm ConfigMaps, certificates' expiry inventory, package versions, and runtime configuration. Restrict all copies because they confer high privilege.

Restore representative application data separately. Etcd contains Kubernetes objects, not the content of external databases and most persistent volumes.

System changes

Creates a protected backup directory, writes an etcd snapshot, and copies Kubernetes configuration on the selected control plane.

Syntax explained

--endpoints=https://127.0.0.1:2379: Targets the local stacked etcd member; external etcd requires its documented healthy endpoint set.
--cacert, --cert, --key: Authenticates the snapshot client with the cluster's etcd PKI without disabling TLS verification.
snapshot status --write-out=table: Checks that the saved snapshot has a readable hash, revision, key count, and size.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Etcd Snapshot{{etcdSnapshot}}Encrypted, off-node path for the verified pre-upgrade etcd snapshot.

sudo install -d -m 0700 /var/backups/kubernetes/2026-07-28 && sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save {{etcdSnapshot}} && sudo ETCDCTL_API=3 etcdctl snapshot status {{etcdSnapshot}} --write-out=table && sudo cp -a /etc/kubernetes /var/backups/kubernetes/2026-07-28/etc-kubernetes

Example output / evidence

Snapshot saved at /var/backups/kubernetes/2026-07-28/etcd-pre-1.36.db
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| a7319c2e | 48291731 |     184026 |      112 MB |
+----------+----------+------------+------------+

Checkpoint: Recovery evidence is readable and off-node

sudo ETCDCTL_API=3 etcdctl snapshot status {{etcdSnapshot}} --write-out=table && sudo stat -c '%a %U:%G %s %n' {{etcdSnapshot}}

Continue whenSnapshot status succeeds, permissions are restrictive, an encrypted off-node copy exists, and an isolated restore rehearsal is documented.

Stop whenEtcd health or quorum is uncertain, snapshot verification fails, backup remains only on the upgraded node, or application restore evidence is missing.

If this step fails

`kubeadm upgrade apply` starts but a static control-plane Pod does not return.

Likely causeThe new image cannot be pulled, the manifest is incompatible with local flags or mounts, the container runtime is unhealthy, a port or certificate fails, or etcd cannot form quorum.

Safe checks

sudo crictl ps -a
sudo journalctl -u kubelet -n 200 --no-pager
sudo ls -la /etc/kubernetes/tmp /etc/kubernetes/manifests

ResolutionKeep the node isolated, preserve kubeadm backup directories and kubelet/runtime logs, and retry the idempotent upgrade after correcting the verified cause. Restore manifests or etcd only through the documented recovery plan with quorum and snapshot evidence.

Security notes

An etcd snapshot can contain every Kubernetes Secret. Encrypt it, restrict it to recovery operators, and destroy expired copies under policy.

Alternatives

Use the platform's supported encrypted etcd snapshot mechanism when it provides equivalent integrity, ownership, off-node retention, and restore testing.

Stop conditions

Do not begin package changes without both cluster-state and application-data recovery evidence.

config

Select the official Kubernetes 1.36 package stream and exact patch

read-only

Configure the official per-minor pkgs.k8s.io 1.36 repository through host management, refresh metadata, list candidates, and record one exact kubeadm, kubelet, and kubectl package version.

Why this step matters

Kubernetes packages are published in per-minor pkgs.k8s.io streams. Explicit repository identity and exact patch selection prevent an operator from installing the wrong minor or a newly published patch that was not reviewed for this window.

What to understand

The legacy apt.kubernetes.io repository is frozen. Configure the 1.36 stream with a dedicated signed-by keyring through configuration management and preserve the previous 1.35 source for recovery analysis rather than editing ad hoc on every node.

Run apt-cache madison and policy before unholding anything. Record the complete package suffix because apt equality matching must use the repository's exact version.

Use the same approved target on kubeadm, kubelet, and kubectl. Kubeadm is upgraded first on each node; kubelet and kubectl follow only at the documented stage.

System changes

The shown command reads repository metadata; the displayed configuration block documents the reviewed 1.36 source which host management applies separately.

Syntax explained

apt-cache madison kubeadm: Lists repository versions without installing or unholding a package.
apt-cache policy kubeadm kubelet kubectl: Shows installed, candidate, and source priority for all three Kubernetes packages.
signed-by=/etc/apt/keyrings/...: Scopes package signature trust to the Kubernetes repository keyring rather than global apt trust.

File /etc/apt/sources.list.d/kubernetes.list

Configuration

deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /

Command

sudo apt-get update && apt-cache madison kubeadm | grep '1.36' && apt-cache policy kubeadm kubelet kubectl

Example output / evidence

   kubeadm | 1.36.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.36/deb  Packages
kubeadm:
  Installed: 1.35.7-1.1
  Candidate: 1.36.1-1.1
kubelet:
  Installed: 1.35.7-1.1
kubectl:
  Installed: 1.35.7-1.1

Checkpoint: One exact 1.36 package version is approved

apt-cache policy kubeadm kubelet kubectl | sed -n '/Installed:/p;/Candidate:/p'

Continue whenShows 1.35 installed and the same explicitly recorded 1.36 target candidate from the official repository for all packages.

Stop whenThe candidate is another minor, comes from an unapproved source, the signature fails, or no exact target has completed release and compatibility review.

If this step fails

The 1.36 kubeadm package is not listed by apt-cache.

Likely causeThe host still uses the frozen legacy repository, the pkgs.k8s.io repository points to another minor stream, or apt metadata and signing-key configuration are incomplete.

Safe checks

grep -R --line-number 'pkgs.k8s.io\|apt.kubernetes.io' /etc/apt/sources.list /etc/apt/sources.list.d
apt-cache policy kubeadm
sudo apt-get update

ResolutionConfigure the official `core:/stable:/v1.36/deb/` repository through the approved host-management source, verify its signed-by keyring, update metadata, and select one exact patch. Do not remove the package hold until the candidate is explicit.

Security notes

Do not fetch repository keys through an unverified mirror or disable apt signature checks.

Alternatives

Mirror the official signed packages internally when required, preserving provenance, exact versions, and repository isolation.

Stop conditions

Keep all packages held until repository provenance and exact version equality are proven.

warning

Upgrade kubeadm on the primary control plane and run the plan

caution

Unhold only kubeadm, install the exact target package on the selected primary control plane, restore its hold, verify the binary, and require a clean upgrade plan before applying.

Why this step matters

The first control-plane kubeadm binary performs version and cluster preflight before any static manifest changes. Upgrading only this held package limits the initial change and makes the plan an explicit go or no-go gate.

What to understand

Run on the selected node that has `/etc/kubernetes/admin.conf`. Verify hostname, package version, and active API context immediately before the install.

Restore the package hold in the same transaction. A later unattended package update must not move kubeadm, kubelet, or kubectl independently.

Read every warning and component configuration state in `kubeadm upgrade plan`. The displayed target must match the approved patch exactly and must not propose a skipped minor.

System changes

Temporarily removes the kubeadm package hold, installs the exact 1.36 kubeadm binary on the primary control plane, and restores the hold; cluster manifests are not yet changed.

Syntax explained

kubeadm='{{targetKubernetesVersion}}': Pins apt to the exact reviewed package instead of installing an unbounded candidate.
apt-mark hold kubeadm: Prevents unrelated automatic updates from changing kubeadm after the controlled install.
kubeadm upgrade plan: Performs upgradeability and version-skew checks and prints the component transition without applying it.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Target Kubernetes Version{{targetKubernetesVersion}}Exact 1.36 package version returned by the approved package repository, including package suffix.

sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm='{{targetKubernetesVersion}}' && sudo apt-mark hold kubeadm && kubeadm version && sudo kubeadm upgrade plan

Example output / evidence

kubeadm set on hold.
kubeadm version: &version.Info{Major:"1", Minor:"36", GitVersion:"v1.36.1"}
COMPONENT                 NODE    CURRENT   TARGET
kube-apiserver                    v1.35.7   v1.36.1
kube-controller-manager           v1.35.7   v1.36.1
kube-scheduler                     v1.35.7   v1.36.1
You can now apply the upgrade by executing: kubeadm upgrade apply v1.36.1

Checkpoint: The kubeadm plan exactly matches the approved transition

kubeadm version -o short && sudo kubeadm upgrade plan

Continue whenLocal kubeadm is the exact 1.36 patch and the plan accepts the healthy 1.35 cluster with the same target.

Stop whenThe plan reports health, skew, configuration, image, certificate, or target mismatch warnings not explicitly resolved by the change owner.

If this step fails

`kubeadm upgrade plan` refuses the target and reports that the version is too far ahead.

Likely causeThe cluster is older than 1.35, the local kubeadm package is not 1.36, or an operator attempted to skip one or more minor versions.

Safe checks

kubectl version
kubeadm version
kubectl get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion

The upgrade plan reports an unhealthy control plane or unreachable API server.

Likely causeA static control-plane Pod, etcd member, load balancer path, certificate, DNS record, or node is already unhealthy before the maintenance starts.

Safe checks

kubectl get --raw='/readyz?verbose'
kubectl -n kube-system get pods -o wide
kubectl get nodes

Security notes

Run kubeadm with sudo only on the verified primary control plane; its admin.conf and PKI are cluster-admin material.

Alternatives

Use a reviewed UpgradeConfiguration file when the cluster needs supported explicit kubeadm options; do not use upgrade to reconfigure unrelated settings.

Stop conditions

Never add `--force` merely to turn a failed plan into an apply command.

warning

Apply the target on exactly one primary control plane

danger

Run kubeadm upgrade apply for the exact version, monitor static Pods and etcd, and accept the first control plane before another instance changes.

Why this step matters

Kubeadm apply rewrites static control-plane manifests and may upgrade etcd, CoreDNS, kube-proxy, configuration, and certificates. It is the highest-impact gate and must run on one primary while all other control-plane instances remain healthy.

What to understand

Watch API load-balancer behavior, etcd member health, kubelet logs, runtime containers, `/etc/kubernetes/manifests`, and kube-system Pods from an independent session while apply runs.

Expect control-plane containers to restart. Client requests may stall briefly during local etcd restart, but quorum and healthy peer API servers must remain available according to the architecture.

Do not start another control-plane upgrade until verbose readyz, etcd health, static Pod versions, scheduler and controller reconciliation, and critical API clients are accepted.

System changes

Rewrites the primary node's static control-plane manifests, advances cluster component configuration to 1.36, may upgrade local etcd and add-ons, and writes kubeadm backup directories.

Syntax explained

upgrade apply {{targetKubernetesRelease}}: Declares the exact reviewed Kubernetes target; replace the example literal only with the approved target derived from the exact package.
--yes: Confirms the already reviewed plan in noninteractive execution; it must not replace human plan approval.
readyz?verbose: Tests individual API readiness dependencies immediately after static Pod replacement.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Target Kubernetes Release{{targetKubernetesRelease}}Exact semantic Kubernetes release reported by kubeadm upgrade plan for the approved 1.36 package.

sudo kubeadm upgrade apply {{targetKubernetesRelease}} --yes && kubectl get --raw='/readyz?verbose' && kubectl -n kube-system get pods -o wide

Example output / evidence

[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.36.1". Enjoy!
[+]ping ok
[+]log ok
[+]etcd ok
readyz check passed
NAME                             READY   STATUS    RESTARTS   AGE
kube-apiserver-cp-01             1/1     Running   0          2m
kube-controller-manager-cp-01    1/1     Running   0          2m
kube-scheduler-cp-01             1/1     Running   0          2m

Checkpoint: First control plane and etcd are healthy on 1.36

kubectl get --raw='/readyz?verbose' && kubectl -n kube-system get pod -l tier=control-plane -o wide

Continue whenReadyz passes, the primary static Pods run the target version, etcd has quorum, and peer control planes continue serving.

Stop whenApply fails, API readiness or etcd quorum is lost, a static Pod loops, the load balancer cannot isolate restart, or any unexplained control-plane error appears.

If this step fails

`kubeadm upgrade apply` starts but a static control-plane Pod does not return.

Likely causeThe new image cannot be pulled, the manifest is incompatible with local flags or mounts, the container runtime is unhealthy, a port or certificate fails, or etcd cannot form quorum.

Safe checks

sudo crictl ps -a
sudo journalctl -u kubelet -n 200 --no-pager
sudo ls -la /etc/kubernetes/tmp /etc/kubernetes/manifests

A control-plane load balancer keeps sending traffic to an instance while its API server restarts.

Likely causeHealth checks are too shallow or slow, connection draining is absent, or maintenance did not account for the external load balancer.

Safe checks

kubectl get --raw='/readyz?verbose'
ss -lntp | grep ':6443'
curl --fail --silent --show-error https://{{controlPlaneEndpoint}}:6443/livez

ResolutionCorrect health-check behavior and confirm the remaining control planes carry traffic before retrying. Upgrade one instance at a time; do not stop a second API server while quorum or client capacity is uncertain.

Security notes

Preserve `/etc/kubernetes/tmp` backups with root-only access; they can contain etcd and manifest material.

Alternatives

For a single-control-plane cluster, schedule explicit API downtime and prove console recovery because another API instance cannot carry traffic.

Stop conditions

Do not touch another control-plane node until first-node API, etcd, scheduler, controller, and external client checks pass.

warning

Drain and upgrade kubelet plus kubectl on the primary node

danger

Evict ordinary workloads through the policy-aware drain, install exact kubelet and kubectl packages, restart kubelet, and leave the node cordoned until it is Ready and system components are healthy.

Why this step matters

Kubernetes requires a drain before a minor kubelet upgrade. Installing the matching kubelet and kubectl only after the control plane converges keeps component order supported, while cordon prevents work from landing before node acceptance.

What to understand

Drain uses the eviction API and respects PDBs. Review DaemonSets, mirror Pods, unmanaged Pods, and emptyDir data rather than adding broad flags until it succeeds.

The kubelet may remain briefly inactive while package configuration changes; systemd restart and journal evidence must show a clean target startup.

Keep the control plane cordoned until its node object is Ready, target kubelet version is visible, CNI and kube-proxy DaemonSets are healthy, and hosted ordinary workloads have rescheduled.

System changes

Cordons and evicts ordinary workloads from the primary node, installs exact kubelet and kubectl packages, restores package holds, and restarts kubelet.

Syntax explained

drain --ignore-daemonsets: Uses policy-aware eviction for ordinary workloads while acknowledging DaemonSet Pods that are expected to remain.
kubelet='{{targetKubernetesVersion}}' kubectl='{{targetKubernetesVersion}}': Installs both node agent and client at the exact approved patch.
systemctl daemon-reload && restart kubelet: Reloads package-provided unit configuration and starts the upgraded node agent.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

Primary Control Plane{{primaryControlPlane}}First control-plane node containing `/etc/kubernetes/admin.conf` and selected for `kubeadm upgrade apply`.Target Kubernetes Version{{targetKubernetesVersion}}Exact 1.36 package version returned by the approved package repository, including package suffix.

kubectl drain {{primaryControlPlane}} --ignore-daemonsets && sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet='{{targetKubernetesVersion}}' kubectl='{{targetKubernetesVersion}}' && sudo apt-mark hold kubelet kubectl && sudo systemctl daemon-reload && sudo systemctl restart kubelet && systemctl is-active kubelet

Example output / evidence

node/cp-01 cordoned
pod/coredns-6f8c7b5f47-2qk8m evicted
node/cp-01 drained
kubelet set on hold.
kubectl set on hold.
active

Checkpoint: Primary kubelet is healthy but node remains controlled

systemctl is-active kubelet && kubectl get node {{primaryControlPlane}} -o custom-columns=NAME:.metadata.name,READY:.status.conditions[-1].type,KUBELET:.status.nodeInfo.kubeletVersion,UNSCHEDULABLE:.spec.unschedulable

Continue whenKubelet is active, the node is Ready on the exact 1.36 patch, and remains unschedulable until the add-on gate passes.

Stop whenDrain is blocked, local data ownership is unknown, kubelet fails, node remains NotReady, or cgroup/runtime/CNI errors appear.

If this step fails

Draining a node waits indefinitely because eviction is blocked by a PodDisruptionBudget.

Likely causeThe protected workload lacks enough healthy replicas, capacity, or topology to satisfy its disruption budget while one node is unavailable.

Safe checks

kubectl get poddisruptionbudget -A
kubectl get pods -A --field-selector spec.nodeName={{node}} -o wide
kubectl describe node {{node}}

Draining reports unmanaged Pods or local data.

Likely causeA static, naked, or operator-specific Pod lacks a normal controller, or a workload uses emptyDir or host-local state that eviction can discard.

Safe checks

kubectl get pods -A --field-selector spec.nodeName={{node}} -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind,EMPTYDIR:.spec.volumes[*].emptyDir
kubectl get daemonset -A

ResolutionIdentify ownership and data semantics before proceeding. Migrate or back up local state, create a controller where appropriate, and use only the narrow drain options explicitly approved for that workload.

A node stays NotReady after kubelet and kubectl are upgraded.

Likely causeThe kubelet failed to restart, cgroup v1 is still in use, the runtime or CNI is incompatible, certificates or kubeconfig are invalid, or node networking cannot reach the API.

Safe checks

sudo systemctl status kubelet --no-pager
sudo journalctl -u kubelet -n 200 --no-pager
kubectl describe node {{node}}

ResolutionKeep the node cordoned. Resolve the first kubelet or runtime error, including Kubernetes 1.36 cgroup v2 requirements, and require Ready plus healthy system DaemonSets before uncordoning.

Security notes

Do not use `--disable-eviction`, `--force`, or `--delete-emptydir-data` without an application-specific, owner-approved exception.

Alternatives

Temporarily add capacity so eviction can satisfy PDB and topology requirements instead of bypassing them.

Stop conditions

Leave the node cordoned whenever kubelet, runtime, CNI, DNS, or system workload evidence is incomplete.

decision

Upgrade the CNI as required and accept cluster add-ons

caution

Follow the CNI's official Kubernetes 1.36 upgrade procedure, then require healthy CNI, CoreDNS, kube-proxy, CSI, API aggregation, and admission webhooks before continuing.

Why this step matters

CNI, CoreDNS, kube-proxy, CSI, aggregated APIs, and webhooks make the control plane usable for workloads. Their health is a separate acceptance gate because kubeadm does not universally manage third-party add-on compatibility or sequence.

What to understand

Follow the installed CNI's official 1.36 procedure at the documented point. Confirm desired and ready DaemonSet replicas across upgraded and old nodes and exercise cross-node Pod connectivity.

CoreDNS should have ready replicas and EndpointSlices, while kube-proxy or its replacement should program Service connectivity. Test both cluster DNS and the Kubernetes Service address from an approved probe Pod.

Check APIService availability and webhook backends because a green core API can still reject workload writes when extension services are unhealthy.

System changes

The command reads add-on and extension health; any CNI upgrade occurs separately through its approved manifest or package procedure.

Syntax explained

get daemonset,deployment,pods -o wide: Compares desired, ready, restart, image, and node placement for system networking and DNS components.
get apiservices: Shows whether aggregated API backends such as resource metrics remain available.

Command

kubectl -n kube-system get daemonset,deployment,pods -o wide && kubectl get apiservices && kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations

Example output / evidence

NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
DaemonSet/cni-node           3         3         3       3            3
DaemonSet/kube-proxy         3         3         3       3            3
Deployment/coredns           2/2       2         2       2            2
NAME                         SERVICE                      AVAILABLE
v1beta1.metrics.k8s.io       metrics/metrics-server       True

Checkpoint: Cluster networking, DNS, storage, and extensions are accepted

kubectl -n kube-system get daemonset,deployment -o custom-columns=KIND:.kind,NAME:.metadata.name,DESIRED:.status.desiredNumberScheduled,READY_DS:.status.numberReady,READY_DEPLOY:.status.readyReplicas

Continue whenEvery required system controller has its intended ready count and synthetic DNS, Service, cross-node, egress, and storage checks pass.

Stop whenAny required DaemonSet, CoreDNS, kube-proxy, CSI, CNI, APIService, or webhook is unavailable or incompatible.

If this step fails

The API is available after the first control-plane upgrade, but CoreDNS or kube-proxy is degraded.

Likely causeAdd-on upgrade waits for all control-plane instances, a new manifest cannot schedule, the image is unavailable, or the CNI and target Kubernetes release are incompatible.

Safe checks

kubectl -n kube-system get deployment,daemonset,pods -o wide
kubectl -n kube-system get events --sort-by=.lastTimestamp
kubectl get endpointslice -n kube-system -l k8s-app=kube-dns

ResolutionPause before upgrading workers, complete all control-plane nodes in the documented order, follow the CNI vendor's compatible upgrade path, and restore DNS and proxy health before continuing.

Pods on an upgraded node cannot reach Services or external destinations.

Likely causeThe CNI, kube-proxy, kernel networking, nftables or iptables mode, MTU, or network policy is incompatible or was not upgraded in the required sequence.

Safe checks

kubectl -n kube-system get pods -o wide --field-selector spec.nodeName={{node}}
kubectl describe node {{node}}
kubectl get networkpolicy -A

ResolutionCordon the node again, preserve CNI and kube-proxy logs, follow the official CNI upgrade procedure, and validate cross-node, Service, DNS, and egress traffic before returning capacity.

Security notes

Do not disable network policy, admission, or certificate verification to make an incompatible add-on appear healthy.

Alternatives

Roll back the add-on through its officially supported procedure while keeping the accepted Kubernetes control plane stable.

Stop conditions

Never proceed to additional control planes or workers while cluster networking, DNS, storage, or admission is degraded.

warning

Upgrade each additional control plane sequentially

danger

On one additional control plane at a time, install exact kubeadm, run kubeadm upgrade node, drain, install kubelet and kubectl, restart, verify, and uncordon before selecting the next instance.

Why this step matters

Additional control planes use `kubeadm upgrade node`, not another cluster-wide apply. Sequential processing preserves API capacity and etcd quorum, and it lets one proven peer carry clients while the selected node's static Pods restart.

What to understand

Repeat hostname, package, backup, endpoint, and quorum checks on every instance. Do not assume identical nodes have identical mounts, runtime configuration, certificates, or repository state.

Install exact kubeadm first, run node phase, then drain and install exact kubelet plus kubectl. Keep the selected instance removed from load-balancer traffic or rely on a verified readiness health check during restart.

Accept API readyz through both the local and stable endpoint, etcd member health, static Pod versions, kubelet Ready, and system DaemonSets before uncordoning and selecting the next node.

System changes

Updates one additional control plane's kubeadm, static manifests, kubelet, kubectl, and running workloads while preserving other instances.

Syntax explained

kubeadm upgrade node: Fetches cluster configuration and updates this additional control plane or worker without repeating cluster-wide apply.
drain {{node}} --ignore-daemonsets: Evicts ordinary workloads through policy before the minor kubelet change.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

Target Kubernetes Version{{targetKubernetesVersion}}Exact 1.36 package version returned by the approved package repository, including package suffix.Node{{node}}Current additional control-plane or worker node processed alone or within the approved bounded batch.

sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm='{{targetKubernetesVersion}}' && sudo apt-mark hold kubeadm && sudo kubeadm upgrade node && kubectl drain {{node}} --ignore-daemonsets && sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet='{{targetKubernetesVersion}}' kubectl='{{targetKubernetesVersion}}' && sudo apt-mark hold kubelet kubectl && sudo systemctl daemon-reload && sudo systemctl restart kubelet

Example output / evidence

[upgrade] Reading configuration from the cluster...
[upgrade/staticpods] Writing new Static Pod manifests
[upgrade] The configuration for this node was successfully updated!
node/cp-02 cordoned
node/cp-02 drained
kubelet set on hold.
kubectl set on hold.

Checkpoint: One additional control plane is fully accepted

kubectl get node {{node}} -o custom-columns=NAME:.metadata.name,READY:.status.conditions[-1].type,KUBELET:.status.nodeInfo.kubeletVersion && kubectl get --raw='/readyz?verbose'

Continue whenThe selected node is Ready on 1.36, its API and etcd member are healthy, and all previously accepted peers remain healthy.

Stop whenQuorum margin is uncertain, another control plane is unhealthy, the selected node fails any gate, or load-balancer behavior cannot protect clients.

If this step fails

`kubeadm upgrade apply` starts but a static control-plane Pod does not return.

Likely causeThe new image cannot be pulled, the manifest is incompatible with local flags or mounts, the container runtime is unhealthy, a port or certificate fails, or etcd cannot form quorum.

Safe checks

sudo crictl ps -a
sudo journalctl -u kubelet -n 200 --no-pager
sudo ls -la /etc/kubernetes/tmp /etc/kubernetes/manifests

A control-plane load balancer keeps sending traffic to an instance while its API server restarts.

Likely causeHealth checks are too shallow or slow, connection draining is absent, or maintenance did not account for the external load balancer.

Safe checks

kubectl get --raw='/readyz?verbose'
ss -lntp | grep ':6443'
curl --fail --silent --show-error https://{{controlPlaneEndpoint}}:6443/livez

Security notes

Keep node PKI and admin.conf local and protected; do not copy one control plane's identity to another.

Alternatives

Rebuild a failed additional control plane from the supported kubeadm join workflow when node repair is riskier than replacement and quorum remains healthy.

Stop conditions

Only one control-plane instance may be in maintenance at a time unless the architecture and quorum plan explicitly prove a larger safe batch.

warning

Drain and upgrade workers in bounded capacity-safe batches

danger

Process at most the approved batch size: verify capacity, upgrade kubeadm, drain through eviction, run kubeadm upgrade node, install kubelet and kubectl, restart, require Ready and system networking, then uncordon.

Why this step matters

Workers carry customer load, so batch size is an availability decision rather than a convenience value. Each node must drain through policy, receive component changes in supported order, and prove networking, storage, and runtime health before capacity returns.

What to understand

Recalculate spare capacity and PDB allowance before every batch because rescheduling the previous node changes resource and topology distribution.

Upgrade kubeadm, drain, run node phase, then upgrade kubelet and kubectl. Keep the node cordoned through runtime, CNI, CSI, kube-proxy, DNS, and application probes.

A node that becomes Ready can still have broken Pod networking, image pulls, volume attachment, or device plugins. Schedule an approved test workload and exercise those node capabilities before uncordon.

System changes

Cordons and evicts workloads, updates kubeadm node configuration and packages, and restarts kubelet on each selected worker.

Syntax explained

{{maintenanceBatch}}: Operational limit selected from real capacity and disruption evidence; the example recommends one node at a time.
kubeadm upgrade node: Downloads the target kubelet configuration and updates this worker's local kubeadm-managed state.
apt-mark hold: Returns every Kubernetes package to controlled-upgrade policy after the exact install.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm='{{targetKubernetesVersion}}' && sudo apt-mark hold kubeadm && kubectl drain {{node}} --ignore-daemonsets && sudo kubeadm upgrade node && sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet='{{targetKubernetesVersion}}' kubectl='{{targetKubernetesVersion}}' && sudo apt-mark hold kubelet kubectl && sudo systemctl daemon-reload && sudo systemctl restart kubelet

Example output / evidence

node/worker-03 cordoned
evicting pod payments/api-7d7f9f6c8b-g2m7n
pod/api-7d7f9f6c8b-g2m7n evicted
node/worker-03 drained
[upgrade] The configuration for this node was successfully updated!
active

Checkpoint: Current worker batch is healthy and isolated for acceptance

systemctl is-active kubelet && kubectl get node {{node}} -o custom-columns=NAME:.metadata.name,READY:.status.conditions[-1].type,KUBELET:.status.nodeInfo.kubeletVersion,UNSCHEDULABLE:.spec.unschedulable

Continue whenKubelet is active, the selected worker is Ready on 1.36, and remains cordoned until node-level synthetic tests pass.

Stop whenEviction violates availability, kubelet or runtime fails, CNI or CSI is unhealthy, or the node cannot run the approved test workload.

If this step fails

Draining a node waits indefinitely because eviction is blocked by a PodDisruptionBudget.

Likely causeThe protected workload lacks enough healthy replicas, capacity, or topology to satisfy its disruption budget while one node is unavailable.

Safe checks

kubectl get poddisruptionbudget -A
kubectl get pods -A --field-selector spec.nodeName={{node}} -o wide
kubectl describe node {{node}}

Draining reports unmanaged Pods or local data.

Likely causeA static, naked, or operator-specific Pod lacks a normal controller, or a workload uses emptyDir or host-local state that eviction can discard.

Safe checks

kubectl get pods -A --field-selector spec.nodeName={{node}} -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind,EMPTYDIR:.spec.volumes[*].emptyDir
kubectl get daemonset -A

A node stays NotReady after kubelet and kubectl are upgraded.

Likely causeThe kubelet failed to restart, cgroup v1 is still in use, the runtime or CNI is incompatible, certificates or kubeconfig are invalid, or node networking cannot reach the API.

Safe checks

sudo systemctl status kubelet --no-pager
sudo journalctl -u kubelet -n 200 --no-pager
kubectl describe node {{node}}

Pods on an upgraded node cannot reach Services or external destinations.

Likely causeThe CNI, kube-proxy, kernel networking, nftables or iptables mode, MTU, or network policy is incompatible or was not upgraded in the required sequence.

Safe checks

kubectl -n kube-system get pods -o wide --field-selector spec.nodeName={{node}}
kubectl describe node {{node}}
kubectl get networkpolicy -A

ResolutionCordon the node again, preserve CNI and kube-proxy logs, follow the official CNI upgrade procedure, and validate cross-node, Service, DNS, and egress traffic before returning capacity.

Security notes

Never execute an unbounded parallel SSH loop across all workers; concurrency must remain visible and policy-limited.

Alternatives

Replace immutable workers from a tested 1.36 node image and drain old nodes through the same application and capacity gates.

Stop conditions

Do not begin the next batch until all nodes in the current batch are accepted or safely removed from service.

verification

Uncordon only accepted nodes and complete cluster-wide verification

caution

For each node require Ready, target kubelet, healthy DaemonSets, Pod and Service networking, storage, DNS, and workload synthetic checks; then uncordon and finish with cluster-wide acceptance.

Why this step matters

Uncordon returns a node to production scheduling and is therefore a state-changing acceptance decision. Cluster-wide completion requires more than matching versions: workloads, networking, DNS, storage, admission, data, monitoring, and disaster recovery must remain sound.

What to understand

Before uncordon, run node-local image pull, Pod DNS, ClusterIP, cross-node, egress, volume mount, and device-plugin checks appropriate to the node role. Remove the probe workload after evidence is retained.

After all nodes, compare the complete version and health inventory with the baseline. Investigate every new restart, Pending Pod, warning event, API error, webhook failure, and latency or error-rate change.

Keep snapshots and kubeadm backup directories through the acceptance window. Clean them only after recovery owners confirm retention and secure off-node copies.

System changes

Marks the accepted node schedulable again; subsequent commands read cluster-wide final health.

Syntax explained

kubectl uncordon {{node}}: Clears the unschedulable flag only after every node acceptance criterion has passed.
custom-columns=...KUBELET,UNSCHEDULABLE: Shows final version, Ready state, and scheduling state together for every node.
readyz?verbose: Rechecks the complete API readiness chain after all maintenance batches.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Node{{node}}Current additional control-plane or worker node processed alone or within the approved bounded batch.

kubectl uncordon {{node}} && kubectl get nodes -o custom-columns=NAME:.metadata.name,STATUS:.status.conditions[-1].type,KUBELET:.status.nodeInfo.kubeletVersion,UNSCHEDULABLE:.spec.unschedulable && kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded && kubectl get --raw='/readyz?verbose'

Example output / evidence

node/worker-03 uncordoned
NAME       STATUS   KUBELET   UNSCHEDULABLE
cp-01      Ready    v1.36.1   <none>
cp-02      Ready    v1.36.1   <none>
worker-01  Ready    v1.36.1   <none>
worker-03  Ready    v1.36.1   <none>
No resources found
readyz check passed

Checkpoint: The entire 1.36 cluster and applications are accepted

kubectl get nodes && kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded && kubectl get --raw='/readyz?verbose'

Continue whenAll nodes are Ready on the target, no unexplained abnormal Pod phase exists, API readiness passes, and external application tests meet baseline.

Stop whenAny node, add-on, workload, network, storage, admission, data-integrity, monitoring, or recovery signal remains unexplained.

If this step fails

The API is available after the first control-plane upgrade, but CoreDNS or kube-proxy is degraded.

Likely causeAdd-on upgrade waits for all control-plane instances, a new manifest cannot schedule, the image is unavailable, or the CNI and target Kubernetes release are incompatible.

Safe checks

kubectl -n kube-system get deployment,daemonset,pods -o wide
kubectl -n kube-system get events --sort-by=.lastTimestamp
kubectl get endpointslice -n kube-system -l k8s-app=kube-dns

A node stays NotReady after kubelet and kubectl are upgraded.

Likely causeThe kubelet failed to restart, cgroup v1 is still in use, the runtime or CNI is incompatible, certificates or kubeconfig are invalid, or node networking cannot reach the API.

Safe checks

sudo systemctl status kubelet --no-pager
sudo journalctl -u kubelet -n 200 --no-pager
kubectl describe node {{node}}

Pods on an upgraded node cannot reach Services or external destinations.

Likely causeThe CNI, kube-proxy, kernel networking, nftables or iptables mode, MTU, or network policy is incompatible or was not upgraded in the required sequence.

Safe checks

kubectl -n kube-system get pods -o wide --field-selector spec.nodeName={{node}}
kubectl describe node {{node}}
kubectl get networkpolicy -A

ResolutionCordon the node again, preserve CNI and kube-proxy logs, follow the official CNI upgrade procedure, and validate cross-node, Service, DNS, and egress traffic before returning capacity.

The upgraded cluster reports API deprecation or rejected manifest errors.

Likely causeWorkloads, admission webhooks, operators, or stored objects still use APIs removed or changed in 1.36.

Safe checks

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
kubectl get apiservices
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations

An operator asks to downgrade all components after the new control plane is running.

Likely causeThe upgrade was treated as a package change, but control-plane state, static manifests, configuration, storage schema, and add-ons have already advanced.

Safe checks

kubectl version
sudo ls -1dt /etc/kubernetes/tmp/kubeadm-backup-* | head
sudo find /var/backups/kubernetes -maxdepth 2 -type f -name '*.db' -o -name '*.snapshot'

ResolutionDo not attempt an improvised in-place minor downgrade. Stabilize or rerun the idempotent upgrade at the declared version. For an unrecoverable control plane, use the rehearsed disaster-recovery plan with verified etcd and application backups, then rebuild at a supported version.

Security notes

Do not delete sensitive recovery material merely to tidy the node; follow encrypted retention and destruction policy.

Alternatives

Keep a repaired but not fully accepted node cordoned and restore capacity elsewhere while deeper diagnostics continue.

Stop conditions

Do not declare completion until control-plane, node, add-on, workload, data, and external consumer evidence all pass.

Finish line

Verification checklist

Cluster and node versions

kubectl version && kubectl get nodes -o custom-columns=NAME:.metadata.name,READY:.status.conditions[-1].type,KUBELET:.status.nodeInfo.kubeletVersion,UNSCHEDULABLE:.spec.unschedulable

The API server and every intended node report the exact approved 1.36 patch, Ready, and schedulable after acceptance.

Control-plane readinesskubectl get --raw='/readyz?verbose'Every readiness check, including etcd, returns ok and the final line reports readiness passed.

System add-onskubectl -n kube-system get daemonset,deployment,pods -o wideCNI, kube-proxy, CoreDNS, CSI, metrics, and other required system workloads have desired healthy replicas with no new restart loop.

Workload disruptionkubectl get pods -A --field-selector=status.phase!=Running,status.phase!=SucceededReturns no unexplained Pending, Failed, or Unknown application Pods after the final batch.

Disaster-recovery evidencesudo ETCDCTL_API=3 etcdctl snapshot status {{etcdSnapshot}} --write-out=tableThe retained pre-upgrade snapshot has a readable hash, revision, key count, size, encrypted off-node copy, and documented restore rehearsal.

Recovery guidance

Common problems and safe checks

`kubeadm upgrade plan` refuses the target and reports that the version is too far ahead.

Likely causeThe cluster is older than 1.35, the local kubeadm package is not 1.36, or an operator attempted to skip one or more minor versions.

Safe checks

kubectl version
kubeadm version
kubectl get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion

The 1.36 kubeadm package is not listed by apt-cache.

Likely causeThe host still uses the frozen legacy repository, the pkgs.k8s.io repository points to another minor stream, or apt metadata and signing-key configuration are incomplete.

Safe checks

grep -R --line-number 'pkgs.k8s.io\|apt.kubernetes.io' /etc/apt/sources.list /etc/apt/sources.list.d
apt-cache policy kubeadm
sudo apt-get update

The upgrade plan reports an unhealthy control plane or unreachable API server.

Likely causeA static control-plane Pod, etcd member, load balancer path, certificate, DNS record, or node is already unhealthy before the maintenance starts.

Safe checks

kubectl get --raw='/readyz?verbose'
kubectl -n kube-system get pods -o wide
kubectl get nodes

Draining a node waits indefinitely because eviction is blocked by a PodDisruptionBudget.

Likely causeThe protected workload lacks enough healthy replicas, capacity, or topology to satisfy its disruption budget while one node is unavailable.

Safe checks

kubectl get poddisruptionbudget -A
kubectl get pods -A --field-selector spec.nodeName={{node}} -o wide
kubectl describe node {{node}}

Draining reports unmanaged Pods or local data.

Likely causeA static, naked, or operator-specific Pod lacks a normal controller, or a workload uses emptyDir or host-local state that eviction can discard.

Safe checks

kubectl get pods -A --field-selector spec.nodeName={{node}} -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.ownerReferences[0].kind,EMPTYDIR:.spec.volumes[*].emptyDir
kubectl get daemonset -A

`kubeadm upgrade apply` starts but a static control-plane Pod does not return.

Likely causeThe new image cannot be pulled, the manifest is incompatible with local flags or mounts, the container runtime is unhealthy, a port or certificate fails, or etcd cannot form quorum.

Safe checks

sudo crictl ps -a
sudo journalctl -u kubelet -n 200 --no-pager
sudo ls -la /etc/kubernetes/tmp /etc/kubernetes/manifests

The API is available after the first control-plane upgrade, but CoreDNS or kube-proxy is degraded.

Likely causeAdd-on upgrade waits for all control-plane instances, a new manifest cannot schedule, the image is unavailable, or the CNI and target Kubernetes release are incompatible.

Safe checks

kubectl -n kube-system get deployment,daemonset,pods -o wide
kubectl -n kube-system get events --sort-by=.lastTimestamp
kubectl get endpointslice -n kube-system -l k8s-app=kube-dns

A node stays NotReady after kubelet and kubectl are upgraded.

Likely causeThe kubelet failed to restart, cgroup v1 is still in use, the runtime or CNI is incompatible, certificates or kubeconfig are invalid, or node networking cannot reach the API.

Safe checks

sudo systemctl status kubelet --no-pager
sudo journalctl -u kubelet -n 200 --no-pager
kubectl describe node {{node}}

Pods on an upgraded node cannot reach Services or external destinations.

Likely causeThe CNI, kube-proxy, kernel networking, nftables or iptables mode, MTU, or network policy is incompatible or was not upgraded in the required sequence.

Safe checks

kubectl -n kube-system get pods -o wide --field-selector spec.nodeName={{node}}
kubectl describe node {{node}}
kubectl get networkpolicy -A

ResolutionCordon the node again, preserve CNI and kube-proxy logs, follow the official CNI upgrade procedure, and validate cross-node, Service, DNS, and egress traffic before returning capacity.

A control-plane load balancer keeps sending traffic to an instance while its API server restarts.

Likely causeHealth checks are too shallow or slow, connection draining is absent, or maintenance did not account for the external load balancer.

Safe checks

kubectl get --raw='/readyz?verbose'
ss -lntp | grep ':6443'
curl --fail --silent --show-error https://{{controlPlaneEndpoint}}:6443/livez

The upgraded cluster reports API deprecation or rejected manifest errors.

Likely causeWorkloads, admission webhooks, operators, or stored objects still use APIs removed or changed in 1.36.

Safe checks

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
kubectl get apiservices
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations

An operator asks to downgrade all components after the new control plane is running.

Likely causeThe upgrade was treated as a package change, but control-plane state, static manifests, configuration, storage schema, and add-ons have already advanced.

Safe checks

kubectl version
sudo ls -1dt /etc/kubernetes/tmp/kubeadm-backup-* | head
sudo find /var/backups/kubernetes -maxdepth 2 -type f -name '*.db' -o -name '*.snapshot'

After the procedure

Alternatives and next steps

Consider these alternatives

Use a tested blue-green cluster migration when the current minor is far behind, the control plane cannot tolerate rolling maintenance, or platform architecture needs substantial change.
Use the managed Kubernetes provider's documented upgrade mechanism rather than kubeadm commands when the provider owns the control plane.
Upgrade one patch within 1.35 first when the cluster is not on the latest patch and the 1.36 release notes require fixes or migrations introduced there.
For large worker fleets, use an automation system that enforces the same drain, health, bounded-batch, and rollback gates instead of running ad hoc parallel shell loops.

Operate it safely

Schedule prompt patch upgrades within 1.36 and repeat release-note, add-on, backup, drain, and acceptance discipline for every supported transition.
Automate a preflight report that inventories versions, deprecated APIs, PDB coverage, headroom, CNI and CSI versions, webhook health, cgroup mode, and backup freshness.
Rehearse etcd and application restore on an isolated cluster and record recovery time plus responsible operators.
Replace mutable package globs in automation with an approved exact-version variable and a per-minor pkgs.k8s.io repository policy.
Monitor certificate expiry, etcd latency, API readiness, scheduler and controller errors, DNS, CNI health, node Ready transitions, and workload disruption during future maintenance.

Reference

Frequently asked questions

Can kubeadm upgrade from 1.34 directly to 1.36?

No. Skipping minor versions is unsupported. Upgrade 1.34 to the latest supported 1.35 patch using that version's official guide, accept the cluster, and then follow the 1.35 to 1.36 procedure.

Does an etcd snapshot back up my application databases?

No. It backs up Kubernetes API state stored in etcd. External databases, object stores, and persistent-volume contents require application-consistent backups and independent restore tests.

Can I downgrade packages if 1.36 fails?

Do not improvise an in-place minor downgrade. Correct and rerun the idempotent kubeadm upgrade at the declared version, or invoke the rehearsed disaster-recovery plan using verified snapshots and backups.

Why drain control-plane nodes if static API Pods continue running?

A minor kubelet upgrade requires drain, and control planes may host CoreDNS or other ordinary critical workloads. Drain also prevents new workloads from landing during node maintenance, but it does not stop static control-plane Pods.

Recovery

Rollback

Kubernetes minor upgrade is not an ordinary package downgrade. Stop advancement, keep unverified nodes cordoned, correct and rerun the idempotent declared upgrade, or invoke the rehearsed disaster-recovery rebuild when the control plane cannot be stabilized.

Stop automation and do not touch another node. Preserve `/etc/kubernetes/tmp/kubeadm-backup-*`, kubelet and runtime logs, manifests, etcd health, exact package versions, and API readiness output.
If a worker or additional control-plane node fails, keep it cordoned and restore service capacity on already accepted nodes while diagnosing the node locally.
If kubeadm apply failed, correct the verified image, runtime, manifest, certificate, network, or etcd cause and rerun the same declared upgrade; kubeadm upgrade is designed to converge idempotently.
If the new application revision fails but the cluster is healthy, roll back that workload through its own controller and source of truth rather than changing Kubernetes packages.
For unrecoverable control-plane or etcd failure, use the rehearsed official etcd restore and cluster recovery plan with the verified snapshot plus separate application backups. Do not improvise an in-place 1.36-to-1.35 downgrade.
After recovery, require the complete preflight and acceptance suite before resuming from the last proven gate.

Evidence

Sources and review

Verified 2026-07-24Review due 2026-10-22

Kubernetes Upgrading kubeadm clusters from 1.35 to 1.36official Kubernetes Version Skew Policyofficial Kubernetes Safely Drain a Nodeofficial Kubernetes Operating etcd clustersofficial Kubernetes Changing the package repositoryofficial