Back up and restore Kubernetes with Velero
Design RPO and RTO, pin Velero 1.18, choose one volume-protection path per PVC, secure independent object storage, create scheduled recovery points, restore into isolation, verify bytes and application behavior, and operate drills without confusing backup completion with recoverability.
Build and continuously prove a reversible Kubernetes recovery system for API resources and persistent volumes, with explicit application consistency, independent retention, least-privilege identity, measured RPO/RTO, and recurring restore evidence.
- Kubernetes 1.25+, provider-supported
- Velero 1.18.2, 1.18.x
- CSI snapshot API snapshot.storage.k8s.io/v1 when used
- Recovery contract Approve workload tiers, RPO, RTO, retention, consistency, recovery order, validation, and accountable owners before selecting scope.
- Supported compatibility matrix Confirm Kubernetes, Velero 1.18.2, provider plugin, CSI driver, snapshot controller, object store, node operating system, and architecture compatibility from current official documentation.
kubectl version && velero version --client-only - Independent recovery storage Provide least-privilege object storage with encryption, retention or immutability, capacity monitoring, audit, and key custody outside the cluster failure domain.
- Application consistency Obtain owner-approved quiesce, native backup, integrity, positive, and negative side-effect tests for every stateful application.
- Isolated recovery target Prepare a disposable cluster or namespace with network isolation, non-production identities, no public ingress, suspended side effects, compatible APIs, and compatible storage.
- Out-of-band administration Maintain separate access to object storage, encryption keys, cluster provisioning, DNS, identities, and incident coordination after total source-cluster loss.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A pinned Velero 1.18 installation with an independently protected object-storage location, a provider or CSI snapshot path chosen from an explicit support matrix, and a node agent only where file-system backup or snapshot data movement actually requires it.
- A backup policy that captures Kubernetes API objects and selected persistent-volume data, labels every protected workload, schedules backups against stated recovery-point objectives, retains enough generations for operational and security incidents, and exposes failure evidence before retention silently expires.
- A recurring restore drill into an isolated namespace or disposable recovery cluster that proves data integrity, application readiness, identity boundaries, external dependencies, and recovery-time objectives rather than treating a Completed backup phase as proof of recoverability.
- A canary workload containing a ConfigMap, Secret reference, Deployment, Service, PersistentVolumeClaim, and deterministic data set is backed up, inspected, restored to an isolated target, and verified byte-for-byte without overwriting the production namespace.
- Operators can distinguish Kubernetes resource capture, CSI snapshot creation, CSI snapshot data movement, provider-native snapshotting, and File System Backup; every protected volume has exactly one intentional path and an owner who understands its portability and consistency limits.
- The organization can show the last successful backup, last successful restore drill, measured RPO and RTO, object-lock or immutability posture, storage-location health, repository maintenance state, alert routing, upgrade compatibility, and an executable retirement procedure.
Architecture
How the parts fit together
Velero has two related but different jobs. It serializes selected Kubernetes resources into backup metadata stored in a BackupStorageLocation, and it coordinates one of several mechanisms for persistent-volume data. A native provider snapshot or CSI snapshot can be fast but may remain inside one failure domain. CSI snapshot data movement copies snapshot content through data-mover Pods to object storage. File System Backup reads mounted file systems through the node agent and writes repository data to object storage. The Velero server reconciles Backup, Restore, Schedule, BackupStorageLocation, VolumeSnapshotLocation, DataUpload, DataDownload, and BackupRepository resources; provider plugins translate storage operations. None of those components can make a crash-consistent volume application-consistent without an application-aware quiesce procedure. The recovery design therefore separates credentials, storage durability, consistency, restore testing, and application validation, and it never restores into production merely to prove that a backup exists.
- Inventory cluster-scoped and namespaced resources, CRDs, operators, volume types, databases, external dependencies, identity material, exclusions, legal retention, and recovery priorities.
- Classify each volume as provider snapshot, CSI snapshot, CSI snapshot data movement, File System Backup, database-native export, or intentionally excluded, and test the selected mechanism against the actual driver and topology.
- Create an object-storage boundary with least-privilege credentials, encryption, versioning or immutability where supported, lifecycle policy, capacity monitoring, and a failure domain independent from the protected cluster.
- Install matching Velero CLI and server versions plus only the reviewed provider plugin and node-agent capabilities; inspect generated RBAC, Secrets, DaemonSets, and storage-location objects before accepting them.
- Protect a deterministic canary first, inspect every Backup and volume-operation result, then create schedules whose names, labels, TTLs, and included namespaces encode ownership and recovery intent.
- Restore into isolation, validate object versions and persistent bytes, execute application-level checks, measure elapsed recovery, document gaps, and destroy the test target only after evidence is retained.
- Alert on missed schedules, PartiallyFailed or Failed phases, unavailable locations, stale restores, repository errors, quota pressure, and version drift; repeat drills after meaningful storage, Kubernetes, plugin, or application changes.
Assumptions
- The cluster runs a Kubernetes version supported by the pinned Velero 1.18 release, every provider plugin is compatible with that release, and the organization can maintain an explicit matrix rather than assuming latest components interoperate.
- Object storage is reachable through a reviewed network path and is administered separately enough that deleting or compromising the cluster does not automatically delete every recovery copy.
- A disposable recovery cluster or an isolated namespace with non-production ingress, DNS, credentials, and external integrations is available. A production namespace is never used as the first restore target.
- Storage drivers expose documented snapshot or mount behavior. The presence of the CSI API does not prove snapshot support, data-movement support, cross-zone restore, or cross-cluster portability.
- Database and message-system owners provide consistency and validation procedures. This guide does not claim that filesystem-level or crash-consistent snapshots replace database-native logical or physical backup methods.
- Secrets required by applications can be restored only under an approved identity and encryption model. Some credentials should be reissued rather than restored, especially after a suspected compromise.
- Cluster-scoped resources, CRDs, webhooks, and operators are restored in an order that is tested for the target. Restoring an obsolete API object into a newer cluster may fail or produce unsafe behavior.
- All example names and endpoints are non-sensitive placeholders. Cloud keys, repository passwords, encryption material, and object-storage credentials are supplied through the chosen platform's secure mechanism and never committed with guide configuration.
Key concepts
- Recovery point objective
- The maximum acceptable age of recovered data. A fifteen-minute RPO requires successful, durable recovery points frequently enough that one delayed or failed job does not silently exceed fifteen minutes.
- Recovery time objective
- The maximum acceptable time to restore service. It includes detecting the event, provisioning a target, restoring objects and volumes, validating dependencies, switching traffic, and obtaining operational approval.
- BackupStorageLocation
- A Velero API object that describes where backup metadata and applicable repository data are stored. Available means Velero can currently communicate with it, not that every stored backup is complete or restorable.
- VolumeSnapshotLocation
- A provider-oriented location used by snapshot plugins. Its snapshots can have different durability, region, account, encryption, and deletion behavior from object-storage backups.
- CSI snapshot
- A storage-system snapshot represented by VolumeSnapshot and VolumeSnapshotContent resources. The snapshot usually remains in the storage provider unless data movement is explicitly used.
- CSI snapshot data movement
- A workflow that creates a CSI snapshot, provisions an intermediate volume, and copies the data through data-mover Pods to a Velero backup repository in object storage.
- File System Backup
- A node-agent workflow that backs up mounted volume files to object storage. It is mutually exclusive with snapshots for the same volume and has workload, mount, permissions, performance, and consistency considerations.
- Restore drill
- A controlled recovery exercise that rebuilds the selected resources and data in isolation, executes technical and application assertions, records duration, and turns failures into owned corrective work.
- Application consistency
- A recovery point in which the application's internal invariants are valid. It may require quiescing writes, flushing journals, coordinating replicas, or using application-native backup tooling.
- Immutability
- A storage control that prevents objects from being changed or deleted for a defined period by ordinary credentials. It reduces ransomware and operator-error risk but requires deliberate retention and legal design.
Before you copy
Values used in this guide
{{veleroVersion}}Pinned Velero release used by both the CLI and server after compatibility review.
Example: v1.18.2{{providerPluginImage}}Immutable image reference for the official provider plugin compatible with the selected Velero release.
Example: provider.example/velero-plugin@sha256:reviewed-digest{{providerName}}Provider identifier required by the selected official Velero object-store plugin after compatibility review.
Example: aws{{bucketName}}Dedicated object-storage bucket that is encrypted, retention-controlled, monitored, and separated from the source cluster failure domain.
Example: organization-kubernetes-recovery{{backupLocationConfig}}Provider-specific, non-secret BackupStorageLocation configuration documented by the selected official plugin.
Example: region=eu-central-1,s3ForcePathStyle=false{{snapshotLocationConfig}}Provider-specific, non-secret VolumeSnapshotLocation configuration; use only when provider snapshots are part of the reviewed volume matrix.
Example: region=eu-central-1{{backupLocation}}Velero BackupStorageLocation name for the independently protected object store.
Example: dr-object-store{{snapshotLocation}}Reviewed VolumeSnapshotLocation name when provider snapshots are part of the design.
Example: primary-region-snapshots{{protectedNamespace}}Canary or application namespace included in the narrow backup test.
Example: payments-canary{{restoreNamespace}}Isolated target namespace with no production ingress or credentials.
Example: restore-drill-202607{{backupName}}Unique manual backup name containing purpose and UTC timestamp.
Example: payments-canary-20260728t1000z{{scheduleName}}Stable schedule name with an owner and documented RPO.
Example: payments-15m{{backupTtl}}Velero object TTL aligned with storage retention and legal policy.
Example: 720h0m0s{{canaryPvc}}PVC containing a deterministic non-sensitive test data set.
Example: recovery-canary-data{{applicationDeployment}}Restored application Deployment inspected with owner-approved, read-only diagnostics when the canary succeeds but the real application does not.
Example: payments-api{{credentialsFile}}secretRoot-owned temporary path provided by a secret delivery process; the file content is never shown, logged, or committed.
Example: /run/secrets/velero/provider-credentials{{expectedDigest}}SHA-256 digest of the deterministic canary data captured before backup.
Example: a4b2...reviewed-full-digestSecurity and production boundaries
- Velero can read broad portions of the Kubernetes API and, depending on configuration, host-mounted volume data. Restrict installation, restore creation, plugin images, node-agent privileges, and access to the velero namespace as production security controls.
- Backup metadata may include Secrets and sensitive configuration. Kubernetes base64 encoding is not encryption. Encrypt object storage and snapshots with managed keys, control key deletion separately, audit reads, and treat downloaded backup archives as sensitive.
- Do not place cloud credentials, repository passwords, access keys, secret values, or signed URLs in CLI flags, environment variables captured by diagnostics, Helm values, shell history, Git, tickets, or tutorial examples. Use workload identity or a reviewed secret-delivery path.
- Protect the backup account from the cluster account. Credentials that can create backups should not necessarily be able to delete immutable versions or destroy encryption keys.
- A restore can reintroduce vulnerable images, stale RBAC, old webhooks, compromised credentials, malicious persistence, or obsolete APIs. Scan and review recovery points after security incidents before reconnecting them to production networks.
- Pre- and post-backup hooks execute commands in workload containers. Treat hook definitions as executable production changes, use fixed commands, bound timeouts, avoid secret output, and test failure behavior.
- Node agents may require hostPath and privileged access for particular methods. Enable only the capability required by the documented storage path, constrain eligible nodes, inspect Pod Security exceptions, and monitor the DaemonSet.
- Object immutability can preserve unwanted or regulated data. Set retention through approved policy, test expiration, and prevent a routine Velero credential from shortening or bypassing the retention boundary.
Stop before continuing if
- Stop if the selected Velero, Kubernetes, provider plugin, CSI driver, snapshot controller, and node-agent versions are not documented as compatible.
- Stop if object storage shares the same deletion credential and failure domain as the protected cluster and no independent copy or immutability boundary exists.
- Stop if a volume has no explicit protection method or if File System Backup and snapshot protection are both assumed for the same volume without understanding their mutual exclusion.
- Stop if the current application cannot be quiesced or validated and its owner has not accepted crash-consistent recovery limitations.
- Stop before restoring cluster-scoped resources, CRDs, admission webhooks, identities, or Secrets into a shared cluster without a resource-by-resource impact review.
- Stop if the recovery target can send production email, payments, jobs, webhooks, or other side effects before isolation controls are proven.
- Stop if a Completed backup has warnings, failed item operations, incomplete volume artifacts, unavailable storage, or no successful restore drill.
- Stop if retention, encryption-key lifecycle, repository password custody, object-lock policy, or restore authorization has no accountable owner.
instruction
Define the recovery contract and protection inventory
Write the RPO, RTO, recovery order, consistency method, retention, exclusions, owners, and proof required for each workload before installing a backup controller.
Why this step matters
Backup tooling cannot decide what the business needs to recover. The contract prevents a broad all-namespaces job from creating false confidence while critical external state, consistency, or restore order remains undocumented.
What to understand
Record workload tier, accountable owner, maximum data loss, maximum recovery duration, retention, legal holds, and whether recovery after compromise needs an older clean point.
Inventory namespaced objects, cluster-scoped dependencies, CRDs, operators, admission webhooks, storage classes, volumes, external DNS, load balancers, cloud identities, databases, and queues. The API listing is evidence, not an instruction to include everything.
Classify generated and declarative state. Git-managed objects may be rebuilt from a signed revision; live PVC data, generated certificates, controller status, and external databases require separate decisions.
Define proof: expected object counts, a deterministic file digest, database integrity query, read-only API check, negative egress test, and owner acceptance. A Pod in Running is not an application test.
Choose a recovery order. Foundation operators and CRDs often precede their custom resources; identity, policy, storage, and network controls should exist before application workloads activate.
System changes
- No cluster changes; produces an owned recovery matrix and evidence checklist.
Syntax explained
--verbs=list- Limits discovery to resource kinds that can be listed; it does not fetch object bodies or prove that Velero should back them up.
-A- Shows claims across namespaces so an unowned volume is less likely to be missed.
kubectl api-resources --verbs=list --namespaced=true -o name | sort && kubectl get pvc -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.storageClassName,MODE:.spec.volumeMode,ACCESS:.spec.accessModes[*],SIZE:.spec.resources.requests.storage' && kubectl get crd -o name | sortconfigmaps cronjobs.batch deployments.apps secrets statefulsets.apps NAMESPACE NAME CLASS MODE ACCESS SIZE payments-canary recovery-canary-data encrypted Filesystem ReadWriteOnce 5Gi customresourcedefinition.apiextensions.k8s.io/certificates.example.io
Checkpoint: Every protected state item has a method and owner
Continue whenThe inventory maps each workload and PVC to one protection path, measurable RPO/RTO, consistency method, validation, retention, and recovery owner.
Stop whenAny tier-one state, external dependency, encryption key, CRD owner, or persistent volume has no recoverable source and no accepted exclusion.
If this step fails
The inventory contains hundreds of unknown PVCs.
Likely causeWorkloads create claims dynamically without ownership, lifecycle, or classification labels.
kubectl get pvc -A --show-labelskubectl get statefulset -A -o wide
ResolutionMap claims through owner references and application teams, add durable ownership labels, and do not promise recovery for unclassified state.
Security notes
- Do not export Secret bodies or ConfigMap content during inventory; names, kinds, ownership, and sensitivity classification are enough.
Alternatives
- Start with one critical namespace and expand only after its full restore passes; breadth without proof is not maturity.
Stop conditions
- Do not install broad backup permissions before the recovery scope and authorization boundary are approved.
decision
Choose one tested protection method for every volume
Build a driver-specific matrix for provider snapshots, CSI snapshots, CSI snapshot data movement, File System Backup, and application-native backups.
Why this step matters
Kubernetes storage APIs describe interfaces, not disaster-recovery guarantees. The actual driver, provider, topology, deletion policy, and restore target determine whether a snapshot survives the event being planned for.
What to understand
Use provider or CSI snapshots for fast same-platform recovery only after testing deletion, zone placement, encryption keys, cross-cluster visibility, and restore from a retained snapshot.
Use CSI snapshot data movement when a supported CSI snapshot should be copied into object storage. Account for intermediate volume provisioning, mover Pod scheduling, repository credentials, transfer throughput, and additional RTO.
Use File System Backup for supported mounted volumes when file-level portability is appropriate. It is mutually exclusive with snapshots for the same volume, does not support every volume type, and can be slower for many small files.
Use database-native backup or replication for transaction consistency and point-in-time recovery. A storage mechanism may still protect ancillary files, but its role must not be overstated.
Document VolumeSnapshotClass deletion policy. Retain can protect a snapshot if namespace objects disappear, but retention and cleanup still need ownership.
System changes
- No changes; selects and records the intended volume mechanism before installation.
Syntax explained
Retain- Preserves storage snapshot content when its Kubernetes object is removed until a later controlled cleanup changes the lifecycle.
Delete- Allows deletion of the VolumeSnapshotContent to remove the backing snapshot; this can reduce orphan cost but narrows recovery protection.
kubectl get pvc,pv -A -o wide && kubectl get storageclass,csidriver,volumesnapshotclass -o wide && kubectl get volumesnapshotclass -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.driver}{"\t"}{.deletionPolicy}{"\n"}{end}'NAME DRIVER DELETIONPOLICY volumesnapshotclass.snapshot.storage.k8s.io/csi csi.example.invalid Retain NAME ATTACHREQUIRED PODINFOONMOUNT MODES AGE csidriver.storage.k8s.io/... true false Persistent 120d
Checkpoint: One intentional path per PVC
Continue whenEvery claim has one primary volume-data method, documented consistency level, failure domain, encryption custody, performance expectation, and restore test.
Stop whenA method is selected because its CRD exists rather than because the exact driver and restore target passed a test.
If this step fails
No VolumeSnapshotClass matches the PVC driver.
Likely causeThe CSI driver does not support snapshots or the snapshot controller and class are absent.
kubectl get pv {{canaryPvc}} -o yamlkubectl get volumesnapshotclass -o wide
ResolutionUse a provider-supported alternative, File System Backup where supported, or application-native backup; do not fabricate an incompatible class.
Security notes
- Snapshot encryption keys and deletion rights can be more powerful than the Velero namespace credential; separate their custody.
Alternatives
- For immutable object data, recreate the PVC from the authoritative object store rather than backing up a cache.
Stop conditions
- Stop if two mechanisms are expected to protect the same volume but Velero will choose only one and the team cannot explain which.
instruction
Prepare independent object storage, identity, and retention
Create the bucket, prefix, workload identity or secret-delivery path, encryption, immutability, lifecycle, audit, and capacity controls outside this guide, then expose only the minimum storage permission to Velero.
Why this step matters
A backup stored behind the same administrator, deletion token, region, and encryption-key lifecycle as the source can disappear in the same incident. Independence is a recovery property, not an installation flag.
What to understand
Prefer short-lived workload identity or a provider-native federation mechanism. If a credentials file is unavoidable, deliver it from a controlled secret system to a root-owned temporary location and remove it after installation.
Grant only required object operations on the dedicated bucket or prefix. Separate read/write from retention bypass, bucket deletion, and key administration.
Enable server-side encryption and record key ownership, rotation, disablement, deletion delay, and disaster access. Losing the encryption key is equivalent to losing every backup.
Use versioning or immutable retention where supported and legally approved. Validate that lifecycle rules do not delete data earlier than Velero TTL and that ordinary cluster credentials cannot bypass retention.
Alert on storage denial, capacity, unusual reads, deletes, retention changes, and key state. Test recovery when the source cluster and its ordinary identities are unavailable.
System changes
- Creates or validates an external recovery storage and identity boundary through provider-approved tooling; no secret value is included here.
Syntax explained
{{credentialsFile}}- A path reference only. The guide never prints, base64-encodes, or stores the credential content.
Values stay on this page and are never sent or saved.
test -r {{credentialsFile}} && stat -c '%n %s bytes %y' {{credentialsFile}}/run/secrets/velero/provider-credentials 184 bytes 2026-07-28 09:40:00.000000000 +0000
Checkpoint: Recovery storage survives source-cluster loss
Continue whenA documented recovery identity can read protected objects while the cluster identity cannot delete immutable generations or destroy the encryption key.
Stop whenThe only credential is inside the source cluster, retention can be bypassed by routine operators, or key recovery is untested.
If this step fails
Storage access works only from an administrator laptop.
Likely causeThe intended workload identity, endpoint policy, DNS, CA trust, or cluster egress path is not configured.
kubectl -n velero get serviceaccountkubectl -n velero get networkpolicy
ResolutionImplement the provider's supported workload identity and network path, keeping personal credentials out of the cluster and installation logs.
Security notes
- A successful `test -r` and metadata-only `stat` check reveals only that a readable file exists. Never print it with cat, echo its content, describe the Secret, or enable shell tracing around credentials.
Alternatives
- Use a dedicated backup account or project with cross-account object retention when the provider supports that stronger separation.
Stop conditions
- Do not continue if the credential file is group-readable, persistent in the repository, or shared with interactive administrator access.
command
Install pinned Velero and only the required capabilities
Verify CLI provenance and client version, inspect the provider plugin's supported Velero matrix, then install the server with an immutable plugin image and the selected location; enable the node agent only when the volume decision requires it.
Why this step matters
Version pinning and capability minimization make the recovery environment reproducible. A moving plugin tag or unnecessary privileged node agent can change behavior or expand access without a reviewed guide revision.
What to understand
Download the Velero CLI from the official release, verify the published checksum through the approved software-supply process, and keep client and server in the same supported release line.
Read the official provider plugin matrix and pin an immutable image digest. The example plugin URI is intentionally invalid until replaced with the approved official plugin.
Render or capture generated manifests for review where your workflow permits. Inspect ClusterRoles, RoleBindings, ServiceAccounts, Secrets, hostPath mounts, security contexts, plugin init containers, and network requirements.
The example includes `--use-node-agent` because this design demonstrates File System Backup or data movement. Remove it when every protected volume uses a method that does not need the agent.
Do not assume `--wait` validates storage or restore. It waits for installation readiness; storage locations, plugins, node coverage, and a canary recovery are separate checkpoints.
System changes
- Creates the velero namespace, Velero CRDs, service account and RBAC, server Deployment, storage-location objects, plugin configuration, and a node-agent DaemonSet when enabled.
Syntax explained
--plugins- Installs the reviewed provider plugin image; use an immutable digest and compatibility-checked version.
--secret-file- Reads credentials from a protected file without placing their value in the command line. Prefer workload identity where supported.
--use-node-agent- Adds node agents needed by FSB or built-in CSI data movement; it is not required for every design.
--wait- Waits for installation resources, not for a successful backup or restore.
Values stay on this page and are never sent or saved.
velero version --client-only && velero install --namespace velero --provider {{providerName}} --plugins {{providerPluginImage}} --bucket {{bucketName}} --backup-location-config {{backupLocationConfig}} --snapshot-location-config {{snapshotLocationConfig}} --secret-file {{credentialsFile}} --use-node-agent --waitClient: Version: v1.18.2 Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.
Checkpoint: Pinned controllers are ready with reviewed privileges
kubectl -n velero get deploy,daemonset,pod -o wide && velero versionContinue whenThe server reports v1.18.2, approved plugin images are immutable, required node agents are Ready on eligible nodes, and no unreviewed privilege or secret exposure exists.
Stop whenClient/server skew, an unsupported plugin, mutable image, CrashLoop, uncovered node, broad unknown RBAC, or unapproved privileged mount appears.
If this step fails
The node-agent DaemonSet is not Ready on all intended nodes.
Likely causeTaints, selectors, Pod Security, kubelet root, image pull, hostPath, architecture, or resources prevent scheduling.
kubectl -n velero get pod -l name=node-agent -o widekubectl -n velero describe daemonset node-agent
ResolutionCorrect only the documented scheduling or platform requirement and retest on a canary node; do not grant cluster-wide privileged exceptions blindly.
Security notes
- Delete the temporary credentials file through the approved secret-delivery process after installation if the provider plugin has copied the required Secret; do not print it to confirm deletion.
Alternatives
- Manage the reviewed installation manifests through GitOps while delivering credentials separately; preserve the same version and restore-test gates.
Stop conditions
- Stop when the plugin documentation does not explicitly support Velero 1.18 and the target platform.
verification
Verify locations, plugins, repositories, and node coverage
Prove that Velero can reach the configured storage, that only intended plugins are registered, and that the selected data path is operational before backing up application state.
Why this step matters
Installation readiness and storage readiness are separate. This checkpoint catches incorrect endpoints, plugin registration, identity, node coverage, and repository state before the first meaningful recovery point.
What to understand
Confirm the server version, plugin kinds, image digests, storage names, provider, bucket or prefix, and recent validation timestamp.
An empty repository list is normal before the first FSB or data-movement backup creates one. After such a backup, every expected namespace repository must become Ready.
Inspect node-agent placement only on nodes that can host protected workloads. Account for taints, operating systems, architectures, kubelet roots, and security context requirements.
Review bounded logs for errors without enabling verbose output that may expose sensitive provider responses. Store diagnostic evidence under restricted access.
Prove that monitoring can observe location phase and controller health; a CLI check performed once is not continuous protection.
System changes
- No changes; reads installation and storage state.
Syntax explained
backup-location get- Shows location phase and validation metadata; it does not validate every historical object or encryption key.
repo get- Shows FSB or data-mover backup repositories after they have been initialized.
velero version && velero plugin get && velero backup-location get && velero snapshot-location get && velero repo get && kubectl -n velero get deploy,daemonset,podServer:
Version: v1.18.2
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED
{{backupLocation}} {{providerName}} recovery/velero Available 2026-07-28 10:02:41 +0000 UTC
NAME STATUS READY UP-TO-DATE AVAILABLE
node-agent 3 3 3 3Checkpoint: Every required control plane is healthy
Continue whenVelero 1.18.2, provider plugins, Available storage, optional snapshot location, node agents, and initialized repositories all match the approved design.
Stop whenAny location is Unavailable, expected plugin is absent, unexpected plugin exists, or required node coverage is incomplete.
If this step fails
The location alternates between Available and Unavailable.
Likely causeIntermittent DNS, proxy, certificate, identity refresh, endpoint, throttling, or network-policy failure.
velero backup-location getkubectl -n velero logs deploy/velero --since=30m
ResolutionStabilize the underlying identity and network path and prove sustained validation before scheduling backups.
Security notes
- Plugin and location output may expose bucket names or account metadata; restrict operational logs according to the backup data classification.
Alternatives
- Query Velero metrics for continuous health, but retain these CLI checks for deterministic drill evidence.
Stop conditions
- Do not create a production schedule while any required storage location is not continuously Available.
config
Create a deterministic recovery canary and consistency procedure
Deploy a non-sensitive canary whose API objects, mounted data, digest, and service response can be asserted after restore; document any application hook separately.
Why this step matters
A deterministic canary turns a backup system into a testable recovery pipeline. It distinguishes object restoration, volume restoration, data integrity, scheduling, and application readiness using safe known state.
What to understand
Use a reviewed immutable image digest in production; the named upstream image and version make the example readable but still require supply-chain pinning.
The test file is non-sensitive and has a known digest. Production validation should include owner-approved application queries without copying private data into evidence.
If FSB is selected, add the exact official Pod annotation or backup flag required by that design. If snapshots are selected, do not opt the same volume into FSB.
Application hooks should flush or pause writes with bounded timeouts and a post-hook that always resumes service. Test hook failure before relying on it.
Keep the canary isolated from production side effects and include it in every restore drill to identify platform failures independently from application complexity.
System changes
- Creates a canary namespace, PVC, Deployment, file, and Pod with no production credentials or external integration.
Syntax explained
sha256sum- Produces a deterministic content digest; it proves bytes, not application-level consistency.
ReadWriteOnce- Exercises a common block-volume access mode without claiming that other modes behave the same.
recovery-canary.yamlValues stay on this page and are never sent or saved.
apiVersion: v1
kind: Namespace
metadata:
name: {{protectedNamespace}}
labels:
oneliners.guru/recovery-owner: platform
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: {{canaryPvc}}
namespace: {{protectedNamespace}}
spec:
accessModes: [ReadWriteOnce]
storageClassName: encrypted
resources:
requests:
storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: recovery-canary
namespace: {{protectedNamespace}}
spec:
replicas: 1
selector:
matchLabels:
app: recovery-canary
template:
metadata:
labels:
app: recovery-canary
spec:
containers:
- name: canary
image: registry.k8s.io/e2e-test-images/agnhost:2.53
command: ["/bin/sh", "-c"]
args: ["test -f /data/canary.txt || printf 'OneLiners recovery canary v1\n' > /data/canary.txt; exec agnhost netexec --http-port=8080"]
ports:
- name: http
containerPort: 8080
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: {{canaryPvc}}Values stay on this page and are never sent or saved.
kubectl apply -f recovery-canary.yaml && kubectl -n {{protectedNamespace}} wait --for=condition=Available deploy/recovery-canary --timeout=180s && kubectl -n {{protectedNamespace}} exec deploy/recovery-canary -- sha256sum /data/canary.txtnamespace/payments-canary created persistentvolumeclaim/recovery-canary-data created deployment.apps/recovery-canary created deployment.apps/recovery-canary condition met a4b2d5b0d9f8cce1c80a06baba2d3341d538907a4107f9b3dcfc8f2ed833edc1 /data/canary.txt
Checkpoint: Canary state is stable before backup
Continue whenThe Deployment is Available, the PVC is Bound, the file digest is recorded, and the Pod serves the expected non-sensitive response.
Stop whenThe canary is not deterministic, its image is unreviewed, its volume method is ambiguous, or it has production credentials or egress.
If this step fails
The canary PVC remains Pending.
Likely causeThe example StorageClass is not present, topology waits for a consumer, quota is exhausted, or the CSI driver is unhealthy.
kubectl -n {{protectedNamespace}} describe pvc {{canaryPvc}}kubectl get storageclass
ResolutionSelect a reviewed encrypted class that exists in the test cluster and prove its snapshot or FSB path before continuing.
Security notes
- Never use copied production secrets or customer records as a canary. The goal is deterministic platform evidence, not realistic sensitive content.
Alternatives
- Use an existing dedicated recovery-test workload if it provides equivalent object, volume, digest, and service assertions.
Stop conditions
- Stop if the canary can reach or trigger a production external system.
command
Create and inspect a narrow manual backup
Back up only the canary namespace through its selected volume path, wait for completion, and inspect resource, hook, snapshot, data-movement, and warning evidence.
Why this step matters
A narrow manual backup proves the mechanism with understandable scope before schedules create many untested recovery points. Detailed inspection prevents a green phase from hiding omitted volumes or warnings.
What to understand
Use a unique timestamped name and one namespace. Do not start with all namespaces or cluster resources because omissions and side effects become harder to diagnose.
For FSB-only designs omit a meaningless snapshot location and use the approved opt-in or default flag. For CSI data movement use its documented backup option and inspect DataUpload objects.
Count expected Kubernetes items and volume artifacts. Investigate every warning, skipped resource, failed hook, failed item operation, snapshot, PodVolumeBackup, or DataUpload.
Record start, completion, backup UID, selected location, volume path, Velero and plugin versions, data digest, and total bytes where available.
Do not download or paste full backup content into a ticket. Evidence should contain names, phases, counts, hashes, and sanitized error text.
System changes
- Creates a Backup object, uploads selected Kubernetes metadata, and creates snapshot or repository data according to the chosen method.
Syntax explained
--include-namespaces- Limits the backup to the reviewed canary namespace; cluster-scoped inclusion still requires explicit understanding.
--storage-location- Selects the approved object-storage location.
--wait- Waits for terminal phase, after which details and logs still require review.
Values stay on this page and are never sent or saved.
velero backup create {{backupName}} --include-namespaces {{protectedNamespace}} --storage-location {{backupLocation}} --volume-snapshot-locations {{snapshotLocation}} --wait && velero backup describe {{backupName}} --detailsBackup request "payments-canary-20260728t1000z" submitted successfully. Waiting for backup to complete... Backup completed with status: Completed Phase: Completed Namespaces: Included: payments-canary Velero-Native Snapshot PVs: 1 of 1 snapshots completed successfully Total items to be backed up: 18 Items backed up: 18
Checkpoint: The recovery point contains every expected artifact
Continue whenPhase is Completed without unexplained warning; object counts, one intended volume artifact, digest context, hooks, and storage location match the protection matrix.
Stop whenPhase is PartiallyFailed or Failed, any expected object or volume is absent, or more than one unintended volume mechanism ran.
If this step fails
Backup is Completed but reports zero volume snapshots.
Likely causeThe design uses FSB, the PVC was excluded, snapshotting is disabled, or the snapshot class and provider do not match.
velero backup describe {{backupName}} --detailskubectl -n velero get podvolumebackups,datauploads -o wide
ResolutionCompare actual artifacts to the declared per-PVC method. Completed metadata alone is insufficient if volume data was expected.
Security notes
- Backup logs and object metadata can contain resource names and provider errors. Sanitize and restrict them; never enable shell tracing around credentials.
Alternatives
- Create separate resource-only and volume-method tests to isolate problems, then combine only after both paths pass.
Stop conditions
- Do not create a schedule from this command until a restore of this exact backup passes.
verification
Inspect backup logs, operations, repositories, and storage evidence
Review the terminal phase and every asynchronous volume operation, confirm the object exists under the expected retention boundary, and capture sanitized evidence.
Why this step matters
Velero coordinates asynchronous systems. Backup phase, logs, operation CRs, repository status, provider storage, and retention must agree before the recovery point is accepted.
What to understand
Zero error and warning counts are necessary but not sufficient. Compare expected resources and volume artifacts with actual detail.
A no-resources result for DataUploads is expected when the chosen path is a local snapshot or FSB. It is a defect when CSI data movement was required.
For FSB inspect PodVolumeBackups and repository readiness. For data movement inspect DataUploads, temporary Pods, bytes, and terminal completion.
Confirm object-store creation, encryption, version or retention state using provider-approved tooling without printing object content or signed access URLs.
Store a concise evidence record with checksums, counts, phases, timestamps, versions, and references to restricted logs.
System changes
- No changes; reads backup and repository evidence.
Syntax explained
ERRORS/WARNINGS- Summary counts that must be reconciled with detailed logs and expected artifacts.
LAST MAINTENANCE- Repository maintenance evidence, not proof of every object or data chunk.
Values stay on this page and are never sent or saved.
velero backup get {{backupName}} && (velero backup logs {{backupName}} | grep -Ei 'error|warning|failed' || true) && kubectl -n velero get podvolumebackups,datauploads -o wide && velero repo getNAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION payments-canary-20260728t1000z Completed 0 0 2026-07-28 10:00:14 +0000 UTC 29d dr-object-store No resources found in velero namespace. NAME STATUS LAST MAINTENANCE payments-canary-kopia Ready 2026-07-28 10:12:00 +0000 UTC
Checkpoint: Independent evidence agrees
Continue whenVelero phase, operation CRs, repository status, provider object, retention metadata, expected counts, and sanitized logs describe one complete recovery point.
Stop whenAny layer is missing, reports a warning, or relies on an unverified interpretation.
If this step fails
Repository is NotReady after a successful data backup.
Likely causeRepository password, object permissions, maintenance, connectivity, or repository metadata is inconsistent.
velero repo getkubectl -n velero describe backuprepository
ResolutionPreserve existing objects, repair the exact credential or repository issue, and run a fresh backup and restore test before accepting repository-backed recovery.
Security notes
- Do not attach backup archives, signed URLs, Secret YAML, or raw repository credentials to drill evidence.
Alternatives
- Use metrics and provider audit logs to automate evidence collection while retaining a human artifact review for releases.
Stop conditions
- Stop if storage-provider state cannot be correlated with the Velero backup UID and retention policy.
command
Create schedules from measured RPO and retention
Create an owner-labelled schedule only after the manual restore passes, choose a TTL that agrees with external retention, and alert on the age of the last successful recovery point.
Why this step matters
A cron expression is not an RPO. The operating control must detect delayed, failed, or incomplete runs quickly enough that a successful predecessor does not age beyond the promised recovery point.
What to understand
Calculate frequency from RPO with failure margin. If one missed run would breach the objective before alert response, schedule more frequently or use a different protection mechanism.
Align Velero TTL, object lifecycle, snapshot deletion, immutability, legal hold, and capacity. Conflicting policies create orphan cost or premature deletion.
Label or annotate schedule ownership through a reviewed manifest when CLI creation cannot capture the required metadata. Track service, tier, data class, and escalation.
Alert on last successful backup age, failed and partially failed phases, unavailable locations, operation failures, repository errors, and storage pressure.
Schedule names should remain stable while backup names are generated. Changes to scope, method, or retention require a new restore test and documented revision.
System changes
- Creates a Schedule that periodically creates Backup objects and associated storage artifacts.
Syntax explained
*/15 * * * *- Requests a backup every fifteen minutes; actual completion and RPO depend on duration and success.
--ttl- Controls Velero backup expiration; it must be coordinated with provider retention and immutability.
--selector- Filters displayed backups to one schedule without reading their contents.
Values stay on this page and are never sent or saved.
velero schedule create {{scheduleName}} --schedule='*/15 * * * *' --include-namespaces {{protectedNamespace}} --storage-location {{backupLocation}} --ttl {{backupTtl}} && velero schedule get && velero backup get --selector velero.io/schedule-name={{scheduleName}}Schedule "payments-15m" created successfully. NAME STATUS CREATED SCHEDULE BACKUP TTL LAST BACKUP SELECTOR payments-15m Enabled 2026-07-28 11:00:00 +0000 UTC */15 * * * * 30d0h0m0s 2m ago <none>
Checkpoint: Schedule and monitoring meet the recovery objective
Continue whenTwo consecutive scheduled backups complete within the RPO margin, retention is coherent, and alert routing detects a deliberately paused or failed canary schedule.
Stop whenBackup duration overlaps dangerously, alerts do not fire, TTL conflicts with storage policy, or the schedule includes unreviewed scope.
If this step fails
Scheduled backups overlap and consume cluster resources.
Likely causeThe interval is shorter than transfer duration, mover concurrency or storage throughput is insufficient, or scope is too broad.
velero backup get --selector velero.io/schedule-name={{scheduleName}}kubectl -n velero get pod -o wide
ResolutionMeasure throughput and split scope, adjust schedule or supported concurrency, and preserve the RPO with capacity rather than allowing an unbounded queue.
Security notes
- Do not encode credentials or sensitive selectors in schedule names, labels, or annotations.
Alternatives
- Use event-driven application-native backups for write-heavy data and a slower Velero schedule for Kubernetes resources.
Stop conditions
- Do not claim a fifteen-minute RPO until backup completion age and alert response demonstrate it.
command
Restore the backup into an isolated target
Prepare network and identity isolation first, map the source namespace to a clean target, keep the default non-destructive existing-resource policy, and wait for the Restore to finish.
Why this step matters
Restore is the only practical proof of recoverability, but it can also recreate credentials, schedulers, ingress, and integrations. Isolation must exist before restored workloads are allowed to start.
What to understand
Prefer a disposable recovery cluster for cluster-scoped resources, operators, admission controls, storage migrations, or full disaster exercises. Namespace mapping is a narrower canary method.
Apply default-deny egress, no ingress, disabled external DNS, non-production identities, and suspended CronJobs through pretested target controls before activation.
Use the default non-destructive policy. Existing resources are normally skipped; `--existing-resource-policy=update` is best effort and does not overwrite PVC data as a naive restore might imply.
Inspect restore warnings, errors, item operations, DataDownloads, PodVolumeRestores, PVC events, image pulls, and admission rejections.
Never treat a Completed Restore as permission to switch traffic. Application and negative side-effect tests follow.
System changes
- Creates an isolated namespace and Restore object, then recreates selected resources and persistent data under namespace mapping.
Syntax explained
--from-backup- Selects the exact accepted recovery point.
--namespace-mappings- Rewrites source namespace references to the isolated target where supported.
--wait- Waits for terminal Restore phase, not for full application correctness.
Values stay on this page and are never sent or saved.
kubectl create namespace {{restoreNamespace}} --dry-run=client -o yaml | kubectl apply -f - && kubectl label namespace {{restoreNamespace}} oneliners.guru/recovery-isolated=true --overwrite && velero restore create {{backupName}}-drill --from-backup {{backupName}} --namespace-mappings {{protectedNamespace}}:{{restoreNamespace}} --waitnamespace/restore-drill-202607 created namespace/restore-drill-202607 labeled Restore request "payments-canary-20260728t1000z-drill" submitted successfully. Waiting for restore to complete... Restore completed with status: Completed
Checkpoint: Restore completes without production connectivity
Continue whenThe Restore is Completed without unresolved warnings, expected objects and volume operations exist in the mapped namespace, and ingress, egress, jobs, and production identities remain blocked.
Stop whenAny side effect is possible, cluster-scoped object would collide, existing objects are skipped unexpectedly, or a volume operation fails.
If this step fails
Restored workload starts before isolation policy applies.
Likely causeNamespace controls were created after restore or admission does not enforce the isolation label.
kubectl -n {{restoreNamespace}} get pod,networkpolicy,cronjob,ingresskubectl get namespace {{restoreNamespace}} --show-labels
ResolutionScale or suspend workloads from the recovery control plane, revoke any restored credentials, record the exposure, and make isolation an admission-enforced prerequisite.
Security notes
- Restored Secrets may be valid production credentials. Prefer test identities or block all external access until reviewed reissuance and approval.
Alternatives
- Restore into a disposable cluster with a separate cloud account for a stronger full-disaster boundary.
Stop conditions
- Stop before a restore that includes cluster-scoped RBAC or webhooks in a shared cluster without explicit impact analysis.
verification
Validate objects, volume bytes, application behavior, and RTO
Compare restored inventory and data with the accepted recovery point, run positive and negative application checks, and record total recovery time and owner acceptance.
Why this step matters
A valid restore preserves intended data and behavior while preventing unintended behavior. Digest, resource, application, dependency, and isolation assertions are all required to turn controller success into recovery evidence.
What to understand
Compare source-at-backup and target resource counts, selected specifications, PVC binding, storage class, access modes, volume mode, and restored bytes.
Run application-native integrity checks and read-only business assertions. For databases, validate schema, transaction consistency, last expected record, and replication state using owner procedures.
Test negative behavior: no production email, payment, webhook, queue consumption, DNS update, scheduled job, privileged identity, or public ingress.
Measure from incident declaration or drill start through target readiness, data restoration, validation, and owner acceptance. Infrastructure-ready time alone understates RTO.
Record versions, backup and restore UIDs, timestamps, output digests, warnings, corrective actions, and signatures without storing secret values or customer records.
System changes
- No changes; executes read-only recovery assertions.
Syntax explained
--details- Shows item-level restore warnings and errors needed to reconcile scope.
sha256sum- Compares known canary bytes with the pre-backup digest.
Values stay on this page and are never sent or saved.
velero restore describe {{backupName}}-drill --details && kubectl -n {{restoreNamespace}} get deploy,pod,svc,pvc && kubectl -n {{restoreNamespace}} exec deploy/recovery-canary -- sha256sum /data/canary.txt && kubectl -n {{restoreNamespace}} exec deploy/recovery-canary -- wget -qO- http://127.0.0.1:8080/hostnamePhase: Completed Warnings: 0 Errors: 0 NAME READY UP-TO-DATE AVAILABLE deployment.apps/recovery-canary 1/1 1 1 persistentvolumeclaim/recovery-canary-data Bound a4b2d5b0d9f8cce1c80a06baba2d3341d538907a4107f9b3dcfc8f2ed833edc1 /data/canary.txt recovery-canary-7c8f7c5d9f-x2m4p
Checkpoint: Recovery evidence meets RPO and RTO
Continue whenResources, volume digest, application invariants, positive tests, negative side-effect tests, isolation, measured RPO, measured RTO, and owner acceptance all pass.
Stop whenAny expected byte, object, invariant, dependency, or isolation test fails even if Restore phase is Completed.
If this step fails
Digest matches but the application fails readiness.
Likely causeConfiguration, schema version, identity, service discovery, dependency, admission, architecture, or image compatibility differs from the recovery point.
kubectl -n {{restoreNamespace}} describe podkubectl -n {{restoreNamespace}} logs deploy/recovery-canary --since=15m
ResolutionKeep isolation, compare the full dependency and version matrix, repair the recovery procedure, and rerun from a clean target.
Security notes
- Use hashes and aggregate assertions for sensitive data; avoid exposing restored records in logs, screenshots, or tickets.
Alternatives
- Run a synthetic transaction against a test endpoint when direct data inspection would violate access boundaries.
Stop conditions
- Do not approve traffic cutover on Kubernetes readiness alone.
instruction
Monitor protection and repeat restore drills
Alert on age and failure, retain bounded evidence, run scheduled recovery exercises, and track corrective actions until the measured recovery contract remains true.
Why this step matters
Backup systems decay as APIs, drivers, plugins, credentials, retention, data size, and application dependencies change. Continuous health and repeated restore evidence are the only defense against silent recovery drift.
What to understand
Alert on last successful backup age per workload, not just global controller availability. Include failed, partially failed, deleting, stuck, and missing schedule conditions.
Monitor storage-location validation, snapshot errors, operation duration, repository readiness and maintenance, node-agent coverage, mover resource pressure, object capacity, and credential expiry.
Run a namespace canary monthly and a disposable-cluster recovery quarterly or after material Kubernetes, storage, plugin, identity, operator, or application changes.
Measure trend in backup duration, restore duration, bytes, small-file count, warning count, and cost. Adjust capacity before the schedule interval becomes impossible.
Attach each drill gap to an owner and due date. A failed drill with no remediation is an acknowledged unmet recovery objective.
System changes
- No direct changes; establishes alerts, drill cadence, evidence retention, and remediation ownership.
Syntax explained
last successful backup age- The most useful RPO-oriented signal; it catches missed and repeatedly failed schedules.
last successful restore drill- A recoverability signal that should age into warning when no recent target validation exists.
velero schedule get && velero backup get && velero restore get && velero backup-location get && velero repo get && kubectl -n velero get backup,restore,schedule,dataupload,datadownloadNAME STATUS SCHEDULE LAST BACKUP payments-15m Enabled */15 * * * * 7m ago NAME PHASE LAST VALIDATED dr-object-store Available 2026-07-28 12:44:20 +0000 UTC NAME STATUS LAST MAINTENANCE payments-canary-kopia Ready 2026-07-28 12:30:00 +0000 UTC
Checkpoint: A failed backup and stale drill page the right owner
Continue whenSynthetic failure and stale-age tests reach the correct route with workload, schedule, age, location, runbook, and no secret value.
Stop whenAlerts depend on manual dashboard viewing, cannot identify an owner, or treat PartiallyFailed as success.
If this step fails
Backups fail but no page fires because the last success still exists.
Likely causeAlerting checks object existence or controller uptime rather than age of the newest accepted recovery point.
velero backup get --selector velero.io/schedule-name={{scheduleName}}velero schedule get
ResolutionAlert on accepted-success timestamp against RPO and separately on failed phases and stuck operations.
Security notes
- Monitoring labels should identify workload and recovery policy without leaking bucket credentials, Secret names that reveal customers, or object URLs.
Alternatives
- Export evidence to the incident platform, but keep Velero and provider state as the authoritative technical sources.
Stop conditions
- Do not close a failed drill until the corrected procedure has passed from a clean target.
instruction
Upgrade safely and retire without deleting recovery evidence
Test supported version transitions with canary backup and restore, preserve storage through rollback, and remove schedules, controllers, credentials, snapshots, repositories, and retained data only through separate approved decisions.
Why this step matters
Recovery tooling upgrades can change CRDs, plugins, repository formats, node-agent behavior, and supported APIs. Retirement can be even more destructive if uninstall and data deletion are conflated.
What to understand
Read every intermediate release's official upgrade instructions and compatibility notes. Do not skip unsupported minor transitions or assume plugin compatibility from image pull success.
Capture current manifests, CRDs, versions, location state, repository state, schedules, accepted backups, object retention, and canary evidence before change.
Test upgrade in a representative non-production cluster against copies or safe canary storage, then repeat backup and restore after production rollout.
If controllers fail, preserve CRDs and backup storage while following the documented rollback. Never delete object data or snapshots as an installation rollback.
For retirement, pause schedules, retain required backups, export the recovery catalog, revoke runtime credentials, remove controllers, and delete stored data only after retention, legal, and owner approval.
System changes
- Creates a pre-upgrade canary Backup and Restore; a later approved upgrade changes Velero components and CRDs.
Syntax explained
pre-upgrade canary- A known recovery point proving the old stack immediately before change.
post-upgrade restore- Evidence that the new stack reads and restores accepted historical data.
Values stay on this page and are never sent or saved.
velero version && velero plugin get && velero backup-location get && velero backup create pre-upgrade-canary --include-namespaces {{protectedNamespace}} --wait && velero restore create pre-upgrade-canary-drill --from-backup pre-upgrade-canary --namespace-mappings {{protectedNamespace}}:{{restoreNamespace}} --waitClient: Version: v1.18.2 Server: Version: v1.18.2 Backup completed with status: Completed Restore completed with status: Completed
Checkpoint: Historical and new recovery points work after change
Continue whenPinned supported versions are healthy and both a pre-upgrade backup and a new post-upgrade backup restore successfully with identical assertions.
Stop whenAny historical backup becomes unreadable, plugin or CRD is unsupported, repository is not Ready, or rollback would require deleting recovery data.
If this step fails
New server cannot read an older repository.
Likely causeUnsupported upgrade path, repository password mismatch, incompatible uploader, plugin skew, or incomplete migration.
velero versionvelero repo get
ResolutionPreserve storage, restore the last supported components and credentials, consult the official upgrade path, and retry in a clone before production.
Security notes
- Removing Velero does not authorize deletion of backups, keys, snapshots, or immutable objects; each has a separate retention decision.
Alternatives
- Run parallel canary installations against separate storage prefixes for a major migration when the provider and Velero documentation support that isolation.
Stop conditions
- Stop if no tested path can restore both the most recent and oldest still-required recovery points.
Finish line
Verification checklist
velero version && velero plugin get && kubectl -n velero get deploy,daemonset,podCLI and server are Velero 1.18.2, only approved plugin digests run, and every required controller and node agent is Ready.velero backup-location get && velero snapshot-location get && velero repo getThe selected backup location is recently Available, optional snapshot location matches the design, and initialized repositories are Ready.velero backup describe {{backupName}} --detailsThe exact backup is Completed with zero unexplained warnings or errors and contains every expected object and one intended volume artifact per PVC.velero restore describe {{backupName}}-drill --detailsThe isolated restore is Completed with zero unresolved warnings or errors and no existing-resource collision.kubectl -n {{restoreNamespace}} exec deploy/recovery-canary -- sha256sum /data/canary.txtThe restored deterministic data digest exactly matches {{expectedDigest}} recorded before backup.kubectl -n {{restoreNamespace}} get deploy,pod,svc,pvc,networkpolicy,ingress,cronjobApplication integrity and read-only service checks pass, production side effects remain blocked, and measured RPO and end-to-end RTO meet the approved targets.Recovery guidance
Common problems and safe checks
The BackupStorageLocation is Unavailable.
Likely causeCredentials, workload identity, endpoint, certificate trust, bucket policy, region, DNS, proxy, or network egress is wrong.
velero backup-location getkubectl -n velero logs deploy/velero --since=15m
ResolutionCompare the location and provider-plugin configuration with the official provider documentation, repair the least-privilege identity or network path, then wait for Available before creating another backup.
The backup phase is PartiallyFailed.
Likely causeOne or more resources, hooks, snapshots, PodVolumeBackups, DataUploads, or plugin operations failed while other items completed.
velero backup describe {{backupName}} --detailsvelero backup logs {{backupName}}
ResolutionTreat the recovery point as unproven. Identify every warning and error, correct the specific resource or volume path, create a new backup, and complete a restore drill.
No persistent-volume data appears in the backup.
Likely causeThe PVC was excluded, its Pod annotation did not select File System Backup, the snapshot class or location did not match, or the workload was not mounted as expected.
velero backup describe {{backupName}} --detailskubectl get pvc,pv,volumesnapshot,volumesnapshotcontent -A
ResolutionMap the PVC to its intended single protection path, correct labels, annotations, snapshot-class selection, or node-agent coverage, then verify the new volume artifact explicitly.
A CSI snapshot remains not ready.
Likely causeThe CSI driver, external snapshotter, VolumeSnapshotClass, topology, quota, permissions, or storage backend cannot complete the snapshot.
kubectl describe volumesnapshot -Akubectl get volumesnapshotcontent -o wide
ResolutionResolve the CSI/storage error with the driver owner. Do not mark the Velero backup recoverable until readyToUse and a restore of that exact class have passed.
A DataUpload or DataDownload is stuck.
Likely causeData-mover scheduling, temporary PVC provisioning, repository access, resource pressure, timeout, topology, or node-agent health is blocking transfer.
kubectl -n velero get datauploads.velero.io,datadownloads.velero.io -o widekubectl -n velero get pod -l app.kubernetes.io/name=velero -o wide
ResolutionInspect the resource status and bounded mover logs, restore scheduling or capacity, and retry through a new backup or restore rather than editing terminal operation state.
File System Backup reports no eligible volume.
Likely causeThe Pod volume is unsupported, unmounted, excluded, not annotated, or node-agent host paths and mount propagation do not match the kubelet layout.
kubectl -n {{protectedNamespace}} get pod -o yamlkubectl -n velero get daemonset node-agent -o yaml
ResolutionVerify the official FSB limitations and kubelet root, then opt in only supported mounted volumes or choose a tested snapshot or application-native method.
A scheduled backup did not run.
Likely causeThe Schedule is paused, invalid, recently changed, controller is unhealthy, or timezone assumptions differ from cron evaluation.
velero schedule getkubectl -n velero describe schedule {{scheduleName}}
ResolutionRepair the controller or schedule, calculate the missed RPO, create a controlled manual backup if appropriate, and alert on future age rather than relying only on cron existence.
Restore reports that resources already exist.
Likely causeThe target is not isolated or a previous drill left objects. Velero's default behavior skips most existing resources instead of overwriting them.
kubectl get all -n {{restoreNamespace}}velero restore describe {{backupName}}-restore --details
ResolutionUse a clean disposable target or reviewed namespace mapping. Do not switch casually to the update policy, which is best-effort and does not overwrite PVC data as people may assume.
PVCs remain Pending after restore.
Likely causeThe target lacks a compatible StorageClass, CSI driver, topology, capacity, snapshot handle, credentials, or access mode.
kubectl -n {{restoreNamespace}} describe pvckubectl get storageclass,csidriver,volumesnapshotclass
ResolutionProvide the tested target storage capability or use an approved resource modifier or mapping; never patch storage semantics without proving data placement and access.
Pods start but application data is inconsistent.
Likely causeThe recovery point was crash-consistent, hooks failed, replicated state was captured at different moments, or application migrations differ from the data.
velero backup describe {{backupName}} --detailskubectl -n {{restoreNamespace}} logs deploy/{{applicationDeployment}} --since=15m
ResolutionKeep the target isolated, use application-native recovery or a known consistent point, repair the quiesce procedure, and create a new validated backup.
The restored application sends production side effects.
Likely causeIngress, egress, credentials, queue consumers, schedulers, or external endpoints were restored before isolation controls.
kubectl -n {{restoreNamespace}} get networkpolicy,ingress,cronjob,secretkubectl -n {{restoreNamespace}} get deploy,statefulset -o wide
ResolutionStop workloads through the recovery-cluster control plane, revoke exposed credentials, block egress, record the incident, and redesign the drill so isolation precedes workload activation.
Backups grow until object-store quota is exhausted.
Likely causeSchedule TTL, object lifecycle, snapshot deletion policy, repository maintenance, failed deletion, or legal hold does not match capacity planning.
velero backup getvelero repo get
ResolutionPreserve required recovery points, reconcile Velero TTL with provider lifecycle and immutability, repair maintenance, add capacity alerts, and never delete arbitrary objects behind Velero.
A backup is visible after installing Velero in a new cluster but cannot restore.
Likely causeMetadata synchronized from object storage while plugins, APIs, snapshot handles, storage classes, repository password, encryption keys, or data paths are incompatible.
velero backup describe {{backupName}} --detailsvelero plugin get
ResolutionRecreate the documented recovery environment and validate every dependency. Listing a backup proves discovery, not portability.
Velero upgrade leaves controllers or CRDs unhealthy.
Likely causeThe upgrade path, CRD change, provider plugin, CLI/server skew, or removed feature was not tested in a representative cluster.
velero versionkubectl -n velero get deploy,daemonset,pod
ResolutionStop backup changes, preserve existing storage, follow the official version-by-version upgrade or rollback procedure, and repeat canary backup and restore before resuming schedules.
Reference
Frequently asked questions
Does a Completed Velero backup prove disaster recovery?
No. It proves that the controller reached a terminal phase. Only a restore drill that checks resources, volume bytes, application invariants, isolation, dependencies, and elapsed time demonstrates the required recovery outcome.
Should every PVC use File System Backup?
No. Choose per volume. FSB improves portability for many mounted file systems but adds node access and transfer cost, has limitations, and is not automatically application-consistent. A tested CSI or provider snapshot can be better for some volumes.
Can GitOps replace Velero?
GitOps can recreate reviewed declarative objects, but it normally does not contain live persistent data, generated resources, every runtime identity, or a transactionally consistent recovery point. Use each tool for its explicit boundary.
Recovery
Rollback
Pause new backup or restore activity, preserve all existing recovery objects, storage, snapshots, repositories, credentials, and evidence, revert only the most recent controller, plugin, schedule, or selection change, then prove the last known-good backup and restore path before resuming.
- Stop new schedules or pause the affected Schedule without deleting existing Backups; record current controller, plugin, CRD, location, repository, operation, and object-retention state.
- For a failed restore drill, keep production untouched, isolate or delete only the disposable target after evidence is captured, and never use update-existing policy as an improvised repair.
- For a failed installation or upgrade, follow the official supported rollback path for Velero and the provider plugin while preserving CRDs, BackupStorageLocations, repositories, snapshots, and object data.
- Restore the last known-good manifests and immutable plugin images, then verify server, node-agent, storage-location, snapshot-location, and repository health.
- Create a fresh narrow canary backup and restore it into a clean isolated target; repeat digest, application, negative side-effect, RPO, and RTO checks.
- If a credential was exposed, revoke it through the provider, issue a least-privilege replacement, inspect audit logs, and determine whether backup content or retention controls were accessed.
- Resume schedules only after the owner accepts the corrected recovery evidence. Delete backups, snapshots, repositories, keys, or object data solely through a separate retention-approved retirement procedure.
Evidence