Deploy Kubernetes applications safely with Helm
Pin Helm and chart inputs, separate secrets from values and release history, render and server-validate manifests, stage upgrades with bounded waits, verify application behavior, roll back only across compatible data, and operate hooks, CRDs, drift, and secret rotation.
Create a reproducible, secret-safe, testable Helm deployment workflow that makes value precedence and rendered change explicit, detects application regressions, and supports evidence-based rollback without pretending Helm can reverse external data side effects.
- Kubernetes 1.25+, chart-supported
- Helm 4.2.3, 4.2.x
- Helm 3 maintained line 3.21.3, 3.21.x
- Unambiguous target Provide cluster identity, kube context, namespace, release name, owner, change window, and out-of-band access.
kubectl config current-context && kubectl cluster-info - Pinned supply chain Verify Helm client, immutable chart, OCI or repository provenance, Chart.lock, dependencies, application images, and Kubernetes API compatibility.
- External secret workflow Create or mount required credentials outside Helm values with least privilege, encryption, rotation, audit, fail-closed behavior, and recovery.
- Application migration contract Document schema, queue, cache, API, hook, CRD, and external side effects plus compatible forward and rollback paths.
- Observability and canary Provide a canary, rollout metrics, SLOs, logs, data checks, dependency checks, negative security assertions, and a bounded observation window.
- Known-good revision Verify the current revision, image artifacts, schema compatibility, and ability to restore application behavior before change.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A reproducible Helm release workflow that pins Helm, chart, dependencies, container images, values layers, target cluster, namespace, and release name; renders and validates the exact desired manifests before mutation.
- A secret boundary in which ordinary configuration lives in reviewed values files but secret values never enter chart defaults, environment overrides, Git, `--set`, shell history, dry-run output, release notes, CI logs, or stored deployment artifacts.
- A staged upgrade and rollback procedure with server-side validation, diff evidence, preflight compatibility, availability gates, Helm tests, application checks, revision history, controlled rollback, and post-rollback verification.
- An operator can explain the final value precedence, chart version, dependency lock, image digest, rendered objects, hooks, CRDs, release revision, and Kubernetes compatibility for any deployment without retrieving or printing secret content.
- A canary upgrade waits for Kubernetes readiness and application evidence, fails safely when the deadline or test fails, and either preserves the known-good revision or rolls back through an explicit reviewed decision.
- Secret delivery can use pre-created Kubernetes Secrets, a Secrets Store CSI pattern, or an approved external-secret controller, while the chart references stable object names and never claims that one provider-specific mechanism is portable to every cluster.
Architecture
How the parts fit together
Helm renders a chart by combining chart defaults, parent values, user-supplied values files, and command-line overrides, then sends Kubernetes objects as a release. The cluster stores release state, so anyone with appropriate namespace Secret access may read historical rendered manifests and values. A chart can also contain hooks and CRDs whose lifecycle differs from ordinary resources. Safe operation therefore separates non-secret configuration from secret material, pins every deployable input, validates output through lint, template, Kubernetes schema and admission, then observes workload and application behavior after apply. Helm rollback creates a new revision from previous release content; it cannot reverse external database migrations, deleted data, immutable fields, CRD conversion, cloud resources, or actions performed by hooks. Those require application-owned compatibility and rollback procedures.
- Confirm cluster context, namespace, Helm 4.2 or maintained Helm 3.21 compatibility, chart provenance, immutable chart version, dependency lock, images, API versions, and application migration policy.
- Separate environment configuration into documented non-secret values files; define the chart contract for references to pre-created or externally synchronized Secrets without embedding values.
- Run dependency build, lint, template, secret-aware output handling, static policy checks, Kubernetes API schema validation, and server-side dry run against the real target.
- Review rendered objects, field ownership, RBAC, Services, ingress, storage, Pod security, NetworkPolicy, hooks, CRDs, deletion behavior, and value precedence.
- Capture current release status, history, values without secret disclosure, running images, data schema, and application SLO before changing anything.
- Run a canary or staged `helm upgrade --install` with a pinned chart version, explicit values, wait and timeout behavior, and either an intentional rollback-on-failure policy or a manual evidence gate.
- Verify Kubernetes rollout, Helm tests, application transactions, metrics, logs, dependency compatibility, secret mounts, and negative security assertions.
- When a regression occurs, stop promotion, preserve evidence, decide whether Helm rollback is data-compatible, roll back to the explicit revision if safe, and validate the complete application again.
Assumptions
- The target Kubernetes context, cluster, namespace, release name, and change window are known and independently visible in the prompt or deployment job.
- Helm 4.2.3 is the primary tested client. Organizations remaining on Helm 3 use the maintained 3.21.3 line and verify flag differences, especially dry run, server-side behavior, and failure rollback.
- The chart declares an appropriate `kubeVersion`, uses APIs served by the target cluster, and has been tested against admission controllers, policy, storage, ingress, service mesh, and CNI.
- Chart artifacts, dependencies, and images are pinned and verified. Floating tags, unpinned repository indexes, or mutable dependencies are not accepted as reproducible release inputs.
- A secret workflow can create the required namespaced Secret or mounted content before Helm. The workflow owns rotation, revocation, encryption, identity, audit, and recovery.
- Application owners define backwards-compatible schema migration, feature-flag, queue, cache, and external API behavior. Helm itself cannot make an irreversible migration reversible.
- Observability exists for rollout, readiness, errors, latency, saturation, queue state, data integrity, and external dependencies. `helm status deployed` is not the acceptance criterion.
- Current release state and values may contain sensitive material from older practices. Operators use restricted access and never paste `helm get all` output into tickets or CI logs.
Key concepts
- Chart version
- The package version in Chart.yaml. It is different from application version and should identify immutable chart content.
- Values precedence
- The order in which chart defaults, parent values, user values files, and `--set` overrides merge, with later and more specific inputs taking precedence.
- Release revision
- A namespace-scoped Helm history entry created by install, upgrade, or rollback. Rollback creates another revision rather than erasing history.
- Hook
- A chart resource executed at lifecycle points such as pre-upgrade or post-install. Hooks can mutate data and their resources may follow special deletion policies.
- CRD lifecycle
- Helm installs CRDs from the chart `crds/` directory but intentionally does not treat CRD upgrade and deletion like ordinary templates; operators need a separate migration plan.
- Server-side dry run
- A simulation sent to the Kubernetes API server for schema and admission evaluation without persisting the objects. It still does not run controllers, hooks, or application tests.
- Rollback-on-failure
- A Helm 4 upgrade policy that returns to the previous successful release when the upgrade fails. It cannot undo external side effects or incompatible data migrations.
- External secret pattern
- A design in which a provider, CSI driver, or controller obtains secret material outside chart values and presents it to a Pod or Kubernetes Secret. Implementations and portability differ.
- Helm test
- A hook-marked workload run on demand to validate release behavior. It needs bounded permissions, timeout, cleanup, and application assertions.
- Three-way safety boundary
- Rendered configuration, Kubernetes controller readiness, and application or data behavior are separate layers that all need evidence.
Before you copy
Values used in this guide
{{helmVersion}}Pinned supported Helm client.
Example: v4.2.3{{releaseName}}Stable release name in one namespace.
Example: checkout{{releaseNamespace}}Target namespace for release state and objects.
Example: checkout-prod{{chartReference}}Verified OCI or chart repository reference.
Example: oci://registry.example.invalid/charts/checkout{{chartVersion}}Immutable chart version approved for this change.
Example: 3.8.2{{baseValues}}Reviewed non-secret common values file.
Example: values/common.yaml{{environmentValues}}Reviewed non-secret environment override.
Example: values/production.yaml{{secretObjectName}}secretStable name created outside Helm by the approved secret workflow.
Example: checkout-runtime{{upgradeTimeout}}Measured deadline for rollout and hook completion.
Example: 10m{{previousRevision}}Explicit last known-good Helm revision.
Example: 17{{expectedImageDigest}}Approved application image identity.
Example: sha256:reviewed-immutable-digestSecurity and production boundaries
- Never put literal passwords, tokens, private keys, certificates, signed URLs, or connection strings in `values.yaml`, environment values, `--set`, `--set-string`, `--set-file`, environment variables printed by CI, shell history, release notes, or tutorial examples.
- Helm dry run and rendered output can contain Secret manifests. Helm 4 offers `--hide-secret` for dry run, but output-handling policy still must prevent secret-bearing templates and external tools from logging values.
- Kubernetes Secrets are stored unencrypted in etcd by default unless encryption at rest is configured. Restrict get/list/watch and Pod creation in the namespace, and prefer short-lived externally managed material.
- An external secret controller, Secrets Store CSI Driver, provider CSI integration, or pre-created Secret is a pattern, not a universal implementation. Validate provider identity, refresh, startup, outage, fail-closed behavior, and portability per cluster.
- Helm release storage can include rendered Secret data and user-supplied values. Restrict access to release Secrets and avoid broad `helm get all` or `helm get values --all` output.
- Hooks are executable deployment behavior and can bypass the apparent declarative flow. Review images, commands, RBAC, network, timeouts, deletion policy, retry, and data effects.
- CRDs and controllers can create cluster-wide security effects. A namespaced Helm release does not make a CRD, ClusterRole, webhook, or custom controller namespaced.
- Chart template functions can produce dynamic values. Avoid random secret generation during upgrades; it can rotate material unexpectedly and leave release storage as the only recovery source.
Stop before continuing if
- Stop if kube context, namespace, release name, chart version, application version, or target change owner is ambiguous.
- Stop if any chart, dependency, or image is mutable, unverified, or incompatible with the target Kubernetes version.
- Stop if a literal secret appears in values, rendered output, diff, dry-run log, command line, release state, or CI artifact.
- Stop if server-side dry run reports policy, schema, ownership, immutable-field, quota, or admission failure.
- Stop if hooks, CRDs, ClusterRoles, webhooks, storage, ingress, or deletion behavior have not been reviewed.
- Stop if the release contains a destructive or backwards-incompatible data migration without an independently tested recovery procedure.
- Stop if canary readiness, application SLO, data checks, secret mount, or negative security assertions fail.
- Stop if Helm rollback is proposed without confirming application and schema compatibility with the previous revision.
verification
Confirm the exact target and supported toolchain
Display the current Kubernetes context, cluster identity, namespace, permissions, Helm version, and served APIs before reading or changing release state.
Why this step matters
A perfectly rendered chart deployed to the wrong cluster or namespace is still an incident. Tool and API versions also determine flag and manifest behavior.
What to understand
Require context and cluster identity in the deployment record, not only a convenient alias.
Confirm Helm 4.2.3 or the separately tested maintained Helm 3.21.3 line; do not assume flags are identical.
Inspect permissions without retrieving Secret content. Release storage access is sensitive.
Confirm the namespace owner, quota, policy, storage, ingress, and mesh prerequisites.
System changes
- No changes; validates target and toolchain.
Syntax explained
auth can-i get secrets- Checks authorization because Helm release state is commonly Secret-backed; it does not read content.
Values stay on this page and are never sent or saved.
kubectl config current-context && kubectl cluster-info && kubectl auth can-i get secrets -n {{releaseNamespace}} && helm version && kubectl versionprod-eu1-platform
Kubernetes control plane is running at https://api.prod-eu1.example.invalid
yes
version.BuildInfo{Version:"v4.2.3"}
Client Version: v1.36.1
Server Version: v1.36.1Checkpoint: Target identity is unambiguous
Continue whenContext, API endpoint, namespace, Helm line, Kubernetes version, permissions, and change owner match the approved request.
Stop whenAny target field differs, the namespace is shared unexpectedly, or tool versions are unsupported.
If this step fails
The context name looks right but API endpoint is wrong.
Likely causeA reused kubeconfig alias points to another cluster.
kubectl cluster-infokubectl config view --minify
ResolutionStop and replace the kubeconfig through the approved identity process; never infer identity from context name alone.
Security notes
- Do not print embedded kubeconfig credentials or client certificates.
Alternatives
- Use a deployment identity bound to one cluster and namespace to reduce context mistakes.
Stop conditions
- Do not continue on an ambiguous context.
instruction
Pin chart, dependencies, images, and compatibility
Verify the immutable chart artifact and dependency lock, inspect Chart.yaml and values, and map every image and API to the target cluster before rendering.
Why this step matters
An immutable input set is necessary for review, reproduction, rollback, and incident comparison.
What to understand
Pin chart version and verify OCI digest or repository provenance through the approved supply chain.
Commit Chart.lock and use dependency build rather than floating update during deployment.
List every image repository, tag, and digest; reject `latest`, branch, or mutable canary tags.
Review kubeVersion, CRDs, API versions, hooks, and application migration notes.
System changes
- No cluster changes; may populate local dependency artifacts according to the lock.
Syntax explained
--version- Prevents Helm from selecting a newer chart automatically.
Values stay on this page and are never sent or saved.
helm show chart {{chartReference}} --version {{chartVersion}} && helm show values {{chartReference}} --version {{chartVersion}} | sed -n '1,80p' && helm dependency list ./chartapiVersion: v2 name: checkout version: 3.8.2 appVersion: 2026.07.4 kubeVersion: ">=1.33.0-0" NAME VERSION REPOSITORY STATUS common 2.4.1 oci://... ok
Checkpoint: Every deployable input is immutable and compatible
Continue whenChart, lock, dependencies, images, Kubernetes APIs, CRDs, and application version are pinned and supported.
Stop whenAny input floats or the target lacks a required API.
If this step fails
Chart.lock does not match Chart.yaml.
Likely causeDependency constraints changed without rebuilding and review.
helm dependency list ./chartgit diff -- Chart.yaml Chart.lock
ResolutionResolve dependencies in a reviewed build, verify immutable artifacts, and commit the intentional lock update.
Security notes
- Chart provenance does not make template behavior safe; review both.
Alternatives
- Vendor the verified chart and dependencies in an internal immutable registry.
Stop conditions
- Stop on mutable dependencies.
decision
Keep secrets out of Helm values and release history
Choose a pre-created Secret, Secrets Store CSI integration, or approved external-secret controller, and make the chart reference only the stable object or mount contract.
Why this step matters
Helm stores release content and renders output in many tools. Removing literal secret values from Helm reduces exposure through Git, history, dry runs, diffs, CI, and release storage.
What to understand
Pre-created Secrets are the most portable contract but need external rotation and audit.
Secrets Store CSI mounts provider data without necessarily synchronizing a Kubernetes Secret; identity and outage behavior are provider-specific.
External-secret controllers synchronize external data to Kubernetes objects through CRDs; this is convenient but adds controller and CRD dependencies.
The chart validates that a named object or mount exists and fails closed. It never supplies a default credential.
Test rotation, missing data, provider outage, old-value revocation, Pod restart, and least-privilege access.
System changes
- No change here; the separately authorized secret workflow creates or mounts secret material.
Syntax explained
jsonpath metadata only- Confirms object identity without printing Secret data.
Values stay on this page and are never sent or saved.
kubectl -n {{releaseNamespace}} get secret {{secretObjectName}} -o jsonpath='{.metadata.name}{"\t"}{.metadata.creationTimestamp}{"\t"}{.type}{"\n"}'checkout-runtime 2026-07-28T15:10:00Z Opaque
Checkpoint: No literal secret enters Helm
Continue whenThe chart references a stable contract and secret values are absent from defaults, overrides, commands, rendered artifacts, release notes, and Git.
Stop whenA value would be passed with `--set`, stored in a values file, or printed for verification.
If this step fails
Chart requires `secret.password` as a value.
Likely causeThe chart couples secret creation to deployment.
helm show values {{chartReference}} --version {{chartVersion}}
ResolutionAdd or select a chart option referencing an existing Secret, or change the chart contract before production use.
Security notes
- Base64 is encoding, not encryption; do not accept encoded literals in values.
Alternatives
- Use short-lived workload identity and no application Secret when the service supports it.
Stop conditions
- Stop on any literal secret.
command
Lint and render the exact non-secret values layers
Run chart linting and client-side rendering with the exact ordered values files, capture only sanitized manifests, and inspect value precedence and generated objects.
Why this step matters
Client rendering exposes the concrete object set and values merge before cluster mutation. It catches template errors but not target admission or controller behavior.
What to understand
Values files are applied left to right; the environment file wins on matching keys.
Keep only non-secret values. `--hide-notes` does not hide Secret objects, so the chart contract must already exclude secret content.
Inspect RBAC, Services, ingress, storage, security contexts, probes, resources, NetworkPolicy, hooks, annotations, and owner labels.
Use policy scanners and Kubernetes schema validation on sanitized output.
System changes
- Creates a local rendered.yaml containing non-secret manifests; does not change the cluster.
Syntax explained
-f common -f production- The right-most file takes precedence.
--include-crds- Includes CRDs for review even though their lifecycle differs.
Values stay on this page and are never sent or saved.
helm lint {{chartReference}} --version {{chartVersion}} -f {{baseValues}} -f {{environmentValues}} && helm template {{releaseName}} {{chartReference}} --version {{chartVersion}} -n {{releaseNamespace}} -f {{baseValues}} -f {{environmentValues}} --include-crds --hide-notes > rendered.yaml==> Linting oci://registry.example.invalid/charts/checkout 1 chart(s) linted, 0 chart(s) failed wrote rendered.yaml: 14 Kubernetes objects
Checkpoint: Rendered output is safe and intentional
Continue whenLint passes; object count, images, APIs, privileges, probes, resources, hooks, CRDs, and values match review with no secret data.
Stop whenA secret, unexpected cluster-scoped object, unsafe Service, floating image, or destructive hook appears.
If this step fails
Rendered image tag differs from the environment value.
Likely causeAnother values path or chart helper overrides it.
rg 'image:' rendered.yamlhelm show values {{chartReference}} --version {{chartVersion}}
ResolutionTrace the template and precedence, fix the chart or approved values, and add a render assertion.
Security notes
- Treat rendered.yaml as sensitive until automated scanning proves it contains no Secret values.
Alternatives
- Render in a locked CI job and retain only hashes and policy results.
Stop conditions
- Do not upload unreviewed rendered manifests.
verification
Run target-cluster server-side dry run
Submit the exact upgrade as a server-side simulation with Secret output hidden, then inspect admission, ownership, quota, immutable fields, and API errors before a real rollout.
Why this step matters
The real API server evaluates served schemas, admission webhooks, policies, quota, and authorization that client rendering cannot know.
What to understand
Helm 4's `--dry-run=server` requires cluster access but persists no release.
`--hide-secret` reduces dry-run disclosure; external tools and chart notes still need review.
Investigate every warning and conflict. Do not normalize warnings that predict unsupported APIs or policy failures.
Dry run does not run hooks, controllers, scheduling, image pulls, readiness, or application tests.
System changes
- No persistent changes; API server evaluates a simulation.
Syntax explained
--dry-run=server- Uses target API validation without persistence.
--hide-secret- Hides Kubernetes Secret manifests from Helm dry-run output.
Values stay on this page and are never sent or saved.
helm upgrade --install {{releaseName}} {{chartReference}} --version {{chartVersion}} -n {{releaseNamespace}} --create-namespace -f {{baseValues}} -f {{environmentValues}} --dry-run=server --hide-secretRelease "checkout" does not exist. Installing it now. NAME: checkout NAMESPACE: checkout-prod STATUS: pending-install REVISION: 1 TEST SUITE: None Secret manifests hidden
Checkpoint: Target admission accepts the exact release
Continue whenSimulation returns no schema, policy, ownership, quota, immutable, RBAC, or secret-output defect.
Stop whenAny server error or unresolved warning appears.
If this step fails
Admission rejects a security context.
Likely causeChart defaults violate namespace policy.
kubectl get namespace {{releaseNamespace}} --show-labelskubectl get events -n {{releaseNamespace}}
ResolutionMake the workload compliant; never weaken the namespace policy merely to pass a chart.
Security notes
- Do not remove `--hide-secret` in shared logs.
Alternatives
- Use `kubectl apply --server-side --dry-run=server` on sanitized rendered manifests as an additional policy gate.
Stop conditions
- Stop on any admission rejection.
verification
Capture the current release and application baseline
Before upgrade, record status, history, chart metadata, non-secret values discipline, running images, workload readiness, application SLO, schema compatibility, and the explicit last known-good revision.
Why this step matters
Rollback needs an explicit known-good revision and application compatibility evidence, not simply the previous number.
What to understand
Record revision, chart, application, image digest, replica and PVC state, SLO, error rate, latency, queue, external dependency, and schema version.
Avoid `helm get all` in broad logs because historical values and manifests may contain secrets.
Review hooks and migrations introduced by the new chart and whether old application code remains compatible afterward.
Define rollback decision authority and maximum observation window.
System changes
- No changes; captures a sanitized pre-change baseline.
Syntax explained
--max 10- Limits displayed history while retaining enough context.
Values stay on this page and are never sent or saved.
helm status {{releaseName}} -n {{releaseNamespace}} && helm history {{releaseName}} -n {{releaseNamespace}} --max 10 && kubectl -n {{releaseNamespace}} get deploy,statefulset,pod,svc,endpointslice,pvc -o wideNAME: checkout NAMESPACE: checkout-prod STATUS: deployed REVISION: 17 REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 17 2026-07-21 10:14:00 UTC deployed checkout-3.7.9 2026.07.2 Upgrade complete
Checkpoint: Revision 17 is proven rollback candidate
Continue whenThe explicit revision, image, schema, SLO, data compatibility, and rollback owner are recorded.
Stop whenPrevious revision is already degraded or incompatible with planned migration.
If this step fails
Previous revision contains an unavailable image.
Likely causeRegistry retention removed rollback artifacts.
helm get manifest {{releaseName}} -n {{releaseNamespace}} --revision {{previousRevision}}
ResolutionRestore verified immutable artifacts or choose another tested revision before change.
Security notes
- Sanitize historical release output.
Alternatives
- Export only approved metadata and hashes from release history.
Stop conditions
- Do not upgrade without a viable recovery candidate.
command
Run the staged pinned upgrade with bounded waits
Deploy the exact chart and values with a measured timeout, wait for Kubernetes readiness, and use Helm 4 rollback-on-failure only after proving that hooks and data changes are compatible with automatic reversal.
Why this step matters
Pinned inputs and bounded readiness gates reduce ambiguity. Automatic rollback is useful only when the application and data layer can safely return.
What to understand
Use `--rollback-on-failure` only after testing migrations, hooks, and external side effects. Otherwise use a manual stop and evidence decision.
The watcher wait observes supported resource readiness, not full application correctness.
Set timeout from measured image pulls, scheduling, migrations, readiness, and load-balancer convergence with a finite incident bound.
Watch events, Jobs, Pods, endpoints, metrics, and logs during rollout from a separate session.
Do not use `--force-replace` as a general repair; replacement can delete stateful or identity-bearing objects.
System changes
- Creates or updates the Helm release and Kubernetes resources, executes declared hooks, and may automatically create a rollback revision on failure.
Syntax explained
--wait=watcher- Helm 4 watches supported resources for readiness.
--rollback-on-failure- Rolls back failed upgrades to the previous successful release; cannot undo external data effects.
--timeout- Bounds waits and hooks according to measured deployment behavior.
Values stay on this page and are never sent or saved.
helm upgrade --install {{releaseName}} {{chartReference}} --version {{chartVersion}} -n {{releaseNamespace}} --create-namespace -f {{baseValues}} -f {{environmentValues}} --wait=watcher --timeout {{upgradeTimeout}} --rollback-on-failureRelease "checkout" has been upgraded. Happy Helming! NAME: checkout NAMESPACE: checkout-prod STATUS: deployed REVISION: 18 DESCRIPTION: Upgrade complete
Checkpoint: Revision 18 is deployed without hidden failure
Continue whenHelm reports deployed, expected revision and chart run, workload readiness is stable, hooks complete, and no warning event or SLO regression appears.
Stop whenA migration, hook, Pod, endpoint, Secret mount, event, or SLO is unhealthy.
If this step fails
Upgrade times out during migration.
Likely causeMigration exceeds deadline or is blocked.
kubectl -n {{releaseNamespace}} get job,podkubectl -n {{releaseNamespace}} logs job/{{releaseName}}-migrate --since=15m
ResolutionDetermine migration state before retry or rollback; never run a non-idempotent migration twice blindly.
Security notes
- Watch logs only under restricted access and redact secret-bearing application output.
Alternatives
- Use a canary release name or progressive delivery controller for higher-risk changes.
Stop conditions
- Stop promotion on the first failed application gate.
verification
Run Helm tests and application verification
Execute bounded non-destructive Helm tests, then validate image digest, endpoints, service behavior, metrics, data compatibility, secret delivery, and negative security assertions.
Why this step matters
Helm and Kubernetes status do not validate business behavior. Layered checks reveal configuration, identity, dependency, data, and security regressions.
What to understand
Helm tests must use a reviewed image, least privilege, no production writes, bounded timeout, and cleanup policy.
Verify immutable image ID, desired and available replicas, endpoints, readiness, ingress, TLS, NetworkPolicy, Secret mount metadata, and application metrics.
Run owner-approved read-only transactions, schema compatibility checks, queue health, and external dependency checks.
Test negative cases: denied identity, unavailable path, absent secret exposure, blocked unnecessary egress, and no debug endpoint.
Observe through the agreed canary interval before broad promotion.
System changes
- Creates temporary Helm test hook resources; tests are non-destructive and cleaned according to chart policy.
Syntax explained
--logs- Displays test logs, which must be designed not to contain secrets.
imageID- Confirms runtime immutable image identity.
Values stay on this page and are never sent or saved.
helm test {{releaseName}} -n {{releaseNamespace}} --logs --timeout 5m && kubectl -n {{releaseNamespace}} rollout status deployment/{{releaseName}} --timeout=5m && kubectl -n {{releaseNamespace}} get pod -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].imageID}{"\n"}{end}'NAME: checkout LAST DEPLOYED: 2026-07-28 16:20:00 UTC NAMESPACE: checkout-prod STATUS: deployed TEST SUITE: checkout-smoke Last Started: 2026-07-28 16:24:10 UTC Last Completed: 2026-07-28 16:24:18 UTC Phase: Succeeded deployment "checkout" successfully rolled out checkout-6c55b8c5cc-k72pq registry.example.invalid/checkout@sha256:reviewed-immutable-digest
Checkpoint: Application and security gates pass
Continue whenHelm test, rollout, image, service, SLO, data, dependency, secret, and negative assertions pass for the full observation window.
Stop whenAny user, data, security, or dependency assertion fails despite deployed status.
If this step fails
Helm test succeeds but SLO degrades.
Likely causeTest coverage is too narrow or load-dependent behavior regressed.
kubectl -n {{releaseNamespace}} get pod,svc,endpointslicekubectl -n {{releaseNamespace}} logs deploy/{{releaseName}} --since=15m
ResolutionStop promotion, preserve evidence, and use the application rollback or compatible forward fix.
Security notes
- Test logs must never echo secret values.
Alternatives
- Use an external synthetic canary in addition to in-cluster Helm tests.
Stop conditions
- Do not accept on Helm status alone.
command
Roll back explicitly when application compatibility permits
If verification fails, stop traffic promotion, confirm schema and external side effects are compatible with revision 17, then create a new rollback revision with bounded wait and verify it.
Why this step matters
Explicit revision and compatibility review prevent a visual rollback from corrupting data or reviving unsupported configuration.
What to understand
Before rollback, determine whether migrations, hooks, queues, caches, secrets, CRDs, and external APIs remain compatible with old code.
Rollback creates revision 19 from revision 17; it does not erase failed revision 18.
`--cleanup-on-fail` only affects certain new resources from a failed rollback. It does not clean external or hook side effects.
After rollback, repeat the same application, SLO, data, dependency, secret, and security checks.
Preserve failed revision evidence and open corrective work before retry.
System changes
- Creates a new Helm revision from the selected historical revision and changes managed Kubernetes resources.
Syntax explained
{{previousRevision}}- Explicit known-good revision, not an assumed previous state.
--cleanup-on-fail- Cleans certain newly created resources when rollback itself fails; it is not general garbage collection.
Values stay on this page and are never sent or saved.
helm rollback {{releaseName}} {{previousRevision}} -n {{releaseNamespace}} --wait=watcher --timeout {{upgradeTimeout}} --cleanup-on-fail && helm history {{releaseName}} -n {{releaseNamespace}} --max 5Rollback was a success! Happy Helming! REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 19 2026-07-28 16:31:00 UTC deployed checkout-3.7.9 2026.07.2 Rollback to 17
Checkpoint: Known-good behavior is restored
Continue whenRollback revision is deployed and all workload, data, SLO, dependency, secret, negative, and route checks match the pre-change baseline.
Stop whenOld code is data-incompatible, side effects remain, or any validation fails.
If this step fails
Old Pods fail after rollback.
Likely causeSchema, Secret, CRD, or external contract changed irreversibly.
kubectl -n {{releaseNamespace}} get pod,jobkubectl -n {{releaseNamespace}} logs deploy/{{releaseName}} --since=15m
ResolutionUse the application recovery or compatible forward-fix plan; do not repeat rollback.
Security notes
- Rollback can restore an older vulnerable image or policy. Incident authority must accept that risk or choose a patched forward fix.
Alternatives
- Use feature-flag disablement or traffic shift when code rollback is data-incompatible.
Stop conditions
- Do not roll back across an incompatible migration.
instruction
Operate release history, drift, secrets, and chart lifecycle
Monitor releases and applications, rotate secrets independently, test upgrades and rollbacks, govern hooks and CRDs, and keep only intentional source-controlled configuration.
Why this step matters
Charts, APIs, credentials, admission, dependencies, and workloads drift. A release is safe only while its source, runtime, rotation, and rollback evidence stay current.
What to understand
Alert on failed and pending releases, rollout failure, missing endpoints, hook Jobs, secret refresh failure, certificate expiry, SLO regression, and drift.
Rotate secret material through its owning system and verify Pods reload or restart intentionally; Helm values remain unchanged.
Review chart and component releases at least every 90 days and before Kubernetes minor upgrades.
Treat CRD upgrade and deletion as separate operator procedures. Helm will not safely manage every CRD lifecycle.
Retire by disabling traffic and jobs, preserving required data, uninstalling reviewed namespaced resources, then cleaning retained or external resources through their owners.
System changes
- No direct changes; establishes ongoing release, secret, upgrade, drift, and retirement controls.
Syntax explained
helm list -A- Inventories releases across accessible namespaces without printing values.
Values stay on this page and are never sent or saved.
helm list -A && helm history {{releaseName}} -n {{releaseNamespace}} --max 10 && kubectl -n {{releaseNamespace}} get deploy,statefulset,job,pod,svc,pvc,networkpolicyNAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION checkout checkout-prod 19 2026-07-28 16:31:00 UTC deployed checkout-3.7.9 2026.07.2
Checkpoint: Release remains reproducible and recoverable
Continue whenSource revision, chart, values, images, secret contract, runtime, history, SLO, upgrade review, and rollback test remain current.
Stop whenLive drift, stale chart, secret exposure, orphan hook, unsupported API, or untested rollback appears.
If this step fails
Live object differs from rendered source.
Likely causeManual patch, mutating admission, controller ownership, or another deployer changed it.
helm get manifest {{releaseName}} -n {{releaseNamespace}}kubectl -n {{releaseNamespace}} get deploy {{releaseName}} -o yaml
ResolutionIdentify field ownership and reconcile through the authoritative source or formal handoff; do not overwrite blindly.
Security notes
- Release history may preserve old secret-bearing mistakes; restrict and remediate access even after chart correction.
Alternatives
- Use GitOps continuous reconciliation with the same secret and rollback boundaries.
Stop conditions
- Do not retire data-bearing resources without owner approval.
Finish line
Verification checklist
kubectl config current-context && kubectl cluster-info && helm version && kubectl versionCluster, namespace, Helm line, Kubernetes version, permissions, and owner match the approved deployment.helm show chart {{chartReference}} --version {{chartVersion}} && helm lint {{chartReference}} --version {{chartVersion}} -f {{baseValues}} -f {{environmentValues}}Chart, dependencies, images, APIs, values, hooks, CRDs, and rendered objects are immutable, compatible, and free of literal secrets.helm upgrade --install {{releaseName}} {{chartReference}} --version {{chartVersion}} -n {{releaseNamespace}} -f {{baseValues}} -f {{environmentValues}} --dry-run=server --hide-secretThe target API accepts schemas, admission, ownership, quota, immutable fields, and policies without exposing Secret manifests.helm status {{releaseName}} -n {{releaseNamespace}} && helm history {{releaseName}} -n {{releaseNamespace}} --max 5Expected revision, chart and application version are deployed, and the explicit rollback revision remains available and compatible.helm test {{releaseName}} -n {{releaseNamespace}} --logs --timeout 5m && kubectl -n {{releaseNamespace}} rollout status deployment/{{releaseName}} --timeout=5mHelm tests, rollout, immutable image, application SLO, data, dependency, secret mount, and negative security assertions pass.kubectl -n {{releaseNamespace}} get secret {{secretObjectName}} -o jsonpath='{.metadata.name}{"\t"}{.metadata.creationTimestamp}{"\n"}'The expected secret contract exists through an approved external workflow and no secret value appears in Helm values, commands, logs, or release artifacts.Recovery guidance
Common problems and safe checks
Helm renders a value different from the environment file.
Likely causeA later values file, `--set`, parent chart, or default overrides the expected path, or YAML type coercion changes it.
helm template {{releaseName}} {{chartReference}} --version {{chartVersion}} -f {{baseValues}} -f {{environmentValues}} --hide-secrethelm show values {{chartReference}} --version {{chartVersion}}
ResolutionTrace documented precedence, remove ad hoc CLI overrides, quote strings deliberately, and add a render assertion.
helm dependency build changes Chart.lock unexpectedly.
Likely causeChart.yaml constraints, repository index, lock file, or Helm version differs from the reviewed build.
git diff -- Chart.lockhelm dependency list .
ResolutionRestore the reviewed lock, pin dependencies and client, fetch from the approved immutable source, and commit only intentional changes.
Client template succeeds but server dry run fails.
Likely causeCluster API versions, admission policy, quota, RBAC, schema, field ownership, or webhook rejects the objects.
helm upgrade --install {{releaseName}} {{chartReference}} --version {{chartVersion}} -n {{releaseNamespace}} -f {{baseValues}} -f {{environmentValues}} --dry-run=server --hide-secret
ResolutionCorrect the chart or target prerequisites; do not disable admission or use force replacement as a shortcut.
Rendered output or CI log contains secret data.
Likely causeSecret values were supplied through values or an output tool printed Secret manifests despite dry-run hiding.
git grep -n 'password\|token\|privateKey' -- values
ResolutionRevoke exposed material, purge artifacts under incident procedure, move delivery outside Helm values, and add redaction and policy tests.
A Pod cannot find the externally managed Secret.
Likely causeSecret workflow, provider identity, CSI mount, controller sync, namespace, name, key, refresh, or ordering is wrong.
kubectl -n {{releaseNamespace}} get pod,secretkubectl -n {{releaseNamespace}} describe pod
ResolutionRepair the chosen provider-specific workflow without printing content; define whether workloads wait, fail closed, or use last-known material.
helm upgrade times out while Pods are eventually healthy.
Likely causeTimeout is shorter than measured scheduling, image pull, migration, readiness, hook, or load-balancer convergence.
kubectl -n {{releaseNamespace}} get pod,job,events --sort-by=.lastTimestamphelm status {{releaseName}} -n {{releaseNamespace}}
ResolutionFix the slow or blocked dependency and set a measured bounded timeout; do not make timeout infinite.
Release is deployed but the application returns errors.
Likely causeKubernetes readiness is too shallow, dependency or schema is incompatible, configuration is wrong, or SLO checks were omitted.
kubectl -n {{releaseNamespace}} get deploy,pod,svc,endpointslicekubectl -n {{releaseNamespace}} logs deploy/{{releaseName}} --since=15m
ResolutionStop promotion, preserve evidence, run application checks, and choose safe config revert, feature flag, or compatible Helm rollback.
A pre-upgrade hook Job never completes.
Likely causeHook image, command, RBAC, network, secret, dependency, timeout, or deletion policy is wrong.
kubectl -n {{releaseNamespace}} get job,podkubectl -n {{releaseNamespace}} describe job
ResolutionInspect bounded logs, repair hook ownership, and do not delete a potentially active migration job until its side effects are known.
Rollback succeeds in Helm but old Pods crash.
Likely causeDatabase schema, external API, Secret format, image, PVC, or CRD is no longer backward compatible.
helm history {{releaseName}} -n {{releaseNamespace}}kubectl -n {{releaseNamespace}} get pod,job
ResolutionUse the application recovery plan, compatible forward fix, expand-contract migration, or data restore; Helm status cannot repair incompatible data.
Rollback leaves resources introduced by the failed revision.
Likely causeHooks, CRDs, external resources, keep annotations, or objects outside release ownership are not removed by ordinary rollback.
helm get manifest {{releaseName}} -n {{releaseNamespace}} --revision {{previousRevision}}kubectl -n {{releaseNamespace}} get all
ResolutionDiff against reviewed revisions, identify owner, and remove only proven orphan resources through their lifecycle procedure.
Helm refuses an immutable field change.
Likely causeThe chart changes a selector, Service field, StatefulSet field, PVC, or another immutable API property.
helm upgrade --install {{releaseName}} {{chartReference}} --version {{chartVersion}} -n {{releaseNamespace}} -f {{baseValues}} -f {{environmentValues}} --dry-run=server --hide-secret
ResolutionDesign a migration or replacement with traffic and data safety. Do not use force replacement until deletion effects are explicitly accepted.
Helm test passes while users still fail.
Likely causeThe test checks only internal connectivity, stale fixtures, or a path that bypasses ingress, identity, data, and external dependencies.
helm test {{releaseName}} -n {{releaseNamespace}} --logskubectl -n {{releaseNamespace}} get pod -l helm.sh/hook=test
ResolutionExpand tests to owner-approved transactions and negative assertions while keeping them non-destructive and bounded.
Upgrade adopts or conflicts with objects managed elsewhere.
Likely causeRelease labels, annotations, server-side field owners, or duplicate deployment systems overlap.
kubectl -n {{releaseNamespace}} get deploy {{releaseName}} -o yamlhelm get manifest {{releaseName}} -n {{releaseNamespace}}
ResolutionStop, identify the authoritative owner, and perform an explicit adoption or handoff procedure rather than forcing conflicts.
Recovery
Rollback
Stop promotion, preserve sanitized evidence, verify the previous revision remains compatible with current data and external state, then create an explicit Helm rollback revision with bounded wait or use the application's forward-recovery plan when Helm rollback is unsafe.
- Record failed revision, chart, application image, hook and migration state, events, SLO, schema, queues, external side effects, and current release history without printing Secret content.
- Block further traffic promotion and new jobs while preserving enough service to avoid multiplying the incident.
- Confirm {{previousRevision}} artifacts still exist and its application supports the current schema, Secret format, CRDs, queues, caches, and external API contracts.
- If compatible, run `helm rollback {{releaseName}} {{previousRevision}} -n {{releaseNamespace}} --wait=watcher --timeout {{upgradeTimeout}} --cleanup-on-fail` under change authority.
- If incompatible, do not force Helm rollback; execute the application-owned compatible forward fix, feature-flag disablement, traffic shift, or data recovery procedure.
- Repeat rollout, image, Helm test, application, SLO, data, dependency, secret, negative security, and external side-effect checks.
- Revoke any exposed credential and inspect release-storage access. Document root cause and add render, fixture, policy, migration, or canary regression tests before retry.
Evidence