Automate Kubernetes certificates with cert-manager and ACME
Install a pinned cert-manager release, prove webhook and controller health, validate ACME staging through HTTP-01, promote one bounded certificate to production, design least-privilege DNS-01 for wildcards, and operate renewal without exposing private keys.
Deliver repeatable, observable, least-privilege public certificate issuance and renewal for Kubernetes workloads while keeping staging, production, solver credentials, TLS Secrets, ingress presentation, and rollback as separately verifiable controls.
- Kubernetes 1.36, current supported
- cert-manager 1.21.0, current stable
- ACME RFC 8555
- Cluster administration Use a reviewed kubeconfig that can install cluster-scoped CRDs, RBAC, admission webhooks, and ClusterIssuers; keep a second administrator and API-server recovery path available.
kubectl auth can-i create customresourcedefinitions.apiextensions.k8s.io && kubectl auth can-i create clusterissuers.cert-manager.io - Owned public DNS Control {{publicHostname}}, authoritative DNS, and every A and AAAA target; document the approved public ingress endpoint and any CDN or proxy in front of it.
dig +trace {{publicHostname}} - Ingress and network path Provide a working {{ingressClass}}, public TCP 80 for HTTP-01, application TCP 443, API-server-to-webhook access, ingress-controller-to-solver access, and controller DNS/HTTPS egress.
kubectl get ingressclass {{ingressClass}} - Deployment ownership Identify the Helm or GitOps owner of namespaces, CRDs, RBAC, webhook configuration, issuers, Certificates, and Secrets; preserve the current release and rollback manifests.
- Secret policy Restrict read access to ACME account keys, DNS credentials, and TLS Secrets; use an approved credential-delivery mechanism and forbid private-key output in logs or support bundles.
- Rate-limit and lifecycle plan Use staging first, review ACME production rate limits, define renewal alerts and external certificate probes, and document the consuming application's Secret reload behavior.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A pinned, observable cert-manager installation whose CustomResourceDefinitions, controller, cainjector, webhook, startup checks, leader election, and namespace boundaries are verified before any public certificate is requested.
- Separate ACME staging and production ClusterIssuers with an HTTP-01 path for ordinary public hostnames, an optional least-privilege DNS-01 path for wildcard names, and a deliberate promotion gate between the two environments.
- A certificate lifecycle that proves issuance, Secret delivery, ingress consumption, renewal, failure diagnosis, monitoring, upgrade readiness, and rollback without treating a green Certificate condition as the only evidence.
- A staging Certificate reaches Ready, its corresponding Order and Challenge complete, the generated TLS Secret contains a matching key pair and expected SAN, and an external TLS client observes the staged chain on the intended hostname.
- After the exact same route and ownership checks pass, a production Certificate is issued by the production ACME endpoint and the workload presents the correct leaf, intermediate, hostname, validity window, and renewal schedule.
- An operator can distinguish ingress reachability, DNS delegation, ACME rate limiting, webhook admission, issuer account, RBAC, Secret, and application reload failures using concrete Kubernetes resources and provider-independent evidence.
Architecture
How the parts fit together
cert-manager is a Kubernetes controller suite, not a certificate authority and not a magic ingress feature. A Certificate declares desired key material and names. cert-manager creates a CertificateRequest, an ACME Order, and one or more Challenges, then coordinates a solver that proves control of each identifier to the configured ACME directory. HTTP-01 temporarily exposes a token through the selected ingress class and therefore depends on public DNS, port 80, ingress routing, NetworkPolicy, and load-balancer behavior. DNS-01 writes a TXT record through a provider credential and can issue wildcard certificates, but that credential is a material security boundary. On success, cert-manager stores the private key and returned chain in a namespaced Secret. The consuming ingress or workload must watch or reload that Secret independently. Staging and production issuers use different ACME accounts and endpoints so testing cannot consume production rate limits or create trusted certificates accidentally.
- Inventory public names, authoritative DNS, ingress class, load-balancer addresses, port 80 behavior, NetworkPolicies, existing Secrets, certificate consumers, and current ACME objects before installation.
- Install one reviewed cert-manager release from the official OCI Helm chart with CRDs explicitly managed, then prove API discovery, webhook reachability, controller leadership, and clean events.
- Create a staging issuer and wait for its ACME account registration to become Ready before requesting any Certificate.
- Request one narrow staging certificate. Observe CertificateRequest, Order, Challenge, temporary solver resources, events, and public validation path instead of retrying blindly.
- Inspect the resulting Secret and test it through the actual ingress endpoint. A valid Secret that is not presented is an application or routing problem, not an issuance problem.
- Create a separate production issuer only after staging passes, then issue the same bounded name set without changing solver semantics at the same time.
- Optionally configure DNS-01 for wildcards using a delegated validation zone or least-privilege provider identity; keep provider credentials outside manifests and logs.
- Rehearse renewal, alerts, controller upgrades, issuer withdrawal, Secret restoration, and rollback while a valid certificate still has enough lifetime for recovery.
Assumptions
- The cluster runs a supported Kubernetes release, uses RBAC, has a functional DNS service, and provides an ingress or Gateway implementation whose class and public endpoint are known. The guide does not install or select an ingress controller.
- The operator controls the requested public names and can inspect authoritative DNS. Example domains use documentation-safe placeholders and must never be copied into a production Certificate unchanged.
- For HTTP-01, the public ACME validator can reach the selected hostname on TCP port 80 and the ingress controller can reach temporary acmesolver Pods. Redirects, proxies, WAF rules, or NetworkPolicies have been identified.
- For DNS-01, the DNS provider supports an official cert-manager integration or webhook that is maintained and reviewed. Credentials can be restricted to the exact validation zone and stored through the cluster's approved secret-delivery mechanism.
- Helm ownership, CRD lifecycle, backup policy, GitOps reconciliation, admission controls, and change windows are known. This guide does not seize resources already managed by another release or tool.
- The application can reload or watch TLS Secrets. A maintenance or rollout procedure exists for consumers that read key material only at process start.
- Staging is mandatory. Production issuance is a promotion, not a troubleshooting technique, and no automation repeatedly deletes Orders, Challenges, accounts, or Secrets to force retries.
Key concepts
- Certificate
- A namespaced desired-state object describing DNS names, Secret name, private-key policy, duration, renewal timing, usages, and issuer reference. It is not the certificate bytes themselves.
- CertificateRequest
- An immutable request generated for a particular issuance attempt. It records approval, denial, issuer response, and failure conditions and is valuable evidence during diagnosis.
- Order
- The cert-manager representation of an ACME order. It binds the requested identifiers to one or more authorization challenges and exposes the remote CA state.
- Challenge
- The concrete ACME proof for one identifier. Its Presented, Processing, Reason, and State fields reveal whether cert-manager created the proof and whether the CA accepted it.
- Issuer versus ClusterIssuer
- An Issuer is namespaced; a ClusterIssuer can be referenced across namespaces. Cluster scope is convenient but broadens who may request certificates unless admission and RBAC controls constrain its use.
- HTTP-01
- An ACME challenge answered at a well-known HTTP path for a specific hostname. It cannot issue wildcard certificates and relies on public port 80 reachability.
- DNS-01
- An ACME challenge answered by a TXT record beneath _acme-challenge. It supports wildcard identifiers but places DNS mutation credentials in the issuance trust path.
- renewBefore
- The desired interval before expiry when cert-manager should start renewal. It creates recovery time; it does not guarantee the consumer has reloaded the renewed Secret.
- ACME staging
- A non-publicly trusted CA environment intended for integration testing. Its roots are not browser-trusted, but its workflow catches routing and ownership mistakes without production rate-limit pressure.
Before you copy
Values used in this guide
{{certManagerVersion}}Reviewed cert-manager release pinned in deployment configuration; never use an unreviewed floating chart version.
Example: v1.21.0{{acmeEmail}}Operational mailbox used for ACME account notices. It is not a secret but should be a maintained team address.
Example: pki-operations@example.net{{ingressClass}}Exact ingress class that owns the public validation route and application Ingress.
Example: nginx-public{{certificateNamespace}}Namespace containing the workload, Certificate, and resulting TLS Secret.
Example: payments-prod{{certificateName}}Stable Certificate resource name used in events and monitoring.
Example: api-example-net{{tlsSecretName}}secretNamespaced Secret written by cert-manager and consumed by the ingress or workload.
Example: api-example-net-tls{{publicHostname}}Public FQDN owned by the organization and routed to the selected ingress.
Example: api.example.net{{dnsZone}}Authoritative DNS zone or delegated ACME validation zone used only for the optional DNS-01 path.
Example: example.net{{dnsCredentialSecret}}secretSecret name containing a least-privilege DNS API credential delivered by the approved secret mechanism.
Example: dns01-api-credentialSecurity and production boundaries
- A ClusterIssuer can turn namespace-level write access into public certificate issuance for organization-owned names. Constrain issuer references, Certificate names, DNS zones, and Secret access with RBAC and admission policy.
- The ACME account private key and DNS provider credential are different secrets. Protect both, but recognize that a broad DNS credential can change production DNS beyond certificate validation.
- Never print tls.key, base64-decode it into terminal history, or include Secret YAML in tickets. Validate key matching through public-key fingerprints or certificate tooling that does not expose private material.
- HTTP-01 creates temporary Pods, Services, and ingress routes. Default-deny policy must permit only the necessary control-plane, ingress-controller, DNS, and ACME path without opening the application namespace broadly.
- Staging chains are intentionally untrusted. Do not disable TLS verification in production clients to accommodate them; promote to production only after staging evidence is complete.
- Back up declarative resources and issuer design, not private keys indiscriminately. If a TLS Secret is restored, treat its private key according to the organization's key-recovery and compromise policy.
Stop before continuing if
- Stop if the requested name is not controlled by the organization, authoritative DNS is ambiguous, or the public address points to an unrelated tenant or environment.
- Stop if the cert-manager CRDs are owned by a different release, GitOps controller, or version lifecycle than the planned Helm change.
- Stop if the API server cannot reach the webhook, the controller cannot reach the Kubernetes API or ACME directory, or the selected solver path is blocked by NetworkPolicy.
- Stop if a DNS credential can mutate the root zone broadly and no delegation or provider-side restriction is available.
- Stop if staging fails. Do not switch to production, delete the ACME account, or loop issuance requests as a workaround.
- Stop if the generated Secret's public key does not match the certificate, the SAN differs from the approved hostname set, or the consumer presents a different Secret.
- Stop if production rate-limit responses appear; preserve Orders and events, calculate the retry window, and correct the root cause before another request.
verification
Inventory the certificate, DNS, ingress, and current ownership path
Record the public resolution, ingress classes, TLS consumers, existing cert-manager APIs, NetworkPolicies, and current Secrets before introducing a controller.
Why this step matters
Certificate automation crosses DNS, networking, admission, Secret storage, and application reload boundaries. An inventory prevents the installation from taking ownership of resources or names managed by another release.
What to understand
Resolve A and AAAA records from an external resolver and compare them with the load-balancer addresses. A forgotten IPv6 record can direct ACME validation to an unrelated endpoint even when IPv4 tests pass.
List every ingress class and identify the exact controller that will own HTTP-01. A class name is configuration, not a generic label; using a default or omitted class may create solver routes on multiple controllers.
Search for existing Certificate, Issuer, ClusterIssuer, webhook, CRD, and Helm release objects. Record their owners and versions rather than assuming an empty command output means cert-manager was never installed.
Map the workload's TLS termination point and Secret reference. If a CDN or external load balancer terminates TLS before Kubernetes, issuance inside the cluster may not change the certificate users see.
System changes
- No cluster state changes; the operator stores a sanitized pre-change inventory with DNS answers, object ownership, versions, and approved hostname scope.
Syntax explained
kubectl get ... -A- Reads namespaced resources across the cluster so hidden issuer or certificate ownership is not missed.
dig A / AAAA- Checks both public IP families because ACME validation may select either advertised address.
Values stay on this page and are never sent or saved.
kubectl version && helm version && kubectl get ingressclass && kubectl get crd | grep cert-manager || true && kubectl get certificate,issuer,clusterissuer -A 2>/dev/null || true && dig +short {{publicHostname}} A && dig +short {{publicHostname}} AAAAClient Version: v1.36.0 Kustomize Version: v5.7.1 NAME CONTROLLER PARAMETERS AGE nginx-public k8s.io/ingress-nginx <none> 420d 203.0.113.20
Checkpoint: One documented validation path
kubectl get ingressclass && dig +short {{publicHostname}} AContinue when{{ingressClass}} exists and {{publicHostname}} resolves only to approved public endpoints.
Stop whenThe hostname is unowned, resolves elsewhere, existing cert-manager resources have an unknown owner, or no supported ingress class is identified.
If this step fails
The hostname resolves differently inside and outside the cluster.
Likely causeSplit-horizon DNS or search-domain behavior is masking the authoritative public path.
dig +trace {{publicHostname}}dig @1.1.1.1 {{publicHostname}} A
ResolutionTreat the authoritative public answer as the ACME path and reconcile split-horizon records before installing a solver.
Security notes
- Do not export existing TLS Secrets as part of inventory. Record safe metadata, explicit secretTemplate labels, certificate fingerprints, and expiry without printing private keys. Certificate owner references are optional and disabled by default.
Alternatives
- Use a namespaced proof-of-concept cluster if production ownership cannot be established safely.
Stop conditions
- Do not continue when the public hostname or ingress endpoint belongs to another team without an approved change.
command
Install one pinned cert-manager release with explicit CRD ownership
Add the official OCI chart at a reviewed version, install it in its dedicated namespace, and wait for all deployments instead of accepting Helm release success alone.
Why this step matters
Pinning the chart and declaring CRD handling creates a reviewable, reproducible lifecycle. Helm's deployed status is necessary but does not by itself prove webhook reachability or controller reconciliation.
What to understand
Review the release notes and supported Kubernetes versions before selecting {{certManagerVersion}}. A version variable is intentionally explicit so a future upgrade becomes a diff rather than an invisible latest-tag change.
CRDs are cluster-scoped and survive namespaced workload changes. Decide whether Helm, GitOps, or another owner manages them, and never let two systems apply incompatible definitions concurrently.
The dedicated namespace separates controller workloads and policy. It does not protect cluster-wide API objects or issuer capability automatically; RBAC and admission policy remain required.
`--wait` covers ready workload resources observed by Helm, while subsequent steps test APIs, admission, events, and leader election independently.
System changes
- Creates the cert-manager namespace, cluster-scoped CRDs and RBAC, webhook configuration, Services, Deployments, and release metadata.
- Introduces controllers capable of reconciling cert-manager resources but does not create an ACME account or certificate yet.
Syntax explained
--version {{certManagerVersion}}- Pins the reviewed chart and application release; do not omit this in production automation.
--set crds.enabled=true- Makes CRD installation an explicit chart decision. Match this with the repository's documented ownership model.
--wait --timeout=10m- Waits for chart resources with a bounded timeout instead of returning immediately while controllers are unavailable.
Values stay on this page and are never sent or saved.
helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager --version {{certManagerVersion}} --namespace cert-manager --create-namespace --set crds.enabled=true --wait --timeout=10m && helm -n cert-manager status cert-managerRelease "cert-manager" does not exist. Installing it now. NAME: cert-manager NAMESPACE: cert-manager STATUS: deployed REVISION: 1 TEST SUITE: None
Checkpoint: Pinned release is deployed once
helm -n cert-manager list && kubectl -n cert-manager get deploy,podContinue whenOne release at {{certManagerVersion}} and ready controller, cainjector, webhook, and startup API check components.
Stop whenHelm reports an ownership conflict, any deployment is unavailable, or CRDs show an unexpected managing release.
If this step fails
Helm reports that a CRD or ClusterRole already exists and cannot be adopted.
Likely causeA prior installation or GitOps owner created cluster-scoped resources with different ownership metadata.
helm list -A | grep cert-managerkubectl get crd certificates.cert-manager.io -o yaml
ResolutionStop and reconcile ownership through the established deployment system; do not overwrite annotations simply to force adoption.
Security notes
- Review chart values and image registries under the organization's supply-chain policy. The official chart does not remove the need for registry allowlisting, signature policy, and version review.
Alternatives
- Use the official static manifests when Helm is not permitted, but pin the matching release and define a deliberate CRD upgrade procedure.
Stop conditions
- Stop on any cluster-scoped ownership conflict or unsupported Kubernetes-to-cert-manager version combination.
verification
Prove APIs, webhook admission, controller leadership, and policy paths
Inspect CRD discovery, workload health, logs, events, webhook endpoints, and a server-side dry run before creating an issuer.
Why this step matters
A ready Pod can still be unable to lead, admit resources, resolve DNS, or reach the API server. This checkpoint establishes platform health before remote ACME state complicates diagnosis.
What to understand
API discovery proves the intended v1 resources are served. Missing or stale resources point to CRD installation, aggregation, or version mismatch rather than issuer configuration.
Webhook Services need ready endpoints, valid serving certificates, DNS, and API-server connectivity. Managed control planes may reach cluster Services through a different network path than worker Pods.
Controller logs should show normal startup and leader acquisition without repeating forbidden, timeout, or conversion errors. Capture only bounded logs to keep troubleshooting evidence readable.
A server-side dry run exercises admission without persisting an object. It is safer than creating and deleting test issuers while the webhook path is uncertain.
System changes
- No persistent change; server-side dry run asks the API server and admission chain to validate a namespace object without storing it.
Syntax explained
--dry-run=server- Runs server admission and validation, unlike client dry run, while avoiding persisted state.
--tail=80- Bounds diagnostic logs so events and errors remain reviewable and sensitive data exposure is reduced.
kubectl api-resources --api-group=cert-manager.io && kubectl -n cert-manager get deploy,pod,svc,endpoints && kubectl -n cert-manager logs deploy/cert-manager --tail=80 && kubectl create namespace cert-manager-smoke --dry-run=server -o yamlNAME SHORTNAMES APIVERSION NAMESPACED KIND certificates cert cert-manager.io/v1 true Certificate clusterissuers cert-manager.io/v1 false ClusterIssuer namespace/cert-manager-smoke created (server dry run)
Checkpoint: Control plane can admit and controllers can reconcile
kubectl -n cert-manager get deploy && kubectl api-resources --api-group=acme.cert-manager.ioContinue whenAll deployments are Available and Order and Challenge APIs are served.
Stop whenWebhook endpoints are absent, dry run times out, logs repeat RBAC errors, or APIs differ from the reviewed release.
If this step fails
Pods are ready but resource creation times out.
Likely causeThe API-server-to-webhook network path is blocked or the webhook Service has unusable endpoints.
kubectl -n cert-manager get endpoints cert-manager-webhookkubectl -n cert-manager logs deploy/cert-manager-webhook --tail=120
ResolutionRestore the documented control-plane path and healthy endpoints; do not weaken all admission controls.
Security notes
- Do not expose the webhook publicly. Fix the private control-plane path defined by the cluster architecture.
Alternatives
- Run the same checks in a representative staging cluster first when production control-plane networking cannot be exercised safely.
Stop conditions
- No issuer should be created until both API discovery and webhook admission are deterministic.
config
Register a dedicated Let's Encrypt staging account
Create a staging ClusterIssuer using the reviewed HTTP-01 ingress class and wait for account registration to report Ready.
Why this step matters
The staging account tests ACME registration, outbound connectivity, issuer RBAC, account Secret creation, and solver configuration without producing publicly trusted certificates or consuming production limits.
What to understand
Use a distinct account-key Secret name so staging and production identities cannot be confused or accidentally shared.
`ingressClassName` explicitly selects one controller and is preferred over relying on whichever class happens to be default in a cluster.
ClusterIssuer is cluster-scoped while its account key is stored according to cert-manager's cluster resource namespace behavior. Protect access to controller and Secret storage accordingly.
Wait for the Ready condition and read its Reason and Message. A created object is not a registered ACME account.
System changes
- Creates a cluster-scoped issuer and causes cert-manager to register or recover a staging ACME account.
- Creates the referenced ACME account private-key Secret through controller reconciliation.
Syntax explained
server: ...acme-staging...- Selects the non-publicly trusted staging directory used for safe integration tests.
privateKeySecretRef- Names the Secret containing the ACME account identity, not the workload certificate private key.
ingressClassName- Routes temporary solver ingress resources through exactly {{ingressClass}}.
clusterissuer-staging.yamlValues stay on this page and are never sent or saved.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
email: {{acmeEmail}}
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-staging-account-key
solvers:
- http01:
ingress:
ingressClassName: {{ingressClass}}clusterissuer.cert-manager.io/letsencrypt-staging created NAME READY AGE letsencrypt-staging True 18s
Checkpoint: Staging account is Ready
kubectl wait --for=condition=Ready clusterissuer/letsencrypt-staging --timeout=120s && kubectl describe clusterissuer letsencrypt-stagingContinue whenReady=True with reason ACMEAccountRegistered and no repeated warning events.
Stop whenRegistration fails, the account Secret has an unexpected owner, or production directory appears anywhere in the manifest.
If this step fails
Issuer remains Not Ready with connection errors.
Likely causeController egress, DNS, proxy, CA trust, or system time prevents access to the staging directory.
kubectl -n cert-manager logs deploy/cert-manager --tail=140kubectl -n cert-manager get networkpolicy
ResolutionRestore approved DNS and HTTPS egress, verify time, and let the same issuer reconcile without cycling account keys.
Security notes
- The account key is sensitive even in staging because it controls that ACME account and provides diagnostic history. Do not print it.
Alternatives
- Use a namespaced Issuer when only one namespace should consume the account.
Stop conditions
- Do not create a Certificate until Ready=True is stable and its account-key Secret ownership is understood.
config
Request one narrow staging certificate
Create a Certificate for a single approved hostname, bounded duration and renewal window, then follow every generated resource.
Why this step matters
A single-name staging request minimizes ambiguity and creates the exact resource graph used in production. Explicit key rotation and renewal timing make lifecycle behavior reviewable.
What to understand
Place the Certificate in the same namespace as its consumer and Secret. Kubernetes Secrets cannot be referenced across namespaces without an explicit replication or gateway design.
ECDSA P-256 is efficient and widely supported, but the algorithm must match client compatibility requirements. Do not switch algorithms during production promotion unless that difference is intentional.
`rotationPolicy: Always` asks cert-manager to generate a new private key for reissuance. Applications must tolerate atomic Secret updates and reload the new pair.
A 90-day requested duration and 30-day renew-before interval give operational recovery time while remaining subject to the issuer's actual policy.
System changes
- Creates a Certificate and causes CertificateRequest, Order, Challenge, temporary solver, private key, and TLS Secret lifecycle changes in {{certificateNamespace}}.
Syntax explained
secretName- Names the Secret cert-manager owns for tls.crt and tls.key; it must match the workload reference exactly.
rotationPolicy: Always- Generates a fresh private key for reissuance rather than reusing the current key.
renewBefore: 720h- Schedules renewal approximately thirty days before expiry, subject to controller calculation and issuer response.
certificate-staging.yamlValues stay on this page and are never sent or saved.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: {{certificateName}}
namespace: {{certificateNamespace}}
spec:
secretName: {{tlsSecretName}}
secretTemplate:
labels:
pki.oneliners.guru/managed-by: cert-manager
duration: 2160h
renewBefore: 720h
privateKey:
algorithm: ECDSA
size: 256
rotationPolicy: Always
dnsNames:
- {{publicHostname}}
issuerRef:
name: letsencrypt-staging
kind: ClusterIssuercertificate.cert-manager.io/api-example-net created certificate.cert-manager.io/api-example-net condition met NAME READY SECRET AGE api-example-net True api-example-net-tls 54s
Checkpoint: Complete staging resource graph
kubectl -n {{certificateNamespace}} get certificate,certificaterequest,order,challenge && kubectl -n {{certificateNamespace}} describe certificate {{certificateName}}Continue whenCertificate Ready=True, CertificateRequest Ready=True, Order valid, Challenge valid, and Secret {{tlsSecretName}} exists.
Stop whenAny requested SAN is unapproved, resources cycle repeatedly, or production issuer is referenced.
If this step fails
Certificate stays Issuing and no Order appears.
Likely causeIssuer is not Ready, CertificateRequest is denied or unapproved, Secret creation is forbidden, or admission policy rejects the request.
kubectl -n {{certificateNamespace}} describe certificate {{certificateName}}kubectl -n {{certificateNamespace}} get certificaterequest -o yaml
ResolutionResolve the specific condition and RBAC or policy denial; do not delete the Certificate to hide the evidence.
Security notes
- Treat create or update permission on Certificate as certificate-issuance authority. Restrict it and validate allowed DNS suffixes.
Alternatives
- Create an Ingress annotation that generates a Certificate only when ingress-shim is intentionally governed; an explicit Certificate is easier to review.
Stop conditions
- Stop if the resource contains a wildcard or additional SAN not present in the approved scope.
verification
Inspect Order, Challenge, solver resources, and the public HTTP path
Preserve the staged issuance evidence and prove that DNS and public HTTP route the challenge to the intended temporary solver.
Why this step matters
Orders and Challenges are the best bridge between Kubernetes desired state and the remote CA. Reading them avoids misclassifying a network, DNS, or provider failure as a generic certificate problem.
What to understand
The Challenge's Presented and Processing fields show whether cert-manager created solver state. The Reason field typically preserves the failing self-check or ACME response.
Temporary resources should select the intended ingress class and disappear after successful authorization. Their absence during a pending Challenge can indicate admission or controller failure.
External DNS must point every published address at a path that can serve the token. Internal curl success is insufficient for a public ACME validator.
Preserve event timestamps and object names before cleanup. They connect controller logs to the exact issuance attempt and reduce destructive retry behavior.
System changes
- No intentional change; reads ACME and solver resources while cert-manager may naturally clean temporary resources after validation.
Syntax explained
-l acme.cert-manager.io/http01-solver=true- Selects only temporary HTTP-01 solver resources created for challenge presentation.
-o wide- Shows node, address, and state context useful for correlating the route without printing Secret data.
Values stay on this page and are never sent or saved.
kubectl -n {{certificateNamespace}} get order,challenge -o wide && kubectl -n {{certificateNamespace}} describe challenge && kubectl -n {{certificateNamespace}} get ingress,service,pod -l acme.cert-manager.io/http01-solver=true -o wide && dig +short {{publicHostname}} ANAME STATE AGE order.acme.cert-manager.io/api-example-net-1-2843911106 valid 47s NAME STATE DOMAIN AGE challenge.acme.cert-manager.io/api-example-net-1-2843911106-0 valid api.example.net 46s 203.0.113.20
Checkpoint: Staging challenge is valid with correct public DNS
kubectl -n {{certificateNamespace}} get challenge -o custom-columns=NAME:.metadata.name,STATE:.status.state,REASON:.status.reasonContinue whenThe Challenge reaches valid and public DNS resolves to the approved ingress endpoint.
Stop whenThe reason reports wrong content, an unrelated ingress class, an unapproved address, or repeated ACME retries.
If this step fails
The solver Service exists but has no endpoints.
Likely causeThe solver Pod is unschedulable, not Ready, selected incorrectly, or blocked by admission policy.
kubectl -n {{certificateNamespace}} describe pod -l acme.cert-manager.io/http01-solver=truekubectl -n {{certificateNamespace}} get endpoints -l acme.cert-manager.io/http01-solver=true
ResolutionResolve scheduling or admission at the solver Pod and verify endpoints before another ACME validation attempt.
Security notes
- Challenge tokens are short-lived proof material. Avoid publishing full support bundles unnecessarily, even though they are not the certificate private key.
Alternatives
- Use DNS-01 when public HTTP reachability is intentionally unavailable and DNS credentials can be scoped safely.
Stop conditions
- Do not promote when the public path was not independently verified or the Challenge succeeded through an unintended ingress.
verification
Validate the staged Secret and the certificate actually presented
Inspect certificate metadata without exposing the private key, compare the Secret with the external endpoint, and verify the intended ingress reference.
Why this step matters
Certificate Ready proves issuance state, not consumer state. Comparing safe certificate metadata at the Secret and real endpoint reveals stale references, reload failures, or TLS termination outside Kubernetes.
What to understand
Read only tls.crt and metadata. Never output tls.key. Certificate public data is sufficient to compare subject, SAN, issuer, serial, validity, and public-key fingerprint.
A staging issuer produces an intentionally untrusted chain, so the test checks identity and routing rather than suppressing verification warnings in application clients.
The Secret's enforced secretTemplate label and Certificate spec.secretName should agree. Owner references are not expected unless the controller was deliberately started with `--enable-certificate-owner-ref`; another writer or restore can still create a coherent-looking but unmanaged key pair.
Use SNI with `-servername`; without it, a shared ingress can present a default certificate and create a false diagnosis.
System changes
- No changes; reads public certificate bytes and metadata from the Secret and external TLS endpoint.
Syntax explained
jsonpath .data.tls\.crt- Selects only the public certificate field and deliberately excludes tls.key.
-servername {{publicHostname}}- Sends TLS SNI so a multi-host ingress selects the certificate for the requested name.
Values stay on this page and are never sent or saved.
kubectl -n {{certificateNamespace}} get secret {{tlsSecretName}} -o jsonpath='{.type}{"\n"}{.metadata.labels.pki\.oneliners\.guru/managed-by}{"\n"}' && kubectl -n {{certificateNamespace}} get secret {{tlsSecretName}} -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -issuer -dates -ext subjectAltName && openssl s_client -connect {{publicHostname}}:443 -servername {{publicHostname}} </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -dateskubernetes.io/tls
api-example-net
subject=CN=api.example.net
issuer=CN=(STAGING) Artificial Apricot R3
notBefore=Jul 28 11:02:00 2026 GMT
notAfter=Oct 26 11:01:59 2026 GMT
X509v3 Subject Alternative Name:
DNS:api.example.netCheckpoint: Secret and endpoint present the same approved staged identity
openssl s_client -connect {{publicHostname}}:443 -servername {{publicHostname}} </dev/null 2>/dev/null | openssl x509 -noout -serial -ext subjectAltNameContinue whenSAN contains only approved names and the serial matches the staged Secret after the consumer reloads.
Stop whenThe key is exposed, SAN is wrong, the managed label is absent, an unexpected owner reference appears, or the endpoint presents another certificate.
If this step fails
The Secret is correct but the endpoint presents a different serial.
Likely causeIngress references a different Secret, the consumer has not reloaded, DNS reaches another environment, or TLS terminates upstream.
kubectl -n {{certificateNamespace}} get ingress -o yamldig +short {{publicHostname}} A
ResolutionTrace the TLS termination and supported reload path. Reissuing the already-correct Secret will not repair routing.
Security notes
- Pipe only the public certificate into OpenSSL. Any workflow that decodes tls.key into a terminal or temporary world-readable file is unacceptable.
Alternatives
- Use a certificate monitoring probe from outside the cluster to compare serial, SAN, chain, and expiry continuously.
Stop conditions
- Production promotion is blocked until the staged Secret and presented endpoint converge.
config
Create a separate production ACME account after staging acceptance
Duplicate the reviewed solver semantics while changing only the account Secret and ACME directory to production.
Why this step matters
Production should differ from the successful staged configuration only where trust environment requires it. Simultaneous solver, ingress, DNS, and account changes make failures hard to attribute.
What to understand
Use a distinct account-key Secret so staging operations cannot rotate or overwrite production account identity.
Review production rate limits and coordinate registered-domain issuance across clusters and teams before the first request.
Wait for issuer Ready and preserve its conditions. Account registration success does not authorize any hostname; each Certificate still undergoes ACME validation.
Keep production issuer creation under a controlled change with narrow RBAC. Namespace users should not gain unexpected capability to request arbitrary organizational names.
System changes
- Creates a production ClusterIssuer and a production ACME account private-key Secret.
- Introduces publicly trusted issuance capability but does not issue a certificate until a Certificate references it.
Syntax explained
https://acme-v02.api.letsencrypt.org/directory- Selects the Let's Encrypt production directory; this exact difference requires explicit approval.
letsencrypt-production-account-key- Separates production ACME account identity from staging.
clusterissuer-production.yamlValues stay on this page and are never sent or saved.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-production
spec:
acme:
email: {{acmeEmail}}
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-production-account-key
solvers:
- http01:
ingress:
ingressClassName: {{ingressClass}}clusterissuer.cert-manager.io/letsencrypt-production created NAME READY AGE letsencrypt-production True 17s
Checkpoint: Production account registered without other drift
kubectl wait --for=condition=Ready clusterissuer/letsencrypt-production --timeout=120s && kubectl get clusterissuer letsencrypt-staging letsencrypt-production -o yamlContinue whenBoth issuers are Ready and solver configuration is equivalent apart from directory and account Secret.
Stop whenStaging was not accepted, production issuer is not Ready, or solver configuration differs unexpectedly.
If this step fails
Production issuer is Not Ready while staging remains Ready.
Likely causeProduction directory access, account registration, email policy, or account Secret state differs.
kubectl describe clusterissuer letsencrypt-productionkubectl -n cert-manager logs deploy/cert-manager --tail=140
ResolutionCorrect the production-specific registration cause without changing the proven solver path or deleting the staging evidence.
Security notes
- Production ACME account access and ClusterIssuer mutation are privileged. Audit both through Kubernetes and deployment-system logs.
Alternatives
- Use a dedicated production Issuer per namespace or DNS zone when blast-radius reduction outweighs account consolidation.
Stop conditions
- Do not point any Certificate at an issuer that is not stably Ready.
config
Promote the proven Certificate to production and verify trust
Change only issuerRef, wait for reissuance, and validate the new public chain, SAN, Secret revision, application reload, and external trust.
Why this step matters
A narrow issuerRef promotion reuses the validated route while producing the publicly trusted chain. End-to-end verification confirms the consumer loaded the new Secret rather than merely observing Certificate Ready.
What to understand
Apply through the owning GitOps or deployment path so the desired state does not revert. A direct kubectl edit can be overwritten immediately by reconciliation.
Record the previous Secret resourceVersion and certificate serial, then watch for a new CertificateRequest and changed public certificate without exposing the key.
Validate with an external client using normal trust verification. A return code of zero and correct hostname are both required.
Inspect the production Order and Challenge once. Even though staging succeeded, production validators and rate-limit state are distinct.
System changes
- Changes Certificate issuerRef, creates a production CertificateRequest, Order, and Challenge, and rotates the private key and TLS Secret.
- Causes the ingress or application to reload publicly trusted key material if its Secret integration is healthy.
Syntax explained
issuerRef.name: letsencrypt-production- The sole intended promotion difference selecting the production ACME account.
rotationPolicy: Always- Ensures production issuance does not reuse the staging private key.
certificate-production.yamlValues stay on this page and are never sent or saved.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: {{certificateName}}
namespace: {{certificateNamespace}}
spec:
secretName: {{tlsSecretName}}
secretTemplate:
labels:
pki.oneliners.guru/managed-by: cert-manager
duration: 2160h
renewBefore: 720h
privateKey:
algorithm: ECDSA
size: 256
rotationPolicy: Always
dnsNames:
- {{publicHostname}}
issuerRef:
name: letsencrypt-production
kind: ClusterIssuercertificate.cert-manager.io/api-example-net configured certificate.cert-manager.io/api-example-net condition met issuer=C=US, O=Let's Encrypt, CN=R13 Verify return code: 0 (ok)
Checkpoint: Externally trusted production certificate
kubectl -n {{certificateNamespace}} wait --for=condition=Ready certificate/{{certificateName}} --timeout=10m && openssl s_client -connect {{publicHostname}}:443 -servername {{publicHostname}} -verify_return_error </dev/nullContinue whenCertificate Ready=True; external handshake verifies, SAN equals {{publicHostname}}, and issuer is the expected production chain.
Stop whenA rate limit occurs, SAN changes, Secret writer conflicts, trust verification fails, or endpoint serial remains stale.
If this step fails
Certificate becomes Ready but external verification still reports the staging issuer.
Likely causeThe consumer has not reloaded or the request reaches a different TLS termination point.
kubectl -n {{certificateNamespace}} get secret {{tlsSecretName}} -o jsonpath='{.metadata.resourceVersion}'openssl s_client -connect {{publicHostname}}:443 -servername {{publicHostname}} </dev/null 2>/dev/null | openssl x509 -noout -issuer -serial
ResolutionUse the supported consumer reload and trace DNS or upstream termination. Do not request another production certificate.
Security notes
- A successful public certificate can expose an unplanned hostname as trustworthy. Confirm application authorization and exposure separately from TLS validity.
Alternatives
- Issue to a parallel Secret and switch the consumer after validation when the application supports atomic blue/green TLS references.
Stop conditions
- Stop if production issuance creates any name or trust path outside the reviewed staging scope.
verification
Rehearse renewal, Secret reload, and expiry monitoring
Record calculated renewal time, trigger one controlled staging renewal rehearsal, then prove the consumer and external monitor observe the replacement.
Why this step matters
Automation is only credible when renewal and consumption are observed before an emergency. A bounded rehearsal validates scheduler, issuer, solver, Secret rotation, and application reload as one lifecycle.
What to understand
Use manual renewal sparingly and preferably with staging during initial validation. Repeated production renewals can consume rate limits and hide scheduler problems.
Record status.renewalTime rather than calculating only from notAfter; cert-manager incorporates the requested duration and renewBefore behavior.
Watch the Secret resourceVersion and certificate serial, then test the external endpoint. A new Secret with an old presented certificate reveals a consumer reload failure.
Monitoring must alert early enough to diagnose DNS, ingress, credentials, and rate limits. A last-day expiry alert is not an operational renewal plan.
System changes
- Triggers one additional issuance and key rotation for the selected Certificate.
- Updates the TLS Secret and may cause a consumer reload or rollout depending on integration.
Syntax explained
cmctl renew- Requests reissuance through cert-manager's supported CLI without deleting Certificate or Secret resources.
--sort-by=.metadata.creationTimestamp- Orders CertificateRequests chronologically so the new attempt can be correlated without deleting history.
Values stay on this page and are never sent or saved.
kubectl -n {{certificateNamespace}} get certificate {{certificateName}} -o custom-columns=NAME:.metadata.name,READY:.status.conditions[-1].status,NOTAFTER:.status.notAfter,RENEWAL:.status.renewalTime && cmctl renew -n {{certificateNamespace}} {{certificateName}} && kubectl -n {{certificateNamespace}} get certificaterequest --sort-by=.metadata.creationTimestampNAME READY NOTAFTER RENEWAL api-example-net True 2026-10-26T11:01:59Z 2026-09-26T11:01:59Z Manually triggered issuance of Certificate payments-prod/api-example-net
Checkpoint: Renewal changes serial and external presentation
kubectl -n {{certificateNamespace}} get certificate {{certificateName}} -o yaml && openssl s_client -connect {{publicHostname}}:443 -servername {{publicHostname}} </dev/null 2>/dev/null | openssl x509 -noout -serial -datesContinue whenA newer CertificateRequest succeeds, Secret revision changes, external serial follows, and renewalTime advances.
Stop whenAnother renewal is already running, production limits are near, consumer cannot reload, or the request changes SAN.
If this step fails
Secret rotates but workload connections fail.
Likely causeThe consumer read a partially managed file path, does not support reload, or expects a different key algorithm or chain.
kubectl -n {{certificateNamespace}} describe certificate {{certificateName}}kubectl -n {{certificateNamespace}} get pod -o wide
ResolutionUse the application's documented atomic reload or controlled rollout and validate compatibility in staging before another production rotation.
Security notes
- Do not make private-key content part of renewal monitoring. Serial, SAN, issuer, fingerprint, validity, and Secret metadata provide sufficient evidence.
Alternatives
- Use an isolated staging hostname for frequent renewal rehearsals and keep production tests limited to planned lifecycle checks.
Stop conditions
- Do not repeat manual renewal to make a failing attempt disappear; diagnose the first preserved request.
config
Add DNS-01 only for wildcard or non-HTTP requirements
Design a delegated or least-privilege DNS solver, deliver its credential without committing it, and validate authoritative TXT propagation in staging first.
Why this step matters
DNS-01 is required for wildcard certificates and useful when public HTTP is impossible, but its credential can become more powerful than the certificates it issues. Provider selection and delegation are security design decisions.
What to understand
The concrete example uses cert-manager's maintained Cloudflare API-token integration. If another provider owns {{dnsZone}}, replace the whole provider block with that provider's current official schema rather than renaming fields.
Prefer a provider token limited to TXT changes in a delegated _acme-challenge zone. Do not grant account-wide DNS administration for convenience.
Deliver the credential with the approved secret manager or sealed workflow. Never place a literal token in Git, shell history, screenshots, events, or guide variables.
Use solver selectors so only the intended DNS zone reaches this credential. Exact dnsNames and dnsZones precedence must be reviewed when multiple solvers coexist.
Authoritative propagation is the evidence that matters. Recursive caches can show stale or split-horizon data and must not be the only check.
System changes
- Adds a DNS-01 solver to an issuer and authorizes cert-manager to create temporary TXT records through a provider credential.
- May create wildcard certificate Secrets whose compromise affects every matching subdomain, increasing key blast radius.
Syntax explained
apiTokenSecretRef- References a namespaced credential Secret; the literal token never belongs in the issuer manifest.
selector.dnsZones- Limits when this solver is selected. Confirm the provider zone and delegated validation zone semantics.
_acme-challenge delegation- Moves validation TXT authority to a narrower zone so the credential need not control ordinary production records.
clusterissuer-dns01-fragment.yamlValues stay on this page and are never sent or saved.
spec:
acme:
solvers:
- dns01:
cloudflare:
apiTokenSecretRef:
name: {{dnsCredentialSecret}}
key: api-token
selector:
dnsZones:
- {{dnsZone}}challenge.acme.cert-manager.io/wildcard-example-net-1-1284729105-0 presented _acme-challenge.example.net. 60 IN TXT "2hJ9yM3uR8-documentation-token" certificate.cert-manager.io/wildcard-example-net condition met
Checkpoint: Staging DNS-01 proves least-privilege authoritative mutation
kubectl -n {{certificateNamespace}} describe challenge && dig +trace TXT _acme-challenge.{{publicHostname}}Continue whenChallenge becomes valid, authoritative servers agree on the temporary TXT record, and provider audit logs show only the approved validation zone.
Stop whenCredential has broad zone access, provider schema is guessed, delegation is inconsistent, or production is selected before staging succeeds.
If this step fails
cert-manager reports permission denied while creating the TXT record.
Likely causeCredential scope, zone identifier, delegated name, or provider API endpoint does not match the selected solver.
kubectl -n {{certificateNamespace}} describe challengekubectl -n cert-manager logs deploy/cert-manager --tail=160
ResolutionCorrect the exact zone mapping or provider permission. Do not grant global DNS administration as a shortcut.
Security notes
- A wildcard private key and a broad DNS API token have large blast radii. Use separate namespaces, issuer policy, short credential rotation, and audit logs.
Alternatives
- Issue explicit hostnames with HTTP-01 when wildcard convenience does not justify DNS credential exposure.
- Delegate validation records to a dedicated zone controlled by a narrow automation account.
Stop conditions
- Do not continue with an unbounded provider credential or an unmaintained webhook integration.
instruction
Operationalize monitoring, upgrades, evidence retention, and withdrawal
Define alerts, dashboards, controller and CRD upgrade ownership, issuer review, Secret consumer inventory, and a tested withdrawal sequence.
Why this step matters
Certificate automation becomes infrastructure only when expiration, reconciliation, public presentation, controller compatibility, and rollback are continuously visible. Installation success is the beginning of operations.
What to understand
Alert on both Kubernetes conditions and external presented certificates. Either source alone can be green while the other is stale.
Track review dates for issuers, solver credentials, DNS delegations, ingress classes, and namespace authorization. Orphaned issuers quietly preserve issuance power.
Upgrade controller, webhook, and CRDs as one reviewed compatibility change. Read every intermediate release note when crossing versions and keep a tested previous manifest.
Retain sanitized CertificateRequest, Order, Challenge, event, and controller evidence for incidents while excluding private keys and provider tokens.
Document withdrawal order: stop new references, preserve valid Secrets, remove issuer capability, revoke credentials where appropriate, and uninstall controllers only when no custom resources depend on them.
System changes
- No direct changes; establishes monitoring, review, upgrade, incident, and decommissioning controls around the deployed certificate platform.
Syntax explained
custom-columns- Produces an operator-readable inventory of readiness and renewal dates without printing Secret data.
helm history- Shows the release lineage needed to map controller and CRD compatibility during rollback.
kubectl -n cert-manager get deploy,pod && kubectl get certificate -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[-1].status,NOTAFTER:.status.notAfter,RENEWAL:.status.renewalTime && helm -n cert-manager history cert-managerNAMESPACE NAME READY NOTAFTER RENEWAL payments-prod api-example-net True 2026-10-26T11:01:59Z 2026-09-26T11:01:59Z REVISION UPDATED STATUS CHART DESCRIPTION 1 Tue Jul 28 10:18:00 2026 superseded cert-manager-v1.20.2 Upgrade complete 2 Tue Jul 28 12:40:00 2026 deployed cert-manager-v1.21.0 Upgrade complete
Checkpoint: Every certificate has an owner and two independent expiry signals
kubectl get certificate -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.conditions[-1].status,NOTAFTER:.status.notAfterContinue whenAll production Certificates have Ready=True, future expiry, an owning team, and an external presented-certificate probe.
Stop whenAny certificate is orphaned, has no external probe, is within the emergency window, or depends on an unsupported controller version.
If this step fails
Kubernetes status is healthy while external expiry alert fires.
Likely causeA stale ingress, CDN, load balancer, DNS target, or consumer reload path presents an older certificate.
openssl s_client -connect {{publicHostname}}:443 -servername {{publicHostname}} </dev/null 2>/dev/null | openssl x509 -noout -serial -dateskubectl -n {{certificateNamespace}} get secret {{tlsSecretName}} -o jsonpath='{.metadata.resourceVersion}'
ResolutionTrace the external termination and reload path; preserve the healthy Secret and avoid unnecessary reissuance.
Security notes
- Metrics and logs must never export Secret values or DNS credentials. Certificate metadata is generally enough for operations.
Alternatives
- Use an organization PKI platform for centralized policy, but keep Kubernetes consumer and presentation checks locally observable.
Stop conditions
- Do not uninstall CRDs while Certificate, Issuer, Order, Challenge, or dependent admission policy resources remain.
Finish line
Verification checklist
kubectl -n cert-manager get deploy,pod,svc,endpoints && kubectl api-resources --api-group=cert-manager.ioController, cainjector, webhook, API startup check, Services, endpoints, and v1 APIs are healthy for the pinned release.kubectl get clusterissuer letsencrypt-staging letsencrypt-production -o custom-columns=NAME:.metadata.name,READY:.status.conditions[-1].status,SECRET:.spec.acme.privateKeySecretRef.name,SERVER:.spec.acme.serverBoth issuers are Ready and have distinct account Secrets and staging versus production directory URLs.kubectl -n {{certificateNamespace}} get certificate,certificaterequest,order,challenge{{certificateName}} is Ready, the latest request is Ready, Order and Challenge are valid, and no objects cycle repeatedly.kubectl -n {{certificateNamespace}} get secret {{tlsSecretName}} -o jsonpath='{.type}{" "}{.metadata.labels.pki\.oneliners\.guru/managed-by}{"\n"}'The Secret is kubernetes.io/tls, carries the enforced cert-manager label from secretTemplate, matches Certificate.spec.secretName, and tls.key is never printed.openssl s_client -connect {{publicHostname}}:443 -servername {{publicHostname}} -verify_return_error </dev/nullNormal trust verification succeeds and the presented certificate contains exactly the approved SAN, production chain, and expected validity.kubectl -n {{certificateNamespace}} get certificate {{certificateName}} -o custom-columns=READY:.status.conditions[-1].status,NOTAFTER:.status.notAfter,RENEWAL:.status.renewalTimeReady=True, renewalTime provides the planned recovery window, and external monitoring tracks presented expiry independently.Recovery guidance
Common problems and safe checks
Creating any cert-manager resource fails with a webhook timeout.
Likely causeThe Kubernetes API server cannot resolve or reach the webhook Service, endpoints are absent, the serving certificate is invalid, or a NetworkPolicy blocks control-plane traffic.
kubectl -n cert-manager get deploy,pod,svc,endpointskubectl -n cert-manager logs deploy/cert-manager-webhook --tail=120kubectl get apiservice | grep cert-managerkubectl get events -A --sort-by=.lastTimestamp | tail -n 80
ResolutionRestore healthy webhook endpoints and the documented control-plane path. Do not disable validation globally; prove a server-side dry run succeeds before creating issuers.
The ClusterIssuer remains Not Ready with ACME account registration errors.
Likely causeThe directory URL is wrong, outbound DNS or HTTPS is blocked, system time is inaccurate, the email is rejected, or the account-key Secret cannot be created.
kubectl describe clusterissuer letsencrypt-stagingkubectl -n cert-manager logs deploy/cert-manager --tail=160kubectl -n cert-manager get networkpolicykubectl get --raw=/readyz?verbose
ResolutionCorrect the directory, network, time, or RBAC cause and allow reconciliation. Preserve the account Secret unless a documented account-key recovery decision requires replacement.
An HTTP-01 Challenge is presented but self-check reports a 404 response.
Likely causeThe wrong ingress class owns the temporary route, another ingress shadows it, the hostname resolves to a different load balancer, or a redirect/rewrite intercepts the path.
kubectl -n {{certificateNamespace}} describe challengekubectl -n {{certificateNamespace}} get ingress,service,pod -l acme.cert-manager.io/http01-solver=true -o widedig +short {{publicHostname}} Acurl -i --resolve {{publicHostname}}:80:203.0.113.20 http://{{publicHostname}}/.well-known/acme-challenge/test
ResolutionMake the selected class and public DNS converge on one intended endpoint, preserve the solver path from rewrites, and rerun the staging request only after external HTTP evidence is correct.
HTTP-01 works from inside the cluster but the ACME server cannot validate it.
Likely causePublic port 80 is blocked, DNS is split-horizon, an edge proxy differs from the internal path, IPv6 points elsewhere, or geographic anycast backends are inconsistent.
dig +trace {{publicHostname}}dig +short {{publicHostname}} Adig +short {{publicHostname}} AAAAkubectl -n {{certificateNamespace}} describe order
ResolutionTest from independent external networks, correct every published A and AAAA target, and ensure the challenge path reaches the same solver globally before retrying staging.
A DNS-01 Challenge remains pending although the provider reports a TXT write.
Likely causeThe record was written to the wrong zone, delegation is incomplete, propagation is slow, the authoritative servers disagree, or split-horizon resolvers hide the public value.
kubectl -n {{certificateNamespace}} describe challengedig +trace TXT _acme-challenge.{{publicHostname}}dig @ns1.example.net TXT _acme-challenge.{{publicHostname}}kubectl -n cert-manager logs deploy/cert-manager --tail=160
ResolutionRepair delegation or provider scope and wait for authoritative agreement. Do not increase credential privileges merely to bypass an incorrect zone mapping.
Certificate Ready is true but clients still see the previous certificate.
Likely causeThe ingress references another Secret or namespace, the controller has not reloaded, traffic reaches a different load balancer, or an upstream CDN terminates TLS.
kubectl -n {{certificateNamespace}} get ingress -o yamlkubectl -n {{certificateNamespace}} get secret {{tlsSecretName}} -o jsonpath='{.metadata.resourceVersion}'openssl s_client -connect {{publicHostname}}:443 -servername {{publicHostname}} -showcerts </dev/nullkubectl get ingressclass
ResolutionTrace the exact TLS termination point and Secret reference, then use the consumer's supported reload or rollout procedure. Do not reissue a certificate that already exists correctly.
Certificate issuance repeats and creates multiple failed CertificateRequests.
Likely causeThe desired spec is changing, the target Secret is deleted by another controller, issuer readiness flaps, or a GitOps template continuously mutates fields.
kubectl -n {{certificateNamespace}} get certificate,certificaterequest -o widekubectl -n {{certificateNamespace}} describe certificate {{certificateName}}kubectl -n {{certificateNamespace}} get events --sort-by=.lastTimestamp | tail -n 80kubectl -n {{certificateNamespace}} get secret {{tlsSecretName}} -o jsonpath='{.metadata.managedFields}'
ResolutionStop the conflicting reconciler or desired-state churn, restore one stable spec and Secret owner, and let cert-manager converge without deleting evidence.
Production ACME returns a rate-limit error.
Likely causeRepeated failed tests used production, duplicate certificates were requested, too many names changed at once, or another environment shares the registered domain limit.
kubectl -n {{certificateNamespace}} describe orderkubectl -n {{certificateNamespace}} get certificaterequest -o widekubectl -n cert-manager logs deploy/cert-manager --since=2hkubectl -n {{certificateNamespace}} get certificate {{certificateName}} -o yaml
ResolutionHonor the reported retry time, return diagnosis to staging, remove configuration churn, and coordinate issuance across environments. Do not rotate accounts to evade limits.
The TLS Secret exists but the private key and certificate do not match.
Likely causeA second writer replaced one key, a restore mixed revisions, or the consumer mounted a different Secret than the inspected object.
kubectl -n {{certificateNamespace}} describe certificate {{certificateName}}kubectl -n {{certificateNamespace}} get secret {{tlsSecretName}} -o jsonpath='{.metadata.ownerReferences}'kubectl -n {{certificateNamespace}} get events --field-selector involvedObject.name={{tlsSecretName}}kubectl -n {{certificateNamespace}} get pod -o yaml | grep -n {{tlsSecretName}}
ResolutionIdentify and stop the conflicting writer, preserve audit evidence, then let the approved Certificate controller produce a coherent Secret. Treat unexpected key replacement as a security event.
Renewal time passes but no new CertificateRequest appears.
Likely causeThe calculated renewal time differs from expectation, the Certificate is paused by an invalid spec, the controller is unavailable, or clock skew distorts scheduling.
kubectl -n {{certificateNamespace}} get certificate {{certificateName}} -o yamlkubectl -n cert-manager get podkubectl -n cert-manager logs deploy/cert-manager --since=6hkubectl get node -o custom-columns=NAME:.metadata.name,READY:.status.conditions[-1].status
ResolutionRead status.notBefore, status.notAfter, and status.renewalTime, restore controller health and time synchronization, then use one documented manual renewal rehearsal in staging.
The solver Pod is Running but the Challenge remains pending under default-deny policy.
Likely causeIngress-controller-to-solver traffic, solver DNS, webhook traffic, or cert-manager egress to ACME is not explicitly allowed.
kubectl get networkpolicy -Akubectl -n {{certificateNamespace}} get pod -l acme.cert-manager.io/http01-solver=true -o widekubectl -n {{certificateNamespace}} describe pod -l acme.cert-manager.io/http01-solver=truekubectl -n cert-manager logs deploy/cert-manager --tail=160
ResolutionAdd the narrow documented flows for the selected ingress and control-plane architecture, verify policy counters or connectivity, and keep unrelated namespace traffic denied.
After an upgrade, CRD conversion or webhook errors prevent reconciliation.
Likely causeThe controller and CRD versions are mismatched, Helm did not own CRD upgrades as expected, or unsupported release jumps skipped migration notes.
helm -n cert-manager history cert-managerkubectl get crd | grep cert-managerkubectl -n cert-manager get deploy -o widekubectl get events -A --sort-by=.lastTimestamp | tail -n 100
ResolutionStop further rollout, compare the pinned release with official upgrade notes, restore the previous compatible controller and CRD set through the owning deployment system, and validate all API objects before resuming.
Reference
Frequently asked questions
Should every application use one ClusterIssuer?
Not automatically. One ClusterIssuer simplifies operations but creates a broad issuance capability. Use namespaced Issuers or policy controls when tenant boundaries, DNS zones, accounts, or risk profiles differ.
Why test HTTP-01 when the Certificate manifest is simple?
The manifest covers only desired state. Successful issuance depends on external DNS, port 80, ingress class, solver resources, policy, ACME account state, and global routing that Kubernetes admission cannot prove.
Can I delete a failed Order to retry faster?
Preserve it until the reason is understood. Deletion discards evidence and can create another identical failure or consume rate limits. Correct the root cause and allow controlled reconciliation in staging.
Does Certificate Ready prove clients are safe?
No. It proves cert-manager stored issued material. External clients may reach another endpoint, a stale consumer, or a CDN. Verify the presented chain, SAN, validity, and key lifecycle at the real hostname.
Recovery
Rollback
Preserve the last valid TLS Secret and serving path, stop new issuance references, revert issuer or controller changes through their owning deployment system, and remove cluster-scoped capability only after every dependent object is inventoried.
- If production promotion fails but the staged or previous production Secret is still valid, stop further Certificate spec changes and restore the last reviewed manifest or parallel Secret reference through GitOps.
- Revert the consumer to the last known-good TLS Secret and use its supported reload or rollout procedure; verify external SNI, SAN, chain, and expiry before declaring service recovery.
- Suspend or remove new references to the affected production issuer while preserving CertificateRequest, Order, Challenge, event, and bounded controller-log evidence. Do not delete the ACME account key as a generic reset.
- For a failed cert-manager upgrade, restore the previous pinned controller and compatible CRD definitions using the documented owner. Validate API conversion and webhook admission before resuming reconciliation.
- Revoke or rotate a DNS provider credential immediately if exposure is suspected, remove its solver, and confirm provider audit logs and authoritative zones contain no unauthorized changes.
- Delete a ClusterIssuer only after all referencing Certificates are migrated and their valid Secrets are protected. Remove cert-manager workloads and CRDs only when no issuer, certificate, request, order, challenge, policy, or automation depends on them.
- After rollback, test public DNS, HTTP challenge path if retained, external TLS presentation, application health, certificate expiry alerts, and the ownership inventory against the pre-change baseline.
Evidence