OneLinersCommand workbench
Guides
Security / System Administration

Build a private CA with OpenSSL and issue service certificates

Create an offline root and constrained online intermediate with OpenSSL 3.5 LTS, issue inventory-approved SAN/EKU service leaves, operate revocation, and prove rotation plus full CA recovery.

360 min17 stepsHigh-impact changeRevision 1
Save or explore
Save to collectionCreate a collection in the sidebar first.
0 of 17 steps completed
Goal

Provide a small internal PKI whose key custody, issuer profiles, mutable state, service identity, revocation freshness, client validation, and disaster recovery are explicit and testable.

Supported environments
  • OpenSSL 3.5.7 LTS or later supported 3.5 patch
  • Linux Current supported enterprise distribution
Prerequisites
  • Approved small-CA scope Certificate population, concurrency, assurance, algorithms, names, lifetimes, revocation, and migration threshold are documented.
  • Offline and online signing boundaries Controlled root workstation, restricted intermediate host, encrypted storage, HSM decision, and single-writer workflow exist.
  • Inventory and distribution Authoritative service/DNS ownership, managed trust stores, CRL/AIA endpoints, monitoring, and reload owners exist.
  • Recovery custody Multiple custodians, separate passphrase shares, versioned media, signed manifests, and isolated recovery workstation are available.
Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System
  • A deliberately small private PKI using OpenSSL 3.5 LTS with an offline root, a path-length-zero online intermediate, serialized issuance state, and purpose-specific service certificates.
  • A complete lifecycle covering CSR review, SAN/EKU enforcement, deployment validation, CRL publication, bounded OCSP, revocation, rotation, state backup, and isolated disaster recovery.
Observable outcome
  • The offline root can sign only intermediates and routine leaf issuance never exposes the root key.
  • Every service leaf is CA:FALSE, hostname-valid, purpose-constrained, inventory-approved, short-lived, and verified on its live endpoint.
  • Revoked credentials are rejected by enforcing clients, while a complete restored CA state preserves serial and revocation history without a second active writer.

Architecture

How the parts fit together

An offline root key on encrypted controlled media signs only a five-year pathlen:0 intermediate. The restricted online intermediate serializes all signing and revocation against one text database, publishes daily CRLs, and optionally feeds a separately keyed responder. Workloads generate private keys locally and submit CSRs; the issuer applies authoritative SAN/EKU profiles. Managed trust stores receive root certificates and services present leaf plus intermediate chains.

Offline rootUltimate trust anchor used only in recorded ceremonies for intermediate issuance/revocation recovery.
Online intermediateSingle serialized leaf issuer and revocation authority with protected mutable state.
Service workloadGenerates/holds its private key, presents leaf plus intermediate, and reloads during rotation.
Trust distributionDeploys approved roots before dependent issuance and removes them after complete dependency retirement.
CRL/optional OCSPPublishes signed status with explicit freshness, availability, and client-enforcement semantics.
Recovery custodyStores versioned encrypted state, separated passphrase shares, manifests, and isolated restore evidence.
  1. A workload creates a private key and self-signed CSR containing requested identity; only the CSR crosses the boundary.
  2. An approver validates inventory/ownership; the intermediate applies its own CA:FALSE, SAN, key-usage, EKU, AIA, and CRL extensions.
  3. The service deploys leaf plus intermediate while clients trust the root and verify hostname/purpose/status.
  4. Revocation updates the authoritative index, produces a fresh signed CRL/status response, and is proven by a failing client check.

Assumptions

  • The PKI serves a small internal fleet with exactly one serialized OpenSSL CA writer and no public-browser trust requirement.
  • An isolated offline workstation, encrypted removable media, multi-person custody, stable HTTPS CRL/AIA endpoints, authoritative DNS/service inventory, and audit storage exist.
  • OpenSSL 3.5.7 LTS is available from a supported operating-system/vendor package, and consuming services support the chosen algorithms and status mechanism.
  • A dedicated CA/HSM platform has been considered and is selected instead when concurrency, HA, high volume, automated issuance, or compliance requirements exceed OpenSSL `ca` limitations.

Key concepts

Trust anchor
A root certificate explicitly trusted by a client; its self-signature does not create trust by itself.
Intermediate CA
A constrained issuer signed by the root so routine issuance does not expose the root key.
Serial/index database
OpenSSL CA mutable state recording issued/revoked certificates; it requires single-writer and atomic backup discipline.
SAN
Subject Alternative Name identifiers modern clients use for DNS/IP identity validation.
EKU
Extended Key Usage restricting a leaf to server, client, OCSP signing, or another purpose.
CRL
Signed list of revoked serials with lastUpdate/nextUpdate; clients must fetch and enforce it.
OCSP
Online per-certificate status protocol whose availability/freshness and client failure policy are separate operational guarantees.

Before you copy

Values used in this guide

{{caBase}}

Encrypted PKI filesystem mount.

Example: /srv/pki
{{rootDir}}

Offline root state directory.

Example: /srv/pki/root
{{rootConfig}}

Reviewed root OpenSSL configuration.

Example: /srv/pki/root/openssl-root.cnf
{{rootKey}}secret

Encrypted root private key or provider URI.

Example: /srv/pki/root/private/root-ca.key
{{rootCertificate}}

Public root trust-anchor certificate.

Example: /srv/pki/root/certs/root-ca.crt
{{rootManifest}}

Signed root state hash manifest.

Example: /srv/pki/root-manifest.sha256
{{intermediateDir}}

Online intermediate state directory.

Example: /srv/pki/intermediate
{{intermediateConfig}}

Reviewed intermediate OpenSSL configuration.

Example: /srv/pki/intermediate/openssl-intermediate.cnf
{{intermediateKey}}secret

Encrypted/HSM intermediate private key.

Example: /srv/pki/intermediate/private/intermediate-ca.key
{{intermediateCsr}}

Intermediate public CSR transferred to root.

Example: /srv/pki/intermediate/csr/intermediate-ca.csr
{{intermediateCertificate}}

Pathlen:0 intermediate CA certificate.

Example: /srv/pki/intermediate/certs/intermediate-ca.crt
{{intermediateCrl}}

PEM intermediate CRL.

Example: /srv/pki/intermediate/crl/intermediate-ca.crl.pem
{{intermediateCrlDer}}

DER CRL atomically published to clients.

Example: /srv/pki/intermediate/crl/intermediate-ca.crl
{{intermediateManifest}}

Signed complete intermediate state manifest.

Example: /srv/pki/intermediate-manifest.sha256
{{intermediateLock}}

Single-writer lock file shared by every intermediate signing and revocation operation.

Example: /srv/pki/intermediate/.openssl-ca.lock
{{publishedCrlPath}}

Same-filesystem web-root path replaced atomically for the public CRL URL.

Example: /var/www/pki/crl/service-ca.crl
{{trustBundle}}

Root plus required intermediate verification bundle.

Example: /etc/pki/example/service-ca-bundle.pem
{{nextTrustBundle}}

Overlap bundle containing old and new approved trust paths.

Example: /etc/pki/example/service-ca-overlap.pem
{{serviceDnsName}}

Primary service DNS identity.

Example: api.svc.example.net
{{serviceAlias}}

Additional inventory-approved service DNS identity.

Example: api-internal.svc.example.net
{{servicePort}}

Live TLS service port.

Example: 443
{{serviceRequestConfig}}

Workload-side CSR configuration.

Example: /etc/pki/tls/api-request.cnf
{{serviceExtensionFile}}

Issuer-reviewed per-request extension profile.

Example: /srv/pki/intermediate/csr/api-extensions.cnf
{{serviceKey}}secret

Workload-owned private key.

Example: /etc/pki/tls/private/api.key
{{serviceCsr}}

Public workload CSR.

Example: /etc/pki/tls/certs/api.csr
{{serviceCertificate}}

Issued short-lived service leaf.

Example: /etc/pki/tls/certs/api.crt
{{revokedCertificate}}

Incident-approved leaf selected for revocation.

Example: /srv/pki/incidents/compromised-api.crt
{{nextServiceCertificate}}

Replacement/rotation leaf tested during overlap.

Example: /etc/pki/tls/certs/api.next.crt
{{crlUrl}}

Stable HTTPS/HTTP CRL Distribution Point.

Example: https://pki.example.net/crl/service-ca.crl
{{ocspUrl}}

Operated OCSP endpoint or an intentionally omitted value/profile.

Example: https://ocsp.example.net
{{intermediateAiaUrl}}

Stable URL publishing the intermediate certificate.

Example: https://pki.example.net/ca/service-intermediate.crt
{{ocspResponderCertificate}}

Dedicated OCSPSigning responder leaf.

Example: /etc/pki/ocsp/responder.crt

Security and production boundaries

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Stop before continuing if

  • Stop when DNS, time, hostname, certificate fingerprint, realm, issuer, serial database, or access-policy evidence differs from the approved inventory.
  • Stop before disabling a permissive rule, rotating a CA, revoking a certificate, or unenrolling a host unless break-glass access has just been tested through an independent path.
  • Stop when a command would expose a secret in argv, overwrite CA state, run two `openssl ca` writers concurrently, or grant an all-users/all-hosts authorization shortcut.
  • Stop after any unexplained certificate issuance, authentication bypass, policy mismatch, lost private key, CA database divergence, or inability to restore the identity service from protected artifacts.
01

decision

Define the CA scope, assurance, and tool boundary

read-only

Approve names, certificate purposes, maximum lifetimes, owners, trust stores, revocation behavior, audit retention, and the maximum certificate population before creating keys.

Why this step matters

OpenSSL's `ca` command is documented as a sample minimal CA application with a text database, no locking, and production-quality limitations. It is suitable only for a deliberately small, serialized internal PKI with compensating controls.

What to understand

Use a managed CA platform when multiple issuers, high volume, automated approval, HA OCSP, policy-as-code, HSM ceremony, or compliance reporting is required.

The root signs only intermediate CA certificates and CRLs. The intermediate signs leaf certificates. Service private keys are generated where they are consumed and never delivered by the CA.

Document whether clients fail open or fail closed when CRL/OCSP is unavailable; OpenSSL cannot impose the consuming application's behavior.

System changes

  • Creates an approved PKI policy and ceremony record; no cryptographic state exists yet.

Syntax explained

OpenSSL 3.5 LTS
Reviewed long-term-support line, currently 3.5.7 and supported through 2030-04-08.
single serialized signer
Compensates for `openssl ca` having no database locking.
Example output / evidence
PKI profile approved: root=offline-15y, intermediate=online-5y, leaf=90d, DNS namespace=svc.example.net, EKU=serverAuth, CRL=24h, concurrent signers=1.

Checkpoint: Checkpoint: Define the CA scope, assurance, and tool boundary

Continue whenSecurity, service, recovery, and compliance owners approve the small-CA boundary and client revocation semantics.

Stop whenThe deployment needs concurrent issuance, public trust, ACME fleet automation, HA OCSP, or an audit assurance level this minimal tool cannot supply.

If this step fails

Two certificates share a serial or issuance events are missing.

Likely causeParallel `openssl ca` processes wrote the text database without locking, a cloned CA state signed independently, or manual serial edits occurred.

Safe checks
  • cut -f4 {{intermediateDir}}/index.txt | sort | uniq -d
  • Compare issuance audit records with every certificate under newcerts.

ResolutionStop the CA, treat the database as potentially divergent, identify every affected certificate, revoke/reissue as necessary, and move issuance behind a single serialized workflow.

Security notes

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Alternatives

  • Use a dedicated CA such as FreeIPA/Dogtag, a cloud private CA, step-ca, Vault PKI, or an enterprise HSM-backed CA when lifecycle and concurrency requirements exceed this guide.

Stop conditions

  • Stop when the observed identity, trust, version, state, or output differs from the approved plan.
  • Stop when independent recovery access, a current state backup, or a named rollback owner is unavailable.
02

verification

Verify the isolated signing host and supported OpenSSL

read-only

Boot the offline root workstation from controlled media, verify OpenSSL 3.5 LTS, time, entropy/provider state, encrypted storage, removable-media policy, and absence of general network connectivity.

Why this step matters

Cryptographic output is only as trustworthy as the host, library, time, randomness, and custody ceremony that created it. An offline root must not become an everyday administration machine.

What to understand

OpenSSL 3.5.7 is the reviewed LTS patch as of 2026-07-28. Revalidate the vendor-supported package and security advisories before each ceremony.

If policy requires FIPS validation, version numbers alone are insufficient; use the exact validated module, approved configuration, and operational environment from its security policy.

Record package hashes, boot-media identity, attendees, start/end time, and every removable medium entering or leaving the room.

System changes

  • Read-only inventory of the signing workstation and mounted encrypted CA volume.

Syntax explained

openssl version -a
Shows build, library, directories, CPU capability, and version—not only the CLI label.
openssl list -providers
Shows active cryptographic providers so unexpected legacy/FIPS provider state is visible.
Command
Fill variables0/1 ready

Values stay on this page and are never sent or saved.

openssl version -a; openssl list -providers; timedatectl show -p NTPSynchronized --value; findmnt -no SOURCE,FSTYPE,OPTIONS {{caBase}}
Example output / evidence
OpenSSL 3.5.7 9 Jun 2026
Providers:
  default
  name: OpenSSL Default Provider
yes
/dev/mapper/pki-root ext4 rw,nodev,nosuid,noexec

Checkpoint: Checkpoint: Verify the isolated signing host and supported OpenSSL

Continue whenApproved OpenSSL 3.5 LTS, synchronized ceremony time, controlled provider set, and encrypted nodev/nosuid/noexec CA storage are recorded.

Stop whenThe host is network-connected unexpectedly, software/provider hashes differ, time is wrong, or removable-media custody is unclear.

If this step fails

The root or intermediate private key cannot be opened.

Likely causeThe passphrase is unavailable, the key file is corrupt, permissions or provider URI are wrong, or the hardware token/provider is not initialized.

Safe checks
  • openssl pkey -in {{rootKey}} -check -noout
  • openssl pkey -in {{intermediateKey}} -check -noout
  • sha256sum the encrypted recovery copies and compare the signed manifest.

ResolutionStop issuance. Recover the key only through the documented multi-person escrow/HSM process and verify it against the public certificate before resuming.

Security notes

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Alternatives

  • Rehearse the exact step in an isolated environment with production-shaped identities, names, and trust paths before production.

Stop conditions

  • Stop when the observed identity, trust, version, state, or output differs from the approved plan.
  • Stop when independent recovery access, a current state backup, or a named rollback owner is unavailable.
03

command

Create the offline root CA state atomically

caution

Create private, certificate, request, CRL, and database directories with restrictive permissions; initialize empty index plus unpredictable starting serial/CRL numbers from OpenSSL randomness.

Why this step matters

The serial, index, and CRL-number files are authoritative mutable state. Predictable layout and restrictive modes make backup, review, and single-writer enforcement possible.

What to understand

Do not use `touch` plus default umask for sensitive files; create them with explicit modes.

Random positive serials reduce collision risk between planned CA generations, but they do not permit parallel writers against one database.

The index starts empty and is never edited casually. OpenSSL documents that reconstructing a corrupt index is difficult and provides no automatic rebuild command.

System changes

  • Creates the root CA directory tree and initial database counters on encrypted offline storage.

Syntax explained

openssl rand -hex 16
Generates a 128-bit hexadecimal starting value from OpenSSL's CSPRNG.
install -m 0600
Creates state files with owner-only access regardless of ambient umask.
Command
Fill variables0/1 ready

Values stay on this page and are never sent or saved.

install -d -m 0700 {{rootDir}} {{rootDir}}/private {{rootDir}}/certs {{rootDir}}/newcerts {{rootDir}}/crl {{rootDir}}/csr; install -m 0600 /dev/null {{rootDir}}/index.txt; openssl rand -hex 16 > {{rootDir}}/serial; openssl rand -hex 16 > {{rootDir}}/crlnumber; chmod 0600 {{rootDir}}/serial {{rootDir}}/crlnumber; find {{rootDir}} -maxdepth 2 -printf '%M %p\n' | sort
Example output / evidence
-rw------- /srv/pki/root/crlnumber
-rw------- /srv/pki/root/index.txt
-rw------- /srv/pki/root/serial
drwx------ /srv/pki/root/certs
drwx------ /srv/pki/root/crl
drwx------ /srv/pki/root/csr
drwx------ /srv/pki/root/newcerts
drwx------ /srv/pki/root/private

Checkpoint: Checkpoint: Create the offline root CA state atomically

Continue whenDirectories are 0700, state files are 0600, index is empty, and serial/crlnumber each contain valid nonzero hex.

Stop whenState exists from another CA generation, permissions are broader, or any initializer would overwrite existing state.

If this step fails

openssl ca reports a serial or index database error.

Likely causeThe serial/index/crlnumber file is missing, malformed, stale, concurrently modified, restored inconsistently, or points at the wrong CA directory.

Safe checks
  • sed -n '1,5p' {{intermediateDir}}/serial {{intermediateDir}}/crlnumber
  • tail -n 20 {{intermediateDir}}/index.txt
  • sha256sum {{intermediateConfig}} {{intermediateDir}}/index.txt {{intermediateDir}}/serial

ResolutionFreeze all signing and revocation. Restore a mutually consistent signed state bundle or reconstruct under an incident procedure before issuing another serial.

Security notes

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Alternatives

  • Rehearse the exact step in an isolated environment with production-shaped identities, names, and trust paths before production.

Stop conditions

  • Stop when the observed identity, trust, version, state, or output differs from the approved plan.
  • Stop when independent recovery access, a current state backup, or a named rollback owner is unavailable.
04

config

Write a root profile that can sign only intermediates

read-only

Use a reviewed OpenSSL configuration with an explicit policy, no copied CSR extensions, a path-length-constrained root profile, a critical key usage, and controlled CRL metadata.

Why this step matters

Issuer-controlled profiles prevent a requester from escalating a CSR into a CA or adding unauthorized SAN/EKU values. Root path length limits the intended hierarchy.

What to understand

OpenSSL warns that `copy_extensions=copyall` can let a malicious CSR request CA:TRUE. This configuration copies none and applies explicit extensions.

`unique_subject=no` supports legitimate certificate rollover with the same subject; serial uniqueness remains mandatory.

The root profile never includes serverAuth/clientAuth and therefore is not a service certificate.

System changes

  • Creates the root CA policy/configuration file used by later root and intermediate signing operations.

Syntax explained

basicConstraints critical, CA:true, pathlen
Marks the certificate as a CA and limits subordinate CA depth.
copy_extensions = none
Prevents request-supplied extensions from entering issued certificates automatically.
File {{rootConfig}}
Configuration
Fill variables0/1 ready

Values stay on this page and are never sent or saved.

[ ca ]
default_ca = root_ca

[ root_ca ]
dir = {{rootDir}}
database = $dir/index.txt
new_certs_dir = $dir/newcerts
certificate = $dir/certs/root-ca.crt
private_key = $dir/private/root-ca.key
serial = $dir/serial
crlnumber = $dir/crlnumber
crl = $dir/crl/root-ca.crl.pem
default_md = sha384
default_days = 1825
default_crl_days = 180
policy = policy_root
email_in_dn = no
unique_subject = no
copy_extensions = none

[ req ]
prompt = no
distinguished_name = root_dn
x509_extensions = root_ca_ext
string_mask = utf8only

[ root_dn ]
C = US
O = Example Operations
CN = Example Offline Root CA

[ policy_root ]
countryName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied

[ root_ca_ext ]
basicConstraints = critical, CA:true, pathlen:1
keyUsage = critical, keyCertSign, cRLSign
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always

[ intermediate_ca_ext ]
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, keyCertSign, cRLSign
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
Example output / evidence
Root profile review: copy_extensions=none; root pathlen=1; intermediate pathlen=0; keyUsage=keyCertSign,cRLSign; SHA-384.

Checkpoint: Checkpoint: Write a root profile that can sign only intermediates

Continue whenA second reviewer confirms directory paths, policy fields, no extension copying, CA constraints, digest, and lifetimes.

Stop whenThe profile can issue unconstrained sub-CAs, contains leaf EKU, copies CSR extensions, or points to another CA state tree.

If this step fails

The root or intermediate certificate lacks CA:TRUE or the expected path length.

Likely causeThe wrong extension section was selected, configuration was not the reviewed file, or request extensions were copied unsafely.

Safe checks
  • openssl x509 -in {{rootCertificate}} -noout -text
  • openssl x509 -in {{intermediateCertificate}} -noout -text
  • openssl verify -x509_strict -CAfile {{rootCertificate}} {{intermediateCertificate}}

ResolutionDiscard the invalid certificate and any descendants, correct the explicit CA extensions, and reissue under a new serial. Never edit an issued certificate in place.

Security notes

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Alternatives

  • Rehearse the exact step in an isolated environment with production-shaped identities, names, and trust paths before production.

Stop conditions

  • Stop when the observed identity, trust, version, state, or output differs from the approved plan.
  • Stop when independent recovery access, a current state backup, or a named rollback owner is unavailable.
05

command

Generate an encrypted root key and self-signed root certificate

danger

Generate the root key interactively or inside an HSM, then create the self-signed root certificate with the reviewed root extension section. Never pass a passphrase literal on the command line.

Why this step matters

The root private key is the ultimate trust secret. Encryption at rest, offline custody, multi-person ceremony, and minimal use are more important than the convenience of unattended signing.

What to understand

RSA 4096/SHA-384 is a conservative interoperability profile, not a universal mandate; approve algorithm agility and client compatibility before choosing ECDSA or a provider-backed key.

For stronger custody, generate a non-exportable key in an HSM/smart card and reference its provider URI. Test recovery and quorum procedures instead of creating an exportable software copy.

Fifteen years is the root certificate lifetime, not permission to ignore annual key/certificate/recovery review.

System changes

  • Creates the encrypted root private key and public self-signed root certificate.

Syntax explained

-aes-256-cbc
Encrypts the PKCS#8 private-key output and prompts for the passphrase because no passout is supplied.
-extensions root_ca_ext
Applies the explicit CA profile instead of relying on defaults or CSR extensions.
Command
Fill variables0/3 ready

Values stay on this page and are never sent or saved.

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -aes-256-cbc -out {{rootKey}}; chmod 0400 {{rootKey}}; openssl req -new -x509 -config {{rootConfig}} -extensions root_ca_ext -key {{rootKey}} -sha384 -days 5475 -out {{rootCertificate}}; chmod 0444 {{rootCertificate}}
Example output / evidence
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
Enter pass phrase for /srv/pki/root/private/root-ca.key:
Certificate request self-signature ok

Checkpoint: Checkpoint: Generate an encrypted root key and self-signed root certificate

openssl pkey -in {{rootKey}} -check -noout; openssl x509 -in {{rootCertificate}} -noout -text

Continue whenPrivate key check passes; certificate is X.509v3, self-issued/self-signed, CA:TRUE pathlen:1, keyCertSign+cRLSign, and its public key matches the private key.

Stop whenThe key is unencrypted/exported contrary to policy, certificate extensions differ, or only one person controls all root recovery material.

If this step fails

The root or intermediate certificate lacks CA:TRUE or the expected path length.

Likely causeThe wrong extension section was selected, configuration was not the reviewed file, or request extensions were copied unsafely.

Safe checks
  • openssl x509 -in {{rootCertificate}} -noout -text
  • openssl x509 -in {{intermediateCertificate}} -noout -text
  • openssl verify -x509_strict -CAfile {{rootCertificate}} {{intermediateCertificate}}

ResolutionDiscard the invalid certificate and any descendants, correct the explicit CA extensions, and reissue under a new serial. Never edit an issued certificate in place.

Security notes

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Alternatives

  • Use a provider-backed non-exportable HSM key with M-of-N activation and audited signing when the trust impact warrants it.

Stop conditions

  • Stop when DNS, time, hostname, certificate fingerprint, realm, issuer, serial database, or access-policy evidence differs from the approved inventory.
  • Stop before disabling a permissive rule, rotating a CA, revoking a certificate, or unenrolling a host unless break-glass access has just been tested through an independent path.
  • Stop when a command would expose a secret in argv, overwrite CA state, run two `openssl ca` writers concurrently, or grant an all-users/all-hosts authorization shortcut.
  • Stop after any unexplained certificate issuance, authentication bypass, policy mismatch, lost private key, CA database divergence, or inability to restore the identity service from protected artifacts.
06

verification

Create and restore-test split root recovery artifacts

read-only

Build a signed manifest containing key/certificate/config/database hashes, create at least two encrypted offline copies under separate custody, then restore one copy onto an isolated workstation and prove the public key/state.

Why this step matters

A backup is only credible when a separate controlled system can restore the exact key, configuration, index, serial, and CRL state and match the public trust anchor.

What to understand

Store passphrase shares separately from encrypted media and require multiple custodians; never photograph or paste recovery secrets.

A filesystem copy made while CA state is changing can mix generations. The root is offline, and every ceremony ends with an atomic versioned state bundle.

Record the test without retaining decrypted private material on the recovery workstation.

System changes

  • Creates a state manifest and controlled recovery copies; the rehearsal temporarily restores encrypted material on an isolated host.

Syntax explained

find -print0 / sort -z
Produces deterministic, whitespace-safe file ordering for the manifest.
fingerprint -sha256
Provides a human-verifiable identifier for the public trust anchor.
Command
Fill variables0/3 ready

Values stay on this page and are never sent or saved.

find {{rootDir}} -type f -print0 | sort -z | xargs -0 sha256sum > {{rootManifest}}; sha256sum {{rootManifest}}; openssl x509 -in {{rootCertificate}} -noout -fingerprint -sha256 -serial -subject -dates
Example output / evidence
3cf3e77f87f6df95f5c913a350bdf8035d62f3c50f7e7cb5c1b8508f7a90cc46  /srv/pki/root-manifest.sha256
sha256 Fingerprint=3A:87:5C:91:...:B2
serial=7C91D0A4F2268838
subject=O=Example Operations,CN=Example Offline Root CA

Checkpoint: Checkpoint: Create and restore-test split root recovery artifacts

Continue whenTwo custody-separated copies verify, an isolated restore matches every hash and root fingerprint, and decrypted remnants are destroyed.

Stop whenOne person holds media and passphrase, hashes mismatch, mutable state is missing, or restore requires the original host.

If this step fails

A CA disaster-recovery rehearsal produces a different issuer key or missing issuance history.

Likely causeThe state bundle omitted index/serial/crlnumber/config/certificates, the private key copy is wrong, or restore ordering mixed generations.

Safe checks
  • Compare signed manifest hashes for all state files.
  • openssl pkey -in {{intermediateKey}} -pubout | sha256sum
  • openssl x509 -in {{intermediateCertificate}} -pubkey -noout | sha256sum

ResolutionKeep the recovered CA isolated and freeze production issuance. Restore a consistent generation or execute the formal CA-compromise/replacement plan.

Security notes

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Alternatives

  • Rehearse the exact step in an isolated environment with production-shaped identities, names, and trust paths before production.

Stop conditions

  • Stop when the observed identity, trust, version, state, or output differs from the approved plan.
  • Stop when independent recovery access, a current state backup, or a named rollback owner is unavailable.
07

command

Create the serialized intermediate CA state

caution

On the restricted online issuer host, create an independent CA database, counters, private directories, audit path, and a single-writer lock/queue enforced by the operating workflow.

Why this step matters

The online intermediate limits routine root exposure but becomes the operational signing boundary. Its text database must have exactly one serialized writer.

What to understand

Place signing behind a reviewed queue or file lock that prevents concurrent `openssl ca`; the CLI itself performs no locking.

Back up index, serial, crlnumber, issued certificates, CRLs, config, and audit record as one generation after every issuance or revocation batch.

Restrict outbound network and administrator access; the issuer needs controlled CSR intake and CRL publication, not general internet access.

System changes

  • Creates the intermediate issuer's mutable CA state on its restricted online host.

Syntax explained

separate index/serial
The intermediate has its own issuance namespace and never writes the root database.
Command
Fill variables0/1 ready

Values stay on this page and are never sent or saved.

install -d -m 0700 {{intermediateDir}} {{intermediateDir}}/private {{intermediateDir}}/certs {{intermediateDir}}/newcerts {{intermediateDir}}/crl {{intermediateDir}}/csr; install -m 0600 /dev/null {{intermediateDir}}/index.txt; openssl rand -hex 16 > {{intermediateDir}}/serial; openssl rand -hex 16 > {{intermediateDir}}/crlnumber; chmod 0600 {{intermediateDir}}/serial {{intermediateDir}}/crlnumber
Example output / evidence
Intermediate state initialized: index.txt=0 bytes; serial=8F21C17A8E33B8F1A6D8C901D66E2A0B; crlnumber=9A440D7E1933659C7D47A9B42B68C71E.

Checkpoint: Checkpoint: Create the serialized intermediate CA state

Continue whenOwner-only state exists on protected storage and the operational system demonstrably prevents two signing writers.

Stop whenThe state is cloned writable, shares root paths, lacks audit/backup, or concurrency cannot be prevented.

If this step fails

Two certificates share a serial or issuance events are missing.

Likely causeParallel `openssl ca` processes wrote the text database without locking, a cloned CA state signed independently, or manual serial edits occurred.

Safe checks
  • cut -f4 {{intermediateDir}}/index.txt | sort | uniq -d
  • Compare issuance audit records with every certificate under newcerts.

ResolutionStop the CA, treat the database as potentially divergent, identify every affected certificate, revoke/reissue as necessary, and move issuance behind a single serialized workflow.

Security notes

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Alternatives

  • Rehearse the exact step in an isolated environment with production-shaped identities, names, and trust paths before production.

Stop conditions

  • Stop when the observed identity, trust, version, state, or output differs from the approved plan.
  • Stop when independent recovery access, a current state backup, or a named rollback owner is unavailable.
08

config

Define explicit service and revocation profiles

read-only

Configure the intermediate with issuer-controlled CA:FALSE leaf profiles, purpose-specific EKU, SAN supplied from a reviewed extension file, CRL distribution, and optional OCSP AIA.

Why this step matters

Service identities need purpose and name constraints. A generic certificate with both server and client EKU increases misuse opportunities and complicates revocation.

What to understand

SAN values are added only through a per-request extension file produced from the authoritative inventory and reviewed before signing.

CRL Distribution Points and AIA advertise locations; they do not guarantee clients fetch or enforce status.

The OpenSSL OCSP responder is a useful lab/interoperability tool, not a complete HA responder. Omit OCSP AIA unless an operated responder and client policy exist.

System changes

  • Creates the intermediate signing, leaf-purpose, and revocation configuration.

Syntax explained

extendedKeyUsage serverAuth/clientAuth
Constrains certificate use to one reviewed authentication role.
default_crl_hours = 24
Sets nextUpdate for frequent CRL publication; alert well before expiry.
File {{intermediateConfig}}
Configuration
Fill variables0/3 ready

Values stay on this page and are never sent or saved.

[ ca ]
default_ca = intermediate_ca

[ intermediate_ca ]
dir = {{intermediateDir}}
database = $dir/index.txt
new_certs_dir = $dir/newcerts
certificate = $dir/certs/intermediate-ca.crt
private_key = $dir/private/intermediate-ca.key
serial = $dir/serial
crlnumber = $dir/crlnumber
crl = $dir/crl/intermediate-ca.crl.pem
default_md = sha384
default_days = 90
default_crl_hours = 24
policy = policy_service
email_in_dn = no
unique_subject = no
copy_extensions = none

[ req ]
prompt = no
distinguished_name = intermediate_dn
string_mask = utf8only

[ intermediate_dn ]
C = US
O = Example Operations
OU = PKI
CN = Example Service Issuing CA

[ policy_service ]
countryName = optional
organizationName = match
organizationalUnitName = optional
commonName = supplied

[ server_leaf ]
basicConstraints = critical, CA:false
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
crlDistributionPoints = URI:{{crlUrl}}
authorityInfoAccess = caIssuers;URI:{{intermediateAiaUrl}}

[ client_leaf ]
basicConstraints = critical, CA:false
keyUsage = critical, digitalSignature
extendedKeyUsage = clientAuth
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
crlDistributionPoints = URI:{{crlUrl}}
Example output / evidence
Intermediate profile review: leaf CA:false; serverAuth and clientAuth separated; copy_extensions=none; leaf lifetime=90d; CRL=24h.

Checkpoint: Checkpoint: Define explicit service and revocation profiles

Continue whenA reviewer confirms explicit CA:FALSE, purpose-separated EKU, controlled SAN workflow, and operated CRL/AIA URLs.

Stop whencopy_extensions is enabled, both EKUs are added without need, URLs are unreachable/unowned, or the CRL interval lacks automation.

If this step fails

A CSR contains an unexpected CA:TRUE, SAN, EKU, or subject.

Likely causeThe requester supplied unreviewed extensions, the wrong service inventory was used, or automation copied request extensions without an issuer-controlled profile.

Safe checks
  • openssl req -in {{serviceCsr}} -noout -verify -subject -text
  • Compare requested DNS/IP identifiers with the authoritative service inventory.

ResolutionReject the CSR. Generate a new request from an approved key and let the issuer apply explicit CA:FALSE, key usage, EKU, SAN, AIA, and CRL extensions.

Security notes

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Alternatives

  • Rehearse the exact step in an isolated environment with production-shaped identities, names, and trust paths before production.

Stop conditions

  • Stop when the observed identity, trust, version, state, or output differs from the approved plan.
  • Stop when independent recovery access, a current state backup, or a named rollback owner is unavailable.
09

command

Generate the intermediate key and CSR

caution

Generate an encrypted intermediate key on the online issuer or HSM, then create and inspect a CSR whose subject matches root policy. Transfer only the CSR to the offline root ceremony.

Why this step matters

The intermediate private key remains at its controlled issuer; only a self-signature-verifiable CSR crosses to the offline root.

What to understand

Review the CSR public-key algorithm/size and subject. CSR verification proves possession of the private key, not authorization to become a CA.

The root ignores request extensions and applies its own pathlen:0 intermediate profile.

For HSM keys, back up token metadata and recovery quorum rather than attempting an export forbidden by policy.

System changes

  • Creates the encrypted intermediate signing key and its public CSR.

Syntax explained

openssl req -noout -verify
Validates the CSR's self-signature before it enters the root ceremony.
no passout/passphrase argument
Causes interactive entry rather than secret disclosure through argv.
Command
Fill variables0/3 ready

Values stay on this page and are never sent or saved.

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -aes-256-cbc -out {{intermediateKey}}; chmod 0400 {{intermediateKey}}; openssl req -new -config {{intermediateConfig}} -key {{intermediateKey}} -out {{intermediateCsr}}; openssl req -in {{intermediateCsr}} -noout -verify -subject -text
Example output / evidence
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
Certificate request self-signature verify OK
subject=O=Example Operations,OU=PKI,CN=Example Service Issuing CA

Checkpoint: Checkpoint: Generate the intermediate key and CSR

Continue whenCSR self-signature is OK, subject/public key match policy, private key remains restricted, and transfer media contains no private key.

Stop whenCSR extensions request CA escalation, subject is unexpected, key is unencrypted/exported improperly, or custody is unclear.

If this step fails

A CSR contains an unexpected CA:TRUE, SAN, EKU, or subject.

Likely causeThe requester supplied unreviewed extensions, the wrong service inventory was used, or automation copied request extensions without an issuer-controlled profile.

Safe checks
  • openssl req -in {{serviceCsr}} -noout -verify -subject -text
  • Compare requested DNS/IP identifiers with the authoritative service inventory.

ResolutionReject the CSR. Generate a new request from an approved key and let the issuer apply explicit CA:FALSE, key usage, EKU, SAN, AIA, and CRL extensions.

Security notes

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Alternatives

  • Rehearse the exact step in an isolated environment with production-shaped identities, names, and trust paths before production.

Stop conditions

  • Stop when the observed identity, trust, version, state, or output differs from the approved plan.
  • Stop when independent recovery access, a current state backup, or a named rollback owner is unavailable.
10

command

Sign the path-length-zero intermediate offline

danger

During a recorded root ceremony, verify the CSR again, sign it with the explicit intermediate profile, inspect every extension, and update/back up root state before taking the certificate out.

Why this step matters

This is the root's rare high-impact operation. Interactive review and explicit extensions prevent unattended approval and constrain the intermediate from creating another CA.

What to understand

Do not use `-batch` for the root ceremony unless an equivalent independently reviewed approval system already exists.

Confirm basicConstraints critical CA:TRUE pathlen:0 and critical keyCertSign/cRLSign before exporting.

Root index and serial change during signing; create a new atomic recovery generation before powering down.

System changes

  • Issues the intermediate CA certificate and advances the root index/serial database.

Syntax explained

-extensions intermediate_ca_ext
Applies root-controlled pathlen:0 CA capabilities.
-x509_strict
Enables stricter RFC 5280 chain checks during verification.
Command
Fill variables0/5 ready

Values stay on this page and are never sent or saved.

exec 9>{{rootDir}}/.openssl-ca.lock; flock -n 9 || { echo 'STOP: another root CA operation holds the lock' >&2; exit 1; }; openssl ca -config {{rootConfig}} -extensions intermediate_ca_ext -days 1825 -md sha384 -in {{intermediateCsr}} -out {{intermediateCertificate}}; openssl x509 -in {{intermediateCertificate}} -noout -subject -issuer -serial -dates -text; openssl verify -x509_strict -CAfile {{rootCertificate}} {{intermediateCertificate}}
Example output / evidence
Sign the certificate? [y/n]: y
1 out of 1 certificate requests certified, commit? [y/n] y
Write out database with 1 new entries
Database updated
{{intermediateCertificate}}: OK

Checkpoint: Checkpoint: Sign the path-length-zero intermediate offline

Continue whenChain is OK; issuer/root fingerprint, serial, validity, CA:TRUE pathlen:0, key usage, SKID, and AKID match the ceremony record.

Stop whenAny extension differs, root state cannot be backed up atomically, or the CSR/certificate fingerprint changed during transfer.

If this step fails

The root or intermediate certificate lacks CA:TRUE or the expected path length.

Likely causeThe wrong extension section was selected, configuration was not the reviewed file, or request extensions were copied unsafely.

Safe checks
  • openssl x509 -in {{rootCertificate}} -noout -text
  • openssl x509 -in {{intermediateCertificate}} -noout -text
  • openssl verify -x509_strict -CAfile {{rootCertificate}} {{intermediateCertificate}}

ResolutionDiscard the invalid certificate and any descendants, correct the explicit CA extensions, and reissue under a new serial. Never edit an issued certificate in place.

Security notes

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Alternatives

  • Rehearse the exact step in an isolated environment with production-shaped identities, names, and trust paths before production.

Stop conditions

  • Stop when DNS, time, hostname, certificate fingerprint, realm, issuer, serial database, or access-policy evidence differs from the approved inventory.
  • Stop before disabling a permissive rule, rotating a CA, revoking a certificate, or unenrolling a host unless break-glass access has just been tested through an independent path.
  • Stop when a command would expose a secret in argv, overwrite CA state, run two `openssl ca` writers concurrently, or grant an all-users/all-hosts authorization shortcut.
  • Stop after any unexplained certificate issuance, authentication bypass, policy mismatch, lost private key, CA database divergence, or inability to restore the identity service from protected artifacts.
11

config

Generate a service key and SAN-bearing CSR at the workload

read-only

Generate the leaf private key on the service host, create a CSR, and place requested SANs in a separate request file for review. The CA will still apply authoritative extensions.

Why this step matters

Generating the key at the workload avoids private-key transport. SAN, not CN alone, is the service identity modern TLS clients verify.

What to understand

Use an unencrypted service key only when unattended startup requires it and compensate with 0400 service-account permissions, encrypted disk, process isolation, and rapid rotation.

For client certificates, use a separate client profile and identity inventory; never reuse the same key/certificate for server and client authentication.

Wildcard SANs broaden compromise impact and are excluded unless the threat model and inventory explicitly justify them.

System changes

  • Defines the service request profile; the subsequent workload command creates its private key and CSR.

Syntax explained

subjectAltName
Authoritative DNS/IP identities clients verify.
prompt = no
Uses reviewed configuration values rather than ad-hoc interactive subject entry.
File {{serviceRequestConfig}}
Configuration
Fill variables0/2 ready

Values stay on this page and are never sent or saved.

[ req ]
prompt = no
distinguished_name = service_dn
req_extensions = request_ext

[ service_dn ]
O = Example Operations
OU = Platform Services
CN = {{serviceDnsName}}

[ request_ext ]
subjectAltName = @service_names

[ service_names ]
DNS.1 = {{serviceDnsName}}
DNS.2 = {{serviceAlias}}
Example output / evidence
Certificate request self-signature verify OK
subject=O=Example Operations,OU=Platform Services,CN=api.svc.example.net
X509v3 Subject Alternative Name:
    DNS:api.svc.example.net, DNS:api-internal.svc.example.net

Checkpoint: Checkpoint: Generate a service key and SAN-bearing CSR at the workload

openssl req -new -newkey rsa:3072 -noenc -config {{serviceRequestConfig}} -keyout {{serviceKey}} -out {{serviceCsr}}; chmod 0400 {{serviceKey}}; openssl req -in {{serviceCsr}} -noout -verify -subject -text

Continue whenCSR self-signature is OK; subject and every SAN exactly match authoritative inventory; private key remains only on the workload.

Stop whenCSR requests CA:TRUE, unexpected SAN/EKU, wildcard, weak key, or the private key would cross the CA boundary.

If this step fails

A CSR contains an unexpected CA:TRUE, SAN, EKU, or subject.

Likely causeThe requester supplied unreviewed extensions, the wrong service inventory was used, or automation copied request extensions without an issuer-controlled profile.

Safe checks
  • openssl req -in {{serviceCsr}} -noout -verify -subject -text
  • Compare requested DNS/IP identifiers with the authoritative service inventory.

ResolutionReject the CSR. Generate a new request from an approved key and let the issuer apply explicit CA:FALSE, key usage, EKU, SAN, AIA, and CRL extensions.

Security notes

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Alternatives

  • Rehearse the exact step in an isolated environment with production-shaped identities, names, and trust paths before production.

Stop conditions

  • Stop when the observed identity, trust, version, state, or output differs from the approved plan.
  • Stop when independent recovery access, a current state backup, or a named rollback owner is unavailable.
12

config

Apply issuer-controlled SAN and issue the service certificate

danger

Create a one-request extension file from approved inventory, review CSR/key/name ownership, then serialize the `openssl ca` signing transaction under the server leaf profile.

Why this step matters

The issuer, not the requester, decides whether names and purposes are authorized. Signing advances critical serial/index state and must be single-threaded and auditable.

What to understand

Before approval, compare DNS ownership, service owner, ticket, CSR fingerprint, public-key strength, requested subject, and profile.

Never invoke two `openssl ca` processes against the same index. A workflow lock must cover signing, database commit, audit write, and atomic state backup.

If no operated OCSP responder exists, use a profile without OCSP AIA rather than advertising a dead endpoint.

System changes

  • Issues one 90-day server certificate and advances intermediate index/serial state.

Syntax explained

-extfile / -extensions
Applies the reviewed per-request names plus fixed issuer-controlled service constraints.
interactive commit
Provides a final human review before mutable CA state changes.
File {{serviceExtensionFile}}
Configuration
Fill variables0/4 ready

Values stay on this page and are never sent or saved.

[ server_leaf_with_names ]
basicConstraints = critical, CA:false
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
subjectAltName = DNS:{{serviceDnsName}},DNS:{{serviceAlias}}
crlDistributionPoints = URI:{{crlUrl}}
authorityInfoAccess = caIssuers;URI:{{intermediateAiaUrl}}
Example output / evidence
Sign the certificate? [y/n]: y
1 out of 1 certificate requests certified, commit? [y/n] y
Write out database with 1 new entries
Database updated

Checkpoint: Checkpoint: Apply issuer-controlled SAN and issue the service certificate

exec 9>{{intermediateLock}}; flock -n 9 || { echo 'STOP: another intermediate CA operation holds the lock' >&2; exit 1; }; openssl ca -config {{intermediateConfig}} -extfile {{serviceExtensionFile}} -extensions server_leaf_with_names -days 90 -md sha384 -in {{serviceCsr}} -out {{serviceCertificate}}

Continue whenOne new serial appears once in index/newcerts/audit, issued names and EKU exactly match approval, and a consistent post-sign state backup exists.

Stop whenThe lock is unavailable, index is divergent, CSR fingerprint changed, extensions differ, or post-sign backup fails.

If this step fails

Two certificates share a serial or issuance events are missing.

Likely causeParallel `openssl ca` processes wrote the text database without locking, a cloned CA state signed independently, or manual serial edits occurred.

Safe checks
  • cut -f4 {{intermediateDir}}/index.txt | sort | uniq -d
  • Compare issuance audit records with every certificate under newcerts.

ResolutionStop the CA, treat the database as potentially divergent, identify every affected certificate, revoke/reissue as necessary, and move issuance behind a single serialized workflow.

Security notes

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Alternatives

  • Rehearse the exact step in an isolated environment with production-shaped identities, names, and trust paths before production.

Stop conditions

  • Stop when DNS, time, hostname, certificate fingerprint, realm, issuer, serial database, or access-policy evidence differs from the approved inventory.
  • Stop before disabling a permissive rule, rotating a CA, revoking a certificate, or unenrolling a host unless break-glass access has just been tested through an independent path.
  • Stop when a command would expose a secret in argv, overwrite CA state, run two `openssl ca` writers concurrently, or grant an all-users/all-hosts authorization shortcut.
  • Stop after any unexplained certificate issuance, authentication bypass, policy mismatch, lost private key, CA database divergence, or inability to restore the identity service from protected artifacts.
13

verification

Validate purpose, hostname, chain, key match, and live deployment

read-only

Build the root-plus-intermediate trust bundle, verify the certificate under strict server purpose/hostname rules, match leaf and private public keys, deploy the chain, and inspect every live endpoint.

Why this step matters

A signed certificate is not accepted until chain, purpose, name, private-key match, server chain presentation, and real client validation all pass.

What to understand

Serve the leaf followed by intermediate certificates; clients receive intermediates but trust the root through their managed trust store.

Test every load-balanced backend and public/internal alias. One stale node can continue presenting a revoked or expired certificate.

Do not use `-k`, `verify_none`, or hostname bypasses as production validation.

System changes

  • Deploys the leaf/intermediate chain and private key to the service, then reloads it according to the service's safe procedure.

Syntax explained

-purpose sslserver -verify_hostname
Checks chain constraints, serverAuth suitability, and exact DNS identity.
-verify_return_error
Makes a live TLS chain error fail immediately.
Command
Fill variables0/5 ready

Values stay on this page and are never sent or saved.

openssl verify -show_chain -x509_strict -purpose sslserver -verify_hostname {{serviceDnsName}} -CAfile {{trustBundle}} {{serviceCertificate}}; test "$(openssl x509 -in {{serviceCertificate}} -pubkey -noout | sha256sum)" = "$(openssl pkey -in {{serviceKey}} -pubout | sha256sum)" && echo 'leaf key matches'; openssl s_client -connect {{serviceDnsName}}:{{servicePort}} -servername {{serviceDnsName}} -verify_hostname {{serviceDnsName}} -CAfile {{trustBundle}} -verify_return_error </dev/null 2>/dev/null | grep -E 'Protocol|Cipher|Verify return code'
Example output / evidence
{{serviceCertificate}}: OK
Chain:
depth=0: O=Example Operations, CN=api.svc.example.net
depth=1: O=Example Operations, CN=Example Service Issuing CA
leaf key matches
Protocol  : TLSv1.3
Cipher    : TLS_AES_256_GCM_SHA384
Verify return code: 0 (ok)

Checkpoint: Checkpoint: Validate purpose, hostname, chain, key match, and live deployment

Continue whenOffline and live checks return OK/0, key matches, exact SAN works, wrong hostname fails, and every backend serves the new serial.

Stop whenAny bypass is needed, the wrong hostname validates, intermediate is omitted, key mismatches, or only some backends rotate.

If this step fails

The issued service certificate validates as a chain but fails hostname or purpose checks.

Likely causeThe SAN omits the service name, serverAuth/clientAuth EKU is wrong, key usage is incompatible, or an unintended intermediate built the chain.

Safe checks
  • openssl verify -x509_strict -purpose sslserver -verify_hostname {{serviceDnsName}} -CAfile {{trustBundle}} {{serviceCertificate}}
  • openssl x509 -in {{serviceCertificate}} -noout -subject -issuer -serial -dates -ext subjectAltName -ext extendedKeyUsage -ext keyUsage

ResolutionDo not deploy. Revoke if the invalid certificate left the signing boundary, correct the issuer profile, and issue a replacement with a new serial.

Security notes

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Alternatives

  • Rehearse the exact step in an isolated environment with production-shaped identities, names, and trust paths before production.

Stop conditions

  • Stop when the observed identity, trust, version, state, or output differs from the approved plan.
  • Stop when independent recovery access, a current state backup, or a named rollback owner is unavailable.
14

command

Generate, validate, and atomically publish a CRL

caution

Generate the intermediate CRL from authoritative index state, inspect issuer/number/update window/signature, convert to the distribution format, and atomically replace the published object.

Why this step matters

Revocation is only effective while clients can obtain a current authentic status object and are configured to enforce it. CRL publication needs freshness monitoring before nextUpdate.

What to understand

Publish through a staging filename/object and atomic rename so clients never receive a partial CRL.

Monitor HTTP status, content hash, issuer, signature, nextUpdate safety margin, and cache headers from each network segment.

A 24-hour CRL creates up to a day of revocation exposure unless clients refresh more frequently; choose interval from threat/SLO evidence.

System changes

  • Advances intermediate CRL number, creates a signed CRL, and publishes its DER form at the controlled distribution point.

Syntax explained

-gencrl
Builds the CRL from revoked entries in the authoritative CA index.
-crl_check
Checks the leaf against the supplied issuer CRL during verification.
Command
Fill variables0/8 ready

Values stay on this page and are never sent or saved.

exec 9>{{intermediateLock}}; flock -n 9 || { echo 'STOP: another intermediate CA operation holds the lock' >&2; exit 1; }; openssl ca -config {{intermediateConfig}} -gencrl -out {{intermediateCrl}}; openssl crl -in {{intermediateCrl}} -noout -issuer -lastupdate -nextupdate -crlnumber; openssl crl -in {{intermediateCrl}} -outform DER -out {{intermediateCrlDer}}; openssl verify -crl_check -CRLfile {{intermediateCrl}} -CAfile {{trustBundle}} {{serviceCertificate}}; sudo install -o root -g root -m 0644 {{intermediateCrlDer}} {{publishedCrlPath}}.new; sudo mv -f {{publishedCrlPath}}.new {{publishedCrlPath}}; remote=$(mktemp); trap 'rm -f "$remote"' EXIT; curl --fail --silent --show-error {{crlUrl}} -o "$remote"; cmp {{publishedCrlPath}} "$remote"
Example output / evidence
issuer=O=Example Operations, CN=Example Service Issuing CA
lastUpdate=Jul 28 13:00:00 2026 GMT
nextUpdate=Jul 29 13:00:00 2026 GMT
crlNumber=0x9A440D7E1933659C7D47A9B42B68C71E
{{serviceCertificate}}: OK

Checkpoint: Checkpoint: Generate, validate, and atomically publish a CRL

Continue whenCRL issuer/signature/number are correct, nextUpdate meets policy, active leaf verifies, and published bytes/hash match the generated DER.

Stop whenCRL is stale, publication is non-atomic, clients cannot reach it, or the index generation differs.

If this step fails

The CRL nextUpdate is expired or no longer monitored.

Likely causeThe scheduled generation/publishing job failed, the intermediate key was unavailable, clock is wrong, or the distribution path points to an old file.

Safe checks
  • openssl crl -in {{intermediateCrl}} -noout -lastupdate -nextupdate -crlnumber
  • curl --fail --silent --show-error {{crlUrl}} | openssl crl -inform DER -noout -lastupdate -nextupdate

ResolutionTreat revocation as degraded, generate and atomically publish a current CRL from authoritative state, then alert before the next safety margin.

Security notes

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Alternatives

  • Rehearse the exact step in an isolated environment with production-shaped identities, names, and trust paths before production.

Stop conditions

  • Stop when the observed identity, trust, version, state, or output differs from the approved plan.
  • Stop when independent recovery access, a current state backup, or a named rollback owner is unavailable.
15

command

Revoke a compromised leaf and prove rejection

danger

Verify the incident-approved certificate fingerprint and serial, revoke it with a precise reason, generate/publish a new CRL immediately, and prove a revocation-checking verifier rejects it.

Why this step matters

Revocation is an irreversible identity-state change in this simple CA workflow. The operator must prove the exact certificate and then verify consuming systems reject it.

What to understand

Replace the service key/certificate before or alongside revocation so availability does not encourage revocation bypass.

The reason `keyCompromise` communicates incident semantics. Do not use certificateHold as a casual reversible toggle.

Capture index/CRL/audit and publish atomically; a local database update with a stale public CRL has no client effect.

System changes

  • Marks one serial revoked in the intermediate index, advances CRL state, and publishes a replacement CRL.

Syntax explained

-crl_reason keyCompromise
Records the reason in a v2 CRL entry for incident/audit interpretation.
expected error 23
Concrete proof that OpenSSL rejected the revoked leaf under CRL checking.
Command
Fill variables0/7 ready

Values stay on this page and are never sent or saved.

openssl x509 -in {{revokedCertificate}} -noout -fingerprint -sha256 -serial -subject; exec 9>{{intermediateLock}}; flock -n 9 || { echo 'STOP: another intermediate CA operation holds the lock' >&2; exit 1; }; openssl ca -config {{intermediateConfig}} -revoke {{revokedCertificate}} -crl_reason keyCompromise; openssl ca -config {{intermediateConfig}} -gencrl -out {{intermediateCrl}}; openssl crl -in {{intermediateCrl}} -outform DER -out {{intermediateCrlDer}}; sudo install -o root -g root -m 0644 {{intermediateCrlDer}} {{publishedCrlPath}}.new; sudo mv -f {{publishedCrlPath}}.new {{publishedCrlPath}}; verify_output="$(openssl verify -crl_check -CRLfile {{intermediateCrl}} -CAfile {{trustBundle}} {{revokedCertificate}} 2>&1)"; verify_rc=$?; printf '%s
' "$verify_output"; test "$verify_rc" -ne 0; printf '%s
' "$verify_output" | grep -q 'error 23'
Example output / evidence
sha256 Fingerprint=8D:31:...:F7
serial=8F21C17A8E33B8F1A6D8C901D66E2A0C
Revoking Certificate 8F21C17A8E33B8F1A6D8C901D66E2A0C.
Database updated
error 23 at 0 depth lookup: certificate revoked
error {{revokedCertificate}}: verification failed

Checkpoint: Checkpoint: Revoke a compromised leaf and prove rejection

Continue whenOnly the approved serial is revoked, published CRL contains it, the replacement certificate works, and the old one fails with error 23 in every enforcing client.

Stop whenFingerprint/serial differs, replacement is not ready, publication fails, or critical clients do not enforce revocation.

If this step fails

A revoked certificate still verifies successfully in a client.

Likely causeThe client does not perform revocation checking, has a stale CRL, cannot reach the distribution point, trusts a different chain, or soft-fails OCSP.

Safe checks
  • openssl crl -in {{intermediateCrl}} -noout -lastupdate -nextupdate -text
  • openssl verify -crl_check -CRLfile {{intermediateCrl}} -CAfile {{trustBundle}} {{revokedCertificate}}
  • Inspect the client's effective revocation policy and cache.

ResolutionPublish a fresh CRL atomically, refresh client caches, and prove the application actually enforces the selected fail-open/fail-closed behavior. Replace the credential immediately.

Security notes

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Alternatives

  • Rehearse the exact step in an isolated environment with production-shaped identities, names, and trust paths before production.

Stop conditions

  • Stop when DNS, time, hostname, certificate fingerprint, realm, issuer, serial database, or access-policy evidence differs from the approved inventory.
  • Stop before disabling a permissive rule, rotating a CA, revoking a certificate, or unenrolling a host unless break-glass access has just been tested through an independent path.
  • Stop when a command would expose a secret in argv, overwrite CA state, run two `openssl ca` writers concurrently, or grant an all-users/all-hosts authorization shortcut.
  • Stop after any unexplained certificate issuance, authentication bypass, policy mismatch, lost private key, CA database divergence, or inability to restore the identity service from protected artifacts.
16

decision

Choose an explicit OCSP availability policy

read-only

If clients require OCSP, issue a separate least-privilege OCSP signing certificate, operate a synchronized responder, monitor freshness/signature, and document client soft-fail or hard-fail behavior.

Why this step matters

OCSP lowers status latency only when responder keys, database synchronization, freshness, availability, and client enforcement are designed. A dead advertised URL can reduce availability or silently soft-fail.

What to understand

OpenSSL's `ocsp` responder mode is described as a mini responder and is appropriate for tests/small controlled use, not a general HA service.

The responder certificate needs OCSPSigning EKU and no CA capability; its key is separate from the intermediate signing key.

TLS stapling changes who fetches status but still needs reliable responder data and service refresh monitoring.

System changes

  • No change when CRL-only is selected. Enabling OCSP adds a responder identity, service, synchronized status database, AIA URL, and monitoring.

Syntax explained

-verify_other
Provides the expected responder signer certificate for response verification.
This Update / Next Update
Bounds status freshness; successful HTTP alone is not enough.
Command
Fill variables0/5 ready

Values stay on this page and are never sent or saved.

openssl ocsp -issuer {{intermediateCertificate}} -cert {{serviceCertificate}} -url {{ocspUrl}} -CAfile {{trustBundle}} -verify_other {{ocspResponderCertificate}} -resp_text
Example output / evidence
Response verify OK
api.svc.example.net.crt: good
This Update: Jul 28 13:05:00 2026 GMT
Next Update: Jul 28 13:10:00 2026 GMT

Checkpoint: Checkpoint: Choose an explicit OCSP availability policy

Continue whenThe chosen CRL-only or CRL+OCSP policy has operated endpoints, freshness alerts, responder-key custody, and tested client outage semantics.

Stop whenOCSP is advertised without HA/monitoring, responder has CA signing capability, or client failure behavior is unknown.

If this step fails

The OCSP responder returns unknown or stale status.

Likely causeThe responder index is stale, issuer/serial differs, the responder certificate/EKU is wrong, or the experimental OpenSSL responder is not highly available.

Safe checks
  • openssl ocsp -issuer {{intermediateCertificate}} -cert {{serviceCertificate}} -url {{ocspUrl}} -CAfile {{trustBundle}} -resp_text
  • Compare responder index checksum and update time with the authoritative CA database.

ResolutionFall back only according to the documented application policy, repair responder synchronization, and do not claim the OpenSSL mini-responder is a production HA revocation service.

Security notes

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Alternatives

  • Use short-lived leaf certificates plus CRL as the primary status mechanism when an HA OCSP service cannot be operated honestly.

Stop conditions

  • Stop when the observed identity, trust, version, state, or output differs from the approved plan.
  • Stop when independent recovery access, a current state backup, or a named rollback owner is unavailable.
17

verification

Rehearse leaf/intermediate rotation and complete CA disaster recovery

read-only

Stage trust before issuance, overlap old/new leaves, verify every client, retire only after evidence, then restore the complete intermediate state onto an isolated host and issue a quarantined test certificate.

Why this step matters

Rotation is the normal proof that trust distribution and reload work; disaster recovery proves the CA can resume without losing serial/revocation history or creating a second divergent writer.

What to understand

Distribute a new root/intermediate before issuing dependent leaves. Remove an old root only after the last dependent certificate and every retained validation use has ended.

The recovered issuer stays quarantined and never signs production while the original exists. Compare key/certificate public keys and complete state hashes.

Test HSM replacement, custodian unavailability, CRL publication, audit continuity, and compromise response—not merely file extraction.

System changes

  • Stages new trust/certificates and performs an isolated restoration of complete CA state; production signing ownership remains singular.

Syntax explained

-show_chain
Displays which trust path OpenSSL actually built during overlap.
sha256sum -c
Validates the restored state generation against its signed manifest.
Command
Fill variables0/4 ready

Values stay on this page and are never sent or saved.

openssl verify -show_chain -x509_strict -purpose sslserver -verify_hostname {{serviceDnsName}} -CAfile {{nextTrustBundle}} {{nextServiceCertificate}}; openssl x509 -in {{nextServiceCertificate}} -noout -serial -dates -fingerprint -sha256; sha256sum -c {{intermediateManifest}}
Example output / evidence
{{nextServiceCertificate}}: OK
Chain:
depth=0: CN=api.svc.example.net
depth=1: CN=Example Service Issuing CA 2026
serial=AA5F1C70D4E91B2F
notBefore=Jul 28 00:00:00 2026 GMT
notAfter=Oct 26 23:59:59 2026 GMT
/srv/pki/intermediate/index.txt: OK
/srv/pki/intermediate/serial: OK
/srv/pki/intermediate/crlnumber: OK

Checkpoint: Checkpoint: Rehearse leaf/intermediate rotation and complete CA disaster recovery

Continue whenOld and new paths work during overlap, all backends rotate, isolated recovery matches state and test issuance, and only one production writer exists.

Stop whenClients lack new trust, any backend is stale, state hashes differ, serial history is missing, or recovered/original signers can operate concurrently.

If this step fails

Clients reject the new intermediate during a CA rotation.

Likely causeThe new intermediate/root was not distributed before leaf issuance, the server omitted intermediate certificates, or path building selects an unintended chain.

Safe checks
  • openssl verify -show_chain -x509_strict -purpose sslserver -verify_hostname {{serviceDnsName}} -CAfile {{trustBundle}} {{serviceCertificate}}
  • Inspect the exact chain served by each endpoint.

ResolutionPause new issuance, restore the previously accepted chain where safe, distribute the new trust anchor/intermediate, and repeat staged compatibility tests before retrying.

Security notes

  • Never put passwords, private-key passphrases, enrollment one-time passwords, LDAP bind credentials, keytabs, or recovery secrets in command arguments, shell history, environment dumps, tickets, screenshots, or this guide. Use an interactive prompt, protected file descriptor, hardware token, or approved secret broker.
  • Separate human administration, automated enrollment, signing, revocation, host operation, and break-glass duties. A single reusable domain administrator or CA key on online hosts defeats the intended security boundary.
  • Test both an allowed and a denied identity operation. A successful privileged administrator action does not prove least privilege, hostname verification, revocation, HBAC, or sudo enforcement.
  • Keep an independently tested recovery path that does not depend on the service being recovered. Identity and PKI outages can otherwise prevent the operators who need to repair them from logging in.

Alternatives

  • Rehearse the exact step in an isolated environment with production-shaped identities, names, and trust paths before production.

Stop conditions

  • Stop when the observed identity, trust, version, state, or output differs from the approved plan.
  • Stop when independent recovery access, a current state backup, or a named rollback owner is unavailable.

Finish line

Verification checklist

Strict service identityopenssl verify -show_chain -x509_strict -purpose sslserver -verify_hostname {{serviceDnsName}} -CAfile {{trustBundle}} {{serviceCertificate}}Leaf OK through exactly the intended intermediate/root path.
Wrong name deniedopenssl verify -x509_strict -purpose sslserver -verify_hostname wrong.svc.example.net -CAfile {{trustBundle}} {{serviceCertificate}}Hostname mismatch and verification failure.
Revoked leaf deniedopenssl verify -crl_check -CRLfile {{intermediateCrl}} -CAfile {{trustBundle}} {{revokedCertificate}}error 23: certificate revoked.
Complete CA recoverysha256sum -c {{intermediateManifest}} on the isolated recovery host and perform quarantined test issuance.Every state hash OK, matching issuer public key, preserved serial history, and no second production writer.

Recovery guidance

Common problems and safe checks

The root or intermediate private key cannot be opened.

Likely causeThe passphrase is unavailable, the key file is corrupt, permissions or provider URI are wrong, or the hardware token/provider is not initialized.

Safe checks
  • openssl pkey -in {{rootKey}} -check -noout
  • openssl pkey -in {{intermediateKey}} -check -noout
  • sha256sum the encrypted recovery copies and compare the signed manifest.

ResolutionStop issuance. Recover the key only through the documented multi-person escrow/HSM process and verify it against the public certificate before resuming.

The root or intermediate certificate lacks CA:TRUE or the expected path length.

Likely causeThe wrong extension section was selected, configuration was not the reviewed file, or request extensions were copied unsafely.

Safe checks
  • openssl x509 -in {{rootCertificate}} -noout -text
  • openssl x509 -in {{intermediateCertificate}} -noout -text
  • openssl verify -x509_strict -CAfile {{rootCertificate}} {{intermediateCertificate}}

ResolutionDiscard the invalid certificate and any descendants, correct the explicit CA extensions, and reissue under a new serial. Never edit an issued certificate in place.

openssl ca reports a serial or index database error.

Likely causeThe serial/index/crlnumber file is missing, malformed, stale, concurrently modified, restored inconsistently, or points at the wrong CA directory.

Safe checks
  • sed -n '1,5p' {{intermediateDir}}/serial {{intermediateDir}}/crlnumber
  • tail -n 20 {{intermediateDir}}/index.txt
  • sha256sum {{intermediateConfig}} {{intermediateDir}}/index.txt {{intermediateDir}}/serial

ResolutionFreeze all signing and revocation. Restore a mutually consistent signed state bundle or reconstruct under an incident procedure before issuing another serial.

A CSR contains an unexpected CA:TRUE, SAN, EKU, or subject.

Likely causeThe requester supplied unreviewed extensions, the wrong service inventory was used, or automation copied request extensions without an issuer-controlled profile.

Safe checks
  • openssl req -in {{serviceCsr}} -noout -verify -subject -text
  • Compare requested DNS/IP identifiers with the authoritative service inventory.

ResolutionReject the CSR. Generate a new request from an approved key and let the issuer apply explicit CA:FALSE, key usage, EKU, SAN, AIA, and CRL extensions.

The issued service certificate validates as a chain but fails hostname or purpose checks.

Likely causeThe SAN omits the service name, serverAuth/clientAuth EKU is wrong, key usage is incompatible, or an unintended intermediate built the chain.

Safe checks
  • openssl verify -x509_strict -purpose sslserver -verify_hostname {{serviceDnsName}} -CAfile {{trustBundle}} {{serviceCertificate}}
  • openssl x509 -in {{serviceCertificate}} -noout -subject -issuer -serial -dates -ext subjectAltName -ext extendedKeyUsage -ext keyUsage

ResolutionDo not deploy. Revoke if the invalid certificate left the signing boundary, correct the issuer profile, and issue a replacement with a new serial.

A revoked certificate still verifies successfully in a client.

Likely causeThe client does not perform revocation checking, has a stale CRL, cannot reach the distribution point, trusts a different chain, or soft-fails OCSP.

Safe checks
  • openssl crl -in {{intermediateCrl}} -noout -lastupdate -nextupdate -text
  • openssl verify -crl_check -CRLfile {{intermediateCrl}} -CAfile {{trustBundle}} {{revokedCertificate}}
  • Inspect the client's effective revocation policy and cache.

ResolutionPublish a fresh CRL atomically, refresh client caches, and prove the application actually enforces the selected fail-open/fail-closed behavior. Replace the credential immediately.

The CRL nextUpdate is expired or no longer monitored.

Likely causeThe scheduled generation/publishing job failed, the intermediate key was unavailable, clock is wrong, or the distribution path points to an old file.

Safe checks
  • openssl crl -in {{intermediateCrl}} -noout -lastupdate -nextupdate -crlnumber
  • curl --fail --silent --show-error {{crlUrl}} | openssl crl -inform DER -noout -lastupdate -nextupdate

ResolutionTreat revocation as degraded, generate and atomically publish a current CRL from authoritative state, then alert before the next safety margin.

The OCSP responder returns unknown or stale status.

Likely causeThe responder index is stale, issuer/serial differs, the responder certificate/EKU is wrong, or the experimental OpenSSL responder is not highly available.

Safe checks
  • openssl ocsp -issuer {{intermediateCertificate}} -cert {{serviceCertificate}} -url {{ocspUrl}} -CAfile {{trustBundle}} -resp_text
  • Compare responder index checksum and update time with the authoritative CA database.

ResolutionFall back only according to the documented application policy, repair responder synchronization, and do not claim the OpenSSL mini-responder is a production HA revocation service.

A service still presents the old certificate after rotation.

Likely causeThe process was not reloaded, a proxy has its own certificate, the wrong file/symlink was updated, or one load-balanced node was missed.

Safe checks
  • openssl s_client -connect {{serviceDnsName}}:{{servicePort}} -servername {{serviceDnsName}} -showcerts </dev/null
  • Compare leaf serial/fingerprint through every backend and external endpoint.

ResolutionKeep the overlap certificate valid, correct deployment/reload on every endpoint, and retire the old certificate only after complete fleet evidence.

A CA disaster-recovery rehearsal produces a different issuer key or missing issuance history.

Likely causeThe state bundle omitted index/serial/crlnumber/config/certificates, the private key copy is wrong, or restore ordering mixed generations.

Safe checks
  • Compare signed manifest hashes for all state files.
  • openssl pkey -in {{intermediateKey}} -pubout | sha256sum
  • openssl x509 -in {{intermediateCertificate}} -pubkey -noout | sha256sum

ResolutionKeep the recovered CA isolated and freeze production issuance. Restore a consistent generation or execute the formal CA-compromise/replacement plan.

Two certificates share a serial or issuance events are missing.

Likely causeParallel `openssl ca` processes wrote the text database without locking, a cloned CA state signed independently, or manual serial edits occurred.

Safe checks
  • cut -f4 {{intermediateDir}}/index.txt | sort | uniq -d
  • Compare issuance audit records with every certificate under newcerts.

ResolutionStop the CA, treat the database as potentially divergent, identify every affected certificate, revoke/reissue as necessary, and move issuance behind a single serialized workflow.

Clients reject the new intermediate during a CA rotation.

Likely causeThe new intermediate/root was not distributed before leaf issuance, the server omitted intermediate certificates, or path building selects an unintended chain.

Safe checks
  • openssl verify -show_chain -x509_strict -purpose sslserver -verify_hostname {{serviceDnsName}} -CAfile {{trustBundle}} {{serviceCertificate}}
  • Inspect the exact chain served by each endpoint.

ResolutionPause new issuance, restore the previously accepted chain where safe, distribute the new trust anchor/intermediate, and repeat staged compatibility tests before retrying.

After the procedure

Alternatives and next steps

Consider these alternatives

  • Use a managed enterprise/private cloud CA when production requires HA issuance, automated enrollment, policy APIs, auditable approval, or managed revocation.
  • Use a non-exportable HSM/provider key with M-of-N activation for root and intermediate custody.
  • Use short-lived automatically renewed certificates to reduce revocation dependence, while preserving an emergency revocation path.

Operate it safely

  • Automate read-only expiry, chain, SAN/EKU, CRL freshness, live serial, and trust-store drift checks without automating root approval.
  • Schedule quarterly leaf rotation and annual intermediate/root recovery ceremonies, including custodian and HSM failure.
  • Maintain a complete certificate inventory mapping serial, key owner, service, environment, deployment endpoints, and revocation status.
  • Plan migration to a production CA platform before certificate population, issuance concurrency, or assurance requirements exceed the documented boundary.

Reference

Frequently asked questions

Is `openssl ca` a production CA platform?

OpenSSL explicitly describes it as a sample minimal CA with a text database, no locking, and production-quality limitations. This guide bounds it to a small serialized internal PKI; larger or higher-assurance environments should use a dedicated CA.

Can we copy all extensions from the CSR?

No. OpenSSL warns that copyall can let a requester obtain CA:TRUE. The issuer applies explicit CA:FALSE, key usage, EKU, SAN, AIA, and CRL extensions after authorization.

Does publishing a CRL mean every client rejects revoked certificates?

No. Each consuming application must fetch a fresh authentic CRL or OCSP response and enforce its configured status policy. Test the actual client.

Recovery

Rollback

Issued certificates and revocations are immutable historical events. Roll back service deployment by restoring the still-valid previous leaf during overlap; recover CA state only from a complete consistent generation.

  1. Pause all issuance/revocation and preserve current index, serial, crlnumber, config, issued certificates, CRLs, and audit logs.
  2. If a service rotation fails, restore the previous valid key/leaf/chain and reload every backend; never bypass hostname or chain validation.
  3. If a profile issued an unauthorized certificate, revoke that serial, publish fresh status, and issue a corrected replacement.
  4. If CA state diverges, keep every signer offline, select one authoritative generation through incident review, and reconcile/revoke affected serials.
  5. If root/intermediate compromise is suspected, execute the CA replacement and trust-anchor removal plan rather than attempting ordinary rollback.

Evidence

Sources and review

Verified 2026-07-24Review due 2026-10-22
OpenSSL 3.5 ca command and warningsofficialOpenSSL 3.5 request generationofficialOpenSSL X.509 v3 extension configurationofficialOpenSSL certificate verificationofficialOpenSSL OCSP command and responder boundaryofficialOpenSSL release strategyofficialOpenSSL current supported downloadsofficial