Build an Amazon S3 backup target with encryption, versioning and lifecycle
Design, deploy and prove an Amazon S3 backup target with explicit object ownership, public-access blocking, SSE-KMS, version history, lifecycle controls, least-privilege roles, optional immutable retention and independent restore evidence.
Provide recoverable S3 backup storage whose cryptographic access, retention, deletion, replication and restore behavior are measured and governed instead of inferred from successful uploads.
- AWS CLI 2.x current supported release
- Amazon S3 General purpose buckets
- AWS KMS Symmetric customer managed keys
- Recovery contract Approved data classification, RPO, RTO, retention, legal hold, storage-class retrieval, replication and cost requirements.
- Short-lived AWS access AWS CLI v2 authenticated to the intended account and Region through SSO, federation, OIDC or another approved temporary role.
aws sts get-caller-identity - Separated roles Existing key-administrator, backup-writer and restore-reader roles with distinct owners and a documented break-glass path.
- Application compatibility The backup producer supports versioned S3, SSE-KMS, checksums and recovery by an exact object version.
- Independent recovery An isolated restore destination and, when required, a separately administered destination account or Region are available for drills.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A dedicated Amazon S3 general purpose bucket for backup data with Bucket owner enforced Object Ownership, all four Block Public Access controls, versioning, transport enforcement, default SSE-KMS encryption, an independently governed customer-managed symmetric KMS key, and S3 Bucket Keys enabled after an explicit request-cost and audit tradeoff review.
- A lifecycle and recovery policy that retains recent current and noncurrent object versions, expires incomplete multipart uploads and old delete markers intentionally, predicts transition and retrieval cost, and never treats expiration as a substitute for a backup product's retention catalog.
- A tested operational boundary covering least-privilege backup writes, restore reads, KMS key policy and grants, optional Object Lock governance or compliance retention, cross-account or cross-Region replication, CloudTrail object-data auditing, cost monitoring, integrity checks, restore drills, and safe retirement.
- A non-sensitive canary is uploaded by the backup writer, encrypted with the expected KMS key and Bucket Key setting, produces a distinct version after overwrite, survives a delete marker, and is restored by version ID to an isolated path by a separate restore identity.
- Public and insecure transport requests are denied, ACLs are disabled, the writer cannot read or delete recovery data, the restore reader cannot write or shorten retention, and key administrators are separate from ordinary backup operators.
- Lifecycle, noncurrent-version count, replication status, Object Lock mode when enabled, KMS availability, audit visibility, storage class retrieval delay, cost, and the latest restore drill are measurable and owned rather than inferred from a green bucket status.
Architecture
How the parts fit together
The backup application writes immutable or append-oriented objects through a dedicated IAM role to an S3 bucket that owns all objects and ignores ACLs. S3 default encryption calls a customer-managed KMS key; a Bucket Key can reduce KMS request volume by using a short-lived bucket-level key, but it changes the KMS request and CloudTrail pattern and should not be represented as per-object KMS authorization. Versioning preserves overwritten and deleted object generations while lifecycle manages their cost and eventual expiry. An optional Object Lock configuration adds WORM retention to specific versions: governance can be bypassed only by a specially authorized request, while compliance cannot be shortened or bypassed even by the account root during the retention interval. Cross-account or cross-Region replication adds a separate failure domain but is asynchronous and needs independent KMS, policy, retention, ownership, and failure monitoring. A restore role reads one chosen version into isolation, validates checksums and application usability, and never writes back into the protected prefix during a drill.
- Define data classification, RPO, RTO, retention, legal hold, retrieval-time, Region, sovereignty, cost, restore, and deletion requirements before creating an irreversible bucket or key.
- Use infrastructure as code to create a KMS key and alias, Bucket owner enforced controls, Block Public Access, versioning, default SSE-KMS and Bucket Key, TLS bucket policy, lifecycle, and ownership tags.
- Preview the CloudFormation change set and separately review every IAM and KMS statement, lifecycle expiration, retention parameter, public access control, and deletion policy.
- Grant the backup writer the narrow prefix plus KMS data-key and decrypt operations required for SSE-KMS multipart upload, while withholding S3 read and delete. Grant the restore reader version-list, version-read, and decrypt only; test denied capabilities.
- Upload a deterministic canary, inspect its version, encryption, checksum and Bucket Key metadata, overwrite it, create a delete marker, and recover an older version to an isolated destination.
- Test optional Object Lock in governance mode with a short approved pilot before considering compliance mode; never use compliance retention as an experiment.
- If replication is required, configure a separately owned destination with versioning, KMS, policy and optional matching Object Lock, then monitor pending bytes, latency and failed operations.
- Operate restore drills, key recovery, lifecycle cost, audit events, storage-class retrieval, replication lag, legal holds, retention expiry and safe decommissioning.
Assumptions
- AWS CLI version 2 is installed from a verified official distribution, returns JSON without a pager in automation, and authenticates through a short-lived role or workload identity rather than long-lived access keys.
- The target AWS partition, account, Region, organizational controls, service control policies, permissions boundaries, and administrator recovery path are known. Examples use placeholders and CloudFormation pseudo parameters instead of literal account IDs.
- The bucket name is globally unique, does not contain confidential organization or customer information, and the chosen Region meets residency, latency, service-availability, replication and KMS requirements.
- The backup application supports S3 versioned object storage, SSE-KMS, multipart uploads, checksums, restore by selected version, and the selected storage-class retrieval latency. It does not require ACLs.
- An existing key-administrator role, backup-writer role and restore-reader role are provisioned and governed outside this stack. Their ARNs are passed as deployment parameters but never hardcoded account identifiers.
- Versioning protects earlier generations but does not prevent an authorized principal from deleting versions, disabling the key, changing lifecycle or deleting the account. Independent roles, Object Lock where justified and replication address different threats.
- Object Lock is optional. Governance mode is tested first; compliance mode requires legal, compliance, retention and cost approval because protected versions cannot be deleted by any principal before expiry.
- Lifecycle rules are coordinated with the backup software's catalog and retention. Deleting an S3 object underneath a backup catalog can produce a backup that appears present but cannot restore.
Key concepts
- Bucket owner enforced
- The recommended S3 Object Ownership setting that disables ACLs and makes the bucket owner own every object. Access is controlled through policies.
- Block Public Access
- Four account or bucket controls that block public ACLs and policies. They complement, rather than replace, a least-privilege bucket policy and organization controls.
- SSE-KMS
- S3 server-side encryption using a symmetric KMS key to protect data keys. Access requires both S3 authorization and applicable KMS authorization.
- S3 Bucket Key
- A bucket-level key mechanism that can reduce direct KMS request cost for SSE-KMS. It changes KMS call frequency and audit granularity and is not supported by every encryption mode.
- Version ID
- An identifier for one object generation in a versioned bucket. A simple delete normally adds a delete marker; explicit version deletion can permanently remove a generation unless retention prevents it.
- Noncurrent version
- An older object generation superseded by a newer version or delete marker. Lifecycle can transition or expire it after both age and optional newer-version count conditions.
- Governance retention
- Object Lock mode that prevents ordinary deletion but allows a principal with `s3:BypassGovernanceRetention` to explicitly bypass it. That permission must be extremely narrow.
- Compliance retention
- Object Lock mode that cannot be shortened or bypassed by any user, including root, until the retain-until date. It is an irreversible cost and availability decision for that interval.
- Delete marker
- A version created by an unqualified delete that becomes the current version and makes a normal GET appear deleted while older versions remain.
- Replication time and failure domain
- S3 replication is asynchronous. A destination in another account or Region can improve isolation, but recovery depends on replication completion, destination KMS, policy and retained versions.
Before you copy
Values used in this guide
{{stackName}}Unique CloudFormation stack name for the backup target.
Example: platform-backup-target{{awsRegion}}Approved AWS Region containing source bucket and KMS key.
Example: eu-central-1{{bucketName}}Globally unique, non-sensitive general purpose bucket name.
Example: org-recovery-eu-central-1-random-suffix{{backupPrefix}}Prefix writable by the backup application.
Example: production/cluster-a/{{keyAdministratorRoleArn}}ARN of the independently governed key-administrator role, supplied as a deployment parameter.
Example: arn:aws:iam::<redacted-account-id>:role/backup-key-admin{{backupWriterRoleArn}}ARN of the workload role allowed to write the backup prefix and use KMS encrypt/data-key operations.
Example: arn:aws:iam::<redacted-account-id>:role/backup-writer{{restoreReaderRoleArn}}ARN of the controlled recovery role allowed to list and decrypt selected versions.
Example: arn:aws:iam::<redacted-account-id>:role/restore-reader{{noncurrentRetentionDays}}Approved age after which eligible noncurrent versions may expire.
Example: 180{{newerVersionsToRetain}}Minimum number of newer noncurrent versions retained before an older version becomes lifecycle-eligible.
Example: 12{{canaryFile}}Local deterministic non-sensitive restore fixture.
Example: ./recovery-canary.txt{{canaryKey}}Object key for the restore fixture.
Example: production/cluster-a/tests/recovery-canary.txt{{versionId}}Exact historical S3 object version recorded during the canary overwrite exercise.
Example: 3HL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY{{restoreDirectory}}Isolated local directory used for a restore drill.
Example: ./restore-drill-202607{{expectedDigest}}SHA-256 digest recorded before upload.
Example: a4b2...reviewed-full-digestSecurity and production boundaries
- Do not create or paste long-lived AWS access keys. Use IAM Identity Center, STS role assumption, an instance or workload role, and a separate break-glass process with CloudTrail visibility.
- The KMS key policy is a cryptographic availability boundary. Removing the S3 or role permission, disabling the key, scheduling deletion, losing the owning account, or breaking cross-account policy can make every encrypted backup unreadable.
- Bucket Keys reduce KMS calls and cost but also reduce the number of per-object KMS events. Record whether this audit tradeoff is acceptable and do not claim one KMS decrypt event per object.
- Default encryption does not by itself prevent a caller from selecting another encryption mode or key when allowed. Enforce the application's required encryption contract with reviewed bucket and IAM policies.
- Block Public Access should also be enforced at account and organization level where possible. A bucket-level green control does not prove another access path, VPC endpoint policy or cross-account principal is safe.
- Object Lock compliance mode cannot be bypassed or shortened. A mistaken multi-year retention can create unavoidable storage cost and prevent lawful deletion; legal and financial approval precedes use.
- Lifecycle expiration is a destructive asynchronous action. Review filters, prefixes, tags, minimum object sizes, noncurrent age and newer-version count against restore evidence before enabling.
- CloudTrail object data events can be high volume and contain bucket and object key metadata. Select protected prefixes, secure the trail destination and retention, and treat object names as potentially sensitive.
Stop before continuing if
- Stop if the authenticated principal, AWS account, partition, Region, bucket name, or organizational policy is ambiguous.
- Stop if any role ARN, account identifier, access key, session token, KMS material, or sensitive object name would be committed or printed.
- Stop if the bucket must use ACLs or public access and no separately approved architecture exists.
- Stop if the KMS key administrator, backup writer and restore reader cannot be separated or the key deletion recovery process is unowned.
- Stop if lifecycle would expire a version before the latest tested restore point or conflicts with the backup application's catalog.
- Stop before enabling Object Lock if retention mode, duration, legal holds, bypass permission, cost and deletion obligations are not approved.
- Stop before compliance mode unless the exact retention has passed governance-mode testing and formal legal/compliance approval.
- Stop if a canary version cannot be restored and verified by digest with the source unavailable.
instruction
Define the recovery contract and trust boundaries
Record the data classes, owners, RPO, RTO, retention, legal holds, restore identities, source account, destination account, Region and expected monthly volume before creating storage. Separate backup writing, restore reading and key administration.
Why this step matters
S3 controls are individually simple, but their combination is a recovery design. A bucket can be private, encrypted and versioned yet still fail the business requirement because the KMS key is unavailable, lifecycle removes versions before the catalog expects, replication never completes, or no independent identity can restore. Writing the contract first turns RPO, RTO and retention into testable controls and identifies decisions that cannot safely be improvised during an incident.
What to understand
Inventory logical backup sets, typical and peak object size, daily change rate, multipart behavior, expected version count, retrieval frequency and the maximum acceptable restoration delay. Storage-class transitions must be evaluated against minimum storage duration and retrieval charges, not only the advertised per-gigabyte price.
Name the writer, restorer, key administrator, auditor and deletion approver as separate roles. Define who can recover when the normal identity provider, workload account or source Region is unavailable. Record whether the independent copy must survive account compromise.
Decide whether overwrite history is sufficient or WORM retention is legally or operationally required. Versioning preserves generations, whereas Object Lock enforces retention on individual versions. Neither replaces an application-consistent backup or a tested restore procedure.
System changes
- No AWS resource changes occur; the output is an approved recovery and authorization design.
Syntax explained
RPO / RTO / retention- RPO limits tolerated data loss, RTO limits recovery duration, and retention defines how long recoverable generations remain. They are separate requirements.
Decision record RPO: 24 hours RTO: 4 hours Recent versions: 35 days Independent copy: required Object Lock: governance pilot only Owner: platform-recovery
Checkpoint: Approve measurable recovery requirements
Continue whenEvery dataset has an owner, classification, RPO, RTO, retention, restore identity, key recovery owner, cost boundary and independent-copy decision.
Stop whenThe team cannot state who may restore, whether legal retention applies, or how the KMS key and source-account loss are recovered.
If this step fails
Stakeholders ask to keep backups forever without a cost or deletion owner.
Likely causeRetention, legal hold and archival requirements have been merged into an undefined policy.
Estimate current and noncurrent bytes, request volume, KMS requests, replication and retrieval by year.
ResolutionObtain a dated retention schedule, legal-hold process and cost owner before encoding lifecycle or immutable retention.
Security notes
- Do not place customer names, credentials or incident details in bucket names, tags or IaC parameter defaults because resource metadata can be visible more broadly than object data.
Alternatives
- Use AWS Backup for supported resources when policy orchestration, vault controls and cross-account recovery better match the workload; still test restoration and key availability.
Stop conditions
- Stop if the design depends on one administrator, one account, one Region or one untested KMS key for both production and recovery.
command
Confirm the AWS CLI identity, Region and organization context
Use AWS CLI v2 with a short-lived administrative session. Confirm the partition, account and caller before previewing any resource. Do not proceed merely because the CLI has credentials.
Why this step matters
AWS account and Region mistakes are common and expensive because bucket names are global, KMS keys are regional and retention can be difficult to reverse. The caller must have permission to preview and deploy the stack but should not rely on long-lived access keys. The displayed identity is evidence for the change record, not proof that the role is authorized by policy.
What to understand
AWS CLI version 2 is the supported major used by this guide. Patch output varies with the current official release; automate against documented JSON fields rather than parsing the human version string.
Confirm the intended partition and organizational guardrails, including service control policies and permissions boundaries. A successful STS call does not prove CloudFormation, KMS and S3 operations will be allowed.
System changes
- No cloud resources are modified.
Syntax explained
sts get-caller-identity- Returns the effective account and principal for the current signed session without revealing credentials.
aws --version && aws sts get-caller-identity && aws configure get regionaws-cli/2.31.0 Python/3.13.7 Linux/6.8.0 exe/x86_64.ubuntu.24
{
"UserId": "AROAXAMPLE:backup-bootstrap",
"Account": "<redacted-account-id>",
"Arn": "arn:aws:sts::<redacted-account-id>:assumed-role/PlatformAdmin/backup-bootstrap"
}
eu-central-1Checkpoint: Match the session to the change approval
aws sts get-caller-identity --output jsonContinue whenThe account and assumed role match the approved backup platform account; the configured Region matches the data-residency decision.
Stop whenThe caller is a root user, a personal long-lived IAM user, an unexpected account or a role without a documented recovery owner.
If this step fails
ExpiredToken or Unable to locate credentials is returned.
Likely causeThe short-lived SSO, federation or workload session is absent or expired.
aws configure listaws sts get-caller-identity
ResolutionReauthenticate through the approved identity provider; do not paste permanent access keys into shell history or environment files.
Security notes
- Redact account IDs and role session names from public tickets. Never capture access keys, session tokens, SSO cache contents or credential-process output.
Alternatives
- Run the deployment through a reviewed CI role with OIDC federation and attach the change-set evidence to the approval.
Stop conditions
- Stop on any identity or Region mismatch.
config
Author the KMS, bucket, lifecycle and transport controls as code
Create a reviewed CloudFormation template. The excerpt shows the critical controls; production code should also include ownership tags, explicit IAM-role policy attachments, access logging or CloudTrail data-event configuration and the approved lifecycle values.
Why this step matters
Infrastructure as code makes encryption, ownership, public-access, transport, versioning and lifecycle reviewable as one system. Explicit Object Ownership avoids relying on create-path defaults. Retain policies prevent an ordinary stack deletion from silently erasing recovery resources. SSE-KMS multipart upload requires both GenerateDataKey and Decrypt because S3 must assemble encrypted parts; the writer therefore receives KMS Decrypt but still lacks S3 GetObject/GetObjectVersion and cannot read backup objects through S3. The restore reader gains version-read and decrypt but not encrypt, write, delete, or key administration.
What to understand
The key policy deliberately names roles passed as parameters. In a mature platform, constrain service use with encryption context and `kms:ViaService`, and preserve an audited key-recovery path. Validate that CloudFormation can create the key without locking administrators out.
Because this bucket enables S3 Bucket Keys, the KMS encryption context is the bucket ARN rather than each object ARN. The key policy therefore combines that exact bucket context with kms:ViaService for regional S3; the S3 identity and bucket policies remain responsible for restricting the writer to backups/.
The lifecycle example expires noncurrent versions only after 35 days while retaining three newer noncurrent generations. Adapt values to backup frequency and catalog behavior. AWS lifecycle timing and storage-class minimum duration can affect both recoverability and cost.
The separate expired-delete-marker rule removes only a delete marker after no object versions remain. It is intentionally separate because ExpiredObjectDeleteMarker cannot be combined with a Days value, and it must never be mistaken for retention of recoverable versions.
Bucket Keys can reduce KMS request cost. They also alter the authorization and audit pattern because S3 reuses a bucket-level key for a limited period; document that tradeoff and use S3 object-level audit controls when per-object access evidence matters.
System changes
- The file defines a retained KMS key, retained versioned S3 bucket, bucket policy and noncurrent lifecycle policy but does not create them until deployment.
Syntax explained
BucketOwnerEnforced- Disables ACLs and makes the bucket owner own uploaded objects, removing an ACL-based cross-account failure mode.
NewerNoncurrentVersions: 3- Preserves at least three newer noncurrent versions in addition to the noncurrent-age condition.
backup-target.yamlAWSTemplateFormatVersion: "2010-09-09"
Parameters:
BucketName: { Type: String }
KeyAdministratorRoleArn: { Type: String }
BackupWriterRoleArn: { Type: String }
RestoreReaderRoleArn: { Type: String }
Resources:
BackupKey:
Type: AWS::KMS::Key
DeletionPolicy: Retain
UpdateReplacePolicy: Retain
Properties:
Description: S3 backup target key
EnableKeyRotation: true
PendingWindowInDays: 30
KeyPolicy:
Version: "2012-10-17"
Statement:
- Sid: KeyAdministration
Effect: Allow
Principal: { AWS: { Ref: KeyAdministratorRoleArn } }
Action: "kms:*"
Resource: "*"
- Sid: BackupEncrypt
Effect: Allow
Principal: { AWS: { Ref: BackupWriterRoleArn } }
Action: [kms:Encrypt, kms:GenerateDataKey, kms:Decrypt]
Resource: "*"
Condition:
StringEquals:
"kms:ViaService": { Fn::Sub: "s3.${AWS::Region}.${AWS::URLSuffix}" }
"kms:EncryptionContext:aws:s3:arn": { Fn::Sub: "arn:${AWS::Partition}:s3:::${BucketName}" }
- Sid: RestoreDecrypt
Effect: Allow
Principal: { AWS: { Ref: RestoreReaderRoleArn } }
Action: [kms:Decrypt, kms:DescribeKey]
Resource: "*"
Condition:
StringEquals:
"kms:ViaService": { Fn::Sub: "s3.${AWS::Region}.${AWS::URLSuffix}" }
"kms:EncryptionContext:aws:s3:arn": { Fn::Sub: "arn:${AWS::Partition}:s3:::${BucketName}" }
BackupBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Retain
UpdateReplacePolicy: Retain
Properties:
BucketName: { Ref: BucketName }
OwnershipControls:
Rules: [{ ObjectOwnership: BucketOwnerEnforced }]
PublicAccessBlockConfiguration:
BlockPublicAcls: true
IgnorePublicAcls: true
BlockPublicPolicy: true
RestrictPublicBuckets: true
VersioningConfiguration: { Status: Enabled }
BucketEncryption:
ServerSideEncryptionConfiguration:
- BucketKeyEnabled: true
ServerSideEncryptionByDefault:
SSEAlgorithm: aws:kms
KMSMasterKeyID: { Fn::GetAtt: [BackupKey, Arn] }
LifecycleConfiguration:
Rules:
- Id: ManageNoncurrentVersions
Status: Enabled
Prefix: backups/
AbortIncompleteMultipartUpload: { DaysAfterInitiation: 7 }
NoncurrentVersionExpiration:
NoncurrentDays: 35
NewerNoncurrentVersions: 3
- Id: RemoveExpiredDeleteMarkers
Status: Enabled
Prefix: backups/
Expiration:
ExpiredObjectDeleteMarker: true
BackupBucketPolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: { Ref: BackupBucket }
PolicyDocument:
Version: "2012-10-17"
Statement:
- Sid: DenyInsecureTransport
Effect: Deny
Principal: "*"
Action: "s3:*"
Resource:
- { Fn::GetAtt: [BackupBucket, Arn] }
- { Fn::Sub: "${BackupBucket.Arn}/*" }
Condition: { Bool: { "aws:SecureTransport": "false" } }backup-target.yaml Resources: BackupKey, BackupBucket, BackupBucketPolicy DeletionPolicy: Retain Object ownership: BucketOwnerEnforced Versioning: Enabled Default encryption: aws:kms Bucket Key: enabled
Checkpoint: Complete peer and security review
Continue whenNo wildcard identity grants permit data access, Retain protects the bucket and key, TLS is required, ACLs are disabled, public access is blocked, lifecycle matches retention, and a key-recovery principal exists.
Stop whenThe template can delete recovery data with the stack, grants the writer S3 read/delete or unconstrained data access beyond the documented multipart KMS requirement, has no recoverable key administrator, or encodes unapproved expiration.
If this step fails
CloudFormation validation reports a malformed policy or lifecycle rule.
Likely causeA property name, YAML structure, policy principal, Fn::Sub string or lifecycle combination is invalid.
aws cloudformation validate-template --template-body file://backup-target.yaml
ResolutionCorrect the template against the current CloudFormation resource reference and rerun static validation before creating a change set.
Security notes
- Do not embed role session credentials, literal secrets or customer identifiers. A KMS key policy can permanently lock out recovery if all administration principals are removed.
Alternatives
- Use a reviewed Terraform or CDK module with equivalent preview, state protection, policy validation and retention semantics.
Stop conditions
- Stop if any reviewer cannot explain every Allow, Deny, lifecycle expiration and retained resource.
command
Validate and create a non-executed CloudFormation change set
Validate syntax, then create a change set without executing it. Review resource replacement, IAM capabilities, parameters and effective policy before changing the account.
Why this step matters
CloudFormation validation confirms structure, not intent. A non-executed change set exposes replacements and mutations while leaving resources unchanged. This is the last safe place to catch a name collision, wrong Region, key replacement, lifecycle deletion or policy expansion without affecting stored recovery data.
What to understand
Capture the generated change-set name and review `describe-change-set` plus the rendered template. A create-only deployment can still create costly or irreversible controls; read-only here means the change set itself does not apply resources.
Do not approve `Replacement: True` for an existing recovery bucket or KMS key without a migration and restore plan. Retain can preserve the old resource, but applications and policies may point to the new one.
System changes
- Creates only CloudFormation change-set metadata; no declared S3 or KMS resource is executed.
Syntax explained
--no-execute-changeset- Creates a deploy change set for inspection but does not apply it.
--parameter-overrides- Supplies reviewed environment-specific names and role ARNs without storing credentials in the template.
Values stay on this page and are never sent or saved.
aws cloudformation validate-template --template-body file://backup-target.yaml && aws cloudformation deploy --template-file backup-target.yaml --stack-name {{stackName}} --region {{awsRegion}} --parameter-overrides BucketName={{bucketName}} KeyAdministratorRoleArn={{keyAdministratorRoleArn}} BackupWriterRoleArn={{backupWriterRoleArn}} RestoreReaderRoleArn={{restoreReaderRoleArn}} --capabilities CAPABILITY_NAMED_IAM --no-execute-changesetWaiting for changeset to be created.. Changeset created successfully. Run the following command to review changes: aws cloudformation describe-change-set --change-set-name awscli-cloudformation-package-deploy-...
Checkpoint: Approve the exact change set
aws cloudformation describe-change-set --stack-name {{stackName}} --region {{awsRegion}}Continue whenOnly the expected KMS key, bucket and bucket policy are added or safely modified; no protected resource is replaced or deleted.
Stop whenThe change set is FAILED, references an unexpected account/Region, replaces a protected resource or expands access beyond the design.
If this step fails
Change set creation fails because the bucket name already exists.
Likely causeS3 general-purpose bucket names are globally unique and the chosen name belongs to another account or existing resource.
aws cloudformation describe-stack-events --stack-name {{stackName}} --region {{awsRegion}}
ResolutionChoose a non-sensitive, globally unique approved name; never attempt to adopt an unknown bucket by weakening ownership controls.
Security notes
- Treat the rendered KMS policy and bucket policy as privileged code. Change-set output may reveal account IDs and role names; store it only in the approved change system.
Alternatives
- Use CloudFormation change sets generated by CI and require two-person approval before execution.
Stop conditions
- Stop on replacement, deletion or unreviewed permission expansion.
command
Execute the reviewed stack and wait for completion
Execute only the approved change set through the controlled deployment path, then wait for a terminal stack state. Preserve events if the operation fails.
Why this step matters
The deployment creates durable recovery resources and a cryptographic dependency. It belongs in a controlled change window even though it does not interrupt an existing application. CloudFormation provides consistent creation order and events, but success does not prove least privilege, restoration or lifecycle correctness.
What to understand
Monitor stack events rather than repeatedly rerunning deploy. If creation fails after the key or bucket exists, inspect retained resources and stack status before retrying with a new name.
Tag ownership, environment, classification and cost allocation in the production template. Avoid sensitive values in tags. Record the template digest and stack ID in the recovery runbook.
System changes
- Creates or updates the KMS key, alias when included, S3 bucket, lifecycle configuration, encryption defaults and transport policy.
Syntax explained
cloudformation deploy- Creates and executes a change set, then waits for the stack operation to finish.
Values stay on this page and are never sent or saved.
aws cloudformation deploy --template-file backup-target.yaml --stack-name {{stackName}} --region {{awsRegion}} --parameter-overrides BucketName={{bucketName}} KeyAdministratorRoleArn={{keyAdministratorRoleArn}} BackupWriterRoleArn={{backupWriterRoleArn}} RestoreReaderRoleArn={{restoreReaderRoleArn}} --capabilities CAPABILITY_NAMED_IAMWaiting for changeset to be created.. Waiting for stack create/update to complete Successfully created/updated stack - backup-target-prod
Checkpoint: Reach a clean terminal stack state
aws cloudformation describe-stacks --stack-name {{stackName}} --region {{awsRegion}} --query 'Stacks[0].[StackStatus,StackId]' --output tableContinue whenStackStatus is CREATE_COMPLETE or UPDATE_COMPLETE and the stack ID is recorded.
Stop whenThe stack rolls back, the bucket or key is unexpectedly retained outside the stack, or a policy validation warning remains unexplained.
If this step fails
The stack enters ROLLBACK_COMPLETE.
Likely causeA resource property, permission, globally unique name, key policy or organizational control blocked creation.
aws cloudformation describe-stack-events --stack-name {{stackName}} --region {{awsRegion}} --max-items 30
ResolutionPreserve events, identify any retained resource, correct the reviewed template, create a fresh change set and avoid force-deleting retained recovery artifacts.
Security notes
- CloudFormation execution is privileged. Use short-lived deployment roles, approval, event logging and drift detection.
Alternatives
- For an existing bucket, stage controls separately and test application compatibility before enabling ACL rejection, encryption or lifecycle.
Stop conditions
- Stop on rollback or any unexplained retained partial resource.
verification
Verify ownership, public access, versioning, encryption and lifecycle
Read every effective bucket control and the KMS key state from AWS. Keep the JSON evidence with the change record rather than relying on the template.
Why this step matters
A successful stack is not equivalent to a conforming backup target. Effective-state reads detect drift, misunderstood defaults and omitted properties. The checks also create a baseline that can be monitored for public-access, versioning, encryption, lifecycle and KMS changes.
What to understand
Confirm the returned KMS ARN is the intended customer-managed key in the chosen Region, not only `aws:kms`. Verify key state is Enabled and rotation is configured according to policy.
Inspect bucket policy status and policy statements separately. Block Public Access is a preventive layer, not proof that every cross-account role, access point or VPC endpoint policy is least privilege.
System changes
- No state changes; the commands read effective controls.
Syntax explained
get-bucket-*- Reads the server-side configuration applied to the named bucket rather than trusting local IaC.
Values stay on this page and are never sent or saved.
aws s3api get-bucket-ownership-controls --bucket {{bucketName}} && aws s3api get-public-access-block --bucket {{bucketName}} && aws s3api get-bucket-versioning --bucket {{bucketName}} && aws s3api get-bucket-encryption --bucket {{bucketName}} && aws s3api get-bucket-lifecycle-configuration --bucket {{bucketName}}{
"OwnershipControls": {"Rules": [{"ObjectOwnership": "BucketOwnerEnforced"}]}
}
{
"PublicAccessBlockConfiguration": {
"BlockPublicAcls": true,
"IgnorePublicAcls": true,
"BlockPublicPolicy": true,
"RestrictPublicBuckets": true
}
}
{"Status":"Enabled"}
{"ServerSideEncryptionConfiguration":{"Rules":[{"BucketKeyEnabled":true,"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms","KMSMasterKeyID":"arn:aws:kms:eu-central-1:<redacted-account-id>:key/<redacted>"}}]}}Checkpoint: Match every effective control to policy
aws kms describe-key --key-id alias/{{stackName}} --region {{awsRegion}}Continue whenOwnership is enforced, all public blocks are true, versioning is Enabled, SSE-KMS uses the approved enabled key, Bucket Key matches the decision, TLS deny exists and lifecycle matches retention.
Stop whenAny control is absent, suspended, points to the wrong key or expires data earlier than the recovery contract.
If this step fails
NoSuchLifecycleConfiguration is returned.
Likely causeLifecycle was omitted, deployment failed or a later change removed it.
aws cloudformation detect-stack-drift --stack-name {{stackName}} --region {{awsRegion}}
ResolutionReconcile the effective bucket with reviewed IaC; do not manually add an expiration rule during investigation.
Security notes
- Store configuration evidence according to internal metadata classification; policies can reveal trusted principals and account structure.
Alternatives
- Use AWS Config managed or custom rules plus EventBridge alerts for continuous control monitoring, while retaining restore drills.
Stop conditions
- Stop before backup writes if any baseline check fails.
command
Upload a deterministic encrypted canary
Create a non-sensitive file with stable contents, calculate its SHA-256 digest and upload it under the protected prefix. Inspect server-side encryption, version ID and checksum metadata.
Why this step matters
A deterministic canary proves that the writer path can create an object and that S3 applies the expected encryption and versioning defaults. The digest becomes independent restore evidence. It must contain no production data because the exercise intentionally creates versions and may replicate them.
What to understand
Run the upload while assuming the backup-writer role, not the administrator. Verify the writer can use GenerateDataKey and the KMS Decrypt operation required internally by SSE-KMS multipart upload, yet has no S3 GetObject/GetObjectVersion, cannot delete versions, cannot alter lifecycle, and cannot administer the KMS key.
Record the returned VersionId and checksum. Server-side encryption protects storage, while the digest checks the recovered bytes. If the backup application already supplies stronger application-level manifests, preserve both.
System changes
- Creates one encrypted object version under the canary key and incurs small S3 and KMS request/storage charges.
Syntax explained
--checksum-algorithm SHA256- Asks S3 to calculate and store a SHA-256 checksum for transfer-integrity validation.
--body- Uploads the local deterministic canary file; it must not contain secrets.
Values stay on this page and are never sent or saved.
printf 'oneliners-s3-restore-canary-v1\n' > {{canaryFile}} && sha256sum {{canaryFile}} && aws s3api put-object --bucket {{bucketName}} --key {{canaryKey}} --body {{canaryFile}} --checksum-algorithm SHA2569a6ff3d8d8fb74b2bb8e94a7c98a42f71567246db8668b579122474033abb831 canary.txt
{
"ServerSideEncryption": "aws:kms",
"SSEKMSKeyId": "arn:aws:kms:eu-central-1:<redacted-account-id>:key/<redacted>",
"BucketKeyEnabled": true,
"VersionId": "3HL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY"
}Checkpoint: Record version, encryption and digest
aws s3api head-object --bucket {{bucketName}} --key {{canaryKey}}Continue whenThe response contains a version, aws:kms, the approved key ARN, expected BucketKeyEnabled value and a matching checksum/digest.
Stop whenThe object uses the wrong key, has no version ID, cannot be read by metadata check, or the writer has broader permissions than intended.
If this step fails
AccessDenied or KMS.AccessDeniedException occurs on put-object.
Likely causeThe bucket policy, writer identity policy, KMS key policy, encryption context or organizational guardrail denies the request.
aws sts get-caller-identityaws kms describe-key --key-id alias/{{stackName}} --region {{awsRegion}}
ResolutionTrace the denied action and grant only the required prefix and KMS encrypt/data-key operations; do not add broad `s3:*` or `kms:*` permissions.
Security notes
- Avoid command-line metadata containing secrets. CloudTrail and shell history can retain object keys and arguments.
Alternatives
- Use a backup-product canary job with an application-level digest, provided it exercises the same writer identity and encryption path.
Stop conditions
- Stop if encryption, versioning or least-privilege assertions fail.
command
Exercise overwrite, delete marker and version inventory
Upload changed canary contents, issue an ordinary delete and list every version and delete marker. This demonstrates what versioning preserves and which operations the recovery process must understand.
Why this step matters
An ordinary delete on a versioned bucket creates a delete marker rather than erasing prior versions. This behavior supports recovery but can confuse applications that interpret the current key as absent. The inventory proves that prior bytes remain and exposes the exact version IDs required for deterministic restore.
What to understand
Versioning does not prevent a principal with `DeleteObjectVersion` from permanently deleting a chosen generation. The writer role should not have that action. Object Lock, when enabled, governs deletion of protected versions separately.
Lifecycle evaluates noncurrent age and retained newer-version count asynchronously. Do not infer exact deletion time from rule activation; monitor inventory and preserve enough margin around the RPO and catalog.
System changes
- Creates a second object version and a current delete marker; earlier versions remain billable and recoverable.
Syntax explained
delete-object without --version-id- Creates a delete marker in a version-enabled bucket instead of permanently removing a prior version.
list-object-versions- Returns both object generations and delete markers; ordinary list-objects hides deleted current keys.
Values stay on this page and are never sent or saved.
printf 'oneliners-s3-restore-canary-v2\n' > {{canaryFile}} && aws s3api put-object --bucket {{bucketName}} --key {{canaryKey}} --body {{canaryFile}} --checksum-algorithm SHA256 && aws s3api delete-object --bucket {{bucketName}} --key {{canaryKey}} && aws s3api list-object-versions --bucket {{bucketName}} --prefix {{canaryKey}}{
"DeleteMarker": true,
"VersionId": "2LWg7lQLnY41.maGB5Z6SWW.dcq0mDgP"
}
{
"Versions": [
{"Key":"backups/canary.txt","VersionId":"Kqx0Y...","IsLatest":false,"Size":31},
{"Key":"backups/canary.txt","VersionId":"3HL4k...","IsLatest":false,"Size":31}
],
"DeleteMarkers": [
{"Key":"backups/canary.txt","VersionId":"2LWg7...","IsLatest":true}
]
}Checkpoint: Prove multiple generations remain
Continue whenTwo object versions and one current delete marker exist, with distinct version IDs recorded for the restore drill.
Stop whenEarlier versions are absent, versioning is Suspended, or the writer can permanently delete a version.
If this step fails
Only one version exists after overwrite.
Likely causeVersioning was not enabled before the first upload, writes used a different key, or lifecycle already removed an older version.
aws s3api get-bucket-versioning --bucket {{bucketName}}aws s3api list-object-versions --bucket {{bucketName}} --prefix {{canaryKey}}
ResolutionCorrect versioning and lifecycle, repeat with a new canary key, and do not claim recoverability for objects written before versioning.
Security notes
- Do not grant backup automation `s3:DeleteObjectVersion` unless a separately approved expiry workflow truly requires it.
Alternatives
- If backup software writes immutable unique keys, test catalog deletion and object retention rather than overwrite behavior.
Stop conditions
- Stop if any expected generation is not recoverable.
command
Restore a selected version into isolation and verify its digest
Assume the restore-reader role, choose the recorded first version ID, download it into a new directory and compare the digest with the expected value. Do not restore over a production path during the drill.
Why this step matters
Restore is the only convincing evidence that storage, version selection, identity, KMS decrypt and object integrity work together. A separate reader role tests privilege separation. An isolated directory prevents the exercise from overwriting live data and allows application-level validation before any production replacement.
What to understand
Compare the digest to the value recorded before upload, not to a digest calculated from an untrusted current source. For databases or archives, also perform format validation and an application-consistency test.
Measure time from restore request to usable data. If lifecycle transitions versions to an archive class, include restore initiation and retrieval completion in the RTO rather than testing only Standard storage.
System changes
- Reads one selected object version and writes a local restored file; it does not change the bucket.
Syntax explained
--version-id- Selects a precise historical object generation even when the current key is hidden by a delete marker.
{{restoreDirectory}}/canary-restored.txt- Keeps the drill output isolated from the source and any production restore destination.
Values stay on this page and are never sent or saved.
mkdir -p {{restoreDirectory}} && aws s3api get-object --bucket {{bucketName}} --key {{canaryKey}} --version-id {{versionId}} {{restoreDirectory}}/canary-restored.txt && sha256sum {{restoreDirectory}}/canary-restored.txt{
"AcceptRanges": "bytes",
"ServerSideEncryption": "aws:kms",
"SSEKMSKeyId": "arn:aws:kms:eu-central-1:<redacted-account-id>:key/<redacted>",
"VersionId": "3HL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY",
"BucketKeyEnabled": true
}
9a6ff3d8d8fb74b2bb8e94a7c98a42f71567246db8668b579122474033abb831 restore/canary-restored.txtCheckpoint: Pass independent restore evidence
test "$(sha256sum {{restoreDirectory}}/canary-restored.txt | awk '{print $1}')" = "{{expectedDigest}}"Continue whenThe selected historical version downloads through the restore role, uses the expected key and exactly matches the pre-upload SHA-256 digest within RTO.
Stop whenThe version is missing, KMS decrypt fails, bytes differ, restoration exceeds RTO or the reader can write/delete.
If this step fails
get-object returns InvalidObjectState.
Likely causeThe selected version is in an archival storage class and needs a temporary restore before retrieval.
aws s3api head-object --bucket {{bucketName}} --key {{canaryKey}} --version-id {{versionId}}
ResolutionInitiate the approved storage-class restore, monitor completion and include the actual delay in the RTO evidence.
Security notes
- Restore destinations inherit the sensitivity of backup data. Use encrypted isolated storage, restrict access, record custody and securely remove drill output under policy.
Alternatives
- Restore into a quarantined recovery account or environment when application validation needs infrastructure and network isolation.
Stop conditions
- Stop before production recovery if integrity or application validation fails.
decision
Pilot Object Lock without making compliance promises
If WORM is required, use a new test bucket or an approved empty pilot, enable Object Lock and exercise short governance retention first. Compliance mode is not a stronger test setting; it is an irreversible legal and cost commitment for the retention interval.
Why this step matters
Object Lock protects specific versions against deletion and overwrite of retention metadata. Governance mode permits an explicitly authorized bypass request; compliance mode cannot be shortened or bypassed by any user, including root, until expiry. Once Object Lock is enabled for a bucket it cannot be disabled and versioning cannot be suspended, so the decision needs legal, capacity, key and account-lifecycle review.
What to understand
Test ordinary delete, explicit version delete without bypass, and an approved governance bypass using a short-lived break-glass role. Confirm CloudTrail captures the action and that the backup writer has no bypass permission.
Plan KMS key availability for the entire retained lifetime. Retained encrypted bytes are not recoverable if the key is permanently deleted. Conversely, compliance-retained data can continue to create storage and account obligations until expiry.
Replication of locked objects needs compatible destination configuration and permissions. Verify retention and legal-hold propagation rather than assuming the source policy is mirrored.
System changes
- The read command changes nothing, but the decision can lead to irreversible bucket Object Lock and retained versions.
Syntax explained
GOVERNANCE- Prevents ordinary deletion but permits a specifically authorized `BypassGovernanceRetention` request.
COMPLIANCE- Prevents retention shortening and deletion for all principals until expiry; no administrative bypass exists.
Values stay on this page and are never sent or saved.
aws s3api get-object-lock-configuration --bucket {{bucketName}}{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {"Mode":"GOVERNANCE","Days":7}
}
}
}Checkpoint: Approve or reject immutable retention
Continue whenLegal, security, recovery, KMS, finance and account owners approve the exact mode, duration, scope, bypass role, audit and disposal procedure.
Stop whenAnyone describes compliance mode as reversible, the key lifecycle is shorter than data retention, or budget/account closure cannot support retained bytes.
If this step fails
A governance-protected test version cannot be deleted by the break-glass role.
Likely causeThe role lacks `s3:BypassGovernanceRetention`, the request omitted the bypass header or an explicit deny applies.
aws s3api get-object-retention --bucket {{bucketName}} --key {{canaryKey}} --version-id {{versionId}}
ResolutionReview the break-glass policy and request path in the pilot; never broaden the routine writer role or switch to compliance as a workaround.
Security notes
- Object Lock is not ransomware immunity if attackers can disable or delete the KMS key, compromise the account, change future retention or prevent new backups.
Alternatives
- Use governance mode, a separate AWS Backup logically air-gapped vault, or cross-account destination according to the threat and regulatory requirement.
Stop conditions
- Do not enable Object Lock on an existing production bucket until irreversibility, retention and cost are explicitly approved.
verification
Validate the independent copy and monitoring path
If replication is required, inspect configuration and the canary's replication status, then restore from the destination under an independently administered identity. Configure alarms for failed operations and replication latency.
Why this step matters
Replication is asynchronous and can remain pending or fail because of source permissions, destination ownership, KMS policy, Object Lock, Region or quota. A configured rule is not proof of an independent backup. Recovery must be demonstrated from the destination after simulated loss of ordinary source access.
What to understand
Use a separate account and independently governed destination key when account compromise is in scope. Cross-Region replication alone does not isolate account administrators; same-Region cross-account replication can isolate ownership without changing residency.
Monitor replication failure metrics, pending bytes and maximum time rather than only spot-checking one object. Existing objects may require S3 Batch Replication or a separate migration; ordinary replication rules apply prospectively.
System changes
- No state changes; reads source replication metadata.
Syntax explained
ReplicationStatus- Reports whether this source object version is pending, completed or failed for replication.
Values stay on this page and are never sent or saved.
aws s3api get-bucket-replication --bucket {{bucketName}} && aws s3api head-object --bucket {{bucketName}} --key {{canaryKey}} --version-id {{versionId}}{
"ReplicationConfiguration": {
"Role": "arn:aws:iam::<redacted-account-id>:role/s3-backup-replication",
"Rules": [{"ID":"independent-copy","Status":"Enabled","Priority":1}]
}
}
{
"VersionId": "3HL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY",
"ReplicationStatus": "COMPLETED"
}Checkpoint: Restore from the independent failure domain
Continue whenThe canary reaches COMPLETED, destination encryption and retention are correct, alarms are active and a destination-only restore matches the digest.
Stop whenThe copy is pending/failed, shares an unapproved administrative dependency, lacks destination KMS recovery or cannot restore independently.
If this step fails
ReplicationStatus remains PENDING beyond the objective.
Likely causeThe destination policy, replication IAM role, KMS grants, versioning, ownership, Object Lock or service quota is incomplete.
aws s3api get-bucket-replication --bucket {{bucketName}}aws s3api head-object --bucket {{bucketName}} --key {{canaryKey}} --version-id {{versionId}}
ResolutionInspect replication metrics and failure reasons, repair the narrow permission or configuration, and do not count pending bytes toward the recoverable copy.
Security notes
- The destination must not trust the same broad human role if separation from source-account compromise is a requirement.
Alternatives
- Use a provider-native backup vault or scheduled, independently authenticated copy when object replication semantics do not meet application consistency.
Stop conditions
- Stop release if required independent recovery has not been tested.
verification
Operationalize restore drills, audit, cost and key recovery
Schedule recurring restore drills and review lifecycle, versions, KMS key state, replication, failed multipart uploads, inventory, data events, access changes and cost. Treat a drill as failed until the restored application or archive is usable.
Why this step matters
Backup controls decay through policy drift, expired identities, key changes, lifecycle edits, growing data volume, retrieval transitions and applications that no longer restore. Scheduled evidence turns the target into an operated recovery service. Cost review is a safety control because unexpected noncurrent growth or immutable retention can threaten account sustainability.
What to understand
Quarterly drills should select a recent and a noncurrent version, use the documented reader role, measure retrieval and application validation, and record sanitized evidence. Annually exercise a scenario in which the ordinary source workload role is unavailable.
Alert on key disablement or deletion scheduling, public-access and bucket-policy changes, versioning suspension, lifecycle changes, failed replication, unexpected version deletion, unusual reads and denied backup writes. Scope CloudTrail data events to balance evidence and cost.
Review S3 Inventory, Storage Lens or equivalent reports for encryption, replication, version, retention and storage-class state. Reconcile stored objects with the backup catalog and investigate orphaned multipart uploads or unexpected delete markers.
System changes
- Read commands do not change state; the operational process creates recurring evidence, alarms and recovery tasks.
Syntax explained
describe-key- Confirms the KMS key remains enabled and identifies its governing metadata; it does not prove every restore role can decrypt.
Values stay on this page and are never sent or saved.
aws s3api get-bucket-versioning --bucket {{bucketName}} && aws s3api get-bucket-lifecycle-configuration --bucket {{bucketName}} && aws kms describe-key --key-id alias/{{stackName}} --region {{awsRegion}}Versioning: Enabled Lifecycle rule: ManageNoncurrentVersions / Enabled KMS key state: Enabled Key rotation: Enabled Latest restore drill: PASS Digest match: PASS RTO: 00:17:42 Replication: COMPLETED Unresolved alerts: 0
Checkpoint: Accept the backup target as an operated service
Continue whenCurrent and historical restores pass inside RTO, alerts have owners, lifecycle and catalog reconcile, the independent copy is usable, key recovery is tested and cost remains within budget.
Stop whenAny backup success metric lacks a recent restore, a key or role has no owner, or lifecycle/replication state cannot be reconciled.
If this step fails
Storage cost rises despite stable current backup size.
Likely causeNoncurrent versions, delete markers, incomplete multipart uploads, replication, minimum-duration charges or immutable objects are accumulating.
aws s3api list-object-versions --bucket {{bucketName}} --prefix {{backupPrefix}} --max-items 100aws s3api list-multipart-uploads --bucket {{bucketName}} --prefix {{backupPrefix}}
ResolutionReconcile version and multipart inventory with retention and legal requirements, model lifecycle changes, test them on a prefix and never delete protected generations ad hoc.
Security notes
- Restore-drill evidence must not include object contents, credentials, KMS policy secrets or customer identifiers. Store audit output under the same change and incident controls as infrastructure evidence.
Alternatives
- A managed backup service can schedule jobs and vault controls, but application validation, key recovery, independent identity and restore evidence remain necessary.
Stop conditions
- Escalate immediately when current backups continue but restore, key, replication or retention evidence fails.
Finish line
Verification checklist
aws s3api get-bucket-ownership-controls --bucket {{bucketName}} && aws s3api get-public-access-block --bucket {{bucketName}} && aws s3api get-bucket-versioning --bucket {{bucketName}} && aws s3api get-bucket-encryption --bucket {{bucketName}}ACLs are disabled, all public-access controls are true, versioning is Enabled and default SSE-KMS references the approved enabled key.aws s3api get-bucket-lifecycle-configuration --bucket {{bucketName}} && aws s3api get-bucket-policy-status --bucket {{bucketName}}Lifecycle matches the recovery contract and the bucket is not public; separately reviewed cross-account and service access remains least privilege.aws s3api get-object --bucket {{bucketName}} --key {{canaryKey}} --version-id {{versionId}} {{restoreDirectory}}/final-canary.txt && sha256sum {{restoreDirectory}}/final-canary.txtThe selected noncurrent version is decrypted by the restore role into isolation and matches the digest recorded before upload.aws s3api head-object --bucket {{bucketName}} --key {{canaryKey}} --version-id {{versionId}}When replication is required, status is COMPLETED and the destination-only restore passes with independent identity and key recovery.aws kms describe-key --key-id alias/{{stackName}} --region {{awsRegion}}Key state, versioning, lifecycle, replication, alarms, inventory, cost and recent restore drill all have current evidence and owners.Recovery guidance
Common problems and safe checks
CloudFormation cannot create the bucket because the name already exists.
Likely causeS3 bucket names are globally unique across AWS accounts in a partition.
aws sts get-caller-identityaws cloudformation describe-stack-events --stack-name {{stackName}} --region {{awsRegion}}
ResolutionChoose a non-sensitive organization convention plus a random suffix; never attempt to adopt an unknown existing bucket.
PutObject is denied even though the writer has S3 permission.
Likely causeKMS key policy, IAM KMS permission, bucket policy, VPC endpoint policy, organization control, prefix, Region or encryption header rejects the request.
aws s3api get-bucket-policy-status --bucket {{bucketName}}aws kms describe-key --key-id alias/{{stackName}} --region {{awsRegion}}
ResolutionTrace both S3 and KMS authorization plus explicit denies; grant only the missing operation at the narrow resource and condition.
Objects show AES256 instead of aws:kms.
Likely causeThe object predates default encryption, the request targeted another bucket, or a caller explicitly selected allowed SSE-S3.
aws s3api head-object --bucket {{bucketName}} --key {{canaryKey}}aws s3api get-bucket-encryption --bucket {{bucketName}}
ResolutionCorrect destination and policy, require the approved KMS contract, and copy existing objects through a reviewed re-encryption procedure.
Restore fails with KMS AccessDenied.
Likely causeRestore role lacks decrypt, key policy excludes it, encryption context differs, key is disabled or pending deletion, or the version used another key.
aws kms describe-key --key-id alias/{{stackName}} --region {{awsRegion}}aws s3api head-object --bucket {{bucketName}} --key {{canaryKey}} --version-id {{versionId}}
ResolutionRestore key availability and the least-privilege decrypt path for the exact key; cancel an unauthorized pending deletion under incident control.
A normal GET returns NoSuchKey after deletion.
Likely causeA delete marker is the current version while older versions still exist.
aws s3api list-object-versions --bucket {{bucketName}} --prefix {{canaryKey}}
ResolutionSelect the intended previous version ID and restore it to an isolated key or remove only the reviewed delete marker.
Lifecycle does not expire an eligible noncurrent version immediately.
Likely causeLifecycle is asynchronous, age or newer-version count is not met, filter mismatches, retention or legal hold blocks deletion, or replication status is pending.
aws s3api get-bucket-lifecycle-configuration --bucket {{bucketName}}aws s3api list-object-versions --bucket {{bucketName}} --prefix {{canaryKey}}
ResolutionVerify every condition and wait for documented asynchronous processing; never delete versions manually to force lifecycle.
KMS request cost is much higher than forecast.
Likely causeBucket Key is disabled, multipart workload is high, objects use another mode, replication adds KMS calls, or clients perform frequent copies.
aws s3api get-bucket-encryption --bucket {{bucketName}}aws s3api head-object --bucket {{bucketName}} --key {{canaryKey}}
ResolutionConfirm Bucket Key and workload patterns, model audit tradeoffs, quotas and replication, then change only through measured policy.
Governance-protected version is deleted unexpectedly.
Likely causeThe caller had `s3:BypassGovernanceRetention` and explicitly or through the console sent the bypass header.
aws s3api get-object-retention --bucket {{bucketName}} --key {{canaryKey}} --version-id {{versionId}}
ResolutionRevoke broad bypass permission, inspect CloudTrail, restore from replication or another version, and tighten policy conditions and approval.
Compliance-protected data cannot be deleted after a mistaken retention.
Likely causeCompliance mode correctly prevents shortening or bypassing retention.
aws s3api get-object-retention --bucket {{bucketName}} --key {{canaryKey}} --version-id {{versionId}}
ResolutionThere is no bypass. Preserve the account and budget until expiry, obtain legal guidance, and correct future default retention through approved forward changes.
Replication remains Pending or reports Failed.
Likely causeDestination bucket versioning, Object Lock, IAM role, destination KMS policy, Region, ownership, quota or replication rule is incomplete.
aws s3api head-object --bucket {{bucketName}} --key {{canaryKey}}aws s3api get-bucket-replication --bucket {{bucketName}}
ResolutionRepair the exact source and destination permissions and configuration, monitor failed operations, and do not count pending objects as an independent copy.
Restore from archival storage is not immediate.
Likely causeLifecycle transitioned the version to a storage class requiring restore initiation and retrieval delay.
aws s3api head-object --bucket {{bucketName}} --key {{canaryKey}} --version-id {{versionId}}
ResolutionInitiate the class-specific temporary restore, monitor completion, and align RTO with retrieval tier or retain a faster recent tier.
Bucket policy status says non-public but an unexpected principal reads data.
Likely causeCross-account role trust, identity policy, access point, VPC endpoint policy, KMS grant, resource policy or organization relationship permits access without being public.
aws s3api get-bucket-policy --bucket {{bucketName}}aws s3api get-bucket-acl --bucket {{bucketName}}
ResolutionReview all authorization layers and CloudTrail object events; public status is not a complete least-privilege assessment.
CloudFormation delete leaves the bucket and key.
Likely causeDeletionPolicy Retain intentionally preserves recovery data and cryptographic access.
aws cloudformation describe-stacks --stack-name {{stackName}} --region {{awsRegion}}aws s3api get-bucket-versioning --bucket {{bucketName}}
ResolutionTreat retained resources as a separate retirement project with inventory, legal, restore, key, version and cost approval; never force-delete.
Recovery
Rollback
Do not treat rollback as bucket deletion. Stop new writers, preserve versions and audit evidence, restore the last reviewed IaC controls, and retire data only through a separately approved, inventory-driven process that respects Object Lock and KMS availability.
- Disable or redirect the backup producer through its application configuration without deleting existing object versions.
- Preserve the failed change set, stack events, effective bucket policy, lifecycle, versioning, encryption, KMS key state and a version inventory.
- If a reversible policy or lifecycle change caused the incident, create and review a forward CloudFormation change set restoring the last approved configuration.
- Do not suspend versioning, schedule KMS deletion, shorten compliance retention or force-empty the bucket as a recovery action.
- Restore a known historical canary and one representative backup into isolation, validate integrity and measure RTO before resuming writes.
- For retirement, stop writes, reconcile the application catalog with every version and delete marker, satisfy retention/legal holds, validate independent copies, estimate cost and obtain data-owner, security and legal approval.
- Keep the KMS key enabled until every required encrypted version is expired or verifiably migrated and all recovery obligations are closed.
Evidence