Cloud & Infrastructure / DevOps & CI/CD

Build reusable Terraform modules with remote state and locking

Design a real module contract, keep provider and backend ownership in environment roots, pin releases, protect sensitive state in versioned encrypted S3 storage, use Terraform 1.15 native lockfiles, and rehearse upgrades and recovery.

240 min12 stepsChanges system stateRevision 1

Save or explore

#terraform #modules #remote-state #state-locking #s3 #infrastructure-as-code #aws #devops

0 of 12 steps completed

Goal

Deliver a reusable Terraform module and production root whose inputs, outputs, versions, state boundary, locking, security, plan review, apply, recovery, and lifecycle are explicit, testable, and safe for a team.

Supported environments

Terraform Community Edition 1.15.x
Amazon S3 backend Native lockfile support
AWS provider 6.x example

Prerequisites

Supported toolchain Terraform 1.15.5 is installed from the approved distribution, every runner uses the reviewed 1.15 line, and provider checksums can be committed for supported platforms.terraform version
Existing inventory Every remote object, current resource address, provider alias, state key, owner, consumer, immutable field, and secret-bearing attribute is inventoried before module extraction.terraform providers && terraform workspace show && terraform state list
Durable backend A separately owned S3 bucket and KMS key provide versioning, encryption, public-access blocking, audit logging, retention, least privilege, and recoverable historical versions.
Short-lived identities Plan, apply, and state-recovery roles are distinct, use workload identity, and have only the provider and backend permissions required for their task.
Recovery access Console and cloud-provider recovery paths exist, a second reviewer is available for backend changes, and historical state can be restored into an isolated key.
Change window The initial production apply, concurrency test, and recovery rehearsal have owners, monitoring, stop conditions, and a protected evidence store.

Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System

A small, composable Terraform module with typed inputs, useful outputs, provider requirements, examples, tests, semantic versioning expectations, and an upgrade path that does not hide environment decisions inside the abstraction.
A separate production root module that pins Terraform, providers, and the module; supplies provider configuration; and stores its own sensitive state in an Amazon S3 backend with object versioning, encryption, least-privilege access, and Terraform 1.15 native S3 lockfiles.
A review workflow that initializes from a clean checkout, validates and tests the module, saves a plan, proves state locking, rejects destructive surprises, applies only the reviewed artifact, verifies outputs, and rehearses state recovery without touching production objects.

Observable outcome

Two independent root modules can call the same versioned child module with different reviewed inputs and provider configurations, while their state remains isolated by backend key and access policy.
Concurrent operations against the same production key serialize on the native S3 lock object; a second operation waits or fails closed instead of proceeding with `-lock=false`.
A no-change plan, module test, backend version inventory, encrypted state recovery exercise, and explicit upgrade rollback provide evidence that reuse did not trade clarity for hidden coupling.

Architecture

How the parts fit together

The reusable child module and the environment root have deliberately different responsibilities. The child module expresses a stable infrastructure capability using resources, typed variables, validations, outputs, provider requirements, documentation, examples, tests, and historical moved blocks. It does not contain credentials, backend configuration, environment account IDs, provider regions, or mutable organizational policy. The root module selects the module release, configures providers, supplies environment-specific values, and owns one remote state key. Terraform Community Edition 1.15.5 initializes providers and the child module from immutable or constrained sources. The S3 backend stores encrypted state, bucket versioning retains recoverable generations, and `use_lockfile = true` creates native lock objects. CI and operators authenticate through short-lived roles with different plan, apply, and state-administration permissions. Plans are treated as sensitive change artifacts and are applied only while their configuration and state context remains valid.

Reusable child moduleRaises the abstraction above individual provider resources, validates inputs, exports a narrow interface, documents behavior, and preserves resource addresses across compatible releases.

Environment root moduleOwns the backend, provider configurations, state boundary, module version, environment values, approval policy, and operational verification.

Private or public module registryDistributes immutable semantic releases and lets callers use an explicit `version` constraint. A VCS source is acceptable only when pinned to an immutable reviewed revision.

S3 state bucketStores a separate versioned state object per root and the associated native lock object. Encryption, retention, audit logging, and least-privilege paths protect both confidentiality and recoverability.

KMS and workload identityEncrypt state with a recoverable managed key and give short-lived runner roles only the actions required for their state path and infrastructure responsibility.

Review pipelineRuns formatting, validation, module tests, policy checks, a locked refresh-backed plan, human review, saved-plan apply, and post-change verification from a reproducible toolchain.

Inventory the capability, ownership boundary, existing state, sensitive values, environments, provider aliases, failure modes, and consumers before deciding that a module is the correct abstraction.
Build the child module in the standard structure, declare compatible Terraform and provider requirements, define typed inputs with validations, export narrow outputs, and add examples and tests.
Bootstrap the state bucket and KMS controls through a separate protected root or provider-native process, because a backend cannot safely create the storage it needs before initialization.
Configure only static, non-secret S3 backend coordinates in the root; pass credentials through workload identity and enable native lockfiles rather than the deprecated DynamoDB locking path.
Pin the module release and provider selection, initialize from a clean checkout, inspect resolved dependencies, test the module, and commit `.terraform.lock.hcl` while excluding `.terraform`, state, plans, and credentials.
Create and review a saved plan under the state lock, reject replacement or deletion that is not an explicit migration, apply the exact artifact, and verify both the managed service and state health.
Exercise concurrent-lock behavior and restore a prior version into an isolated recovery key, then document module upgrade, backend recovery, force-unlock, and rollback ownership.

Assumptions

Terraform Community Edition 1.15.5 is the reviewed stable toolchain. Every runner uses the same supported minor line and a committed provider dependency lock file; experimental Terraform 1.16 builds are excluded from production.
The example uses Amazon S3 because Terraform's current S3 backend provides native lockfiles. The architecture is portable only to backends that provide equivalent locking, encryption, access control, version recovery, and audit evidence.
The S3 bucket, KMS key, and backend roles are bootstrapped separately and survive deletion of any application stack. Destroying an application root must never destroy its own recovery state.
The child module represents a meaningful infrastructure capability rather than a thin wrapper around one resource. Environment policy and provider authentication remain visible in the root.
Each root module owns one reviewed state boundary. Sharing outputs does not grant a consumer read access to the complete sensitive state when a narrower publication mechanism is available.
All applies occur through an approved serialized workflow. Local operators may plan with read access, but production writes and force-unlock require stronger roles and a change record.
The organization can recover KMS keys and retained state versions. Object versioning without tested decrypt permission and a restore procedure is not considered a backup.
No example command contains a credential. Short-lived identity, session boundaries, shell history, debug logs, saved plans, and pipeline artifacts follow the organization's secret-handling policy.

Key concepts

Root module: The configuration directory from which Terraform runs. It owns backend and provider configuration, calls child modules, supplies environment-specific values, and is the correct boundary for a state workspace.
Child module: A reusable collection of resources called by a root or another child module. A child module declares provider requirements but normally receives provider configurations from its caller and does not decide where state is stored.
State binding: The association between a Terraform resource instance address and one remote object identity. Safe operations preserve a one-to-one mapping; duplicate imports and empty or wrong state keys violate that assumption.
Backend: The subsystem that stores state and, depending on type, provides locking and operation features. Backend configuration is evaluated before ordinary Terraform expressions, so it cannot use normal input variables or resource outputs.
State lock: A serialization mechanism that prevents concurrent writers from calculating and committing changes against the same snapshot. It protects consistency only when every writer honors it.
Saved plan: A binary plan artifact that captures a reviewed proposal for a particular configuration, state lineage, provider set, and variable context. It is sensitive and becomes stale when any relevant input changes.
Refresh: The read of provider APIs that updates Terraform's in-memory understanding of remote objects before comparison. A refresh can reveal drift even when the configuration repository did not change.
Dependency lock file: The `.terraform.lock.hcl` record of selected provider versions and checksums. It should be committed and reviewed; it does not lock child module versions or serialize state operations.
Module contract: The documented inputs, outputs, provider requirements, behavior, security assumptions, and upgrade guarantees that callers rely on. Internal resource names are implementation details until state compatibility makes address changes operationally significant.
Native S3 lockfile: The S3 backend locking mode enabled by `use_lockfile = true`. Terraform coordinates through a sibling `.tflock` object; DynamoDB-based S3 locking is deprecated in current Terraform documentation.
Composition: Combining focused modules in a root and passing outputs explicitly instead of building a deeply nested module tree. Composition keeps ownership, dependencies, and replacement effects easier to review.

Before you copy

Values used in this guide

{{stateBucket}}

Pre-existing globally unique S3 bucket dedicated to Terraform state, with versioning, encryption, audit logging, public-access blocking, and protected lifecycle controls.

Example: acme-prod-terraform-state-eu-west-1

{{stateKey}}

Stable, unique object path for this root module. Changing it selects different state and can make every configured object appear absent.

Example: platform/network/prod/terraform.tfstate

{{awsRegion}}

Reviewed region containing the backend bucket and used explicitly by the S3 backend.

Example: eu-west-1

{{kmsKeyArn}}

Recoverable customer-managed KMS key ARN used for state encryption, with a key policy that covers approved backend roles and recovery administration.

Example: arn:aws:kms:eu-west-1:111122223333:key/12345678-1234-1234-1234-123456789012

{{moduleSource}}

Registry address or immutable VCS source for the reusable module. Prefer a registry so callers can use first-class semantic version constraints.

Example: app.terraform.io/acme/platform-service/aws

{{moduleVersion}}

Reviewed compatible module release constraint; production change evidence records the exact selected release.

Example: ~> 3.4.0

{{environment}}

Stable environment identity passed to naming and policy logic without embedding an account-specific provider configuration in the child.

Example: prod

{{serviceName}}

Validated workload name used by the example module contract and operational ownership tags.

Example: payments-api

Security and production boundaries

Terraform state and saved plan files are sensitive operational records. They can contain secrets, identifiers, network topology, policy documents, provider-returned attributes, and values hidden from normal CLI display. Encrypt them, restrict read as well as write access, keep them out of source control, and avoid attaching them to ordinary tickets.
Use short-lived workload identity or identity federation for provider and backend access. Do not put access keys, session tokens, KMS material, or backend credentials in `.tf` files, `-backend-config=KEY=value` shell history, generated plans, screenshots, or example output.
A `sensitive = true` value is redacted in selected interfaces but can still exist in state and plan files. Prefer ephemeral values and provider write-only arguments where Terraform 1.15 and the provider support them, then verify the provider schema instead of assuming a password was omitted.
Keep state administration separate from ordinary infrastructure administration. The ability to overwrite state, restore older versions, delete lock files, or force-unlock can redirect a future apply even when the principal cannot directly modify every managed object.
Never publish `TF_LOG=TRACE` output without security review. Provider requests, responses, environment details, and values can appear in debug logs; enable logging only for a bounded diagnostic and store the result as a secret-bearing incident artifact.
The backend identity needs access to both the state object and the native `.tflock` object. Grant those paths explicitly, restrict listing to the necessary prefix, and avoid broad bucket administration in ordinary plan or apply roles.
KMS key deletion or an inaccessible encryption context can make every retained state generation useless. Separate key administration, enable recovery controls, monitor disable and deletion events, and prove that the designated recovery role can decrypt an isolated snapshot.
Module consumers execute provider operations defined by module code. Review the complete release diff, provider requirements, data sources, provisioners, external programs, and downloaded sources before trusting a new version; a friendly output interface is not a sandbox.

Stop before continuing if

Stop if the backend bucket or KMS key would be managed by the same root whose state it stores, unless a separately reviewed bootstrap and destroy-protection design proves the dependency cannot erase recovery data.
Stop if a plan shows an empty state, mass creation, replacement, deletion, a different workspace, or an unexplained state lineage or serial.
Stop if any runner can bypass state locking, uses `-lock=false`, relies on a floating module branch, or stores backend credentials in configuration or command history.
Stop if the module contains provider credentials, root provider blocks, environment account literals, secrets as ordinary outputs, or a destructive provisioner that callers cannot audit.
Stop if state versioning, encryption, retained snapshots, audit logs, KMS recovery, and an isolated restore rehearsal are not all available.
Stop if a module upgrade changes resource addresses without compatible moved blocks or proposes replacement without an approved data and availability migration.
Stop if a saved plan is copied into an unprotected artifact store, outlives its approval window, or no longer matches the reviewed commit, variables, provider set, and state.
Stop after any denied-client or second-runner test demonstrates broader access than intended. Correct identity and path policy before production apply.

decision

Define the module, ownership, and state boundaries

read-only

Inventory the capability, consumers, remote objects, provider aliases, environment differences, existing addresses, secrets, failure domains, and operators. Decide whether the proposed abstraction is meaningful and draw a separate root-module and state boundary for every independently operated environment.

Why this step matters

A module boundary becomes a long-lived contract and a state boundary becomes a blast-radius decision. Extracting code before identifying ownership often produces a thin wrapper that hides provider behavior while leaving every consumer coupled to one release team. Likewise, placing unrelated services in one state makes a harmless module upgrade acquire the same lock and credentials as critical infrastructure. The inventory establishes which behavior belongs in reusable code, which choices remain visible in each root, and whether existing resource addresses need a moved-block migration.

What to understand

List all current instance addresses and their responsible team before moving code. Terraform associates remote objects with addresses, not with filenames, so copying a resource into a module changes its address unless an explicit `moved` block records the transition. Record provider aliases and every immutable attribute that could force replacement.

Define the module around an operational capability such as a compliant service logging target, not around a single `aws_s3_bucket` resource. Write the intended inputs, outputs, invariants, exclusions, upgrade promise, and observability contract before implementation. Keep account, region, credentials, backend, and approval policy in the environment root.

Choose state boundaries by ownership, lifecycle, privilege, and recovery objective. A state key should be small enough that its operators and change cadence are coherent, but not fragmented so aggressively that hundreds of roots exchange full remote state or create circular dependencies.

Mark every secret-bearing or topology-bearing attribute. Decide whether the provider offers an ephemeral or write-only argument and whether a narrow external publication mechanism can replace broad downstream access to state outputs.

System changes

No infrastructure or state is changed; the command reads the current Terraform version, provider graph, workspace, and state addresses for the design record.

Syntax explained

terraform providers: Shows provider requirements by module so embedded provider ownership and alias needs are visible.
terraform state list: Lists current instance addresses that must remain stable or be migrated with reviewed moved blocks.

Command

terraform version && terraform providers && terraform workspace show && terraform state list

Example output / evidence

Terraform v1.15.5
on linux_amd64
Providers required by configuration:
.
└── provider[registry.terraform.io/hashicorp/aws] ~> 6.0
prod
aws_kms_alias.state
aws_s3_bucket.service_logs

Checkpoint: The contract and blast radius are explicit

terraform state list | sort && terraform providers

Continue whenEvery managed address, provider alias, consumer, owner, state key, sensitive attribute, migration, and rollback responsibility is present in the reviewed design.

Stop whenOwnership overlaps, the proposed module is only a renamed resource, existing addresses are unknown, or one state key would cross unrelated privilege and recovery boundaries.

If this step fails

Initialization asks to migrate state when only a backend setting was expected to change.

Likely causeTerraform detected a different backend identity, key, workspace prefix, or partial configuration than the checkout previously initialized. Accepting the prompt without understanding the destination can split or overwrite state.

Safe checks

terraform version
git diff -- '*.tf' '*.hcl'
Inspect the non-secret backend configuration supplied by the approved runner.
List versioned state objects in the intended backend through the provider console.

ResolutionStop, compare the old and new backend coordinates, pull a protected backup from the known-good backend, and choose `-reconfigure` only for the same state location or `-migrate-state` only for a deliberate reviewed migration. Do not guess at an interactive prompt in production.

A child module works in one account but fails in another.

Likely causeThe module embedded a provider configuration, assumed a region, account ID, naming convention, default tags, or availability-zone count instead of receiving those decisions through the caller.

Safe checks

terraform providers
terraform validate
Review provider blocks inside the child module.
Compare documented inputs with every environment-specific literal.

ResolutionMove provider configuration to the root module, declare only `required_providers` in the child, expose intentional differences as typed and validated inputs, and pass provider aliases explicitly. Re-test both environments before publishing a new module version.

Security notes

Do not paste state content into the design document. Record addresses and classifications, while keeping raw state in the protected backend.

Alternatives

Leave a small configuration in its root and extract shared policy tests or documentation when a module would not create a clearer capability.

Stop conditions

Do not write module code until every existing object has exactly one intended destination address and rollback owner.

config

Create the standard module structure and version contract

read-only

Create `main.tf`, `variables.tf`, `outputs.tf`, `versions.tf`, a README, examples, and tests. Require Terraform 1.15.x for the reviewed release, declare the provider source and minimum compatible provider line, and keep provider configuration outside the child module.

Why this step matters

The standard structure makes the module understandable to humans, registries, documentation tools, and tests. A child module declares which provider it needs, but the root supplies the actual provider configuration and credentials. The child should use the least restrictive provider constraint compatible with its code so callers can select one version across their graph; the root then applies an upper bound and commits the resolved provider checksums. Requiring the reviewed Terraform minor prevents untested CLI semantics from entering production.

What to understand

Keep the module tree shallow. Nested modules without a public README can be internal implementation details, but a collection of deeply nested wrappers obscures ownership and makes upgrades harder to reason about. Compose focused capabilities in the root when possible.

Document purpose, non-goals, inputs, outputs, examples, security assumptions, upgrade notes, and destructive behavior. Generated input tables help, but they cannot replace an explanation of what trust boundary the module creates and how an operator verifies it.

Do not include a backend block, provider credentials, provider region, default account, or mutable organization policy. Provider aliases are passed by the caller through the module `providers` map when the capability genuinely spans configurations.

Use semantic releases. Compatible versions preserve the contract and state upgrade path; breaking input, output, provider, behavior, or moved-block changes require a major release and a migration document.

System changes

Adds the reusable module source tree to version control; no provider API or state operation occurs.

Syntax explained

required_version = "~> 1.15.0": Accepts compatible Terraform 1.15 patch releases while excluding a future unreviewed minor.
source = "hashicorp/aws": Uses the provider's global source address so the caller and every child agree on provider identity.
version = ">= 6.0": States the child module's minimum provider capability without forcing every caller to one patch; the root owns the upper bound.

File modules/platform-service/versions.tf

Configuration

terraform {
  required_version = "~> 1.15.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 6.0"
    }
  }
}

Example output / evidence

Success! The configuration is valid.

modules/platform-service/
├── README.md
├── main.tf
├── outputs.tf
├── variables.tf
├── versions.tf
├── examples/complete/main.tf
└── tests/module.tftest.hcl

Checkpoint: The child is portable and tool-readable

terraform -chdir=modules/platform-service fmt -check -recursive && terraform -chdir=modules/platform-service validate

Continue whenFormatting and validation pass, every variable and output is documented, and no backend or provider configuration exists inside the child.

Stop whenThe module embeds credentials, region, account identity, a backend, a provider block, or an undocumented resource replacement.

If this step fails

A child module works in one account but fails in another.

Safe checks

terraform providers
terraform validate
Review provider blocks inside the child module.
Compare documented inputs with every environment-specific literal.

Security notes

Review external data sources, provisioners, local-exec, remote-exec, and downloaded artifacts as code execution or data-exfiltration boundaries; omit them unless strictly necessary.

Alternatives

Use an internal mono-repository subdirectory during early development, then publish an immutable registry release when multiple roots need independent upgrades.

Stop conditions

Do not publish a release without a license or internal usage terms, a security owner, and a supported upgrade path.

config

Implement typed inputs, validation, resources, and narrow outputs

read-only

Define a small contract that accepts a service identity, environment, and tags, rejects malformed values before provider calls, creates the intended capability, and returns only stable non-secret identifiers needed by callers.

Why this step matters

Types and validations turn assumptions into an executable contract before a provider request can create a costly or non-compliant object. Defaults should be safe and unsurprising; required choices should remain explicit. Outputs are an API, so exporting an entire resource object or secret increases coupling and exposes implementation details. A narrow output such as an ARN or service endpoint gives composition without granting downstream configurations access to unrelated state data.

What to understand

Use object types with `optional` attributes for structured inputs only when the structure remains comprehensible. A large untyped map shifts validation into provider errors and makes compatibility impossible to communicate.

Validate names, enumerations, mutually dependent settings, CIDRs, and policy boundaries that Terraform can know during planning. Provider-side facts still require preconditions, postconditions, checks, or post-apply verification.

Keep computed naming predictable and expose the final identifier. Do not let callers pass raw policy JSON or arbitrary resource arguments merely to make the module universal; that usually dissolves the abstraction and security review.

Mark display-sensitive outputs, but remember the value still normally resides in state. Do not output credentials, private keys, recovery codes, or complete provider objects. Use a secret system and a write-only or ephemeral flow where supported.

System changes

Adds and validates the module interface; no remote object changes occur until a caller plans and applies the module.

Syntax explained

type: Defines the structural contract and lets Terraform reject incompatible caller values early.
validation: Expresses a domain rule with a specific operator-facing error before provider API calls.
map(string): Accepts additional non-sensitive tags while retaining a predictable key and value type.

File modules/platform-service/variables.tf

Configuration

variable "service_name" {
  type        = string
  description = "Stable lowercase service identifier."

  validation {
    condition     = can(regex("^[a-z][a-z0-9-]{2,30}$", var.service_name))
    error_message = "service_name must be 3-31 lowercase letters, digits, or hyphens."
  }
}

variable "environment" {
  type        = string
  description = "Deployment environment."
  validation {
    condition     = contains(["dev", "stage", "prod"], var.environment)
    error_message = "environment must be dev, stage, or prod."
  }
}

variable "tags" {
  type        = map(string)
  description = "Additional non-sensitive ownership tags."
  default     = {}
}

Example output / evidence

Error: Invalid value for variable

service_name must be 3-31 lowercase letters, digits, or hyphens.

This was checked by the validation rule at variables.tf:5,3-13.

Checkpoint: The module fails closed on invalid input

terraform -chdir=modules/platform-service test -verbose

Continue whenValid fixtures plan successfully, invalid names and environments fail with the intended messages, and outputs contain no secret or complete resource object.

Stop whenInvalid input reaches the provider, arbitrary policy bypasses the contract, or a sensitive value appears in output or test artifacts.

If this step fails

A module output leaks a value or creates broad state coupling.

Likely causeThe module exported entire resource objects or sensitive values instead of a narrow stable interface, and downstream configurations read the complete remote state rather than an intentionally published value.

Safe checks

terraform output
Review output blocks and their consumers.
Review `terraform_remote_state` data sources.
Inspect state access policy for downstream identities.

ResolutionPublish only the minimum non-secret identifiers through narrow outputs or a dedicated configuration store. Mark display-sensitive values, avoid granting downstream readers access to the full state merely to retrieve one output, and rotate anything already exposed.

Security notes

Treat tags and descriptions as potentially public metadata. Reject secrets and personal data from free-form tag inputs through policy and review.

Alternatives

Split capabilities into two composable modules when one input object develops unrelated modes and mutually exclusive security behavior.

Stop conditions

Do not widen the interface with an `extra_arguments` escape hatch that bypasses validation and review.

verification

Test behavior, failure cases, and upgrade compatibility

read-only

Run formatting, validation, and `terraform test` from a clean module checkout. Cover defaults, invalid values, conditional branches, output invariants, provider aliases, and a representative upgrade from the previous supported module release.

Why this step matters

Static validation proves only that syntax and types are coherent. Module tests let maintainers assert planned values, expected failures, and provider-backed behavior where an isolated test account is available. Upgrade testing is equally important: a clean installation can pass while existing state proposes destructive replacement because addresses or immutable arguments changed. The release gate therefore covers both a new fixture and a representative prior state path.

What to understand

Use mock providers for pure contract tests and a dedicated disposable account for a smaller number of integration tests. Never point module tests at a production account, shared state key, or credential with unrelated privileges.

Assert security defaults and negative cases: public exposure disabled, encryption required, deletion protection preserved, invalid CIDRs rejected, and secret outputs absent. A test suite that covers only the happy path encourages unsafe flexibility.

Test the module with caller-owned default and aliased provider configurations. A child that silently relies on a default provider may fail when a root operates across regions or accounts.

For upgrades, initialize the previous release, create representative state in an isolated environment, switch to the candidate, and require a plan with no unexplained destroy or replacement. Retain moved blocks needed by older supported callers.

System changes

Downloads pinned provider packages into the ignored `.terraform` directory and may create then destroy disposable integration fixtures; it does not use production state.

Syntax explained

-backend=false: Initializes a reusable child without attempting to configure an environment backend.
terraform test -verbose: Runs module-native test files and displays plans or state needed to diagnose assertions; output must be handled as potentially sensitive.

Command

terraform -chdir=modules/platform-service fmt -check -recursive && terraform -chdir=modules/platform-service init -backend=false -input=false && terraform -chdir=modules/platform-service validate && terraform -chdir=modules/platform-service test -verbose

Example output / evidence

Success! The configuration is valid.
tests/module.tftest.hcl... in progress
  run "valid_prod_contract"... pass
  run "reject_invalid_name"... pass
  run "outputs_are_narrow"... pass
tests/module.tftest.hcl... tearing down
Success! 3 passed, 0 failed.

Checkpoint: The candidate behaves safely for new and existing callers

terraform -chdir=modules/platform-service test

Continue whenAll positive, negative, security, provider-alias, cleanup, and prior-release upgrade cases pass in an isolated environment.

Stop whenA test leaves infrastructure behind, needs production credentials, exposes a secret, or an upgrade proposes unexplained replacement.

If this step fails

A module test passes but the production plan fails validation.

Likely causeTests did not cover the production provider version, caller aliases, variable edge cases, policy controls, or actual backend and state shape.

Safe checks

terraform version
terraform providers
terraform validate
terraform plan -out=review.tfplan

ResolutionReproduce the pinned toolchain and caller contract in a non-production test root, add positive and negative variable tests, exercise upgrades from the prior module version, and require a production read-only plan as the final acceptance gate.

Security notes

Test state and verbose output are sensitive too. Use isolated credentials, automatic expiry, protected logs, and a cleanup alert rather than relying only on successful teardown.

Alternatives

Use plan fixtures plus policy tests when a provider-backed test would be too costly, while retaining at least one scheduled integration path for real API behavior.

Stop conditions

Do not publish when the test runner cannot prove cleanup or when a failed test can persist publicly reachable infrastructure.

warning

Bootstrap remote state outside the application root

caution

Create the dedicated S3 bucket, versioning, encryption, KMS recovery, public-access block, retention, audit logging, and backend roles through a separate protected bootstrap root or provider-native process. Record the bucket, key policy, recovery owner, and immutable lifecycle.

Why this step matters

An application root cannot safely depend on a backend bucket that it must access before Terraform initializes, and a routine destroy must never remove its recovery record. A separate bootstrap lifecycle breaks that dependency and gives state administration tighter privileges and retention. Versioning protects against accidental overwrite or deletion only when versions remain decryptable and operators can identify and restore the correct lineage. The bootstrap plan is therefore reviewed like critical security infrastructure.

What to understand

Enable S3 versioning before the first production state write. Protect the bucket and KMS key from ordinary application destroy, monitor policy and lifecycle changes, and prohibit public access at both account and bucket layers where organizational policy supports it.

Use server-side encryption with a recoverable KMS key and explicit key policy. Document who can decrypt current and historical versions, who can administer the key, and how deletion scheduling is detected and canceled.

Create separate plan, apply, and recovery roles. Limit S3 access to the exact bucket and prefixes, include the `.tflock` path required by native locking, and reserve object-version restore or force-unlock for an incident role.

Do not store backend credentials in Terraform code. The runner acquires short-lived identity before init; static backend coordinates may be committed, while sensitive partial configuration is supplied through protected channels and kept out of shell history.

System changes

The reviewed bootstrap apply creates or hardens the durable state bucket, KMS controls, retention, audit configuration, and backend identities outside application state.

Syntax explained

-out=state-bootstrap.tfplan: Saves the exact proposed bootstrap changes for review and later controlled apply.
-lock-timeout=5m: Waits a bounded period for the bootstrap state lock rather than disabling serialization.

Command

terraform -chdir=bootstrap/state plan -out=state-bootstrap.tfplan -lock-timeout=5m && terraform -chdir=bootstrap/state show state-bootstrap.tfplan

Example output / evidence

Terraform will perform the following actions:
  # aws_s3_bucket.terraform_state will be created
  # aws_s3_bucket_versioning.terraform_state will be created
  # aws_s3_bucket_server_side_encryption_configuration.terraform_state will be created
Plan: 6 to add, 0 to change, 0 to destroy.

Checkpoint: State storage survives application failure and operator error

terraform -chdir=bootstrap/state plan -detailed-exitcode -lock-timeout=5m

Continue whenAfter approved bootstrap apply, the plan exits 0 and evidence shows versioning, encryption, public-access blocking, audit logs, retention, least privilege, KMS recovery, and deletion protection.

Stop whenThe application root owns its backend, versioning is suspended, KMS recovery is unassigned, or an ordinary apply role can delete the bucket and historical versions.

If this step fails

Terraform cannot decrypt or read the remote state object.

Likely causeThe runtime lost KMS permissions, the key was disabled or deleted, the object uses a different encryption context, or backend credentials select the wrong account.

Safe checks

terraform init -reconfigure
Verify the caller identity through the cloud provider's identity command.
Inspect KMS key status and backend audit logs without downloading state.
Confirm the expected bucket and key from the change record.

ResolutionRestore the approved identity or recover the encryption key according to provider policy. Do not create an empty replacement state. If the key is irrecoverable, escalate to state and infrastructure recovery using a versioned snapshot and a complete remote-object inventory.

Security notes

Protect the saved bootstrap plan as sensitive; IAM policy documents, ARNs, and provider-returned values reveal the state security architecture.

Alternatives

Use an organization-provisioned backend service or HCP Terraform workspace when a central platform team owns equivalent durability and access controls.

Stop conditions

Do not migrate production state until a prior version can be restored and decrypted into an isolated recovery key.

config

Configure the S3 backend with native lockfiles

caution

Add the static backend type and non-secret coordinates to the root module, enable `use_lockfile = true`, and use a stable key unique to the root. Authenticate through the runner environment, then initialize deliberately.

Why this step matters

Backend initialization happens before normal Terraform variables and resources, so coordinates must be static or supplied as partial backend configuration. Current Terraform S3 documentation supports native lockfiles and marks DynamoDB locking deprecated. Enabling `use_lockfile` gives every supported writer the same serialization mechanism without operating a second database. The key is an identity boundary: a typo does not merely choose another file, it can make Terraform believe production objects do not exist.

What to understand

Use `terraform init -reconfigure` only when the same backend location should be re-read without moving state. Use `-migrate-state` only for an explicitly reviewed move after pulling a protected snapshot and inventorying all workspaces.

Avoid credentials and secret encryption material in backend arguments or CLI key-value flags. Terraform records merged backend configuration under `.terraform`, so the directory stays ignored and runner disks are ephemeral or protected.

The backend identity needs read and write on the state object plus the documented native lockfile actions on `<key>.tflock`. Keep state-object deletion more restrictive; Terraform does not need to delete the state object during normal operation.

Do not configure `dynamodb_table` for a new Terraform 1.15 deployment. A temporary dual mechanism may be part of a separately tested migration from older clients, but all production writers must converge on one supported version and native locking.

System changes

Writes local backend metadata under ignored `.terraform` and connects this root to the existing remote state key; a migration flag could also write remote state and therefore requires separate approval.

Syntax explained

use_lockfile = true: Enables Terraform's native S3 state lock object for serialized operations.
encrypt = true: Requests server-side encryption for state writes; bucket policy and KMS controls must enforce the intended protection independently.
kms_key_id: Selects the reviewed customer-managed KMS key whose access and recovery lifecycle are operated separately.

File environments/prod/backend.tf

Configuration

Fill variables0/4 ready

Values stay on this page and are never sent or saved.

State Bucket{{stateBucket}}Pre-existing globally unique S3 bucket dedicated to Terraform state, with versioning, encryption, audit logging, public-access blocking, and protected lifecycle controls.State Key{{stateKey}}Stable, unique object path for this root module. Changing it selects different state and can make every configured object appear absent.Aws Region{{awsRegion}}Reviewed region containing the backend bucket and used explicitly by the S3 backend.Kms Key Arn{{kmsKeyArn}}Recoverable customer-managed KMS key ARN used for state encryption, with a key policy that covers approved backend roles and recovery administration.

terraform {
  backend "s3" {
    bucket       = "{{stateBucket}}"
    key          = "{{stateKey}}"
    region       = "{{awsRegion}}"
    use_lockfile = true
    encrypt      = true
    kms_key_id   = "{{kmsKeyArn}}"
  }
}

Example output / evidence

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically use this backend unless the backend configuration changes.

Checkpoint: Initialization selects the one intended protected state

terraform -chdir=environments/prod init -reconfigure -input=false && terraform -chdir=environments/prod workspace show && terraform -chdir=environments/prod state list

Continue whenInitialization succeeds as the approved identity, the expected workspace and known addresses appear, and the backend audit log shows access only to the intended state and lock paths.

Stop whenTerraform proposes migration unexpectedly, state is empty, a different workspace appears, KMS access fails, or lockfile access is denied.

If this step fails

Initialization asks to migrate state when only a backend setting was expected to change.

Safe checks

terraform version
git diff -- '*.tf' '*.hcl'
Inspect the non-secret backend configuration supplied by the approved runner.
List versioned state objects in the intended backend through the provider console.

A plan unexpectedly proposes many creates after backend initialization.

Likely causeThe configuration selected an empty state key, wrong workspace, wrong account, or wrong region. Terraform therefore sees configuration but no bindings to existing objects.

Safe checks

terraform workspace show
terraform state list
terraform providers
Compare the backend key and state serial with the protected inventory.

ResolutionDo not apply. Restore the intended backend coordinates, reinitialize, and confirm the expected state addresses before planning again. If the original state is unavailable, treat recovery as an incident rather than attempting to recreate production from an empty state.

S3 locking returns AccessDenied for the `.tflock` object.

Likely causeThe backend identity can read and write the state object but its least-privilege policy does not include the separate lock-file key operations required by `use_lockfile = true`.

Safe checks

Confirm the exact state key and `<key>.tflock` path.
Inspect denied S3 actions in the provider audit log.
Compare the IAM resource paths with the official S3 backend permissions.
Confirm that no bucket policy condition selects a different prefix.

ResolutionUpdate the backend policy narrowly for the documented lock object permissions, retain stronger deletion restrictions on the state object, and retry initialization and a no-change plan. Do not fall back to `-lock=false` or deprecated DynamoDB locking as a shortcut.

Security notes

The `encrypt` argument is not a substitute for a bucket policy that rejects unencrypted or wrongly keyed writes and for controls that prevent KMS deletion.

Alternatives

Use a minimal empty backend block plus a protected non-secret backend configuration file when roots share a standardized runner template.

Stop conditions

Never answer an unexpected migration prompt interactively in production; stop and reconcile source and destination first.

config

Pin the module in a production root and pass provider ownership

read-only

Configure providers in the root, call the reviewed registry release with an explicit compatible version, pass environment values, and keep the caller contract small. Commit the provider dependency lock file generated by initialization.

Why this step matters

The root is where environment decisions remain reviewable: provider region and aliases, module release, backend, tags, and values. A registry version constraint avoids silently tracking a default branch while allowing reviewed compatible patches. The provider lock file records exact provider packages and checksums, but does not pin modules; both controls are needed. Passing provider ownership from the root also lets two callers use the same child in different accounts without modifying module code.

What to understand

Use a registry module version constraint with a bounded compatible line and record the exact selected version from init in the change. For VCS sources, pin an immutable commit or reviewed tag because the `version` argument applies only to registry modules.

Commit `.terraform.lock.hcl` after reviewing source, version, signatures, and checksums. Exclude `.terraform`, plan files, state, crash logs, override files carrying local experiments, and credentials.

Set provider configuration only in the root. When the module requires aliases, declare `configuration_aliases` in the child provider requirements and pass the root configurations through `providers = { ... }` explicitly.

Do not retrieve one identifier by giving this root broad access to another root's complete state. Publish narrow non-sensitive values through an approved configuration interface or tightly controlled output mechanism.

System changes

Initialization downloads the pinned child module and provider packages locally and updates `.terraform.lock.hcl`; it does not change remote infrastructure.

Syntax explained

version = "{{moduleVersion}}": Constrains a registry module to the reviewed semantic release line; the actual selection is recorded in init evidence.
version = "~> 6.0": Lets the root accept provider 6.x releases while excluding a future major until compatibility is reviewed.
provider "aws": Keeps authentication, region, aliases, and environment ownership in the root instead of the reusable child.

File environments/prod/main.tf

Configuration

Fill variables0/5 ready

Values stay on this page and are never sent or saved.

Aws Region{{awsRegion}}Reviewed region containing the backend bucket and used explicitly by the S3 backend.Module Source{{moduleSource}}Registry address or immutable VCS source for the reusable module. Prefer a registry so callers can use first-class semantic version constraints.Module Version{{moduleVersion}}Reviewed compatible module release constraint; production change evidence records the exact selected release.Service Name{{serviceName}}Validated workload name used by the example module contract and operational ownership tags.Environment{{environment}}Stable environment identity passed to naming and policy logic without embedding an account-specific provider configuration in the child.

terraform {
  required_version = "~> 1.15.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

provider "aws" {
  region = "{{awsRegion}}"
}

module "service" {
  source  = "{{moduleSource}}"
  version = "{{moduleVersion}}"

  service_name = "{{serviceName}}"
  environment  = "{{environment}}"
  tags = {
    Owner       = "platform"
    ManagedBy   = "terraform"
    Environment = "{{environment}}"
  }
}

Example output / evidence

Initializing modules...
Downloading app.terraform.io/acme/platform-service/aws 3.4.2 for service...
- service in .terraform/modules/service
Initializing provider plugins...
- Installed hashicorp/aws v6.55.0 (signed by HashiCorp)

Checkpoint: Dependencies are immutable enough for review

terraform -chdir=environments/prod init -input=false && terraform -chdir=environments/prod providers lock -platform=linux_amd64 -platform=darwin_arm64 && git diff -- .terraform.lock.hcl

Continue whenThe intended module release and provider packages resolve, signatures and checksums are recorded, and no floating branch or unreviewed major version appears.

Stop whenThe module resolves from a default branch, init upgrades unexpectedly, provider ownership remains inside the child, or the lock-file diff is unexplained.

If this step fails

Upgrading a module version proposes replacement of existing resources.

Likely causeA resource address changed without a compatible `moved` block, an immutable provider argument changed, a `for_each` key changed, or the version constraint selected a release outside the reviewed range.

Safe checks

terraform plan -out=module-upgrade.tfplan
terraform show module-upgrade.tfplan
git diff between the pinned module releases
Review module release notes and moved blocks.

ResolutionDo not apply. Restore the previous pin, identify each replacement cause, add or consume a compatible `moved` block for address-only refactors, and stage immutable changes through a migration design. Release a corrected semantic version and test it against a copy of representative state.

Terraform installs an unexpected module release.

Likely causeThe caller omitted a registry `version` constraint, used a floating VCS default branch, changed the dependency lock file expecting it to pin modules, or ran init with upgrade behavior.

Safe checks

terraform get
terraform init
Review every module source and version argument.
Inspect the module manifest under `.terraform/modules` without committing it.

ResolutionUse a registry source with an explicit compatible version constraint or a VCS source pinned to an immutable commit or reviewed tag. Record the selected release in the change, review its diff, and rerun validation and plan from a clean checkout.

Security notes

Module code executes with the root provider's privileges. Review the complete selected release, not only the module block shown in the caller.

Alternatives

Pin a VCS module to a full commit SHA when no registry exists, and add an automated process that reviews and updates the immutable reference.

Stop conditions

Do not run a production plan after `terraform init -upgrade` until every provider and module change is independently reviewed.

verification

Create a locked, reproducible, saved plan

read-only

From the approved commit and pinned toolchain, format, validate, test, initialize, and save a normal refresh-backed plan under a bounded state lock. Render the plan for human and policy review, checking every create, update, replacement, deletion, output, and unknown.

Why this step matters

A normal plan refreshes remote objects, compares configuration, state, and provider observations, and proposes convergence. Saving it binds review to the artifact later applied; rerunning an unsaved plan at approval time could produce a different action. Locking prevents another writer from changing state during calculation. Review must look beyond the summary: provider replacements, unknown values, policy expansion, module address changes, and secret-bearing outputs can be dangerous even when the counts look small.

What to understand

Run from a clean checkout at the exact approved commit with the intended variable source and short-lived identity. Record Terraform, provider, and module versions. Do not use `-refresh=false` for the final plan because it can hide remote changes.

Treat the binary plan and any JSON or text rendering as sensitive. Store it in an access-controlled, short-lived artifact channel tied to the commit and approval; never commit it or post it to a public review.

Reject all unexplained delete-and-create or `-/+` replacements. Check resource addresses, immutable attributes, IAM and network policy, encryption, retention, public exposure, and any value known only after apply.

Use policy checks as an additional gate, not as a substitute for an operator who understands provider semantics and the service rollback. Confirm the state lock owner and the absence of overlapping drift or maintenance jobs.

System changes

Reads provider APIs and remote state under a lock and writes a sensitive local saved-plan artifact; no remote object is changed.

Syntax explained

-out=review.tfplan: Saves the exact proposal so apply can execute the reviewed artifact rather than recalculating.
-lock-timeout=5m: Retries lock acquisition for a bounded window and fails closed if another writer remains active.
-input=false: Prevents hidden interactive values in automation; all inputs must come from approved sources.

Command

terraform -chdir=environments/prod fmt -check -recursive && terraform -chdir=environments/prod validate && terraform -chdir=environments/prod plan -out=review.tfplan -lock-timeout=5m -input=false && terraform -chdir=environments/prod show review.tfplan

Example output / evidence

Terraform will perform the following actions:
  # module.service.aws_s3_bucket.logs will be created
  + resource "aws_s3_bucket" "logs" {
      + bucket = "payments-api-prod-logs"
    }
Plan: 3 to add, 0 to change, 0 to destroy.

Checkpoint: The saved plan contains only intended non-destructive actions

terraform -chdir=environments/prod show -json review.tfplan | jq -r '.resource_changes[] | [.address, (.change.actions|join(","))] | @tsv'

Continue whenEvery address and action maps to the design, no unexplained delete or replacement exists, and reviewers approve the exact commit, inputs, state, identities, and artifact digest.

Stop whenAny action is unexplained, the plan includes replacement or deletion without migration approval, state changed after planning, or the artifact may contain an exposed secret.

If this step fails

Terraform reports that the state lock is already held.

Likely causeAnother approved operation is active, an earlier process was interrupted, or an operator is pointing at the same state key from an unexpected checkout. A lock is evidence of serialization, not an inconvenience to bypass.

Safe checks

terraform workspace show
terraform version
ps -ef | grep '[t]erraform'
Review the remote runner and change-system activity for the exact backend key.

ResolutionIdentify the lock owner and wait for the active operation. If the owner confirms that no process can still write, follow the backend-specific recovery procedure and use `terraform force-unlock` only with the exact lock ID, a protected state backup, and a second reviewer. Never set `-lock=false` to make progress.

A CI plan waits indefinitely or times out acquiring a lock.

Likely causeRuns are not serialized at the pipeline layer, the lock timeout is shorter than normal applies, or an abandoned operation still owns the backend lock.

Safe checks

Inspect queued and running jobs for the same state key.
Compare lock metadata with the runner identity and start time.
Review recent apply duration and cancellation events.
Confirm that no scheduled drift job overlaps the change window.

ResolutionSerialize operations per backend key, use a bounded `-lock-timeout` that matches operating expectations, cancel duplicate queued work, and investigate stale locks explicitly. A timeout should fail closed and alert; it must not retry with locking disabled.

Security notes

A person who can read a plan may learn credentials or sensitive infrastructure details even when terminal output redacts values.

Alternatives

Use HCP Terraform's run queue and plan review while preserving equivalent version, identity, policy, and state-boundary evidence.

Stop conditions

Never apply by typing `terraform apply` without the saved artifact after reviewers approved a specific saved plan.

verification

Prove concurrent writers are serialized

read-only

In an isolated test state using the production backend policy pattern, hold one Terraform operation and start a second with a short lock timeout. Confirm that the second waits and then fails with lock metadata, and that neither process proceeds by disabling locking.

Why this step matters

A backend configuration line is not operational evidence. The native `.tflock` object requires correct permissions, a client that honors it, and a pipeline that does not retry with unsafe flags. A controlled contention test proves the behavior without risking the production key. It also teaches operators what healthy lock metadata and timeout failure look like, reducing the chance that a normal lock is mistaken for corruption during an incident.

What to understand

Use an isolated state key and harmless fixture. Start the first operation through a controlled harness that holds its lock long enough for observation, rather than pausing an actual production apply.

Verify the second role has the normal state and lock permissions. An AccessDenied error tests IAM failure, not serialization. The expected result is lock acquisition wait followed by an owned-lock error and no plan artifact.

Inspect the `.tflock` audit events and lock metadata, but do not delete the lock object manually. Let the first operation exit cleanly, then confirm the next plan acquires and releases the lock.

Configure CI concurrency by backend key as a second layer. Backend locking protects state; pipeline serialization also prevents duplicated cloud API work, confusing approvals, and a queue of stale plans.

System changes

Creates and releases a temporary native lock object for the isolated test key; no production infrastructure is changed.

Syntax explained

-lock-timeout=15s: Makes the negative test wait briefly and then fail closed while the legitimate first operation owns the lock.
-out=second.tfplan: Provides an assertion that no saved plan is produced when the second process cannot obtain the lock.

Command

terraform -chdir=environments/lock-test plan -lock-timeout=15s -out=second.tfplan

Example output / evidence

Acquiring state lock. This may take a few moments...

Error: Error acquiring the state lock

Lock Info:
  ID:        1a2b3c4d-1111-2222-3333-abcdef123456
  Path:      acme-prod-terraform-state-eu-west-1/platform/lock-test/terraform.tfstate
  Operation: OperationTypePlan

Checkpoint: A second writer cannot calculate or commit concurrently

test ! -e environments/lock-test/second.tfplan && terraform -chdir=environments/lock-test plan -lock-timeout=30s -detailed-exitcode

Continue whenThe contending plan produced no artifact; after the first owner exits, a new plan acquires the native lock and returns the expected empty or reviewed diff.

Stop whenBoth operations proceed, the second creates a plan while the lock is held, AccessDenied masks the test, or cleanup requires manual lock deletion.

If this step fails

Terraform reports that the state lock is already held.

Safe checks

terraform workspace show
terraform version
ps -ef | grep '[t]erraform'
Review the remote runner and change-system activity for the exact backend key.

A CI plan waits indefinitely or times out acquiring a lock.

Likely causeRuns are not serialized at the pipeline layer, the lock timeout is shorter than normal applies, or an abandoned operation still owns the backend lock.

Safe checks

Inspect queued and running jobs for the same state key.
Compare lock metadata with the runner identity and start time.
Review recent apply duration and cancellation events.
Confirm that no scheduled drift job overlaps the change window.

Security notes

Lock metadata reveals operator and state-path information. Keep it in the protected test record and avoid screenshots in public issue trackers.

Alternatives

Use a backend vendor's official concurrency integration test when direct lock holding is unavailable, but still verify pipeline behavior for the exact root.

Stop conditions

Do not certify the backend while any supported writer can set `-lock=false` or bypass the common pipeline.

command

Apply only the approved plan and verify the module contract

caution

Confirm the commit, artifact digest, approval, state key, identity, and maintenance window, then apply the exact saved plan. Monitor provider operations and verify module outputs plus the managed service's security, availability, and ownership signals.

Why this step matters

Applying a saved plan prevents Terraform from silently recalculating against new configuration, but the artifact is valid only while its context remains trustworthy. Provider APIs can still fail partway, leaving a new state snapshot and a partially converged service. Immediate verification therefore covers the narrow module output, remote service properties, state addresses, logs, and a new normal plan. The operator must distinguish retryable provider failure from a design flaw before reapplying anything.

What to understand

Before apply, verify the plan digest, source commit, selected state, approval expiry, caller identity, and absence of another change. If the saved plan is stale or state changed, discard it and restart review.

Watch the provider activity and backend audit trail. Do not cancel merely because a create takes longer than expected; check provider status and documented timeout behavior. If interrupted, inspect state and remote objects before planning again.

Validate the actual security properties rather than relying only on Terraform outputs: encryption, public access, retention, tags, policy, metrics, and service-specific health. Run an unauthorized negative test from an isolated identity.

Create a fresh normal plan after verification. Exit code 0 is the desired convergence evidence; a non-empty diff after immediate apply indicates provider normalization, missing configuration, drift, or a non-idempotent module.

System changes

Applies the exact approved module actions to provider resources and writes a new versioned remote state snapshot under the native lock.

Syntax explained

review.tfplan: Executes the previously reviewed binary plan instead of generating a new proposal at apply time.
-input=false: Prevents an operator from introducing an unreviewed value during the apply.
terraform output: Reads the module's deliberately narrow root outputs for post-change verification; output remains potentially sensitive.

Command

terraform -chdir=environments/prod apply -lock-timeout=5m -input=false review.tfplan && terraform -chdir=environments/prod output && terraform -chdir=environments/prod state list

Example output / evidence

module.service.aws_s3_bucket.logs: Creation complete after 2s [id=payments-api-prod-logs]
module.service.aws_s3_bucket_versioning.logs: Creation complete after 1s

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

service_log_bucket_arn = "arn:aws:s3:::payments-api-prod-logs"

Checkpoint: The release converges and satisfies its operational contract

terraform -chdir=environments/prod plan -detailed-exitcode -lock-timeout=5m

Continue whenThe command exits 0, state lists only intended addresses, the service passes positive and negative checks, and the backend contains a new encrypted recoverable version.

Stop whenApply is partial, a new plan is non-empty, public or unauthorized access succeeds, an expected control is absent, or state versioning and lock release cannot be confirmed.

If this step fails

Terraform reports that the state lock is already held.

Safe checks

terraform workspace show
terraform version
ps -ef | grep '[t]erraform'
Review the remote runner and change-system activity for the exact backend key.

Upgrading a module version proposes replacement of existing resources.

Safe checks

terraform plan -out=module-upgrade.tfplan
terraform show module-upgrade.tfplan
git diff between the pinned module releases
Review module release notes and moved blocks.

Security notes

Do not print complete state or JSON output as a verification shortcut. Query only required non-secret outputs and provider control status.

Alternatives

Use a canary root or lower-risk environment for a new major module release, then promote the same immutable version after its observation window.

Stop conditions

Do not retry a partial apply until the current state, remote objects, and provider error are reconciled in a new reviewed plan.

verification

Rehearse isolated state recovery and module rollback

read-only

Copy a retained encrypted state version into a dedicated recovery key, initialize an isolated checkout against that key, compare lineage, serial, addresses, providers, and a refresh-only plan, then delete the temporary recovery copy through the protected procedure.

Why this step matters

A versioned object is not a recovery capability until the team can select the correct generation, decrypt it, initialize without overwriting production, and interpret its relationship to current infrastructure. Restoring into an isolated key avoids replacing live state during a rehearsal. A refresh-only plan reveals how the snapshot differs from remote objects without proposing infrastructure convergence, while address and lineage comparison detects the wrong workspace or generation. Module rollback is separate: an older code release may not support current state or provider schemas.

What to understand

Record the current state version, lineage, serial, module release, provider selections, and object-version identifier before the exercise. Select a known prior version under dual control and copy it to a recovery-only prefix with equivalent encryption.

Point a dedicated recovery checkout at the isolated key. Never change the production backend block in place for a rehearsal. Use a read-only provider role when possible and prevent apply through policy.

List addresses and run refresh-only plan. Differences are evidence to analyze, not a prompt to apply. Confirm that a recovery operator can decrypt historical versions without gaining ordinary production apply rights.

For module rollback, review release notes and state compatibility. Prefer fixing forward or using a documented compatible prior version; never assume `git revert` can interpret provider-upgraded state and newly created resource schemas.

System changes

Creates a protected temporary copy of a historical state object under an isolated key and local backend metadata; the verification does not change production infrastructure.

Syntax explained

init -reconfigure: Reads the recovery checkout's isolated backend coordinates without migrating the live state.
plan -refresh-only: Compares the restored snapshot with remote observations without proposing changes to remote objects.

Command

terraform -chdir=recovery/prod init -reconfigure -input=false && terraform -chdir=recovery/prod state list && terraform -chdir=recovery/prod plan -refresh-only -lock-timeout=5m

Example output / evidence

Initializing the backend...
Successfully configured the backend "s3"!
module.service.aws_s3_bucket.logs
module.service.aws_s3_bucket_versioning.logs

No changes. Your infrastructure still matches the configuration.

Checkpoint: A historical snapshot is usable without risking live state

terraform -chdir=recovery/prod state list && terraform -chdir=recovery/prod plan -refresh-only -detailed-exitcode -lock-timeout=5m

Continue whenThe designated recovery role decrypts the intended version, expected addresses and lineage are documented, differences are explainable, and no production state or object is written.

Stop whenThe version cannot be decrypted, recovery selects production state, address inventory is incomplete, or any role can apply from the rehearsal checkout.

If this step fails

Terraform cannot decrypt or read the remote state object.

Likely causeThe runtime lost KMS permissions, the key was disabled or deleted, the object uses a different encryption context, or backend credentials select the wrong account.

Safe checks

terraform init -reconfigure
Verify the caller identity through the cloud provider's identity command.
Inspect KMS key status and backend audit logs without downloading state.
Confirm the expected bucket and key from the change record.

A backend migration succeeds but older state is no longer recoverable.

Likely causeObject versioning was not enabled, retention was too short, or migration validation considered only the current object and not lineage, serial, workspaces, and recovery access.

Safe checks

terraform state pull
Inspect bucket versioning and retained object versions.
List every Terraform workspace before and after migration.
Compare state lineage and serial in protected copies.

ResolutionStop further writes, preserve the current and source snapshots, restore or enable versioning according to provider controls, and rehearse recovery into an isolated key. Accept the backend only after every workspace and its rollback snapshot is accounted for.

Security notes

The recovery copy contains the same secrets as production state. Apply equal access, retention, encryption, audit, and secure deletion controls.

Alternatives

Use a provider-supported isolated restore or HCP Terraform state-version recovery workflow if it preserves equivalent read-only rehearsal and audit guarantees.

Stop conditions

Never overwrite the live state object merely to prove that a backup is readable.

instruction

Operate releases, locks, state, and deprecations as one lifecycle

read-only

Document owners and alerts for module releases, provider upgrades, state versions, KMS health, lock age, force-unlock, drift, failed applies, recovery tests, and caller deprecations. Review the guide and controls every 90 days.

Why this step matters

Reusable modules and remote state fail gradually when ownership, versions, credentials, locks, or recovery evidence become stale. A quarterly operating review catches unsupported Terraform lines, provider deprecations, module consumers pinned to obsolete releases, disabled KMS keys, suspended bucket versioning, aged locks, and recurring drift before an emergency apply. Publishing a module is therefore the beginning of lifecycle responsibility, not the end of implementation.

What to understand

Track all supported module majors and their callers. Announce deprecation, retain moved blocks for the promised window, provide an upgrade test, and remove compatibility only through a documented breaking release.

Review Terraform and provider security advisories and current backend documentation. Test upgrades from the oldest supported caller, regenerate provider checksums for supported runner platforms, and reject unreviewed automatic upgrades.

Alert on backend policy changes, public-access changes, versioning suspension, KMS disable or deletion scheduling, unusual reads, state-object deletion, lock age, force-unlock, and apply outside the approved role.

Run scheduled no-change plans with `-detailed-exitcode` and bounded locking. Route exit 2 to human drift review and exit 1 to an operational failure queue; never auto-apply the resulting proposal.

System changes

Reads versions, provider selections, current remote objects, state, and locking; monitoring configuration is maintained through its own reviewed lifecycle.

Syntax explained

-detailed-exitcode: Returns 0 for no diff, 1 for error, and 2 for a non-empty plan so monitoring distinguishes drift from failure.
-lock-timeout=5m: Lets scheduled observation wait for a legitimate operation while remaining bounded and serialized.

Command

terraform -chdir=environments/prod version && terraform -chdir=environments/prod providers && terraform -chdir=environments/prod plan -detailed-exitcode -lock-timeout=5m

Example output / evidence

Terraform v1.15.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v6.55.0

No changes. Your infrastructure matches the configuration.

Checkpoint: The module and state platform have current owners and evidence

terraform -chdir=environments/prod plan -detailed-exitcode -lock-timeout=5m

Continue whenThe latest approved run exits 0 or has a triaged diff, supported versions are documented, alerts work, and a successful recovery rehearsal is less than one quarter old.

Stop whenNo owner can explain a non-empty plan, a supported caller cannot upgrade, state controls are stale, or recovery evidence exceeds the review interval.

If this step fails

A plan or state pull exposes a secret in a terminal, artifact, or log.

Likely causeState and plan files contain provider-returned attributes, and the `sensitive` marker controls display rather than guaranteeing omission from state. Debug logging or unredacted artifacts expanded the exposure.

Safe checks

Stop log and artifact publication.
Identify every principal and system that received the data.
Review whether the value is sensitive, ephemeral, or write-only in the configuration.
Preserve only access-controlled forensic evidence.

ResolutionTreat the value as compromised, rotate it at its authoritative system, revoke exposed artifacts, restrict state access, and redesign the resource to use ephemeral or write-only provider features where supported. Do not try to erase evidence by hand-editing state.

A module output leaks a value or creates broad state coupling.

Safe checks

terraform output
Review output blocks and their consumers.
Review `terraform_remote_state` data sources.
Inspect state access policy for downstream identities.

Security notes

Scheduled plans use a least-privilege read role and protected output. AI or chat systems must not ingest raw plans or state as a convenience.

Alternatives

Use HCP Terraform health assessments and registry governance while retaining independent KMS, identity, output, and recovery reviews appropriate to that platform.

Stop conditions

Disable automated observation if it can overlap applies without locking or publish sensitive plan output to a broadly readable log.

Finish line

Verification checklist

Module contract testterraform -chdir=modules/platform-service testPositive, negative, security, provider-alias, and upgrade cases pass without leaked secrets or orphaned resources.

Production convergenceterraform -chdir=environments/prod plan -detailed-exitcode -lock-timeout=5mThe command exits 0 after apply and the approved backend lock is acquired and released normally.

State durabilityterraform -chdir=recovery/prod state list && terraform -chdir=recovery/prod plan -refresh-onlyAn isolated retained state version decrypts, contains expected addresses, and can be compared without writing live infrastructure.

Dependency evidenceterraform -chdir=environments/prod version && terraform -chdir=environments/prod providersTerraform, provider, and module selections match the reviewed supported versions and immutable source policy.

Recovery guidance

Common problems and safe checks

Terraform reports that the state lock is already held.

Safe checks

terraform workspace show
terraform version
ps -ef | grep '[t]erraform'
Review the remote runner and change-system activity for the exact backend key.

Initialization asks to migrate state when only a backend setting was expected to change.

Safe checks

terraform version
git diff -- '*.tf' '*.hcl'
Inspect the non-secret backend configuration supplied by the approved runner.
List versioned state objects in the intended backend through the provider console.

A plan unexpectedly proposes many creates after backend initialization.

Likely causeThe configuration selected an empty state key, wrong workspace, wrong account, or wrong region. Terraform therefore sees configuration but no bindings to existing objects.

Safe checks

terraform workspace show
terraform state list
terraform providers
Compare the backend key and state serial with the protected inventory.

A plan or state pull exposes a secret in a terminal, artifact, or log.

Safe checks

Stop log and artifact publication.
Identify every principal and system that received the data.
Review whether the value is sensitive, ephemeral, or write-only in the configuration.
Preserve only access-controlled forensic evidence.

Terraform cannot decrypt or read the remote state object.

Likely causeThe runtime lost KMS permissions, the key was disabled or deleted, the object uses a different encryption context, or backend credentials select the wrong account.

Safe checks

terraform init -reconfigure
Verify the caller identity through the cloud provider's identity command.
Inspect KMS key status and backend audit logs without downloading state.
Confirm the expected bucket and key from the change record.

A CI plan waits indefinitely or times out acquiring a lock.

Likely causeRuns are not serialized at the pipeline layer, the lock timeout is shorter than normal applies, or an abandoned operation still owns the backend lock.

Safe checks

Inspect queued and running jobs for the same state key.
Compare lock metadata with the runner identity and start time.
Review recent apply duration and cancellation events.
Confirm that no scheduled drift job overlaps the change window.

A child module works in one account but fails in another.

Safe checks

terraform providers
terraform validate
Review provider blocks inside the child module.
Compare documented inputs with every environment-specific literal.

Upgrading a module version proposes replacement of existing resources.

Safe checks

terraform plan -out=module-upgrade.tfplan
terraform show module-upgrade.tfplan
git diff between the pinned module releases
Review module release notes and moved blocks.

Terraform installs an unexpected module release.

Safe checks

terraform get
terraform init
Review every module source and version argument.
Inspect the module manifest under `.terraform/modules` without committing it.

A module output leaks a value or creates broad state coupling.

Safe checks

terraform output
Review output blocks and their consumers.
Review `terraform_remote_state` data sources.
Inspect state access policy for downstream identities.

A module test passes but the production plan fails validation.

Likely causeTests did not cover the production provider version, caller aliases, variable edge cases, policy controls, or actual backend and state shape.

Safe checks

terraform version
terraform providers
terraform validate
terraform plan -out=review.tfplan

S3 locking returns AccessDenied for the `.tflock` object.

Likely causeThe backend identity can read and write the state object but its least-privilege policy does not include the separate lock-file key operations required by `use_lockfile = true`.

Safe checks

Confirm the exact state key and `<key>.tflock` path.
Inspect denied S3 actions in the provider audit log.
Compare the IAM resource paths with the official S3 backend permissions.
Confirm that no bucket policy condition selects a different prefix.

A backend migration succeeds but older state is no longer recoverable.

Likely causeObject versioning was not enabled, retention was too short, or migration validation considered only the current object and not lineage, serial, workspaces, and recovery access.

Safe checks

terraform state pull
Inspect bucket versioning and retained object versions.
List every Terraform workspace before and after migration.
Compare state lineage and serial in protected copies.

After the procedure

Alternatives and next steps

Consider these alternatives

Use HCP Terraform workspaces when centralized execution, encrypted state, run queues, policy, private registry, and organization-level access controls fit the operating model. Keep the same separation between reusable modules and environment roots.
Use the official GCS or AzureRM backend when operating in those clouds; verify current backend-specific locking, object versioning, encryption-key recovery, identity, and least-privilege semantics rather than mechanically translating S3 arguments.
Keep a small capability directly in one root when extraction would create a thin wrapper, hide simple behavior, or force unrelated teams into a shared release lifecycle. Reuse is valuable only when the contract is clearer than duplication.
Use a private registry for organization modules that need discovery, semantic versions, generated documentation, and controlled distribution. An immutable VCS commit is a simpler alternative for a small internal estate.
Publish non-sensitive shared identifiers through a dedicated parameter or configuration store instead of `terraform_remote_state` when consumers should not receive permission to read the producer's complete state snapshot.

Operate it safely

Publish a module release policy that defines semantic versioning, deprecation windows, moved-block retention, provider compatibility, signed release evidence, and supported upgrade paths.
Add `terraform test` cases for defaults, validation failures, provider aliases, conditional resources, outputs, and upgrade behavior from the previous supported module release.
Create alerts for state object deletion, KMS disable or deletion scheduling, unusual state reads, lock age, force-unlock, failed plans, and production apply outside the approved runner role.
Schedule a quarterly isolated state restore and module upgrade rehearsal. Record time to recover, missing permissions, stale documentation, and any secret leakage discovered in state or plans.
Split state by ownership and blast radius as the platform grows; do not use workspaces or one large state merely to avoid designing explicit contracts between independently operated components.

Reference

Frequently asked questions

Should a reusable child module configure its own backend?

No. Backend configuration belongs to the root module and is initialized before ordinary module evaluation. A child module should be reusable across roots and state locations; embedding backend ownership would couple the abstraction to one environment and does not create separate child-module state.

Does `.terraform.lock.hcl` lock state or module versions?

No. It records selected provider packages and checksums. State serialization comes from the backend, and child module versions must be constrained by the module block or immutable source reference.

Should we keep DynamoDB locking for an S3 backend?

Current Terraform documentation marks DynamoDB-based S3 locking as deprecated and recommends native S3 locking through `use_lockfile = true`. A transition may temporarily configure both for older clients, but the operational goal should be a supported Terraform line using native locking.

Can a sensitive output be considered absent from state?

No. Sensitive marking primarily controls display. Verify whether the value is stored, and prefer supported ephemeral or write-only mechanisms when the provider can operate without persisting it. Always protect the complete state as sensitive.

Recovery

Rollback

Rollback is a reviewed convergence operation, not restoration of an old state file over current reality. Stop new writes, preserve current state and provider evidence, restore the last compatible module and root configuration, create a fresh locked plan, and apply only actions that are demonstrably safer than the failed release. Use a historical state version only for state-corruption recovery after reconciling every remote object created since that snapshot.

Disable the production apply queue, identify the current lock owner, preserve the latest state object version, saved plan, source commit, provider logs, and actual remote-object inventory.
If the failure is only application behavior and the prior module release supports current state and provider schemas, pin that immutable version and render a new normal plan; do not reuse the old plan.
Review every proposed action. Reject deletion or replacement caused by address changes, and add or restore moved blocks when the object should remain bound at a compatible address.
Apply the reviewed rollback plan under the native lock, verify the service and unauthorized negative tests, and require a subsequent no-change plan.
If state is corrupt, copy the correct retained version into an isolated key first, reconcile lineage, serial, addresses, and remote objects, then follow the official state-recovery incident process under dual control.
Rotate any credential exposed through state, plan, debug log, or test artifact; restoring an older snapshot does not make an exposed secret safe again.

Evidence

Sources and review

Verified 2026-07-24Review due 2026-10-22

Install Terraformofficial Terraform standard module structureofficial Terraform module developmentofficial Terraform module block referenceofficial Terraform module refactoring and moved blocksofficial Terraform remote stateofficial Terraform S3 backend and native lockingofficial Terraform backend configurationofficial Terraform sensitive dataofficial Terraform dependency lock fileofficial