Cloud & Infrastructure / DevOps & CI/CD

Detect, review, and remediate Terraform drift safely

Compare protected refresh-only and normal plans, classify authority and impact, contain incidents, accept reality in HCL or restore intent through a saved plan, and verify full convergence without automatic apply.

180 min10 stepsChanges system stateRevision 1

Save or explore

#terraform #drift #refresh-only #state #incident-response #infrastructure-as-code #security #devops

0 of 10 steps completed

Goal

Turn Terraform drift into a deterministic, evidence-backed, human-owned workflow that detects remote changes, protects sensitive state, makes authority explicit, remediates without hidden replacement, and prevents recurrence.

Supported environments

Terraform Community Edition 1.15.x
Remote backend Locking and versioning required
Provider audit logs Current provider service

Prerequisites

Pinned production context Terraform 1.15.5, provider and module releases, root path, variables, backend, workspace, and aliases match the approved production workflow.terraform -chdir={{rootPath}} version && terraform -chdir={{rootPath}} providers && terraform -chdir={{rootPath}} workspace show
Protected state and plans Remote state is encrypted, versioned, locked, recoverable, and access-controlled; binary and JSON plans have equal confidentiality and short retention.
Read-only drift identity The observation runner can read required state and provider objects but cannot apply, force-unlock, administer state versions, or change unrelated infrastructure.
Provider and service evidence Provider audit logs, service metrics, synthetic checks, and denied-access tests can identify actor, impact, and post-remediation behavior.
Authority owners Configuration, service, security, incident, and external-controller owners are reachable to decide whether reality is accepted, intent restored, or ownership delegated.
Recovery and containment State recovery, service rollback, break-glass containment, credential rotation, and a second reviewer are available before any write remediation.

Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System

A local-first drift pipeline that runs locked refresh-only and normal plans with deterministic exit codes, protected artifacts, execution-context evidence, and no automatic apply.
A triage method that separates remote-versus-state observations from configuration-versus-desired-state proposals, classifies security and service impact, and identifies the actor and authority behind each difference.
Three safe resolution paths: update configuration to accept approved reality, apply a reviewed normal plan to restore intent, or document a narrowly scoped external owner with independent monitoring; each path has verification and rollback.

Observable outcome

Scheduled runs distinguish healthy exit 0, operational error exit 1, and drift exit 2, while every non-empty plan reaches an owner with source commit, state serial, provider versions, and protected evidence.
No one uses `terraform refresh`, broad `ignore_changes`, `-refresh=false`, `-target`, direct state editing, or automatic apply to make a drift alert disappear.
After remediation, a full normal plan exits 0, service and security checks pass, provider audit events match the approved action, and the decision is encoded in configuration or an explicit ownership contract.

Architecture

How the parts fit together

Terraform evaluates three related but different views: configuration expresses intended management, state records bindings and the latest committed attributes, and provider reads describe remote objects now. A refresh-only plan asks how state and outputs would change to reflect remote observations without proposing remote modification. A normal plan refreshes in memory and proposes how to make remote objects match configuration. The pipeline runs both under remote locking with the same pinned Terraform 1.15.5, provider packages, root, variables, identity, and state. Exit codes and JSON action summaries route work, while binary and JSON plans remain protected as sensitive data. Humans correlate provider audit logs, service behavior, emergency changes, and controller ownership. Remediation never begins from an unclassified diff: security exposure is contained as an incident; accepted manual change is encoded in HCL; unintended change is restored through a saved normal plan; a legitimate external authority receives only a narrow documented ignore and independent monitoring.

Configuration repositoryHolds reviewed desired state, module and provider constraints, lifecycle exceptions, ownership metadata, tests, and rollback history.

Remote state backendProvides sensitive versioned snapshots and locking. State is evidence and binding metadata, not a collaborative text file to edit directly.

Provider read APIsReturn current remote attributes during refresh and supply audit events that help identify manual or automated changes.

Drift runnerUses a short-lived read role, pinned toolchain, correct root context, locking, bounded time, detailed exit codes, protected plans, and no apply permission.

Triage ownerClassifies cause, authority, service and security impact, dependencies, accepted reality, intended restoration, or external ownership.

Remediation runnerApplies only a newly reviewed saved normal or refresh-only plan under a stronger role, then performs independent positive and negative verification.

Prove root, backend, workspace, commit, variables, Terraform, provider, identity, and freeze context before interpreting any diff.
Run and save a refresh-only plan to observe remote-versus-state changes; route exit 2 as evidence, never as approval to update state.
Run and save a full normal plan to observe desired convergence; compare its actions with refresh-only output and provider audit history.
Classify each attribute by owner, cause, security impact, service impact, replacement behavior, and dependency. Contain active exposure immediately through incident control.
Choose one authority: update HCL to accept approved reality, restore intent through a saved normal plan, or document a narrow external owner and monitoring.
Review the complete saved remediation plan, apply through the serialized write role, verify service and denied access, and require an empty full normal plan.
Record cause and prevention, correct emergency reconciliation or external-controller rules, and keep scheduled observation read-only.

Assumptions

Terraform 1.15.5 and pinned provider versions are supported. Scheduled and interactive runs use the same root, variables, backend, workspace, module releases, and provider aliases.
Remote state is versioned, encrypted, locked, recoverable, and treated as sensitive. Drift jobs have read access and cannot apply or force-unlock.
Provider audit logs and service monitoring identify out-of-band changes and user impact. Terraform output alone is not considered sufficient incident evidence.
Emergency changes are allowed only through a documented path that records actor, reason, expiry, and required configuration reconciliation.
Another controller may legitimately own selected attributes only when that ownership is explicit, narrow, testable, and independently monitored.
The first analysis is a full graph plan. `-target` is reserved for exceptional recovery and cannot establish that the whole root has converged.
Saved plan and JSON artifacts are protected because they can contain secrets and detailed infrastructure metadata even when CLI display redacts values.
No automatic process applies drift remediation. Security containment can be urgent, but desired-state adoption and restoration still require a named owner and verification.

Key concepts

Root module: The configuration directory from which Terraform runs. It owns backend and provider configuration, calls child modules, supplies environment-specific values, and is the correct boundary for a state workspace.
Child module: A reusable collection of resources called by a root or another child module. A child module declares provider requirements but normally receives provider configurations from its caller and does not decide where state is stored.
State binding: The association between a Terraform resource instance address and one remote object identity. Safe operations preserve a one-to-one mapping; duplicate imports and empty or wrong state keys violate that assumption.
Backend: The subsystem that stores state and, depending on type, provides locking and operation features. Backend configuration is evaluated before ordinary Terraform expressions, so it cannot use normal input variables or resource outputs.
State lock: A serialization mechanism that prevents concurrent writers from calculating and committing changes against the same snapshot. It protects consistency only when every writer honors it.
Saved plan: A binary plan artifact that captures a reviewed proposal for a particular configuration, state lineage, provider set, and variable context. It is sensitive and becomes stale when any relevant input changes.
Refresh: The read of provider APIs that updates Terraform's in-memory understanding of remote objects before comparison. A refresh can reveal drift even when the configuration repository did not change.
Dependency lock file: The `.terraform.lock.hcl` record of selected provider versions and checksums. It should be committed and reviewed; it does not lock child module versions or serialize state operations.
Resource drift: A difference between Terraform's recorded or configured understanding and the current remote object, often caused by console, API, incident, another controller, provider normalization, or failed partial operations.
Refresh-only plan: A plan mode that proposes updates to state and root outputs so they reflect remote objects, without proposing changes to those remote objects. It is observation and possible state acceptance, not automatic remediation.
Normal plan: The default mode that refreshes remote observations and proposes actions to make remote infrastructure match configuration.
Accept reality: A reviewed decision that an out-of-band remote setting is now desired. Durable acceptance requires updating configuration, not only refreshing state.
Restore intent: A reviewed decision that configuration remains authoritative and Terraform should change the remote object back through a normal saved plan.
External authority: A named controller that legitimately owns one narrow attribute. Terraform may ignore that attribute only with explicit boundaries and independent monitoring.

Before you copy

Values used in this guide

{{rootPath}}

Repository path of the one Terraform root being observed and remediated.

Example: environments/prod/network

{{driftCaseId}}

Stable incident or change identifier tying plans, state versions, provider events, owner, decision, and verification together.

Example: DRIFT-2026-0718

{{refreshPlan}}secret

Protected path for the refresh-only observation artifact.

Example: /secure/DRIFT-2026-0718/refresh-only.tfplan

{{normalPlan}}secret

Protected path for the full normal convergence proposal.

Example: /secure/DRIFT-2026-0718/normal.tfplan

{{remediationPlan}}secret

Protected path for the final reviewed plan after the authority decision.

Example: /secure/DRIFT-2026-0718/remediation.tfplan

{{reviewWindow}}

Maximum time a saved remediation plan may remain eligible before state and remote context must be refreshed.

Example: 30m

Security and production boundaries

Terraform state and saved plan files are sensitive operational records. They can contain secrets, identifiers, network topology, policy documents, provider-returned attributes, and values hidden from normal CLI display. Encrypt them, restrict read as well as write access, keep them out of source control, and avoid attaching them to ordinary tickets.
Use short-lived workload identity or identity federation for provider and backend access. Do not put access keys, session tokens, KMS material, or backend credentials in `.tf` files, `-backend-config=KEY=value` shell history, generated plans, screenshots, or example output.
A `sensitive = true` value is redacted in selected interfaces but can still exist in state and plan files. Prefer ephemeral values and provider write-only arguments where Terraform 1.15 and the provider support them, then verify the provider schema instead of assuming a password was omitted.
Keep state administration separate from ordinary infrastructure administration. The ability to overwrite state, restore older versions, delete lock files, or force-unlock can redirect a future apply even when the principal cannot directly modify every managed object.
Never publish `TF_LOG=TRACE` output without security review. Provider requests, responses, environment details, and values can appear in debug logs; enable logging only for a bounded diagnostic and store the result as a secret-bearing incident artifact.
A weakened firewall, identity policy, encryption setting, audit control, public-access block, retention rule, or secret can be an active incident. Do not delay containment merely to preserve a clean Terraform workflow; contain safely, preserve evidence, and reconcile configuration immediately afterward.
A read-only drift runner still reads sensitive state and provider attributes. Give it no apply, force-unlock, broad state administration, or unrelated provider permissions, and keep its artifacts short-lived and access-controlled.
Do not send raw drift plans, state, provider responses, or debug logs to public AI systems. Sanitize only the minimum action and error context if assistance is required.
Audit broad lifecycle ignores regularly. An `ignore_changes = all` pattern can turn a managed resource into an unmonitored state placeholder while still creating the impression of control.

Stop before continuing if

Stop if root path, backend key, workspace, state lineage, variables, provider alias, caller identity, or source commit differs from the expected production context.
Stop if another plan, apply, import, state operation, provider maintenance, or manual emergency change is active.
Stop if active security exposure or customer impact exists; switch to incident containment while preserving Terraform evidence.
Stop if the plan proposes unexplained delete or replacement, touches unrelated owners, or includes a provider or module upgrade not part of drift remediation.
Stop if the team cannot name whether configuration, remote reality, or a specific external controller is authoritative for each changed attribute.
Stop if anyone suggests broad `ignore_changes`, `-refresh=false`, `-lock=false`, routine `-target`, direct state editing, or automatic apply as a shortcut.
Stop if the saved plan is stale, state serial changed, remote objects changed again, or approval exceeded {{reviewWindow}}.
Stop after remediation unless a complete normal plan exits 0 and service, security, and denied-access checks pass.

verification

Prove execution context before reading drift

read-only

From a clean approved checkout, record Terraform and provider versions, commit, root, backend, workspace, state addresses, variables, caller identity, and active changes. Acquire the lock through a bounded read-only plan.

Why this step matters

A wrong workspace, backend key, provider alias, variable file, region, account, or source commit can produce a convincing but meaningless mass diff. Context validation is the highest-value safety gate because remediation against the wrong root can create or destroy unrelated infrastructure. Recording the state serial and active change queue also establishes whether a later saved plan remains valid.

What to understand

Use the same immutable toolchain and variable sources as production applies. Do not run init with upgrade behavior inside a drift case because provider normalization changes would become mixed with remote drift.

Compare known addresses and backend metadata with the root inventory without dumping state. Verify caller identity through provider-native identity commands and confirm the drift role has no apply or state-administration privilege.

Inspect running CI, maintenance, imports, and emergency changes for the same objects. A legitimate active writer explains transient differences and should complete before a stable observation.

System changes

Reads local configuration, provider selection, workspace, and state address metadata; no remote infrastructure changes.

Syntax explained

-chdir={{rootPath}}: Runs each command in the one explicit root rather than relying on the shell's current directory.
state list: Confirms expected bindings without exposing full sensitive attribute values.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Root Path{{rootPath}}Repository path of the one Terraform root being observed and remediated.

terraform -chdir={{rootPath}} version && terraform -chdir={{rootPath}} providers && terraform -chdir={{rootPath}} workspace show && terraform -chdir={{rootPath}} state list

Example output / evidence

Terraform v1.15.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v6.55.0
prod
module.network.aws_security_group.app

Checkpoint: The drift case names one exact production context

terraform -chdir={{rootPath}} workspace show && terraform -chdir={{rootPath}} state list

Continue whenRoot, commit, backend, workspace, lineage, serial, providers, variables, identity, addresses, and active-change status match the production inventory.

Stop whenAny context differs, state is empty, a lock owner is unexplained, or provider/module versions changed outside scope.

If this step fails

A plan contains hundreds of unrelated drift changes.

Likely causeThe state boundary is too broad, a provider upgrade changed normalization, a shared external controller made bulk edits, or wrong variables, workspace, account, or region were selected.

Safe checks

terraform version
terraform workspace show
terraform providers
Compare backend key, provider alias, variable source, and recent bulk-change logs.

ResolutionFreeze applies, verify execution context first, and bisect provider or configuration changes in an isolated checkout. Split analysis by owner and risk; do not use `-target` or a mass refresh-only apply to make the report smaller.

A plan unexpectedly proposes many creates after backend initialization.

Likely causeThe configuration selected an empty state key, wrong workspace, wrong account, or wrong region. Terraform therefore sees configuration but no bindings to existing objects.

Safe checks

terraform workspace show
terraform state list
terraform providers
Compare the backend key and state serial with the protected inventory.

ResolutionDo not apply. Restore the intended backend coordinates, reinitialize, and confirm the expected state addresses before planning again. If the original state is unavailable, treat recovery as an incident rather than attempting to recreate production from an empty state.

Security notes

Record identities and addresses, not raw state or environment credentials.

Alternatives

Run observation in the approved CI environment when local workstations cannot reproduce identity and variable context reliably.

Stop conditions

Do not interpret a diff until context evidence is complete.

verification

Save a refresh-only observation plan

read-only

Run a locked refresh-only plan with deterministic exit codes and save it to protected storage. This reveals how state and root outputs would change to match current provider observations without proposing remote modification.

Why this step matters

Refresh-only mode isolates provider observations from normal convergence and is safer than the legacy `terraform refresh` command because it presents a reviewable plan rather than immediately overwriting state. Exit code 2 means differences were detected, not failure. The artifact establishes what Terraform saw at one moment but does not decide whether those remote values should become desired configuration.

What to understand

Use `-detailed-exitcode`: route 0 to healthy observation, 1 to runner or provider failure, and 2 to triage. Never coerce 2 to success and discard its evidence.

Protect binary and rendered output. Plans may contain secrets, policies, endpoints, IDs, and values hidden in normal terminal display. Keep artifact retention long enough for the case but shorter than state retention.

Record state serial, provider audit-window start, run identity, commit, and timestamps. Remote changes after planning can make the observation stale even though it remains useful forensic evidence.

Do not apply refresh-only at this stage. First compare a normal plan and determine authority for every difference.

System changes

Reads provider APIs and state under a lock and writes a sensitive local plan; it does not change state or remote objects.

Syntax explained

-refresh-only: Changes the planning objective to state and output updates that reflect remote objects.
-detailed-exitcode: Distinguishes no difference, operational error, and detected difference as exit 0, 1, and 2.
-out={{refreshPlan}}: Preserves the exact sensitive observation for protected review.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

Root Path{{rootPath}}Repository path of the one Terraform root being observed and remediated.Refresh Plan{{refreshPlan}}Protected path for the refresh-only observation artifact.

terraform -chdir={{rootPath}} plan -refresh-only -detailed-exitcode -lock-timeout=5m -input=false -out={{refreshPlan}}

Example output / evidence

Note: Objects have changed outside of Terraform

  # aws_security_group.app has changed
  ~ resource "aws_security_group" "app" {
      ~ description = "managed" -> "incident temporary access"
    }

This is a refresh-only plan, so Terraform will not take any actions to undo these.

Checkpoint: Remote-versus-state differences are captured without writing

terraform -chdir={{rootPath}} show {{refreshPlan}}

Continue whenEvery changed address and attribute is visible, exit status is recorded correctly, and no apply or state write occurred.

Stop whenThe plan errors, context changes, artifact leaks, or active security exposure requires immediate containment.

If this step fails

The scheduled drift plan exits 2 but the human-readable log says little.

Likely causeAutomation used `-detailed-exitcode`, correctly reporting a non-empty plan, while log truncation, redaction, or an incorrect expectation treated exit 2 as a generic failure.

Safe checks

terraform plan -refresh-only -detailed-exitcode -out=drift.tfplan
terraform show drift.tfplan
terraform show -json drift.tfplan
Inspect artifact retention and log truncation settings.

ResolutionRoute exit 2 to a drift-review queue, exit 1 to operational failure, and exit 0 to healthy. Preserve the protected saved plan, render a sanitized action summary, and assign an owner; never auto-apply because a monitor observed a difference.

Drift appears on every plan immediately after a successful apply.

Likely causeThe provider normalizes an argument, an API supplies unstable ordering or defaults, another controller owns the field, or module configuration writes a value the service transforms.

Safe checks

terraform plan -out=repeat.tfplan
terraform show repeat.tfplan
Read provider release notes and current resource schema.
Compare remote API values across two read-only observations.

ResolutionPin and test the provider, configure the normalized representation, model set semantics, or document a narrowly scoped external owner. Use `ignore_changes` only for a specific legitimate authority with independent monitoring; do not hide the complete resource.

Security notes

Render only sanitized summaries into notifications; keep the actual plan behind state-equivalent access controls.

Alternatives

Use HCP Terraform health assessment as the observation source when its workspace context and evidence are equivalent.

Stop conditions

Never replace this command with unreviewed `terraform refresh`.

verification

Save a full normal convergence plan

read-only

Without changing configuration or context, create a full normal saved plan. Compare it with refresh-only output to see what Terraform would change remotely to satisfy configuration.

Why this step matters

A refresh-only diff says remote reality differs from recorded state; a normal plan says what configuration would do about the current observation. Both are required because an empty refresh-only plan can coexist with configuration drift, and applying refresh-only can be followed by a normal proposal to revert the remote value. The normal plan is still only a proposal and must not be auto-applied.

What to understand

Keep Terraform, provider, module, variables, identity, backend, and commit identical to the refresh-only run. Otherwise the comparison mixes multiple causes.

Inspect all actions, replacement paths, unknown values, dependencies, outputs, and provider warnings. A one-line attribute difference can force replacement of a stateful endpoint.

Compare plan JSON at address and attribute paths, but retain human review of provider semantics and service impact. Counts alone cannot establish risk.

If the full plan is enormous, verify context and recent provider upgrades before triaging attributes. Do not narrow the final acceptance with routine `-target`.

System changes

Reads provider APIs and state under a lock and writes a sensitive normal-plan artifact; no remote change is applied.

Syntax explained

normal planning mode: Proposes how remote objects should change to match the current configuration after refresh.
-out={{normalPlan}}: Preserves the complete proposal for classification and prevents a reviewer from relying on truncated logs.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

Root Path{{rootPath}}Repository path of the one Terraform root being observed and remediated.Normal Plan{{normalPlan}}Protected path for the full normal convergence proposal.

terraform -chdir={{rootPath}} plan -detailed-exitcode -lock-timeout=5m -input=false -out={{normalPlan}}

Example output / evidence

Terraform will perform the following actions:
  # aws_security_group.app will be updated in-place
  ~ resource "aws_security_group" "app" {
      ~ description = "incident temporary access" -> "managed"
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Checkpoint: Desired convergence is distinct from observation

terraform -chdir={{rootPath}} show {{normalPlan}}

Continue whenEvery proposed create, update, replacement, deletion, and output change is classified separately from refresh-only state observations.

Stop whenA replacement or unrelated action is unexplained, context differs between plans, or a provider upgrade entered the case.

If this step fails

A refresh-only plan is empty but a normal plan proposes changes.

Likely causeRemote objects match the current state snapshot, but configuration differs from state. Refresh-only mode asks only whether state should adopt remote observations; it does not evaluate normal convergence intent in the same way.

Safe checks

terraform plan -refresh-only -out=refresh.tfplan
terraform plan -out=normal.tfplan
terraform show normal.tfplan
Compare configuration, state, and provider observations attribute by attribute.

ResolutionUse the normal plan as the desired-state proposal. Decide whether configuration should change to accept reality or remote infrastructure should be restored in a reviewed apply. Do not apply refresh-only and declare drift resolved while normal configuration still disagrees.

A drift alert disappears without a reviewed change.

Likely causeAnother refresh or apply rewrote state, the remote object changed again, the scheduled job used another workspace, or plan artifacts expired before triage.

Safe checks

Compare state object versions and serials.
Review provider audit logs across the alert window.
Compare scheduled-run backend and workspace metadata.
Recover the protected original drift plan artifact.

ResolutionKeep immutable alert metadata and protected plan evidence long enough for investigation. Reconstruct configuration, state, and remote observations at the event time, identify the writer, and correct monitoring context before closing the alert.

Security notes

A normal plan can reveal the former and current values of a secret-bearing attribute; handle both as compromised if exposed.

Alternatives

Split the state root as a future architecture change when complete plans cannot be reviewed by one accountable owner.

Stop conditions

Do not approve a targeted subset while the full normal plan remains unreviewed.

decision

Classify cause, authority, and impact for every difference

read-only

Correlate refresh-only and normal plans with configuration history, state versions, provider audit logs, incidents, controller events, service monitoring, and ownership. Assign each attribute a cause and one authority decision.

Why this step matters

Terraform can show differences but cannot know whether a manual incident action should be retained, whether another controller legitimately owns a field, or whether a provider normalization is harmless. Authority is a product, security, and operations decision. Classification prevents an operator from blindly reverting a life-saving emergency change or permanently accepting a security weakening through state refresh.

What to understand

Classify origin: manual console/API, emergency action, another IaC root, controller, provider default or normalization, failed partial apply, module/provider upgrade, data-source change, or configuration error.

Classify impact: confidentiality, integrity, availability, cost, compliance, identity, network exposure, data durability, endpoint stability, and downstream dependency.

Name one authority for each attribute: Terraform configuration, approved remote reality that must enter HCL, or a specific external controller. Ambiguous co-ownership is a design defect to resolve.

Identify replacement and rollback behavior before choosing remediation. A force-new field, key change, or policy dependency can make a small diff operationally destructive.

System changes

Reads plan artifacts and external audit evidence; no state, configuration, or infrastructure changes.

Syntax explained

show -json: Provides exact addresses and actions for a protected classification manifest.
provider audit logs: Identify actor, time, API action, and request context that Terraform state alone does not preserve.

Command

Fill variables0/3 ready

Values stay on this page and are never sent or saved.

Root Path{{rootPath}}Repository path of the one Terraform root being observed and remediated.Refresh Plan{{refreshPlan}}Protected path for the refresh-only observation artifact.Normal Plan{{normalPlan}}Protected path for the full normal convergence proposal.

terraform -chdir={{rootPath}} show -json {{refreshPlan}} | jq -r '.resource_changes[] | [.address, (.change.actions|join(","))] | @tsv' && terraform -chdir={{rootPath}} show -json {{normalPlan}} | jq -r '.resource_changes[] | [.address, (.change.actions|join(","))] | @tsv'

Example output / evidence

aws_security_group.app	update
aws_security_group.app	update
Classification: manual emergency change; security-sensitive; incident owner SEC-42; authority decision pending.

Checkpoint: Every diff has one owner and one safe decision

terraform -chdir={{rootPath}} show {{normalPlan}}

Continue whenThe case records cause, actor, authority, security and service impact, replacement behavior, dependencies, remediation path, verifier, and rollback for every action.

Stop whenAny attribute lacks an owner, active exposure exists, audit evidence is missing, or teams disagree on authority.

If this step fails

Terraform detects a security control weakened outside code.

Likely causeAn emergency or manual change altered policy, network exposure, encryption, logging, identity, or retention, and the normal workflow was bypassed or failed to reconcile.

Safe checks

Preserve the refresh-only and normal plans.
Review provider audit logs and change identity.
Verify current exposure from an isolated authorized and denied client.
Check incident and emergency-change records.

ResolutionTreat active exposure as an incident. Contain through the approved emergency path, preserve evidence, then update configuration or apply a reviewed restoration plan. Rotate exposed credentials and validate dependent systems; do not wait for routine drift automation to remediate.

The team cannot decide whether remote reality or configuration is authoritative.

Likely causeOwnership, emergency-change policy, service intent, and external-controller boundaries were never documented, so drift is treated as a technical diff without a product or security decision.

Safe checks

Identify service and configuration owners.
Review change, incident, and provider audit records.
Compare current behavior with SLO and security requirements.
List downstream systems that depend on each alternative.

ResolutionPause remediation, name a decision owner, classify risk and user impact, and document either accept-reality or restore-intent with verification and rollback. If exposure is active, contain safely first while preserving the final desired-state decision for review.

Security notes

Contain active security weakening immediately through incident control, then reconcile Terraform; do not wait for ordinary review while exposure continues.

Alternatives

Escalate to the service and security owner rather than forcing a technical decision when authority is ambiguous.

Stop conditions

No remediation plan is allowed until every changed security and lifecycle attribute is classified.

warning

Contain active security or availability impact

caution

When drift exposes data, identity, network access, encryption, retention, or service availability, switch to the incident process. Apply the smallest provider-native or Terraform-controlled containment, preserve evidence, and open immediate configuration reconciliation.

Why this step matters

A read-only drift workflow must not become a reason to leave customers exposed. Incident containment may legitimately occur outside the normal apply path when latency matters, but it needs a precise scope, actor, evidence, verification, and expiry. Terraform configuration must then represent the accepted final state so the next plan does not undo containment or repeatedly report the same emergency change.

What to understand

Preserve refresh-only and normal plan artifacts, state object version, provider audit logs, source commit, monitoring, and affected identities before changing more than necessary.

Choose the smallest reversible containment that closes active exposure or restores availability. Avoid broad policy replacement, unrelated cleanup, provider upgrades, or destructive rebuild during the incident.

Verify containment with an authorized positive test and unauthorized negative test from an isolated client. Rotate credentials or keys that may have been exposed; configuration convergence alone does not revoke leaked material.

Open a reconciliation change immediately. Decide whether containment becomes desired HCL or is replaced by a better reviewed control, then require a full plan and post-incident review.

System changes

The diagnostic command is read-only; any incident containment is a separately authorized provider or Terraform change recorded by the incident process.

Syntax explained

smallest containment: Limits blast radius and keeps the final Terraform reconciliation understandable.
positive and negative tests: Prove intended service access remains while the exposed path is closed.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

Root Path{{rootPath}}Repository path of the one Terraform root being observed and remediated.Normal Plan{{normalPlan}}Protected path for the full normal convergence proposal.

terraform -chdir={{rootPath}} show {{normalPlan}}

Example output / evidence

Incident DRIFT-2026-0718: public ingress detected on tcp/22; containment owner assigned; provider audit evidence preserved; normal Terraform remediation pending review.

Checkpoint: Exposure is closed and reconciliation is mandatory

terraform -chdir={{rootPath}} plan -detailed-exitcode -lock-timeout=5m

Continue whenService and denied-access checks prove containment, evidence is preserved, credentials are rotated where needed, and the remaining diff is assigned to a reviewed reconciliation.

Stop whenContainment broadens access, destroys evidence, breaks required service, or no configuration owner accepts reconciliation.

If this step fails

Terraform detects a security control weakened outside code.

Likely causeAn emergency or manual change altered policy, network exposure, encryption, logging, identity, or retention, and the normal workflow was bypassed or failed to reconcile.

Safe checks

Preserve the refresh-only and normal plans.
Review provider audit logs and change identity.
Verify current exposure from an isolated authorized and denied client.
Check incident and emergency-change records.

Security notes

Do not include raw secrets, state, or provider request bodies in the incident timeline; link protected evidence.

Alternatives

Use a pre-approved break-glass Terraform plan for common exposure patterns when it can run faster than safe manual API changes.

Stop conditions

Stop routine drift handling and page incident response whenever active exposure or customer impact is confirmed.

config

Accept approved reality by updating configuration

caution

When the remote change is correct, modify HCL, module inputs, and tests to express it. Create a new normal plan and require no remote changes; use refresh-only apply only when reviewed state/output updates remain and never as a substitute for HCL.

Why this step matters

Durable acceptance means configuration becomes the source of truth for the approved remote setting. Applying refresh-only can update state but leaves old HCL ready to revert the value on the next normal plan. Updating code also gives reviewers history, tests, ownership, and downstream module impact. A refresh-only apply is useful only for residual state or output synchronization after that decision and still requires a protected saved plan.

What to understand

Change the smallest explicit configuration surface and update variable validation, module contract, policy tests, documentation, and service expectations. Do not combine provider or module upgrades.

Run a full normal plan. The desired result is no remote action because reality already matches the new HCL. If other changes appear, classify them separately.

If refresh-only still proposes state or output changes, review and apply its saved artifact only after verifying it will not modify remote objects. Then rerun a complete normal plan.

Commit actor, reason, audit event, authority decision, expiry if temporary, and post-change verifier. Temporary accepted changes need an explicit removal date and alert.

System changes

Updates reviewed Terraform source and may write state/output metadata through a separately approved refresh-only apply; it does not intentionally change remote infrastructure.

Syntax explained

explicit HCL: Makes approved reality durable and reviewable in desired configuration.
refresh-only apply: Optionally records reviewed provider observations in state without changing remote objects; it does not replace the HCL update.

File reviewed Terraform configuration

Configuration

# Encode the approved remote value as desired configuration.
# Keep the change explicit, tested, and owned; do not silence it with ignore_changes.
resource "aws_security_group" "app" {
  description = "approved incident-hardened policy"
}

Example output / evidence

aws_security_group.app: Refreshing state... [id=sg-0123456789abcdef0]

No changes. Your infrastructure matches the configuration.

Checkpoint: Approved reality is now reproducible desired state

terraform -chdir={{rootPath}} plan -detailed-exitcode -lock-timeout=5m

Continue whenThe full normal plan exits 0 and tests, security controls, service behavior, ownership, and documentation match the accepted setting.

Stop whenThe plan proposes remote change, broad ignore rules are added, or temporary acceptance lacks an expiry and owner.

If this step fails

A refresh-only plan is empty but a normal plan proposes changes.

Safe checks

terraform plan -refresh-only -out=refresh.tfplan
terraform plan -out=normal.tfplan
terraform show normal.tfplan
Compare configuration, state, and provider observations attribute by attribute.

A refresh-only apply updates state but the next plan proposes reverting the remote change.

Likely causeRefresh-only accepted the provider observation into state, while unchanged configuration still declares the previous desired value. Normal planning therefore proposes convergence back to configuration.

Safe checks

terraform plan -refresh-only
terraform plan
git diff -- '*.tf' '*.tfvars'
Review the approved desired-state decision.

ResolutionIf the remote change is accepted, update configuration to represent it and review a normal plan. If it is not accepted, use the normal plan as a remediation proposal. Do not repeatedly refresh state as a substitute for deciding authority.

Security notes

Do not accept a weakened control merely because it already exists remotely; security owner approval and compensating controls remain necessary.

Alternatives

Restore intent in the next step when remote reality is unauthorized or incompatible with service requirements.

Stop conditions

Do not apply refresh-only repeatedly while configuration remains different.

command

Restore intended configuration through a reviewed saved plan

caution

When HCL remains authoritative, create a fresh full saved plan after classification, inspect replacement and dependency paths, obtain service and security approval, and apply the exact artifact through the serialized write role.

Why this step matters

Restoration changes real infrastructure and can interrupt service or overwrite an emergency safeguard. A new saved plan is required because the earlier observation artifact may be stale and was created under a read role. Replacement paths, immutable attributes, dependencies, and provider behavior must be reviewed before write approval. Applying the exact artifact preserves the relationship between decision and execution.

What to understand

Recheck root context, state serial, remote observations, incident status, source commit, provider versions, and approval window. Discard the plan if any changes.

Reject unexplained replacement or deletion. For a necessary replacement, use a separate migration with data copy, parallel capacity, endpoint cutover, verification, and rollback rather than a drift shortcut.

Verify backups and service rollback before apply. Monitor provider operations and service metrics; do not cancel a slow operation without understanding provider behavior.

After apply, run positive and denied-access tests, compare provider audit events, verify state lock release and version creation, and create a complete normal plan.

System changes

Applies the exact reviewed normal plan to restore remote objects and writes a new versioned state snapshot under the lock.

Syntax explained

-out={{remediationPlan}}: Saves the exact restoration proposal for approval and apply.
{{remediationPlan}}: Makes apply execute the reviewed artifact instead of recalculating a potentially different proposal.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

Root Path{{rootPath}}Repository path of the one Terraform root being observed and remediated.Remediation Plan{{remediationPlan}}Protected path for the final reviewed plan after the authority decision.

terraform -chdir={{rootPath}} plan -out={{remediationPlan}} -lock-timeout=5m -input=false && terraform -chdir={{rootPath}} show {{remediationPlan}} && terraform -chdir={{rootPath}} apply -lock-timeout=5m -input=false {{remediationPlan}}

Example output / evidence

aws_security_group.app: Modifying... [id=sg-0123456789abcdef0]
aws_security_group.app: Modifications complete after 1s

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Checkpoint: Intent is restored without hidden collateral change

terraform -chdir={{rootPath}} plan -detailed-exitcode -lock-timeout=5m

Continue whenThe command exits 0, service and security checks pass, provider audit shows only approved actions, and no replacement or dependency failure remains.

Stop whenThe saved plan is stale, replacement is unexplained, service metrics degrade, denied access succeeds, or a fresh plan remains non-empty.

If this step fails

A drift remediation plan proposes replacement of a critical object.

Likely causeThe changed attribute is immutable, an address or `for_each` key changed, provider schema evolved, or configuration attempts to restore a value the API can only change through replacement.

Safe checks

terraform show drift-remediation.tfplan
terraform show -json drift-remediation.tfplan
Review replacement paths and provider documentation.
Identify data, identity, endpoint, and dependency migration requirements.

ResolutionDo not apply. Restore the last non-replacing configuration if safe, adopt approved reality, add a moved block for address-only changes, or design a parallel replacement with data migration, cutover, rollback, and service owner approval.

Security notes

Use a stronger apply role only for the approved window and revoke the session afterward; do not grant write capability to the drift monitor.

Alternatives

Use provider-native emergency restoration only when Terraform cannot meet containment latency, followed immediately by HCL reconciliation.

Stop conditions

Never apply an old observation plan as remediation after configuration, state, or remote objects changed.

decision

Document narrow external ownership instead of hiding drift

read-only

If another controller legitimately owns one attribute, name it, define boundaries and conflict behavior, narrow any lifecycle ignore to that attribute, and add independent monitoring. Keep all security-critical fields under explicit authority.

Why this step matters

Some systems intentionally change capacity or generated metadata. Terraform can coexist only when ownership is precise. Broad `ignore_changes` hides unrelated security and availability changes while leaving a false impression of management. A narrow rule plus controller audit, bounds, alerts, and failure behavior gives one authority per attribute and keeps the remainder visible.

What to understand

Document controller identity, exact attribute, allowed range, reconciliation interval, audit source, failure mode, emergency owner, and what happens when Terraform and controller disagree.

Keep boundaries such as min/max capacity, launch identity, network policy, encryption, logging, and public access under explicit Terraform or another single named authority.

Test by changing the externally owned value through the controller in a non-production fixture, proving Terraform remains quiet only for that attribute while detecting a separate security drift.

Review every ignore rule quarterly and remove it when the external controller is retired. Use policy checks to reject `ignore_changes = all` and whole-resource patterns.

System changes

May update lifecycle configuration and monitoring through a separate reviewed change; the diagnostic command itself reads only.

Syntax explained

ignore_changes = [specific_attribute]: Delegates one exact field to another authority; it must never be a blanket drift-suppression tool.
independent monitoring: Detects harmful changes to the delegated attribute because Terraform intentionally will not propose restoring it.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

Root Path{{rootPath}}Repository path of the one Terraform root being observed and remediated.Normal Plan{{normalPlan}}Protected path for the full normal convergence proposal.

terraform -chdir={{rootPath}} plan -out={{normalPlan}} -lock-timeout=5m && terraform -chdir={{rootPath}} show {{normalPlan}}

Example output / evidence

No changes. Your infrastructure matches the configuration.
External owner contract: autoscaler owns desired_capacity only; min_size, max_size, launch template, identity, and network policy remain Terraform-owned.

Checkpoint: Coexistence has one authority per attribute

terraform -chdir={{rootPath}} plan -detailed-exitcode -lock-timeout=5m

Continue whenThe plan exits 0, controlled external changes are monitored, and an injected non-delegated security change still appears in testing.

Stop whenOwnership overlaps, ignores cover whole resources or security controls, or no independent alert detects an out-of-bounds value.

If this step fails

A lifecycle `ignore_changes` rule conceals a dangerous difference.

Likely causeThe rule was added broadly to silence noisy plans without naming the external owner or monitoring security-relevant attributes.

Safe checks

Review every lifecycle block.
Compare ignored attributes with provider audit and configuration policy.
Run a controlled plan after removing the rule in an isolated branch.
Identify the system that is supposed to own each ignored field.

ResolutionRemove broad ignores, assign a single authority, configure a narrow attribute-level rule only when justified, and add independent drift detection for that field. Remediate any concealed security change through incident or reviewed change control.

Security notes

Never delegate identity policy, encryption, public exposure, retention, or audit controls without an equally explicit authoritative security controller.

Alternatives

Remove Terraform management of the entire object through a reviewed release-without-destroy when another controller genuinely owns its complete lifecycle.

Stop conditions

Reject any change whose only justification is making the plan quiet.

verification

Verify complete convergence, service behavior, and denial

read-only

After the chosen remediation, run a complete normal plan, service synthetic checks, unauthorized negative tests, provider audit review, state version and lock checks, and monitoring validation. Do not use targeting for acceptance.

Why this step matters

An empty plan proves only that Terraform sees configuration, state, and provider observations as converged. It cannot prove application health, correct identity behavior, DNS, data integrity, client access, or monitoring. Independent positive and negative checks close that gap. Complete-graph planning ensures no dependency or unrelated drift was hidden by a targeted remediation.

What to understand

Require detailed exit 0 from a fresh full plan. Exit 2 reopens classification and exit 1 is an operational failure, not success.

Run service checks from representative authorized clients and denied identities. Validate security, availability, latency, data, endpoints, certificates, logs, and alerts appropriate to the changed control.

Compare provider audit events to the saved plan and approved identity. Confirm no console, API, controller, or second Terraform root changed the object during remediation.

Confirm the backend released its lock, wrote a recoverable encrypted state version, and did not expose plans or logs. Preserve sanitized evidence and protect raw artifacts.

System changes

Reads state, provider APIs, service endpoints, audit logs, and monitoring; no remote change is applied.

Syntax explained

full plan: Evaluates the complete dependency graph rather than a targeted subset.
-detailed-exitcode: Makes complete convergence a deterministic release gate.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Root Path{{rootPath}}Repository path of the one Terraform root being observed and remediated.

terraform -chdir={{rootPath}} plan -detailed-exitcode -lock-timeout=5m

Example output / evidence

No changes. Your infrastructure matches the configuration.
Service synthetic: PASS
Unauthorized access: DENIED
State lock: released
Audit actions: match approved remediation

Checkpoint: Terraform and the real service agree safely

terraform -chdir={{rootPath}} plan -detailed-exitcode -lock-timeout=5m

Continue whenExit 0, passing positive and denied checks, matching audit events, healthy metrics, released lock, and recoverable state are all recorded.

Stop whenAny check fails, audit contains an unknown writer, state is not recoverable, or a targeted plan is offered as final evidence.

If this step fails

The drift plan fails because provider credentials expired.

Likely causeScheduled observation uses a static or overly short session without refresh, identity federation is misconfigured, or provider and backend identities have different expiration behavior.

Safe checks

Verify caller identity without exposing credentials.
Review federation and runner token issuance logs.
terraform init -reconfigure -input=false
Run a read-only provider query for one known object.

ResolutionRepair short-lived workload identity and keep the run failed closed. Do not replace it with a static administrator key. Distinguish credential failure from an empty plan so monitoring cannot report false health.

A targeted plan appears safe but the full plan later shows additional changes.

Likely causeUsing `-target` narrowed graph evaluation and hid dependencies or other drift. Targeting is an exceptional recovery tool, not a general drift-remediation workflow.

Safe checks

terraform plan -out=full.tfplan
terraform show full.tfplan
Compare targeted and complete dependency graphs.
Review why targeting was introduced.

ResolutionDiscard the targeted approval and review a complete normal plan. Separate state by ownership if full plans are consistently too large; do not normalize routine operations around partial graph visibility.

Security notes

Use non-sensitive synthetic data and isolated denied identities; do not paste production tokens into verification commands.

Alternatives

Extend observation before closure when the service's failure mode is delayed or asynchronous.

Stop conditions

Do not close the drift case on an empty plan alone.

instruction

Prevent recurrence and operate read-only detection

read-only

Record root cause and update access policy, emergency reconciliation, controller boundaries, module behavior, provider pinning, tests, state design, and alerts. Schedule locked read-only plans with no apply permission and review controls every 90 days.

Why this step matters

Drift often recurs because a manual path remains open, emergency changes lack HCL reconciliation, another controller has ambiguous ownership, or noisy provider behavior trained operators to ignore alerts. Prevention addresses that system rather than only restoring one value. The scheduled monitor remains read-only so a false positive, provider bug, or compromised runner cannot turn observation into infrastructure mutation.

What to understand

Restrict console and API writes, use short-lived roles, require change attribution, and alert on security-sensitive provider actions. Preserve break-glass access but automatically open a reconciliation case.

Fix provider normalization and module idempotence, add tests for the changed attribute, pin reviewed versions, and split states that are too broad for one owner to triage.

Route exit 0, 1, and 2 distinctly. Store a protected plan for 2, send a sanitized summary, and assign an SLA based on security and service impact. Never auto-apply.

Review lifecycle ignores, stale drift cases, artifact protection, recovery evidence, state locks, identities, provider support, and detection coverage every 90 days.

System changes

Reads provider and state on schedule; any prevention policy, tests, or access changes follow their own reviewed configuration lifecycle.

Syntax explained

-refresh-only: Observes remote-versus-state changes without proposing remote modification.
no apply permission: Keeps detection from becoming an unreviewed remediation channel.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

Root Path{{rootPath}}Repository path of the one Terraform root being observed and remediated.Refresh Plan{{refreshPlan}}Protected path for the refresh-only observation artifact.

terraform -chdir={{rootPath}} plan -refresh-only -detailed-exitcode -lock-timeout=5m -input=false -out={{refreshPlan}}

Example output / evidence

No changes. Your infrastructure still matches the configuration.
Drift monitor exit: 0
Last reconciliation: DRIFT-2026-0718
Next control review: 2026-10-22

Checkpoint: Detection is trustworthy and recurrence controls are owned

terraform -chdir={{rootPath}} plan -refresh-only -detailed-exitcode -lock-timeout=5m -input=false

Continue whenThe latest run exits 0 or has an owned triage case, the monitor cannot apply, and root-cause controls plus the next review date are recorded.

Stop whenThe monitor uses static administrator credentials, can apply, suppresses exit 2, overlaps writers without locks, or publishes sensitive output.

If this step fails

The scheduled drift plan exits 2 but the human-readable log says little.

Likely causeAutomation used `-detailed-exitcode`, correctly reporting a non-empty plan, while log truncation, redaction, or an incorrect expectation treated exit 2 as a generic failure.

Safe checks

terraform plan -refresh-only -detailed-exitcode -out=drift.tfplan
terraform show drift.tfplan
terraform show -json drift.tfplan
Inspect artifact retention and log truncation settings.

A targeted plan appears safe but the full plan later shows additional changes.

Likely causeUsing `-target` narrowed graph evaluation and hid dependencies or other drift. Targeting is an exceptional recovery tool, not a general drift-remediation workflow.

Safe checks

terraform plan -out=full.tfplan
terraform show full.tfplan
Compare targeted and complete dependency graphs.
Review why targeting was introduced.

Security notes

Limit the monitor to read and plan. State read is still privileged and must use protected identity, logs, and artifacts.

Alternatives

Use HCP Terraform health assessments with equivalent routing, artifact, identity, and no-auto-remediation controls.

Stop conditions

Disable a detector that reports false health because errors and non-empty plans are collapsed into one success status.

Finish line

Verification checklist

Full Terraform convergenceterraform -chdir={{rootPath}} plan -detailed-exitcode -lock-timeout=5mThe full graph exits 0 without targeting, refresh suppression, broad ignore rules, or unexplained provider warnings.

Read-only detectionterraform -chdir={{rootPath}} plan -refresh-only -detailed-exitcode -lock-timeout=5m -input=falseExit 0 is healthy, exit 1 creates an operational failure, exit 2 creates a drift case, and the runner has no apply permission.

Service and security behaviorRun the documented authorized synthetic and unauthorized negative checks for the affected service.Required behavior succeeds, unauthorized access fails, monitoring is healthy, and no exposed credential remains valid.

Audit and state evidence

Compare provider audit actions, state object versions, lock events, source commit, and saved-plan digest with {{driftCaseId}}.

Only approved identities and actions appear, the lock released, and a recoverable encrypted state version exists.

Recovery guidance

Common problems and safe checks

Terraform reports that the state lock is already held.

Likely causeAnother approved operation is active, an earlier process was interrupted, or an operator is pointing at the same state key from an unexpected checkout. A lock is evidence of serialization, not an inconvenience to bypass.

Safe checks

terraform workspace show
terraform version
ps -ef | grep '[t]erraform'
Review the remote runner and change-system activity for the exact backend key.

ResolutionIdentify the lock owner and wait for the active operation. If the owner confirms that no process can still write, follow the backend-specific recovery procedure and use `terraform force-unlock` only with the exact lock ID, a protected state backup, and a second reviewer. Never set `-lock=false` to make progress.

Initialization asks to migrate state when only a backend setting was expected to change.

Likely causeTerraform detected a different backend identity, key, workspace prefix, or partial configuration than the checkout previously initialized. Accepting the prompt without understanding the destination can split or overwrite state.

Safe checks

terraform version
git diff -- '*.tf' '*.hcl'
Inspect the non-secret backend configuration supplied by the approved runner.
List versioned state objects in the intended backend through the provider console.

ResolutionStop, compare the old and new backend coordinates, pull a protected backup from the known-good backend, and choose `-reconfigure` only for the same state location or `-migrate-state` only for a deliberate reviewed migration. Do not guess at an interactive prompt in production.

A plan unexpectedly proposes many creates after backend initialization.

Likely causeThe configuration selected an empty state key, wrong workspace, wrong account, or wrong region. Terraform therefore sees configuration but no bindings to existing objects.

Safe checks

terraform workspace show
terraform state list
terraform providers
Compare the backend key and state serial with the protected inventory.

A plan or state pull exposes a secret in a terminal, artifact, or log.

Likely causeState and plan files contain provider-returned attributes, and the `sensitive` marker controls display rather than guaranteeing omission from state. Debug logging or unredacted artifacts expanded the exposure.

Safe checks

Stop log and artifact publication.
Identify every principal and system that received the data.
Review whether the value is sensitive, ephemeral, or write-only in the configuration.
Preserve only access-controlled forensic evidence.

ResolutionTreat the value as compromised, rotate it at its authoritative system, revoke exposed artifacts, restrict state access, and redesign the resource to use ephemeral or write-only provider features where supported. Do not try to erase evidence by hand-editing state.

Terraform cannot decrypt or read the remote state object.

Likely causeThe runtime lost KMS permissions, the key was disabled or deleted, the object uses a different encryption context, or backend credentials select the wrong account.

Safe checks

terraform init -reconfigure
Verify the caller identity through the cloud provider's identity command.
Inspect KMS key status and backend audit logs without downloading state.
Confirm the expected bucket and key from the change record.

ResolutionRestore the approved identity or recover the encryption key according to provider policy. Do not create an empty replacement state. If the key is irrecoverable, escalate to state and infrastructure recovery using a versioned snapshot and a complete remote-object inventory.

A CI plan waits indefinitely or times out acquiring a lock.

Likely causeRuns are not serialized at the pipeline layer, the lock timeout is shorter than normal applies, or an abandoned operation still owns the backend lock.

Safe checks

Inspect queued and running jobs for the same state key.
Compare lock metadata with the runner identity and start time.
Review recent apply duration and cancellation events.
Confirm that no scheduled drift job overlaps the change window.

ResolutionSerialize operations per backend key, use a bounded `-lock-timeout` that matches operating expectations, cancel duplicate queued work, and investigate stale locks explicitly. A timeout should fail closed and alert; it must not retry with locking disabled.

The scheduled drift plan exits 2 but the human-readable log says little.

Likely causeAutomation used `-detailed-exitcode`, correctly reporting a non-empty plan, while log truncation, redaction, or an incorrect expectation treated exit 2 as a generic failure.

Safe checks

terraform plan -refresh-only -detailed-exitcode -out=drift.tfplan
terraform show drift.tfplan
terraform show -json drift.tfplan
Inspect artifact retention and log truncation settings.

A refresh-only plan is empty but a normal plan proposes changes.

Safe checks

terraform plan -refresh-only -out=refresh.tfplan
terraform plan -out=normal.tfplan
terraform show normal.tfplan
Compare configuration, state, and provider observations attribute by attribute.

Drift appears on every plan immediately after a successful apply.

Likely causeThe provider normalizes an argument, an API supplies unstable ordering or defaults, another controller owns the field, or module configuration writes a value the service transforms.

Safe checks

terraform plan -out=repeat.tfplan
terraform show repeat.tfplan
Read provider release notes and current resource schema.
Compare remote API values across two read-only observations.

A drift remediation plan proposes replacement of a critical object.

Safe checks

terraform show drift-remediation.tfplan
terraform show -json drift-remediation.tfplan
Review replacement paths and provider documentation.
Identify data, identity, endpoint, and dependency migration requirements.

Terraform detects a security control weakened outside code.

Likely causeAn emergency or manual change altered policy, network exposure, encryption, logging, identity, or retention, and the normal workflow was bypassed or failed to reconcile.

Safe checks

Preserve the refresh-only and normal plans.
Review provider audit logs and change identity.
Verify current exposure from an isolated authorized and denied client.
Check incident and emergency-change records.

A refresh-only apply updates state but the next plan proposes reverting the remote change.

Safe checks

terraform plan -refresh-only
terraform plan
git diff -- '*.tf' '*.tfvars'
Review the approved desired-state decision.

A lifecycle `ignore_changes` rule conceals a dangerous difference.

Likely causeThe rule was added broadly to silence noisy plans without naming the external owner or monitoring security-relevant attributes.

Safe checks

Review every lifecycle block.
Compare ignored attributes with provider audit and configuration policy.
Run a controlled plan after removing the rule in an isolated branch.
Identify the system that is supposed to own each ignored field.

The drift plan fails because provider credentials expired.

Safe checks

Verify caller identity without exposing credentials.
Review federation and runner token issuance logs.
terraform init -reconfigure -input=false
Run a read-only provider query for one known object.

A plan contains hundreds of unrelated drift changes.

Safe checks

terraform version
terraform workspace show
terraform providers
Compare backend key, provider alias, variable source, and recent bulk-change logs.

A drift alert disappears without a reviewed change.

Likely causeAnother refresh or apply rewrote state, the remote object changed again, the scheduled job used another workspace, or plan artifacts expired before triage.

Safe checks

Compare state object versions and serials.
Review provider audit logs across the alert window.
Compare scheduled-run backend and workspace metadata.
Recover the protected original drift plan artifact.

The team cannot decide whether remote reality or configuration is authoritative.

Safe checks

Identify service and configuration owners.
Review change, incident, and provider audit records.
Compare current behavior with SLO and security requirements.
List downstream systems that depend on each alternative.

A targeted plan appears safe but the full plan later shows additional changes.

Likely causeUsing `-target` narrowed graph evaluation and hid dependencies or other drift. Targeting is an exceptional recovery tool, not a general drift-remediation workflow.

Safe checks

terraform plan -out=full.tfplan
terraform show full.tfplan
Compare targeted and complete dependency graphs.
Review why targeting was introduced.

After the procedure

Alternatives and next steps

Consider these alternatives

Use HCP Terraform health assessments or drift detection when centralized workspace features fit, while preserving the same authority decision, no-auto-apply, sensitive-artifact, and independent verification rules.
Use provider-native configuration controls for emergency containment when Terraform plan latency is unacceptable, then open an immediate reconciliation case and encode the final decision in Terraform.
Use policy as code to block dangerous remediation actions, but retain human review because policy cannot determine every service-specific intent or acceptable replacement.
Split a state root when recurring plans are too large for one owner to review or unrelated teams repeatedly block each other's locks. State redesign is preferable to routine targeted plans.

Operate it safely

Create a drift taxonomy and dashboard covering root, provider, attribute class, actor, cause, time to triage, authority decision, replacement risk, and recurrence.
Add emergency-change automation that opens a reconciliation record and schedules a protected refresh-only and normal plan as soon as containment ends.
Reduce recurring noise by fixing provider normalization, unstable sets, missing configuration, and ambiguous external ownership rather than expanding ignores.
Add service-level synthetic and denied-access checks to remediation pipelines so an empty plan cannot conceal functional or security failure.
Review state boundaries and split roots whose full plans cannot be owned and reviewed within the operational objective.

Reference

Frequently asked questions

Is `terraform apply -refresh-only` a safe way to fix drift?

It can record approved remote observations in state without changing remote objects, but unchanged configuration may still propose reverting them. Use it only after review, and update configuration when reality is accepted.

Why run both refresh-only and normal plans?

Refresh-only isolates remote-versus-state observations; normal mode shows desired convergence against configuration. Their difference helps operators decide authority and prevents state acceptance from being mistaken for configuration reconciliation.

Can a drift system automatically apply low-risk changes?

This guide does not permit it. Provider replacement semantics, sensitive values, dependencies, external ownership, and service impact can make a small-looking diff dangerous. Automation should detect, classify, and route, not write.

When is `ignore_changes` acceptable?

Only for specific attributes legitimately owned by another named system, with documented boundaries and independent monitoring. It should not silence whole resources or security controls.

Recovery

Rollback

Rollback reverses the selected remediation, not the existence of drift evidence. Freeze new writes, preserve current and pre-remediation state versions and provider reality, restore the last safe configuration or approved remote value through a fresh full saved plan, and verify service plus complete convergence. Never overwrite live state with an older snapshot merely to silence a plan.

Stop the apply queue, identify the lock owner, preserve current state object version, source commit, refresh-only and normal plans, provider audit events, service metrics, and actual remote-object inventory.
If accepted-reality HCL is wrong but remote infrastructure remained safe, restore the prior configuration, generate a new full normal plan, and inspect every proposed action before applying.
If a restore-intent apply harmed service, use the documented service rollback or provider-native recovery, then update HCL to the safe accepted state and create a new full plan.
For necessary replacement, restore the previous endpoint, data, identity, and policy through its dedicated migration rollback; do not rely on state restoration to recreate lost provider objects.
Apply the reviewed rollback artifact under the state lock, run authorized and denied service checks, rotate exposed credentials, and require a complete normal plan exit 0.
Use historical state recovery only for confirmed state corruption after reconciling every remote change since that serial and testing the snapshot in an isolated key.

Evidence

Sources and review

Verified 2026-07-24Review due 2026-10-22

Terraform manage resource driftofficial Terraform plan command and refresh-only modeofficial Terraform refresh command guidanceofficial Terraform state overviewofficial Terraform remote stateofficial Terraform sensitive dataofficial Terraform resource lifecycleofficial