Configure Ansible inventory, SSH, roles, Vault, and idempotent deployments
Build a pinned ansible-core 2.21 control environment, authenticated SSH inventory, exact roles and collections, named Vault IDs, least-privilege become, check/diff gates, rolling deployment, idempotency proof, and rollback.
Operate repeatable configuration deployments in which target selection, machine identity, dependencies, privilege, secrets, predicted changes, real changes, health, convergence, and recovery are independently reviewable.
- ansible-core 2.21.x maintained line; exact patch pinned
- Control Python 3.12, 3.13, 3.14
- OpenSSH host-key verification enabled
- Supported isolated control node Python 3.12–3.14, exact hashed package lock, reviewed SSH client, clean plugin paths, and no operator-global Ansible dependency.
python --version && ansible --version && python -m pip check - Authenticated target identity Stable inventory aliases, trusted host fingerprints or SSH CA, private routing, non-root automation user, and exact task-level sudo policy.
- Protected automation repository Signed/reviewed Git revision, secret scanning, exact collection/role locks, protected release variables, and a documented owner.
- Secret and recovery services Named Vault password source, credential rotation, encrypted backups, previous artifacts/configuration, service health check, and out-of-band access.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A reproducible ansible-core 2.21 control environment with an exact Python dependency lock, pinned collections and roles, an explicit configuration file, and no dependency on an operator's global Ansible state.
- A production inventory design that supports reviewed static YAML and inventory plugins, separates identity from variables, validates SSH host keys, and makes host/group selection visible before a play can change anything.
- A least-privilege SSH and become path in which the login account is not root, privilege escalation is scoped to tasks that need it, secrets are protected with named Vault IDs, and password sources never enter Git or process arguments.
- An idempotent role and rolling deployment workflow with syntax, inventory, check/diff, serial, any_errors_fatal, health, second-run, audit, and rollback gates.
- The same locked control environment produces the same inventory graph, plugin set, role/collection versions, rendered templates, and task selection on every approved runner.
- Unknown or changed SSH host keys fail closed, Vault secrets remain encrypted at rest and redacted at execution, and ordinary tasks do not inherit unnecessary root authority.
- A single-host canary and bounded rolling batch complete with service-level checks; a second identical run reports changed=0; failures stop later batches and preserve usable hosts.
- Operators understand that check mode is a module-dependent simulation, diff can disclose secrets, dynamic inventory is live cloud data rather than a trusted source by itself, and rollback must be designed per change.
Architecture
How the parts fit together
The control repository stores playbooks, roles, inventory definitions, encrypted variable files, dependency locks, configuration, tests, and recovery instructions. A pinned Python virtual environment runs ansible-core 2.21, the maintained stable line as of 2026-07-28, with exact patch and collection versions selected by the organization. Static inventory describes stable host aliases and group structure; inventory plugins discover changing infrastructure through read-only APIs and compose approved groups. The SSH connection plugin reaches a non-root automation account and relies on OpenSSH known_hosts verification. Become is disabled by default and enabled only for narrowly selected tasks. Named Vault IDs separate environments and trust domains, while password clients obtain decryption material from an approved local secret service. Roles express desired state with idempotent modules and handlers. Syntax, inventory graph, check/diff, canary, rolling batches, post-change health, second-run idempotency, and rollback form separate gates.
- Pin and verify the control environment, configuration, collections, roles, and SSH client behavior before inventory is parsed.
- Build static or plugin inventory with allowlisted filters, inspect graph and host variables, then prove every selected host key and login identity.
- Encrypt environment secrets with named Vault IDs whose password clients use approved secret delivery and emit no secret output.
- Author roles with FQCN idempotent modules, handlers, explicit ownership and modes, task-level become, no_log around secret-bearing actions, and a documented rollback.
- Run syntax, inventory, lint/test, check and limited diff while acknowledging simulation gaps and confidentiality limits.
- Deploy a canary, then bounded batches with any_errors_fatal and health gates; stop on the first unsafe signal.
- Run the same play again, require changed=0, export redacted evidence, and rehearse restoration of the previous configuration/package/service state.
Assumptions
- The control node runs Python 3.12–3.14, which is the documented control-node range for ansible-core 2.21. The organization pins an exact supported 2.21.z build and reviews its changelog before rollout.
- Managed Linux hosts have a supported Python interpreter, a dedicated non-root automation account, stable DNS or inventory identity, and an approved sudoers policy. Password-based SSH and direct root login are not required.
- SSH host-key fingerprints are obtained from provisioning evidence, a trusted console, or a host certificate authority. Blind first-use acceptance is not a production trust model.
- Cloud/API inventory credentials are read-only and supplied by workload identity or an approved secret mechanism. Inventory discovery does not authorize configuration changes.
- The repository is protected by review, signed revisions, CI, and secret scanning. Vault-encrypted values may be committed, but Vault passwords, SSH private keys, tokens, and decrypted output are not.
- The managed service has a concrete health check, a previous known-good package/configuration artifact, and a rollback compatible with any state or schema changes.
Key concepts
- ansible-core versus the Ansible community package
- ansible-core supplies the execution engine, language, and builtin plugins. The community package adds many collections. This guide pins core and explicitly installs only required collections.
- Inventory source
- A static file or plugin configuration that produces hosts, groups, and variables. Inventory output is an execution input and must be reviewed like code.
- Host-key verification
- OpenSSH verifies that the endpoint presents the expected host public key. Disabling it protects neither credentials nor task integrity from a man-in-the-middle.
- Idempotency
- After desired state is reached, repeating the play produces no changes. It does not mean the first change is safe, reversible, or semantically correct.
- Vault ID
- A label associated with encrypted content and a password source, allowing environments or trust domains to use different decryption credentials.
- Check and diff mode
- Check asks modules to predict changes without applying them; unsupported modules may skip or mispredict. Diff shows supported before/after data and can disclose secrets.
- serial and any_errors_fatal
- serial limits each batch; any_errors_fatal stops the play across hosts after a fatal task failure. Together they limit, but do not eliminate, partial deployment.
Before you copy
Values used in this guide
{{ansibleCoreVersion}}Exact reviewed patch in the maintained ansible-core 2.21 line.
Example: 2.21.z pinned in requirements-control.txt{{inventoryPath}}Reviewed inventory directory or plugin source for the target environment.
Example: inventories/production{{targetGroup}}Explicit inventory group selected for this deployment.
Example: web_production{{canaryHost}}One healthy host selected from the target group for the first real run.
Example: web-prod-a{{automationUser}}Dedicated non-root SSH identity with reviewed task-specific sudo.
Example: ansible-deploy{{vaultId}}Named Vault trust domain used for the production encrypted variables.
Example: production{{vaultPasswordClient}}secretExecutable that reads a password from the approved secret service and writes only the password to stdout.
Example: /opt/automation/bin/vault-password-client{{releaseVersion}}Reviewed application or configuration release passed to the role.
Example: 2026.07.28-1Security and production boundaries
- Keep host-key checking enabled. Ansible credentials and become operations are valuable enough that a host-identity warning is a stop signal, not a nuisance.
- Do not put SSH private keys, Vault passwords, become passwords, API tokens, or secret values in inventory, ansible.cfg, command arguments, CI logs, facts, or callback output.
- Vault is encryption at rest, not a runtime sandbox. Use no_log for secret-bearing tasks, diff:false for secret templates, restrictive remote file modes, and short-lived credentials.
- Dynamic inventory plugins execute on the control node and use cloud credentials. Pin their collection, limit plugin enablement, filter discovery, and review the resulting graph.
- Task-level become is safer than play-wide become. A broad NOPASSWD sudo rule can turn any compromised module, template, or plugin into root execution.
- Collections and roles are executable supply-chain inputs. Pin versions and integrity in a controlled artifact mirror; review changelogs before update.
Stop before continuing if
- Stop if the resolved ansible-core or collection versions differ from the lock, the porting guide was not reviewed, or an unapproved plugin path is active.
- Stop if inventory selects an unexpected host, environment, account, region, or group, or if host variables contain plaintext secrets.
- Stop on an unknown or changed SSH host key until the new fingerprint is independently authenticated.
- Stop if the automation account can log in as root, has broad sudo, or a task requests become without a documented need.
- Stop if check/diff reveals secret material, reports an unreviewed destructive action, or skips a critical module whose real behavior has not been tested.
- Stop if a canary fails health, any host becomes unreachable, a batch is partially changed without recovery evidence, or the second run is not idempotent.
command
Pin ansible-core 2.21 and isolate the control environment
Create a dedicated virtual environment from an exact reviewed lock. ansible-core 2.21 is the current maintained GA line as of 2026-07-28; the lock must name the approved 2.21.z patch and hashes rather than floating on 2.21 or latest.
Why this step matters
Ansible loads code and configuration from Python packages, collections, plugin paths, environment variables, and the current directory. A global installation can silently select another module or callback. Exact hashes bind the reviewed packages, while ansible --version and ansible-config expose the effective executable and configuration. The displayed patch is an example of the approved lock; operators must choose the latest reviewed supported 2.21 patch available in their artifact system rather than copying a stale example.
What to understand
Review the 2.21 porting guide and maintained support matrix before changing a previous core line.
Store wheels in a controlled mirror and verify dependency hashes. Disable arbitrary extra indexes.
Run from a clean runner with ANSIBLE_CONFIG, plugin paths, and Python environment explicitly controlled.
System changes
- Creates an isolated local Python environment and installs pinned execution dependencies.
- Does not contact or modify managed hosts.
Syntax explained
--require-hashes- Requires every resolved Python artifact to match a lockfile hash.
ansible-config dump --only-changed- Shows non-default effective configuration that can alter behavior.
python3.13 -m venv .venv && . .venv/bin/activate && python -m pip install --require-hashes --requirement requirements-control.txt && ansible --version && ansible-config dump --only-changedansible [core 2.21.2] config file = /workspace/automation/ansible.cfg configured module search path = ['/workspace/automation/library'] ansible python module location = /workspace/automation/.venv/lib/python3.13/site-packages/ansible ansible collection location = /workspace/automation/collections executable location = /workspace/automation/.venv/bin/ansible python version = 3.13.5 CONFIG_FILE() = /workspace/automation/ansible.cfg
Checkpoint: Runner uses only reviewed core and configuration
. .venv/bin/activate && ansible --version && python -m pip checkContinue whenansible-core is the approved 2.21.z patch, Python is supported, dependencies are consistent, and config/plugin paths point only inside the project.
Stop whenAny version, hash, configuration file, plugin path, or package index differs from the lock and review.
If this step fails
ansible reports another configuration file.
Likely causeExecution started from another directory or ANSIBLE_CONFIG points globally.
Print ansible --version and the ANSIBLE_CONFIG variable.
ResolutionSet the reviewed project config explicitly and remove unapproved environment overrides.
Security notes
- Treat Python packages and callbacks as executable supply-chain inputs.
Alternatives
- Use a signed immutable execution-environment image built from the same exact locks.
Stop conditions
- Stop if a supported exact patch and integrity evidence are unavailable.
config
Set explicit safe Ansible and SSH defaults
Keep host-key checking enabled, disable retry files and world-readable temporary files, constrain inventory and collection paths, and choose predictable timeout, callback, and interpreter behavior. Do not place credentials in ansible.cfg.
Why this step matters
Configuration precedence is powerful enough to disable host identity, load plugins, expose arguments, or write sensitive temporary files. A project-local file makes intent reviewable. IdentitiesOnly prevents an SSH agent with many keys from trying unrelated identities. The project known_hosts file provides a reproducible trust set. Pipelining reduces remote temporary operations but requires a compatible sudo policy and must be tested. Inventory plugins are explicitly enabled so an unexpected installed plugin does not become a source.
What to understand
Check every environment variable in CI because environment precedence can override this file.
Keep callback output conservative; third-party callbacks can transmit or persist facts and module results.
Do not use host_key_checking=False or StrictHostKeyChecking=accept-new for production.
System changes
- Creates local project configuration only.
Syntax explained
enable_plugins- Allowlist of inventory plugin implementations Ansible may load.
IdentitiesOnly=yes- Limits SSH authentication to explicitly configured identities.
allow_world_readable_tmpfiles=False- Refuses insecure fallback for temporary module files.
ansible.cfg[defaults]
inventory = ./inventories/production
roles_path = ./roles
collections_path = ./collections
host_key_checking = True
retry_files_enabled = False
timeout = 15
interpreter_python = auto_silent
stdout_callback = default
display_args_to_stdout = False
allow_world_readable_tmpfiles = False
[inventory]
enable_plugins = host_list,script,auto,yaml,ini,toml,amazon.aws.aws_ec2
[ssh_connection]
pipelining = True
ssh_args = -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=./ssh/known_hostsANSIBLE_CONFIG=./ansible.cfg ansible-config view && ANSIBLE_CONFIG=./ansible.cfg ansible-config dump --only-changedDEFAULT_HOST_LIST(/workspace/automation/ansible.cfg) = ['/workspace/automation/inventories/production'] DEFAULT_ROLES_PATH(/workspace/automation/ansible.cfg) = ['/workspace/automation/roles'] COLLECTIONS_PATHS(/workspace/automation/ansible.cfg) = ['/workspace/automation/collections'] HOST_KEY_CHECKING(/workspace/automation/ansible.cfg) = True RETRY_FILES_ENABLED(/workspace/automation/ansible.cfg) = False DEFAULT_TIMEOUT(/workspace/automation/ansible.cfg) = 15
Checkpoint: Effective defaults fail closed
ANSIBLE_CONFIG=./ansible.cfg ansible-config dump | grep -E 'HOST_KEY_CHECKING|ALLOW_WORLD_READABLE_TMPFILES|COLLECTIONS_PATHS'Continue whenHost-key checking is true, world-readable temporary files are false, and paths are project-local.
Stop whenAny global config, callback, inventory plugin, or SSH option weakens the declared boundary.
If this step fails
Pipelining causes sudo failures.
Likely causesudo requiretty or remote policy conflicts with pipelined execution.
Test one non-mutating become command on a canary.
ResolutionCorrect the narrow sudo policy or disable pipelining explicitly; do not broaden sudo.
Security notes
- Review ansible-config dump in every runner; a committed file alone does not prove effective behavior.
Alternatives
- Use a centrally managed signed ansible.cfg embedded in an execution environment.
Stop conditions
- Stop if runner policy requires disabling host-key verification or unsafe temporary files.
verification
Authenticate host keys and prove the non-root SSH identity
Populate the project known_hosts file only from authenticated provisioning evidence or an SSH CA. Test the canary with a read-only ping and identity command. A changed key must be investigated, not automatically replaced.
Why this step matters
The SSH server key authenticates the machine before Ansible sends credentials or modules. DNS identity alone is insufficient. ansible.builtin.ping verifies Python/module transport, not ICMP. The command task is skipped in check mode because generic commands cannot predict safely; that skipped output is itself evidence of a check-mode limitation. A real id command can be approved separately because it is read-only.
What to understand
Prefer host certificates signed by a controlled SSH CA for autoscaled hosts.
Record hostname, address, key type, full fingerprint, source, and lifecycle event.
Use one identity file or agent key dedicated to the automation scope.
System changes
- Reads host trust and opens authenticated SSH sessions; ping transfers a temporary module.
Syntax explained
ssh-keygen -F- Looks up the exact host entry in the selected known_hosts file.
ansible.builtin.ping- Tests Ansible's authenticated connection and remote Python execution.
Values stay on this page and are never sent or saved.
ssh-keygen -F web-prod-a.example.net -f ./ssh/known_hosts && ansible {{canaryHost}} -i {{inventoryPath}} -m ansible.builtin.ping && ansible {{canaryHost}} -i {{inventoryPath}} -m ansible.builtin.command -a 'id' --check# Host web-prod-a.example.net found: line 4
web-prod-a.example.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIReviewedFingerprintMaterial
web-prod-a | SUCCESS => {
"changed": false,
"ping": "pong"
}
web-prod-a | SKIPPEDCheckpoint: Canary identity and host key are independently trusted
ansible {{canaryHost}} -i {{inventoryPath}} -m ansible.builtin.command -a 'id -un' -oContinue whenThe known host key matches provisioning evidence and the remote identity is exactly {{automationUser}}, not root.
Stop whenKey verification fails, DNS/address differs, agent offers an unrelated key, or remote identity is root.
If this step fails
ping succeeds against the wrong host alias.
Likely causeansible_host mapping or inventory precedence differs from review.
Run ansible-inventory --host for the alias and resolve DNS independently.
ResolutionCorrect inventory identity and reauthenticate the host key before any play.
Security notes
- Do not publish known_hosts if internal topology is confidential; protect it as operational metadata.
Alternatives
- Use an SSH host CA and @cert-authority known_hosts entry with controlled certificate issuance.
Stop conditions
- Stop on any host-key or remote-identity ambiguity.
config
Model stable hosts and groups in YAML inventory
Use YAML inventory beginning with all, stable aliases, explicit ansible_host values, and group variables that are non-secret. Separate environments into inventory directories so a limit cannot accidentally bridge production and staging.
Why this step matters
Inventory aliases are operational identities referenced by limits, facts, logs, and host_vars. Stable aliases should not be IP addresses that can be reassigned. YAML makes group structure reviewable and avoids INI type surprises. Inventory variables are high precedence inputs capable of changing users, interpreters, ports, paths, and role behavior; plaintext secrets do not belong there.
What to understand
Use one inventory root per environment and require its explicit path in deployment commands.
Keep connection variables minimal and move role defaults to roles.
Review duplicate host membership and variable precedence with --host output.
System changes
- Creates inventory source files only; inventory commands are read-only.
Syntax explained
--graph- Shows selected groups and host membership without executing a play.
--host- Displays merged variables for one host, which may contain sensitive metadata.
inventories/production/hosts.ymlall:
children:
production:
children:
web_production:
hosts:
web-prod-a:
ansible_host: web-prod-a.example.net
web-prod-b:
ansible_host: web-prod-b.example.net
vars:
ansible_user: ansible-deploy
environment: production
service_port: 8443Values stay on this page and are never sent or saved.
ansible-inventory -i {{inventoryPath}} --graph && ansible-inventory -i {{inventoryPath}} --host {{canaryHost}}@all:
|--@ungrouped:
|--@web_production:
| |--web-prod-a
| |--web-prod-b
|--@production:
| |--@web_production:
{
"ansible_host": "web-prod-a.example.net",
"ansible_user": "ansible-deploy",
"environment": "production",
"service_port": 8443
}Checkpoint: Static inventory contains only the intended production hosts
ansible-inventory -i {{inventoryPath}} --list | jq -r '._meta.hostvars | keys[]'Continue whenOnly reviewed stable aliases appear and every merged host variable has an approved non-secret source.
Stop whenUnexpected hosts, duplicate identities, plaintext secrets, or staging/production overlap appears.
If this step fails
A group variable has an unexpected type.
Likely causeINI/YAML parsing or extra-vars precedence changed the value.
Inspect ansible-inventory --host and the source hierarchy.
ResolutionUse YAML with explicit types and remove ambiguous higher-precedence overrides.
Security notes
- Host variable output can reveal internal addresses and usernames; redact evidence appropriately.
Alternatives
- Use an inventory plugin for ephemeral hosts while retaining static policy groups through constructed groups.
Stop conditions
- Stop if host selection cannot be enumerated before execution.
config
Constrain inventory plugins and compose approved groups
For elastic infrastructure, pin the provider collection and use read-only workload identity, explicit regions, account boundary, running-state and tag filters. The plugin output must be reviewed exactly like static inventory and cache age must be visible.
Why this step matters
A dynamic plugin queries a live control plane, so its result can change between preview and deployment. Region, account identity, state, and tags must constrain discovery. Tag-based authorization is safe only if tag mutation is separately controlled. strict:true turns undefined composition data into a visible failure instead of silently dropping identity. Private addresses keep management off public networks, but routing and SSH trust must still be proven.
What to understand
Verify the cloud caller identity separately and prohibit cross-account discovery.
Use collection FQCN and exact version; inventory plugins execute local Python code.
Export and hash the approved host list immediately before a change; re-evaluate if it changes.
System changes
- Queries cloud inventory read-only and writes a protected snapshot.
Syntax explained
filters- Server-side constraints reducing discovered instances.
strict: true- Fails on invalid composed values rather than silently proceeding.
private_ip_address- Uses internal management routing instead of public instance exposure.
inventories/production/aws_ec2.ymlplugin: amazon.aws.aws_ec2
regions:
- eu-central-1
filters:
instance-state-name: running
tag:Environment: production
tag:ManagedBy: ansible
hostnames:
- tag:AutomationAlias
- instance-id
keyed_groups:
- prefix: role
key: tags.Role
compose:
ansible_host: private_ip_address
strict: trueansible-inventory -i inventories/production/aws_ec2.yml --graph --vars && ansible-inventory -i inventories/production/aws_ec2.yml --list > /secure-staging/inventory.json && jq '._meta.hostvars | keys | length' /secure-staging/inventory.json@all: |--@aws_ec2: | |--i-0examplea | |--i-0exampleb |--@web_production: | |--i-0examplea | |--i-0exampleb 2
Checkpoint: Dynamic inventory snapshot matches account and change scope
sha256sum /secure-staging/inventory.json && jq -r '._meta.hostvars | to_entries[] | [.key,.value.tags.Environment,.value.tags.ManagedBy] | @tsv' /secure-staging/inventory.jsonContinue whenEvery host is running in the approved account/region, has controlled production/managed tags, and the snapshot digest is recorded.
Stop whenCredential identity, filter, tag ownership, cache age, host count, or private routing is uncertain.
If this step fails
Inventory changes between graph and play.
Likely causeInstances launched/terminated, tags changed, or cache and live queries differ.
Repeat inventory snapshot and compare IDs and cache metadata.
ResolutionFreeze the target set with an explicit --limit generated from the approved snapshot or reschedule after review.
Security notes
- Do not store cloud API keys in the plugin YAML. Prefer short-lived workload identity.
Alternatives
- Generate a signed static deployment target manifest from cloud inventory as a separate approval stage.
Stop conditions
- Stop when the target set changes after approval.
command
Install exact role and collection dependencies
Pin every collection and external role to an exact reviewed release or immutable commit in requirements.yml, install into project-local paths, and record the resolved list. Do not use floating branches or force upgrades during a deployment.
Why this step matters
Collections and roles are executable code running on the control node and managed hosts. Version ranges and branch names allow upstream changes without a repository review. Project-local installation makes resolution visible. Public registries are distribution channels, not sufficient organizational trust; mirror approved artifacts and retain provenance/digests where possible.
What to understand
Review module behavior, plugin loading, dependencies, changelog, license, maintainer, and release artifact.
Use FQCN in tasks so module resolution does not depend on collection search order.
Do not install dependencies from playbook runtime on production runners.
System changes
- Downloads and installs executable automation dependencies locally.
Syntax explained
-r requirements.yml- Uses the reviewed dependency declaration.
-p project path- Keeps dependencies out of operator-global locations.
ansible-galaxy collection install -r collections/requirements.yml -p ./collections && ansible-galaxy role install -r roles/requirements.yml -p ./roles/external && ansible-galaxy collection list && find roles/external -maxdepth 2 -name .git -prune -o -type f -name meta.yml -printStarting galaxy collection install process amazon.aws:10.1.2 was installed successfully community.crypto:3.0.4 was installed successfully - downloading role 'hardening', owned by example - extracting example.hardening to /workspace/automation/roles/external/example.hardening - example.hardening (7.4.1) was installed successfully
Checkpoint: Resolved dependencies equal the approved lock
ansible-galaxy collection list --format json > /tmp/collections.json && sha256sum collections/requirements.yml roles/requirements.yml /tmp/collections.jsonContinue whenEvery resolved collection and role exactly matches review and the digests are recorded.
Stop whenA version floats, dependency appears transitively without review, or code resolves outside project paths.
If this step fails
Galaxy installs a newer dependency than expected.
Likely causeA range, indirect dependency, pre-release, or existing global installation influenced resolution.
Inspect requirements and collection list in a clean environment.
ResolutionPin exact compatible versions and rebuild from an empty project-local path.
Security notes
- Scan role/collection source for command execution, lookups, callbacks, and network access.
Alternatives
- Vendor reviewed collections and roles in a signed internal artifact repository.
Stop conditions
- Stop if integrity or maintainership of executable dependencies cannot be established.
config
Author a state-driven role with handlers and rollback inputs
Use FQCN modules, explicit ownership/modes, validated templates, handlers only on real change, and immutable release artifacts. Avoid shell and command where a state-aware module exists. Preserve the previous configuration before replacement.
Why this step matters
State-aware modules compare desired and actual state and report changed only when necessary. A validation command is acceptable because it performs a tool-specific read-only check and changed_when:false states that contract. Rendering to a candidate path prevents an invalid file from replacing the live configuration. backup:true supplies a host-local restoration input, but backups require retention, permissions, and an off-host strategy. Handlers restart only after real changes and should listen to one stable notification.
What to understand
Pin package versions and artifact repositories; latest undermines repeatability.
Use validate parameters where modules support them; otherwise stage and check explicitly.
Add argument specs and role defaults so invalid input fails before task mutation.
System changes
- Syntax/list commands are read-only; the role later installs a package, writes configuration, and manages a service.
Syntax explained
argv- Avoids shell parsing for the validation command.
changed_when: false- Declares that successful validation observes rather than mutates state.
backup: true- Creates a timestamped remote copy before replacing an existing destination.
roles/web/tasks/main.ymlValues stay on this page and are never sent or saved.
---
- name: Install reviewed package
ansible.builtin.package:
name: "example-web={{ release_version }}"
state: present
become: true
- name: Render candidate configuration
ansible.builtin.template:
src: web.conf.j2
dest: /etc/example-web/web.conf.candidate
owner: root
group: root
mode: "0640"
become: true
notify: Restart example web
- name: Validate candidate configuration
ansible.builtin.command:
argv: [/usr/bin/example-web, --check, /etc/example-web/web.conf.candidate]
changed_when: false
become: true
- name: Deploy configuration
ansible.builtin.copy:
remote_src: true
src: /etc/example-web/web.conf.candidate
dest: /etc/example-web/web.conf
owner: root
group: root
mode: "0640"
backup: true
become: true
notify: Restart example web
- name: Enable and start service
ansible.builtin.service:
name: example-web
enabled: true
state: started
become: trueValues stay on this page and are never sent or saved.
ansible-playbook site.yml -i {{inventoryPath}} --syntax-check --list-tasks --tags webplaybook: site.yml
play #1 (web_production): Deploy reviewed web configuration TAGS: []
tasks:
web : Install reviewed package
web : Validate candidate configuration
web : Deploy configuration
web : Enable and start service
web : Verify local healthCheckpoint: Role task graph is state-driven and reversible
ansible-playbook site.yml -i {{inventoryPath}} --syntax-check && grep -R -nE 'ansible\.builtin\.(shell|command)|become: true|backup: true' roles/webContinue whenSyntax passes; every command and become has a reviewed reason; configuration replacement retains recovery evidence.
Stop whenRole uses unguarded shell, floating packages, unconditional restarts, secret output, or no rollback input.
If this step fails
Template changes every run.
Likely causeTimestamp, unordered data, host-specific nondeterminism, or volatile fact enters content.
Render twice with identical inputs and compare protected outputs.
ResolutionRemove volatile data, sort structures, and move runtime metadata outside managed configuration.
Security notes
- Backups can contain secrets; set permissions, retention, and no_log/diff controls.
Alternatives
- Deploy an immutable machine/container image and use Ansible only for orchestration and verification.
Stop conditions
- Stop if desired state cannot be validated before replacement.
verification
Constrain become to exact privileged tasks
Keep the play non-privileged and set become:true only on tasks requiring root-owned packages, files, or services. Verify the automation account's sudo list and deny interactive shells, arbitrary editors, package managers, and unrestricted command execution.
Why this step matters
Play-wide become makes every module, lookup result, and task root-capable. Task-level escalation exposes the privileged boundary during review. sudo command allowlists can still be unsafe when arguments permit alternate configuration, shell escapes, plugins, or arbitrary paths, so the actual executable behavior matters. Package and file modules may require broader implementation commands than a simplistic sudo list; test through the final role rather than granting ALL.
What to understand
Use a dedicated become method/user and disable become flags supplied through uncontrolled inventory.
Prefer root-owned staging directories not writable by the SSH user.
Audit sudo invocations and alert on commands outside deployment windows.
System changes
- Reads sudo authorization and play task list only.
Syntax explained
sudo -n -l- Lists non-interactive sudo authorization without prompting for a password.
task-level become- Escalates only the specific state transition requiring privilege.
Values stay on this page and are never sent or saved.
ansible {{canaryHost}} -i {{inventoryPath}} -m ansible.builtin.command -a 'sudo -n -l' -o && ansible-playbook site.yml -i {{inventoryPath}} --list-tasks | grep -n .web-prod-a | SUCCESS | rc=0 >>
User ansible-deploy may run the following commands on web-prod-a:
(root) NOPASSWD: /usr/bin/systemctl status example-web,
/usr/bin/systemctl restart example-web,
/usr/bin/example-web --check *
1:playbook: site.yml
2: play #1 (web_production): Deploy reviewed web configurationCheckpoint: SSH identity cannot obtain a general root shell
ansible {{canaryHost}} -i {{inventoryPath}} -m ansible.builtin.command -a 'sudo -n /bin/sh -c id' -oContinue whenThe controlled negative test is denied and approved role tasks succeed through exact escalation.
Stop whenAutomation account has sudo ALL, shell/editor escape, writable privileged executable, or become credentials appear in output.
If this step fails
Role needs broader sudo than the documented tasks.
Likely causeModule implementation, package backend, temp path, or handler executes additional commands.
Test on a disposable host and inspect sudo audit logs.
ResolutionRedesign the role or add exact safe commands; never grant blanket access to unblock a run.
Security notes
- Arguments and writable files can turn an apparently narrow sudo rule into arbitrary execution; threat-model each command.
Alternatives
- Use a root-owned local agent or immutable image pipeline when safe sudo decomposition is impractical.
Stop conditions
- Stop if least-privilege escalation cannot support the role safely.
config
Separate Vault IDs and protect password sources
Encrypt only secret variables with a named production Vault ID. Configure the password client outside Git, owned by the runner identity with mode 0700, retrieving a short-lived value from the approved secret system and writing no diagnostics to stdout.
Why this step matters
Named IDs prevent one password from silently decrypting every environment and make rotation scope visible. The password client path can be committed as configuration, but the client implementation and secret source need their own review. The password must be the only stdout content because Ansible consumes stdout as credential material. Vault ciphertext can be committed; decrypted files and password sources cannot. encrypt_string is useful for individual values, but editing whole encrypted files can reduce review clarity.
What to understand
Use separate IDs for production, staging, and independent administrative domains.
Mark secret tasks no_log:true and diff:false; Vault automatically decrypts before module execution.
Test rekey and recovery with a synthetic file before rotating production.
System changes
- Encrypts the selected local variables file in place.
Syntax explained
--vault-id label@source- Associates ciphertext with a named identity and password provider.
Vault 1.2 header- Carries the Vault ID label in encrypted-file metadata, not the secret.
inventories/production/group_vars/web_production/vault.yml$ANSIBLE_VAULT;1.2;AES256;production
REDACTED_CIPHERTEXT_CREATED_BY_ANSIBLE_VAULTValues stay on this page and are never sent or saved.
ansible-vault encrypt --vault-id {{vaultId}}@{{vaultPasswordClient}} inventories/production/group_vars/web_production/vault.yml && head -n 1 inventories/production/group_vars/web_production/vault.yml && stat -c '%a %U %n' {{vaultPasswordClient}}$ANSIBLE_VAULT;1.2;AES256;production 700 automation /opt/automation/bin/vault-password-client
Checkpoint: Only ciphertext and non-secret labels are stored
git diff --check -- inventories/production/group_vars/web_production/vault.yml && grep -R -nE 'password|token|secret' inventories/production --exclude=vault.yml || trueContinue whenVault file has the expected production header, password client permissions are restrictive, and no plaintext secret is found.
Stop whenPassword source enters Git, process arguments, logs, shell history, world-readable storage, or a shared Vault ID crosses trust domains.
If this step fails
Vault password client emits an error plus the password.
Likely causeDiagnostics are written to stdout rather than stderr or logging is enabled.
Test with synthetic credential and separately capture stdout/stderr in an isolated runner.
ResolutionMake stdout contain only the password, suppress secret logging, and rotate if output was retained.
Security notes
- Vault passwords should be retrieved just in time and never exported as long-lived environment variables.
Alternatives
- Use a pinned external secret lookup plugin returning short-lived credentials at task time.
Stop conditions
- Stop if secret decryption cannot occur without durable plaintext.
verification
Validate syntax, inventory, variables, tags, and host scope
Run syntax checks, list hosts and tasks, render the exact inventory snapshot, and confirm the target limit before check mode. These gates catch parsing and selection errors without assuming module simulation can prove behavior.
Why this step matters
A syntactically valid play can still target the wrong environment or include an unexpected role. Listing hosts and tasks transforms implicit selection into review evidence. Tags can skip prerequisites and handlers, so their use must be designed, not improvised. Extra variables have very high precedence and should not carry unreviewed host, user, become, path, or secret overrides.
What to understand
Record the exact command, Git revision, inventory digest, dependency list, and runner identity.
Compare selected hosts with maintenance and load-balancer state.
Reject empty groups; an empty successful run is not deployment evidence.
System changes
- Parses playbooks and inventory only.
Syntax explained
--list-hosts- Enumerates the final host set after patterns and limits.
--list-tasks- Shows the selected task graph without executing it.
Values stay on this page and are never sent or saved.
ansible-playbook site.yml -i {{inventoryPath}} --syntax-check && ansible-playbook site.yml -i {{inventoryPath}} --list-hosts --limit {{targetGroup}} && ansible-playbook site.yml -i {{inventoryPath}} --list-tasks --tags webplaybook: site.yml
play #1 (web_production): Deploy reviewed web configuration
pattern: ['web_production']
hosts (2):
web-prod-a
web-prod-b
tasks:
web : Install reviewed package
web : Validate candidate configuration
web : Deploy configuration
web : Enable and start service
web : Verify local healthCheckpoint: Execution graph exactly matches approval
ansible-playbook site.yml -i {{inventoryPath}} --list-hosts --limit {{targetGroup}} | tee /secure-staging/approved-hosts.txtContinue whenNon-empty host list, expected task order, approved tags, and no extra-vars that alter trust or privilege.
Stop whenSelection is empty, broad, changed since inventory approval, or tasks differ from review.
If this step fails
List-hosts returns zero hosts.
Likely causeGroup name, plugin cache/filter, inventory path, or limit is wrong.
Inspect inventory graph and exact case-sensitive group names.
ResolutionCorrect inventory/limit and repeat every preview gate; do not remove the limit blindly.
Security notes
- Treat extra-vars files as code and secrets; command-line values are process-visible.
Alternatives
- Generate a signed deployment manifest containing the exact host IDs and playbook revision.
Stop conditions
- Stop if execution scope cannot be reproduced exactly.
verification
Use check and diff as partial evidence, not a guarantee
Run check mode on the canary, enable diff only after secret-bearing tasks set diff:false and no_log:true, and inspect skipped/unsupported tasks. Check mode predicts supported module changes but cannot exercise every condition, command, handler, or runtime dependency.
Why this step matters
The output shows both value and limitation: supported package/template modules predict change, while a generic validation command skips. Conditions based on registered results may also differ because prior tasks did not run. Diff can expose complete before/after files, so secret templates must disable it and the log must be treated as configuration evidence. The canary limit reduces disclosure and load. A clean check is required but not sufficient; disposable integration and a real canary remain necessary.
What to understand
Classify every skipped task and every module's check/diff support.
Review predicted package versions, file paths, ownership, modes, handler notifications, and deletions.
Redact only copies of logs, never mutate the original evidence without recording transformation.
System changes
- Connects to the canary and asks modules to simulate; supported modules should not persist changes.
Syntax explained
--check- Requests module-specific change prediction without normal mutation.
--diff- Displays supported before/after content and can disclose sensitive values.
--limit canary- Restricts simulation and output to one approved host.
Values stay on this page and are never sent or saved.
ansible-playbook site.yml -i {{inventoryPath}} --limit {{canaryHost}} --check --diff --vault-id {{vaultId}}@{{vaultPasswordClient}} | tee /secure-staging/check-redacted.logPLAY [Deploy reviewed web configuration] TASK [web : Install reviewed package] changed: [web-prod-a] TASK [web : Render candidate configuration] changed: [web-prod-a] --- before: /etc/example-web/web.conf +++ after: /home/ansible-deploy/.ansible/tmp/source TASK [web : Validate candidate configuration] skipping: [web-prod-a] PLAY RECAP web-prod-a : ok=5 changed=2 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
Checkpoint: Predicted changes and simulation gaps are approved
grep -E 'changed:|skipping:|failed:|unreachable:' /secure-staging/check-redacted.logContinue whenOnly expected tasks predict change; every skipped task has a real-run test; no secret or destructive change appears.
Stop whenSecret output, unexpected deletion/restart, unsupported critical task, failure, unreachable host, or changed target scope appears.
If this step fails
A check-mode task contacts or changes an external system.
Likely causeModule/action plugin lacks safe check behavior or task forces check_mode:false.
Inspect module attributes and task-level check_mode settings.
ResolutionRemove forced execution, mock the dependency in tests, and exclude the task from production preview until behavior is safe.
Security notes
- Never upload raw diff logs to public CI artifacts or support tickets.
Alternatives
- Run full normal mode against disposable instances created from the production image.
Stop conditions
- Stop when preview cannot bound a critical task's effect.
command
Deploy one canary and require service health
Run the normal play against one approved host with the locked environment and Vault ID. Keep it out of load-balancer service if required, then verify local configuration, process, port, logs, and an application-level transaction before widening.
Why this step matters
A real canary exercises package managers, commands, handlers, sudo, service managers, filesystems, certificates, and runtime dependencies that check mode cannot model. One healthy host limits blast radius but can still affect shared databases or control planes, so role effects must already be bounded. A 200 response is one signal; validate configuration, process stability, logs, metrics, dependencies, and a safe user transaction over an acceptance interval.
What to understand
Record pre-change package/config/service state and load-balancer membership.
Use an immutable release variables file, not ad-hoc command-line overrides.
Do not widen until restart counts, latency, errors, and dependencies remain accepted.
System changes
- Applies the reviewed role to one production host and may restart the service.
Syntax explained
--limit canaryHost- Constricts mutation to exactly one reviewed inventory alias.
-e @release file- Loads a reviewed versioned variable file; extra-vars still have high precedence.
Values stay on this page and are never sent or saved.
ansible-playbook site.yml -i {{inventoryPath}} --limit {{canaryHost}} --vault-id {{vaultId}}@{{vaultPasswordClient}} -e @releases/{{releaseVersion}}.yml && ansible {{canaryHost}} -i {{inventoryPath}} -m ansible.builtin.uri -a 'url=https://127.0.0.1:8443/healthz validate_certs=true return_content=false status_code=200'PLAY RECAP
web-prod-a : ok=14 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
web-prod-a | SUCCESS => {
"changed": false,
"status": 200,
"server": "example-web"
}Checkpoint: Canary is stable through the acceptance interval
ansible {{canaryHost}} -i {{inventoryPath}} -m ansible.builtin.command -a 'systemctl is-active example-web' -oContinue whenPlay has zero failed/unreachable/ignored tasks, health transaction passes, service remains active, and monitoring shows no regression.
Stop whenAny failure, unreachable state, unexpected changed task, degraded metric, shared-state mutation, or health instability occurs.
If this step fails
Play succeeds but health check fails.
Likely causeService state module sees a running process while configuration, dependency, certificate, port, or application readiness is wrong.
Read bounded service status, listener, journal, and dependency health.
ResolutionStop rollout and execute the documented host rollback or signed forward fix.
Security notes
- Use no_log on secret tasks; the normal run handles decrypted data and privileged operations.
Alternatives
- Canary in a production-identical shadow pool before touching a live host.
Stop conditions
- Stop on any canary acceptance failure.
command
Roll through bounded batches and stop globally on failure
Set serial to an accepted count or percentage and any_errors_fatal:true at the play level. Drain, change, verify, and return each batch before the next begins. Do not combine a large fork count with an unbounded host pattern.
Why this step matters
serial creates multiple play batches; any_errors_fatal prevents later batches after a fatal error across the current operation. The batch size must preserve capacity and failure-domain diversity. A host can be changed before another host fails, so this is not atomic deployment. Drain/add operations should be delegated to a controlled endpoint and verified. rescue and ignore_errors can defeat the intended stop behavior and need explicit review.
What to understand
Calculate healthy capacity after removing one entire batch and one additional failure.
Use max_fail_percentage only with precise semantics and tests; any_errors_fatal is clearer for small critical fleets.
Keep forks at or below operational API/service limits and preserve ordered evidence per host.
System changes
- Applies the role across approved production hosts in bounded sequential batches.
Syntax explained
serial- Limits each batch; can be count, percentage, or list of staged batch sizes.
any_errors_fatal- Stops the play for all hosts after a fatal task failure is handled for the current batch.
Values stay on this page and are never sent or saved.
ansible-playbook site.yml -i {{inventoryPath}} --limit {{targetGroup}} --vault-id {{vaultId}}@{{vaultPasswordClient}} -e @releases/{{releaseVersion}}.ymlPLAY [Deploy reviewed web configuration] 1/2 PLAY RECAP web-prod-a : ok=12 changed=0 unreachable=0 failed=0 PLAY [Deploy reviewed web configuration] 2/2 PLAY RECAP web-prod-b : ok=14 changed=3 unreachable=0 failed=0 Deployment gate: 2/2 healthy; no fatal errors
Checkpoint: Every completed host is healthy and no later batch starts after failure
ansible {{targetGroup}} -i {{inventoryPath}} -m ansible.builtin.uri -a 'url=https://127.0.0.1:8443/healthz validate_certs=true status_code=200' -oContinue whenAll changed hosts pass health, capacity remains accepted, and play recap has zero failed, unreachable, rescued, or ignored surprises.
Stop whenAny host fails/unreachable, later batch begins after a fatal error, capacity drops, or service indicators regress.
If this step fails
Partial batch changed before a fatal failure.
Likely causeAnsible applies tasks host-by-host and is not a transaction.
Use recap and per-host task results to identify exact state.
ResolutionKeep later batches stopped; restore or forward-fix changed hosts according to the per-host rollback plan.
Security notes
- Delegated load-balancer tasks must use separate narrow credentials and no_log.
Alternatives
- Use blue/green infrastructure and switch traffic after whole-pool verification.
Stop conditions
- Stop on the first host or service failure; do not raise thresholds during an incident.
verification
Run the identical play again and require changed=0
Repeat the exact play, inventory snapshot, release variables, dependency lock, Vault ID, and target group. A clean second run must report changed=0 on every host and retain the service health established after the first run.
Why this step matters
A second run is the most direct idempotency test of actual module, template, package, service, and host behavior. changed=0 is required but not sufficient: a role can converge to the wrong stable state. Correlate with health and reviewed desired values. Preserve a redacted log, Git SHA, inventory digest, lock digests, runner identity, timestamps, and release version. Changes in facts, inventory, upstream packages, or secrets between runs invalidate the comparison.
What to understand
Run soon enough that external inputs remain stable and record any unavoidable drift.
Investigate every changed task; do not add changed_when:false merely to silence it.
Check that secret redaction and log permissions remain effective.
System changes
- Executes the same role again; correct idempotent tasks should persist no changes.
Syntax explained
changed=0- Observed convergence for the exact second-run inputs, not a universal proof.
rescued/ignored=0- Ensures error-handling did not conceal a failure in the evidence.
Values stay on this page and are never sent or saved.
ansible-playbook site.yml -i {{inventoryPath}} --limit {{targetGroup}} --vault-id {{vaultId}}@{{vaultPasswordClient}} -e @releases/{{releaseVersion}}.yml | tee /secure-staging/idempotency-redacted.logPLAY RECAP web-prod-a : ok=14 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 web-prod-b : ok=14 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Checkpoint: Actual production state is converged and healthy
grep -E 'changed=[1-9]|unreachable=[1-9]|failed=[1-9]|rescued=[1-9]|ignored=[1-9]' /secure-staging/idempotency-redacted.log || trueContinue whenNo output, changed=0 for every approved host, and service-level checks remain healthy.
Stop whenAny change or concealed failure appears, or inputs differ from the first run.
If this step fails
Only service tasks report changed every run.
Likely causeRole uses restarted rather than started, handler is unconditional, or module/platform reports state inaccurately.
Inspect notifications and service manager state before/after.
ResolutionUse started plus change-triggered handler and fix task state reporting; retest twice.
Security notes
- Store redacted evidence in an access-controlled change system and delete raw secret-bearing logs.
Alternatives
- Run automated idempotency tests in CI and retain production second-run verification for critical roles.
Stop conditions
- Do not declare completion without changed=0 or an explicitly approved documented exception.
warning
Rehearse host rollback and preserve partial-state evidence
Rollback is role-specific: drain the affected host, stop later batches, restore the reviewed package and configuration backup, validate, restart, verify, and return traffic. Never assume rerunning an older role can reverse data migrations or deleted state.
Why this step matters
Ansible is not transactional. A failed play can leave package, file, service, load-balancer, and remote-system states at different points. A rollback play must identify pre-change artifacts, understand schema compatibility, and be independently tested. Preview is useful but inherits check/diff limitations. The canary rollback drill uses a reversible release; production incidents may require a forward fix instead. Preserve first-failure logs before reruns or handlers overwrite evidence.
What to understand
Record exact hosts and completed tasks from recap and callback evidence.
Verify previous package/artifact signatures and configuration backup permissions before restore.
Keep traffic drained until local and external health plus monitoring pass.
System changes
- The shown command previews rollback; a real approved rollback restores package/configuration and restarts the service.
Syntax explained
rollback.yml- Separate reviewed recovery workflow, not an improvised reverse task list.
--limit canaryHost- Prevents a rollback drill from expanding beyond one approved host.
Values stay on this page and are never sent or saved.
ansible-playbook rollback.yml -i {{inventoryPath}} --limit {{canaryHost}} --vault-id {{vaultId}}@{{vaultPasswordClient}} -e @releases/previous-reviewed.yml --check --diffPLAY [Rollback reviewed web release] TASK [Verify retained package artifact] ok: [web-prod-a] TASK [Preview previous configuration restore] changed: [web-prod-a] TASK [Preview service restart] changed: [web-prod-a] PLAY RECAP web-prod-a : ok=6 changed=2 unreachable=0 failed=0 skipped=0
Checkpoint: Rollback restores a known-good service without widening impact
ansible {{canaryHost}} -i {{inventoryPath}} -m ansible.builtin.uri -a 'url=https://127.0.0.1:8443/healthz validate_certs=true status_code=200' -oContinue whenPrevious reviewed version/configuration is active, health and monitoring pass, host returns to service deliberately, and evidence identifies every changed state.
Stop whenRollback artifact is unavailable/untrusted, schema is incompatible, backup contains unknown secrets, or health remains degraded.
If this step fails
Previous package cannot start with current data.
Likely causeForward-only schema or state migration crossed the rollback boundary.
Read migration ledger and compatibility documentation before changing binaries.
ResolutionKeep traffic isolated and use the approved data recovery or signed forward-fix procedure.
Security notes
- Rollback credentials and backups are sensitive; access them only under audited incident authority.
Alternatives
- Deploy a fixed immutable image or switch to a known-good blue/green pool.
Stop conditions
- Stop rollback at the first incompatible state or failed health checkpoint.
Finish line
Verification checklist
ansible --version && ansible-galaxy collection listCore, Python, config, collection, and role versions exactly match review.ansible-inventory -i {{inventoryPath}} --graphOnly approved groups and hosts appear in the selected production inventory.ansible {{canaryHost}} -i {{inventoryPath}} -m ansible.builtin.command -a 'id -un' -oSSH host key is authenticated and remote identity is {{automationUser}}, never root.ansible-playbook site.yml -i {{inventoryPath}} --limit {{targetGroup}} --vault-id {{vaultId}}@{{vaultPasswordClient}} -e @releases/{{releaseVersion}}.ymlSecond identical real run reports changed=0, unreachable=0, failed=0, rescued=0, and ignored=0 for every host.ansible {{targetGroup}} -i {{inventoryPath}} -m ansible.builtin.uri -a 'url=https://127.0.0.1:8443/healthz validate_certs=true status_code=200' -oEvery host and external service path satisfies the reviewed acceptance checks.Recovery guidance
Common problems and safe checks
ansible reports a host-key mismatch.
Likely causeThe host was rebuilt, DNS/IP was reused, a load balancer is being targeted, or the connection is intercepted.
Compare the presented fingerprint with trusted provisioning or console evidence.Confirm inventory_hostname and ansible_host resolve to the intended server.
ResolutionReplace the known_hosts entry only after independent authentication and document the lifecycle event. Never disable host-key checking.
Inventory graph contains hosts from another environment.
Likely causePlugin filters, credentials, regions, compose rules, or cached inventory are too broad.
Inspect ansible-inventory --graph and --host for affected aliases.Run the cloud query with the same read-only identity and filters.
ResolutionTighten account, region, tag, state, and group filters; clear only the reviewed cache; repeat graph approval.
A Vault file cannot be decrypted.
Likely causeWrong Vault ID, password source, file label, rotated password, or corrupted ciphertext.
Inspect only the Vault header and file permissions.Test the password client in an isolated process with stdout redirected securely.
ResolutionRestore the correct named password source or rekey from an authorized backup. Do not paste the password into a command or commit a decrypted copy.
Check mode reports no change but the real run changes hosts.
Likely causeA module lacks check support, a command task is skipped, or registered runtime data changes control flow.
Review module check_mode support and skipped tasks.Run a disposable integration target or canary with normal mode.
ResolutionReplace command/shell with an idempotent module or add explicit changed_when/check behavior backed by tests; keep canary gating.
Diff output exposes a password, token, or private configuration.
Likely causeA secret-bearing template/task permits diff or a callback logs module arguments.
Stop log publication and identify every copy of the output.Inspect task diff/no_log settings without repeating the secret.
ResolutionRotate the exposed value, remove retained output under incident policy, set diff:false and no_log:true, and retest with synthetic values.
The second run reports changed tasks.
Likely causeNon-idempotent command use, unstable template ordering, timestamp content, unconditional handlers, or service modules that always report change.
Run with task-level verbosity on a disposable host and compare registered results.Inspect changed_when, creates/removes, template inputs, and handler notifications.
ResolutionUse a state-aware module or guard, stabilize inputs, and require changed=0 before widening deployment.
Become fails only for one task.
Likely causeThe task's executable/path differs from the exact sudoers command, the remote temp path is inaccessible, or become user is wrong.
Run ansible.builtin.command id without become, then with the reviewed become task.Inspect sudo logs and sudo -l through an approved interactive session.
ResolutionNarrowly correct the role or sudoers command. Do not grant broad NOPASSWD or switch the whole play to root.
A handler does not run after configuration changes.
Likely causeThe task did not report changed, notified a different name, failed before handlers, or a later fatal error prevented execution.
Inspect task changed status, notification names, and play recap.Use --list-tasks and review handler definitions.
ResolutionFix state reporting and notification wiring; use force_handlers only when restarting after another failure is demonstrably safe.
One host fails and later batches still start.
Likely causeany_errors_fatal is absent, failure was ignored/rescued, or the play structure reset failure handling.
Inspect play keywords, blocks, rescue, ignore_errors, and recap.Reproduce on a staging group with a controlled failure.
ResolutionSet any_errors_fatal at the correct play/block, remove unsafe ignores, and test that later batches remain untouched.
A pinned collection resolves to another version.
Likely causeAn operator-global collection path wins, the lock is a range, or installation used force/pre-release options.
Run ansible-galaxy collection list and ansible-config dump --only-changed.Inspect ANSIBLE_COLLECTIONS_PATH and the isolated environment.
ResolutionRebuild the isolated environment from exact locks and remove global paths from configuration.
Hosts become unreachable mid-batch.
Likely causeSSH/network configuration, firewall, service restart, load, or host replacement interrupted the control channel.
Stop later batches and use out-of-band health/console evidence.Compare the last successful task and network changes.
ResolutionUse the documented out-of-band rollback, restore connectivity, authenticate any new host key, then resume only with explicit limits.
Rollback restores a file but the service remains unhealthy.
Likely causePackage, schema, runtime state, environment, ownership, or dependent service also changed.
Compare the full release manifest and captured pre-change state.Read bounded service status, journal, and dependency health.
ResolutionRestore every coupled state component or use a signed forward fix. Do not repeatedly restart a service with incompatible state.
Recovery
Rollback
Stop later batches, preserve per-host evidence, drain affected hosts, and choose a tested role-specific restore or forward fix. Restore only authenticated previous packages/configuration, respect schema compatibility, validate before replacement, and return traffic only after local and external health. Ansible cannot make a multi-host deployment atomic.
- Freeze deployment and export the exact inventory snapshot, recap, completed tasks, release variables, runner/dependency digests, service health, and redacted logs.
- Keep unhealthy or partially changed hosts drained; confirm remaining fleet capacity.
- Map each host to package, configuration, service, load-balancer, and dependent-state changes.
- Verify previous artifacts and backups, permissions, signatures, and data/schema compatibility.
- Preview rollback on one affected host with check/diff while protecting secrets and classifying skipped tasks.
- Run the reviewed rollback play on one host, validate configuration, restart only required services, and perform local/external health checks.
- Repeat in bounded batches only after the recovery canary is stable.
- If backward compatibility fails, stop and use the documented forward fix or data recovery.
- Run the recovery play again and require changed=0; record final host/version/health state.
Evidence