Configure auditd rules and investigate Linux audit events
Design a question-led RHEL 9 audit policy, validate architecture-aware rules and capacity, reconstruct multi-record events, protect remote evidence, and introduce immutable mode safely.
Produce a complete, useful, privacy-aware Linux audit trail with zero silent loss and a repeatable investigation workflow.
- Red Hat Enterprise Linux 9.x
- Linux Audit userspace RHEL 9 current
- Maintained RHEL 9 canary Kernel auditing enabled at boot, official audit package, reliable UTC time, representative workload and no current lost records.
sudo auditctl -s && rpm -q audit && timedatectl - Evidence and privacy contract Approved security questions, required fields, retention, access, redaction, legal hold and incident owners.
- Protected storage and collector Capacity-isolated audit filesystem plus authenticated remote collector in a separate administration domain.
- Recovery and reboot access Console access, prior complete rules/configuration, change window and tested reboot rollback before immutable mode.
- Synthetic fixtures Dedicated test login, safe paths and representative steady/peak workloads that contain no production secrets.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A Linux Audit deployment on Red Hat Enterprise Linux 9 with a dedicated log filesystem, fail-action and rotation policy, bounded kernel backlog, measured rule volume, persistent ordered rule fragments and zero lost records under representative peak activity.
- A least-noise rule set for audit configuration, identity/authentication files, privileged execution, selected administrative syscalls and protected application configuration, with explicit 64-bit and—only where supported—32-bit syscall architecture handling.
- An investigation workflow that preserves raw logs, groups multi-record events by serial, distinguishes login identity `auid` from real/effective IDs, converts architecture/syscall/exit values safely, correlates PATH/CWD/EXECVE/SYSCALL/PROCTITLE records and produces privacy-redacted reports.
- A tamper-resistance design that watches audit configuration/log access, forwards events to a separately controlled collector, limits local audit administration, and enables immutable rules (`-e 2`) only after canary validation and a scheduled reboot/change boundary.
- `auditctl -s` reports enabled auditing, the intended failure mode and backlog limit, `lost=0`, low normal backlog and no unexplained plugin queue overflow or log suspension.
- Persistent rules load in deterministic order, use keys and filters, cover the correct syscall tables, avoid blanket `execve` or recursive high-churn rules, and generate complete test events without saturating the host.
- `ausearch` and `aureport` reconstruct who logged in, what executable/syscall/path/object/result occurred and which rule matched, while raw evidence remains protected and interpreted output is clearly marked as derived.
- After `-e 2`, runtime rule and enabled-state changes are rejected until reboot. The team understands this prevents ordinary auditctl tampering but not a privileged attacker controlling boot/kernel, so remote protected collection remains essential.
Architecture
How the parts fit together
Kernel audit hooks evaluate ordered task, exit, filesystem and exclusion rules and queue matching records in the kernel backlog. `auditd` drains the queue to a root-protected log on a dedicated filesystem and may dispatch to a remote transport plugin. Persistent fragments in `/etc/audit/rules.d` are merged by `augenrules` into the active rules file at boot/load. Keys label rule intent for searching. One event may contain SYSCALL, EXECVE, CWD, PATH, PROCTITLE and configuration records sharing the same timestamp/serial. `auid` tracks the original login identity across sudo/setuid while uid/euid describe current credentials. Dashboards monitor kernel lost/backlog/rate plus daemon disk/plugin state. The final ordered rule sets immutable mode; further rule changes require a controlled reboot.
- Define security questions, event fields, retention/privacy, response owners and maximum volume before writing rules.
- Inventory audit status, boot state, rules, lost/backlog, daemon/plugin/disk/rotation and collector path.
- Map required operations to syscall names per supported architecture and apply narrow identity/path/success filters and searchable keys.
- Build ordered persistent fragments, check them and load on a canary without immutable finalization.
- Generate known successful and failed events, reconstruct them from raw logs and compare event rate/backlog/disk/plugin behavior under peak load.
- Roll out gradually, enable immutable `-e 2` only after all tests and reboot plan pass, forward remotely and monitor continuously.
- Investigate by preserving raw logs and host/time/rule hashes before interpretation; change immutable rules only through an approved reboot.
Assumptions
- The host runs a maintained RHEL 9 update with the official `audit` userspace package, kernel audit enabled from boot, reliable UTC time and controlled console/reboot access.
- Security owners define concrete questions and required evidence rather than installing sample compliance rules as proof of compliance.
- The audit log has a dedicated or capacity-isolated filesystem, protected root permissions, monitored storage and a remote collector under a different administrative boundary.
- The organization has decided availability behavior for backlog, disk full and disk error. `single` or `halt` can intentionally stop service and requires executive/system-owner approval.
- 32-bit compatibility is either disabled/not supported and documented, or tested with separate `arch=b32` rules. Syscall names/numbers are never assumed identical between ABIs.
- Privacy/legal owners approve collection of command arguments, paths, usernames, terminal, network and authentication events plus retention and access.
Key concepts
- Audit event
- One logical operation represented by multiple records sharing the same timestamp and serial, such as SYSCALL, EXECVE, CWD and PATH.
- auid/loginuid
- Original authenticated login identity retained across privilege changes; unset identities require explicit filter handling.
- uid/euid
- Current real/effective credentials at syscall time, which may differ from the login identity after sudo or setuid.
- arch=b64/b32
- Selects the architecture-specific syscall table. It must appear before `-S`; bi-arch systems need separate reviewed rules.
- Backlog
- Kernel queue waiting for auditd. A growing queue or `lost` record count means audit evidence is incomplete.
- Rule key
- Human-readable tag attached to matching records and used by `ausearch -k`; it does not change enforcement.
- Immutable audit configuration
- `-e 2` prevents runtime audit rule/config changes until reboot. It is the final rule and creates an operational reboot boundary.
Before you copy
Values used in this guide
{{minInteractiveUid}}Minimum login UID from local account policy.
Example: 1000{{auditKey}}Stable lower-noise rule key used for investigation.
Example: identity_change{{investigationStart}}UTC beginning of incident window.
Example: 2026-07-28 10:00:00{{investigationEnd}}UTC end of incident window.
Example: 2026-07-28 10:15:00{{protectedConfig}}Exact application configuration path watched for writes/attributes.
Example: /etc/acme-web/app.tomlSecurity and production boundaries
- Audit records can contain command-line arguments, paths, usernames, terminals, addresses and security labels. Collection and forwarding need minimization, access control, encryption and retention.
- A blanket `execve` rule for all users or recursive watch on a hot tree can overwhelm kernel backlog, disk and collector and may collect secrets from arguments. Begin from questions and filters.
- Audit observes; it does not prevent the operation. Pair findings with SELinux/AppArmor, DAC, authentication and change controls.
- Local root can influence logs, daemon, storage or reboot. Immutable rules raise the bar for runtime `auditctl` changes but do not replace remote append-only/tamper-evident collection.
- Interpreted output (`ausearch -i`) resolves IDs/names using current/local data and is convenient but derived. Preserve raw records and hashes for forensic use.
Stop before continuing if
- Stop if audit is disabled, lost records are nonzero, backlog approaches limit, auditd/plugins are suspended, disk actions are unowned or time is unreliable.
- Do not load broad sample/execve/directory rules before canary volume tests and privacy approval.
- Stop if syscall architecture mapping is unknown, arch appears after `-S`, b32 coverage is assumed, or filters include unset login IDs incorrectly.
- Do not append `-e 2` until all syntax, event, load, forwarding, rollback and reboot tests pass; after it is loaded, do not promise runtime rollback.
- Stop an investigation if raw evidence is modified, event records are separated from serial, or interpreted names are treated as original bytes.
- Do not delete or shorten audit retention during an incident or to solve capacity; invoke the approved capacity/failure action.
instruction
Define the questions, evidence, privacy, and retention contract
Write each security question as an actor, action, object, result, and time window. For every question, name the required fields, consumers, retention period, access control, redaction rule, expected event rate, and response owner before translating it into an Audit rule.
Why this step matters
Linux Audit can record low-level activity at very high volume, but it cannot decide which evidence is proportionate, legally permitted, or operationally useful. Starting from copied compliance snippets often creates duplicate rules, records every process execution, or collects command arguments that contain credentials while still failing to answer who changed the protected object. A question-led contract identifies the login identity, current credentials, executable, syscall, path, result, host, serial, time and key that an investigator actually needs. It also distinguishes host evidence from application business audit. Retention and access are part of the design because an accurate audit trail can expose usernames, internal paths, commands and authentication context. Expected event rate and response ownership make capacity and alerting testable rather than aspirational.
What to understand
Express each question as actor, action, object, success/failure, time and host.
Record whether `auid`, uid, euid, session, executable, arguments, path and security label are necessary.
Set a minimum retention and a deletion/legal-hold process instead of keeping data indefinitely.
Identify who may search raw logs and who receives only redacted derived reports.
Estimate normal and incident event rates and define an evidence-gap escalation owner.
System changes
- Creates a reviewed evidence and privacy specification; no host configuration changes.
Syntax explained
auid- Original authenticated login identity retained across sudo and setuid transitions.
rule key- Stable intent label used to search and measure a related group of rules.
column -t -s, security/linux-audit-evidence-contract.csvquestion actor action object result retention identity-file-change login auid write/attribute /etc/passwd,/etc/group both 180d acme-config-change login auid write/attribute /etc/acme-web/app.toml both 90d sudo-execution login auid execute /usr/bin/sudo both 90d audit-config-change login auid write/attribute /etc/audit/** both 365d
Checkpoint: Every proposed rule answers a named security question
test -s security/linux-audit-evidence-contract.csv && grep -E 'retention|identity|audit-config' security/linux-audit-evidence-contract.csvContinue whenApproved questions, fields, privacy basis, retention, access, alert owner, expected rate and explicit exclusions such as blanket command capture.
Stop whenThe team cannot state why arguments or identities are collected, has no retention/access owner, or proposes blanket `execve` as a substitute for a threat model.
If this step fails
A compliance template contains hundreds of unexplained rules.
Likely causeControl text was copied without mapping it to local evidence questions.
Trace each key to a control, owner and investigation query.
ResolutionRemove unsupported scope in staging and rebuild from approved questions.
The proposed rule may capture secrets in arguments.
Likely causeCommand-line collection was selected without data minimization.
Review EXECVE/PROCTITLE need and representative application invocations.
ResolutionPrefer exact file/executable events or application audit and redact derived reports; do not collect arguments by default.
Security notes
- Treat the contract and sample events as sensitive operational metadata.
- Audit observes activity; it does not authorize or block it.
Alternatives
- Use application-level structured audit for business actions and correlate it with narrow host events.
Stop conditions
- The team cannot state why arguments or identities are collected, has no retention/access owner, or proposes blanket `execve` as a substitute for a threat model.
verification
Capture kernel, daemon, rules, storage, and forwarding baseline
Before changing rules, verify auditing began at boot, `auditd` is active, the kernel reports no lost records, active rules match persistent fragments, the log filesystem has capacity, rotation/actions are understood, and every dispatcher or remote path is healthy.
Why this step matters
Absence of an event is evidence only when the collection path was healthy throughout the window. `auditctl -s` exposes kernel enabled, failure, rate, backlog limit, current backlog and lost counters. The daemon status alone cannot reveal kernel drops, and a green daemon cannot guarantee a stalled dispatcher or full log filesystem is safe. Comparing `auditctl -l` with ordered files in `/etc/audit/rules.d` detects runtime drift. Boot configuration matters because starting audit late misses early activity. Capture filesystem, inode, ownership, rotation, space actions, plugin/collector state, time synchronization, package version and the newest readable serial. Never reset the lost counter merely to make a dashboard green; first preserve and declare the evidence gap.
What to understand
Record kernel command line, package/kernel release and UTC synchronization.
Capture `auditctl -s` fields and the complete active rule list with hashes.
Inventory ordered persistent fragments and the generated rules file.
Check audit log filesystem capacity/inodes, owner/mode, rotation and configured space actions.
Inspect dispatcher and remote collector queues plus continuity of event serial/time.
System changes
- None; stores a protected pre-change evidence bundle.
Syntax explained
lost- Kernel records dropped since boot; any nonzero value creates an evidence gap.
backlog- Records currently waiting for auditd to drain from the kernel queue.
failure- Kernel behavior when critical audit errors occur; policy must own the availability tradeoff.
sudo auditctl -s && sudo auditctl -l && sudo systemctl status auditd --no-pager && sudo ausearch --start recent -m DAEMON_START,CONFIG_CHANGE -ienabled 1 failure 1 pid 721 rate_limit 0 backlog_limit 8192 lost 0 backlog 0 backlog_wait_time 60000 type=DAEMON_START msg=audit(07/28/2026 09:01:42.110:44) : auditd start, ver=3.1.5 format=enriched
Checkpoint: The existing audit trail is complete enough to change safely
sudo auditctl -s | grep -E '^(enabled|failure|lost|backlog) ' && sudo test -r /var/log/audit/audit.log && df -h /var/log/auditContinue whenenabled=1, lost=0, stable low backlog, active/persistent rules reconciled, protected writable storage, healthy plugins/collector and reliable time.
Stop whenAuditing is disabled, lost is nonzero without declared scope, backlog grows, disk/plugin is unhealthy, time is unreliable, or active rules cannot be reconciled.
If this step fails
auditctl reports lost records.
Likely causeBacklog overflow, daemon/disk/plugin delay, memory pressure or a rule storm dropped evidence.
Preserve status, logs, event rate, filesystem and plugin metrics before any reset.
ResolutionDeclare the gap, repair throughput/noise and repeat the entire validation interval.
Active rules differ from `/etc/audit/rules.d`.
Likely causeA runtime change, failed load, ordering issue or stale generated file created drift.
Compare `auditctl -l`, fragments and `augenrules --check`.
ResolutionReconcile on a canary and load through the approved persistent mechanism.
Security notes
- Baseline artifacts reveal monitored paths and security controls; restrict them.
- Do not stop auditd with ordinary systemd commands as a diagnostic shortcut.
Alternatives
- A signed host collector may gather these read-only checks if it preserves exact raw output and time.
Stop conditions
- Auditing is disabled, lost is nonzero without declared scope, backlog grows, disk/plugin is unhealthy, time is unreliable, or active rules cannot be reconciled.
decision
Map operations to syscall tables and supported architectures
Use `ausyscall` and the host ABI inventory to map each approved operation. Put `-F arch=b64` or `b32` before `-S`, create separate reviewed rules only for ABIs the host can execute, and never copy numeric syscall identifiers between architectures.
Why this step matters
Audit syscall rules are evaluated against an architecture-specific syscall table. The same operation can have different numbers on x86_64 and i386, and an unqualified or misordered rule can bind to the wrong table. `arch` must be selected before `-S` so audit tooling resolves names correctly. A 64-bit kernel may support 32-bit binaries even if normal applications are 64-bit, but adding b32 rules without verifying compatibility can fail or create misleading coverage. Conversely, omitting a supported ABI creates an evasion path. Map semantic operations to syscall names with `ausyscall`, record whether compatibility execution is enabled, and exercise both tested paths. Prefer grouped syscalls with identical fields/action because fewer kernel rules evaluate more efficiently than many duplicates.
What to understand
Inventory CPU architecture, kernel personality and installed compatibility runtime.
Resolve syscall names for every supported ABI using host tooling.
Place each `arch` filter before its `-S` list.
Group only syscalls sharing the same security question, filters, success scope and key.
Test each supported ABI with a harmless fixture and document unsupported ABIs explicitly.
System changes
- None; produces the ABI-to-operation rule design.
Syntax explained
arch=b64- Selects the 64-bit syscall table before syscall-name resolution.
arch=b32- Selects the 32-bit compatibility syscall table only when supported and required.
ausyscall- Maps names and numbers using the audit userspace table instead of guesswork.
uname -m && getconf LONG_BIT && ausyscall x86_64 sethostname && ausyscall i386 sethostname && sudo auditctl -lx86_64 64 170 74 -a always,exit -F arch=b64 -S sethostname,setdomainname -F auid>=1000 -F auid!=4294967295 -k system_identity
Checkpoint: Every syscall rule has tested ABI coverage
ausyscall x86_64 sethostname && sudo auditctl -l | grep 'arch=b64'Continue whenNamed syscalls, supported ABI matrix, arch-before-S ordering, no copied numeric identifiers and test cases for each enabled ABI.
Stop whenCompatibility support is unknown, a rule lacks an architecture, `arch` follows `-S`, or numeric syscalls were copied from another system.
If this step fails
The rule loads but never records the test.
Likely causeIt was resolved against another syscall table or the operation uses a different syscall.
Trace the harmless fixture and compare with `ausyscall` for exact ABI.
ResolutionCorrect architecture/order/name and retest on the target kernel.
A b32 rule fails to load.
Likely causeThe kernel/platform does not expose the assumed compatibility table or syscall.
Check ABI support and audit tooling output.
ResolutionRemove unsupported coverage with documented evidence; do not claim it is monitored.
Security notes
- An omitted executable ABI can create a monitoring blind spot.
- Syscall names are more reviewable than raw numbers but still require target-version testing.
Alternatives
- Use exact file watches or application events when the question concerns an object rather than every syscall implementation.
Stop conditions
- Compatibility support is unknown, a rule lacks an architecture, `arch` follows `-S`, or numeric syscalls were copied from another system.
config
Configure protected storage, rotation, and explicit failure actions
Place audit logs on capacity-isolated storage, restrict access, size rotation from measured rate and retention, and select `space_left`, `admin_space_left`, disk-full, disk-error, flush and notification actions with system owners. High-availability and fail-secure hosts may require different actions.
Why this step matters
Audit storage policy is an availability and evidence-retention decision, not a cosmetic configuration. An aggressive `halt` or `single` action can stop production, while `ignore` can silently lose mandatory evidence. A separate filesystem prevents audit growth from filling the root filesystem but still needs capacity, inode and mount monitoring. Rotation must retain enough local buffer for collector outages without overwriting an incident. `incremental_async` plus a measured frequency balances durability and throughput; the choice must be tested against organizational loss tolerance. Thresholds shown in MiB are examples for the fixture, not universal sizing. Reload daemon storage settings through the supported audit service procedure and confirm configuration events; kernel rule immutability is a separate boundary.
What to understand
Calculate bytes/events per second at normal and peak load plus collector outage window.
Use dedicated storage, root ownership, restrictive traversal and backup/legal-hold policy.
Set warning/admin thresholds early enough for a human or automation to respond.
Obtain explicit owner approval for single/halt actions and test console recovery.
Monitor log filesystem, auditd suspension, rotation, notification delivery and collector lag.
System changes
- Replaces audit daemon storage/rotation/failure settings and reloads them.
Syntax explained
incremental_async- Writes asynchronously and requests periodic data synchronization based on `freq`.
keep_logs- Retains rotated logs instead of silently deleting the oldest; requires capacity management.
single / halt- Availability-impacting fail-secure actions requiring tested console recovery and owner approval.
/etc/audit/auditd.conflog_file = /var/log/audit/audit.log
log_group = root
write_logs = yes
flush = incremental_async
freq = 100
max_log_file = 128
num_logs = 10
max_log_file_action = keep_logs
space_left = 2048
space_left_action = email
admin_space_left = 1024
admin_space_left_action = single
disk_full_action = halt
disk_error_action = haltsudo install -o root -g root -m 0640 security/auditd.conf /etc/audit/auditd.conf && sudo service auditd reload && sudo auditctl -sConfig file /etc/audit/auditd.conf reloaded log_file = /var/log/audit/audit.log flush = incremental_async freq = 100 max_log_file = 128 num_logs = 10 space_left = 2048 admin_space_left = 1024 disk_full_action = halt
Checkpoint: Capacity and failure behavior are measured and owned
sudo service auditd status; sudo auditctl -s; df -h /var/log/auditContinue whenValid protected configuration, ample thresholds, functioning notification/collector, tested rotation and an approved response for every failure action.
Stop whenThe log shares uncontrolled root capacity, thresholds exceed free space, halt/single lacks approval/recovery test, or reload interrupts evidence collection.
If this step fails
Reload fails or auditd becomes suspended.
Likely causeInvalid setting, permission, storage or action executable problem.
Inspect service/audit logs and validate the candidate file offline.
ResolutionRestore the prior reviewed configuration and reload through the supported service procedure.
The host enters single-user mode during a capacity test.
Likely causeAdmin threshold/action was reached as configured.
Use approved console recovery and preserve log/storage evidence.
ResolutionIncrease tested capacity or adjust thresholds/actions with the system owner; never truncate active evidence.
Security notes
- Audit logs and notifications may contain personal and secret-adjacent data.
- A remote collector should use authenticated encrypted transport and separate administration.
Alternatives
- For availability-first systems, choose an approved alert/suspend action with explicit evidence-gap handling instead of pretending fail-secure behavior is free.
Stop conditions
- The log shares uncontrolled root capacity, thresholds exceed free space, halt/single lacks approval/recovery test, or reload interrupts evidence collection.
config
Stage ordered base kernel controls without immutability
Create the first ordered fragment to remove unmanaged runtime rules in the staged policy, set a tested backlog limit and failure behavior, and leave immutable mode out. Review how `augenrules` sorts filenames before adding functional rules.
Why this step matters
Persistent fragments are concatenated in lexical order, so controls that delete rules, define queue behavior and finalize immutability must have deliberate positions. `-D` is appropriate only inside a complete owned rule set: loading a partial fragment after deletion can create a monitoring gap. The backlog value is not a magic hardening number; it must absorb measured bursts while auditd drains normally, and memory impact must be observed. Failure mode `1` logs kernel audit failures without deliberately panicking this general-purpose fixture; systems requiring panic/fail-secure behavior need separate tested policy. Immutable `-e 2` is intentionally absent because it would prevent iterative correction and make rollback require reboot before functional and load evidence exists.
What to understand
Inventory vendor/compliance fragments before taking ownership of `-D`.
Use numeric prefixes that make base, exclusions, functional rules and finalization obvious.
Size backlog from peak queue measurements and host memory constraints.
Record the approved kernel failure behavior and response.
Hash staged fragments and review the generated merged file before load.
System changes
- Adds a persistent base fragment but does not load it in this step.
Syntax explained
-D- Deletes all active rules during a complete policy load; dangerous with incomplete fragments.
-b 8192- Sets the kernel backlog record capacity for this measured fixture.
-f 1- Selects printk-style kernel failure reporting for the fixture; policy may require another tested value.
/etc/audit/rules.d/10-base-config.rules## Ordered base controls; immutable finalization belongs in 99-finalize.rules.
-D
-b 8192
-f 1sudo install -o root -g root -m 0600 security/10-base-config.rules /etc/audit/rules.d/10-base-config.rules && sudo augenrules --check/usr/sbin/augenrules: Rules have changed and should be updated 10-base-config.rules: -D -b 8192 -f 1 No -e 2 rule present during canary validation.
Checkpoint: Rule order and kernel controls are safe to canary
sudo augenrules --check && grep -R --line-number -- '-e 2' /etc/audit/rules.d || trueContinue whenComplete owned fragment inventory, reviewed order, measured backlog/failure setting and no immutable rule during staging.
Stop when`-D` would erase unmanaged required rules, backlog/failure choice is untested, ordering is ambiguous, or any fragment already enables `-e 2`.
If this step fails
Loading starts with -D then fails.
Likely causeA later syntax error left a partial rule set after deletion.
Preserve CONFIG_CHANGE events and compare active rules immediately.
ResolutionRestore the previous complete fragment set and load atomically on a canary.
Backlog memory or latency rises in tests.
Likely causeLimit/wait behavior and rule volume are mismatched to the host.
Measure backlog, lost, audit rate, CPU and auditd drain.
ResolutionReduce noisy rules and choose a measured limit; do not mask loss with an enormous queue.
Security notes
- Restrict rule fragments to root and version/hash them.
- Do not treat `-f 2` panic as universally safer; it can create deliberate denial of service.
Alternatives
- Retain a vendor-owned base when policy ownership and upgrade testing are stronger than local replacement.
Stop conditions
- `-D` would erase unmanaged required rules, backlog/failure choice is untested, ordering is ambiguous, or any fragment already enables `-e 2`.
config
Add exact identity and protected-configuration rules
Monitor writes and attribute changes to account databases, audit configuration, and the exact application configuration path. Use stable keys and avoid recursive watches on volatile trees. The examples answer object-change questions without recording every file read.
Why this step matters
File rules should correspond to objects whose modification materially changes identity, authentication or security posture. The `wa` permissions capture writes and metadata changes without logging every read of password databases. A key groups related rules for queries and volume dashboards. Exact paths and small stable directories are preferable to recursive watches over `/etc`, application data or logs; high-churn trees generate noise, increase backlog pressure and make investigations harder. Watching audit configuration detects attempts to alter persistence, but monitoring audit log writes naively can record auditd's own activity recursively. Protection of audit logs is therefore handled with ownership, mount controls, narrow human-access filters and remote collection rather than a broad write watch.
What to understand
Verify every watched object exists or document creation-time behavior.
Use `w` for content changes and `a` for owner/mode/xattr changes.
Give keys stable intent names independent of ticket numbers.
Avoid reads unless a defined threat question requires them and volume/privacy testing passes.
Test package/account-management tools so atomic rename/replacement events still answer the question.
System changes
- Adds persistent object rules but does not load them until the canary step.
Syntax explained
-w path- Declares a filesystem watch accepted by the Audit rules tooling for an exact path/tree.
-p wa- Records write and attribute-change permissions, not ordinary reads or execution.
-k key- Labels matching records for `ausearch -k` and rate analysis.
/etc/audit/rules.d/30-identity-config.rulesValues stay on this page and are never sent or saved.
-w /etc/passwd -p wa -k identity_change
-w /etc/group -p wa -k identity_change
-w /etc/shadow -p wa -k identity_change
-w /etc/gshadow -p wa -k identity_change
-w /etc/audit/ -p wa -k audit_config_change
-w {{protectedConfig}} -p wa -k acme_config_changesudo install -o root -g root -m 0600 security/30-identity-config.rules /etc/audit/rules.d/30-identity-config.rules && sudo augenrules --check-w /etc/passwd -p wa -k identity_change -w /etc/group -p wa -k identity_change -w /etc/shadow -p wa -k identity_change -w /etc/audit/ -p wa -k audit_config_change -w /etc/acme-web/app.toml -p wa -k acme_config_change Persistent fragments merge to a changed candidate; active rules remain unchanged until canary load.
Checkpoint: Each object rule is narrow, searchable, and measurable
sudo augenrules --check && awk '!/^($|#)/ {print}' /etc/audit/rules.d/30-identity-config.rulesContinue whenOnly approved identity/audit/application paths, `wa` scope, stable keys, root-only file and a positive/negative test for atomic replacement.
Stop whenA watched tree is volatile/recursive, read monitoring lacks justification, keys are ambiguous, or the rule could recursively audit auditd's own log writes.
If this step fails
The rule produces no event when a package replaces a file.
Likely causeThe test/path/rule load differs or event was searched outside its key/time.
Check active rules, complete serial and atomic rename target.
ResolutionCorrect exact watched parent/object strategy and retest the real update path.
Event volume spikes after watching a directory.
Likely causeThe directory contains unrelated or high-churn files.
Measure key/path frequency and top actors.
ResolutionReplace it with exact files or narrow syscall/path filters derived from the question.
Security notes
- Account and audit configuration events are highly sensitive and should be forwarded/protected.
- A watch records after an operation is attempted; it does not prevent tampering.
Alternatives
- Use file-integrity monitoring for content baselines and keep Audit focused on actor/action attribution.
Stop conditions
- A watched tree is volatile/recursive, read monitoring lacks justification, keys are ambiguous, or the rule could recursively audit auditd's own log writes.
config
Record narrowly scoped privileged and identity-changing execution
Add architecture-qualified exit rules for exact administrative executables and selected identity-changing syscalls. Filter to authenticated interactive login identities and exclude the unset login UID, then measure both success and failure without enabling global command capture.
Why this step matters
A blanket `execve` rule for every process is tempting because it appears comprehensive, but it can expose command arguments, overwhelm the queue and bury meaningful activity. The evidence contract asks instead when an authenticated human invokes an exact privilege broker and when identity-defining kernel operations are attempted. `auid` identifies the original login across sudo, while uid/euid explain current credentials. The `auid>=1000` threshold is a local policy variable, not a universal constant, and the explicit exclusion of `4294967295` prevents service/kernel tasks with unset login identity from being mislabeled as people. `exe` narrows execution to the canonical binary but package paths, wrappers and alternatives must be inventoried. Separate b32 rules are required only if that ABI can run.
What to understand
Read the local UID_MIN/account policy before selecting the interactive threshold.
Confirm the canonical executable path and any supported privilege brokers.
Keep success and failure unless the evidence contract deliberately limits one.
Pair auid with uid/euid/session/terminal rather than treating any one field as complete attribution.
Measure event volume and inspect whether EXECVE records contain sensitive arguments.
System changes
- Adds persistent exact-executable and administrative syscall rules for later canary load.
Syntax explained
-a always,exit- Evaluates matching conditions when the syscall exits and records its result.
-F exe=/usr/bin/sudo- Narrows the rule to the resolved executable image rather than every exec.
auid!=4294967295- Excludes the unset login identity used by non-login tasks.
/etc/audit/rules.d/40-privileged.rulesValues stay on this page and are never sent or saved.
-a always,exit -F arch=b64 -S execve -F exe=/usr/bin/sudo -F auid>={{minInteractiveUid}} -F auid!=4294967295 -k privileged_exec
-a always,exit -F arch=b64 -S sethostname,setdomainname -F auid>={{minInteractiveUid}} -F auid!=4294967295 -k system_identitysudo install -o root -g root -m 0600 security/40-privileged.rules /etc/audit/rules.d/40-privileged.rules && sudo augenrules --check-a always,exit -F arch=b64 -S execve -F exe=/usr/bin/sudo -F auid>=1000 -F auid!=4294967295 -k privileged_exec -a always,exit -F arch=b64 -S sethostname,setdomainname -F auid>=1000 -F auid!=4294967295 -k system_identity 32-bit companion rules: not installed; compatibility execution is disabled and tested on this host.
Checkpoint: Privileged execution coverage answers attribution without global capture
sudo augenrules --check && grep -R --line-number 'execve' /etc/audit/rules.dContinue whenOnly exact approved executables/syscalls, correct ABI, local interactive threshold, unset exclusion, privacy review and measured expected volume.
Stop whenThe rule captures all execve, UID policy is unknown, wrappers/ABIs are untested, or arguments may expose secrets without approval.
If this step fails
sudo event has auid=4294967295.
Likely causeThe process did not originate in a valid audited login session or loginuid propagation is broken.
Compare session, uid/euid, parent chain and PAM/loginuid configuration.
ResolutionRepair login session attribution; do not assign a human based only on uid.
Expected administrator action bypasses the exact exe rule.
Likely causeA wrapper, alternative path or another broker performed it.
Resolve `/proc/PID/exe`, package ownership and supported workflow.
ResolutionAdd a separately justified exact rule or standardize the broker; do not widen to all execution.
Security notes
- Command arguments can contain tokens and passwords; prefer tools that never accept secrets on argv.
- Audit attribution supports investigation but does not prove the account owner personally acted.
Alternatives
- Use sudo's own protected structured I/O/event logging as complementary evidence, correlated by session and time.
Stop conditions
- The rule captures all execve, UID policy is unknown, wrappers/ABIs are untested, or arguments may expose secrets without approval.
config
Protect rule persistence and monitor human access to audit logs
Keep rule and daemon configuration root-owned, record changes to `/etc/audit`, and add an authenticated-human read/open rule for the audit log directory rather than watching auditd's own writes. Forward to a collector controlled outside the host.
Why this step matters
Tamper resistance needs several independent layers. Root-only ownership prevents ordinary users changing persistent rules, and the configuration watch provides attribution for attempts. A naive `-w /var/log/audit -p wa` can match auditd's normal writes and create excessive self-referential volume. This design instead asks when an authenticated interactive identity opens audit log files and excludes unset service activity; target-kernel support for `openat2` must be checked. Local root can still alter storage, stop a plugin, reboot into another kernel or erase files after compromise. Remote authenticated collection in a separate administrative boundary is therefore the durable evidence path. Immutable mode later prevents ordinary runtime rule changes, but it cannot turn a compromised boot chain into trustworthy evidence.
What to understand
Protect `/etc/audit`, rule fragments, daemon configuration and plugin configuration with strict ownership/modes.
Test exact human log-access syscalls on each supported architecture and avoid auditing auditd's own write path.
Restrict local readers and expose redacted reports rather than raw logs where possible.
Forward over authenticated encrypted transport to a separately administered collector.
Compare serial/time continuity and alert on local/remote gaps, plugin queue pressure and configuration changes.
System changes
- Tightens audit configuration permissions and stages narrow configuration/log-access rules.
Syntax explained
-F dir=/var/log/audit- Filters matching syscalls to paths below the exact audit log directory.
openat/openat2- Modern file-open operations to test on the target kernel/ABI.
separate collector- Places retained evidence outside the host's local failure and administration domain.
/etc/audit/rules.d/50-audit-protection.rulesValues stay on this page and are never sent or saved.
-w /etc/audit/ -p wa -k audit_config_change
-a always,exit -F arch=b64 -S openat,openat2 -F dir=/var/log/audit -F auid>={{minInteractiveUid}} -F auid!=4294967295 -k audit_log_accesssudo chown -R root:root /etc/audit && sudo find /etc/audit -type f -exec chmod go-w {} + && sudo install -o root -g root -m 0600 security/50-audit-protection.rules /etc/audit/rules.d/50-audit-protection.rulesdrwxr-x--- root root /etc/audit -w /etc/audit/ -p wa -k audit_config_change -a always,exit -F arch=b64 -S openat,openat2 -F dir=/var/log/audit -F auid>=1000 -F auid!=4294967295 -k audit_log_access Remote collector continuity test: serial range received with no gap.
Checkpoint: Tampering and human log access are visible without recursive noise
sudo augenrules --check && sudo find /etc/audit -xdev -perm /022 -ls && sudo ausearch -m CONFIG_CHANGE -ts recent -iContinue whenNo writable configuration artifacts, narrow tested access rule, protected local logs, authenticated remote continuity and alerts for gaps.
Stop whenThe log rule records auditd continuously, remote transport is unauthenticated, collectors share host credentials, or configuration remains writable by non-root users.
If this step fails
audit_log_access dominates event volume.
Likely causeThe filter also matches daemon/backup/collector processes or a broad reader.
Inspect auid, exe, syscall and path across complete serials.
ResolutionNarrow authenticated actors/executables or redesign log export; never suppress unexplained access silently.
Remote serial ranges have gaps.
Likely causeNetwork/plugin/collector backpressure or retention failed.
Compare local raw serial/time, plugin queue and collector ingestion.
ResolutionPreserve local logs, repair transport/capacity and document the remote evidence gap.
Security notes
- Remote collector credentials must not be embedded in rule files or command output.
- Raw audit access should itself be privileged, logged and reviewed.
Alternatives
- Ship signed rotations through a controlled batch channel when live transport is unavailable, accepting and documenting delayed detection.
Stop conditions
- The log rule records auditd continuously, remote transport is unauthenticated, collectors share host credentials, or configuration remains writable by non-root users.
verification
Check, review, and load the complete canary ruleset
On a matching canary, use `augenrules --check` to determine whether fragments would change the merged file, independently review every rule against the target audit tooling, prove no final `-e 2` is present, load once through `augenrules`, and immediately compare active rules, configuration events, lost/backlog and daemon health.
Why this step matters
A ruleset is an ordered kernel program. Syntax-valid individual lines can still be unsafe as a whole because `-D` deletes previous coverage, duplicates increase cost, exclusions can shadow later rules and immutable mode can close the rollback path. Generate and review the exact merged file that will load. Load only on a canary with console access and a protected baseline, then capture the CONFIG_CHANGE event and compare normalized active rules. Immediately inspect lost/backlog and rule count before generating traffic. `augenrules --load` changes the live kernel rules; it is not merely validation. Never add `-e 2` to this canary cycle. Keep the former complete fragment set ready so a failed load can be restored without disabling audit.
What to understand
Back up and hash the complete prior persistent and active rules.
Review lexical order, duplicates, broad exclusions, keys, architectures and absence of finalization.
Run `augenrules --check` before the mutating load, understanding that it checks whether the merged file would change rather than proving kernel rule semantics.
Load once on a canary and capture exit code plus CONFIG_CHANGE serial.
Diff normalized active rules and monitor lost/backlog/daemon/plugin/disk immediately.
System changes
- Replaces the active canary kernel audit rules with the complete staged persistent set.
Syntax explained
augenrules --check- Reports whether persistent fragments would change the generated rules file; it is not a complete kernel syntax or behavior validator.
augenrules --load- Merges persistent fragments and loads the resulting active kernel policy.
auditctl -l- Shows the normalized active rules actually enforced by the kernel.
sudo augenrules --check && sudo augenrules --load && sudo auditctl -l && sudo auditctl -s/usr/sbin/augenrules: No change enabled 1 failure 1 backlog_limit 8192 lost 0 backlog 1 -w /etc/passwd -p wa -k identity_change -a always,exit -F arch=b64 -S execve -F exe=/usr/bin/sudo -F auid>=1000 -F auid!=-1 -F key=privileged_exec
Checkpoint: Canary active rules exactly match the reviewed non-immutable set
sudo auditctl -s && sudo auditctl -l | tee /root/audit-active-canary.txt && ! grep -R -- '-e 2' /etc/audit/rules.dContinue whenSuccessful load, expected normalized active set, CONFIG_CHANGE evidence, enabled=1, lost=0, stable backlog and healthy daemon/plugins/storage.
Stop whenLoad is partial, active rules differ materially, any exclusion shadows coverage, `-e 2` appears, lost increases, backlog grows or remote forwarding gaps.
If this step fails
augenrules exits nonzero after -D was applied.
Likely causeA later invalid fragment caused partial replacement.
Capture active rules/status and CONFIG_CHANGE immediately.
ResolutionLoad the previous complete known-good fragments, verify evidence continuity and fix offline.
Rule text differs in `auditctl -l`.
Likely causeKernel/auditctl normalized aliases such as unset/-1 or merged syscall lists, or a rule was rejected.
Compare semantics, load messages and event tests rather than raw formatting only.
ResolutionResolve rejected semantic differences before proceeding.
Security notes
- Loading rules can reduce monitoring coverage; treat it as a security change with an evidence-gap owner.
- Use console access because failure behavior may affect availability.
Alternatives
- Boot a disposable clone with the staged rules first when production-like canary capacity is unavailable.
Stop conditions
- Load is partial, active rules differ materially, any exclusion shadows coverage, `-e 2` appears, lost increases, backlog grows or remote forwarding gaps.
verification
Generate known successful and denied fixture events
Use a dedicated test login identity and synthetic objects to perform one successful and one failed action for every key. Record UTC windows and expected actor/object/result, then search by key and event serial without modifying production identities or secrets.
Why this step matters
A loaded rule proves nothing until a controlled event demonstrates the intended fields and result. Each key needs fixtures for success and failure where meaningful because success-only searches can hide denied attacks and failure-only rules miss completed changes. Use synthetic accounts and non-secret files; never change `/etc/shadow` or production hostname merely to test coverage. Capture the exact UTC interval, host, test case and expected serial. Search raw first so numeric IDs, encoded proctitle and original records remain available. A rule may emit several records sharing one serial; counting lines as separate events inflates metrics. Negative tests must distinguish an application/DAC denial from an audit collection failure.
What to understand
Create a disposable authenticated login fixture with known auid and no production privilege.
Use application test paths or reversible metadata changes, not real identity corruption.
Exercise every key and both result directions defined by the evidence contract.
Record one expected logical event and its complete record-type set.
Confirm local and remote collectors receive the same event serial/time/key.
System changes
- Performs bounded reversible fixture operations that generate audit records.
Syntax explained
--raw- Returns original audit field encoding suitable for evidence preservation and exact serial grouping.
-k- Searches events tagged by the stable rule key.
-ts/-te- Bounds the test to the recorded incident/test time window.
Values stay on this page and are never sent or saved.
sudo -iu audit-fixture touch {{protectedConfig}}.candidate && sudo ausearch -k {{auditKey}} -ts {{investigationStart}} -te {{investigationEnd}} --rawtype=SYSCALL msg=audit(1785259801.422:1882): arch=c000003e syscall=257 success=yes exit=3 auid=1007 uid=1007 euid=1007 exe="/usr/bin/touch" key="acme_config_change" type=CWD msg=audit(1785259801.422:1882): cwd="/home/audit-fixture" type=PATH msg=audit(1785259801.422:1882): item=0 name="/etc/acme-web/app.toml" inode=28103 nametype=NORMAL
Checkpoint: Every rule key produces the expected complete logical event
sudo ausearch -k {{auditKey}} -ts {{investigationStart}} -te {{investigationEnd}} --raw | grep -E 'type=(SYSCALL|PATH)'Continue whenCorrect host/time/auid/uid/euid/exe/syscall/path/result/key, all records grouped by serial, no lost count increase and remote continuity.
Stop whenA test touches production credentials/configuration, raw events lack required fields, serials are split, lost/backlog increases, or remote copy is absent.
If this step fails
ausearch returns no matching events.
Likely causeRule was not active, wrong key/time/host/ABI/path was used, or audit pipeline lost data.
Inspect active rule, status, raw recent CONFIG_CHANGE and exact fixture.
ResolutionFix the identified mapping or pipeline and repeat the whole fixture window.
The event records a service auid instead of the test user.
Likely causeThe fixture did not originate through an audited login session.
Inspect loginuid, PAM session, session ID and process ancestry.
ResolutionRun through a real isolated login path and revalidate attribution.
Security notes
- Fixtures must contain no real passwords, tokens or customer paths.
- Preserve raw output separately from redacted training/screenshots.
Alternatives
- Replay a signed synthetic workload in an isolated clone when production canary operations are prohibited.
Stop conditions
- A test touches production credentials/configuration, raw events lack required fields, serials are split, lost/backlog increases, or remote copy is absent.
instruction
Reconstruct a multi-record event without losing evidential meaning
Preserve the raw file and hash, bound the query by host/time/key, select one event serial, and read SYSCALL, EXECVE, CWD, PATH, PROCTITLE and related records together. Interpret `auid`, uid/euid, architecture, syscall, exit/result, executable and object names as separate claims.
Why this step matters
Linux Audit emits a set of typed records, not a ready-made narrative. The serial after the timestamp identifies one logical event. SYSCALL states the kernel operation, credentials, executable and result; EXECVE may show arguments; CWD supplies the relative-path base; PATH records identify one or more objects and name types; PROCTITLE may preserve encoded process title. `auid` supports login attribution while uid/euid show credentials at operation time. `success=no` with a negative errno is an attempted action, not a completed change. `ausearch -i` resolves IDs and encodings using interpretation tables and current host data, so it is useful for analysis but not a replacement for raw evidence. Investigators should state uncertainty rather than infer a human, path or success that records do not establish.
What to understand
Copy the relevant raw log read-only, record host/time/timezone/package state and hash it.
Search by bounded time/key, then extract all records with one serial.
Map architecture/syscall with target-version `ausyscall` and preserve numeric originals.
Reconstruct relative paths using CWD and every PATH item/name type.
Separate login identity, current credentials, executable, arguments, operation and result in the report.
System changes
- Creates protected raw and interpreted investigation artifacts; does not alter the audited host.
Syntax explained
-a serial- Selects all records belonging to one logical audit event.
--raw- Preserves original field representation for evidence.
-i- Produces a convenient interpreted view that must be labeled derived.
sudo ausearch -a 1882 --raw | tee incident/audit-1882.raw && sha256sum incident/audit-1882.raw && sudo ausearch -a 1882 -itype=SYSCALL msg=audit(07/28/2026 10:10:01.422:1882) : arch=x86_64 syscall=openat success=yes exit=3 auid=operator uid=root euid=root exe=/usr/bin/sudo key=acme_config_change type=EXECVE msg=audit(...:1882) : argc=3 a0=sudo a1=install a2=app.toml type=CWD msg=audit(...:1882) : cwd=/home/operator type=PATH msg=audit(...:1882) : item=0 name=/etc/acme-web/app.toml nametype=NORMAL
Checkpoint: The event narrative is traceable to complete raw records
sha256sum incident/audit-1882.raw && grep -E 'type=(SYSCALL|CWD|PATH)' incident/audit-1882.rawContinue whenProtected raw hash, complete serial, explicit identities/credentials/operation/object/result, target ABI mapping and a redacted derived narrative with uncertainty.
Stop whenRaw logs were modified, records from different serials are mixed, interpreted names replace originals, time/host is unknown, or success is inferred from intent.
If this step fails
PATH is missing from the event.
Likely causeThat syscall/event did not generate a path record, collection failed or the query split records.
Search the complete serial raw and review syscall semantics/lost counter.
ResolutionReport the missing object evidence and adjust a justified exact rule for future events.
Interpreted username differs from incident-era identity.
Likely causeUID/name resolution changed after the event.
Use raw numeric UID, identity-management history and timestamp.
ResolutionReport both raw ID and time-bounded identity evidence; do not rewrite raw logs.
Security notes
- Investigators need least-privilege raw access and a logged evidence-handling path.
- Redact command arguments and paths before tickets or external sharing.
Alternatives
- Use `aureport` for aggregate triage, then return to exact serials for conclusions.
Stop conditions
- Raw logs were modified, records from different serials are mixed, interpreted names replace originals, time/host is unknown, or success is inferred from intent.
verification
Load-test event volume, backlog, storage, and remote continuity
Replay representative normal and incident bursts on the canary, measure events and bytes per key, kernel backlog/lost, auditd CPU/I/O, filesystem growth, dispatcher queue and collector lag. Remove or narrow high-noise rules before rollout rather than hiding loss with a huge queue.
Why this step matters
A rule set that works for one manual command can fail during deployment, package update, authentication storm or incident response. Kernel backlog protects short bursts but is not storage, and raising it indefinitely consumes memory without fixing a slow daemon/plugin or noisy rule. Load testing must exercise the same binaries, paths, identities and remote transport while recording `lost` continuously—one final zero can miss a prior reset or reboot. Per-key rates reveal which question creates volume. Calculate local retention from measured bytes, rotation and collector outage duration. Test warning actions before destructive thresholds using a safe isolated filesystem fixture. Acceptance requires both complete local events and matching remote serial/time ranges.
What to understand
Replay representative steady, deployment and incident bursts with synthetic data.
Sample lost/backlog/rate, auditd CPU/I/O, disk/inodes and plugin queue during the entire run.
Calculate events/bytes per key and identify low-value high-volume fields.
Interrupt or throttle the collector to test local buffering and recovery without filling production storage.
Compare local and remote serial/time/key counts and document expected transport semantics.
System changes
- Generates a bounded burst of synthetic audited activity on the canary and consumes temporary audit capacity.
Syntax explained
peak backlog- Maximum queued records during the measured burst, not a tuning target by itself.
lost=0 throughout- No observed kernel record drop during every sample in the test window.
per-key rate- Connects resource cost back to the security question that created it.
./tests/audit-load-fixture --duration 300 --profile peak & pid=$!; while kill -0 $pid 2>/dev/null; do sudo auditctl -s | grep -E 'lost|backlog'; sleep 5; done; wait $pidsamples=60 peak_events_per_second=184 lost=0 throughout peak_backlog=217 of 8192 audit_log_growth=31 MiB/5m collector_p95_lag=1.8s top_key=privileged_exec 42 events/s acceptance=PASS
Checkpoint: The complete pipeline survives peak and collector-degraded scenarios
sudo auditctl -s && df -h /var/log/audit && ./tests/compare-audit-serials --local /var/log/audit/audit.log --collector fixtureContinue whenNo loss, bounded backlog, sufficient local outage buffer, acceptable CPU/I/O, complete collector continuity and documented capacity margins.
Stop whenLost increases, backlog trends upward, auditd/application latency breaches budget, storage thresholds approach, plugin suspends, or collector has unexplained gaps.
If this step fails
Backlog recovers but lost increased.
Likely causeThe burst exceeded queue capacity or drain throughput.
Identify time/key rate, daemon/plugin/disk latency and memory pressure.
ResolutionNarrow unnecessary rules or improve throughput, then rerun; increasing backlog alone is not proof.
Local logs are complete but collector lags indefinitely.
Likely causeTransport/plugin/collector capacity or retry design is insufficient.
Inspect queue, acknowledgements, TLS/network and collector ingestion.
ResolutionFix remote capacity/retry and retest outage recovery before rollout.
Security notes
- Never use real customer commands or secrets in the load fixture.
- Treat a single dropped record as an evidence-quality failure, not an acceptable percentage.
Alternatives
- Use a production-like isolated clone when the canary cannot tolerate representative burst testing.
Stop conditions
- Lost increases, backlog trends upward, auditd/application latency breaches budget, storage thresholds approach, plugin suspends, or collector has unexplained gaps.
decision
Roll out the observable ruleset before final immutability
Deploy the reviewed non-immutable fragments to a small host cohort, verify reboot persistence, compare active hashes and event rates, investigate every lost/backlog/collector alert, and expand only after a defined observation window. Keep `99-finalize.rules` absent.
Why this step matters
Immutable mode should lock a known-good operational system, not conceal an untested one. A staged cohort reveals distribution-specific service startup, boot ordering, package paths, compatibility ABI, workload volume and collector behavior that a lab cannot fully model. Reboot testing proves persistent fragments load from boot and early events are captured. Compare semantic active rules and signed artifact hashes, not just successful deployment exit codes. An observation window must include a representative maintenance or workload cycle. Expansion stops on any evidence gap or unexplained event-rate difference. Runtime rollback is still possible during this phase, but it must restore the whole previous policy through the supported loader without stopping or disabling auditing.
What to understand
Deploy signed/hash-pinned fragments and daemon configuration to a small diverse cohort.
Reboot through change control and verify boot-time enabled state and complete active rules.
Exercise fixture events and compare local/remote continuity on every host.
Observe peak events, lost/backlog, disk/actions and plugin lag for the agreed window.
Restore the complete previous ruleset on any failure; do not edit individual live rules ad hoc.
System changes
- Loads rules and reboots canary/cohort hosts; immutable mode remains disabled.
Syntax explained
cohort rollout- Limits the blast radius while sampling real host/workload differences.
reboot persistence- Proves ordered fragments load correctly from boot rather than only in the current session.
sudo augenrules --check && sudo augenrules --load && sudo rebootpost-reboot: enabled 1 lost 0 backlog 0 ruleset_sha256=8e652a... remote_continuity=PASS unexpected_keys=0 immutable_finalize=ABSENT
Checkpoint: The non-immutable policy is production-proven and rollback-tested
sudo auditctl -s && sudo auditctl -l | sha256sum && sudo ausearch -m DAEMON_START,CONFIG_CHANGE -ts boot -iContinue whenCorrect boot load, lost=0, stable backlog/storage/plugins, complete remote evidence, all fixtures pass and prior-policy rollback was rehearsed.
Stop whenAny host drifts, boot loses rules/events, event rate exceeds capacity, collector gaps, privacy issue appears, or rollback has not been demonstrated.
If this step fails
Rules are missing only after reboot.
Likely causeFragment ownership/name/order, augenrules generation or boot service state is wrong.
Inspect boot journal, generated audit.rules, permissions and CONFIG_CHANGE.
ResolutionRestore correct persistent packaging and repeat reboot before rollout.
One host has much higher key volume.
Likely causeWorkload/path/account/ABI differs or a noisy process matches the filter.
Compare complete sample events and host inventory.
ResolutionNarrow or create a documented host profile; do not average away the outlier.
Security notes
- Reboot and fail actions require console/recovery coordination.
- Do not allow configuration management to reset lost counters or silently overwrite local rules.
Alternatives
- Maintain separate versioned rule profiles for materially different host roles instead of one broad universal policy.
Stop conditions
- Any host drifts, boot loses rules/events, event rate exceeds capacity, collector gaps, privacy issue appears, or rollback has not been demonstrated.
config
Enable immutable rules only after every gate passes
Create a final lexically last fragment containing only `-e 2`, review the complete generated order, schedule a reboot boundary, load it once, and prove runtime deletion/change is rejected while auditing, fixtures, forwarding and availability remain healthy.
Why this step matters
`-e 2` makes the kernel audit enabled state immutable for the rest of the boot. It is valuable because ordinary runtime `auditctl` attempts cannot delete rules or disable auditing, but it also turns any policy defect into a reboot-required change. The line must be the final rule after all functional fragments. Before loading, archive the reviewed merged rules, prove console/reboot access, confirm no lost/backlog/disk/plugin issue and schedule the operational boundary. After loading, verify `enabled 2`, generate fixtures, and demonstrate a harmless rule deletion attempt is rejected. Do not attempt to bypass immutability. Future kernel rule changes are staged persistently, reviewed and activated during a controlled reboot. This control does not protect against a hostile boot configuration, kernel compromise or local evidence deletion, so remote collection and secure boot/change controls remain necessary.
What to understand
Require signed approval for syntax, behavior, negative tests, load, privacy, capacity, forwarding and rollback evidence.
Ensure `99-finalize.rules` is lexically last and contains only the final enabled-state directive.
Archive hashes of all fragments, generated rules, package/kernel versions and collector state.
Load during a scheduled window with console access and verify rejected runtime mutation.
For later changes, edit reviewed persistent files and reboot; never promise an online rule rollback.
System changes
- Sets kernel audit configuration immutable until the next reboot, creating a mandatory reboot boundary for rule changes.
Syntax explained
-e 2- Locks the kernel audit enabled/rule configuration for the current boot.
99-finalize.rules- Lexically last fragment that prevents functional rules appearing after immutability.
/etc/audit/rules.d/99-finalize.rules## Final rule: load only after merge review, event, load, forwarding, rollback and reboot gates pass.
-e 2printf '%s
' '-e 2' | sudo tee /etc/audit/rules.d/99-finalize.rules >/dev/null && sudo chmod 0600 /etc/audit/rules.d/99-finalize.rules && sudo augenrules --check && sudo augenrules --load && sudo auditctl -senabled 2 failure 1 lost 0 backlog 0 auditctl: Error deleting rule (Operation not permitted) Runtime kernel rule changes now require reboot.
Checkpoint: Immutable mode protects a verified, observable policy
sudo auditctl -s | grep '^enabled 2$' && ! sudo auditctl -D && sudo auditctl -s | grep -E '^(lost 0|backlog 0)$'Continue whenenabled=2, mutation rejected, active rules/hash correct, fixtures and remote continuity pass, lost=0 and the next change has a documented reboot workflow.
Stop whenAny validation gate is incomplete, final rule is not last, console/reboot access is unavailable, or current lost/backlog/disk/plugin/collector state is unhealthy.
If this step fails
A required rule is wrong after `enabled 2`.
Likely causeImmutable finalization was applied before complete validation.
Preserve status/events and determine monitoring/availability impact.
ResolutionUse the approved reboot procedure with a corrected complete persistent ruleset; do not disable audit at runtime.
Host fails or stalls during immutable rollout.
Likely causeFailure action, audit volume, backlog, disk or plugin behavior was not representative in testing.
Use console recovery, preserve evidence and inspect the earliest boot/audit state.
ResolutionRoll back persistent artifacts through a controlled reboot and repeat capacity testing.
Security notes
- Immutable Audit is tamper resistance, not prevention of privileged boot/kernel compromise.
- Every emergency reboot/change must preserve the incident evidence and remote collector continuity.
Alternatives
- Remain at enabled=1 only when an explicit risk decision prioritizes online rule changes; compensate with strict administration, integrity monitoring and remote alerts.
Stop conditions
- Any validation gate is incomplete, final rule is not last, console/reboot access is unavailable, or current lost/backlog/disk/plugin/collector state is unhealthy.
verification
Operate the audit trail as a continuously verified evidence system
Monitor kernel, daemon, storage and collector health; review per-key value/noise and access quarterly; rehearse serial-based investigations; preserve raw evidence and privacy boundaries; and schedule rule changes through the immutable reboot process.
Why this step matters
Audit quality decays as applications, account policy, kernels, architectures, paths and incident questions change. Continuous operation therefore monitors enabled/failure/lost/backlog, daemon suspension, disk thresholds, plugin queue, collector gaps, time and per-key volume. A zero-alert dashboard is not enough: regular drills must retrieve a known event, preserve raw records, group by serial and explain auid versus uid/euid, syscall, path and result. Reviewers should remove rules that no longer answer a question, but immutable hosts require a staged persistent revision and controlled reboot. Retention expiration and legal holds must be executed predictably. Access to raw logs and derived reports is reviewed, and any lost record or broken collector range becomes an explicit incident/evidence gap.
What to understand
Alert on enabled/failure changes, any lost increase, sustained backlog, disk/actions and plugin/collector gaps.
Report per-key event/byte rate and sample completeness rather than only total file size.
Rehearse one known privilege/config event investigation from raw evidence each quarter.
Review privacy, access, retention/legal holds and redaction with owners.
Stage versioned rule changes, rerun all gates and activate immutable revisions through reboot.
System changes
- None during the review; future approved rule revisions use the controlled reboot lifecycle.
Syntax explained
aureport- Provides aggregate triage views; conclusions still require exact raw event review.
CONFIG_CHANGE- Records changes to audit configuration and helps detect unexpected administration.
review due- Forces revalidation when software, threat questions, privacy or operations change.
sudo auditctl -s && sudo aureport --summary --start today && sudo ausearch -m CONFIG_CHANGE,DAEMON_ABORT,DAEMON_END -ts today -iAudit Summary Report Range of time in logs: 07/28/2026 00:00:01.003 - 07/28/2026 12:30:42.190 Number of changes in configuration: 0 Number of failed authentications: 3 Kernel: enabled=2 lost=0 backlog=0 Collector continuity: PASS Review due: 2026-10-28
Checkpoint: Audit completeness, usefulness, and privacy remain demonstrable
sudo auditctl -s && ./tests/compare-audit-serials --since 24h && ./tests/audit-fixtures --all-keysContinue whenNo loss, bounded backlog, complete remote ranges, passing fixtures, useful per-key evidence, current owners/retention/access and successful investigation drill.
Stop whenAny evidence gap is hidden, privacy/access review is overdue, rules no longer match host behavior, or immutable changes are attempted without reboot/change control.
If this step fails
Dashboard is green but fixture search fails.
Likely causeMetrics cover daemon uptime but not rule semantics, serial completeness or key drift.
Inspect active rules/hash, fixture time/key and raw local/remote records.
ResolutionTreat as evidence failure, repair policy/pipeline and backfill only where independently possible.
Retention deletion conflicts with an incident hold.
Likely causeLifecycle metadata or ownership is missing.
Freeze deletion through the approved evidence process and inventory copies.
ResolutionApply legal hold, document scope and correct retention automation without editing raw evidence.
Security notes
- Do not expose raw audit records in general dashboards or tickets.
- Loss, configuration drift and unauthorized raw-log access are security incidents, not routine noise.
Alternatives
- Export narrowly aggregated metrics and redacted incident notes while keeping raw records in the protected evidence system.
Stop conditions
- Any evidence gap is hidden, privacy/access review is overdue, rules no longer match host behavior, or immutable changes are attempted without reboot/change control.
Finish line
Verification checklist
sudo auditctl -s && df -h /var/log/auditenabled=2 after finalization, lost=0, bounded backlog and safe capacity.sudo augenrules --check && sudo auditctl -lReviewed ordered rules, correct architectures/filters/keys and final -e 2 last../tests/audit-fixtures --all-keys --require-success-and-failureEvery question produces complete serial-grouped local and remote evidence../tests/compare-audit-serials --since 24h && ./tests/audit-investigation-drillNo unexplained gaps and analyst reconstructs identities, operation, object and result from raw records.sudo auditctl -s | grep '^enabled 2$' && ! sudo auditctl -DRuntime rule mutation is rejected and future changes require controlled reboot.Recovery guidance
Common problems and safe checks
auditctl shows lost greater than zero.
Likely causeKernel backlog overflow, memory pressure or auditd/plugin/disk slowness dropped records.
Capture status, daemon state, backlog, rate, disk and plugin queues immediately.
ResolutionDeclare evidence gap, reduce unneeded noisy rules or fix throughput, resize based on tests and repeat the affected evidence window.
Backlog grows toward limit while disk is fast.
Likely causeDispatcher/remote plugin is slow or a rule storm exceeds processing.
Inspect auditd state/plugin queue and top rule keys/event rates.
ResolutionFix/decouple plugin and narrow noisy rules through controlled change; do not disable audit.
Rule never matches on x86_64.
Likely causeWrong ABI/syscall table, arch placed after -S, path/exe/filter or success direction is wrong.
Use `ausyscall`, auditctl -l and a synthetic event.
ResolutionCorrect ordering and separate b64/b32 rules based on supported ABI.
Events show auid=unset.
Likely causeKernel thread/service began without an authenticated login session.
Compare uid/euid/ses/exe and rule intent.
ResolutionUse service-specific exe/uid filters; do not attribute it to a human.
ausearch -k finds SYSCALL but no PATH.
Likely causeSyscall/event type may not produce PATH records, item collection failed, or search split event.
Search full serial raw and inspect syscall semantics.
ResolutionReport available evidence without inventing a path; add appropriate exact rule if question requires it.
Audit log fills dedicated filesystem.
Likely causeRetention/rotation, event rate or forwarding failure exceeds capacity.
Inspect auditd.conf actions, rule-key rates, archive/collector and disk.
ResolutionExecute approved space action, preserve evidence and correct sizing/rules; never truncate active logs casually.
augenrules --load fails after immutable enabled.
Likely cause`-e 2` blocks runtime changes until reboot.
Confirm `auditctl -s enabled 2` and staged rule files.
ResolutionSchedule controlled reboot with validated persistent rules; do not attempt to disable runtime audit.
Remote collector has gaps while local logs are complete.
Likely causePlugin/network/TLS/collector queue or retention failed.
Compare event serial/time ranges and plugin/collector health.
ResolutionPreserve local evidence, repair forwarding, document remote gap and test retransmission design.
Recovery
Rollback
Before immutable mode, restore the complete previous reviewed daemon configuration and ordered rule fragments, validate, load atomically and repeat evidence/capacity checks. After `-e 2`, runtime rollback is intentionally unavailable: preserve evidence, stage the previous complete persistent set, schedule console-backed reboot, verify boot rules and continuity, and never disable auditing as a shortcut.
- Stop rollout and preserve kernel status, active/persistent rules, CONFIG_CHANGE events, raw logs, collector ranges and artifact hashes.
- Determine whether `auditctl -s` reports enabled=1 or immutable enabled=2.
- If enabled=1, restore the entire previous root-owned fragment/configuration set and validate it offline.
- Load the prior complete policy atomically with `augenrules --load`; do not delete individual coverage ad hoc.
- If enabled=2, stage the prior complete persistent set without attempting runtime mutation.
- Confirm console access, storage/failure behavior, remote evidence and approved reboot window.
- Reboot, verify early audit start, active semantic rules, enabled state, lost/backlog and remote continuity.
- Run all key fixtures and capacity checks before closing the incident or reenabling broader rollout.
- Retain failed policy and evidence-gap documentation for root-cause and privacy/security review.
Evidence