Create AppArmor profiles and investigate denials
Build an enforce-mode Ubuntu AppArmor profile, review bounded learning evidence, model execution/network boundaries, interpret denials and roll out without unconfined shortcuts.
Confine a custom service with maintainable path/operation policy while preserving required behavior and proving forbidden behavior stays blocked.
- Ubuntu Server 24.04 LTS
- AppArmor Ubuntu 24.04 current
- Ubuntu AppArmor host Ubuntu 24.04 LTS, enabled kernel module, official apparmor/apparmor-utils packages and reliable journal/audit.
aa-status && cat /sys/module/apparmor/parameters/enabled - Stable custom service Root-owned executable/unit, dedicated user and documented path/child/capability/IPC/network contract.
- Isolated learning environment Matching staging host with synthetic data and external side effects blocked; production remains enforce.
- Test and rollback artifacts Positive/negative lifecycle suite, prior profile/application sources and console access.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- An Ubuntu 24.04 LTS AppArmor profile attached to a stable custom service executable, stored in `/etc/apparmor.d`, parsed and loaded reproducibly, enforced in production, and maintained through a package-safe local include.
- A path and operation model that permits only required configuration reads, state/log writes, selected child execution, Unix/network socket families and capabilities, while denying secrets, unrelated homes, uploaded execution and unexpected children.
- A denial-analysis and profile-learning workflow that uses complain mode only on an isolated staging instance for a bounded test plan, reviews every `aa-logprof` proposal, then proves enforcement with positive and negative tests and monitored rollout.
- `aa-status` shows the expected profile loaded and the service PID confined in enforce mode after restart and reboot; no production workflow relies on complain/unconfined/disabled state.
- The profile uses maintainable abstractions, tunables and local overrides without broad recursive or `/** mrwklix` rules. Execute transitions and capabilities are explicit.
- Operators interpret `apparmor=DENIED`, operation, profile, name, requested/denied mask, identity and peer data; they first check DAC, systemd, mount namespace, path aliases and application configuration.
- Required operations succeed while secret reads, static writes, uploaded execution, wrong child programs and unauthorized network/IPC remain blocked by AppArmor or an explicitly documented complementary control.
Architecture
How the parts fit together
Ubuntu's kernel AppArmor LSM matches the service executable attachment and applies the loaded profile. Rules are path-oriented and distinguish read, write, link, lock, memory-map and execute transition modes. Packaged abstractions supply reviewed common runtime access; a local include holds site-specific additions. DAC and systemd mount namespaces apply independently. AppArmor network rules constrain families/types but are not a complete destination firewall, so nftables/systemd socket design retains endpoint restrictions. Kernel audit messages feed journald or auditd. A staging-only complain window executes a finite test plan; proposals are mapped to application intent, narrowed, parsed and then tested in enforce mode.
- Verify kernel/module/profile/audit state and record existing local changes before creating a profile.
- Write an operation/path/child/network/capability contract and install the service with strict DAC/systemd boundaries.
- Create a minimal profile with stable attachment, necessary abstractions and a local include.
- On isolated staging only, use a bounded complain/test window and review denials/proposals one operation at a time.
- Parse/reload the profile, switch to enforce, run positive and negative tests, then reboot and repeat.
- Canary production, monitor denials and audit health, and rollback the exact profile revision without disabling AppArmor.
Assumptions
- The host runs Ubuntu Server 24.04 LTS with AppArmor enabled in the kernel, profiles loaded, official `apparmor` and `apparmor-utils` packages, and healthy journal/audit collection.
- The service uses a stable non-writable executable path and a dedicated Unix account; interpreted runtimes, wrappers and changing symlink targets are modeled explicitly.
- The application contract covers every path alias, mount namespace, child executable, library/plugin, capability, signal, ptrace, D-Bus/Unix socket and network family needed in normal and maintenance operation.
- A matching isolated staging host can enter complain mode for this one profile during a bounded learning window. Production stays enforce.
- External firewall and systemd controls restrict actual addresses/ports and writable paths; AppArmor is not treated as their replacement.
Key concepts
- Attachment
- The executable/path/profile name used by the kernel to decide which AppArmor profile confines a new process.
- Complain mode
- Violations are logged but permitted. It is a temporary staging learning mode, not a production availability fix.
- Enforce mode
- Violations not allowed by the profile are blocked and normally logged.
- Abstraction
- Reusable include containing common rules. It improves maintainability but each included permission still belongs to the profile's authority.
- Execute mode
- `ix`, `px`, `cx`, `ux` and variants define whether a child inherits, transitions to another profile, a child profile, or becomes unconfined. `ux` is especially dangerous.
- Path mediation
- Rules match resolved paths and permissions; aliases, mounts, deleted files and namespaces can affect observed names and require testing.
- Explicit deny
- A deny rule documents/prohibits access and can suppress its audit message, so negative behavior must be tested rather than inferred from logs alone.
Before you copy
Values used in this guide
{{profileName}}Stable AppArmor profile name.
Example: acme-worker{{executablePath}}Exact root-owned service executable.
Example: /usr/local/libexec/acme-worker{{stateDirectory}}Only persistent writable data tree.
Example: /var/lib/acme-worker{{logDirectory}}Application-owned log tree if not using journald.
Example: /var/log/acme-worker{{profilePath}}Main profile file under /etc/apparmor.d.
Example: /etc/apparmor.d/usr.local.libexec.acme-workerSecurity and production boundaries
- Never use `aa-disable`, kernel `apparmor=0`, `ux`, an empty unconfined profile or permanent complain mode as a compatibility fix.
- AppArmor logging can expose internal paths, commands, users and network peers. Restrict raw audit/journal access and redact retained tickets.
- `aa-logprof` observes both legitimate and malicious/mistaken activity. Accepting every suggestion trains the profile on whatever happened during the window.
- Broad abstractions and recursive globs are real permissions. Review included files and prefer separate read-only and writable trees.
- AppArmor path policy supplements DAC, capabilities, systemd sandboxing, firewalling and application authorization. Test every layer separately.
Stop before continuing if
- Stop if AppArmor is disabled, profile/audit state is unknown, messages are lost, or the executable/unit path is mutable or ambiguous.
- Do not use production complain mode to restore service; fix the narrow profile or roll back the application/profile revision.
- Stop if a proposal contains `ux`, broad home/root/filesystem globs, unknown capabilities, arbitrary child execution or network beyond the contract.
- Stop if DAC/systemd/mount/application configuration is the real cause, or an explicit deny is a deliberate security boundary.
- Do not release until enforce/reboot positive tests pass and all defined negative tests remain blocked with no unexplained denial.
verification
Verify kernel AppArmor, loaded profiles, modes, processes, and logging
Capture Ubuntu/kernel/packages, kernel security module state, `aa-status`, loaded profile modes, current confined processes, boot arguments, service unit/executable identity, journal/audit health, local profile changes and last reboot.
Why this step matters
Ubuntu 24.04 integrates AppArmor with the kernel; service status alone does not prove the LSM is enabled or the process attached. `aa-status` separates loaded profiles, modes and confined processes. Boot parameters, parser/kernel feature ABI and package versions determine whether a profile will survive reboot. A lost or absent logging path makes learning/investigation incomplete. Local includes, disable symlinks and cached profiles may differ from source. Record the actual executable through `/proc/PID/exe`, because wrappers and symlinks often explain why a profile did not attach.
What to understand
Record OS/kernel/apparmor/parser package versions.
Inspect kernel enabled parameter and boot command line.
List enforce/complain/unconfined profiles and process attachments.
Check `/etc/apparmor.d/disable`, local includes and cache/source timestamps.
Verify journal/audit retention and no relevant rate/backlog loss.
System changes
- None; records protected baseline evidence.
Syntax explained
aa-status- Summarizes kernel profile and process attachment/mode state.
/sys/module/apparmor/parameters/enabled- Kernel AppArmor enablement indicator.
sudo aa-status && cat /sys/module/apparmor/parameters/enabled && sudo journalctl -k -g 'apparmor=' --since '1 hour ago' --no-pager | tail -n 20apparmor module is loaded. 148 profiles are loaded. 62 profiles are in enforce mode. 0 profiles are in complain mode. 31 processes have profiles defined. 31 processes are in enforce mode. Y
Checkpoint: The kernel will enforce and record the profile
test "$(cat /sys/module/apparmor/parameters/enabled)" = Y && sudo aa-statusContinue whenAppArmor enabled, package/features known, audit/journal healthy, no unexpected complain/disabled state and exact executable/unit identified.
Stop whenKernel/AppArmor disabled, logging unreliable, existing mode/customizations unknown, or executable attachment path is ambiguous.
If this step fails
Service is active but AppArmor filesystem is not mounted.
Likely causeKernel boot disabled AppArmor or unsupported boot state.
Inspect kernel command line and release documentation.
ResolutionRestore AppArmor through approved boot change/reboot before profiling.
Profile is loaded but process is unconfined.
Likely causeAttachment/executable path differs or service predates profile load.
Inspect `/proc/PID/exe`, unit and profile name.
ResolutionCorrect attachment and restart in maintenance window.
Security notes
- Do not expose the full profile/process inventory publicly.
Alternatives
- Use a signed host baseline collector with identical checks.
Stop conditions
- Kernel/AppArmor disabled, logging unreliable, existing mode/customizations unknown, or executable attachment path is ambiguous.
instruction
Write path, execute, capability, IPC, signal, and network contract
Document every startup/request/maintenance file operation, actual resolved path, child executable and desired transition, library/plugin, capability, signal/ptrace, D-Bus/Unix socket and network family/type. Add forbidden secret, home, upload-execute, arbitrary-child and unrelated-peer tests.
Why this step matters
Path-based policy is only as stable as the paths and mount namespaces the process uses. Record real resolved paths, symlinks and package upgrade paths. Execute mode is a privilege boundary: `ix` retains the caller profile, `px` transitions to a named profile, `cx` to a child profile and `ux` removes confinement. Capabilities and network rules can be much broader than files. A contract prevents a learning tool from converting accidental or hostile activity into permission. Maintenance helpers often deserve their own child profile rather than inherited broad authority.
What to understand
Trace `/proc/PID/root`, cwd, mountinfo and unit namespace.
Separate root-owned code/config/static from writable state/cache/log/uploads.
Inventory every exec and interpreter/library/plugin path.
Record exact capability/IPC/signal/peer and complementary firewall controls.
Create positive and behavioral negative tests for each boundary.
System changes
- None; creates profile/test specification.
Syntax explained
ix/px/cx/ux- Inherit, profile transition, child transition and unconfined execute modes.
path rule mode- Specific read/write/map/lock/link/execute permissions, not a generic access bit.
column -t -s, security/acme-worker-apparmor-contract.csvoperation object/path mode/transition required config-read /etc/acme-worker/** r yes state-write /var/lib/acme-worker/** rwk yes helper-exec /usr/lib/acme-worker/renderer px yes secret-read /etc/shadow r no upload-exec /var/lib/acme-worker/uploads/** x no
Checkpoint: Every proposed rule has intent and a test
grep -E 'secret-read|upload-exec|helper-exec' security/acme-worker-apparmor-contract.csvContinue whenComplete paths/aliases, modes, transitions, capabilities, IPC/network, owners and forbidden tests.
Stop whenUnknown dynamic plugins/children/paths or `ux`/broad network/filesystem is requested without redesign.
If this step fails
Runtime uses random versioned paths.
Likely causeDeployment layout lacks stable entry/symlink semantics.
Inspect package/releases and `/proc/PID/exe`.
ResolutionCreate a stable root-owned entry point and explicit version subtree rules.
One process performs admin and Internet-facing work.
Likely causeService responsibilities are not separated.
Classify child paths/capabilities and lifecycle.
ResolutionSplit services/profiles rather than broadening one domain.
Security notes
- Exclude secret values and customer paths from contract examples.
Alternatives
- Use multiple child profiles or separate units for distinct trust roles.
Stop conditions
- Unknown dynamic plugins/children/paths or `ux`/broad network/filesystem is requested without redesign.
config
Install strict DAC and systemd confinement before AppArmor
Install a root-owned stable executable, dedicated non-login account, read-only configuration/static assets, writable state/log directories and a hardened unit. Validate with `namei` and systemd before interpreting AppArmor denials.
Why this step matters
DAC and systemd restrictions can deny the same operation before AppArmor. Making files world-writable to silence EACCES destroys a separate control and may change AppArmor owner-rule semantics. The attachment executable must be root-owned and stable so the confined service cannot replace the program that selects its policy. Separate writable directories prevent broad globs. systemd mount namespaces can expose different paths than the host and must be reflected in the contract. Capabilities should be removed there before deciding AppArmor capability rules.
What to understand
Use a dedicated account and no broad supplemental groups.
Make executable/config root-owned and immutable to service.
Use StateDirectory/LogsDirectory or exact ReadWritePaths.
Drop capabilities and use NoNewPrivileges.
Validate unit and actual namespace paths.
System changes
- Installs service account/files/unit and writable directories.
Syntax explained
ProtectSystem=strict- Read-only service mount view except explicit writable paths.
NoNewPrivileges- Blocks privilege gain through exec.
/etc/systemd/system/acme-worker.service[Service]
User=acme-worker
ExecStart=/usr/local/libexec/acme-worker
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/var/lib/acme-worker /var/log/acme-workerValues stay on this page and are never sent or saved.
sudo install -o root -g root -m 0755 build/acme-worker {{executablePath}} && sudo install -d -o acme-worker -g acme-worker -m 0750 {{stateDirectory}} {{logDirectory}}-rwxr-xr-x root root /usr/local/libexec/acme-worker drwxr-x--- acme-worker acme-worker /var/lib/acme-worker NoNewPrivileges=yes ProtectSystem=strict ReadWritePaths=/var/lib/acme-worker /var/log/acme-worker
Checkpoint: Non-AppArmor controls match the contract
namei -l {{executablePath}} {{stateDirectory}} && systemd-analyze verify /etc/systemd/system/acme-worker.serviceContinue whenStable root-owned entry, narrow service-owned state/log, valid hardened unit and no root/broad capability need.
Stop whenService can replace code/config, needs root, or DAC/systemd cause remains unresolved.
If this step fails
No AppArmor record accompanies denial.
Likely causeDAC/systemd/mount restriction blocked first.
Check namei, unit properties and namespace.
ResolutionFix the narrow underlying control, not profile.
Unit wrapper changes executable path.
Likely causeAttachment targets wrong binary.
Inspect ExecStart and `/proc/PID/exe`.
ResolutionProfile wrapper with transition or execute stable binary directly.
Security notes
- Never use chmod 777 or root service as a diagnostic shortcut.
Alternatives
- Use systemd managed StateDirectory and DynamicUser if application lifecycle supports it.
Stop conditions
- Service can replace code/config, needs root, or DAC/systemd cause remains unresolved.
config
Create a minimal attached profile with local include
Create the profile with global tunables, stable named attachment, the smallest reviewed abstractions, executable/config/static/state/log rules and a local include. Do not add complain flags, `ux`, home globs or generic filesystem access to production source.
Why this step matters
Start with fewer permissions and a complete test plan. `abstractions/base` grants common runtime behavior but should still be understood; stacking convenient abstractions can become broad. The executable needs read/map behavior appropriate to the runtime and attachment. Separate trees prevent write permission leaking to static content. A local include allows site-specific additions without editing a packaged profile, but it is equally privileged and must be versioned/reviewed. Recursive globs must be limited to owned trees. Explicit denies can document boundaries but may not log, so tests remain essential.
What to understand
Name profile independently but bind exact executable.
Inspect every abstraction included by current Ubuntu package.
Use exact or owned subtree globs, avoiding `/home/**` and `/**` allows.
Keep local include empty initially and under configuration management.
Comment rules with contract/test IDs.
System changes
- Installs profile source but does not yet load it into kernel.
Syntax explained
profile name attachment- Named profile applies when exact executable path starts.
#include if exists <local/...>- Optional site override preserved separately.
/etc/apparmor.d/usr.local.libexec.acme-worker#include <tunables/global>
profile acme-worker /usr/local/libexec/acme-worker {
#include <abstractions/base>
/usr/local/libexec/acme-worker mr,
/etc/acme-worker/** r,
/srv/acme-worker/public/** r,
/var/lib/acme-worker/** rwk,
/var/log/acme-worker/** rw,
#include if exists <local/usr.local.libexec.acme-worker>
}Values stay on this page and are never sent or saved.
apparmor_parser -Q profiles/usr.local.libexec.acme-worker && sudo install -o root -g root -m 0644 profiles/usr.local.libexec.acme-worker {{profilePath}} && sudo apparmor_parser -Q {{profilePath}}AppArmor parser: syntax OK Profile: acme-worker Attachment: /usr/local/libexec/acme-worker Includes: abstractions/base, local/usr.local.libexec.acme-worker
Checkpoint: The initial profile is minimal and parseable
sudo apparmor_parser -Q {{profilePath}} && grep -REn '\*\*/.*[x]|ux|/home|/ \*\*' {{profilePath}}Continue whenSyntax valid, exact attachment, reviewed abstractions, separated read/write paths, no ux/broad globs/capabilities/network.
Stop whenParser error, mutable attachment, unreviewed abstraction, broad recursive/execute/ux rule or local override drift.
If this step fails
Parser rejects `include if exists`.
Likely causeParser/package syntax differs.
Check official Ubuntu profile examples and parser version.
ResolutionUse the supported local include syntax for installed release.
Interpreter cannot map required libraries.
Likely causeRuntime paths/abstraction are incomplete.
Capture exact denial and library path in staging.
ResolutionAdd narrow library/abstraction rule after review, not global `/usr/** mr`.
Security notes
- Profile and local include are root policy code; protect ownership/mode/review.
Alternatives
- Start from a maintained vendor profile when one exists for the exact binary.
Stop conditions
- Parser error, mutable attachment, unreviewed abstraction, broad recursive/execute/ux rule or local override drift.
warning
Use complain mode only for a bounded isolated staging test
On a matching isolated host, load the minimal profile, place only this profile in complain mode, clear/mark the test interval, execute the finite positive and negative test plan, and immediately return to enforce. Production never enters complain mode.
Why this step matters
Complain mode permits profile violations, so it cannot protect a production workload and will also record attacker/mistake behavior as learning candidates. Use it only on isolated staging with synthetic data, blocked external side effects, one profile, a short window and a finite test plan. Negative tests in complain mode will be permitted; their events must be marked forbidden and must not become rules. Return to enforce before reviewing/accepting results so an interrupted workflow cannot leave weak mode. Record aa-status before and after.
What to understand
Match kernel/parser/profile packages to production.
Use synthetic data and block external services/network effects.
Mark each test time/ID and expected allow/deny.
Limit complain to exact profile and duration.
Assert enforce immediately after even on test failure.
System changes
- Temporarily permits profile violations for one isolated staging profile, then restores enforce.
Syntax explained
aa-complain- Logs would-be denials but permits them; staging learning only.
aa-enforce- Returns profile to blocking mode.
Values stay on this page and are never sent or saved.
sudo apparmor_parser -r {{profilePath}} && sudo aa-complain {{executablePath}} && { trap 'sudo aa-enforce {{executablePath}}' EXIT; ./tests/acme-worker-contract --learning; rc=$?; sudo aa-enforce {{executablePath}}; trap - EXIT; exit "$rc"; }Setting /usr/local/libexec/acme-worker to complain mode. Executed 18 named positive cases and 7 negative cases. Captured 23 ALLOWED profile violations. Setting /usr/local/libexec/acme-worker to enforce mode.
Checkpoint: Learning ended and enforcement is restored
sudo aa-status | grep -A5 'profiles are in complain mode'Continue whenNamed bounded test evidence exists, negative proposals marked reject, and zero production/target profile remains complain.
Stop whenHost has real secrets/traffic, isolation or test IDs are missing, another profile changes mode, or enforce restoration fails.
If this step fails
Test abort leaves profile complain.
Likely causeCleanup trap/process did not run.
Check aa-status immediately.
ResolutionForce enforce, stop service and correct automation before repeating.
Learning produces unrelated paths.
Likely causeBackground/attack/misconfigured workload ran in window.
Correlate PID/test IDs and reject unmatched events.
ResolutionRepeat on a clean isolated instance; never accept noise.
Security notes
- Complain mode is a temporary loss of mandatory enforcement for that profile.
Alternatives
- Author rules from a complete contract and investigate individual enforce denials without complain mode.
Stop conditions
- Host has real secrets/traffic, isolation or test IDs are missing, another profile changes mode, or enforce restoration fails.
decision
Review aa-logprof proposals one operation at a time
Run `aa-logprof` against the bounded staging interval and the version-controlled staging profile directory, but compare every suggestion with the contract, resolved path, DAC/systemd layer and negative tests. Choose the narrowest exact rule or reject it; the tool writes accepted changes and may reload the staging profile, so inspect the resulting diff and confirm enforce mode immediately.
Why this step matters
`aa-logprof` is an interactive rule authoring assistant, not a security decision engine. It sees events, not business intent, and can propose globs, abstractions or execute modes whose scope exceeds one event. It may include negative-test and malicious behavior. Resolve symlinks/namespaces and check whether DAC/systemd/configuration should change first. Prefer exact owned subtrees and safe child transitions. Review the final profile diff outside the tool, including abstraction contents and local include. Preserve rejected proposals as negative tests.
What to understand
Limit input to exact bounded interval/profile.
Map event to test ID and contract outcome.
Reject unknown, forbidden, secret, home, broad network/capability and ux proposals.
Narrow globs and separate write from execute.
Peer-review final diff and rerun parser/contract tests.
System changes
- Edits the version-controlled staging profile for accepted proposals and may reload that profile; this step is never run against production.
Syntax explained
aa-logprof -f- Reads selected log file and proposes profile updates interactively.
aa-logprof -d- Uses the explicit version-controlled staging profile directory instead of silently editing another profile copy.
reject- Preserves policy boundary when observed behavior is invalid or forbidden.
Values stay on this page and are never sent or saved.
sudo aa-logprof -d "$PWD/profiles" -f /var/log/audit/audit.log && sudo aa-enforce {{executablePath}} && git diff -- profiles/usr.local.libexec.acme-workerProfile: acme-worker Path: /var/lib/acme-worker/cache/item-42 Mode: rw Contract: state cache write — ACCEPT narrowed to /var/lib/acme-worker/cache/** rwk, Path: /etc/shadow Contract: forbidden secret read — REJECT Profile diff reviewed: 6 accepted rules, 17 rejected proposals
Checkpoint: Every accepted proposal is contract-backed and narrow
git diff --check -- profiles/usr.local.libexec.acme-worker && ./tests/profile-lint profiles/usr.local.libexec.acme-workerContinue whenNo ux/broad globs/unknown capabilities; each addition links to positive test and every forbidden proposal remains negative.
Stop whenProposal origin/intent is uncertain, tool suggests unconfined/broad access, or diff includes behavior outside test plan.
If this step fails
Tool suggests an abstraction with many paths.
Likely causeConvenience grouping is broader than event.
Inspect abstraction source and compare each permission.
ResolutionUse exact rule or a smaller reviewed abstraction.
Accepted rule overwrites vendor profile directly.
Likely causeLocal customization ownership is wrong.
Check main/local include packaging.
ResolutionMove site rule to managed local include or own full profile explicitly.
Security notes
- Never run interactive acceptance unattended or pipe yes into aa-logprof.
Alternatives
- Manually author rules from audited events and contract.
Stop conditions
- Proposal origin/intent is uncertain, tool suggests unconfined/broad access, or diff includes behavior outside test plan.
config
Model child execution without escaping confinement
Inventory every child executable and choose `ix` only when the parent profile is appropriate, `px` for an independently named profile, or `cx` for a child profile. Avoid `ux`; deny uploaded/data-tree execution and test interpreter/script combinations.
Why this step matters
Execution changes the code acting with profile authority. `ix` lets child inherit all parent rights, which may be excessive for a complex helper. `px` transitions to another profile, while `cx` can define a child namespace. Uppercase fallback variants can execute differently if target profile is absent and must be understood; fail-closed transitions are generally clearer. `ux` removes confinement and is unacceptable for routine helpers. Interpreted scripts involve both interpreter and script access/attachment. Uploaded or writable files must never be executable under service authority.
What to understand
Inventory execve from application source and bounded tests.
Classify helper trust and required rights separately.
Create/load target child profile before fail-closed transition.
Test absent/corrupt child profile behavior.
Deny execution from all service-writable/upload trees and test it.
System changes
- Adds explicit execution/transition rules and possibly a child profile.
Syntax explained
ix- Child inherits current profile authority.
px/cx- Transitions into independent/namespaced child profile.
ux- Executes unconfined and is prohibited by this design.
/etc/apparmor.d/local/usr.local.libexec.acme-worker/usr/bin/logger ixr,
/usr/lib/acme-worker/renderer rPx -> acme-renderer,
deny /var/lib/acme-worker/uploads/** x,Values stay on this page and are never sent or saved.
grep -REn '[ipcu]x|->|profile ' {{profilePath}} /etc/apparmor.d/local/usr.local.libexec.acme-worker/usr/lib/acme-worker/renderer rPx -> acme-renderer, /usr/bin/logger ixr, deny /var/lib/acme-worker/uploads/** x, No ux rules found
Checkpoint: No child can gain unintended authority
! grep -REn '\bux\b' {{profilePath}} /etc/apparmor.d/local/usr.local.libexec.acme-worker && ./tests/acme-worker-exec-boundariesContinue whenEvery child has justified transition, uploaded/data execution denied, interpreter cases tested and missing profile fails safely.
Stop whenUnknown dynamic execution, ux/fallback escape, writable executable/helper or untested interpreter path exists.
If this step fails
Helper fails under inherited profile.
Likely causeIt needs rights the Internet-facing parent should not have.
Inventory helper-specific operations.
ResolutionCreate a narrow child/separate service profile rather than widen parent.
Script executes through allowed interpreter.
Likely causeInterpreter ix plus readable writable script bypasses upload deny intent.
Test interpreter with writable scripts and inspect rules.
ResolutionRestrict interpreter/script paths or use child profile and non-writable code.
Security notes
- Execution permissions receive the same scrutiny as capabilities.
Alternatives
- Move helper behind a local authenticated service boundary.
Stop conditions
- Unknown dynamic execution, ux/fallback escape, writable executable/helper or untested interpreter path exists.
config
Add only required capabilities, IPC, signals, and network families
Map each non-file operation to an AppArmor rule and complementary control. Grant no capability unless systemd cannot remove the need; constrain signals/ptrace/D-Bus/Unix peers; use the narrow supported network family/type and firewall exact endpoints.
Why this step matters
Capabilities can bypass broad kernel checks and should be eliminated through service design before being allowed. AppArmor network mediation may express family/type/protocol but is not always an exact destination firewall; nftables/proxy owns address/port. Unix, D-Bus, signal and ptrace rules can often identify peers and operations more precisely. A generic `network,` or `capability,` is not least privilege. Signals from systemd/administrator may appear as unconfined peer and should be limited to lifecycle set. Test denied peers and wrong socket families.
What to understand
Remove capabilities in systemd and verify application can use unprivileged port.
List actual socket families/types and peer paths.
Use firewall/proxy for exact remote endpoints.
Limit signal set and peer; deny ptrace unless explicit diagnostic design.
Test wrong peer/family/capability negative paths.
System changes
- Adds selected non-file mediation rules; firewall/systemd changes are separate reviews.
Syntax explained
network inet stream- Permits IPv4 stream socket operations broadly; exact peers require complementary controls.
peer- Constrains supported IPC/signal rule to another label/address identity.
Values stay on this page and are never sent or saved.
grep -REn 'capability|network|unix|dbus|signal|ptrace' {{profilePath}} /etc/apparmor.d/local/usr.local.libexec.acme-workernetwork inet stream, unix (connect, send, receive) type=stream peer=(addr="@acme-metrics"), signal (receive) set=(term, int) peer=unconfined, Capabilities: none
Checkpoint: Non-file authority is explicit and layered
./tests/acme-worker-ipc-network --positive --negative && sudo nft list ruleset | grep acme-workerContinue whenNo capabilities unless justified, only required IPC/signal/network forms, exact endpoint firewall and negative peers blocked.
Stop whenRule uses generic capability/network/ptrace, peer cannot be bounded, or complementary firewall/systemd control is absent.
If this step fails
App works only with capability net_bind_service.
Likely causeIt binds privileged port directly.
Inspect listener/systemd socket/reverse proxy.
ResolutionUse high port/socket activation/reverse proxy rather than capability if feasible.
Network rule permits arbitrary peers.
Likely causeAppArmor grammar is family/type-level for this case.
Test firewall and route/proxy restrictions.
ResolutionDocument and enforce exact egress/ingress outside AppArmor.
Security notes
- Do not claim AppArmor gives destination filtering where rule grammar does not.
Alternatives
- Use a local proxy with a Unix socket so profile can constrain the local peer.
Stop conditions
- Rule uses generic capability/network/ptrace, peer cannot be bounded, or complementary firewall/systemd control is absent.
verification
Parse, reload, and prove the kernel loaded the reviewed revision
Run parser syntax-only checks against the target ABI, inspect includes and diff, reload only the exact profile, then verify `aa-status`, kernel profile mode and service attachment. Record hashes of main/local sources and package versions.
Why this step matters
A profile file on disk does not prove the kernel loaded that revision; parser caches and include changes matter. Syntax-only parsing catches grammar/feature incompatibility before policy replacement. Reload replaces the live profile for matching processes according to kernel semantics and can break service immediately, so use staging/canary and preserve prior source/cache. Explicitly enforce after learning. Hash main and local sources because a benign-looking main profile can import a broad local override.
What to understand
Parse using target host ABI/packages.
Review resolved includes and source/local diff.
Hash and version all source files.
Reload exact profile rather than all profiles.
Confirm loaded enforce mode and service PID attachment.
System changes
- Replaces live kernel profile for the service and sets enforce mode.
Syntax explained
apparmor_parser -Q- Checks/parses without replacing kernel profile.
-r- Replaces an already loaded profile.
Values stay on this page and are never sent or saved.
sudo apparmor_parser -Q {{profilePath}} && sudo apparmor_parser -r {{profilePath}} && sudo aa-enforce {{executablePath}} && sudo aa-status | grep -A2 acme-worker/usr/local/libexec/acme-worker acme-worker (enforce) /usr/local/libexec/acme-worker (4118) acme-worker
Checkpoint: Kernel profile equals reviewed enforce source
sudo aa-status | grep -A3 acme-worker && sha256sum {{profilePath}} /etc/apparmor.d/local/usr.local.libexec.acme-workerContinue whenSyntax/ABI valid, reviewed hashes recorded, profile loaded enforce and exact PID attached.
Stop whenParser warnings/errors, source/hash drift, profile complain/unconfined, wrong PID/attachment or rollback source missing.
If this step fails
Reload succeeds but process still uses old attachment.
Likely causeProcess must restart or exec transition differs.
Inspect aa-status and `/proc/PID/attr/current`.
ResolutionRestart controlled service and verify exact current profile.
Cache/source mismatch after reboot.
Likely causeCache invalidation/include timestamps or packaging is inconsistent.
Reload source, inspect cache and boot logs.
ResolutionFix deployment/cache ownership and repeat reboot test.
Security notes
- Profile reload is a live security change; audit and authorize it.
Alternatives
- Deploy through distro package/config management that invalidates caches correctly.
Stop conditions
- Parser warnings/errors, source/hash drift, profile complain/unconfined, wrong PID/attachment or rollback source missing.
verification
Interpret enforce denials and explicit-deny behavior
Reproduce one test, collect `apparmor=DENIED` event and application error, interpret operation/profile/name/requested and denied masks, identities and network peer fields, then classify it against DAC/systemd/configuration and contract. Remember explicit deny may suppress its audit record.
Why this step matters
An AppArmor record identifies the profile and operation but still does not establish that permission is legitimate. Requested/denied masks show what failed; fsuid/ouid and path help compare owner rules/DAC. Network events contain family/address/port when supported. A denial may be secondary after a wrong configured path. Explicit deny intentionally can avoid log noise, so a missing record does not mean the negative path succeeded—test the application result and profile enforcement. Limit searches to test window/profile/PID and preserve raw event separately.
What to understand
Capture UTC test interval and expected outcome.
Verify profile/PID/executable and resolved name.
Interpret masks, identity, peer and operation.
Check DAC/systemd/mount/config first.
Classify expected deny versus legitimate missing rule.
System changes
- Executes bounded positive/negative tests; expected denied operation has no effect.
Syntax explained
requested_mask/denied_mask- Operation permissions requested and blocked.
profile/name- Confining profile and mediated object path/peer.
sudo journalctl -k --since '-5 min' -g 'apparmor="DENIED"' --no-pageraudit: type=1400 apparmor="DENIED" operation="open" profile="acme-worker" name="/etc/shadow" pid=4118 comm="acme-worker" requested_mask="r" denied_mask="r" fsuid=991 ouid=0 Contract: forbidden secret read; denial expected; no profile change
Checkpoint: Each denial has a root-cause decision
grep -E 'Event:|Contract:|Cause:|Action:' security/apparmor-denial-*.mdContinue whenEvent mapped to test/profile/path/mask/identity and classified as expected, DAC/config/path, or narrow rule gap.
Stop whenEvent mapping is uncertain, logs lost, explicit deny is intentionally protecting boundary, or fix would broaden unrelated paths/operations.
If this step fails
Log says ALLOWED instead of DENIED.
Likely causeProfile remains complain.
Check aa-status and current process profile.
ResolutionReturn enforce and rerun; complain evidence is not acceptance.
No event for explicit deny.
Likely causeExplicit deny logging suppression is expected.
Verify operation failed and process is enforce-confined.
ResolutionRetain behavioral negative test rather than weakening deny.
Security notes
- Redact sensitive names/peers in tickets while retaining protected raw evidence.
Alternatives
- Use auditd search when installed, correlating event serial across records.
Stop conditions
- Event mapping is uncertain, logs lost, explicit deny is intentionally protecting boundary, or fix would broaden unrelated paths/operations.
verification
Run the full positive and negative contract in enforce mode
Restart the exact service, verify profile attachment, run all lifecycle/request/maintenance positives and secret/home/static-write/upload-exec/wrong-child/wrong-peer negatives, monitor service/resources/audit, then reboot and repeat.
Why this step matters
A broad profile can pass every positive test, so containment is measured by negative tests. The test must hit the exact local instance/PID rather than another backend. Lifecycle includes start, stop, reload, log rotation, upgrade and failure recovery because those use paths not exercised by a health request. Reboot validates kernel enablement, profile load ordering, cache and service attachment. Monitor audit/log rate and resource effects; missing logs with failed explicit-deny tests can be normal, while audit loss is not.
What to understand
Prove exact PID/current profile before requests.
Exercise all contract phases and representative concurrency.
Use synthetic secrets/uploads/peers.
Require zero unexplained denies and no audit loss.
Repeat after profile reload, application upgrade and reboot.
System changes
- Runs service and synthetic state changes under enforce; negative operations are blocked.
Syntax explained
require-enforce- Test aborts if profile is complain/unconfined.
reboot repeat- Proves persistent loading and attachment.
Values stay on this page and are never sent or saved.
sudo aa-enforce {{executablePath}} && sudo systemctl restart acme-worker && ./tests/acme-worker-contract --positive --negative --require-enforcePASS profile acme-worker enforce, PID 4118 attached PASS 18 required operations PASS deny /etc/shadow and /home PASS deny static write and upload execute PASS deny unprofiled child and unauthorized Unix peer PASS no unexplained denial; reboot repeat PASS
Checkpoint: Functionality and confinement survive reboot
sudo aa-status | grep -A3 acme-worker && ./tests/acme-worker-contract --assert-report security/apparmor-test.jsonContinue whenAll positives pass, all negatives blocked, enforce/PID exact, no unexplained denials/loss before and after reboot.
Stop whenAny negative succeeds, profile is complain/unconfined, audit unreliable, service hits another backend or denial is unexplained.
If this step fails
Test passes through load balancer but local service failed.
Likely causeRequest reached another backend.
Pin host/port and correlate PID/access log.
ResolutionRun direct isolated canary tests against exact process.
Upgrade fails only under enforce.
Likely causeNew version adds path/child behavior not in contract.
Compare artifact/unit/path and denial to release notes.
ResolutionRollback release or review/update contract/profile; never use complain production.
Security notes
- Synthetic negative fixtures must not contain real credentials.
Alternatives
- Run the same matrix in a VM snapshot before production canary.
Stop conditions
- Any negative succeeds, profile is complain/unconfined, audit unreliable, service hits another backend or denial is unexplained.
verification
Canary rollout with profile-safe rollback
Deploy signed main/local profile and service revision to one host, parse and load before starting, assert enforce attachment, run tests, monitor denials/policy changes and health, then expand gradually. Rollback restores the prior exact profile/application and reloads it; it never disables AppArmor.
Why this step matters
Main profile, local include, abstractions from packages, executable/unit and kernel parser features form one release. Install/parse profile before service start so the process cannot begin unconfined. Canary detects host-specific mounts, package versions and local overrides. Monitor both denial rate and `aa-status`; a missing profile is worse than a quiet log. Rollback is restoring the previous reviewed source and reloading/restarting exact service. Removing/disabling the profile creates an unconfined window and is not acceptable availability recovery.
What to understand
Sign/hash main/local/test/application artifacts.
Validate target package/features and parse before service restart.
Canary enforce positives/negatives/reboot.
Alert on mode/profile source/load changes and denials.
Rehearse exact prior-source reload and service rollback.
System changes
- Deploys/replaces live AppArmor and service artifacts across fleet.
Syntax explained
profile reload- Updates live policy without removing confinement.
canary- Limits scope while real workload reveals missing legitimate paths.
Values stay on this page and are never sent or saved.
sha256sum {{profilePath}} /etc/apparmor.d/local/usr.local.libexec.acme-worker && sudo aa-status | grep -A3 acme-worker && sudo journalctl -k -g 'apparmor="DENIED"' --since recentprofile sha256 93b8...f1 local include sha256 1a09...2c acme-worker (enforce), PID 4118 Canary tests PASS No unexplained AppArmor denials
Checkpoint: Fleet is enforce-confined and reversible
sudo aa-status | grep -A3 acme-worker && ./tests/acme-worker-contract --positive --negative --require-enforceContinue whenSigned sources/hashes converge, exact PIDs enforce-confined, tests pass, no unexplained denials and rollback rehearsal passes.
Stop whenTarget parser/package differs, source/local drift, canary denial or negative success, profile missing/complain, or rollback would disable confinement.
If this step fails
Rollback source fails to parse after package update.
Likely causeAbstraction/feature ABI changed.
Test prior source against new packages before rollout.
ResolutionPause update and create reviewed compatibility revision; do not disable profile.
Only one host emits path denial.
Likely causeMount/symlink/config/local override drift exists.
Compare mountinfo, paths, unit and profile hashes.
ResolutionConverge host or document/review narrow site override.
Security notes
- Audit profile mode/source/load changes to remote protected storage.
Alternatives
- Use deb/RPM-style profile packaging and configuration management for ordered rollback.
Stop conditions
- Target parser/package differs, source/local drift, canary denial or negative success, profile missing/complain, or rollback would disable confinement.
Finish line
Verification checklist
sudo aa-status | grep -A3 acme-workerProfile loaded enforce and exact service PID attached.sudo apparmor_parser -Q {{profilePath}} && sha256sum {{profilePath}}Target parser accepts reviewed source/hash../tests/acme-worker-contract --positive --negative --require-enforceRequired operations pass and forbidden operations remain blocked.sudo journalctl -k -g 'apparmor="DENIED"' --since recent --no-pagerNo unexplained denial and logging pipeline is healthy.Recovery guidance
Common problems and safe checks
aa-status shows no process confined.
Likely causeAttachment path/profile name differs from the actual executable, wrapper or symlink.
Inspect `/proc/PID/exe`, unit ExecStart and loaded profile names.
ResolutionAttach/profile the real stable executable or model the wrapper transition explicitly.
Application gets EACCES but no AppArmor denial.
Likely causeDAC, systemd, mount, seccomp, capability or explicit deny logging behavior caused it.
Check namei, unit sandbox, journal and test expected denial outcome.
ResolutionFix the actual layer; do not widen the AppArmor profile.
aa-logprof proposes access to a secret path.
Likely causeThe learning workload was compromised, misconfigured or used real credentials unexpectedly.
Correlate process/test/operation and application configuration.
ResolutionReject the proposal, correct application behavior and keep negative test.
Profile parses on staging but not production.
Likely causeParser/kernel features, abstractions, tunables or package versions differ.
Compare apparmor_parser version, feature ABI and included files.
ResolutionConverge official packages/profile sources before rollout.
A child works only with `ux`.
Likely causeNo child profile/transition was designed.
Identify child binary, arguments, required files/capabilities and trust.
ResolutionCreate `px`/`cx` transition or separate service; never unconfine it casually.
Profile denial references a different path than configuration.
Likely causeSymlink, bind mount, namespace, deleted file or runtime path resolution differs.
Inspect `/proc/PID/root`, mountinfo, readlink and audit name.
ResolutionModel the actual stable path or remove ambiguous aliases.
Network works too broadly despite profile.
Likely causeAppArmor network rule permits family/type but not exact remote endpoint.
Inspect profile grammar, sockets and firewall.
ResolutionEnforce address/port with firewall/proxy and document complementary boundary.
Explicit deny test has no audit record.
Likely causeAppArmor explicit deny can suppress logging.
Verify operation failure and profile enforcement directly.
ResolutionKeep behavioral negative test; do not remove deny just to create logs.
Recovery
Rollback
Keep AppArmor enabled and the service confined. Stop the canary, restore the previous signed main/local profile and application/unit, syntax-check and reload that profile, restart exact service, then repeat attachment, positive, negative and reboot checks. Never use aa-disable, unconfined/ux or production complain as rollback.
- Pause rollout and preserve denial, profile/mode, source hashes, package and process evidence.
- Stop exact service without removing the loaded profile.
- Restore previous application/unit and main/local profile sources.
- Parse the prior profile against target ABI; resolve compatibility before start.
- Reload prior profile and explicitly enforce it.
- Restart canary and verify `/proc/PID/attr/current`/aa-status attachment.
- Run full positive/negative/reboot tests and monitor denials.
- Retain failed profile evidence and update contract before another revision.
Evidence