Configure Apache request logging and safe rotation
Create a useful virtual-host log format, protect log paths, include timing and forwarded-client context safely, test rotation, and define incident queries.
Produce actionable Apache logs without granting write access to privileged log directories or silently losing records during rotation.
- Apache HTTP Server 2.4.x
- Ubuntu Server 24.04 LTS
- Console and recovery access Keep an independent console session and a copy of the current Apache configuration before changing listeners, modules, or access rules.
- DNS and network plan Resolve every production hostname to the intended public address and list the exact management and client networks that require access.
getent ahosts {{domain}} && ip route show - Port ownership Confirm that TCP port 80 is free or already belongs to the intended Apache instance.
sudo ss -lntp 'sport = :80' - Rollback artifact Snapshot the host or create an archive of the active configuration and web roots before the first material change.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A per-virtual-host Apache access and error logging baseline that records hostname, direct peer, final status, bytes, microsecond duration, request metadata, and a generated request ID.
- A protected root:adm log directory with explicit read access for one operations identity, an idempotent site rewrite, and a distribution-owned rotation policy tested in debug and controlled force modes.
- A small incident-query kit that finds recent 5xx/proxy failures without relying on fragile whitespace field numbers and escapes untrusted control characters before terminal display.
- Successful and missing requests produce concrete records in the correct tenant files with parseable status, byte, duration, and request-ID evidence.
- Apache remains active through a forced rotation, opens current log files, retains rotated evidence, and has no unexpected unlinked file descriptors.
- Deployment users cannot write privileged logs, operations access is explicit, disk/retention ownership is documented, and client-IP trust is not inferred from spoofable headers.
Architecture
How the parts fit together
Apache writes one site-specific access log using the globally named `oneliners_ops` format and one site-specific error log with a bounded module log level. `%a` records the connection peer, while a trusted-proxy design must separately overwrite and validate any forwarded client identity. mod_logio records response bytes and `%D` records microseconds. mod_unique_id supplies a correlation value. Root owns the log directory, adm receives read access, logrotate renames/compresses according to Ubuntu policy, and Apache reopens files through the packaged post-rotation action.
- Apache selects a named vhost, handles or proxies the request, and resolves the final response status.
- The access logger emits the configured fields and generated request ID; the error logger records module/service failures at bounded levels.
- Operators and collectors read files without application write permission and preserve raw evidence according to policy.
- logrotate evaluates size/time/retention, renames the active files, and invokes the packaged reopen action.
- Bounded incident queries select relevant records and escape terminal control characters before display.
Assumptions
- The Apache host has explicit named virtual hosts and known active ErrorLog/CustomLog directives; this guide changes one selected site.
- Ubuntu's packaged `/etc/logrotate.d/apache2` remains the authoritative rotation policy unless a separately reviewed central collector replaces file retention.
- The direct connection peer is meaningful, or a documented trusted reverse proxy overwrites forwarding metadata before Apache/application use.
- Log retention, privacy, access, legal, and central-shipping requirements are defined before increasing captured fields.
- The operations user `{{opsUser}}` is a real managed identity; group changes are reviewed and a new login is expected before supplementary membership applies.
Key concepts
- Final status `%>s`
- The last HTTP status sent after internal redirects, often more useful than the original status.
- Duration `%D`
- Request-processing time in microseconds, useful for relative latency triage but not a complete distributed trace.
- Direct peer `%a`
- The network address connected to Apache; behind a proxy it is the proxy unless remote-IP trust is configured.
- Request ID
- A server-generated correlation value for joining access, error, proxy, and application evidence.
- Log reopen
- The service action that closes a renamed file and begins writing the newly created active path after rotation.
- Unlinked open file
- A deleted/rotated inode still held by a process, consuming disk even though it is absent from directory listings.
Before you copy
Values used in this guide
{{domain}}Virtual host used in per-site access/error filenames and deterministic requests.
Example: api.example.com{{opsUser}}Managed human/service identity granted read-only log access through adm.
Example: oncallSecurity and production boundaries
- Logs are evidence but also sensitive data: URLs, query strings, referrers, user agents, addresses, and errors can contain identifiers, tokens, or attacker-controlled content.
- Keep `/var/log/apache2` non-writable by Apache workers, deployment users, and application accounts; otherwise compromise can erase or forge evidence.
- Do not log Authorization, cookies, request bodies, secrets, or full sensitive query parameters. Redact before central shipping.
- Client headers can contain terminal escape sequences. Use bounded output and escaped rendering such as `sed -n l` during interactive investigation.
- Never trust X-Forwarded-For merely because it exists. Configure a known proxy to overwrite it and define trusted hops before deriving client identity.
- Group access expands who can read every Apache log under the directory; use a narrower ACL or collector when site isolation requires it.
Stop before continuing if
- Stop if active log destinations, ownership, disk pressure, rotation behavior, or central collectors are unknown.
- Do not reload if the format/parser fails, a site gets duplicate CustomLog destinations, or another tenant is modified.
- Do not grant adm access to an unmanaged/shared account or make the log directory world-readable/writable.
- Stop rotation rehearsal if retention policy would remove required evidence, disk headroom is insufficient, or Apache's reopen action is missing.
- Do not accept logging until 2xx/404 records, proxy/application errors, request IDs, byte/duration fields, and rotation recovery are proven.
verification
Inventory log destinations and ownership
List every ErrorLog and CustomLog directive, directory mode, disk usage, rotation policy, and current write handles.
Why this step matters
Logging changes are unsafe when current destinations, file handles, owners, retention, capacity, and collectors are unknown; inventory also prevents silent duplicate logging.
What to understand
Dump runtime log configuration and search enabled/available files for ErrorLog, CustomLog, piped logs, syslog, and conditional environments.
Record directory/file ownership, mount/inode capacity, largest logs, open handles, collector state, rotation history, and failed units.
Run logrotate debug because it reads policy without rotating; inspect the packaged postrotate action and retention before forcing anything.
System changes
- No persistent change; reads Apache, filesystem, process, and logrotate state.
Syntax explained
DUMP_RUN_CFG- Prints effective runtime configuration details including log information.
stat- Shows directory owner, group, and mode.
du -sh- Summarizes current Apache log disk use.
logrotate --debug- Evaluates policy without rotating or updating state.
sudo apache2ctl -t -D DUMP_RUN_CFG | grep -i log || true; sudo grep -R '^[[:space:]]*\(ErrorLog\|CustomLog\)' /etc/apache2; sudo stat -c '%A %U %G %n' /var/log/apache2; sudo du -sh /var/log/apache2; sudo logrotate --debug /etc/logrotate.d/apache2drwxr-x--- root adm /var/log/apache2 12M /var/log/apache2
Checkpoint: Checkpoint: inventory
Continue whenEvery active destination/collector, owner, volume, retention rule, reopen action, and disk/inode margin is known.
Stop whenUnknown piped/remote logs, disk pressure, world-writable evidence paths, missing rotation, or unowned collectors exist.
If this step fails
Runtime dump and file search disagree.
Likely causeDisabled files, conditional includes, generated config, or another Apache binary/root is involved.
sudo apache2ctl -Vsudo apache2ctl -t -D DUMP_INCLUDESsudo grep -R -nE 'ErrorLog|CustomLog' /etc/apache2
ResolutionIdentify the authoritative include graph and active binary before editing.
Security notes
- Inventory output can expose internal paths, users, and log destinations; protect it.
Alternatives
- Use configuration-management inventory only when it is reconciled with live file handles and runtime dumps.
Stop conditions
- Unknown piped/remote logs, disk pressure, world-writable evidence paths, missing rotation, or unowned collectors exist.
command
Back up logging and rotation configuration
Preserve Apache log directives and the distribution logrotate policy before introducing a custom format.
Why this step matters
Format definitions, site destinations, module state, and rotation policy must roll back together without deleting the evidence created during the change.
What to understand
Archive `/etc/apache2` and the exact logrotate file, list contents, verify checksum, and record current module/site state.
Do not include active log bodies in a general configuration archive unless evidence retention and sensitive-data handling explicitly require it.
Rollback should restore directives, not erase new/rotated logs that may be needed for diagnosis.
System changes
- Creates a root-owned configuration archive and integrity checksum.
Syntax explained
tar -C /- Stores canonical relative paths for controlled restore.
-czf- Creates a compressed archive at the explicit protected path.
sha256sum- Records archive integrity.
sudo tar -C / -czf /root/apache-logging-before.tgz etc/apache2 etc/logrotate.d/apache2 && sudo sha256sum /root/apache-logging-before.tgz70ca... /root/apache-logging-before.tgz
Checkpoint: Checkpoint: backup
Continue whenThe archive contains Apache and rotation configuration, is protected, and restores into a parser-before-reload workflow.
Stop whenThe archive is incomplete/unprotected, config is generated elsewhere, or rotation state/collector rollback is absent.
If this step fails
The archive changes while being read.
Likely causeA deployment or management agent is updating config concurrently.
ps -ef | grep -E '[a]nsible|[c]hef|[p]uppet|[t]ar'sudo tar -tzf /root/apache-logging-before.tgz | head
ResolutionCoordinate with the authoritative deployment, take a consistent snapshot, and verify it.
Security notes
- Configuration may expose internal topology and auth paths; keep the archive root-only.
Alternatives
- Use a signed immutable config release with tested rollback and retain the same evidence.
Stop conditions
- The archive is incomplete/unprotected, config is generated elsewhere, or rotation state/collector rollback is absent.
config
Define an operations-focused log format
Include virtual host, final status, duration, request ID, bytes, user agent, and the direct peer. Trust X-Forwarded-For only when a known proxy overwrites it.
Why this step matters
A named format provides stable semantics and enough operational context without copying sensitive request bodies, cookies, credentials, or spoofable forwarded identities.
What to understand
`%v` identifies the selected tenant, `%a` the direct peer, `%>s` final status, `%O` response bytes, `%D` microseconds, and UNIQUE_ID a server-generated correlation value.
Quoted request, referrer, and user-agent fields can contain attacker input; downstream parsing must understand escaping and terminal output must be sanitized.
Enable logio and unique_id explicitly, keep the format file root-owned, and audit for an existing format name before replacement.
System changes
- Creates/enables one global named format and enables logio/unique_id modules; no site uses it until the next step.
Syntax explained
%%v- Selected canonical virtual-host name.
%%a- Direct client/proxy peer address.
%%>s- Final response status after internal redirects.
%%O- Network bytes sent, supplied by mod_logio.
%%D- Request duration in microseconds.
%%{UNIQUE_ID}e- Server-generated correlation environment value.
/etc/apache2/conf-available/50-operations-log.confprintf 'LogFormat "%%v %%a %%l %%u %%t \\"%%r\\" %%>s %%O %%D \\"%%{Referer}i\\" \\"%%{User-Agent}i\\" id=%%{UNIQUE_ID}e" oneliners_ops\n' | sudo tee /etc/apache2/conf-available/50-operations-log.conf && sudo a2enmod logio unique_id && sudo a2enconf 50-operations-logEnabling module logio. Enabling module unique_id. Enabling conf 50-operations-log.
Checkpoint: Checkpoint: format
Continue whenThe format parses, modules load once, and the name is defined exactly once with reviewed non-secret fields.
Stop whenThe name collides, module load fails, sensitive fields are added, or client identity semantics are misunderstood.
If this step fails
Configtest reports unknown `%O` or empty request IDs.
Likely causemod_logio or mod_unique_id is not loaded for the active Apache instance.
sudo apache2ctl -M | grep -E 'logio|unique_id'sudo apache2ctl configtest
ResolutionEnable/verify the exact modules and active config root before site activation.
Security notes
- More fields increase privacy and injection risk; capture only operationally justified data.
Alternatives
- Use a supported structured logging module/schema when exact machine parsing is mandatory.
Stop conditions
- The name collides, module load fails, sensitive fields are added, or client identity semantics are misunderstood.
config
Assign per-site access and error logs
Replace existing site log destinations idempotently with the reviewed format, then add one marked per-module error level that captures proxy failures without globally enabling noisy debug logging.
Why this step matters
Per-site files establish tenant ownership and keep the selected format from duplicating global logs; idempotent replacements avoid repeated directives during reruns.
What to understand
The sed expressions replace existing top-level site ErrorLog/CustomLog lines with deterministic destinations rather than appending another copy.
A sentinel inserts the bounded LogLevel once. `proxy:info` can be noisy; use it only for a proxy site and measure volume.
Inspect the diff and every active vhost block; generated/multi-vhost files may require a declarative/manual edit instead.
System changes
- Replaces the selected site's log destinations and adds one marked module-specific LogLevel.
Syntax explained
CustomLog … oneliners_ops- Writes access records in the reviewed named format.
ErrorLog- Sends this site's server/module errors to a dedicated file.
LogLevel warn proxy:info- Keeps general warnings while increasing proxy detail for this vhost.
grep sentinel || sed- Prevents duplicate LogLevel insertion.
/etc/apache2/sites-available/{{domain}}.confValues stay on this page and are never sent or saved.
sudo sed -i -E 's#^[[:space:]]*CustomLog .*$# CustomLog ${APACHE_LOG_DIR}/{{domain}}-access.log oneliners_ops#; s#^[[:space:]]*ErrorLog .*$# ErrorLog ${APACHE_LOG_DIR}/{{domain}}-error.log#' /etc/apache2/sites-available/{{domain}}.conf && sudo grep -q 'ONELINERS LOGLEVEL' /etc/apache2/sites-available/{{domain}}.conf || sudo sed -i '\#</VirtualHost>#i\ # ONELINERS LOGLEVEL\n LogLevel warn proxy:info' /etc/apache2/sites-available/{{domain}}.confErrorLog /var/log/apache2/example.com-error.log CustomLog /var/log/apache2/example.com-access.log oneliners_ops
Checkpoint: Checkpoint: site logs
Continue whenOne intended vhost writes one access and one error destination with one marked log level; no other tenant changes.
Stop whenThe file hosts multiple trust domains, generated config overwrites edits, duplicate logs remain, or diff crosses another block.
If this step fails
The sed command changes a commented example or wrong block.
Likely causeThe site file is nonstandard or contains multiple virtual hosts.
sudo git diff --no-index /etc/apache2/sites-available/api.example.com.conf.before /etc/apache2/sites-available/api.example.com.conf || truesudo apache2ctl -S
ResolutionRestore the archive and manage the complete vhost declaratively with explicit log directives.
Security notes
- Separate filenames are not access isolation if all readers retain directory-wide access.
Alternatives
- Use a site-specific include managed by the authoritative deployment when editing the main file is fragile.
Stop conditions
- The file hosts multiple trust domains, generated config overwrites edits, duplicate logs remain, or diff crosses another block.
verification
Protect the log directory
Keep log directories root-owned and non-writable by deploy users; grant read access through the adm group rather than loosening mode bits.
Why this step matters
Logs must be writable through Apache's privileged open/reopen lifecycle but immutable to application/deployment identities; read access is an audited operations privilege.
What to understand
Set the directory root:adm 0750 and inspect existing log-file modes plus rotation `create` behavior; directory mode alone does not rewrite all files.
Append one managed operations user to adm only after reviewing access to other system logs included by that group.
A new login is needed for supplementary membership; verify with `id` and remove access through account management when duties end.
System changes
- Normalizes directory owner/mode and changes supplementary groups for the selected operations identity.
Syntax explained
chown root:adm- Keeps administrative ownership and assigns read group.
chmod 0750- Allows owner full and group traversal/read, denying other users.
usermod --append --groups adm- Adds adm without replacing existing supplementary groups.
Values stay on this page and are never sent or saved.
sudo chown root:adm /var/log/apache2 && sudo chmod 0750 /var/log/apache2 && sudo usermod --append --groups adm {{opsUser}} && sudo stat -c '%A %U %G %n' /var/log/apache2drwxr-x--- root adm /var/log/apache2
Checkpoint: Checkpoint: permissions
Continue whenThe directory is root:adm 0750, active/rotated files follow policy, app/deploy users cannot write, and only approved operators read.
Stop whenThe user is shared/unmanaged, adm is too broad, collectors lose access, or active file modes conflict with rotation.
If this step fails
The operator still receives permission denied.
Likely causeThe session predates group membership, file mode differs, or an ACL denies access.
id oncallnamei -l /var/log/apache2/api.example.com-access.loggetfacl /var/log/apache2 2>/dev/null || true
ResolutionStart a new authenticated session and correct the narrow policy; never chmod 777.
Security notes
- adm may read sensitive logs beyond Apache; prefer a narrow ACL/group when required.
Alternatives
- Grant a dedicated collector group/ACL or expose sanitized incident views through controlled tooling.
Stop conditions
- The user is shared/unmanaged, adm is too broad, collectors lose access, or active file modes conflict with rotation.
verification
Reload and generate deterministic records
Validate syntax, reload, make successful and missing requests, then confirm status and timing fields appear in the expected per-site files.
Why this step matters
Parser success plus deterministic 2xx/404 requests proves that the selected vhost opens the files and produces the expected status, byte, duration, and correlation fields.
What to understand
Parse before reload, then request by real hostname with certificate verification. Record the before/after file offsets.
Generate a known success and missing path; do not use a secret URL. Render sample output escaped during terminal review.
Check error logs and journal for open permission, unknown format, module, or reload errors.
System changes
- Reloads Apache and creates deterministic access/error log records.
Syntax explained
apache2ctl configtest- Rejects malformed format/site directives before activation.
curl --output /dev/null- Generates a request without displaying the body.
tail -n 2- Bounds evidence to the two deterministic requests.
Values stay on this page and are never sent or saved.
sudo apache2ctl configtest && sudo systemctl reload apache2 && curl --silent --output /dev/null https://{{domain}}/ && curl --silent --output /dev/null https://{{domain}}/missing-oneliners && sudo tail -n 2 /var/log/apache2/{{domain}}-access.logexample.com 127.0.0.1 - - [date] "GET / HTTP/2.0" 200 1234 5421 example.com 127.0.0.1 - - [date] "GET /missing-oneliners HTTP/2.0" 404
Checkpoint: Checkpoint: validate
Continue whenSyntax is valid and two new site records contain correct host, peer, quoted request, 200/404, bytes, duration, and nonempty request IDs.
Stop whenRecords go elsewhere/duplicate, fields shift or disappear, permissions fail, or another tenant/log changes.
If this step fails
Apache reload succeeds but the new files do not exist.
Likely causeNo request selected the vhost, paths are invalid, or the site remains disabled.
sudo apache2ctl -Ssudo grep -R -n CustomLog /etc/apache2/sites-enabledsudo journalctl -u apache2 -n 100 --no-pager
ResolutionCorrect the active vhost and log path, then repeat named requests.
Security notes
- Use non-secret deterministic paths and escape displayed attacker-controlled fields.
Alternatives
- Validate on a canary site and switch the format declaratively after evidence passes.
Stop conditions
- Records go elsewhere/duplicate, fields shift or disappear, permissions fail, or another tenant/log changes.
warning
Rehearse log rotation
Run debug mode first, then force one controlled rotation and verify Apache reopens log files without losing service.
Why this step matters
Debug mode validates policy safely; one controlled force proves rename/create/reopen behavior before production retention depends on it.
What to understand
Inspect debug output for file matches, frequency, rotate count, compression, create mode, and postrotate action.
Force rotation only in a controlled window because it updates state and can compress/delete according to policy.
Afterward generate a request, confirm the active file grows, Apache is healthy, rotated file is retained, and no large unlinked handles remain.
System changes
- Updates logrotate state, rotates matched Apache files, may compress/delete old generations, and triggers Apache reopen.
Syntax explained
--debug- Evaluates policy without changing files/state.
--force- Rotates eligible configured logs regardless of elapsed schedule.
lsof +L1- Finds processes holding unlinked files.
Values stay on this page and are never sent or saved.
sudo logrotate --debug /etc/logrotate.d/apache2 && sudo logrotate --force /etc/logrotate.d/apache2 && sudo systemctl is-active apache2 && sudo lsof +L1 | grep apache2 || true; sudo ls -lh /var/log/apache2/{{domain}}-*reading config file /etc/logrotate.d/apache2 active example.com-access.log example.com-access.log.1
Checkpoint: Checkpoint: rotation
Continue whenPolicy matches intended files, active and .1 files exist, a post-rotation request reaches the active file, service stays active, and no unexpected unlinked handle remains.
Stop whenDebug predicts deletion beyond policy, postrotate is missing, disk headroom is low, collector is not rotation-aware, or service/reopen fails.
If this step fails
Force rotation removes more history than expected.
Likely causeRotate count, glob, dateext/compression, or shared policy differs from assumption.
sudo sed -n '1,200p' /etc/logrotate.d/apache2sudo cat /var/lib/logrotate/status | grep apache2
ResolutionRestore retained evidence if available, correct and debug the policy, and do not force again until reviewed.
Security notes
- Rotation must preserve evidence permissions and retention; do not use copytruncate casually because it can lose records.
Alternatives
- Let a local collector own rotation/retention after a tested cutover from file-based policy.
Stop conditions
- Debug predicts deletion beyond policy, postrotate is missing, disk headroom is low, collector is not rotation-aware, or service/reopen fails.
instruction
Record high-value incident queries
Provide bounded commands for recent 5xx responses, slow requests, top client peers, and startup errors without parsing arbitrary untrusted control characters in a terminal.
Why this step matters
Operators need short, reproducible triage commands that match the actual format and do not interpret attacker-controlled bytes as terminal control.
What to understand
Match the quoted-request/status boundary instead of a fixed whitespace field because timestamp and request contain spaces.
Bound output with tail and escape nonprinting bytes with `sed -n l`; preserve raw logs unchanged for evidence.
Combine recent service journal, site errors, and access outcomes by time/request ID, then move to structured tooling for broad analysis.
System changes
- No persistent change; reads recent bounded evidence.
Syntax explained
grep -aE '" 5[0-9]{2} '- Matches a tested quote/status boundary for 5xx in this format.
tail -n 20- Bounds interactive output.
sed -n l- Escapes nonprinting and control characters.
journalctl --since- Bounds service events by a relative time window.
Values stay on this page and are never sent or saved.
sudo grep -aE '" 5[0-9]{2} ' /var/log/apache2/{{domain}}-access.log | tail -n 20 | sed -n l; sudo journalctl -u apache2 --since '-30 min' --no-pager; sudo grep -a 'proxy:error' /var/log/apache2/{{domain}}-error.log | tail -n 20 | sed -n l203.0.113.20 ... "GET /api HTTP/2.0" 502 [proxy:error] AH01102: error reading status line from remote server
Checkpoint: Checkpoint: incident queries
Continue whenKnown test 5xx/proxy errors are found, output is bounded/escaped, and request IDs/timestamps support correlation.
Stop whenThe format differs, output is unbounded, raw control bytes render, clock/timezone mismatch prevents correlation, or access is unauthorized.
If this step fails
The query misses a visible 502 line.
Likely causeThe active format has different quoting/status order or log compression/rotation moved the record.
sudo head -n 1 /var/log/apache2/api.example.com-access.log | sed -n lsudo zgrep -a ' 502 ' /var/log/apache2/api.example.com-access.log* | tail
ResolutionValidate the actual named format and use a parser/query matched to its grammar and rotated files.
Security notes
- Incident queries should not paste unredacted tokens, URLs, or identities into tickets/chats.
Alternatives
- Use a schema-aware central query with role-based access and documented redaction.
Stop conditions
- The format differs, output is unbounded, raw control bytes render, clock/timezone mismatch prevents correlation, or access is unauthorized.
Finish line
Verification checklist
curl --silent --output /dev/null https://{{domain}}/ && sudo tail -n 1 /var/log/apache2/{{domain}}-access.logA new line contains the virtual host, status, bytes, and duration.sudo logrotate --debug /etc/logrotate.d/apache2 && systemctl is-active apache2Rotation policy parses and Apache remains active.Recovery guidance
Common problems and safe checks
The access file remains empty after requests.
Likely causeAnother vhost handled the request, CustomLog uses another path/condition, Apache cannot open the file, or the reload failed.
sudo apache2ctl -Ssudo grep -R -n CustomLog /etc/apache2/sites-enabledsudo journalctl -u apache2 -n 100 --no-pagersudo lsof /var/log/apache2
ResolutionCorrect the selected vhost/path and directory ownership, parse, reload, and issue a named local request.
Every request is logged twice.
Likely causeThe site contains two CustomLog directives or a global and site log both capture the same traffic.
sudo grep -R -n CustomLog /etc/apache2sudo apache2ctl -t -D DUMP_RUN_CFG
ResolutionChoose intentional global versus per-site destinations and remove the duplicate after preserving the prior config.
Rotated files exist but Apache keeps writing to the old inode.
Likely causeThe packaged postrotate action failed, service control differs, or a custom policy bypassed reopen.
sudo logrotate --debug /etc/logrotate.d/apache2sudo lsof +L1 | grep apache2sudo journalctl -u apache2 -n 100 --no-pager
ResolutionRepair the packaged graceful/reopen action and rehearse again; do not delete an open large inode as a disk-pressure workaround.
The apparent client IP is always the load balancer.
Likely causeThe format correctly logs the direct peer, but trusted remote-IP configuration has not been designed.
sudo apache2ctl -M | grep remoteipsudo grep -R -n RemoteIP /etc/apache2Review load-balancer overwrite behavior
ResolutionConfigure mod_remoteip only with exact trusted proxy addresses and an overwritten header; retain original peer evidence if required.
A log line changes terminal state or displays garbage.
Likely causeAn attacker-controlled URI/header contains escape or control bytes.
sudo tail -n 20 /var/log/apache2/api.example.com-access.log | sed -n l
ResolutionUse escaped/noninteractive tooling, preserve raw evidence, and fix downstream parser/terminal handling rather than deleting records.
Reference
Frequently asked questions
Why not parse the status with `$9` in awk?
The timestamp and quoted request contain spaces, so whitespace field numbers are fragile. Match a tested log grammar or use a structured parser.
Why log both access and error files?
Access logs show protocol outcomes and timing; error logs provide module/upstream causes. Incidents usually need both.
Does adm membership apply immediately?
Usually a new login/session is needed to receive supplementary groups; verify with `id` rather than broadening directory modes.
Recovery
Rollback
Restore the previous format and site directives while retaining generated logs for incident evidence.
- Restore configuration from /root/apache-logging-before.tgz without deleting rotated log evidence.
- Run apache2ctl configtest and reload Apache.
- Remove the operations user from adm only after confirming they no longer require log access.
- Do not change /var/log/apache2 to a world-writable mode during rollback.
Evidence