Configure nginx structured operations logging
Log request timing and upstream timing per site, protect log paths, validate rotation, and prepare bounded incident queries for 4xx, 5xx, and slow requests.
Make nginx failures diagnosable without leaking secrets into logs or losing records during rotation.
- nginx 1.24+
- Ubuntu Server 24.04 LTS
- Console and recovery access Keep an independent console session and a copy of the current nginx configuration before changing listeners, modules, or access rules.
- DNS and network plan Resolve every production hostname to the intended public address and list the exact management and client networks that require access.
getent ahosts {{domain}} && ip route show - Port ownership Confirm that TCP port 80 is free or already belongs to the intended nginx instance.
sudo ss -lntp 'sport = :80' - Rollback artifact Snapshot the host or create an archive of the active configuration and web roots before the first material change.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A reviewed, site-scoped nginx access log that records stable operational fields such as ISO time, normalized URI, status, response bytes, end-to-end duration, upstream destination and duration, and a request correlation identifier.
- A protected per-site log layout with deliberate reader membership, bounded buffering, the packaged rotation/reopen path, concrete commissioning events, and portable incident queries that do not interpret untrusted log text as terminal control sequences.
- A repeatable evidence and rollback procedure that distinguishes nginx parser success, filesystem permissions, buffer flush, request generation, rotation behavior, and application/upstream failures instead of treating a non-empty log file as sufficient proof.
- A known 200, a known 404, and any controlled upstream request produce concrete records in the intended site file with the documented fields and without query strings, request bodies, Cookie, or Authorization values.
- nginx validates and reloads cleanly, the log directory remains root-owned and non-writable by workers or deployers, authorized operators can read after a new login, and logrotate reopens files without deleted handles.
- Operators have bounded, portable commands for recent 5xx responses, requests slower than two seconds, and recent nginx unit messages, plus clear stop conditions when rotation, permissions, disk capacity, or privacy policy is unsafe.
Architecture
How the parts fit together
The HTTP log module defines a named `oneliners_ops` format once in the nginx `http` context through `/etc/nginx/conf.d`. A reviewed target server block selects that format for its own access file and a warning-level error file. nginx workers open the files through the privileged master, buffer access records in memory for at most five seconds, and reopen paths when the packaged logrotate policy signals the service. Root owns `/var/log/nginx`; the `adm` group supplies read/traverse access to a named operations account without granting write access. Commissioning requests exercise the complete client → selected server → optional upstream → response → buffered write path. Incident queries parse only stable `key=value` fields and cap output before displaying escaped control characters.
- A client request reaches the exact nginx server block and receives a request identifier from nginx.
- nginx serves locally or contacts an upstream while recording status, byte count, total duration, upstream address/status/duration, peer, host, method, and normalized URI.
- The access record enters a bounded buffer and is flushed within five seconds; errors are written to the separate warning-level file.
- logrotate renames/compresses according to the packaged policy and requests nginx to reopen active files rather than deleting a still-open inode.
- An operator reads the protected files or journal with bounded queries, treats values as untrusted text, and correlates a request ID with application evidence.
Assumptions
- The Ubuntu 24.04 nginx baseline is healthy, the intended site file is known, and its expanded configuration contains exactly one site-scoped `access_log` and one `error_log` directive before this guide edits it.
- The domain is already routed to this server block, TLS and upstream behavior are tested separately, and a controlled 200 plus 404 can be generated without affecting customer data.
- The operations user is a managed human or service identity. Adding it to `adm` is acceptable under local policy, and the session can be restarted so supplementary groups are refreshed.
- The host has disk headroom for current and rotated logs, `/etc/logrotate.d/nginx` is the authoritative packaged policy, and no external agent is simultaneously rewriting the same nginx files.
- The chosen fields satisfy privacy and retention policy. If full URI/query, headers, body, client identity, or user-generated fields are later added, they require a separate data-protection and terminal-safety review.
Key concepts
- log_format
- An HTTP-context nginx directive that assigns a reusable name to an access-record template; defining it does not write anything until an `access_log` selects it.
- $uri
- The normalized current URI path used here instead of `$request_uri`, intentionally excluding the original query string where tokens and personal data often appear.
- $request_time
- Elapsed seconds from reading the first client bytes until the last response bytes are sent to the log stage.
- $upstream_response_time
- Time spent receiving responses from selected upstreams; it can contain multiple values or `-`, so it is evidence rather than a guaranteed scalar.
- $request_id
- An nginx-generated hexadecimal identifier useful for correlating proxy, application, and incident evidence when it is also forwarded deliberately.
- buffer and flush
- A performance trade-off where access records may remain in process memory until the buffer fills or the configured maximum flush interval expires.
- reopen
- The signal-driven operation that closes old log descriptors and opens current paths after rotation, avoiding writes into renamed or deleted files.
Before you copy
Values used in this guide
{{domain}}Exact site hostname and safe filename prefix for the reviewed server block and its logs.
Example: app.example.com{{opsUser}}Managed operator identity allowed to read Ubuntu administration logs through supplementary group membership.
Example: aliceSecurity and production boundaries
- Logs are attacker-influenced input. Host, URI and request-related values can contain misleading or terminal-control content; keep queries bounded and render non-printable characters visibly before pasting output into tickets.
- Do not add Authorization, Cookie, Set-Cookie, request/response bodies, unredacted query strings, session identifiers, password-reset links, or API tokens to a general operations log.
- Membership in Ubuntu's `adm` group grants access to more than one nginx site and often to other system logs. Use per-service ACLs, a forwarding agent, or a narrower group where separation is required.
- The nginx worker and deployment account must not write `/var/log/nginx`. Writable privileged logs enable evidence destruction, symlink attacks, or forged operational records after an application compromise.
- Request identifiers aid correlation but are not authentication or proof of origin. Preserve timestamps, host identity, rotation state, and upstream evidence when incident integrity matters.
Stop before continuing if
- Stop if the site file has zero or multiple access/error directives, another config manager owns it, `nginx -T` shows the directive in the wrong context, or the target hostname selects a different block.
- Do not reload when `nginx -t` fails, the format is undefined/duplicated, log paths escape `/var/log/nginx`, or permissions would make the directory worker/deployer writable.
- Stop if disk capacity or retention is unknown, logrotate debug reports an error, the service cannot reopen logs, or deleted nginx file handles remain after a controlled rotation.
- Do not accept records containing secrets, query tokens, raw control characters, unexpected cross-tenant traffic, or missing operational fields.
verification
Inventory active nginx logs
List expanded access_log/error_log directives, ownership, disk use, open handles, and rotation policy.
Why this step matters
The effective expanded configuration, open file paths, ownership, capacity, and rotation policy must be known before editing because the visible site file may be overridden or generated elsewhere.
What to understand
Review `nginx -T` as configuration evidence, not just individual snippets. Note every format and access/error directive that applies to the intended server.
Record filesystem mode, current volume, deleted handles, filesystem headroom, and the packaged rotation rule. A technically valid format is unsafe when storage or reopen behavior is unknown.
Resolve the exact site file from `server_name` and includes. This guide intentionally refuses a search-and-replace across all nginx configuration.
System changes
- No persistent change; reads nginx's expanded configuration, log metadata, storage usage, open handles, and rotation policy.
Syntax explained
nginx -T- Parses and prints the full include-expanded nginx configuration.
stat -c- Displays exact mode, owner, group and path rather than a vague directory listing.
logrotate --debug- Parses and simulates policy decisions without rotating files.
sudo nginx -T 2>&1 | grep -E 'access_log|error_log|log_format'; sudo stat -c '%A %U %G %n' /var/log/nginx; sudo du -sh /var/log/nginx; sudo logrotate --debug /etc/logrotate.d/nginxdrwxr-x--- root adm /var/log/nginx 8.0M /var/log/nginx
Checkpoint: Checkpoint: inventory
Continue whenOne intended site, known existing directives, protected paths, adequate capacity, and a valid packaged rotation rule are recorded.
Stop whenConfiguration ownership, site selection, storage capacity, current log paths, or rotation/reopen behavior cannot be determined.
If this step fails
The expanded configuration contains several matching access_log lines.
Likely causeLogging is inherited and overridden at multiple contexts or the hostname exists in multiple server blocks.
sudo nginx -T 2>&1 | grep -n -E 'server_name|access_log|error_log'
ResolutionMap each directive to its exact HTTP/server/location context and choose one explicit site-level policy before editing.
Security notes
- Inventory output can reveal internal hostnames, upstreams and paths; store it with administrative access.
Alternatives
- Export the expanded config and log-policy metadata through a read-only configuration-management audit if direct host review is prohibited.
Stop conditions
- Configuration ownership, site selection, storage capacity, current log paths, or rotation/reopen behavior cannot be determined.
command
Back up logging configuration
Preserve nginx configuration and the packaged logrotate file.
Why this step matters
Logging changes cross nginx and logrotate policy, so recovery needs a verified archive created before mutation rather than an assumption that package defaults remain available.
What to understand
The archive captures nginx configuration and the active packaged rotation file. Verify its member list and checksum and keep it root-readable.
Do not place active logs in the configuration archive; they can be large and are evidence governed by a separate retention/backup policy.
Record whether a controller will overwrite local changes. The correct rollback may be a prior signed deployment revision rather than extracting files by hand.
System changes
- Creates `/root/nginx-logging-before.tgz` and its checksum; it does not alter running nginx.
Syntax explained
tar -C /- Stores stable relative paths from the filesystem root.
etc/nginx- Includes the complete nginx configuration graph.
etc/logrotate.d/nginx- Includes the packaged rotation/reopen policy being tested.
sudo tar -C / -czf /root/nginx-logging-before.tgz etc/nginx etc/logrotate.d/nginx && sudo sha256sum /root/nginx-logging-before.tgzaba2... /root/nginx-logging-before.tgz
Checkpoint: Checkpoint: backup
Continue whenThe protected archive lists both configuration areas, verifies by checksum, and can be restored from an independent session.
Stop whenThe archive cannot be created/read, free space is inadequate, contents are incomplete, or configuration management will race the change.
If this step fails
tar completes with missing-path warnings or no archive.
Likely causeThe expected packaged paths differ, permissions are insufficient, or storage is full/read-only.
sudo test -d /etc/nginxsudo test -f /etc/logrotate.d/nginxdf -h /root
ResolutionResolve the actual authoritative paths and storage problem; create and inspect a complete archive before proceeding.
Security notes
- The archive contains internal routing and possibly credentials or allowlists; keep it mode 0600 in protected backup storage.
Alternatives
- Use an immutable signed configuration release plus a separately versioned logrotate policy when it provides equivalent restoration evidence.
Stop conditions
- The archive cannot be created/read, free space is inadequate, contents are incomplete, or configuration management will race the change.
config
Define a structured text format
Log request and upstream duration, final upstream address, status, bytes, request ID, and direct peer. Do not log Authorization, Cookie, query secrets, or request bodies.
Why this step matters
A named schema makes incident queries and downstream parsing stable while deliberately excluding high-risk request data and fields whose uncontrolled cardinality or syntax would make operations unreliable.
What to understand
`$uri` excludes the original query string; method, host, peer, status, bytes, request and upstream timings provide a useful minimal operational record.
The format is space-delimited key-value text. Do not call it JSON, and do not add values that may contain unescaped spaces without redesigning the parser.
The command replaces one dedicated owned file deterministically. Inspect with `sed -n l` and `nginx -T` so quoting and hidden characters are visible.
System changes
- Creates or replaces `/etc/nginx/conf.d/50-operations-log.conf`; no site selects it until the next step and no running process changes until reload.
Syntax explained
log_format oneliners_ops- Defines the reusable schema name in HTTP context.
$time_iso8601- Records an explicit timezone-bearing event timestamp.
$request_time- Records total nginx-observed request duration in seconds.
$upstream_*- Captures selected upstream address, response status and response duration when a proxy is involved.
$request_id- Adds an nginx-generated correlation value.
/etc/nginx/conf.d/50-operations-log.confprintf 'log_format oneliners_ops '\''$time_iso8601 host=$host peer=$remote_addr method=$request_method uri=$uri status=$status bytes=$body_bytes_sent request_time=$request_time upstream=$upstream_addr upstream_status=$upstream_status upstream_time=$upstream_response_time request_id=$request_id'\'';\n' | sudo tee /etc/nginx/conf.d/50-operations-log.conflog_format oneliners_ops '$time_iso8601 host=$host ... request_time=$request_time ...';
Checkpoint: Checkpoint: format
Continue whenOne parseable format definition exists with the documented fields and no header, body, Cookie, Authorization, or original query string.
Stop whenPrivacy approval is absent, the name already means something else, quoting is damaged, or nginx reports the definition in an invalid context.
If this step fails
nginx reports `unknown log format oneliners_ops`.
Likely causeThe file is not included in HTTP context, has a syntax error, or the access directive uses a different name.
sudo nginx -T 2>&1 | grep -n -A2 -B2 oneliners_opssudo sed -n l /etc/nginx/conf.d/50-operations-log.conf
ResolutionRestore one correctly quoted definition under the active HTTP include and validate before any reload.
Security notes
- Treat schema expansion as a data collection change; never add secrets merely because a variable is available.
Alternatives
- Use syslog or correctly escaped JSON with schema tests when a collector requires it; keep the same minimal field and privacy discipline.
Stop conditions
- Privacy approval is absent, the name already means something else, quoting is damaged, or nginx reports the definition in an invalid context.
config
Assign per-site buffered logs
Replace only complete access_log and error_log directives inside the already reviewed target-site file. Stop if either directive is absent or repeated; small buffering reduces write overhead while preserving predictable flush behavior.
Why this step matters
Selecting the format at the exact server boundary provides per-tenant evidence; a guarded exact-directive replacement prevents a broad regex from silently changing another server or included context.
What to understand
The preconditions require exactly one complete access and error directive in the reviewed site file. A zero or multiple count fails before `sed` mutates anything.
The access file uses a 32 KiB buffer and five-second maximum flush. The error log remains unbuffered by this directive and records warning-or-higher events.
After editing, inspect the diff and expanded config. A parser success alone does not prove that the intended server selected these paths.
System changes
- Replaces exactly one site-level access and error log directive in `/etc/nginx/sites-available/{{domain}}`; running nginx remains unchanged until reload.
Syntax explained
grep -Ec ... -eq 1- Acts as a guard that refuses ambiguous or missing directives.
sed -i -E- Edits only full anchored directive lines in the selected file.
buffer=32k- Buffers access records to reduce write frequency.
flush=5s- Bounds how long a low-volume record normally remains buffered.
warn- Stores warning, error, critical, alert and emergency error-log events.
/etc/nginx/sites-available/{{domain}}Values stay on this page and are never sent or saved.
SITE=/etc/nginx/sites-available/{{domain}}; test "$(grep -Ec '^[[:space:]]*access_log[[:space:]]' "$SITE")" -eq 1 && test "$(grep -Ec '^[[:space:]]*error_log[[:space:]]' "$SITE")" -eq 1 && sudo sed -i -E 's#^[[:space:]]*access_log[[:space:]].*;[[:space:]]*$# access_log /var/log/nginx/{{domain}}-access.log oneliners_ops buffer=32k flush=5s;#; s#^[[:space:]]*error_log[[:space:]].*;[[:space:]]*$# error_log /var/log/nginx/{{domain}}-error.log warn;#' "$SITE"access_log /var/log/nginx/example.com-access.log oneliners_ops buffer=32k flush=5s; error_log /var/log/nginx/example.com-error.log warn;
Checkpoint: Checkpoint: site logs
Continue whenThe target file and expanded target server each show one intended access path/format and one intended warning-level error path.
Stop whenDirective count is not one, site file is generated, another server is modified, the path is shared across tenants, or the config diff includes unrelated lines.
If this step fails
The guard exits before sed runs.
Likely causeThe directives are inherited, split across includes, commented, multiline, or duplicated.
grep -nE '^[[:space:]]*(access_log|error_log)' /etc/nginx/sites-available/app.example.comsudo nginx -T 2>&1 | grep -n -E 'server_name|access_log|error_log'
ResolutionEdit the reviewed server block explicitly or create a site-scoped included fragment; do not weaken the guard into a global replacement.
Security notes
- Per-site logs still contain client-controlled data and IP addresses; separation reduces accidental cross-tenant access but does not replace retention/privacy controls.
Alternatives
- Use a dedicated site-scoped include file referenced once by the server block when configuration is templated or multiline.
Stop conditions
- Directive count is not one, site file is generated, another server is modified, the path is shared across tenants, or the config diff includes unrelated lines.
verification
Protect log directory access
Keep the directory root-owned, grant read access through adm, and never let the web deployment account write privileged log paths.
Why this step matters
Log confidentiality and integrity depend on the directory ownership boundary; authorized read access should not turn application, deployment, or operations identities into privileged log writers.
What to understand
Root owns the directory and `adm` receives traverse/read according to Ubuntu's administrative-log convention. Existing log file modes and logrotate `create` behavior must remain compatible.
`usermod -aG` changes account membership but not the groups of existing sessions. Test from a new login rather than weakening modes when an old session fails.
Review whether `adm` exposes other tenant/system logs. A narrower ACL or collector is preferable in separated environments.
System changes
- Sets `/var/log/nginx` owner/group/mode and adds `{{opsUser}}` to the supplementary `adm` group; no log content is changed.
Syntax explained
chown root:adm- Keeps privileged ownership while assigning the approved reader group.
chmod 0750- Allows owner full access, group read/traverse, and no access to others.
usermod --append --groups- Adds membership without removing the user's other supplementary groups.
Values stay on this page and are never sent or saved.
sudo chown root:adm /var/log/nginx && sudo chmod 0750 /var/log/nginx && sudo usermod --append --groups adm {{opsUser}} && sudo stat -c '%A %U %G %n' /var/log/nginxdrwxr-x--- root adm /var/log/nginx
Checkpoint: Checkpoint: permissions
Continue whenThe directory is `root:adm` mode 0750, workers/deployers cannot write, and a fresh operator session can read only as policy permits.
Stop whenThe directory is a symlink/unexpected mount, worker or deployment identity would gain write access, or `adm` grants an unacceptable scope.
If this step fails
A fresh operations session cannot read the new log.
Likely causeFile create mode/group differs, a parent/ACL blocks access, logrotate recreated it differently, or identity/group data is stale.
id alicenamei -l /var/log/nginx/app.example.com-access.loggetfacl -p /var/log/nginx/app.example.com-access.loggrep -A8 /var/log/nginx /etc/logrotate.d/nginx
ResolutionAlign directory/file group and rotation create policy with the approved reader model; never use world-readable or writable modes.
Security notes
- Administrative group membership is privileged read access and should be reviewed, logged, and removed when no longer required.
Alternatives
- Grant a read-only ACL to a dedicated group or let a root-run collector forward a redacted stream to an access-controlled store.
Stop conditions
- The directory is a symlink/unexpected mount, worker or deployment identity would gain write access, or `adm` grants an unacceptable scope.
verification
Validate and generate known log events
Reload the valid configuration, generate 200 and 404 requests, wait for the bounded buffer flush, and inspect only the new records.
Why this step matters
A validated configuration only proves syntax. Controlled requests must exercise server selection, response classes, buffer flush, file creation, schema population, and the operator's actual read path.
What to understand
Run `nginx -t` before reload. Generate an expected existing path and a unique missing path, then wait at least the configured flush interval.
Record concrete new lines and verify host, method, URI, status, timing, upstream placeholders/values, and request ID. Do not accept only a growing file size.
If public DNS/CDN obscures the origin, use a reviewed local `curl --resolve` test and then a separate external acceptance request.
System changes
- Gracefully reloads nginx, creates two harmless HTTP request records, and may create the new site log files.
Syntax explained
nginx -t &&- Prevents reload and request generation when parsing fails.
curl --output /dev/null- Exercises the full HTTP response while discarding the body.
sleep 5- Waits for the configured maximum access-log flush interval.
tail -n 2- Bounds evidence to the two commissioning records.
Values stay on this page and are never sent or saved.
sudo nginx -t && sudo systemctl reload nginx && curl --silent --output /dev/null https://{{domain}}/ && curl --silent --output /dev/null https://{{domain}}/missing-oneliners && sleep 5 && sudo tail -n 2 /var/log/nginx/{{domain}}-access.log2026-07-25T... host=example.com status=200 request_time=0.012 2026-07-25T... host=example.com status=404
Checkpoint: Checkpoint: generate
Continue whenThe existing path logs status 200 and the unique missing path logs 404 in the intended file with every documented operational field.
Stop whenParser/reload fails, requests mutate data, another tenant/file receives records, fields contain secrets, or statuses and selected server do not match.
If this step fails
Requests succeed but records do not appear after five seconds.
Likely causeAnother server block handled them, the effective access_log differs, nginx cannot write the path, or reload did not use the edited config.
sudo nginx -T 2>&1 | grep -n -E 'server_name|access_log'systemctl status nginx --no-pagersudo journalctl -u nginx --since '-10 min' --no-pager -n 100
ResolutionProve server selection and effective path, fix only the identified scope/permission issue, then repeat the controlled events.
Security notes
- Use only non-mutating public paths and never place secrets in the commissioning URL or headers.
Alternatives
- Use an internal health endpoint and `curl --resolve` when public requests are unsafe or an edge terminates traffic.
Stop conditions
- Parser/reload fails, requests mutate data, another tenant/file receives records, fields contain secrets, or statuses and selected server do not match.
warning
Rehearse log rotation and reopen
Debug the packaged rule, force one controlled rotation, and confirm nginx receives the reopen signal without retaining deleted files.
Why this step matters
Rotation is not complete until the policy parses, current files are rotated, nginx reopens new paths, new requests land there, and no process retains deleted inodes.
What to understand
Run debug first because it does not rotate. A forced rotation is state-changing and belongs in a controlled window with disk headroom and current evidence retained.
The packaged Ubuntu rule carries the service-specific reopen action. Do not substitute `copytruncate` for convenience; it can lose or duplicate records under write load.
After rotation, generate another request, wait for flush, inspect current and `.1` files, and use `lsof +L1` to detect deleted handles.
System changes
- Forces the packaged nginx rotation policy, renames/creates log paths, runs its reopen action, and changes future write destinations.
Syntax explained
logrotate --debug- Parses and simulates decisions without changing files.
logrotate --force- Runs a controlled rotation regardless of normal schedule.
lsof +L1- Finds processes holding open files whose directory link count is below one.
Values stay on this page and are never sent or saved.
sudo logrotate --debug /etc/logrotate.d/nginx && sudo logrotate --force /etc/logrotate.d/nginx && systemctl is-active nginx && sudo lsof +L1 | grep nginx || true; sudo ls -lh /var/log/nginx/{{domain}}-*reading config file /etc/logrotate.d/nginx active example.com-access.log example.com-access.log.1
Checkpoint: Checkpoint: rotate
Continue whenPolicy debug is clean, nginx stays active, current and rotated files exist with protected ownership, a new request reaches the current file, and nginx has no deleted handles.
Stop whenDebug reports errors, disk headroom is insufficient, the packaged reopen command is absent/wrong, nginx is unhealthy, or evidence retention forbids forced rotation.
If this step fails
New records continue in `.1` or lsof shows a deleted nginx log.
Likely causeThe reopen signal did not reach the active master, policy targets another service/PID, or multiple nginx instances exist.
systemctl status nginx --no-pagercat /run/nginx.pidsudo lsof +L1 | grep nginxsudo journalctl -u nginx --since '-10 min' --no-pager
ResolutionRepair the packaged reopen path for the correct master, reopen safely, generate a new request, and verify the current file before the next rotation.
Security notes
- Do not delete large or suspicious logs to make the test pass; preserve evidence and solve retention/capacity through approved policy.
Alternatives
- Exercise rotation on staging or wait for the scheduled rotation while monitoring reopen evidence when forcing production rotation is not approved.
Stop conditions
- Debug reports errors, disk headroom is insufficient, the packaged reopen command is absent/wrong, nginx is unhealthy, or evidence retention forbids forced rotation.
instruction
Record incident queries
Keep bounded commands for recent 5xx, slow requests, and startup errors. Parse the space-delimited request_time field with portable awk and render control characters visibly instead of trusting log text as terminal output.
Why this step matters
Operators need copyable bounded queries before an incident, but those queries must run on Ubuntu's default awk, parse only stable fields, and display untrusted log content safely.
What to understand
Extended grep selects a literal ` status=` key followed by a three-digit 5xx class. Tail caps output before `sed -n l` renders control characters visibly.
The POSIX awk loop finds the `request_time=` field, splits on `=`, converts the numeric value, and prints records above two seconds without gawk-only match capture arrays.
The journal query is time- and line-bounded. Treat results as leads, then correlate request ID, application/upstream evidence and deployment timeline.
System changes
- No persistent change; reads bounded records from the site access log and nginx systemd journal.
Syntax explained
grep -aE- Treats input as text and enables the explicit 5xx regular expression.
tail -n 20- Limits returned access records.
awk for/split- Parses the named timing field portably across POSIX-compatible awk implementations.
sed -n l- Escapes non-printable characters before terminal display.
journalctl --since ... -n 100- Bounds unit evidence by both time and count.
Values stay on this page and are never sent or saved.
LOG=/var/log/nginx/{{domain}}-access.log; sudo grep -aE ' status=5[0-9]{2} ' "$LOG" | tail -n 20 | sed -n l; sudo awk '{ for (i=1; i<=NF; i++) if ($i ~ /^request_time=/) { split($i, value, "="); if ((value[2] + 0) > 2) print } }' "$LOG" | tail -n 20 | sed -n l; sudo journalctl -u nginx --since '-30 min' --no-pager -n 100host=example.com status=502 request_time=3.004 upstream_status=502
Checkpoint: Checkpoint: queries
Continue whenKnown 5xx and slow test records are returned when present, clean requests are excluded, control characters are escaped, and journal output is bounded.
Stop whenThe schema has changed, fields contain spaces/unescaped delimiters, commands scan unbounded archives on a loaded host, or output is pasted externally without redaction.
If this step fails
A visibly slow request is not selected.
Likely causeThe site uses another format, timing field is absent/malformed, threshold differs, or the slow segment occurred upstream while total data was not flushed.
sudo tail -n 5 /var/log/nginx/app.example.com-access.log | sed -n lsudo nginx -T 2>&1 | grep -n oneliners_ops
ResolutionConfirm the effective schema and field value, adjust the reviewed threshold/query, and generate a controlled slow fixture rather than guessing.
Security notes
- Never eval, source, or interpolate extracted log fields into commands; logs are attacker-controlled data.
Alternatives
- Use a tested local parser or observability platform with the same schema, redaction, limits and access controls for larger retained datasets.
Stop conditions
- The schema has changed, fields contain spaces/unescaped delimiters, commands scan unbounded archives on a loaded host, or output is pasted externally without redaction.
Finish line
Verification checklist
curl --silent --output /dev/null https://{{domain}}/ && sleep 5 && sudo tail -n 1 /var/log/nginx/{{domain}}-access.logRecord contains host, status, request_time, upstream fields, and request_id.sudo logrotate --debug /etc/logrotate.d/nginx && systemctl is-active nginxPolicy parses and nginx remains active.Recovery guidance
Common problems and safe checks
The new site access file remains empty after a successful request and five seconds.
Likely causeThe request selected another server block, access logging is disabled or overridden deeper in context, buffering did not flush because the edited directive is inactive, or nginx could not create/write the path.
sudo nginx -T 2>&1 | grep -n -E 'server_name|access_log|oneliners_ops'curl -vk --resolve app.example.com:443:127.0.0.1 https://app.example.com/sudo journalctl -u nginx --since '-10 min' --no-pager -n 100sudo namei -l /var/log/nginx/app.example.com-access.log
ResolutionIdentify the selected server and effective logging directive, correct only that reviewed file, validate, reload, regenerate a known request, wait for the configured flush, and require a concrete record.
nginx -t reports an unknown log format or duplicate directive.
Likely causeThe format file is outside the HTTP include path, syntax/quoting is damaged, the name is duplicated, or an `access_log` references it before the effective definition.
sudo nginx -T 2>&1 | grep -n -A2 -B2 oneliners_opssudo sed -n l /etc/nginx/conf.d/50-operations-log.conf
ResolutionKeep one syntactically quoted format in an HTTP-context include, remove only the unintended duplicate, and require a clean expanded configuration before reload.
The operator still receives permission denied after being added to adm.
Likely causeThe existing login has stale supplementary groups, a parent/file mode or ACL blocks access, or local policy uses another group.
id alicegetent group admnamei -l /var/log/nginx/app.example.com-access.loggetfacl -p /var/log/nginx/app.example.com-access.log
ResolutionStart a fresh authenticated session and correct the narrow directory/file group or ACL; do not make logs world-readable or grant the user write access.
Disk usage grows or nginx continues writing a renamed/deleted file after rotation.
Likely causeThe packaged postrotate action failed, the wrong policy was forced, retention is too large, or another nginx instance does not receive the reopen signal.
sudo logrotate --debug /etc/logrotate.d/nginxsudo lsof +L1 | grep nginxsudo nginx -T 2>&1 | grep -E 'access_log|error_log'sudo du -ah /var/log/nginx | sort -h | tail
ResolutionRepair the packaged reopen action and capacity/retention policy, signal the correct instance, confirm new writes reach current paths, and preserve incident evidence before cleanup.
Reference
Frequently asked questions
Why not log the complete request line?
It commonly includes query strings containing tokens or personal data. The selected method plus normalized `$uri` retains useful routing evidence without copying the original query.
Is the access log enough to diagnose an upstream incident?
No. It shows nginx's observation. Correlate the request identifier, upstream timings/status, application logs, service health and network evidence.
Why wait five seconds in the test?
The configured access buffer may delay a low-volume record until `flush=5s`; testing immediately can create a false failure.
Can I use awk on this format?
Yes for controlled fixed fields. This guide uses POSIX-compatible field iteration instead of gawk-only capture arrays and never executes extracted values.
Recovery
Rollback
Restore prior nginx logging files while preserving collected evidence and secure directory permissions.
- Restore configuration from /root/nginx-logging-before.tgz without deleting current or rotated logs.
- Run nginx -t and reload.
- Remove the operations user from adm only after log access is no longer required.
- Never make /var/log/nginx writable by the nginx worker or deployment account.
Evidence