Web & Proxy / Incident Response & Troubleshooting

Fix NGINX 502 Bad Gateway with upstream evidence

Trace one request from NGINX to the selected upstream and distinguish connection refusal, timeout, DNS, Unix-socket access, protocol mismatch, and invalid responses.

35 min8 stepsChanges system stateRevision 1

Save or explore

#nginx #502 #bad-gateway #upstream #reverse-proxy

0 of 8 steps completed

Goal

Restore the proxied request without hiding application failure by increasing every timeout, disabling TLS verification, or routing traffic to an unverified backend.

Supported environments

NGINX Open Source 1.26.x, 1.28.x

Prerequisites

Request Have one reproducible URL, Host header, method, and timestamp.
Proxy access Read NGINX effective configuration and error logs.
Upstream owner Know the expected service, address, port/socket, and protocol.

Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System

A repeatable NGINX reverse proxy incident record that starts with the literal `502 Bad Gateway` symptom and preserves the first useful failure instead of hiding it with an early restart or broad permission change.
A layer-by-layer decision path from client context through configuration and runtime evidence to one narrowly scoped remediation, followed by positive and negative verification.
A reusable evidence bundle containing commands, concrete example output, timestamps, identities, effective configuration, and stop conditions that another operator can review without access to the original terminal.

Observable outcome

You can identify which boundary failed, explain why competing hypotheses were rejected, and state what changed before declaring the incident resolved.
The repaired path succeeds under the intended identity while an unauthorized or incorrect path still fails, proving that the fix did not simply remove a security control.
The final evidence distinguishes a transient recovery from a durable repair by checking logs, counters, configuration provenance, and the original user-visible operation.

Architecture

How the parts fit together

NGINX reverse proxy troubleshooting is treated as an evidence pipeline rather than a list of guesses. The operator captures the symptom, resolves client and target context, inspects the first authoritative server-side failure, tests the smallest cause boundary, applies one reviewed change, and repeats the same observation path. This keeps a secondary error from replacing the primary cause and makes rollback possible.

Client contextProves the executable, identity, target, selected configuration, environment variables, and exact request that produced the visible error.

Transport and targetConfirms that the request reached the intended host, socket, service, repository, or upstream instead of a stale context or similarly named environment.

Authoritative evidenceUses effective configuration, service state, event records, and logs from the component that made the decision rather than relying on a client summary.

Controlled remediationChanges one demonstrated cause, retains a backup or previous reference, and avoids global permission, trust, timeout, or force settings.

Verification boundaryRepeats the original operation, exercises a negative case, and watches for recurrence long enough to distinguish a repair from a restart-only recovery.

Copy the literal error and the command that produced it before retrying, restarting, pruning, resetting, or editing anything.
Resolve the selected identity, configuration, target, and current runtime state so later commands inspect the same path the user exercised.
Read the nearest authoritative log or effective configuration and align timestamps across the client and server evidence.
Test one hypothesis with a read-only command; if evidence disagrees, return to the previous layer instead of stacking speculative changes.
Back up the affected configuration or reference, apply the narrow repair, validate syntax, and reload only the component that owns the decision.
Repeat the original action plus a negative control, record the outcome, and keep rollback material until the observation window is complete.

Assumptions

A 502 is generated by the intended NGINX instance.
The upstream can be tested locally without unsafe writes.
Configuration changes use backup, nginx -t, and reload rather than process kill.
Clocks are close enough that client and server timestamps can be correlated. If they are not, record the offset before comparing logs.
The operator has a recovery path that does not depend on the component being changed, such as console access, a second session, or a preserved remote reference.
Commands are first run in the affected environment with placeholders reviewed. OneLiners never executes them and does not know local policy, tenancy, or maintenance constraints.

Key concepts

Gateway: NGINX accepts the client request and becomes a client of another HTTP/FastCGI/gRPC service.
Upstream: The selected backend address, Unix socket, or named group that must resolve, accept, and return the expected protocol.
Connect failure: NGINX could not establish the upstream connection, commonly refused, timed out, inaccessible, or unresolved.
Response failure: The connection opened but the upstream closed early, sent an invalid header, or used another protocol.
Effective configuration: The complete `nginx -T` output after includes, which determines actual server/location/proxy selection.

Set guide values0/2 ready

Fill these once. Every matching command and configuration block updates immediately; values stay in this page only.

Url{{url}}Reproducible public URL.Upstream{{upstream}}Resolved local upstream URL.

Security and production boundaries

Keep upstream TLS verification, authentication, and network segmentation intact during diagnosis.
Do not expose internal error bodies or configuration dumps to public clients.
Do not paste private keys, tokens, complete environment dumps, authorization headers, or unredacted customer data into tickets or external analysis tools.
A successful operation after disabling authentication, trust, host verification, sandboxing, or least privilege is a security regression, not a valid repair.
Prefer effective configuration and narrow identity tests over recursive ownership changes, mode 777, global trust bypasses, or force options copied from unrelated incidents.

Stop before continuing if

Stop if the request could mutate production data.
Stop if the proposed fallback backend has not passed readiness and authorization checks.
Stop if the target, account, environment, repository, or service instance cannot be identified unambiguously.
Stop before a destructive cleanup, force update, broad permission change, or production reload when backup and recovery evidence is missing.
Stop if the proposed action would conceal the first error or remove logs, failed objects, repository references, or configuration needed for diagnosis.

instruction

Freeze the symptom and define the incident boundary

read-only

Before changing NGINX reverse proxy, preserve the literal `502 Bad Gateway` message, the exact action that produced it, the affected identity, target, UTC time, and expected result. Decide which production boundary is in scope and name the independent recovery path you will keep available.

Why this step matters

A retry, restart, cleanup, permission change, or configuration edit can replace the primary failure with a secondary one. A fixed starting record lets every later check answer a specific hypothesis.

What to understand

Record whether the problem affects one user or workload, one target, one host, or every comparable path; this determines whether to begin at the client, transport, or service boundary.

Write down the expected successful behavior in observable terms such as an exit status, HTTP response, remote object ID, authenticated principal, or stable service state.

Keep console access, a second session, an earlier configuration, or a recoverable reference independent of the component being diagnosed.

System changes

No persistent change. This step reads current configuration, identity, runtime state, or logs and records evidence for the incident timeline.

Example output / evidence

Incident scope
Tool: NGINX reverse proxy
Observed: `502 Bad Gateway`
Affected target: production-app-01
Observed at: 2026-08-17T14:02:11Z
Expected: the reviewed operation completes without weakening an access or safety control
Recovery path: console session and last known-good configuration retained

Security notes

Redact secrets and customer data, but retain error text, timestamps, object IDs, modes, owners, and target names needed to reproduce the decision.

Alternatives

When direct production inspection is restricted, reproduce the same version and configuration boundary in an isolated environment and mark which evidence is illustrative.

Stop conditions

Stop before any mutation if the target, identity, affected environment, expected result, or recovery path is still ambiguous.

command

Capture the failing client transaction

read-only

Record DNS result, TLS peer, response status, headers, timing, and request timestamp for one bounded request.

Why this step matters

A reproducible request anchors every log and distinguishes an immediate gateway failure from a client DNS, TLS, or network problem.

What to understand

Preserve the Host name and path because different server/location blocks can select different upstreams.

A short start-transfer time often aligns with immediate refusal or configuration failure; a long one suggests connect/read timeout, but logs remain authoritative.

Record the command, UTC timestamp, exit status, and the exact output before moving to the next layer. A later successful command must not erase evidence of the original failure.

Interpret this result together with the previous checkpoint. One matching line is evidence for a hypothesis, not permission to apply every commonly suggested fix.

System changes

No persistent change. This step reads current configuration, identity, runtime state, or logs and records evidence for the incident timeline.

Syntax explained

--dump-header -: Prints response status and headers while discarding the body.
--write-out: Records stable timing and selected remote IP fields.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Url{{url}}Reproducible public URL.

date -u +%FT%TZ; curl --silent --show-error --output /dev/null --dump-header - --write-out 'code=%{http_code} remote=%{remote_ip} connect=%{time_connect} start=%{time_starttransfer} total=%{time_total}\n' {{url}}

Example output / evidence

2026-08-17T14:02:11Z
HTTP/2 502
server: nginx
code=502 remote=203.0.113.20 connect=0.021 start=0.043 total=0.043

Checkpoint: Checkpoint: Capture the failing client transaction

date -u +%FT%TZ; curl --silent --show-error --output /dev/null --dump-header - --write-out 'code=%{http_code} remote=%{remote_ip} connect=%{time_connect} start=%{time_starttransfer} total=%{time_total}\n' {{url}}

Continue whenThe intended NGINX address returns a reproducible 502 with an exact UTC timestamp.

Stop whenDNS/TLS points to another provider, the response is not from NGINX, or the request mutates data.

If this step fails

The request fails before HTTP status.

Likely causeClient-side DNS, route, TCP, or TLS failure precedes NGINX.

Safe checks

getent ahosts app.example.net
openssl s_client -connect app.example.net:443 -servername app.example.net </dev/null

ResolutionFollow the failed client layer instead of an NGINX upstream tutorial.

Security notes

Do not include bearer tokens or customer request bodies in the capture.

Alternatives

Use a read-only health endpoint with the same routing/location rules.

Stop conditions

DNS/TLS points to another provider, the response is not from NGINX, or the request mutates data.

command

Read the matching NGINX upstream error

read-only

Find the request by timestamp, client, server, path, and upstream fields in the error journal or configured error log.

Why this step matters

The error record identifies the selected upstream and failure stage, turning a generic 502 into a specific refused, timeout, DNS, socket, TLS, or response hypothesis.

What to understand

Match the exact server, request, and upstream because unrelated upstream failures can occur simultaneously.

Keep the errno and phrase: connection refused differs from upstream timed out and upstream sent invalid header.

Record the command, UTC timestamp, exit status, and the exact output before moving to the next layer. A later successful command must not erase evidence of the original failure.

Interpret this result together with the previous checkpoint. One matching line is evidence for a hypothesis, not permission to apply every commonly suggested fix.

System changes

No persistent change. This step reads current configuration, identity, runtime state, or logs and records evidence for the incident timeline.

Syntax explained

journalctl -u nginx: Reads unit-associated startup and stderr records.
error.log: Reads NGINX request-processing errors when configured to a file.

Command

sudo journalctl -u nginx --since '-10 minutes' --no-pager | tail -n 100; sudo tail -n 100 /var/log/nginx/error.log

Example output / evidence

2026/08/17 14:02:11 [error] 1421#1421: *981 connect() failed (111: Connection refused) while connecting to upstream, client: 198.51.100.14, server: app.example.net, request: "GET /health HTTP/2.0", upstream: "http://127.0.0.1:8080/health"

Checkpoint: Checkpoint: Read the matching NGINX upstream error

sudo journalctl -u nginx --since '-10 minutes' --no-pager | tail -n 100; sudo tail -n 100 /var/log/nginx/error.log

Continue whenOne log record matches the request and names the upstream plus concrete failure stage.

Stop whenNo matching record exists or another proxy/CDN generated the 502.

If this step fails

The log contains no request record.

Likely causeAnother NGINX instance/path owns logging, severity excludes the record, or the request never reached this host.

Safe checks

sudo nginx -T 2>&1 | grep -n 'error_log'
systemctl cat nginx

ResolutionLocate the effective log and serving instance before changing upstream configuration.

Security notes

Logs can contain query strings and client identifiers; follow retention and redaction policy.

Alternatives

Use a request ID propagated into access and application logs when available.

Stop conditions

No matching record exists or another proxy/CDN generated the 502.

command

Resolve the selected server, location, and proxy target

read-only

Validate syntax and inspect the complete configuration around server_name, location, proxy_pass, upstream, resolver, sockets, timeouts, and TLS settings.

Why this step matters

Included files, location precedence, variables, and named upstreams often make the live target different from the file an operator opens first.

What to understand

`nginx -T` prints the active include graph and should be retained with secrets redacted.

A proxy_pass scheme must match the upstream protocol; HTTPS to a plaintext server or FastCGI through HTTP yields response failures.

Record the command, UTC timestamp, exit status, and the exact output before moving to the next layer. A later successful command must not erase evidence of the original failure.

Interpret this result together with the previous checkpoint. One matching line is evidence for a hypothesis, not permission to apply every commonly suggested fix.

System changes

No persistent change. This step reads current configuration, identity, runtime state, or logs and records evidence for the incident timeline.

Syntax explained

nginx -t: Validates syntax and referenced files without reloading.
nginx -T: Prints the complete effective configuration after includes.

Command

sudo nginx -t && sudo nginx -T 2>&1 | grep -nE 'server_name|location |proxy_pass|upstream |server 127\.0\.0\.1|resolver|proxy_(connect|read)_timeout'

Example output / evidence

nginx: configuration file /etc/nginx/nginx.conf test is successful
118: server_name app.example.net;
126: location / {
127: proxy_pass http://app_backend;
203: upstream app_backend {
204: server 127.0.0.1:8080;

Checkpoint: Checkpoint: Resolve the selected server, location, and proxy target

sudo nginx -t && sudo nginx -T 2>&1 | grep -nE 'server_name|location |proxy_pass|upstream |server 127\.0\.0\.1|resolver|proxy_(connect|read)_timeout'

Continue whenThe request's server/location maps to one expected upstream address or group and protocol.

Stop whenConfiguration contains unknown generated includes, secrets, or a different production target.

If this step fails

nginx -t fails.

Likely causeA configuration edit, missing file, duplicate directive, or permission problem already prevents safe reload.

Safe checks

sudo nginx -t
sudo journalctl -u nginx --since '-30 minutes'

ResolutionRestore the last valid configuration or correct the exact validation error before any reload.

Security notes

Redact private keys, credentials, and internal hostnames from full configuration captures.

Alternatives

Use configuration management's rendered artifact and checksum when direct `nginx -T` access is restricted.

Stop conditions

Configuration contains unknown generated includes, secrets, or a different production target.

command

Test the selected upstream directly from the proxy host

read-only

Check the listener and make a protocol-correct read-only request to the exact address or Unix socket named by the log.

Why this step matters

Testing from the NGINX network namespace separates backend availability and protocol from external routing, client TLS, and load-balancer behavior.

What to understand

Use the address from the matching error log, not a convenient public URL.

For a Unix socket, inspect `ss -lx`, path traversal, NGINX worker identity, and curl's `--unix-socket` support.

Record the command, UTC timestamp, exit status, and the exact output before moving to the next layer. A later successful command must not erase evidence of the original failure.

Interpret this result together with the previous checkpoint. One matching line is evidence for a hypothesis, not permission to apply every commonly suggested fix.

System changes

No persistent change. This step reads current configuration, identity, runtime state, or logs and records evidence for the incident timeline.

Syntax explained

ss -lntp: Shows the local TCP listener and owning process for the selected port.
--max-time 5: Bounds the direct probe so a stuck upstream does not hang the incident session.
--fail: Returns nonzero for HTTP error responses while retaining the body when not silent.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Upstream{{upstream}}Resolved local upstream URL.

sudo ss -lntp 'sport = :8080'; curl --silent --show-error --fail --max-time 5 --write-out '\ncode=%{http_code} connect=%{time_connect} total=%{time_total}\n' {{upstream}}

Example output / evidence

LISTEN 0 4096 127.0.0.1:8080 0.0.0.0:* users:(("api",pid=7712,fd=9))
{"status":"ok"}
code=200 connect=0.001 total=0.004

Checkpoint: Checkpoint: Test the selected upstream directly from the proxy host

sudo ss -lntp 'sport = :8080'; curl --silent --show-error --fail --max-time 5 --write-out '\ncode=%{http_code} connect=%{time_connect} total=%{time_total}\n' {{upstream}}

Continue whenThe intended process listens and returns the expected protocol/status from the NGINX host.

Stop whenThe endpoint mutates data, belongs to another tenant, or direct probing violates access policy.

If this step fails

Connection refused.

Likely causeThe service is stopped, bound to another address/port, crashed, or NGINX targets stale configuration.

Safe checks

systemctl status api --no-pager
journalctl -u api --since '-15 minutes'

ResolutionRestore the upstream service or correct the reviewed address; do not increase proxy timeouts.

Security notes

Do not bypass upstream authentication or TLS identity checks for the direct probe.

Alternatives

Use the service's documented local readiness command when no HTTP endpoint exists.

Stop conditions

The endpoint mutates data, belongs to another tenant, or direct probing violates access policy.

command

Distinguish DNS, Unix socket, TLS, and timeout causes

read-only

Run the branch that matches the error log: resolve names, inspect socket permissions, verify upstream TLS/SNI, or measure connect and response deadlines.

Why this step matters

The same 502 status can represent four distinct boundaries; only the exact log phrase and target type determine which branch is relevant.

What to understand

A hostname used in proxy_pass may resolve at startup or through an explicit resolver depending on configuration and variables.

For HTTPS upstreams, certificate identity and SNI may differ from the TCP address; for Unix sockets, every parent directory must be traversable by NGINX workers.

Record the command, UTC timestamp, exit status, and the exact output before moving to the next layer. A later successful command must not erase evidence of the original failure.

Interpret this result together with the previous checkpoint. One matching line is evidence for a hypothesis, not permission to apply every commonly suggested fix.

System changes

No persistent change. This step reads current configuration, identity, runtime state, or logs and records evidence for the incident timeline.

Syntax explained

getent ahosts: Uses the host's configured name-service path for the upstream name.
namei -l: Shows Unix-socket parent traversal and final socket metadata.
-servername / -verify_return_error: Tests upstream SNI and fails on certificate verification errors.

Command

getent ahosts app.internal; namei -l /run/app/app.sock 2>/dev/null || true; openssl s_client -connect app.internal:8443 -servername app.internal -verify_return_error </dev/null 2>/dev/null | grep -E 'subject=|issuer=|Verify return code'

Example output / evidence

10.10.4.21 STREAM app.internal
srwxrwx--- app nginx app.sock
subject=CN=app.internal
issuer=CN=Example Internal CA
Verify return code: 0 (ok)

Checkpoint: Checkpoint: Distinguish DNS, Unix socket, TLS, and timeout causes

getent ahosts app.internal; namei -l /run/app/app.sock 2>/dev/null || true; openssl s_client -connect app.internal:8443 -servername app.internal -verify_return_error </dev/null 2>/dev/null | grep -E 'subject=|issuer=|Verify return code'

Continue whenThe relevant name, socket, certificate, and deadline evidence agrees with the intended upstream design.

Stop whenThe certificate, DNS answer, or socket owner identifies an unexpected service.

If this step fails

Direct HTTP works but NGINX reports invalid header.

Likely causeproxy_pass protocol, HTTP version, TLS, FastCGI, gRPC, or application response framing mismatches.

Safe checks

sudo nginx -T
curl --verbose {{upstream}}

ResolutionConfigure the correct upstream protocol/module and verify it directly; do not suppress invalid-response errors.

Security notes

Never set `proxy_ssl_verify off` as a permanent repair for an unexplained trust failure.

Alternatives

Use a service-discovery health record or Unix socket only when its lifecycle and permissions are managed explicitly.

Stop conditions

The certificate, DNS answer, or socket owner identifies an unexpected service.

decision

Apply one proven repair and reload safely

caution

Back up the affected include, edit only the demonstrated address/protocol/socket/timeout cause, validate the full configuration, reload, and repeat both direct and proxied probes.

Why this step matters

A validated reload preserves active connections and provides an immediate rollback file while proving both halves of the proxy path.

What to understand

If the upstream service was repaired, an NGINX edit may be unnecessary; record that no proxy configuration changed.

Increase a timeout only when measured healthy requests legitimately exceed the reviewed deadline and capacity effects are understood.

Record the command, UTC timestamp, exit status, and the exact output before moving to the next layer. A later successful command must not erase evidence of the original failure.

Interpret this result together with the previous checkpoint. One matching line is evidence for a hypothesis, not permission to apply every commonly suggested fix.

System changes

Creates a rollback copy, may update one NGINX include, and reloads validated proxy configuration.

Syntax explained

cp -a: Creates an exact rollback copy of the affected include.
systemctl reload nginx: Asks NGINX to load validated configuration without terminating working connections.
curl --fail: Makes both direct and proxied verification fail on HTTP error status.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

Upstream{{upstream}}Resolved local upstream URL.Url{{url}}Reproducible public URL.

sudo cp -a /etc/nginx/conf.d/app.conf /etc/nginx/conf.d/app.conf.before-oneliners && sudo nginx -t && sudo systemctl reload nginx && curl --silent --show-error --fail {{upstream}} >/dev/null && curl --silent --show-error --fail --write-out 'proxy_code=%{http_code} total=%{time_total}\n' {{url}} >/dev/null

Example output / evidence

nginx: configuration file /etc/nginx/nginx.conf test is successful
proxy_code=200 total=0.018

Checkpoint: Checkpoint: Apply one proven repair and reload safely

sudo cp -a /etc/nginx/conf.d/app.conf /etc/nginx/conf.d/app.conf.before-oneliners && sudo nginx -t && sudo systemctl reload nginx && curl --silent --show-error --fail {{upstream}} >/dev/null && curl --silent --show-error --fail --write-out 'proxy_code=%{http_code} total=%{time_total}\n' {{url}} >/dev/null

Continue whenDirect and proxied requests return the intended status, error logs stay clean, and the old configuration is available.

Stop whennginx -t fails, the direct upstream is unhealthy, or the change routes to an unverified backend.

If this step fails

Reload succeeds but 502 persists.

Likely causeThe request selects another server/location, multiple workers/configs exist, or the upstream fails only under proxy headers/load.

Safe checks

sudo nginx -T
sudo tail -n 100 /var/log/nginx/error.log

ResolutionCorrelate a new request and compare its exact upstream; restore the prior include if the change was unrelated.

Security notes

Do not disable upstream TLS verification or route around authentication to achieve a 200 response.

Alternatives

Drain traffic and roll back the upstream deployment when proxy configuration is correct.

Stop conditions

nginx -t fails, the direct upstream is unhealthy, or the change routes to an unverified backend.

verification

Observe for recurrence and close with evidence

read-only

After the original NGINX reverse proxy operation succeeds, repeat the same read-only observation path used at the start. Watch the relevant logs, counters, identities, and target state for a bounded period, then record the proven cause, exact change, verification result, rollback point, and remaining uncertainty.

Why this step matters

A single successful retry may be a transient recovery. Repeating the original checks and retaining the rollback point distinguishes a durable repair from a restart-only improvement.

What to understand

Use the same client identity, target, path, and configuration selection as the original failure so the positive result is comparable.

Include one negative or unauthorized control where safe; this proves the repair did not work by removing authentication, trust, isolation, or branch protection.

Keep the evidence concise enough for another operator to reproduce, but include software versions and exact object or configuration references.

System changes

No persistent change. This step reads current configuration, identity, runtime state, or logs and records evidence for the incident timeline.

Example output / evidence

Observation window: 15 minutes
Original operation: PASS
Original signature recurrences: 0
Negative control: PASS
Rollback point retained: yes
Cause and repair recorded: yes
Remaining uncertainty: none observed within the stated boundary

Checkpoint: Checkpoint: the repair remains valid

sudo cp -a /etc/nginx/conf.d/app.conf /etc/nginx/conf.d/app.conf.before-oneliners && sudo nginx -t && sudo systemctl reload nginx && curl --silent --show-error --fail {{upstream}} >/dev/null && curl --silent --show-error --fail --write-out 'proxy_code=%{http_code} total=%{time_total}\n' {{url}} >/dev/null

Continue whenThe intended operation stays healthy, the original signature does not recur, and the negative control still enforces the expected boundary.

Stop whenThe result depends on an unexplained retry, disabled control, different target, or unrecorded manual state.

If this step fails

The error returns during the observation window.

Likely causeThe change treated a symptom, another instance still has the old state, or an automated process reverted or recreated the failing condition.

Safe checks

date -u +%FT%TZ; curl --silent --show-error --output /dev/null --dump-header - --write-out 'code=%{http_code} remote=%{remote_ip} connect=%{time_connect} start=%{time_starttransfer} total=%{time_total}\n' {{url}}
sudo cp -a /etc/nginx/conf.d/app.conf /etc/nginx/conf.d/app.conf.before-oneliners && sudo nginx -t && sudo systemctl reload nginx && curl --silent --show-error --fail {{upstream}} >/dev/null && curl --silent --show-error --fail --write-out 'proxy_code=%{http_code} total=%{time_total}\n' {{url}} >/dev/null

ResolutionPreserve the recurrence, compare it with the first evidence set, restore the known-good state if necessary, and reopen the unresolved layer instead of stacking another repair.

Security notes

Store only redacted operational evidence and remove temporary debug output according to local retention policy after review.

Alternatives

Use the service's approved monitoring or audit trail when an interactive observation window is not practical.

Stop conditions

Do not close the incident while the result is intermittent, the rollback point is missing, or a safety control remains weakened.

Finish line

Verification checklist

Original operation succeeds

sudo cp -a /etc/nginx/conf.d/app.conf /etc/nginx/conf.d/app.conf.before-oneliners && sudo nginx -t && sudo systemctl reload nginx && curl --silent --show-error --fail {{upstream}} >/dev/null && curl --silent --show-error --fail --write-out 'proxy_code=%{http_code} total=%{time_total}\n' {{url}} >/dev/null

The operation completes with exit status 0 and without the original error. The output identifies the intended host, service, repository, or endpoint.

Failure evidence stays absent

date -u +%FT%TZ; curl --silent --show-error --output /dev/null --dump-header - --write-out 'code=%{http_code} remote=%{remote_ip} connect=%{time_connect} start=%{time_starttransfer} total=%{time_total}\n' {{url}}

A fresh diagnostic capture shows the healthy path and no recurrence of the original signature. Logs and counters remain stable during the observation window.

Recovery guidance

Common problems and safe checks

upstream timed out

Likely causeThe connection or response exceeded proxy timeout because of dependency latency, saturation, or an undersized deadline.

Safe checks

curl --max-time 5 {{upstream}}
journalctl -u api --since '-10 minutes'

ResolutionRepair upstream latency/capacity or set an evidence-based deadline; do not maximize every timeout.

no live upstreams

Likely causeEvery upstream peer is unavailable according to fail counters, DNS, or configuration state.

Safe checks

sudo nginx -T
curl --fail http://each-peer/health

ResolutionRestore at least one verified peer and review fail_timeout/max_fails behavior.

Permission denied while connecting to upstream Unix socket

Likely causeNGINX worker identity cannot traverse the path or access the socket, or mandatory access control denies it.

Safe checks

namei -l /run/app/app.sock
ps -o user,group,cmd -C nginx

ResolutionCorrect the service-managed socket group/mode or policy; do not chmod 777.

After the procedure

Alternatives and next steps

Consider these alternatives

Roll back the upstream release when the proxy path is healthy and the application introduced the failure.
Temporarily drain a failed peer from a multi-peer group after health and capacity review.

Operate it safely

Propagate request IDs across NGINX and application logs.
Alert separately on refused, timed-out, invalid-response, and no-live-upstream signatures.
Turn the verified checks into a read-only health probe or alert using stable fields rather than matching a whole human-formatted line.
Record the cause, exact repair, rollback point, software version, and follow-up owner in the incident note so the same failure becomes cheaper to diagnose.
Review adjacent environments for the demonstrated cause, but apply changes only where the same evidence is present.

Reference

Frequently asked questions

Should I increase proxy_read_timeout for every 502?

No. Immediate connection refusal, DNS, socket permission, TLS, and invalid-response failures are not fixed by a longer read timeout.

Why does the upstream work from my laptop but not NGINX?

NGINX uses the proxy host's network namespace, DNS, source address, socket permissions, trust store, and effective configuration. Test from that boundary.

Does nginx -t prove the upstream is healthy?

No. It validates configuration syntax and referenced files, not DNS, listener state, TLS identity, application readiness, or response time.

Can I disable proxy_ssl_verify to test?

Capture the certificate chain and verify the intended CA/SNI instead. A bypass can turn a trust failure into an interception risk.

Recovery

Rollback

Restore the backed-up include or prior upstream deployment and reload only after `nginx -t` succeeds.

Copy `app.conf.before-oneliners` back to the managed include path.
Run `sudo nginx -t` and reload NGINX.
Repeat direct and proxied probes and confirm the error log matches the restored state.

Evidence

Sources and review

Verified 2026-08-17Review due 2027-02-13

NGINX proxy moduleofficial NGINX upstream moduleofficial NGINX core error loggingofficial