Configure nginx HTTP load balancing with safe failure handling
Balance requests across private backends, set explicit connect/read timeouts, use passive health behavior, preserve client context, and rehearse one-node failure.
Publish a redundant HTTP service without exposing backends or allowing a failed node to stall every client request.
- nginx 1.24+
- Ubuntu Server 24.04 LTS
- Console and recovery access Keep an independent console session and a copy of the current nginx configuration before changing listeners, modules, or access rules.
- DNS and network plan Resolve every production hostname to the intended public address and list the exact management and client networks that require access.
getent ahosts {{domain}} && ip route show - Port ownership Confirm that TCP port 443 is free or already belongs to the intended nginx instance.
sudo ss -lntp 'sport = :443' - Rollback artifact Snapshot the host or create an archive of the active configuration and web roots before the first material change.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- An nginx upstream group with two private application nodes, least-connections scheduling, bounded passive failure detection, reusable upstream connections, and no public backend exposure.
- A public server route with explicit HTTP/1.1 keepalive semantics, host/client/scheme context, connect/send/read timeouts, and a conservative retry policy that avoids duplicating non-idempotent requests.
- A redundancy rehearsal that proves both nodes serve traffic, one controlled node can fail without breaching the latency budget, and recovery is visible in nginx and backend evidence.
- Both private backends pass direct health/identity tests, nginx config is clean, and public requests reach both nodes without assuming an exact 50/50 count.
- Connection, gateway, and timeout failures can move to the other node within two tries while ordinary non-idempotent requests are not blindly replayed.
- Stopping one reviewed backend leaves public health successful within budget; restarting it restores participation, and rollback returns to the preserved single-upstream route.
Architecture
How the parts fit together
nginx terminates the public request and selects an upstream peer from a static group using `least_conn`. Each private peer has passive `max_fails/fail_timeout` state; open-source nginx learns from real request failures rather than active health probes. The proxy route uses HTTP/1.1 with an empty Connection header to support upstream keepalive, passes reviewed host/scheme/client context, bounds each phase, and permits at most two upstream attempts for selected failure classes. Application instances remain responsible for shared state, idempotency, readiness, and graceful drain.
- A client reaches nginx; nginx validates public TLS/Host and selects the location.
- least_conn selects an available peer, accounting for active connections rather than guaranteeing round-robin counts.
- nginx opens/reuses an HTTP/1.1 upstream connection and sends reviewed context headers.
- On selected connection/timeout/502-504 failures, nginx may try the other peer within the two-attempt ceiling and method-safety rules.
- The chosen backend returns identity/health/application output; nginx logs upstream address/status/time and replies to the client.
Assumptions
- Both nodes run equivalent compatible releases, use a stable private network/DNS plan, and are not directly reachable from the Internet.
- Health endpoints are cheap, non-secret, and represent readiness for real traffic rather than merely process liveness.
- Sessions, uploads, caches, migrations, jobs, and generated data are external/shared or intentionally node-local; the application does not depend on accidental stickiness.
- nginx is the trusted edge or receives client identity from a separately configured trusted proxy; application trusted-proxy/hop behavior is explicit.
- SSH/service control for the failure rehearsal targets an approved non-shared node and maintenance window; otherwise an identical staging environment is used.
Key concepts
- least_conn
- A scheduler that sends a new request to an available peer with fewer active connections; counts need not be equal.
- passive health check
- Peer availability inferred from real proxied request failures rather than periodic active probes.
- fail_timeout
- The window for counting max_fails and the period a peer may be considered unavailable after the threshold.
- upstream keepalive
- A cache of idle backend connections; it does not cap total active upstream connections.
- retry safety
- Whether sending a request to another peer can duplicate side effects. Method and application idempotency matter.
- graceful drain
- Removing a node from new traffic while allowing in-flight work to finish before stop/deploy.
Before you copy
Values used in this guide
{{domain}}Public nginx hostname routed to the group.
Example: app.example.com{{appName}}Short safe identifier used in upstream/config filenames.
Example: checkout{{backendA}}First private backend address/name.
Example: 10.20.0.11{{backendB}}Second private backend address/name.
Example: 10.20.0.12{{backendPort}}Private application HTTP port.
Example: 8080{{appService}}Exact backend systemd unit used in the controlled rehearsal.
Example: checkout.serviceSecurity and production boundaries
- Private backends still require host firewall/security-group policy permitting only nginx/management, plus patching and authentication appropriate to their trust zone.
- X-Forwarded-For can contain client-supplied leading values with `$proxy_add_x_forwarded_for`; applications must trust the known final proxy hops rather than the first arbitrary value.
- Preserved Host must be validated by the application. Scheme/client headers should be overwritten by the trusted edge, not appended blindly at multiple layers.
- Retries can duplicate writes. Do not add `non_idempotent` or broad status retries unless the application implements idempotency keys/transaction guarantees.
- Health and node identity endpoints must not expose versions, environment, secrets, database credentials, or internal topology beyond operational need.
- Do not put admin/status paths in the public location; restrict them by private listener/network/authentication.
Stop before continuing if
- Stop if either backend is unhealthy, public, runs another release/schema, or uses inconsistent session/upload/job state.
- Do not reload if upstream names/ports, config scope, keepalive semantics, timeout/retry policy, or trusted proxy model is unresolved.
- Do not infer success from exact 50/50 distribution; require both identities and bounded public correctness.
- Abort failure rehearsal if service control targets shared dependencies, public requests fail/exceed budget, or restart/readiness is incomplete.
- Do not accept redundancy without application transactions, failure/recovery logs, capacity headroom, and a documented drain/deploy process.
verification
Test every backend directly
Confirm private addresses, health endpoints, unique node markers, latency, and that no backend is publicly reachable.
Why this step matters
A load balancer cannot create healthy redundancy from inconsistent, public, slow, or stateful nodes; each peer must be proven directly before nginx masks differences.
What to understand
Verify address/route/firewall/listener/process, release/config/schema, health/node markers, latency, logs, and public inaccessibility for both nodes.
Exercise a representative read and approved transaction on each, including session/upload/shared-state behavior.
Confirm health means ready for traffic and has bounded dependencies; a simple process-up endpoint can create false positives.
System changes
- No persistent change; creates bounded direct private HTTP connections.
Syntax explained
for u in …- Runs identical evidence against both explicit backend URLs.
curl --max-time 3- Bounds direct health latency.
--write-out http_code time_total- Records status and elapsed time without body ambiguity.
Values stay on this page and are never sent or saved.
for u in http://{{backendA}}:{{backendPort}} http://{{backendB}}:{{backendPort}}; do curl --fail --silent --max-time 3 "$u/health"; curl --silent --write-out ' %{http_code} %{time_total}\n' --output /dev/null "$u/"; donenode-a ok 200 0.012 node-b ok 200 0.010
Checkpoint: Checkpoint: backend baseline
Continue whenBoth private nodes return distinct identity plus healthy equivalent behavior/latency and are blocked from the public Internet.
Stop whenAny listener, release, config/schema, state, health, latency, log, route, or exposure differs unexpectedly.
If this step fails
One node health passes but real requests fail.
Likely causeHealth is too shallow or release/config/dependency differs.
curl -v http://10.20.0.11:8080/systemctl status checkout.service --no-pagerCompare release checksums/config
ResolutionFix readiness semantics and node parity before balancing.
Security notes
- Direct test paths must remain private and non-secret; do not bypass backend access controls from arbitrary networks.
Alternatives
- Use a staging pair with production-like state when direct production transactions are unsafe.
Stop conditions
- Any listener, release, config/schema, state, health, latency, log, route, or exposure differs unexpectedly.
command
Back up the public server block
Preserve the exact active configuration before adding an upstream group.
Why this step matters
The current public route is the known-good path; preserve the exact active server file and verify its symlink/deployment ownership before adding upstream state.
What to understand
Resolve sites-enabled to the available file and store a metadata-preserving copy/checksum outside served roots.
Record public response, TLS, logs, current single backend, and application release rollback.
Archive any existing upstream fragment with the same appName to prevent overwrite/collision.
System changes
- Creates a side-by-side `.before-upstream` copy and checksum.
Syntax explained
cp --archive- Preserves owner/mode/timestamps for exact targeted restore.
sha256sum- Records integrity of the known-good public server file.
Values stay on this page and are never sent or saved.
sudo cp --archive /etc/nginx/sites-available/{{domain}} /etc/nginx/sites-available/{{domain}}.before-upstream && sudo sha256sum /etc/nginx/sites-available/{{domain}}.before-upstream2c71... /etc/nginx/sites-available/example.com.before-upstream
Checkpoint: Checkpoint: backup
Continue whenThe saved file is the active public block, protected, verifiable, and paired with a tested restore/parser/reload path.
Stop whenThe active source differs, config is generated, backup is unverified, or existing upstream fragment collides.
If this step fails
Backup path is itself included by nginx.
Likely causeAn include wildcard loads all files regardless of extension/location.
sudo nginx -T 2>&1 | grep before-upstreamsudo grep -R -n 'include ' /etc/nginx/nginx.conf /etc/nginx/conf.d
ResolutionMove backup outside included directories and parse again.
Security notes
- Config copies expose private backend addresses and auth rules; keep root-only and outside HTTP roots.
Alternatives
- Use a version-controlled immutable config release and deployment rollback.
Stop conditions
- The active source differs, config is generated, backup is unverified, or existing upstream fragment collides.
config
Define a bounded upstream group
Use least_conn for uneven request times, cap failures, and specify keepalive connections. Private DNS names must resolve consistently on the proxy host.
Why this step matters
A separately named upstream centralizes peer membership, scheduling, failure thresholds, and idle connection reuse without exposing backend details in public site logic.
What to understand
least_conn fits uneven request duration but does not guarantee equal counts. max_fails/fail_timeout are passive and depend on actual proxy failure classification.
keepalive 32 caches idle connections per worker; coordinate with backend connection capacity and set HTTP/1.1/Connection behavior in the proxy route.
Static names may not dynamically re-resolve on address change. Prefer stable private addresses or an explicit supported resolver design.
System changes
- Creates `/etc/nginx/conf.d/30-{{appName}}-upstream.conf` with two private peers and connection policy.
Syntax explained
least_conn- Chooses an available peer with fewer active connections.
max_fails=3- Counts qualifying failures before temporary unavailability.
fail_timeout=30s- Defines failure-count and temporary-unavailable window.
keepalive 32- Caches up to 32 idle upstream connections per worker.
/etc/nginx/conf.d/30-{{appName}}-upstream.confValues stay on this page and are never sent or saved.
printf 'upstream {{appName}}_backend {\n least_conn;\n server {{backendA}}:{{backendPort}} max_fails=3 fail_timeout=30s;\n server {{backendB}}:{{backendPort}} max_fails=3 fail_timeout=30s;\n keepalive 32;\n}\n' | sudo tee /etc/nginx/conf.d/30-{{appName}}-upstream.confupstream app_backend {
server 10.20.0.11:8080 max_fails=3 fail_timeout=30s;
server 10.20.0.12:8080 max_fails=3 fail_timeout=30s;Checkpoint: Checkpoint: upstream
Continue whenOne unique group contains exactly two private peers, reviewed passive thresholds, and capacity-compatible keepalive.
Stop whenappName/group collides, peer is public/unhealthy, DNS is unstable, backend connection capacity is unknown, or state parity is absent.
If this step fails
nginx reports duplicate upstream name.
Likely causeAnother included file defines the same appName group.
sudo nginx -T 2>&1 | grep -n 'upstream checkout_backend'
ResolutionChoose one authoritative uniquely named group and remove/archive the duplicate.
Security notes
- Restrict backend firewall to nginx and management; private addressing alone is not access control.
Alternatives
- Use a supported discovery/active-health load balancer for dynamic fleets.
Stop conditions
- appName/group collides, peer is public/unhealthy, DNS is unstable, backend connection capacity is unknown, or state parity is absent.
config
Route the public host to the upstream
Set HTTP/1.1 keepalive semantics, preserve Host and client context, and apply finite connect, send, and read timeouts.
Why this step matters
The public location must use keepalive-compatible HTTP/1.1, bounded phases, and a single explicit forwarding trust model while preserving application Host behavior deliberately.
What to understand
`Connection ""` removes hop-by-hop close semantics so upstream keepalive can work. Host is preserved only if the application validates it.
`$proxy_add_x_forwarded_for` appends the direct peer to any incoming XFF; the application must trust known rightmost hops, not arbitrary first values.
Three seconds to connect, 30 to send, and 60 to read are initial phase-specific limits; uploads/streaming/long jobs need route-specific design.
The replacement assumes one simple existing `location /`; inspect the diff because nested/generated locations require declarative editing.
System changes
- Replaces the public root location with the upstream route, headers, HTTP version, and timeout policy.
Syntax explained
proxy_pass http://group- Routes to nginx's named peer group.
proxy_http_version 1.1- Enables upstream HTTP/1.1 needed for keepalive and many upgrades.
proxy_set_header Connection ""- Clears hop-by-hop Connection header for reuse.
Host $host- Passes nginx-normalized public host to the application.
X-Forwarded-For $proxy_add_x_forwarded_for- Appends the direct peer to the chain; trust must be right-to-left/known hops.
proxy_*_timeout- Bounds connection establishment, request send, and response read phases separately.
/etc/nginx/sites-available/{{domain}}Values stay on this page and are never sent or saved.
sudo perl -0pi -e 's#location / \{[^}]+\}#location / {\n proxy_pass http://{{appName}}_backend;\n proxy_http_version 1.1;\n proxy_set_header Connection "";\n proxy_set_header Host $host;\n proxy_set_header X-Real-IP $remote_addr;\n proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n proxy_set_header X-Forwarded-Proto $scheme;\n proxy_connect_timeout 3s;\n proxy_send_timeout 30s;\n proxy_read_timeout 60s;\n }#s' /etc/nginx/sites-available/{{domain}}proxy_pass http://app_backend; proxy_connect_timeout 3s; proxy_read_timeout 60s;
Checkpoint: Checkpoint: proxy
Continue whenOne intended public location targets the group with correct HTTP/headers/timeouts and no unrelated location changes.
Stop whenReplacement crosses nested/other locations, Host/trusted-proxy model is unknown, or legitimate route latency exceeds limits.
If this step fails
nginx -t reports unexpected `}` or duplicate location.
Likely causeRegex replacement matched a nested/non-simple block incorrectly.
diff -u /etc/nginx/sites-available/app.example.com.before-upstream /etc/nginx/sites-available/app.example.comsudo nginx -t
ResolutionRestore and manage the complete site block explicitly/declaratively.
Security notes
- Do not trust leftmost XFF or pass client-controlled forwarding headers without a defined chain.
Alternatives
- Write a complete versioned server file rather than regex-editing a complex production config.
Stop conditions
- Replacement crosses nested/other locations, Host/trusted-proxy model is unknown, or legitimate route latency exceeds limits.
config
Limit retries to safe failure classes
Retry connection and gateway failures only. Do not retry arbitrary non-idempotent application requests unless the application design explicitly supports it.
Why this step matters
Retries improve availability for connection/gateway failures but must be bounded and respect method/idempotency to avoid multiplying load or side effects.
What to understand
Select only error, timeout, and 502-504 classes and at most two tries, matching the two-peer group.
nginx normally avoids sending non-idempotent requests to another upstream unless explicitly allowed; do not add `non_idempotent` casually.
Even an idempotent-looking request can have application side effects; use request IDs/idempotency keys and logs to detect ambiguity.
The grep guard makes the Quick path idempotent rather than duplicating policy on rerun.
System changes
- Adds one bounded `proxy_next_upstream` policy and two-attempt ceiling to the public location.
Syntax explained
error timeout- Retries network/protocol errors and timeouts.
http_502 http_503 http_504- Retries selected gateway/unavailable statuses.
proxy_next_upstream_tries 2- Caps total upstream attempts at the number of peers.
grep guard || sed- Prevents duplicate insertion when exact policy already exists.
/etc/nginx/sites-available/{{domain}}Values stay on this page and are never sent or saved.
sudo grep -q 'proxy_next_upstream_tries 2;' /etc/nginx/sites-available/{{domain}} || sudo sed -i '/proxy_connect_timeout 3s;/a\ proxy_next_upstream error timeout http_502 http_503 http_504;\n proxy_next_upstream_tries 2;' /etc/nginx/sites-available/{{domain}}proxy_next_upstream error timeout http_502 http_503 http_504; proxy_next_upstream_tries 2;
Checkpoint: Checkpoint: retry policy
Continue whenPolicy appears once, is limited to reviewed failures/two tries, and does not include non_idempotent.
Stop whenApplication idempotency is unknown, broader statuses/methods are requested, policy duplicates, or retry latency exceeds SLO.
If this step fails
A POST is duplicated after response loss.
Likely causeApplication/client retried or explicit non-idempotent policy exists; first peer may have committed.
sudo nginx -T 2>&1 | grep -n proxy_next_upstreamReview app audit/idempotency logs
ResolutionRemove unsafe expansion and implement end-to-end idempotency/transaction semantics.
Security notes
- Retries can amplify abuse/overload; bound attempts and preserve rate/body/auth controls.
Alternatives
- Return a clear failure and let an idempotency-aware client/application retry.
Stop conditions
- Application idempotency is unknown, broader statuses/methods are requested, policy duplicates, or retry latency exceeds SLO.
verification
Validate expanded upstream configuration
Test syntax and inspect the expanded upstream plus proxy block before reload.
Why this step matters
Complete syntax/expanded review must prove upstream membership, route scope, retry safety, private endpoints, and no accidental public listener before reload.
What to understand
Run nginx -t and inspect full upstream/location order, headers, timeouts, retry flags, server_name, TLS, and log paths.
Search all includes for duplicate group/proxy directives and assert neither backend address appears in a public listen directive.
Capture a protected diff against the backup; parser success cannot identify wrong-but-valid peer addresses.
System changes
- No persistent change; parses and prints active nginx configuration.
Syntax explained
nginx -t- Rejects syntax/reference errors before activation.
nginx -T- Prints expanded upstream and proxy policy for review.
grep -A12- Provides a bounded review excerpt; full protected dump remains authoritative.
Values stay on this page and are never sent or saved.
sudo nginx -t && sudo nginx -T 2>&1 | grep -A12 -E 'upstream {{appName}}_backend|proxy_pass http://{{appName}}_backend'nginx: configuration file /etc/nginx/nginx.conf test is successful
upstream app_backend {
server 10.20.0.11:8080;Checkpoint: Checkpoint: validate
Continue whenParser is clean and exactly two private peers, one route, bounded safe retries, and correct trust/timeouts appear.
Stop whenAny warning, duplicate, wrong address/port, public backend, unsafe retry, missing timeout, or unrelated diff exists.
If this step fails
Config parses but group points at production database port.
Likely causeA placeholder or copied backendPort is wrong.
sudo nginx -T 2>&1 | grep -A10 upstreamsudo ss -ntp
ResolutionCorrect and directly prove exact application endpoints before reload.
Security notes
- Protect expanded config because it exposes topology; validate semantics, not syntax only.
Alternatives
- Use configuration-management schema tests and staging probes before deployment.
Stop conditions
- Any warning, duplicate, wrong address/port, public backend, unsafe retry, missing timeout, or unrelated diff exists.
verification
Reload and observe traffic distribution
Send several bounded requests and inspect a backend marker header or body to prove both nodes receive traffic.
Why this step matters
Repeated public requests and a trustworthy backend marker prove both nodes participate; least_conn distribution is evaluated for presence/correctness, not exact equality.
What to understand
Reload only after validation, then test public TLS, Host, headers, session behavior, and a loop of bounded requests.
Use an authenticated/non-spoofable node marker in staging or correlate upstream_addr in edge logs; a body marker should not leak unnecessary topology publicly.
Test with realistic concurrency because sequential fast requests can favor keepalive/timing patterns.
System changes
- Gracefully reloads nginx and creates public/backend request and log records.
Syntax explained
seq 1 10- Generates a small deterministic sample, not a load test.
sort | uniq -c- Counts observed node markers without expecting exact balance.
curl --fail --silent- Requires successful responses and emits marker bodies.
Values stay on this page and are never sent or saved.
sudo systemctl reload nginx && for i in $(seq 1 10); do curl --fail --silent https://{{domain}}/node; done | sort | uniq -c4 node-a 6 node-b
Checkpoint: Checkpoint: distribution test
Continue whenBoth nodes appear, every response is correct, public latency/session behavior is healthy, and logs identify selected peers.
Stop whenA node never participates due to failure/config, any response differs, state/session breaks, or latency/error budget regresses.
If this step fails
Counts are 10/0 but both direct health checks pass.
Likely causeleast_conn/keepalive/timing, passive failed state, config not reloaded, or marker caching.
sudo tail -n 100 /var/log/nginx/app.example.com-access.logsudo tail -n 100 /var/log/nginx/app.example.com-error.logsudo nginx -T 2>&1 | grep -A8 upstream
ResolutionUse upstream log fields and concurrent uncached tests; investigate health before changing scheduler.
Security notes
- Avoid exposing permanent public node identity; prefer restricted diagnostics/log correlation.
Alternatives
- Run a controlled load test with structured upstream logs and service objectives.
Stop conditions
- A node never participates due to failure/config, any response differs, state/session breaks, or latency/error budget regresses.
warning
Rehearse one backend failure
Stop one test backend during a controlled window, confirm public requests remain successful within the timeout budget, restart it, and watch nginx errors clear.
Why this step matters
Redundancy is unproven until one peer fails, nginx avoids it within the budget, the application remains correct, and the recovered peer re-enters safely.
What to understand
Verify SSH host key, exact node/service, reverse dependencies, and maintenance approval before stop. Never paste an unverified hostname/service into privileged automation.
During outage, run repeated public health and representative transactions with max-time, record upstream_addr/status/time and ensure no duplicate writes.
Restart the node, wait for direct readiness, then observe it rejoin after passive state/keepalive effects. Compare logs and state consistency.
If production interruption is not permitted, rehearse on a faithful staging pair and separately test drain/readiness during deployment.
System changes
- Stops/starts one remote backend service and intentionally creates passive failure/recovery state and log events.
Syntax explained
ssh backendA- Targets the reviewed private host using existing authenticated host-key-verified access.
systemctl stop/start appService- Controls the exact reviewed application lifecycle.
curl --max-time 5- Bounds each public observation during failure.
tail domain-error.log- Shows passive disable/recovery evidence.
Values stay on this page and are never sent or saved.
ssh {{backendA}} 'sudo systemctl stop {{appService}}' && for i in $(seq 1 5); do curl --fail --max-time 5 https://{{domain}}/health; done; ssh {{backendA}} 'sudo systemctl start {{appService}}' && sudo tail -n 30 /var/log/nginx/{{domain}}-error.lognode-b ok node-b ok upstream server temporarily disabled while connecting to upstream
Checkpoint: Checkpoint: failure rehearsal
Continue whenAll public checks remain correct/within budget on B, A stop is visible, A restarts/readies, and later requests show both peers without state loss.
Stop whenSSH identity/service/dependencies differ, public failure/latency/duplicates occur, B lacks capacity, A restart/readiness fails, or state diverges.
If this step fails
Stopping A also makes B unhealthy.
Likely causeShared database/cache/storage/DNS dependency or coordinated release/state incompatibility failed.
Check both direct health endpointsReview shared dependency monitoring and backend journals
ResolutionRestore A, stabilize shared dependency, and redesign/monitor the actual single point of failure.
Security notes
- Remote privileged service control requires audited access; do not embed passwords/keys or disable host verification.
Alternatives
- Use an approved orchestrator drain/failure injection or staging rehearsal with identical health and state design.
Stop conditions
- SSH identity/service/dependencies differ, public failure/latency/duplicates occur, B lacks capacity, A restart/readiness fails, or state diverges.
Finish line
Verification checklist
for u in http://{{backendA}}:{{backendPort}} http://{{backendB}}:{{backendPort}}; do curl --fail --max-time 3 "$u/health"; doneEvery backend returns its expected health marker.for i in $(seq 1 10); do curl --fail --max-time 5 https://{{domain}}/health; doneAll requests succeed and total latency stays within the defined timeout budget.Recovery guidance
Common problems and safe checks
All traffic goes to one node.
Likely causeleast_conn legitimately sees fewer active connections there, the other peer is failed/unreachable, responses are cached, or node marker is not reliable.
sudo nginx -T 2>&1 | grep -A8 'upstream checkout_backend'curl -s http://10.20.0.11:8080/nodecurl -s http://10.20.0.12:8080/nodesudo tail -n 100 /var/log/nginx/app.example.com-error.log
ResolutionVerify peer health/config and observe under representative concurrency; do not demand exact equal counts.
One dead node causes long public delays.
Likely causeConnect/read timeouts are too high, failure is an accepted-but-stalled connection, retry classes exclude it, or both peers share the failing dependency.
curl --max-time 10 -w '%{http_code} %{time_total}\n' -o /dev/null https://app.example.com/healthReview `$upstream_addr`, `$upstream_status`, `$upstream_response_time` logs
ResolutionFix the failure/dependency and tune phase-specific bounds from real latency; do not mask overload with unlimited retries.
Duplicate POST side effects appear during failures.
Likely causeA non-idempotent request was retried by explicit policy/app/client, or the first backend committed before its response was lost.
sudo nginx -T 2>&1 | grep -n proxy_next_upstreamReview application idempotency/audit logs by request ID
ResolutionRemove unsafe retry expansion, implement application idempotency keys/transactions, and treat duplicates as an incident.
New backend DNS address is not used.
Likely causeStatic upstream names were resolved when configuration loaded and no dynamic resolver mechanism is configured.
getent ahosts backend-a.internalsudo nginx -T 2>&1 | grep -A8 upstreamsudo ss -ntp | grep ':8080'
ResolutionUse stable addresses or a supported dynamic-resolution/service-discovery design and test reload/failure behavior.
Application sees spoofed/incorrect client IP.
Likely causeIt trusts the first XFF value, multiple proxies append inconsistently, or nginx remote_addr is another load balancer.
Review edge `proxy_set_header` directivesReview application trusted proxy/hop configurationSend a controlled spoofed X-Forwarded-For request
ResolutionDefine the exact proxy chain, overwrite at the authoritative edge, and trust only known hops from the rightmost side.
Reference
Frequently asked questions
Should ten requests split five and five?
No. least_conn reacts to active connection counts and timing; prove both peers participate and service objectives hold.
Does keepalive 32 limit the backend to 32 connections?
No. It caches up to that many idle connections per worker; active connections can exceed it.
Are open-source nginx health checks active?
This configuration uses passive failures from real requests. Use a supported active-health design if periodic probes are required.
Recovery
Rollback
Restore the previous single-upstream server block and remove the upstream group only after nginx validates.
- Restore {{domain}}.before-upstream over the active server block.
- Remove /etc/nginx/conf.d/30-{{appName}}-upstream.conf.
- Run nginx -t and reload nginx.
- Keep both backends private and running until clients have been verified on the restored path.
Evidence