Configure HAProxy load balancing with health checks and TLS termination
Deploy HAProxy 3.2 LTS as a validated HTTPS load balancer with application-aware health checks, observable backend state, controlled certificate renewal, and zero-downtime reloads.
Publish one HTTPS service across two application nodes while keeping failed nodes out of rotation, preserving client-facing TLS identity, and making every configuration and certificate change reversible.
- Ubuntu Server 24.04 LTS
- HAProxy 3.2 LTS
- OpenSSL 3.x
- Two independently testable application nodes Each backend exposes an unauthenticated but non-sensitive readiness endpoint and a node marker on a private address. Operators can stop one test node without creating an unrelated outage.
curl --fail http://{{backendOneAddress}}:{{backendPort}}{{healthPath}} && curl --fail http://{{backendTwoAddress}}:{{backendPort}}{{healthPath}} - Owned DNS, certificate, and maintenance path The public hostname resolves to the load balancer, the certificate chain and private key are available through an approved renewal workflow, and console access exists if remote routing is interrupted.
- A reviewed client and proxy trust model Decide where TLS ends, whether backends need TLS, which forwarded headers applications trust, and whether PROXY protocol is supported end-to-end before copying a frontend.
- Monitoring and rollback ownership Name owners for public probes, certificate expiry, backend availability, queue time, response errors, reload failures, old workers, and the last known-good configuration and PEM bundle.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A package-managed HAProxy 3.2 LTS instance with a plaintext redirect frontend, a TLS frontend, explicit modern protocol defaults, one application backend, and two named servers reached on private addresses.
- Layer-seven readiness checks that request a deliberately small application endpoint, require the intended status, remove a failing node only after a bounded threshold, and reintroduce it gradually with slow start.
- A local, permission-restricted Runtime API socket that exposes process, server, queue, health, and traffic state without publishing administrative commands on a network listener.
- A certificate lifecycle that stages a complete PEM bundle, validates identity and chain before install, reloads without dropping established connections, and alerts before expiry or renewal failure.
- A client using the public hostname is redirected to HTTPS, receives the expected certificate and application response, and can reach either healthy backend without learning its private address.
- When one readiness endpoint fails, HAProxy marks only that node DOWN after the configured fall threshold, sends new traffic to the remaining node, and returns the recovered node through slow start.
- Every configuration change is rejected before reload if parsing, binding, file access, certificate loading, or referenced object validation fails.
- Operators can distinguish public TLS failure, frontend saturation, queueing, backend transport failure, application readiness failure, and reload failure from Runtime API, journal, and external probes.
Architecture
How the parts fit together
Clients resolve one public hostname and connect to HAProxy. Port 80 performs only a deterministic HTTPS redirect. Port 443 terminates TLS with the staged certificate bundle, normalizes explicitly trusted forwarding headers, and selects the application backend. The backend distributes new connections with leastconn and probes each node's readiness path independently of user traffic. HAProxy's master-worker lifecycle lets systemd load a validated configuration while old workers drain established connections. A local Unix Runtime API socket reports the live state that static configuration cannot show.
- A client reaches port 80 and receives a redirect, or connects directly to 443 with SNI for the public hostname.
- HAProxy selects the matching certificate, negotiates TLS, adds controlled forwarding metadata, and applies frontend policy.
- Leastconn chooses an eligible server. Servers that fail the L7 readiness contract are absent from new selection; existing connection behavior follows application protocol and timeout policy.
- The response returns through HAProxy while counters and health state update in memory and are visible through the local Runtime API.
- A change is validated offline, atomically installed, and reloaded through the master-worker service. New workers accept new sessions while prior workers drain.
Assumptions
- This is a single HAProxy node tutorial. It makes the application tier redundant but does not make the load-balancer host itself highly available. Add an independently designed VIP, anycast, cloud load balancer, or second proxy tier when that failure domain is unacceptable.
- The example targets HAProxy 3.2 LTS on Ubuntu 24.04 and uses package-managed systemd master-worker behavior. Exact default paths, user, crypto library, and unit flags are confirmed locally before deployment.
- Backends speak HTTP on a protected network. If network trust requires backend TLS, certificate identity validation and rotation must be configured rather than changing only the port to 443.
- The readiness path is inexpensive, cannot mutate state, returns no secrets, and accurately represents dependencies required to accept a new request. It is distinct from liveness when restarting a process would be harmful.
- The application understands the selected forwarding headers and trusts them only from the proxy. Client-supplied versions are removed or overwritten at the boundary.
- Session state is externalized or replicated. Load balancing does not make an in-memory login, upload, transaction, or WebSocket recover on another node.
Key concepts
- TLS termination
- HAProxy proves the public server identity, decrypts the client connection, and applies HTTP policy. The backend leg is a separate security boundary and may require its own TLS.
- Active health check
- A scheduled synthetic request generated by HAProxy. The server becomes eligible or ineligible according to response validation and rise/fall thresholds.
- Readiness
- A contract that says a node can accept new work now. It can fail while the process remains alive, for example when a required datastore is unavailable.
- Slow start
- A recovery interval during which HAProxy gradually increases a server's effective weight so cold caches and connection pools are not flooded.
- Master-worker reload
- A lifecycle in which the master starts a new validated worker and asks older workers to drain, preserving listeners and established sessions where protocol behavior permits.
- Runtime API
- A command socket exposing the in-memory state of the running process. It complements configuration validation and must be protected because some commands alter traffic.
- Connection draining
- Stopping assignment of new connections while allowing existing sessions a bounded time to finish. It is not application-level transaction migration.
Before you copy
Values used in this guide
{{publicHostname}}Public DNS name covered by the TLS certificate.
Example: app.example.net{{listenAddress}}Approved local address on which the public HAProxy frontends bind.
Example: 192.0.2.80{{backendHostname}}Application hostname sent in direct baseline requests and backend health checks.
Example: app.internal.example.net{{backendOneAddress}}Private address of the first application node.
Example: 10.30.0.21{{backendTwoAddress}}Private address of the second application node.
Example: 10.30.0.22{{backendPort}}Private application HTTP port.
Example: 8080{{backendService}}Exact application service unit stopped only during the controlled health-failure exercise.
Example: example-app.service{{healthPath}}Read-only application readiness endpoint.
Example: /readyz{{leafCertificate}}Staged public leaf certificate issued for the public hostname.
Example: /var/tmp/app.example.net.crt{{intermediateChain}}Ordered intermediate CA certificates needed to build the served chain.
Example: /var/tmp/app.example.net-chain.crt{{privateKey}}secretRestricted private key matching the leaf certificate.
Example: /run/credentials/haproxy/app.example.net.key{{certificatePem}}Staged PEM containing private key, leaf, and chain.
Example: /etc/haproxy/certs/app.example.net.pem{{candidatePem}}Candidate PEM path checked before atomic installation.
Example: /var/tmp/app.example.net.pem{{nextPem}}Newly assembled renewal candidate verified before it replaces the active PEM.
Example: /var/tmp/app.example.net-next.pem{{candidateConfig}}Candidate HAProxy configuration path.
Example: /etc/haproxy/candidates/haproxy.cfgSecurity and production boundaries
- The PEM contains the private key. Restrict its owner and mode, exclude it from logs and repositories, use an approved secret-delivery path, and rotate immediately after suspected exposure.
- TLS termination makes HAProxy a plaintext visibility boundary. Protect memory, logs, dumps, Runtime API access, observability exports, and the backend network accordingly.
- Do not forward arbitrary client-supplied `X-Forwarded-*` or PROXY metadata. Normalize headers and configure applications to trust only the proxy source range.
- A health endpoint can reveal version, dependency, or topology details. Return the minimum status needed by the proxy and retain diagnostic details in authenticated telemetry.
- The Runtime API is not a read-only protocol. A writable socket can disable servers or change runtime state; limit filesystem access and give monitoring a least-privilege path.
Stop before continuing if
- Stop if the public hostname, ownership, certificate renewal authority, or rollback destination is ambiguous.
- Stop if the backend readiness endpoint mutates state, exposes secrets, or reports success while dependencies required for real requests are unavailable.
- Stop if direct backend access bypasses authentication, rate limits, or trusted forwarding-header policy enforced at HAProxy.
- Stop if `haproxy -c -V` reports any error or warning that has not been understood; never reload to discover whether invalid syntax is tolerated.
- Stop if certificate key, SAN, chain, permissions, or expiry verification fails.
- Stop a failure exercise if the remaining backend lacks capacity, error rates grow, or session semantics are unknown.
verification
Baseline DNS, listeners, backend identity, and the health contract
Resolve the public name, prove ports 80 and 443 are free on the load balancer, query every backend directly with the same Host header, and define exactly which response means the application is ready for traffic.
Why this step matters
HAProxy can only make a useful routing decision when backend identity, application readiness, and the client hostname are correct before insertion. A TCP accept alone can route clients to a process whose database or migrations are broken.
What to understand
The health endpoint should be cheap, unauthenticated only from the load-balancer network, and represent dependencies required to serve ordinary requests without triggering expensive downstream work.
Use the same Host header and protocol HAProxy will send. A direct IP curl that selects a default virtual host can produce a false positive.
Record response body, status, latency, node marker, certificate expectations, and release revision for both backends.
System changes
- No changes; creates backend readiness, DNS, listener, and health-contract evidence.
Syntax explained
curl --resolve- Connects to the selected backend address while preserving the intended Host name.
-D -- Prints response headers so status, node identity, and caching behavior are visible.
Values stay on this page and are never sent or saved.
getent ahostsv4 {{publicHostname}}; sudo ss -lntp | grep -E ':(80|443)\b' || true; curl --fail --silent --show-error --resolve {{backendHostname}}:{{backendPort}}:{{backendOneAddress}} http://{{backendHostname}}:{{backendPort}}{{healthPath}} -D -; curl --fail --silent --show-error --resolve {{backendHostname}}:{{backendPort}}:{{backendTwoAddress}} http://{{backendHostname}}:{{backendPort}}{{healthPath}} -D -192.0.2.80 STREAM app.example.net
HTTP/1.1 200 OK
Content-Type: application/json
X-Node: app-01
{"status":"ready","database":"ok","revision":"2026.07.28"}
HTTP/1.1 200 OK
X-Node: app-02
{"status":"ready","database":"ok","revision":"2026.07.28"}Checkpoint: Both backends satisfy one explicit readiness contract
curl --fail --resolve {{backendHostname}}:{{backendPort}}:{{backendOneAddress}} http://{{backendHostname}}:{{backendPort}}{{healthPath}}Continue whenEach backend returns the approved status and body within the timeout, with the expected node and revision.
Stop whenPorts conflict, DNS points elsewhere, backend identity differs, or readiness does not cover required dependencies.
If this step fails
Direct health is 200 but real application requests fail.
Likely causeThe health endpoint is shallow, uses another virtual host, or omits required dependencies.
curl -v -H 'Host: {{backendHostname}}' http://{{backendOneAddress}}:{{backendPort}}{{healthPath}}curl -v -H 'Host: {{backendHostname}}' http://{{backendOneAddress}}:{{backendPort}}/
ResolutionRedesign readiness to represent the minimum service contract while remaining bounded and non-destructive.
Security notes
- Keep detailed dependency errors out of the unauthenticated health body; expose only bounded status to HAProxy.
Alternatives
- Use agent checks when application capacity needs more than binary ready/down state, after authenticating and constraining the agent path.
Stop conditions
- Do not place a backend in rotation until its health contract is meaningful and repeatable.
command
Install HAProxy 3.2 LTS and local Runtime API tooling
Install the supported package, record build options and linked TLS library, and add a local Unix-socket client without exposing an administrative TCP endpoint.
Why this step matters
Build features determine TLS, health, threading, and runtime capabilities. Recording the packaged systemd unit also verifies that reload uses master-worker semantics and validates configuration.
What to understand
This guide targets HAProxy 3.2 LTS. Do not copy directives from newer 3.3/3.4 documentation without checking availability in `haproxy -vv` and the 3.2 configuration manual.
The Runtime API uses a Unix socket restricted by mode and group. A public admin-level TCP socket is an unnecessary takeover surface.
Package service units may already provide graceful reload. Preserve vendor updates by using drop-ins only when a proven change is required.
System changes
- Installs HAProxy, service units, default configuration, logging integration, and socat.
Syntax explained
haproxy -vv- Shows exact version, support branch, build flags, TLS library, paths, and threading capabilities.
systemctl cat- Displays the effective packaged unit and drop-ins without editing them.
sudo apt-get update && sudo apt-get install --yes haproxy socat && haproxy -vv | sed -n '1,35p' && systemctl cat haproxy | sed -n '1,100p'HAProxy version 3.2.4-1ppa1~noble Status: long-term supported branch - will stop receiving fixes around Q2 2030. Built with OpenSSL version : OpenSSL 3.0.13 30 Jan 2024 Running on OpenSSL version : OpenSSL 3.0.13 30 Jan 2024 Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256) Features : +EPOLL +NETFILTER +PCRE2 +OPENSSL +LUA +PROMEX ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS ExecReload=/usr/sbin/haproxy -Ws -f $CONFIG -c -q
Checkpoint: The installed build supports TLS and master-worker reload
haproxy -vv | grep -E 'version|OpenSSL|multi-thread'Continue whenHAProxy 3.2 LTS, OpenSSL enabled, and the service starts with -W or -Ws master-worker mode.
Stop whenThe branch is unsupported, TLS is absent, or the service uses an unknown custom startup wrapper.
If this step fails
Package version is older than the documented configuration syntax.
Likely causeThe host uses a distribution branch or repository different from the reviewed 3.2 LTS source.
apt-cache policy haproxyhaproxy -vv
ResolutionChoose a supported signed package source or adapt to the exact official version manual; do not mix syntax by trial.
Security notes
- Review package provenance and security updates because HAProxy terminates untrusted internet traffic and holds private keys.
Alternatives
- Use the distribution-supported HAProxy release when its security lifetime and required features satisfy the design.
Stop conditions
- Do not deploy a configuration against an unverified executable or service wrapper.
command
Assemble and verify the TLS certificate bundle
Build a mode-0600 PEM containing leaf certificate, intermediate chain, and private key; verify key match, SAN, validity, and chain before HAProxy can read it.
Why this step matters
HAProxy must load a matching private key and complete chain for the requested hostname. Validating before config reload prevents an unreadable or mismatched renewal from taking the listener out of service.
What to understand
The bind `crt` format accepts a PEM with certificate chain and key. Keep the leaf before intermediates and never include the root CA unnecessarily.
SAN, not only subject CN, must contain the public hostname. Check UTC validity and renewal headroom.
The haproxy process needs read permission, but ordinary users and backend services do not. Backups of this bundle are secret material.
System changes
- Creates the restricted certificate directory and writes a combined private-key-bearing PEM.
Syntax explained
openssl x509 -ext subjectAltName- Displays identities modern clients validate and the certificate validity interval.
openssl pkey/x509 -modulus- Compares public modulus digests to prove the private key matches the leaf certificate.
Values stay on this page and are never sent or saved.
sudo install -d -o root -g haproxy -m 0750 /etc/haproxy/certs && sudo sh -c 'cat {{leafCertificate}} {{intermediateChain}} {{privateKey}} > /etc/haproxy/certs/{{publicHostname}}.pem' && sudo chown root:haproxy /etc/haproxy/certs/{{publicHostname}}.pem && sudo chmod 0640 /etc/haproxy/certs/{{publicHostname}}.pem && openssl x509 -in /etc/haproxy/certs/{{publicHostname}}.pem -noout -subject -issuer -dates -ext subjectAltName && test "$(openssl x509 -in {{leafCertificate}} -pubkey -noout | openssl pkey -pubin -outform DER | openssl sha256)" = "$(openssl pkey -in {{privateKey}} -pubout -outform DER | openssl sha256)"subject=CN = app.example.net
issuer=C = US, O = Example Trust, CN = Example RSA CA 01
notBefore=Jul 20 00:00:00 2026 GMT
notAfter=Oct 18 23:59:59 2026 GMT
X509v3 Subject Alternative Name:
DNS:app.example.net
certificate_key_match=PASS
-rw-r----- 1 root haproxy 7821 Jul 28 13:02 /etc/haproxy/certs/app.example.net.pemCheckpoint: Certificate, chain, key, hostname, and permissions pass
sudo -u haproxy test -r /etc/haproxy/certs/{{publicHostname}}.pem && openssl x509 -checkend 604800 -noout -in /etc/haproxy/certs/{{publicHostname}}.pemContinue whenHAProxy can read the bundle, SAN matches, key matches, chain is complete, and at least seven days remain.
Stop whenThe key is mismatched or exposed, SAN is wrong, certificate is near expiry, or the chain is incomplete.
If this step fails
HAProxy validation reports no private key found.
Likely causeThe PEM lacks the key, uses an unsupported encrypted key without startup secret handling, or file permissions block access.
grep -n 'BEGIN .*PRIVATE KEY' /etc/haproxy/certs/{{publicHostname}}.pemsudo -u haproxy test -r /etc/haproxy/certs/{{publicHostname}}.pem
ResolutionRebuild the PEM from protected source material, verify key match and permissions, then revalidate before reload.
Security notes
- Never print, commit, email, or expose the PEM; command examples deliberately show only certificate metadata and digests.
Alternatives
- Use HAProxy 3.2 crt-store to separate certificate and key paths when that improves secret lifecycle and the exact syntax is validated.
Stop conditions
- Do not reload with an unverified or broadly readable private key.
config
Define global runtime, logging, timeouts, and safe HTTP defaults
Configure master-worker-safe runtime access, global TLS minimum, explicit connection timeouts, HTTP logging, and server-state preservation without exposing an administrative web page.
Why this step matters
Explicit process, socket, TLS, logging, state, and timeout policy creates predictable failure behavior. Missing timeouts can leave connections indefinitely, while an admin socket that is too broad can disable or redirect backends.
What to understand
Operator-level Runtime API is enough for read and common operational state. Use admin only for commands that truly require it and keep the Unix group restricted.
Server state files preserve runtime health, weight, drain, and maintenance status across reload when the service captures them correctly.
Timeouts are an application contract. Tune from observed connect and response distributions; overly long values hide failure and overly short values create false outages.
System changes
- Defines HAProxy process identity, Runtime API socket, TLS floor, logging, server-state loading, and connection timeout behavior.
Syntax explained
stats socket ... level operator- Creates a local restricted Runtime API used for status and controlled operations.
expose-fd listeners- Allows listener file descriptors to survive supported seamless reload behavior.
timeout connect/client/server- Bounds separate backend connection and client/server inactivity phases.
/etc/haproxy/haproxy.cfgglobal
log /dev/log local0
user haproxy
group haproxy
daemon
stats socket /run/haproxy/admin.sock user haproxy group haproxy mode 660 level operator expose-fd listeners
server-state-file /var/lib/haproxy/server-state
ssl-default-bind-options ssl-min-ver TLSv1.2
defaults
log global
mode http
option httplog
option http-keep-alive
timeout connect 5s
timeout client 30s
timeout server 30s
timeout http-request 10s
timeout queue 10s
load-server-state-from-file globalValues stay on this page and are never sent or saved.
sudo haproxy -c -V -f {{candidateConfig}}Configuration file is valid Warnings were found: 0 HAProxy version 3.2.4 runtime_socket=/run/haproxy/admin.sock mode=660 level=operator tls_min=TLSv1.2
Checkpoint: Core configuration validates without warnings
sudo haproxy -c -V -f {{candidateConfig}}Continue whenExit zero, no warnings, supported directives, restricted socket, and explicit timeouts.
Stop whenAny warning is unexplained, socket is network-exposed, or timeouts are copied without workload review.
If this step fails
Runtime socket exists but operators receive permission denied.
Likely causeSocket group, mode, parent directory, or operator group membership differs.
namei -l /run/haproxy/admin.sockid
ResolutionGrant the narrow HAProxy operator group and recreate through a validated reload; do not make the socket world-writable.
Security notes
- Runtime API can drain or disable servers and reveal topology; local filesystem permissions are part of the trust boundary.
Alternatives
- Use a read-only monitoring socket and a separate tightly controlled admin socket when duties must be separated.
Stop conditions
- Do not expose stats or Runtime API over unauthenticated TCP.
config
Terminate TLS, redirect HTTP, and preserve client context
Bind HTTPS with the verified certificate, redirect ordinary HTTP to HTTPS, set forwarding headers from trusted HAProxy state, and add a bounded health endpoint for the load balancer itself.
Why this step matters
The frontend is the public trust boundary: it selects the certificate, protocol, redirect, and client metadata seen by applications. HAProxy must overwrite forwarding headers so clients cannot spoof them.
What to understand
Binding to an explicit service address avoids accidental management-interface exposure. If Keepalived later owns a VIP, the host must support binding behavior and startup order intentionally.
ALPN advertises HTTP/2 and HTTP/1.1 at the TLS boundary. Backend protocol remains separately configured.
The load-balancer health path proves HAProxy request processing but deliberately does not imply backend or database readiness.
System changes
- Creates public HTTP and HTTPS listeners, redirect behavior, forwarding headers, and the frontend routing rule.
Syntax explained
bind ... ssl crt- Creates the TLS listener and loads the verified certificate/key bundle.
http-request set-header- Overwrites untrusted client-provided forwarding context with values HAProxy observed.
alpn h2,http/1.1- Advertises supported application protocols during TLS negotiation.
/etc/haproxy/haproxy.cfgValues stay on this page and are never sent or saved.
frontend fe_http
bind {{listenAddress}}:80
http-request redirect scheme https code 301 unless { ssl_fc }
frontend fe_https
bind {{listenAddress}}:443 ssl crt /etc/haproxy/certs/{{publicHostname}}.pem alpn h2,http/1.1
http-request set-header X-Forwarded-Proto https
http-request set-header X-Forwarded-For %[src]
http-request set-header X-Forwarded-Host %[req.hdr(Host)]
http-request return status 200 content-type text/plain string "ready\n" if { path /lb-healthz }
default_backend be_appValues stay on this page and are never sent or saved.
sudo haproxy -c -V -f {{candidateConfig}}Configuration file is valid frontend fe_http bind=:80 redirect=https frontend fe_https bind=:443 ssl certificate=app.example.net.pem default_backend=be_app
Checkpoint: Candidate listeners and routing parse exactly once
sudo haproxy -c -V -f {{candidateConfig}}Continue whenOne port-80 and one port-443 bind, valid certificate, redirect, headers, and default backend.
Stop whenA management interface is exposed, certificate cannot load, or client headers are passed through untrusted.
If this step fails
Clients enter a redirect loop.
Likely causeThe backend also redirects because it does not trust X-Forwarded-Proto or another proxy rewrites scheme context.
curl -vkIL http://{{publicHostname}}/curl -vkI https://{{publicHostname}}/
ResolutionDefine one redirect owner and configure the application to trust forwarding headers only from HAProxy addresses.
Security notes
- Applications must trust X-Forwarded-* only from the load-balancer network; otherwise direct clients can spoof identity and scheme.
Alternatives
- Use end-to-end TLS to backends when the internal network or compliance boundary requires encryption and identity verification.
Stop conditions
- Do not expose TLS until hostname, certificate, redirect, and forwarding trust are verified.
config
Configure balanced backends with application health checks
Use least-connections balancing, an HTTP readiness request with explicit expected status, rise/fall hysteresis, connection limits, and slow start for both named servers.
Why this step matters
Active application-level checks prevent routing to nodes that accept TCP but cannot serve. Rise/fall hysteresis avoids flapping, and slow start protects a recovered backend from an immediate surge.
What to understand
leastconn favors the server with fewer active connections and suits variable request durations. Round-robin may be more predictable for uniform stateless requests.
HTTP/1.1 checks require a Host header to reach the intended virtual host. The expected status must distinguish ready from degraded or maintenance.
maxconn limits each server and queues excess requests under the configured queue timeout; size it from backend capacity and observability.
System changes
- Defines backend selection, health request, health thresholds, per-server capacity, and recovery ramp.
Syntax explained
fall 3 rise 2- Requires three failures to mark down and two successes to return up, reducing transient flapping.
slowstart 30s- Ramps effective capacity after recovery instead of sending full traffic immediately.
http-check expect status 200- Treats only the defined readiness status as healthy.
/etc/haproxy/haproxy.cfgValues stay on this page and are never sent or saved.
backend be_app
balance leastconn
option httpchk
http-check send meth GET uri {{healthPath}} ver HTTP/1.1 hdr Host {{backendHostname}}
http-check expect status 200
default-server check inter 2s fall 3 rise 2 slowstart 30s maxconn 500
server app-01 {{backendOneAddress}}:{{backendPort}}
server app-02 {{backendTwoAddress}}:{{backendPort}}Values stay on this page and are never sent or saved.
sudo haproxy -c -V -f {{candidateConfig}}Configuration file is valid backend=be_app algorithm=leastconn check=GET /ready Host: app.internal.example.net expect=200 servers=app-01,app-02 inter=2s fall=3 rise=2 slowstart=30s
Checkpoint: Health check matches direct backend readiness
curl --fail -H 'Host: {{backendHostname}}' http://{{backendOneAddress}}:{{backendPort}}{{healthPath}}Continue whenBoth nodes return exactly the expected status/body and tolerate the configured interval.
Stop whenHealth checks mutate state, overload dependencies, or report ready while representative requests fail.
If this step fails
All backends oscillate UP and DOWN together.
Likely causeThe shared health dependency, Host header, timeout, or endpoint is wrong rather than every node failing independently.
echo 'show stat' | socat stdio /run/haproxy/admin.sockcurl -v -H 'Host: {{backendHostname}}' http://{{backendOneAddress}}:{{backendPort}}{{healthPath}}
ResolutionCorrect the shared health contract and tune bounded timeouts from observed latency before re-enabling automatic traffic.
Security notes
- Restrict readiness endpoints to the load-balancer network and avoid returning dependency credentials or stack traces.
Alternatives
- Use roundrobin for uniform short requests or source hashing only when stickiness is a deliberate application requirement.
Stop conditions
- Do not route production traffic with an unrepresentative or flapping health check.
command
Validate and perform a zero-downtime reload
Back up the live configuration, validate the exact candidate including certificate access, install it atomically, reload through systemd, and verify that the master remains while worker generation changes.
Why this step matters
HAProxy master-worker reload starts a validated new worker while the old worker drains existing connections. Keeping the master PID and probing through the change demonstrates the intended zero-downtime lifecycle.
What to understand
Validation must run before replacing the live file and must include certificate readability. Treat warnings as failures until explicitly understood.
The new worker accepts new connections; old workers can remain while long-lived sessions drain. Monitor them rather than killing immediately.
Run an external low-rate probe during reload and compare error, handshake, and latency counts. A successful systemctl command alone does not prove clients were uninterrupted.
System changes
- Backs up and replaces the live configuration and performs a graceful master-worker reload.
Syntax explained
haproxy -c -V- Validates all configuration files and emits diagnostics without starting listeners.
systemctl reload- Invokes the packaged graceful reload path instead of a stop/start outage.
show info- Confirms the running worker generation, uptime, connections, and process health.
Values stay on this page and are never sent or saved.
sudo install -m 0600 /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.before-change && sudo haproxy -c -V -f {{candidateConfig}} && sudo install -m 0640 -o root -g haproxy {{candidateConfig}} /etc/haproxy/haproxy.cfg && old=$(systemctl show -p MainPID --value haproxy); sudo systemctl reload haproxy; new=$(systemctl show -p MainPID --value haproxy); printf 'master_before=%s master_after=%s\n' "$old" "$new"; sudo systemctl is-active haproxy; echo 'show info' | sudo socat stdio /run/haproxy/admin.sock | grep -E '^(Pid|Uptime|Nbproc|CurrConns)'Configuration file is valid master_before=1810 master_after=1810 active Pid: 2241 Uptime: 0d 0h00m03s Nbproc: 1 CurrConns: 42 reload_probe_failures=0
Checkpoint: Reload changes workers without client failures
systemctl is-active haproxy && curl --fail --silent https://{{publicHostname}}/lb-healthzContinue whenMaster remains, new worker serves, old sessions drain, TLS probe succeeds, and external failure count stays zero.
Stop whenValidation warns, reload returns nonzero, new worker fails, or probes observe resets or TLS errors.
If this step fails
Reload succeeds but old workers never exit.
Likely causeLong-lived sessions, WebSockets, tunnel traffic, or missing client/server timeouts prevent draining.
ps -ef | grep '[h]aproxy'echo 'show sess' | socat stdio /run/haproxy/admin.sock | head
ResolutionIdentify session classes and set reviewed graceful-drain policy; do not kill old workers without understanding client impact.
Security notes
- The backup configuration may contain topology and certificate paths; keep it mode 0600 and rotate it through configuration management.
Alternatives
- Use Runtime API changes for temporary drain or weight operations, but persist intended state in configuration before the next reload.
Stop conditions
- Never replace or reload a candidate that has not passed the exact binary's validation.
decision
Prove backend withdrawal and controlled return
Stop the disposable backend application or make its readiness endpoint fail, observe HAProxy mark it DOWN after fall, verify all requests reach the healthy node, then restore and watch rise plus slow start.
Why this step matters
A controlled backend failure validates check thresholds, withdrawal, capacity, user continuity, state visibility, and slow recovery without taking the load balancer itself down.
What to understand
Choose one disposable or drained backend and confirm the remaining node can handle test traffic. Do not perform the first failure test at peak capacity.
The expected transition occurs after fall multiplied by interval plus request time. Verify queue and latency remain within objectives.
On return, rise requires consecutive successes and slowstart gradually restores load. This provides time to expose a bad warmup.
System changes
- Stops and later restarts one backend application, temporarily concentrating traffic on the other node.
Syntax explained
show stat- Reports server status, health result, check code, failures, sessions, and throttle from the running process.
fall/rise- Controls the number of consecutive check results before state changes.
Values stay on this page and are never sent or saved.
sudo systemctl stop {{backendService}}; sleep 8; echo 'show stat' | sudo socat stdio /run/haproxy/admin.sock | grep 'be_app,app-01'; for i in $(seq 1 8); do curl --silent --show-error https://{{publicHostname}}/node; echo; donebe_app,app-01,0,0,0,0,0,0,DOWN,1,1,3,0,L7STS,503,Health check returned error app-02 app-02 app-02 app-02 after restore: app-01 status=UP throttle=17% -> 100% over 30s
Checkpoint: Failed backend receives no traffic and returns gradually
echo 'show stat' | socat stdio /run/haproxy/admin.sock | grep 'be_app,app-01'Continue whenDOWN after three failures, zero new application requests, then UP after two successes with slowstart.
Stop whenHealthy capacity is insufficient, traffic still reaches DOWN node, or returned node fails representative requests.
If this step fails
HAProxy marks the node DOWN but clients receive 503.
Likely causeThe remaining backend is also unhealthy, maxconn saturated, queue timed out, or no healthy server remains.
echo 'show stat' | socat stdio /run/haproxy/admin.sockcurl -v http://{{backendTwoAddress}}:{{backendPort}}{{healthPath}}
ResolutionRestore safe capacity, stop the test, and revise N+1 capacity plus queue thresholds before claiming availability.
Security notes
- Restrict backend service control and Runtime API groups; either can remove production capacity.
Alternatives
- Use Runtime API drain to test planned maintenance separately from hard failure detection.
Stop conditions
- Abort when the surviving backend cannot carry the measured workload safely.
command
Install a renewed certificate with validation and graceful reload
Build the next PEM in a staging path, validate key and SAN, test a temporary candidate referencing it, atomically replace the bundle, reload, and verify the certificate served externally.
Why this step matters
Certificate renewal is a recurring production change. Staging, validating, atomic replacement, graceful reload, and external SNI verification prevent a valid file on disk from differing from what clients receive.
What to understand
Validate key match and SAN as in the initial step, plus a minimum remaining lifetime that prevents installing a stale renewal artifact.
Keep the previous PEM in protected recovery storage until the new worker and external probes succeed.
HAProxy can update certificates through Runtime API, but in-memory updates must be persisted or the next reload reverts them.
System changes
- Replaces the active private-key certificate bundle and reloads workers to serve the renewed identity.
Syntax explained
openssl x509 -checkend 2592000- Rejects a certificate that expires within thirty days.
openssl s_client -servername- Tests the real TLS listener with SNI and extracts the certificate clients receive.
Values stay on this page and are never sent or saved.
sudo install -m 0640 -o root -g haproxy {{nextPem}} /etc/haproxy/certs/{{publicHostname}}.pem.next && openssl x509 -in /etc/haproxy/certs/{{publicHostname}}.pem.next -noout -checkend 2592000 -ext subjectAltName && sudo haproxy -c -V -f {{candidateConfig}} && sudo mv /etc/haproxy/certs/{{publicHostname}}.pem.next /etc/haproxy/certs/{{publicHostname}}.pem && sudo systemctl reload haproxy && echo | openssl s_client -connect {{publicHostname}}:443 -servername {{publicHostname}} 2>/dev/null | openssl x509 -noout -serial -dates -fingerprint -sha256Certificate will not expire Configuration file is valid serial=5A8F1C7D03 notBefore=Oct 10 00:00:00 2026 GMT notAfter=Jan 8 23:59:59 2027 GMT sha256 Fingerprint=8B:17:92:4C:6A:11:4B:2D:...
Checkpoint: External clients receive the intended new certificate
echo | openssl s_client -connect {{publicHostname}}:443 -servername {{publicHostname}} 2>/dev/null | openssl x509 -noout -serial -datesContinue whenSerial, SAN, chain, and dates match the approved renewed certificate with no probe errors.
Stop whenCandidate validation fails, key/SAN differs, or external listener still serves an unknown certificate.
If this step fails
Disk certificate is new but clients receive the old serial.
Likely causeReload failed, another load balancer or CDN terminates TLS, DNS points elsewhere, or SNI selects another bundle.
systemctl status haproxy --no-pagergetent ahostsv4 {{publicHostname}}echo 'show ssl cert' | socat stdio /run/haproxy/admin.sock
ResolutionIdentify the actual TLS terminator and SNI selection, then reload or update the authoritative endpoint without deleting the known-good bundle.
Security notes
- Restrict staging directories, renewal hooks, and private key logs; certificate automation executes with sensitive authority.
Alternatives
- Use the HAProxy Runtime API transactional certificate update when fully automated, then update persistent files and validate the next reload.
Stop conditions
- Do not remove the previous key material until external verification and rollback evidence are complete.
verification
Verify TLS, routing distribution, health telemetry, and reload readiness
Close with external HTTP/TLS probes, repeated node responses, Runtime API statistics, config validation, certificate expiry, service status, logs, and an owned alert for every failure surface.
Why this step matters
A complete load-balancer verification spans public DNS, redirect, TLS identity, frontend processing, backend selection, application readiness, runtime state, process lifecycle, logs, and certificate renewal headroom.
What to understand
Distribution is statistical and a tiny sample is not proof of an exact split. It does prove both node markers can be reached under ordinary conditions.
Alert on frontend availability, TLS expiry and handshake errors, backend UP count, check failures, queue time, response codes, saturation, reload failures, old workers, and Runtime API access.
Retain the validated configuration digest, certificate fingerprint, external probe, health-failure exercise, and renewal procedure with every release.
System changes
- No changes; captures the final client, TLS, backend, runtime, config, and process evidence.
Syntax explained
curl -I http://- Verifies the plaintext listener redirects without downloading an application body.
sort | uniq -c- Summarizes observed backend node markers across repeated requests.
show stat- Reports current health and traffic counters from the running worker.
Values stay on this page and are never sent or saved.
curl --fail --silent --show-error -I http://{{publicHostname}}/; curl --fail --silent --show-error https://{{publicHostname}}/lb-healthz; for i in $(seq 1 12); do curl --fail --silent https://{{publicHostname}}/node; echo; done | sort | uniq -c; echo 'show stat' | sudo socat stdio /run/haproxy/admin.sock | grep '^be_app'; sudo haproxy -c -V -f /etc/haproxy/haproxy.cfg; systemctl is-active haproxyHTTP/1.1 301 Moved Permanently
location: https://app.example.net/
ready
6 app-01
6 app-02
be_app,app-01,...,UP,...,L7OK,200,...
be_app,app-02,...,UP,...,L7OK,200,...
Configuration file is valid
activeCheckpoint: Every layer is healthy and observable
sudo haproxy -c -V -f /etc/haproxy/haproxy.cfg && systemctl is-active haproxyContinue whenValid config, correct redirect and certificate, both backends UP, no queues or errors, active service, and renewal alerts owned.
Stop whenAny client path, certificate, backend, runtime metric, or reload test lacks evidence.
If this step fails
Both servers are UP but distribution is heavily skewed.
Likely causeLeastconn correctly follows long-lived connection counts, one node is slow-starting, weights differ, or persistent connections reduce sample independence.
echo 'show stat' | socat stdio /run/haproxy/admin.sockecho 'show servers state' | socat stdio /run/haproxy/admin.sock
ResolutionReview active sessions, weights, throttle, and workload duration before changing the algorithm.
Security notes
- Publish aggregate metrics, not private backend addresses or unrestricted Runtime API data.
Alternatives
- Use synthetic probes from multiple networks to cover DNS, routing, IPv4/IPv6, and certificate-chain differences.
Stop conditions
- Do not close rollout until a failed backend and renewed certificate have both been exercised safely.
Finish line
Verification checklist
curl --fail --silent --show-error -I http://{{publicHostname}}/; openssl s_client -connect {{publicHostname}}:443 -servername {{publicHostname}} -verify_return_error </dev/nullHTTP redirects to HTTPS; the served certificate covers the hostname, chains to the approved trust store, and has sufficient validity remaining.for i in $(seq 1 12); do curl --fail --silent https://{{publicHostname}}/node; echo; done | sort | uniq -cBoth node identities are observed during ordinary operation and every response is successful.echo 'show stat' | sudo socat stdio /run/haproxy/admin.sock | grep '^be_app'Both servers are UP with L7OK checks, no unexpected queues, and no continuing connection or response errors.sudo haproxy -c -V -f /etc/haproxy/haproxy.cfg && systemctl is-active haproxy && pgrep -a haproxyThe installed configuration validates, the service is active, one current worker serves traffic, and no unowned old worker remains.Recovery guidance
Common problems and safe checks
HAProxy reports a backend DOWN although opening the TCP port succeeds.
Likely causeThe configured HTTP readiness path, Host header, expected status, timeout, or application dependency does not satisfy the layer-seven check.
curl --verbose http://{{backendOneAddress}}:{{backendPort}}{{healthPath}}echo 'show stat' | socat stdio /run/haproxy/admin.sock
ResolutionReproduce the exact request from the proxy namespace, fix the application readiness contract or reviewed check, and retain rise/fall hysteresis.
Clients receive intermittent 503 responses during a node failure.
Likely causeThe fall threshold is too slow for the application failure, retries target exhausted nodes, or the remaining server cannot absorb the offered load.
journalctl -u haproxy --since '-10 minutes'echo 'show stat' | socat stdio /run/haproxy/admin.sock
ResolutionMeasure check detection and capacity, correct the failure contract, and tune thresholds from evidence rather than hiding overload with unlimited retries.
A reload succeeds but the old worker remains for a long time.
Likely causeLong-lived connections, WebSockets, tunnel traffic, an unbounded timeout, or a stuck session prevents graceful draining.
pgrep -a haproxyecho 'show info' | socat stdio /run/haproxy/admin.sock
ResolutionIdentify the session class, apply protocol-appropriate close or hard-stop policy in a planned window, and never kill all workers blindly.
The renewed certificate is installed but clients still see the old one.
Likely causeThe wrong PEM path or SNI certificate is active, reload failed, a preceding proxy terminates TLS, or old workers still serve connections.
openssl s_client -connect {{publicHostname}}:443 -servername {{publicHostname}} </dev/nullsystemctl status haproxy --no-pager
ResolutionTrace the actual termination point, compare fingerprints, validate the loaded path, and repeat the controlled reload after correcting the candidate.
Reference
Frequently asked questions
Does two backends make the service highly available?
It removes one application-node failure but leaves HAProxy, its host, network path, DNS, certificate process, and shared dependencies as potential single points of failure.
Why check HTTP instead of only TCP?
A TCP listener proves that a socket accepted a connection. It does not prove the application can serve the required request or reach its critical dependencies.
Will reload interrupt WebSockets?
Master-worker reload is designed to drain old workers, but WebSockets and other long-lived sessions can keep an old worker alive. Define and test bounded drain behavior for the actual protocol.
Should the backend also use TLS?
Yes when the backend network is not an adequate trust boundary or policy requires encryption and identity end-to-end. Configure certificate verification, not encryption alone.
Recovery
Rollback
Rollback restores the last configuration and certificate pair that passed both offline validation and an external probe. It cannot undo application writes already accepted by a backend or restore sessions that the application itself does not externalize.
- Freeze additional changes, preserve the failing configuration, journal, Runtime API snapshot, public probe, certificate fingerprint, and process list so configuration, application, and network failures remain distinguishable.
- Restore the versioned last-known-good HAProxy configuration and PEM bundle with their original owners and modes. Run `haproxy -c -V` against the restored paths before touching the running process.
- Reload through the package-managed systemd unit, not by killing workers. Confirm existing connections drain, the new worker accepts connections, both backends return UP, and the public TLS probe succeeds.
- If the load balancer cannot recover, return DNS or the upstream virtual IP to the documented previous service only within the pre-approved TTL and health-check plan. Do not expose private application ports directly as an improvised bypass.
Evidence