Configure ISC Kea DHCP with reservations and high availability
Deploy two supported Kea 3.0 LTS DHCPv4 peers in hot-standby mode with identical subnet policy, durable independent lease stores, fixed-address reservations, peer-only native HA listeners, automatic takeover guarded against split brain, and a controlled lease resynchronization path.
Provide one authoritative DHCP service through a planned node failure without duplicate allocation, reservation drift, exposed control endpoints, lost lease history, or an unsynchronized former primary returning to serve clients.
- Kea DHCP 3.0.3 LTS, 3.0.x LTS
- Ubuntu Server 24.04 LTS
- Two independently recoverable Linux peers Use two nodes on distinct failure domains with stable client-path and management addresses, synchronized time, console access, and enough local durable storage for lease growth and logs.
timedatectl status && ip -br address - A complete DHCP authority and relay inventory Document every subnet, pool, option, reservation, relay target, relay gateway address, snooping policy, legacy server, and maintenance owner before introducing a second authority.
- A tested fencing decision Define who or what disables one peer when the HA network is partitioned but both nodes can still reach clients. Automatic partner-down alone does not prove single-writer behavior.
- A migration and rollback window Export current leases and reservations, reduce lease duration in advance only when operationally approved, retain the previous service configuration, and provide a disposable client on each relay path.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A pair of Kea 3.0.3 LTS DHCPv4 servers with identical subnet IDs, pools, DHCP options, and host reservations. The primary normally answers clients while the standby receives lease updates into its own durable memfile database.
- A dedicated peer control path for heartbeats, lease updates, synchronization, and state commands. Its listener follows the client-facing failure domain, is restricted to HA commands, and is permitted only between the two peer addresses.
- A guarded failover lifecycle: normal hot standby, communications interrupted, client-evidence-based partner-down, one active allocator, fenced recovery, lease synchronization, and return to the normal relationship.
- An operating evidence set covering relay symmetry, reservation selection, dynamic DORA, lease parity, state transitions, listener exposure, local storage, alert ownership, and rollback to one authority.
- A new or renewing client on every tested relay path receives one valid response while either peer is deliberately unavailable, and no packet trace shows competing offers from both nodes.
- The reservation receives the approved fixed address and hostname on either peer; dynamic clients remain within the defined pool; subnet IDs, identifiers, and options are consistent.
- Each lease allocated during normal operation or takeover exists on the peer expected to recover it. A returning node remains non-authoritative until HA synchronization completes.
- Operators can identify every HA state, explain why automatic takeover is or is not permitted, fence one allocator, restore the pair serially, and prove the control listener is not reachable from client networks.
Architecture
How the parts fit together
Kea's HA hook is an application-level lease replication and state-coordination system, not a shared virtual IP and not ISC DHCP failover. Both DHCP daemons receive client or relay traffic. In hot-standby mode the primary normally answers, while the standby suppresses service and accepts lease updates over a dedicated HTTP listener. Heartbeats establish peer communication. On peer loss, the standby first enters communications-interrupted and observes unacknowledged client attempts; only then can it enter partner-down and serve all clients. Each node retains an independent memfile lease database. The returning peer synchronizes from the active authority before resuming its normal role.
- A client broadcasts or a relay forwards DHCPDISCOVER to both nodes. Under normal hot standby, the primary processes it and the standby does not answer.
- The primary commits the lease to its local memfile database and sends a lease4-update command to the standby's dedicated HA listener.
- Peers exchange heartbeats. If communication fails, the standby enters communications-interrupted but does not immediately assume that the primary is dead.
- Client retry evidence and configured timers allow the standby to enter partner-down. Fencing guarantees the old primary cannot continue allocating.
- When the old peer returns, it enters syncing, receives the authoritative lease set from the active node, and reaches normal hot-standby only after completion.
Assumptions
- Both nodes run the same supported Kea 3.0.3 or reviewed compatible 3.0.x LTS build. The procedure intentionally never runs mixed 2.6/3.0 or 3.0/3.2 peers across documented HA compatibility boundaries.
- Every client segment or relay is configured to deliver requests to both nodes. The chosen HA path fails with or meaningfully reflects the client path instead of hiding access-network failure behind a healthy management network.
- The example uses hot standby and local memfile stores. Load-balancing mode, hub-and-spoke relationships, DHCPv6, shared SQL lease databases, host-database backends, and multi-thread performance tuning require additional design.
- Network time is synchronized, both nodes use durable storage, packet capture is permitted for a controlled test, and a disposable client identity exists on each representative relay path.
- An external human or automated fence can stop one node, remove its relay reachability, or block its DHCP output when peer state is ambiguous. That control is tested independently of Kea.
- Configuration is generated from one source with only this-server-name differing. Reservations, pools, subnet IDs, option data, hooks, timers, and peer definitions cannot drift.
Key concepts
- Hot standby
- An active-passive Kea HA mode in which the primary normally handles all DHCP traffic and the standby mirrors leases and takes over after failure detection.
- Communications interrupted
- An ambiguous state entered after heartbeat or command failures. The peer relationship is unavailable, but the other server may still reach clients, so immediate dual service would risk split brain.
- Partner down
- A state in which one peer assumes the partner cannot serve and enables all client scopes. Entering it automatically requires configured timing and client-observation logic or an explicit controlled decision.
- Lease synchronization
- Transfer of lease records from the authoritative peer to a recovering peer. It is separate from continuous per-lease updates and must complete before stale state can serve safely.
- Memfile lease database
- A local persistent lease store represented by an append-oriented CSV file and periodically compacted by lease-file cleanup. Each HA peer owns its own file.
- Reservation
- A host policy mapping a stable client identifier to address, hostname, options, classes, or boot information. It is configuration, not a replicated dynamic lease and not network authentication.
- Fencing
- A control that makes one peer incapable of serving when ownership is ambiguous, for example service stop, relay removal, switch ACL, VM power action, or network isolation.
Before you copy
Values used in this guide
{{clientInterface}}Interface receiving direct or relayed DHCP client traffic on the local peer.
Example: enp2s0{{relayAddress}}Relay gateway address used to select the client subnet and validate routing.
Example: 10.20.0.1{{keaUser}}Unprivileged operating-system account running Kea.
Example: _kea{{keaGroup}}Group permitted to read Kea configuration and write owned state.
Example: _kea{{keaService}}Exact systemd service unit installed by the selected package source.
Example: isc-kea-dhcp4-server.service{{leaseDirectory}}Durable local directory containing the memfile database.
Example: /var/lib/kea{{leaseFile}}Absolute local lease CSV path; each peer uses its own local copy.
Example: /var/lib/kea/kea-leases4.csv{{haHookPath}}Absolute packaged path to libdhcp_ha.so.
Example: /usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so{{leaseCommandsHookPath}}Absolute packaged path to libdhcp_lease_cmds.so.
Example: /usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so{{candidateConfig}}Staged candidate configuration validated before install.
Example: /etc/kea/candidates/kea-dhcp4.conf{{primaryConfig}}Rendered primary candidate used for policy comparison.
Example: /var/tmp/kea-a.conf{{standbyConfig}}Rendered standby candidate used for policy comparison.
Example: /var/tmp/kea-b.conf{{thisServerName}}Local peer name; the only intended HA relationship difference between rendered peer files.
Example: kea-a{{primaryName}}Peer name carrying the primary role.
Example: kea-a{{standbyName}}Peer name carrying the standby role.
Example: kea-b{{primaryAddress}}Primary's HA/client-path address allowed by the standby firewall.
Example: 192.0.2.21{{standbyAddress}}Standby's HA/client-path address allowed by the primary firewall.
Example: 192.0.2.22{{peerAddress}}Remote peer address used from the node where a verification command runs.
Example: 192.0.2.22{{primaryHaPort}}Dedicated primary HA HTTP listener port.
Example: 8000{{standbyHaPort}}Dedicated standby HA HTTP listener port.
Example: 8000{{peerHaPort}}Remote peer HA listener port for local verification.
Example: 8000{{keaFirewallFile}}Reviewed nftables candidate protecting the HA listener.
Example: /etc/nftables.d/40-kea-ha.nft{{clientSubnet}}CIDR served by the example subnet declaration.
Example: 10.20.0.0/24{{dynamicPool}}Inclusive allocatable range inside the client subnet.
Example: 10.20.0.100 - 10.20.0.199{{defaultGateway}}Router option delivered to clients.
Example: 10.20.0.1{{dnsServers}}Comma-separated DNS resolver addresses in Kea option syntax.
Example: 10.20.0.53, 10.20.0.54{{searchDomain}}DNS search domain delivered to clients.
Example: corp.example.net{{reservedMac}}Stable tested MAC identifier for the example reservation.
Example: 52:54:00:aa:10:05{{reservedAddress}}Unique reserved address, preferably outside the dynamic range.
Example: 10.20.0.20{{reservedHostname}}Hostname assigned to the reserved client.
Example: printer-01Security and production boundaries
- DHCP controls addressing, gateways, DNS, and boot options. A compromised server can redirect an entire client segment even if lease replication remains healthy.
- The HA listener exposes commands that affect lease and service state. Restrict it by source and destination, do not publish it, and protect traffic crossing any untrusted network.
- Reservations and lease databases contain device identifiers, addresses, hostnames, and timing. Encrypt backups, restrict queries, and retain records according to privacy policy.
- Do not treat a MAC reservation as access control. Attackers on the segment can spoof identifiers; enforce switch, NAC, firewall, and service identity separately.
- A failover mechanism without fencing can become an availability-triggered duplicate-address generator. Split-brain prevention is a production security and integrity requirement.
Stop before continuing if
- Stop if another DHCP authority remains reachable, relay traffic is asymmetric, or a planned subnet cannot be selected deterministically.
- Stop if package versions, hook binaries, peer configuration, subnet IDs, pools, options, or reservations differ beyond the local peer name.
- Stop if lease storage is unwritable, ephemeral, shared accidentally, or cannot be backed up and synchronized.
- Stop if the HA listener is reachable from client or public networks or cleartext commands cross an untrusted path.
- Stop failover testing if both peers answer, no fence exists, or client retry behavior cannot satisfy the configured automatic takeover guard.
- Stop recovery if the returning peer can serve before lease synchronization completes.
verification
Inventory relays, interfaces, routes, and existing DHCP authority
Map every client subnet to its relay or directly connected interface, confirm both Kea nodes receive the same broadcasts or relayed packets, and prove that no legacy DHCP server can answer the planned ranges.
Why this step matters
High availability cannot compensate for an asymmetric relay path or an undiscovered authoritative server. Capturing packets on both nodes proves the real client path before the HA state machine is allowed to allocate addresses.
What to understand
Document the relay gateway address used for subnet selection, the DHCP server targets configured on every relay, and any switch security feature such as DHCP snooping that permits only approved server ports.
UDP 67 must be reachable from clients or relays to both peers. The HA listener is a different HTTP service and must be reachable only peer-to-peer, never from client VLANs.
A legacy ISC DHCP, router DHCP service, lab appliance, or forgotten container can issue conflicting leases even when Kea itself is perfectly synchronized.
System changes
- No changes; creates the authoritative network and current-server inventory for the rollout.
Syntax explained
ss -lunp- Lists listening UDP sockets and owning processes so another DHCP authority is visible.
tcpdump '(udp port 67 or udp port 68)'- Captures the DHCP exchange without decoding unrelated client traffic.
Values stay on this page and are never sent or saved.
ip -br address; ip route; sudo ss -lunp | grep -E ':(67|8000|8001)\b' || true; sudo tcpdump -ni {{clientInterface}} -c 8 '(udp port 67 or udp port 68)'enp1s0 UP 192.0.2.21/24 enp2s0 UP 10.20.0.21/24 default via 192.0.2.1 dev enp1s0 10.20.0.0/24 dev enp2s0 proto kernel scope link src 10.20.0.21 12:04:31 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:aa:10:05, length 300 12:04:31 IP 10.20.0.1.67 > 10.20.0.21.67: BOOTP/DHCP, Request from 52:54:00:aa:10:05, length 300
Checkpoint: Both peers see the intended client or relay traffic
sudo tcpdump -ni {{clientInterface}} -c 8 '(udp port 67 or udp port 68)'Continue whenDiscover or relay-forward packets arrive on both nodes with the expected relay address and no unexpected offers.
Stop whenOnly one node receives traffic, relay addresses are ambiguous, or another server sends offers for the planned pool.
If this step fails
The secondary never sees requests although peer HTTP connectivity works.
Likely causeRelays target only the primary, multicast/broadcast is filtered, or capture is running on the management interface.
ip route get {{relayAddress}}tcpdump -ni any -c 20 'udp port 67 or 68'
ResolutionCorrect relay helper targets and network ACLs so both nodes receive the same client path before enabling HA.
Security notes
- DHCP captures contain client MAC addresses, hostnames, and requested options; retain them as restricted operational evidence.
Alternatives
- Use a dedicated relay test VLAN and disposable clients when production packet capture is prohibited.
Stop conditions
- Do not start Kea while another unowned DHCP authority can answer the target network.
command
Install a consistent Kea 3.0 LTS package set on both peers
Install the same supported Kea 3.0 LTS release, DHCPv4 daemon, HA and lease-command hooks, and the isolated perfdhcp test client on both nodes, then record binary and library paths before writing configuration.
Why this step matters
HA peers exchange commands and lease objects whose behavior can change across major branches. An identical package set and verified hook location eliminate version skew as an invisible failover fault.
What to understand
This tutorial targets the supported Kea 3.0.3 LTS branch and its matching ARM. Kea 2.6 and 3.0 are not wire-compatible HA peers, and Kea 3.2 is a separate stable branch, so every cross-branch move requires the upstream serialized upgrade procedure.
Distribution package names and service names differ. Record the actual unit, binary, and hook locations instead of copying an architecture-specific path blindly.
Install optional SQL drivers only when an external lease backend was deliberately selected and its own highly available failure domain has been designed.
System changes
- Installs Kea binaries, service units, configuration directories, hook libraries, and package-managed users on both peers.
Syntax explained
kea-dhcp4 -V- Prints the exact DHCPv4 daemon version used to check peer compatibility.
dpkg-query -W- Records every installed ISC Kea package and version for reproducibility.
sudo apt-get update && sudo apt-get install --yes isc-kea-dhcp4 isc-kea-hooks isc-kea-perfdhcp && kea-dhcp4 -V && dpkg-query -W 'isc-kea-*' | sort && find /usr/lib -type f \( -name 'libdhcp_ha.so' -o -name 'libdhcp_lease_cmds.so' \) -print3.0.3 isc-kea-common 3.0.3-isc202603191045 isc-kea-dhcp4 3.0.3-isc202603191045 isc-kea-hooks 3.0.3-isc202603191045 isc-kea-perfdhcp 3.0.3-isc202603191045 /usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so /usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so
Checkpoint: Both nodes run the same supported version and expose both hooks
kea-dhcp4 -V; test -r {{haHookPath}} -a -r {{leaseCommandsHookPath}}Continue whenVersion 3.0.3 on each peer, the matching perfdhcp utility, and readable HA plus lease-command libraries at recorded paths.
Stop whenVersions differ, a hook is missing, or either package source is outside the approved support policy.
If this step fails
Kea reports it cannot open libdhcp_ha.so.
Likely causeThe hook package is absent or the copied library path does not match the architecture and package layout.
find /usr/lib -name libdhcp_ha.so -printdpkg -L isc-kea-hooks | grep libdhcp_ha
ResolutionInstall the matching hook package and use its actual absolute path; do not copy a library from another release.
Security notes
- Use only the signed ISC or approved distribution repository and monitor Kea security announcements.
Alternatives
- Use Kea 3.2 only as a separately tested upgrade with both peers moved serially according to its current compatibility and release notes.
Stop conditions
- Do not form an HA relationship across the documented 2.6-to-3.0 or 3.0-to-3.2 branch boundaries.
config
Choose and protect the lease database on each node
Use one local memfile lease database per peer for this HA-hook design, place it on durable local storage, and verify that lease-file cleanup runs. Do not point both peers at the same file.
Why this step matters
The HA hook synchronizes lease updates between independent peer stores. Durable local memfile databases avoid making a shared SQL endpoint a hidden single failure domain and retain each peer's last known lease view.
What to understand
The memfile backend appends lease changes and Kea Lease File Cleanup periodically compacts redundant records. Monitor both the CSV file and cleanup process because disk exhaustion can stop lease persistence.
A shared SQL lease database is a different Kea availability strategy with database latency, UTC, schema, and failure semantics. Do not combine it casually with peer replication.
Back up the lease file and configuration together, but treat synchronized peer state and backup as different controls. Corruption or administrative deletion can replicate.
System changes
- Creates a restricted durable lease directory and defines persistent memfile storage plus cleanup cadence.
Syntax explained
"type": "memfile"- Stores each peer's lease state locally in an append-oriented CSV database.
"persist": true- Keeps allocations across daemon restart instead of operating memory-only.
"lfc-interval": 3600- Requests periodic lease-file cleanup at a one-hour interval.
/etc/kea/kea-dhcp4.confValues stay on this page and are never sent or saved.
"lease-database": {
"type": "memfile",
"name": "{{leaseFile}}",
"persist": true,
"lfc-interval": 3600
},Values stay on this page and are never sent or saved.
sudo install -d -o {{keaUser}} -g {{keaGroup}} -m 0750 {{leaseDirectory}} && sudo -u {{keaUser}} test -w {{leaseDirectory}} && systemctl list-timers --all | grep -i kea || truedrwxr-x--- 2 _kea _kea 4096 Jul 28 12:30 /var/lib/kea kea-lfc.timer loaded active waiting Kea lease file cleanup write_test=passed
Checkpoint: Each peer owns a writable independent lease path
sudo -u {{keaUser}} test -w {{leaseDirectory}} && stat -c '%a %U:%G %n' {{leaseDirectory}}Continue whenThe Kea account can write, other users cannot browse the directory, and peer paths are local rather than shared.
Stop whenBoth peers resolve to one shared file, storage is ephemeral, or permissions require running Kea as root.
If this step fails
Kea starts but leases disappear after restart.
Likely causePersistence is disabled, the configured path differs from the service path, or the lease file cannot be written.
journalctl -u {{keaService}} -n 100 --no-pagerls -l {{leaseDirectory}}
ResolutionCorrect the owned durable path, restore the last verified lease file if necessary, and resynchronize only through the HA procedure.
Security notes
- Lease data maps client identifiers to addresses and hostnames; restrict backups and files to DHCP operators.
Alternatives
- Use PostgreSQL or MySQL only when that database is independently highly available, monitored, schema-managed, and supported by the installed Kea build.
Stop conditions
- Do not start allocation when lease storage is read-only, ephemeral, corrupt, or shared unintentionally.
config
Define the subnet, pool, options, and non-conflicting reservations
Create identical address policy on both peers, keep reservations outside the dynamic pool where practical, and use one stable identifier type consistently for managed hosts.
Why this step matters
Peer replication synchronizes leases, not an inconsistent configuration. Identical subnet IDs, pools, options, and reservation policy prevent peers from making different decisions after takeover.
What to understand
A reservation outside the dynamic pool avoids a live dynamic lease colliding with a newly introduced fixed address. If in-pool reservations are required, configure reservation conflict behavior deliberately.
Relay-selected subnets need stable subnet IDs and correct relay addresses. Direct-interface selection in the example is unsuitable when the server does not have an interface on every client subnet.
Use a reservation identifier clients actually present consistently. MAC addresses may change with privacy features, virtualization, docking, or hardware replacement.
System changes
- Defines the allocatable range, default gateway, DNS, search domain, and one fixed client reservation on both peers.
Syntax explained
"id": 20- Provides a stable unique subnet identifier that must match on both HA peers.
"pool"- Limits addresses Kea may allocate dynamically within the subnet.
"reservations"- Maps a reviewed client identifier to fixed address, hostname, and optional client-specific options.
/etc/kea/kea-dhcp4.confValues stay on this page and are never sent or saved.
"subnet4": [{
"id": 20,
"subnet": "{{clientSubnet}}",
"interface": "{{clientInterface}}",
"pools": [{ "pool": "{{dynamicPool}}" }],
"option-data": [
{ "name": "routers", "data": "{{defaultGateway}}" },
{ "name": "domain-name-servers", "data": "{{dnsServers}}" },
{ "name": "domain-name", "data": "{{searchDomain}}" }
],
"reservations": [{
"hw-address": "{{reservedMac}}",
"ip-address": "{{reservedAddress}}",
"hostname": "{{reservedHostname}}"
}]
}],Values stay on this page and are never sent or saved.
sudo kea-dhcp4 -t {{candidateConfig}} && jq '.Dhcp4.subnet4[0] | {subnet,pools,reservations}' {{candidateConfig}}{
"subnet": "10.20.0.0/24",
"pools": [{"pool":"10.20.0.100 - 10.20.0.199"}],
"reservations": [
{"hw-address":"52:54:00:aa:10:05","ip-address":"10.20.0.20","hostname":"printer-01"}
]
}
Configuration check successfulCheckpoint: Both peers validate byte-equivalent address policy
sha256sum {{primaryConfig}} {{standbyConfig}}; kea-dhcp4 -t {{candidateConfig}}Continue whenOnly the documented local peer-name field differs; subnets, IDs, pools, options, and reservations match.
Stop whenA reservation overlaps an active dynamic lease, peers have different subnet IDs, or relay selection is untested.
If this step fails
A reserved client receives an address from the dynamic pool.
Likely causeThe client presents a different identifier, the reservation lives in another subnet scope, or early reservation lookup is not configured as expected.
journalctl -u {{keaService}} | grep -i '{{reservedMac}}'jq '.Dhcp4.subnet4[].reservations' {{candidateConfig}}
ResolutionCapture the client's actual identifier, correct reservation scope on both peers, and resolve any existing conflicting lease before retest.
Security notes
- Reservations are inventory and policy, not strong network authentication; enforce access with switch, NAC, firewall, and service controls.
Alternatives
- Use the Kea host database for large managed inventories after its availability, schema, audit, and rollback path are designed.
Stop conditions
- Do not publish a reservation until existing lease ownership and address uniqueness are proven.
config
Create a peer-only HA listener and firewall boundary
Bind each native dedicated HA listener to its client-path address, restrict commands, and allow its TCP port only from the partner. Keep any broader HTTP/TLS management endpoint disabled or on a separately authenticated management boundary.
Why this step matters
HA heartbeats, lease updates, and synchronization are control-plane operations. A dedicated listener reduces contention while a peer-only firewall prevents clients or untrusted management hosts from issuing control commands.
What to understand
ISC recommends the dedicated listener for HA performance and reliability. Place it on the network whose loss should influence failover rather than an out-of-band path that remains healthy while clients lose service.
The example uses clear HTTP within a tightly controlled network. Where traffic crosses an untrusted segment, use supported TLS/authentication or an isolated encrypted transport and verify peer identity.
Firewall both acceptance and explicit rejection, and test from peer plus unauthorized host. A local curl alone does not prove network policy.
System changes
- Adds peer-restricted firewall rules and configures dedicated multithreaded HA HTTP processing.
Syntax explained
"http-dedicated-listener": true- Separates HA command traffic from the ordinary Control Agent listener.
"restrict-commands": true- Limits the dedicated endpoint to HA-related commands supported by the branch.
nft -c / nft -f- Checks the candidate ruleset before applying it to the live firewall.
/etc/kea/kea-dhcp4.conf"ha-mt": {
"enable-multi-threading": true,
"http-dedicated-listener": true,
"http-listener-threads": 2,
"http-client-threads": 2
},
"restrict-commands": true,Values stay on this page and are never sent or saved.
sudo nft -c -f {{keaFirewallFile}} && sudo nft -f {{keaFirewallFile}} && sudo nft list chain inet filter input | sed -n '1,120p' && curl --fail --silent --show-error http://{{peerAddress}}:{{peerHaPort}}/ -H 'Content-Type: application/json' -d '{"command":"status-get","service":["dhcp4"]}'table inet filter {
chain input {
ip saddr 192.0.2.22 tcp dport 8000 accept comment "Kea HA peer"
tcp dport 8000 drop comment "Protect Kea HA listener"
}
}
[{ "result": 0, "text": "Everything is fine", "arguments": { "pid": 2141, "uptime": 318 } }]Checkpoint: Peer reaches the HA API and client networks cannot
curl --fail http://{{peerAddress}}:{{peerHaPort}}/ -d '{"command":"status-get","service":["dhcp4"]}'Continue whenPeer command succeeds, unauthorized-source test is blocked, and packet path follows the client-facing failure domain.
Stop whenThe listener is public, cleartext crosses an untrusted network, or management and client sources can issue commands.
If this step fails
Peers allocate leases but repeatedly log heartbeat failures.
Likely causeTCP listener, firewall, route, or URL points at the wrong interface even though UDP DHCP works.
ss -lntp | grep ':{{peerHaPort}}'nft list rulesetcurl -v http://{{peerAddress}}:{{peerHaPort}}/
ResolutionCorrect the listener address, peer URL, route, and peer-only firewall as one reviewed control plane.
Security notes
- The HA API can change DHCP availability and leases; never expose it to client VLANs or the public internet.
Alternatives
- Use a dedicated isolated service VLAN that shares the customer-path failure characteristics and is monitored end to end.
Stop conditions
- Do not enable HA when the listener lacks an authenticated or strictly isolated peer boundary.
config
Configure hot-standby peers, timers, and automatic split-brain guard
Load lease-command before HA hooks, define both server identities identically, select hot-standby mode, and retain the client-observation gate before automatic partner-down.
Why this step matters
Hot standby is operationally simpler than load balancing: the primary serves normally and the standby mirrors leases. The delayed partner-down transition observes unanswered clients so a severed peer link does not immediately create two allocators.
What to understand
Hook order matters because HA uses lease-command operations to send updates and perform synchronization. Use the real packaged paths.
heartbeat-delay should be materially shorter than max-response-delay so several heartbeat opportunities occur. The values are milliseconds.
Keep max-unacked-clients nonzero unless the client population has been proven unable to increment SECS and an external fencing control prevents split brain. Setting zero removes an important ambiguity guard.
Only this-server-name differs between peer files. Roles, URLs, timers, pools, reservations, and hook settings remain consistent.
System changes
- Loads two privileged hooks, creates the peer relationship, enables lease synchronization, and defines automatic failover timing.
Syntax explained
"mode": "hot-standby"- Makes the primary the normal responder and the standby the synchronized takeover peer.
"max-unacked-clients": 10- Requires evidence from unserved client attempts before automatic partner-down.
"sync-leases": true- Requests full lease synchronization when a peer needs to catch up.
/etc/kea/kea-dhcp4.confValues stay on this page and are never sent or saved.
"hooks-libraries": [
{ "library": "{{leaseCommandsHookPath}}" },
{
"library": "{{haHookPath}}",
"parameters": {
"high-availability": [{
"this-server-name": "{{thisServerName}}",
"mode": "hot-standby",
"heartbeat-delay": 10000,
"max-response-delay": 60000,
"max-ack-delay": 10000,
"max-unacked-clients": 10,
"sync-leases": true,
"peers": [
{ "name": "{{primaryName}}", "url": "http://{{primaryAddress}}:{{primaryHaPort}}/", "role": "primary" },
{ "name": "{{standbyName}}", "url": "http://{{standbyAddress}}:{{standbyHaPort}}/", "role": "standby" }
]
}]
}
}
],Values stay on this page and are never sent or saved.
sudo kea-dhcp4 -t {{candidateConfig}} && jq '.Dhcp4["hooks-libraries"]' {{candidateConfig}}[
{"library":"/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so"},
{"library":"/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so","parameters":{"high-availability":[{"this-server-name":"kea-a","mode":"hot-standby","heartbeat-delay":10000,"max-response-delay":60000,"max-unacked-clients":10}]}}
]
Configuration check successfulCheckpoint: Both candidate files parse and define one symmetric relationship
kea-dhcp4 -t {{primaryConfig}} && kea-dhcp4 -t {{standbyConfig}}Continue whenBoth parse; peer names, roles, URLs, hook order, and timers match, with only this-server-name local.
Stop whenTimers differ, both nodes claim the same local name, hook order is reversed, or automatic failover has no split-brain gate.
If this step fails
Both peers stay in WAITING or READY and DHCP remains disabled.
Likely causePeer identity, URL, role, hook loading, or synchronized configuration differs.
journalctl -u {{keaService}} | grep -E 'HA_STATE|HA_'curl http://{{peerAddress}}:{{peerHaPort}}/
ResolutionCompare both HA blocks, correct identity and reachability, then restart one peer at a time and observe state transitions.
Security notes
- Automatic partner-down trades availability against duplicate-address risk; keep a documented fencing and manual override procedure.
Alternatives
- Use load-balancing mode only when both peers serving 50/50 is required and automatically created HA client classes plus pool behavior have been tested.
Stop conditions
- Do not disable the unacked-client guard without an independent proven fence.
command
Validate, stage, and start peers one at a time
Back up current files, validate candidate JSON, install it atomically, start the primary, then the standby, and inspect explicit HA state transitions before sending production traffic.
Why this step matters
Serial rollout preserves one known service path and makes each state transition observable. Atomic install plus daemon validation prevents malformed JSON from taking both DHCP authorities offline.
What to understand
Run the daemon's `-t` validation against the exact candidate installed, not a separate editor buffer.
Start the intended primary first, but do not declare it ready until the HA hook enables local DHCP and state is expected. Then start the standby.
Keep configuration backups mode 0600 because they contain topology, reservations, and control-plane URLs.
System changes
- Backs up and replaces the live Kea configuration, restarts one daemon, and begins HA state synchronization.
Syntax explained
kea-dhcp4 -t- Parses and semantically validates DHCPv4 configuration without binding service sockets.
install -m 0640- Replaces the file with explicit ownership and mode in one controlled operation.
journalctl --since- Limits state review to the rollout window while preserving timestamps.
Values stay on this page and are never sent or saved.
sudo install -m 0600 -o root -g root /etc/kea/kea-dhcp4.conf /etc/kea/kea-dhcp4.conf.before-ha && sudo kea-dhcp4 -t {{candidateConfig}} && sudo install -m 0640 -o root -g {{keaGroup}} {{candidateConfig}} /etc/kea/kea-dhcp4.conf && sudo systemctl restart {{keaService}} && sudo systemctl is-active {{keaService}} && sudo journalctl -u {{keaService}} --since '-2 minutes' --no-pager | grep -E 'HA_STATE_TRANSITION|HA_LOCAL_DHCP|HA_LEASE'Configuration check successful active HA_STATE_TRANSITION kea-a: server transitions from WAITING to SYNCING state, partner state is READY HA_STATE_TRANSITION kea-a: server transitions from SYNCING to HOT-STANDBY state, partner state is READY HA_LEASE_UPDATES_ENABLED kea-a: lease updates will be sent to the partner HA_LOCAL_DHCP_ENABLE local DHCP service is enabled
Checkpoint: Primary and standby reach normal hot-standby state
journalctl -u {{keaService}} --since '-5 minutes' | grep HA_STATE_TRANSITIONContinue whenPrimary serves, standby is synchronized, lease updates are enabled, and neither reports communication interruption.
Stop whenValidation warns, service fails, DHCP remains disabled, or peer states disagree.
If this step fails
The daemon validates but exits after restart.
Likely causeRuntime interface, port, lease path, hook dependency, or permissions differ from parse-time checks.
systemctl status {{keaService}} --no-pagerjournalctl -u {{keaService}} -n 150 --no-pagerss -lunp | grep ':67 '
ResolutionRestore the saved configuration on that peer, restart it, and correct the runtime resource conflict before touching the partner.
Security notes
- Do not paste complete reservation or control configuration into public incident channels.
Alternatives
- Use configuration management with preflight validation, serialized deployment, and automatic rollback to the same known-good file.
Stop conditions
- Never restart both peers simultaneously during an initial rollout.
verification
Prove reservation behavior and peer lease synchronization
Request one dynamic lease and the fixed reservation, query leases on both peers through the protected control path, and confirm address, identifier, subnet, and lifetime match.
Why this step matters
Healthy peer heartbeats do not prove lease replication or reservation selection. A real DORA exchange and matching lease query demonstrate client service plus state transfer.
What to understand
Use a disposable client identity and release or expire its lease according to test policy afterward. Never spoof a production MAC.
Compare client identifier, address, subnet ID, valid lifetime, cltt, and state on both nodes, allowing only expected timestamp propagation differences.
Test the reservation separately with its actual identifier type and verify that the dynamic pool did not previously own that address.
System changes
- Allocates disposable test leases on the primary and replicates them to the standby lease database.
Syntax explained
perfdhcp -4 -r 1- Generates DHCPv4 exchanges at a low controlled rate rather than stress testing production.
lease4-get-all- Reads lease records through the protected lease-command hook for cross-peer comparison.
Values stay on this page and are never sent or saved.
sudo perfdhcp -4 -r 1 -R 2 -p 3 -l {{clientInterface}} -Y 1 -y 2; curl --silent http://{{primaryAddress}}:{{primaryHaPort}}/ -H 'Content-Type: application/json' -d '{"command":"lease4-get-all","service":["dhcp4"]}' | jq; curl --silent http://{{standbyAddress}}:{{standbyHaPort}}/ -H 'Content-Type: application/json' -d '{"command":"lease4-get-all","service":["dhcp4"]}' | jqRate: 1.000 exchanges/second DISCOVER-OFFER: 3 sent, 3 received REQUEST-ACK: 3 sent, 3 received primary lease: 10.20.0.100 hw-address=52:54:00:00:00:01 subnet-id=20 state=0 standby lease: 10.20.0.100 hw-address=52:54:00:00:00:01 subnet-id=20 state=0 reservation: printer-01 -> 10.20.0.20
Checkpoint: Client service and both lease databases agree
curl --silent http://{{standbyAddress}}:{{standbyHaPort}}/ -d '{"command":"lease4-get-all","service":["dhcp4"]}'Continue whenStandby contains the primary's disposable dynamic lease and reservation behavior matches policy.
Stop whenA lease exists on only one peer, identifiers differ, or duplicate offers appear.
If this step fails
The client receives an ACK but the standby has no lease.
Likely causeLease updates are disabled, lease-command hook is missing, HA listener failed, or peer is in a state that suppresses synchronization.
journalctl -u {{keaService}} | grep -E 'HA_LEASE|COMMAND_'curl http://{{standbyAddress}}:{{standbyHaPort}}/
ResolutionKeep automatic failover disabled, repair hook and peer communication, then perform a controlled lease sync before retesting.
Security notes
- Control queries expose the full active client inventory; restrict output and delete disposable test records through supported commands.
Alternatives
- Use a dedicated isolated client and packet capture when perfdhcp is not approved on the production segment.
Stop conditions
- Do not test failover until peer lease contents agree.
decision
Test primary loss, automatic takeover, and fencing
Stop only the primary in an approved window, observe standby communications-interrupted and partner-down transitions, verify clients receive unique leases, and keep the old primary fenced until resynchronization is ready.
Why this step matters
A planned failure test is the only way to validate timers, client-observation guard, relay reachability, and operational fencing together. The original primary must not return as an unsynchronized second authority.
What to understand
Record time from last heartbeat through communications-interrupted to partner-down and compare it with max-response-delay plus the client observation gate.
If the peer link alone fails while both nodes still see clients, automatic takeover can create split brain. Keep network and service fencing able to stop one allocator.
During takeover, test new allocation and renewal from different relay segments while watching for two offers or conflicting addresses.
System changes
- Stops the primary DHCP service and allows the standby to assume full allocation authority after its safety gates.
Syntax explained
systemctl stop- Creates a controlled daemon failure while leaving the host and evidence available.
journalctl -f --line-buffered- Streams state changes in real time so takeover timing is measured.
Values stay on this page and are never sent or saved.
sudo systemctl stop {{keaService}}; date -Is; sudo journalctl -u {{keaService}} -f | grep --line-buffered -E 'HA_COMMUNICATION|HA_STATE_TRANSITION|HA_LOCAL_DHCP'2026-07-28T14:22:10+00:00 HA_HEARTBEAT_COMMUNICATIONS_FAILED kea-b: failed to communicate with kea-a HA_STATE_TRANSITION kea-b: HOT-STANDBY -> COMMUNICATIONS-INTERRUPTED HA_COMMUNICATION_INTERRUPTED_CLIENT4_UNACKED 10 clients were unacked HA_STATE_TRANSITION kea-b: COMMUNICATIONS-INTERRUPTED -> PARTNER-DOWN HA_LOCAL_DHCP_ENABLE local DHCP service is enabled for all clients
Checkpoint: One and only one peer serves after the tested failure
sudo tcpdump -ni {{clientInterface}} -c 20 '(udp port 67 or udp port 68)'Continue whenStandby becomes partner-down after the guard, clients receive leases, and no packets originate from the fenced primary.
Stop whenBoth peers answer, standby never transitions, leases conflict, or management cannot fence either side.
If this step fails
Standby remains communications-interrupted indefinitely.
Likely causeNot enough clients increment SECS to satisfy max-unacked-clients, or client traffic does not reach the standby.
tcpdump -ni {{clientInterface}} 'udp port 67 or 68'journalctl -u {{keaService}} | grep UNACKED
ResolutionRestore the primary, analyze client behavior and relay path, then choose a documented external fence or carefully tuned threshold; do not simply set the guard to zero.
Both peers issue offers after a network partition.
Likely causeThe failed path isolated peers but not clients and automatic partner-down or manual override activated both.
tcpdump -ni {{clientInterface}} -e 'udp port 67 or 68'journalctl -u {{keaService}} | grep HA_STATE_TRANSITION
ResolutionFence one node immediately, preserve lease files and packet evidence, reconcile allocations, and repair the failure-domain design before re-enabling automatic takeover.
Security notes
- Fencing authority can intentionally stop DHCP; restrict it, log it, and require a runbook that identifies the surviving source of truth.
Alternatives
- Use a relay, orchestrator, or network fence that routes client traffic to only one peer when reliable client-observation takeover is impossible.
Stop conditions
- Abort if there is no deterministic way to ensure only one allocator remains active.
command
Return the failed peer through maintenance and lease synchronization
Keep the recovered node from serving, inspect its database and configuration, start it into HA recovery, complete lease synchronization from the surviving authority, and only then restore normal roles.
Why this step matters
A returning peer has stale lease knowledge and must not answer until it has synchronized from the node that served during the outage. State-machine evidence provides the safe rejoin gate.
What to understand
Do not copy lease CSV files while daemons are active. Let the HA hook perform supported synchronization and preserve source-of-truth direction.
If software maintenance is required, use Kea's maintenance commands so the partner deliberately owns service before changing packages.
After synchronization, compare lease counts and sample recent outage allocations before restoring ordinary role expectations.
System changes
- Starts the recovered daemon, replaces or updates its stale lease database through HA synchronization, and returns the pair to hot standby.
Syntax explained
systemctl start- Returns only the recovered peer; HA state still controls whether local DHCP is enabled.
HA_LEASES_SYNC_COMPLETE- Required log evidence that the recovering database received the authoritative lease set.
Values stay on this page and are never sent or saved.
sudo systemctl start {{keaService}} && sudo journalctl -u {{keaService}} -f | grep --line-buffered -E 'HA_STATE_TRANSITION|HA_LEASES_SYNC|HA_LOCAL_DHCP'HA_STATE_TRANSITION kea-a: WAITING -> SYNCING, partner state PARTNER-DOWN HA_LEASES_SYNC_START starting lease database synchronization from kea-b HA_LEASES_SYNC_COMPLETE 1482 leases synchronized successfully HA_STATE_TRANSITION kea-a: SYNCING -> HOT-STANDBY HA_LOCAL_DHCP_ENABLE local DHCP service is enabled in normal relationship
Checkpoint: Recovered peer synchronized before normal service
journalctl -u {{keaService}} --since '-10 minutes' | grep -E 'HA_LEASES_SYNC_COMPLETE|HA_STATE_TRANSITION'Continue whenSync completes without errors and both nodes return to normal hot-standby state with matching recent leases.
Stop whenThe recovering node serves before synchronization, sync fails, or lease counts and identifiers diverge.
If this step fails
Lease sync repeatedly fails or times out.
Likely causeThe HA listener is blocked, lease database is unwritable, volume is large relative to timeout, or versions/configuration are incompatible.
journalctl -u {{keaService}} -n 200 --no-pagerdf -h {{leaseDirectory}}; stat {{leaseDirectory}}curl http://{{peerAddress}}:{{peerHaPort}}/
ResolutionKeep the recovering peer fenced, fix capacity, connectivity, and compatibility, then repeat supported synchronization from the surviving authority.
Security notes
- The surviving peer is the authoritative source during recovery; record who approved that direction before overwriting stale state.
Alternatives
- Restore a verified lease backup only when both peer states are unusable and conflict reconciliation has been planned.
Stop conditions
- Never enable unsynchronized local DHCP merely to make the recovered service appear healthy.
verification
Verify steady state, reservation, lease parity, and alert ownership
Finish with state, service, listener, lease, packet, capacity, and reservation evidence from both peers, then alert on any state outside normal hot standby, failed sync, lease-file growth, and partner communication.
Why this step matters
A durable DHCP service includes address correctness, synchronized state, protected control plane, capacity, monitoring, and an owned failover decision—not merely two active systemd units.
What to understand
Monitor peer state labels and transition duration; communications-interrupted may be expected briefly, but partner-down, terminated, unavailable, or repeated syncing need action.
Alert on lease file write failures, cleanup failure, disk pressure, duplicate offers, pool utilization, reservation conflicts, control listener exposure, and package version skew.
Schedule controlled failover and recovery exercises, including a network partition that tests the external fence without causing production duplicate allocation.
System changes
- No new changes; captures final service, socket, state, synchronization, storage, and client evidence.
Syntax explained
ss -lunp / -lntp- Verifies DHCP UDP and HA TCP listeners bind only intended addresses.
stat -c '%s'- Provides a lease-file growth baseline for capacity monitoring.
Values stay on this page and are never sent or saved.
systemctl is-active {{keaService}}; ss -lunp | grep ':67 '; ss -lntp | grep ':{{peerHaPort}} '; journalctl -u {{keaService}} --since '-15 minutes' | grep -E 'HA_STATE_TRANSITION|HA_LEASES_SYNC_COMPLETE|HA_COMMUNICATION'; stat -c '%s %n' {{leaseFile}}active
UNCONN 0 0 10.20.0.21:67 0.0.0.0:* users:(("kea-dhcp4",pid=2141,fd=18))
LISTEN 0 128 192.0.2.21:8000 0.0.0.0:* users:(("kea-dhcp4",pid=2141,fd=24))
HA_STATE_TRANSITION kea-a: SYNCING -> HOT-STANDBY
HA_LEASES_SYNC_COMPLETE 1482 leases synchronized successfully
184212 /var/lib/kea/kea-leases4.csv
peer_state=HOT-STANDBY lease_parity=PASS reservation=PASS duplicate_offers=0Checkpoint: The complete HA operating contract is satisfied
journalctl -u {{keaService}} --since '-15 minutes' | grep HA_STATE_TRANSITIONContinue whenBoth peers are in normal relationship, leases and reservations match, only intended listeners are exposed, and alert owners are recorded.
Stop whenPeer state, lease parity, reservation, firewall, storage, or client tests lack evidence.
If this step fails
Peer state is normal but pool utilization approaches exhaustion.
Likely causeAddress demand, abandoned leases, reservation layout, or subnet sizing exceeds the operational model.
curl http://{{primaryAddress}}:{{primaryHaPort}}/ -d '{"command":"stat-lease4-get","service":["dhcp4"]}'grep -c ',0,' {{leaseFile}}
ResolutionForecast and expand or redesign the subnet safely; HA cannot create additional addresses.
Security notes
- Keep full lease and reservation evidence in restricted monitoring; expose aggregate health metrics broadly.
Alternatives
- Use ISC Stork for supported fleet visibility after securing its agents, database, credentials, and operator roles.
Stop conditions
- Do not close rollout until fencing and recovery have been demonstrated, not just described.
Finish line
Verification checklist
journalctl -u {{keaService}} --since '-15 minutes' | grep -E 'HA_STATE_TRANSITION|HA_LEASES_SYNC_COMPLETE|HA_COMMUNICATION'Primary and standby are in normal hot-standby relationship, lease updates are enabled, recent synchronization succeeded, and no communication failure remains.sudo perfdhcp -4 -r 1 -R 2 -p 3 -l {{clientInterface}} -Y 1 -y 2The disposable dynamic clients complete DORA exactly once, the reserved client receives only its approved address, and packet capture shows no duplicate offers.sudo nft list ruleset; curl --fail http://{{peerAddress}}:{{peerHaPort}}/ -d '{"command":"status-get","service":["dhcp4"]}'Lease samples and counts agree, the peer command succeeds, and equivalent requests from client networks are blocked.journalctl -u {{keaService}} | grep -E 'COMMUNICATIONS-INTERRUPTED|PARTNER-DOWN|HA_LEASES_SYNC_COMPLETE' | tail -20The planned test records guarded takeover by one peer and a completed synchronization before the recovered peer resumes normal role.Recovery guidance
Common problems and safe checks
Clients time out even though both Kea services are active and peers exchange heartbeats.
Likely causeRelay helpers, subnet selection, interface binding, switch DHCP snooping, or UDP 67 firewall policy blocks the actual client path.
tcpdump -ni {{clientInterface}} 'udp port 67 or 68'journalctl -u {{keaService}} | grep -E 'DHCP4_PACKET|SUBNET'nft list ruleset
ResolutionFix relay delivery and subnet selection on both nodes; peer HTTP health is not a substitute for DHCP packet reachability.
The pair alternates between normal and communications-interrupted.
Likely causeHeartbeat timer coordination, overloaded shared listener, packet loss, route asymmetry, or firewall state intermittently breaks peer commands.
journalctl -u {{keaService}} | grep HA_HEARTBEATcurl -w '%{time_total}\n' http://{{peerAddress}}:{{peerHaPort}}/ss -lntp | grep ':{{peerHaPort}}'
ResolutionUse the dedicated listener, restore stable peer connectivity, and keep max-response-delay several heartbeat intervals before retesting.
A recovered peer reaches READY but never completes synchronization.
Likely causeLease-command hook, write permissions, capacity, timeout, incompatible version, or control listener prevents the full lease transfer.
journalctl -u {{keaService}} -n 200 --no-pagerdf -h {{leaseDirectory}}kea-dhcp4 -V
ResolutionKeep it fenced, correct storage and compatibility, and repeat synchronization from the surviving authority.
A reservation conflicts with a live dynamic lease.
Likely causeThe fixed address was introduced inside the pool without clearing previous ownership or peer configurations diverged.
curl http://{{primaryAddress}}:{{primaryHaPort}}/ -d '{"command":"lease4-get-all","service":["dhcp4"]}'jq '.Dhcp4.subnet4[].reservations' {{candidateConfig}}
ResolutionFence the address, reconcile the existing lease through supported commands, move the reservation outside the pool where possible, and deploy identical policy.
Reference
Frequently asked questions
Why use hot standby instead of load balancing?
Hot standby is simpler to reason about because one peer normally allocates and both have access to the whole pool. Load balancing can improve distribution but requires its 50/50 client scopes and failure behavior to be tested separately.
Why not set max-unacked-clients to zero for faster takeover?
Zero removes the client-observation interval designed to reduce split brain when peer communication fails but both servers still reach clients. Use it only with a proven external fence and documented client behavior.
Can both peers use one SQL lease database?
That is a different shared-database strategy. It adds database availability and latency as a dependency and does not provide the same peer-state visibility. Do not combine strategies without an explicit design.
Are reservations synchronized as leases?
Reservation definitions are configuration and must be deployed identically. When a reserved address becomes a lease, that lease state is synchronized through HA.
Recovery
Rollback
Rollback preserves one DHCP authority and its authoritative lease set. It does not merge conflicting leases automatically; any interval where two peers served requires conflict analysis and client remediation before returning to the former service.
- Fence the node that is not selected as the lease source of truth, stop its Kea service, and preserve its configuration, lease file, HA logs, and packet evidence read-only.
- On the surviving node, restore the last validated pre-HA configuration only after removing HA hooks and peer listener policy from the candidate, then run `kea-dhcp4 -t` before a serialized restart.
- Restore relay targets or the previous DHCP server only after proving the selected lease data is compatible and no second authority can answer. Keep old services stopped until packet capture confirms one responder.
- If duplicate allocation occurred, compare both lease databases and client evidence, quarantine contested addresses, and use application/network owners to reconcile clients. Do not concatenate lease CSV files.
Evidence