Configure a highly available virtual IP with Keepalived and VRRP
Operate a Keepalived VRRPv3 pair with explicit priority, non-preemptive recovery, bounded health tracking, protocol-aware firewalling, packet evidence, split-brain fencing, and tested failover.
Provide one stable IPv4 service address across two Linux peers while ensuring exactly one healthy node owns it and ambiguous partitions are detected and fenced.
- Ubuntu Server 24.04 LTS
- Keepalived 2.2.8
- VRRP VRRPv3
- Two peers on a supported shared network Both nodes have stable primary addresses, matching subnet reachability, synchronized time, independent recovery access, and a layer-two or explicitly reviewed unicast path that carries VRRP.
ip -br address && ip route - A reserved and demonstrably unused VIP The address is reserved in IPAM, excluded from DHCP, absent from static configuration and cloud ownership, and approved for movement between both peer interfaces.
arping -D -I {{interfaceName}} -c 3 {{virtualIpAddress}} - A local readiness contract The protected service exposes a bounded, read-only local readiness path and is configured identically on both peers, including keys, data access, firewall policy, and bind behavior.
- Independent fencing and external observation Operators can prevent either peer from serving without relying on VRRP communication and can probe the VIP from a real client path while retaining console access.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A two-node Keepalived pair that uses VRRPv3 to move one reserved IPv4 virtual address between identical service nodes without placing that address permanently in either host's ordinary network configuration.
- A bounded local readiness script, executed as an unprivileged account, that reduces effective VRRP priority only after repeated failures and therefore transfers ownership before clients are directed to an unhealthy service.
- A protocol-aware nftables boundary that admits VRRP only from the named peer on the intended interface while preserving independent management and service access during a partition.
- An operating procedure that proves packet fields, single ownership, client convergence, service failover, peer failure, recovery, non-preemptive handback, and split-brain fencing with concrete evidence.
- During normal operation exactly one peer lists {{virtualIpAddress}}, emits VRRPv3 advertisements for VRID {{vrid}}, and answers the external service probe; the other peer remains ready in BACKUP.
- Stopping the tracked service on the owner lowers its effective priority and moves the VIP to the healthy peer within the documented interval without allowing both nodes to answer simultaneously.
- Stopping or isolating the current owner produces a controlled takeover, while restoring it does not cause an unplanned immediate handback because preemption behavior is explicitly configured and tested.
- When the control path is ambiguous, operators can identify the partition from captures and journals, fence one peer, select a single authority, and recover without relying on priority as a substitute for mutual visibility.
Architecture
How the parts fit together
Each host runs the same protected application and one Keepalived VRRP instance on a reviewed interface. The elected MASTER assigns the floating address locally and advertises VRID, effective priority, address set, and interval using IP protocol 112. The BACKUP receives those advertisements and waits for the master-down interval before claiming the address. A local readiness script changes effective priority when the application cannot accept traffic, allowing the healthy peer to win. Gratuitous ARP updates the local network after ownership changes. The firewall restricts VRRP to the exact peer and interface, but an independent fence remains necessary because no election protocol can prove that an unreachable peer has stopped serving clients.
- Both peers parse matching policy, but {{primaryAddress}} starts with priority {{primaryPriority}} and becomes MASTER only after it observes the control path and local readiness; {{secondaryAddress}} remains BACKUP at {{secondaryPriority}}.
- The MASTER assigns {{virtualIpAddress}}/{{virtualIpCidr}}, sends gratuitous ARP, and emits VRRPv3 advertisements with VRID {{vrid}}, one-second cadence, and TTL 255 on {{interfaceName}}.
- The BACKUP receives advertisements and continuously evaluates its own readiness script. It does not answer through the VIP while a valid higher-priority owner is visible.
- A local application failure reduces the owner's effective priority by {{healthWeight}}. The healthy peer then wins the election, assigns the VIP, announces its link-layer ownership, and begins serving new client connections.
- A complete peer failure causes advertisements to stop. After the calculated master-down interval the BACKUP becomes MASTER, but only if external fencing and application-state assumptions permit safe independent service.
- A returning peer enters the observed election without forcing an immediate transfer. Operators compare application readiness, packet fields, VIP ownership, and client results before an intentional maintenance handback.
Assumptions
- The two peer interfaces participate in the same supported layer-two domain, or an explicit unicast VRRP design has been reviewed for the actual routed and virtualized network. Cloud platforms that reject movable addresses, gratuitous ARP, multicast, or protocol 112 require their native floating-IP mechanism instead.
- The proposed VIP is reserved in IPAM, excluded from DHCP pools, absent from static network configuration, and not owned by a load balancer, hypervisor, cloud API, router, stale Keepalived instance, or another VRID participant.
- Both nodes run the same protected service, configuration, credentials, content, data-access policy, and firewall behavior. Keepalived moves an address; it does not replicate application state, transactions, files, sessions, or databases.
- The readiness endpoint is local, bounded, read-only, representative of the ability to accept new work, and safe for frequent execution. It does not perform writes or create a dependency avalanche during an incident.
- Operators retain console or independent management access and a tested fence that can stop Keepalived, isolate the client interface, withdraw relay or switch access, or power off one node without depending on the failed VRRP path.
- Switches, virtual networks, host firewalls, and upstream security controls permit the reviewed VRRP packets and gratuitous ARP behavior. Anti-spoofing and port-security policy has been validated rather than disabled broadly.
Key concepts
- VRRP instance
- A set of routers or service nodes sharing a virtual-router identifier, address list, advertisement interval, and election policy. The instance is independent of an application cluster unless health tracking connects them.
- Virtual Router Identifier
- An 8-bit identifier used to distinguish VRRP instances on one network. Reusing a VRID with unrelated participants can merge elections even when the configuration names differ.
- Effective priority
- The configured base priority after tracking weights and state adjustments are applied. It determines the preferred owner only among peers that can actually exchange compatible advertisements.
- Master-down interval
- The time a BACKUP waits without valid MASTER advertisements before taking ownership. It combines multiple advertisement intervals with a priority-dependent skew rather than being an instantaneous heartbeat timeout.
- Preemption
- The behavior in which a higher-priority node retakes MASTER from a healthy lower-priority owner. Disabling automatic preemption can prevent avoidable second transitions after recovery.
- Gratuitous ARP
- An unsolicited address announcement used to update peer, switch, gateway, and client neighbor state after the VIP's MAC ownership changes. Its acceptance and convergence are network-dependent.
- Split brain
- A condition in which multiple peers believe they may own the VIP or serve the same client identity. Equal configuration and priority cannot prevent it when advertisements are partitioned.
- Fencing
- An independent action that prevents a selected peer from serving. Fencing converts ambiguous reachability into provable single ownership and is required for safe recovery from partitions.
Before you copy
Values used in this guide
{{interfaceName}}Exact interface carrying VRRP and the movable service address on both peers.
Example: ens192{{virtualIpAddress}}Reserved IPv4 service address that must be present on exactly one peer.
Example: 192.0.2.50{{virtualIpCidr}}Prefix length used when Keepalived assigns the virtual address.
Example: 24{{vrid}}Collision-free VRRP virtual-router identifier reviewed for this layer-two domain.
Example: 42{{routerId}}Human-readable local Keepalived router identifier, unique per peer.
Example: edge-a{{nodePriority}}Base priority rendered for the local node.
Example: 150{{primaryPriority}}Reviewed base priority for the preferred node.
Example: 150{{secondaryPriority}}Reviewed base priority for the standby node.
Example: 100{{primaryAddress}}Stable ordinary address of the preferred peer used for control-path policy and evidence.
Example: 192.0.2.21{{secondaryAddress}}Stable ordinary address of the standby peer used for control-path policy and evidence.
Example: 192.0.2.22{{trackedService}}Exact systemd unit for the application whose readiness controls ownership.
Example: haproxy.service{{servicePort}}Port reached by the local and external service verification request.
Example: 443{{serviceHealthPath}}Bounded read-only readiness endpoint used by the tracker.
Example: /readyz{{healthScriptPath}}Root-owned path of the fixed readiness script executed by Keepalived.
Example: /usr/local/libexec/keepalived/check-service{{healthWeight}}Positive magnitude subtracted from effective priority after repeated readiness failures.
Example: 60{{scriptUser}}Unprivileged account used to execute the readiness command.
Example: keepalived-check{{scriptGroup}}Restricted group owning the readiness script and any minimal readable resources.
Example: keepalived-check{{candidateConfig}}Staged complete Keepalived configuration validated before installation.
Example: /etc/keepalived/candidates/keepalived.conf{{firewallCandidate}}Staged nftables policy that admits reviewed VRRP peer traffic.
Example: /etc/nftables.d/30-keepalived-vrrp.nft{{partitionSeconds}}Approved duration of the isolated control-path test, long enough to cross the master-down interval but bounded by the maintenance plan.
Example: 8Security and production boundaries
- VRRP does not provide a modern confidentiality or peer-identity boundary. Restrict protocol 112 by interface and exact peer source, isolate the control network when practical, and monitor for unexpected VRID participants instead of relying on legacy authentication fields.
- A root-operated high-availability daemon can execute tracking and notification scripts. Keep `script_security` enabled, use a fixed root-owned script, run the probe as an unprivileged account, avoid interpolating network data, and never make the script directory writable by the service user.
- The VIP can bypass host-based expectations if the two nodes have different firewall, certificate, application, data, or logging configuration. Treat parity as a release invariant and verify the external service identity after every transition.
- Packet captures, IPAM records, service paths, and topology identify critical infrastructure. Retain evidence with access control and avoid pasting unrestricted captures into public support channels.
- A partition can make both nodes legitimately conclude that they are alone. Protocol filtering must never be used as a routine failover trigger; fence first whenever simultaneous client reachability is plausible.
Stop before continuing if
- Stop before installation if the VIP answers duplicate-address detection, appears in IPAM under another owner, overlaps DHCP, or cannot legally move between both peer interfaces.
- Stop if the peers are not on the same verified control path, protocol 112 or gratuitous ARP is filtered unpredictably, or the platform documents that VRRP-style address movement is unsupported.
- Stop if the protected service, data, credentials, certificates, firewall, or routes differ between nodes; moving the address would then move clients to a different security or consistency boundary.
- Stop if the readiness command is slow, mutating, secret-bearing, recursively dependent on the VIP, or executable through writable paths.
- Stop activation if parser output contains an unexplained warning, either peer cannot retain independent management access, or no external observer and fence are ready.
- Fence immediately if both nodes list the VIP, both journals report MASTER, an unexpected source advertises the VRID, or clients resolve the VIP to alternating link-layer owners.
- Stop a failure exercise when the remaining node lacks capacity, application error rate rises, state consistency is uncertain, or the recovery objective would be exceeded.
verification
Prove the layer-two domain, routes, ownership, and current VIP state
Before introducing VRRP, map both peers, the shared client segment, switching behavior, upstream routing, address management, and every existing owner of the proposed virtual IP.
Why this step matters
VRRP elects an address owner but does not make two hosts share a routed path that does not already exist. Duplicate-address detection and a topology inventory prevent deployment from colliding with a static, DHCP, cloud, or appliance assignment.
What to understand
Both interfaces must attach to the broadcast domain or reviewed unicast design that carries VRRP and gratuitous ARP. A provider network that forbids extra MAC/IP ownership requires a platform-native alternative.
Record switch ports, VLAN, subnet prefix, default gateway, hypervisor anti-spoofing, cloud source/destination checks, and IPAM ownership. A passing local route check does not prove upstream acceptance.
Run duplicate detection during a controlled quiet interval and cross-check IPAM, ARP tables, DNS, load balancers, and configuration management; silence alone is not proof that an address is unassigned.
System changes
- No changes; captures the baseline topology, interface identity, routing, neighbors, and absence of a current VIP owner.
Syntax explained
arping -D- Uses duplicate-address-detection probes without claiming the address.
-I {{interfaceName}}- Pins the probe to the exact shared layer-two interface.
Values stay on this page and are never sent or saved.
ip -br link; ip -br address; ip route; ip neigh show dev {{interfaceName}}; arping -D -I {{interfaceName}} -c 3 {{virtualIpAddress}}ens192 UP 52:54:00:1a:20:11 ens192 UP 192.0.2.21/24 default via 192.0.2.1 dev ens192 Unicast reply from 192.0.2.1 [00:11:22:33:44:55] 0.701ms Sent 3 probes (3 broadcast(s)) Received 0 response(s)
Checkpoint: The VIP is free and both peers share a supported network
arping -D -I {{interfaceName}} -c 3 {{virtualIpAddress}}Continue whenNo reply, a reserved IPAM record, matching subnet reachability on both peers, and platform confirmation that VIP movement is supported.
Stop whenAny host answers, peers are on different broadcast domains, or the network enforces source ownership that Keepalived cannot update.
If this step fails
The duplicate probe receives an ARP reply.
Likely causeThe address is already assigned, cached through a proxy ARP device, or present on a stopped-but-connected appliance.
ip neigh show {{virtualIpAddress}}arping -I {{interfaceName}} -c 5 {{virtualIpAddress}}
ResolutionIdentify the responding MAC through switching and asset records; allocate a different address unless ownership can be transferred explicitly.
Security notes
- Treat IPAM ownership and anti-spoofing controls as security boundaries; never weaken an entire VLAN merely to permit one unmanaged VIP.
Alternatives
- Use a cloud floating IP, provider route update, load balancer, or BGP advertisement where layer-two address movement is unsupported.
Stop conditions
- Do not configure the candidate until the VIP is reserved and the network team confirms the ownership mechanism.
command
Install and baseline the same Keepalived build on both peers
Install the distribution package, record the exact version and build options, inspect the unit, and confirm no previous process or configuration will compete for the same VRID.
Why this step matters
VRRP behavior, parser support, script security, and defaults vary by release and build. Matching package provenance makes priority, preemption, tracking, and notification behavior reproducible across the pair.
What to understand
This guide targets the Ubuntu 24.04 Keepalived 2.2.8 package and only standard VRRPv3 constructs documented by upstream. Confirm features rather than copying directives from a newer development manual.
Installation can create a unit but the service must remain stopped until both peer candidates and firewall policy validate. One premature MASTER can claim the production address.
Record the package source, binary checksum, enabled features, service sandboxing, configuration include paths, and any existing automation that restarts Keepalived automatically.
System changes
- Installs Keepalived, its systemd service, configuration directory, documentation, and package-managed operating files.
Syntax explained
keepalived --version- Prints release, kernel, and compile options needed for compatibility evidence.
systemctl cat- Shows the effective vendor unit plus local overrides without modifying it.
sudo apt-get update && sudo apt-get install --yes keepalived && keepalived --version 2>&1 | head -8 && systemctl cat keepalived | sed -n '1,120p' && pgrep -a keepalived || trueKeepalived v2.2.8 (01/29,2024) Linux 6.8.0-64-generic #67-Ubuntu Running on Linux 6.8.0-64-generic x86_64 configure options: --enable-snmp --enable-dbus ... # /usr/lib/systemd/system/keepalived.service ExecStart=/usr/sbin/keepalived --dont-fork
Checkpoint: Versions match and services remain controlled
keepalived --version 2>&1 | head -1; systemctl is-enabled keepalivedContinue whenBoth nodes report the approved matching version; rollout automation owns when the service becomes enabled and started.
Stop whenVersions or build options differ, an unknown override exists, or a service is already advertising the target VRID.
If this step fails
Keepalived starts immediately after package installation.
Likely causeA previous configuration exists or package policy starts enabled services automatically.
systemctl status keepalived --no-pagerjournalctl -u keepalived -n 100 --no-pager
ResolutionStop the unit, preserve the unexpected configuration and packet evidence, and prove the VIP was not claimed before continuing.
Security notes
- Install from the approved signed repository and inspect any pre-existing notification scripts before enabling the daemon.
Alternatives
- Build upstream only when a required feature is absent and the organization owns patching, reproducible builds, signing, and regression tests.
Stop conditions
- Do not enable either unit simply because package installation succeeded.
config
Permit VRRP protocol 112 only between the two approved peers
Stage narrow nftables policy for VRRP advertisements on the shared interface while retaining stateful application and management policy. VRRP is an IP protocol, not UDP or TCP port 112.
Why this step matters
Blocked advertisements cause each peer to believe the other is absent and can create two MASTER nodes. Over-broad acceptance allows an unexpected system to participate in the election or suppress the intended owner with a higher priority.
What to understand
VRRP uses IP protocol number 112 and the IPv4 multicast destination 224.0.0.18. A rule for TCP or UDP port 112 does nothing. The received advertisement's TTL must be 255.
Validate the candidate in the context of the complete firewall. This fragment deliberately cannot prove the surrounding chain priority, established-state, interface, namespace, virtualization, or upstream ACL behavior.
Keep client-service policy separate. VRRP advertisements may be healthy while the VIP's TCP service is blocked, and the service may be reachable while peer advertisements are silently dropped.
System changes
- Adds narrow local acceptance for standard multicast VRRP from the two peer source addresses and drops other VRRP directed to the group on that interface.
Syntax explained
meta l4proto vrrp- Matches IP protocol 112 rather than a transport-layer port.
ip daddr 224.0.0.18- Restricts the rule to the standard IPv4 VRRP multicast group.
{{firewallCandidate}}Values stay on this page and are never sent or saved.
table inet keepalived_vrrp {
chain input {
type filter hook input priority filter; policy accept;
iifname "{{interfaceName}}" ip saddr {{primaryAddress}} ip daddr 224.0.0.18 meta l4proto vrrp accept
iifname "{{interfaceName}}" ip saddr {{secondaryAddress}} ip daddr 224.0.0.18 meta l4proto vrrp accept
iifname "{{interfaceName}}" ip daddr 224.0.0.18 meta l4proto vrrp drop
}
}table inet keepalived_vrrp {
chain input {
type filter hook input priority filter; policy accept;
iifname "ens192" ip saddr 192.0.2.21 ip daddr 224.0.0.18 meta l4proto vrrp accept
iifname "ens192" ip saddr 192.0.2.22 ip daddr 224.0.0.18 meta l4proto vrrp accept
iifname "ens192" ip daddr 224.0.0.18 meta l4proto vrrp drop
}
}Checkpoint: Only peer advertisements traverse the actual filter path
sudo nft --check --file {{firewallCandidate}} && sudo tcpdump -ni {{interfaceName}} -c 5 'ip proto 112'Continue whenThe whole candidate validates and packet capture later shows VRRPv3 advertisements from only the elected peer with TTL 255.
Stop whenThe active firewall owner is unclear, protocol 112 is blocked upstream, or a third source advertises the same VRID.
If this step fails
No advertisements appear after both services start.
Likely causeThe wrong interface is monitored, multicast is suppressed, nftables or an upstream ACL blocks protocol 112, or neither instance parsed successfully.
tcpdump -ni any -vv 'ip proto 112'nft monitor trace
ResolutionFind and correct the exact drop point; do not force MASTER state as a workaround.
Security notes
- Do not use legacy PASS authentication as a security control. Restrict the network path and peers; VRRP authentication mechanisms do not replace an isolated control plane.
Alternatives
- Use a reviewed unicast peer configuration only when multicast truly cannot be carried, and retain equivalent source filtering and packet-capture tests.
Stop conditions
- Stop if firewall ownership or rollback is unclear; a mistaken replacement can sever both management and client access.
config
Create a bounded, non-privileged service readiness tracker
Track only a local, deterministic readiness condition that represents whether this host should own the VIP. Use a fixed executable path, short timeout, and unprivileged script user.
Why this step matters
A VIP that remains on a node whose service cannot answer is an availability failure. A tracker that hangs, mutates state, relies on remote dependencies, or runs as root can instead create flapping or a privilege boundary.
What to understand
The check asks the local service whether it can accept a request. It does not use the VIP, DNS, or remote internet, because those dependencies can create feedback loops during a transition.
Use repeated failure and success thresholds in Keepalived so one transient check does not move the address. Choose a negative weight that makes the unhealthy node lose, or enter FAULT for conditions where it must never own.
Keep output empty on success and bounded on failure. Monitor duration and exit code separately; a two-second check executed every two seconds must not accumulate concurrent processes.
System changes
- Creates a root-owned, non-writable executable readiness script invoked by Keepalived under an unprivileged account.
Syntax explained
--fail- Returns a non-zero status for HTTP error responses.
--max-time 2- Bounds the entire local readiness request so tracker processes cannot accumulate indefinitely.
{{healthScriptPath}}Values stay on this page and are never sent or saved.
#!/bin/sh
set -eu
exec /usr/bin/curl --fail --silent --show-error --max-time 2 http://127.0.0.1:{{servicePort}}{{serviceHealthPath}} >/dev/nullhealth script exit=0 duration=0.014s HTTP readiness returned 200 no stdout no stderr
Checkpoint: The tracker is fast, deterministic, and privilege-separated
sudo -u {{scriptUser}} timeout 3 {{healthScriptPath}}; echo $?Continue whenExit 0 only while the local service is ready, non-zero during the planned local failure, and completion well before the check interval.
Stop whenThe script writes data, requires root, follows untrusted input, reaches a remote dependency, hangs, or returns success while the service is unavailable.
If this step fails
Keepalived repeatedly logs script timeout and changes state.
Likely causeThe readiness endpoint blocks, DNS or remote access was added, the timeout exceeds the interval, or process permissions differ from the interactive test.
sudo -u {{scriptUser}} time {{healthScriptPath}}journalctl -u keepalived | grep -i script
ResolutionRestore a local bounded check, correct permissions, and require consecutive evidence before re-enabling tracking.
Security notes
- Keep the script root-owned and not writable by the service account; never interpolate network or user input into a shell command.
Alternatives
- Track a systemd unit only when its active state accurately represents readiness; a running process may still be unable to serve.
Stop conditions
- Do not attach an untested script to a production election because an exit-code inversion can move the VIP immediately.
config
Render the VRRPv3 instance with explicit priority and non-preemptive recovery
Use one deterministic template on both peers. Both begin as BACKUP, have unique unicast source addresses but share interface, VRID, advertisement interval, VIP, and health policy; only node priority differs.
Why this step matters
Explicit BACKUP state on both nodes lets the higher effective priority win the first election while `nopreempt` prevents a recovered preferred node from taking the VIP back immediately. This avoids a second disruption after a successful failover.
What to understand
Priority is a relative election value from 1 through 254 for backups; 255 represents the address owner in protocol semantics. Keep enough separation for the tracking weight to make an unhealthy preferred node lose.
Both nodes must use the same VRID only within this service pair and collision domain. A reused VRID can join unrelated elections even when virtual IPs differ.
Three advertisements and master-down calculation determine detection. Do not shorten timers without packet-loss, CPU-stall, switch-convergence, and false-failover testing.
`nopreempt` changes recovery: the current healthy MASTER remains owner until it fails or an operator performs a controlled handback. Document that as desired behavior rather than calling it a stuck priority.
System changes
- Defines a VRRPv3 election, one virtual address, gratuitous ARP behavior, a bounded health tracker, priority-based failover, and non-preemptive recovery.
Syntax explained
state BACKUP- Starts each node as a candidate rather than forcing an immediate uncoordinated MASTER claim.
priority {{nodePriority}}- Sets this node's base election preference before tracking weight is applied.
nopreempt- Prevents a later higher-priority candidate from taking ownership from a healthy current MASTER.
advert_int 1- Sends one advertisement per second; peer detection derives from this interval and priority.
{{candidateConfig}}Values stay on this page and are never sent or saved.
global_defs {
enable_script_security
script_user {{scriptUser}} {{scriptGroup}}
router_id {{routerId}}
}
vrrp_script chk_service {
script "{{healthScriptPath}}"
interval 2
timeout 2
fall 3
rise 5
weight -{{healthWeight}}
}
vrrp_instance VI_SERVICE {
state BACKUP
interface {{interfaceName}}
virtual_router_id {{vrid}}
priority {{nodePriority}}
advert_int 1
nopreempt
garp_master_delay 1
garp_master_repeat 3
virtual_ipaddress {
{{virtualIpCidr}} dev {{interfaceName}}
}
track_script {
chk_service
}
}VRRP_Instance(VI_SERVICE) Entering BACKUP STATE Script chk_service now returning 0 VRRP_Script(chk_service) succeeded Netlink reflector reports interface ens192 UP
Checkpoint: Peer candidates are identical except identity and priority
sudo keepalived --config-test --use-file={{candidateConfig}}Continue whenBoth candidates parse with the intended interface, same VRID/VIP/timers/tracker, unique router IDs, and priorities {{primaryPriority}} and {{secondaryPriority}}.
Stop whenThe parser warns, VRID collides, priority plus tracking weight cannot change the winner, or peers differ in more than documented identity fields.
If this step fails
The lower-priority node wins the initial election.
Likely causeThe preferred node's tracker reduces effective priority, its advertisements are filtered, interface is down, configuration differs, or the other node owns the address with priority semantics.
journalctl -u keepalived -n 100 --no-pagertcpdump -ni {{interfaceName}} -vv 'ip proto 112'
ResolutionCompare effective health, packet priority, VRID, VIP, and configuration; do not force state while the discrepancy is unexplained.
Security notes
- Keep scripts immutable to their execution user and do not include legacy cleartext VRRP authentication as a substitute for network isolation.
Alternatives
- Omit `nopreempt` only when automatic return to the preferred node is an explicit, tested requirement and a second transition is acceptable.
Stop conditions
- Do not install the configuration until collision, tracking weight, and recovery policy have peer-reviewed evidence.
command
Validate, install, and start one peer at a time
Parse each candidate offline, preserve the previous file, install atomically with strict ownership, start the lower-priority peer first, observe BACKUP, then start the preferred peer and observe a single election.
Why this step matters
Serialized activation limits the first claim to one observed peer and makes parser, firewall, interface, script, and election failures attributable. Starting both simultaneously hides which prerequisite failed.
What to understand
Start the secondary and prove it remains BACKUP without the VIP while alone only if the preferred peer is already advertising; in a fresh pair, the first healthy candidate will become MASTER after the down interval. Plan the exact sequence and client exposure.
Use console or an independent management path. Assigning the VIP and sending gratuitous ARP can redirect production connections immediately even though the service behind it has not been externally tested.
Preserve the previous file and installed digest. Configuration validation does not validate switch acceptance, remote firewall, service readiness semantics, or client neighbor-cache convergence.
System changes
- Installs the reviewed candidate, enables Keepalived, begins VRRP advertisements, and allows exactly one peer to assign the virtual address.
Syntax explained
--config-test- Parses the candidate without starting an election.
install -m 0640- Replaces the effective file with explicit owner, group, and restricted permissions.
Values stay on this page and are never sent or saved.
sudo keepalived --config-test --use-file={{candidateConfig}} && sudo install -o root -g root -m 0640 {{candidateConfig}} /etc/keepalived/keepalived.conf && sudo systemctl enable --now keepalived && systemctl is-active keepalived && journalctl -u keepalived -n 40 --no-pagerConfiguration file /etc/keepalived/keepalived.conf VRRP_Instance(VI_SERVICE) Entering BACKUP STATE VRRP_Instance(VI_SERVICE) received lower priority 150 VRRP_Instance(VI_SERVICE) Entering MASTER STATE Sending gratuitous ARP on ens192 for 192.0.2.50 active
Checkpoint: One MASTER owns the VIP and the other is BACKUP
ip address show dev {{interfaceName}} | grep -F '{{virtualIpAddress}}' || true; journalctl -u keepalived -n 30 --no-pagerContinue whenExactly one node lists the VIP, journals agree on MASTER/BACKUP roles, the health script succeeds, and no rapid transitions occur.
Stop whenBoth or neither own the VIP, roles flap, the script fails, or client traffic reaches an unready service.
If this step fails
Both nodes report MASTER immediately after activation.
Likely causeAdvertisements are filtered, interfaces/VLANs differ, VRID/config differs, or virtualization prevents multicast reception.
tcpdump -ni {{interfaceName}} -vv 'ip proto 112'nft monitor trace
ResolutionFence one node at once, preserve both packet traces, correct the control path, and repeat from a serialized stopped state.
Security notes
- A successful parser does not make root-executed notification or tracking scripts safe; this guide enables script security and an explicit unprivileged user.
Alternatives
- Run the first election on an isolated validation VLAN when production address movement cannot be safely observed.
Stop conditions
- Fence a peer immediately when single ownership cannot be proved.
verification
Decode advertisements, effective priority, TTL, and cadence
Capture the actual control traffic on both peers and correlate its VRID, priority, interval, source, destination, and TTL with journal state and VIP ownership.
Why this step matters
Journal state is one node's interpretation. A packet capture proves what the network carries and reveals collision, filtered reception, wrong priority, unexpected cadence, or a TTL that violates the on-link VRRP security check.
What to understand
Capture on both nodes during stable operation. The MASTER should send roughly one advertisement per configured interval, and the BACKUP should receive those same packets from the expected peer.
The standard IPv4 destination is 224.0.0.18, protocol is 112, and TTL is 255. A different source, VRID, address count, priority, or interval can identify an unrelated or stale instance.
The VIP must appear on only the MASTER. Neighbor caches on a client or gateway should map it to the elected host after gratuitous ARP convergence.
System changes
- No changes; records control-plane packets, interface ownership, and state transitions as the stable baseline.
Syntax explained
-vv- Decodes VRRP version, VRID, priority, interval, and TTL rather than printing only packet endpoints.
ip proto 112- Selects VRRP without confusing it with port 112.
Values stay on this page and are never sent or saved.
sudo timeout 12 tcpdump -ni {{interfaceName}} -vv 'ip proto 112'; ip -details address show dev {{interfaceName}}; journalctl -u keepalived --since '-5 minutes' --no-pager192.0.2.21 > 224.0.0.18: VRRPv3, Advertisement, vrid 42, prio 150, intvl 100cs, length 12, ttl 255
192.0.2.21 > 224.0.0.18: VRRPv3, Advertisement, vrid 42, prio 150, intvl 100cs, length 12, ttl 255
inet 192.0.2.50/24 scope global secondary ens192
VRRP_Instance(VI_SERVICE) Entering MASTER STATECheckpoint: The wire agrees with the elected roles
sudo timeout 5 tcpdump -ni {{interfaceName}} -c 3 -vv 'ip proto 112'Continue whenOnly the expected MASTER advertises VRID {{vrid}} at one-second cadence with intended priority and TTL 255; one node owns {{virtualIpAddress}}.
Stop whenA third source appears, fields differ from policy, packets arrive on only one peer, or both interfaces list the VIP.
If this step fails
Advertisements have the expected VRID but a surprising priority.
Likely causeHealth tracking changed effective priority, a config variable is wrong, or an unrelated instance reused the VRID.
journalctl -u keepalived | grep -E 'priority|Script|VRRP'keepalived --config-test --use-file=/etc/keepalived/keepalived.conf
ResolutionIdentify the source MAC and effective tracking state before changing priority; resolve collisions rather than outbidding them.
Security notes
- Packet captures expose infrastructure addresses and election policy; store them with restricted incident evidence.
Alternatives
- Mirror the switch port to a controlled analyzer when host capture cannot observe packets dropped before the interface.
Stop conditions
- Do not begin failover testing until stable packet cadence and single ownership are proven from both perspectives.
decision
Exercise local service failure and weighted ownership transfer
In a maintenance window, fail only the tracked readiness condition on the current MASTER. Observe consecutive failures, effective-priority reduction, BACKUP takeover, gratuitous ARP, and client recovery before restoring the service.
Why this step matters
Node loss is not the only failure. This exercise proves that the VIP follows application readiness, the tracking weight is mathematically sufficient, and the receiving node can serve clients without depending on the failed local process.
What to understand
Announce the test and watch client error rate continuously. Stopping the service is intentional disruption; use a representative non-critical environment first and retain an immediate restart command.
With primary priority {{primaryPriority}}, secondary priority {{secondaryPriority}}, and negative weight {{healthWeight}}, calculate the unhealthy effective priority before rollout. It must fall below the healthy peer without reaching ambiguous behavior.
The configured `fall 3` requires three failures before weight changes, while `rise 5` avoids handing eligibility back after one transient success. Measure actual detection and neighbor convergence.
System changes
- Stops the tracked application on the current owner, causes health weight to change, and can transfer the virtual address and all new client traffic.
Syntax explained
systemctl stop- Creates a controlled local readiness failure with a known reversal command.
journalctl -f- Streams tracker and state transitions while an independent client probe measures availability.
Values stay on this page and are never sent or saved.
sudo systemctl stop {{trackedService}}; journalctl -fu keepalivedVRRP_Script(chk_service) failed (exited with status 7) VRRP_Instance(VI_SERVICE) Changing effective priority from 150 to 90 VRRP_Instance(VI_SERVICE) received higher priority 100 VRRP_Instance(VI_SERVICE) Entering BACKUP STATE peer: VRRP_Instance(VI_SERVICE) Entering MASTER STATE peer: Sending gratuitous ARP on ens192 for 192.0.2.50
Checkpoint: Readiness failure moves service exactly once
curl --fail --connect-timeout 2 http://{{virtualIpAddress}}:{{servicePort}}{{serviceHealthPath}}; ip address show dev {{interfaceName}}Continue whenThe healthy peer owns the VIP, its service answers, the failed peer is BACKUP with reduced priority, and client interruption stays inside the approved budget.
Stop whenBoth peers own the address, the healthy peer is not ready, client errors exceed budget, or transitions continue after the expected takeover.
If this step fails
The service fails but the VIP does not move.
Likely causeThe script still returns success, fall threshold has not elapsed, weight is too small, or the peer's base priority is lower than expected.
sudo -u {{scriptUser}} {{healthScriptPath}}; echo $?journalctl -u keepalived | grep -E 'Script|priority'
ResolutionRestore service first, correct readiness semantics and priority math offline, then repeat the exercise.
Security notes
- Do not use a remotely writable health endpoint or file as a failover switch; authorization to fail readiness is authorization to move production traffic.
Alternatives
- Reduce priority from a supervised maintenance command when application readiness cannot be represented safely by an automatic script.
Stop conditions
- Abort and restart the service immediately when the remaining node cannot carry the full production load.
decision
Exercise loss of the MASTER process and advertisement stream
After restoring healthy steady state, stop Keepalived itself on the elected owner and measure master-down detection, VIP acquisition, gratuitous ARP convergence, client continuity, and monitoring alarms.
Why this step matters
Stopping the election daemon tests a different path from application health: advertisement withdrawal, master-down timing, address removal, peer acquisition, and layer-two neighbor update.
What to understand
A graceful stop can send priority zero and accelerate takeover; a crash or power loss cannot. Measure both only where the production risk and fencing plan permit, and never infer crash timing solely from graceful-stop evidence.
Probe from outside the hosts. A local curl can succeed through loopback or local routing while gateways and clients retain a stale neighbor entry.
Confirm monitoring reports the state transition and absence of advertisements. An HA mechanism that moves traffic silently creates an unowned degraded state.
System changes
- Stops the current VRRP daemon, removes its VIP, and transfers address ownership to the surviving peer.
Syntax explained
systemctl stop keepalived- Performs an orderly daemon withdrawal and is reversible with a controlled start.
ip address show- Confirms the old owner removed the address instead of relying only on log text.
Values stay on this page and are never sent or saved.
sudo systemctl stop keepalived; ip address show dev {{interfaceName}}; journalctl -u keepalived -n 60 --no-pagerVRRP_Instance(VI_SERVICE) sent 0 priority VRRP_Instance(VI_SERVICE) removing protocol VIPs peer: VRRP_Instance(VI_SERVICE) Transition to MASTER STATE peer: VRRP_Instance(VI_SERVICE) Entering MASTER STATE peer: Sending gratuitous ARP on ens192 for 192.0.2.50
Checkpoint: Peer loss is detected within the agreed objective
timeout 20 sh -c 'until curl --fail --silent http://{{virtualIpAddress}}:{{servicePort}}{{serviceHealthPath}}; do sleep 1; done'Continue whenThe survivor owns and serves the VIP once, client recovery meets the objective, and the stopped node lists no VIP.
Stop whenThe old owner retains the address, the survivor stays BACKUP, external clients retain an unusable neighbor, or error budget is exhausted.
If this step fails
The peer becomes MASTER but external clients cannot connect.
Likely causeGratuitous ARP is suppressed, upstream anti-spoofing pins ownership, client ACL differs, or the service binds only the old node's primary address.
arping -I {{interfaceName}} -c 3 {{virtualIpAddress}}ss -lntp | grep ':{{servicePort}}'
ResolutionRestore the known-good owner, correct neighbor propagation and service binding, and repeat with upstream observation.
Security notes
- A VIP transition can move authenticated sessions to a host with different keys or policy; keep service security material synchronized through its own controlled process.
Alternatives
- Use hypervisor power-off only in a later resilience exercise with console access and explicit fencing ownership.
Stop conditions
- Do not simulate a host crash until graceful loss and rollback have repeatable external evidence.
instruction
Recover the preferred peer without automatic preemption
Restore its service and Keepalived, verify that `nopreempt` keeps it BACKUP while the current owner remains healthy, then decide whether a planned handback is operationally useful.
Why this step matters
A recovered preferred node is not automatically a safer owner. Non-preemptive recovery avoids a second client disruption while operators verify service state, configuration, monitoring, time, routes, and advertisement reception.
What to understand
Higher base priority does not force takeover under `nopreempt`; this is expected. Treat the active secondary as the current authority until an approved maintenance window.
A controlled handback can stop Keepalived on the current MASTER after verifying the preferred node is healthy BACKUP. Restart the former owner afterward; do not remove `nopreempt` briefly in production.
Compare effective configuration and scripts before handback. The recovered node may have missed package, certificate, firewall, application, or routing changes during the incident.
System changes
- Restarts the previously failed application and VRRP participant; no VIP movement is expected while the current MASTER stays healthy.
Syntax explained
sleep 8- Allows multiple rise checks and advertisements before evaluating stable recovery.
nopreempt- Keeps the recovered higher-priority node from creating an automatic handback.
Values stay on this page and are never sent or saved.
sudo systemctl start {{trackedService}} keepalived; sleep 8; journalctl -u keepalived -n 50 --no-pager; ip address show dev {{interfaceName}}VRRP_Script(chk_service) succeeded VRRP_Instance(VI_SERVICE) Changing effective priority from 90 to 150 VRRP_Instance(VI_SERVICE) Entering BACKUP STATE VRRP_Instance(VI_SERVICE) received advert with priority 100 192.0.2.50 is not present on recovered node
Checkpoint: Recovered peer is healthy BACKUP
systemctl is-active {{trackedService}} keepalived; sudo -u {{scriptUser}} {{healthScriptPath}}; ip address show dev {{interfaceName}}Continue whenBoth services are active, tracker succeeds, advertisements are received, and the recovered peer does not own the VIP.
Stop whenIt preempts unexpectedly, cannot receive adverts, differs from the deployed policy, or the tracker is unstable.
If this step fails
The recovered high-priority peer immediately takes the VIP.
Likely cause`nopreempt` is missing, initial state is MASTER, configuration differs, or the current owner briefly stopped advertising.
grep -E 'state|nopreempt|priority' /etc/keepalived/keepalived.conftcpdump -ni {{interfaceName}} -vv 'ip proto 112'
ResolutionStabilize one owner, compare configs, restore BACKUP plus `nopreempt`, and repeat recovery under observation.
Security notes
- Do not use a handback as an excuse to run divergent secrets or firewall policy; both peers must remain equally trusted service owners.
Alternatives
- Use default preemption when rapid return to specialized hardware is required, but test the second transition and consider `preempt_delay`.
Stop conditions
- Do not hand back merely to restore a diagram; retain the healthy current owner until the change has a user benefit.
decision
Prove split-brain detection and fencing without serving from two owners
Simulate loss of the advertisement path only in an isolated or explicitly fenced exercise. Keep the client path observable, ensure one peer is prevented from serving, and prove alarms detect that both would otherwise become MASTER.
Why this step matters
VRRP cannot distinguish a dead peer from an advertisement-only partition. If both peers still reach clients, both can claim the same address. The exercise validates external fencing and detection rather than accepting dual ownership as normal failover.
What to understand
Use an isolated VLAN whenever possible. In production, first fence one node's client-facing output or service so a second MASTER state cannot produce duplicate ARP replies or concurrent writes.
A trap-cleaned, sleep-bounded table and independent console reduce the chance of leaving the partition in place. Keep `nft delete table inet ol_vrrp_test` ready in a second session, verify removal from both nodes, and capture the resulting priority and state transitions.
Alert from at least two signals: expected advertisements absent and VIP/Master present on both nodes. A monitor that shares the partitioned path can report both peers healthy independently.
If unicast VRRP is used, test asymmetric loss in each direction. One-way reachability can produce state that differs from a simple symmetric multicast drop.
System changes
- Creates a uniquely named temporary nftables table and hooked input chain, blocks VRRP reception for the bounded sleep interval, and removes the complete table through an EXIT/signal trap; an external fence prevents duplicate client service while the peer may elect itself MASTER.
Syntax explained
trap cleanup EXIT HUP INT TERM- Deletes the entire temporary table after normal completion, shell error, hangup, or an interactive termination signal.
sleep {{partitionSeconds}}- Keeps the narrowly scoped fault active only for the reviewed interval; an independent session verifies cleanup immediately afterward.
ip protocol vrrp drop- Drops only advertisements for the test instead of disconnecting the management or application path.
Values stay on this page and are never sent or saved.
sudo sh -c 'set -eu; if nft list table inet ol_vrrp_test >/dev/null 2>&1; then echo "stale ol_vrrp_test table exists" >&2; exit 1; fi; cleanup() { nft delete table inet ol_vrrp_test 2>/dev/null || true; }; trap cleanup EXIT HUP INT TERM; nft add table inet ol_vrrp_test; nft "add chain inet ol_vrrp_test input { type filter hook input priority -5; policy accept; }"; nft add rule inet ol_vrrp_test input ip protocol vrrp drop; sleep {{partitionSeconds}}'peer-a: VRRP_Instance(VI_SERVICE) Entering MASTER STATE peer-b: VRRP_Instance(VI_SERVICE) Entering MASTER STATE FENCE: peer-b client egress remains blocked ALERT: duplicate MASTER state / advertisements missing test table ol_vrrp_test removed by EXIT trap after 8 seconds
Checkpoint: Ambiguous ownership triggers fencing and an actionable alarm
ip address show dev {{interfaceName}}; sudo tcpdump -ni {{interfaceName}} -c 6 -vv 'ip proto 112'; sudo nft list rulesetContinue whenThe non-authoritative node cannot serve, operators see the dual-MASTER risk, the temporary drop is absent afterward, and normal single ownership returns.
Stop whenNo working fence exists, the test rule is not automatically reversible, console access is absent, or production clients can observe both owners.
If this step fails
Both peers own the VIP after the test rule is removed.
Likely causeAdvertisements remain blocked elsewhere, VRID/config differs, multicast membership failed, or neighbor/control state has not reconverged.
nft list rulesettcpdump -ni {{interfaceName}} -vv 'ip proto 112'
ResolutionKeep one peer fenced, restore verified advertisement reception, then restart only the fenced participant after a single owner is stable.
Security notes
- Fault injection is privileged production change. Require change control, independent rollback, exact targets, timeout, and evidence; never paste an unbounded remote firewall command.
Alternatives
- Model the partition in a lab with cloned topology and packet filters when production fencing cannot guarantee that clients see one owner.
Stop conditions
- Never perform this exercise merely to demonstrate VRRP; skip it until the fence is stronger than the tested failure.
verification
Verify one owner, service reachability, packet health, and alert coverage
Close the rollout with simultaneous evidence from both peers, an external client, the advertisement stream, neighbor resolution, configuration validation, application readiness, and monitoring.
Why this step matters
A final check must prove configuration, election, address ownership, network propagation, application readiness, process state, and monitoring together. Any one signal alone can remain green during a user-visible outage.
What to understand
Run the owner check on both peers at the same time and the client probe from outside their network namespace. Archive which node is intended to remain MASTER under `nopreempt`.
Alert on two or zero owners, no advertisements, unexpected source/VRID/priority, repeated transitions, tracker failure, service failure, VIP probe failure, and stale neighbor behavior.
Record configuration digests, package versions, interface/MAC mapping, firewall rules, packet sample, baseline detection time, failure exercises, fence owner, and controlled handback procedure.
System changes
- No changes; records final configuration, ownership, packet, service, and process evidence.
Syntax explained
--config-test- Revalidates the installed configuration rather than only the earlier candidate.
external curl- Proves the client path reaches the elected node and healthy application.
Values stay on this page and are never sent or saved.
sudo keepalived --config-test --use-file=/etc/keepalived/keepalived.conf; ip address show dev {{interfaceName}}; sudo timeout 4 tcpdump -ni {{interfaceName}} -c 3 -vv 'ip proto 112'; curl --fail --show-error http://{{virtualIpAddress}}:{{servicePort}}{{serviceHealthPath}}; systemctl is-active keepalived {{trackedService}}Configuration file /etc/keepalived/keepalived.conf is valid inet 192.0.2.50/24 scope global secondary ens192 VRRPv3, Advertisement, vrid 42, prio 100, intvl 100cs, ttl 255 ready active active
Checkpoint: The pair is stable and operationally owned
systemctl is-active keepalived {{trackedService}} && sudo -u {{scriptUser}} {{healthScriptPath}}Continue whenOne healthy MASTER, one healthy BACKUP, clean advertisements, successful external service probes, and alerts plus rollback assigned.
Stop whenOwnership is ambiguous, transitions recur, monitoring cannot distinguish control and service failures, or evidence is missing.
If this step fails
Everything is healthy locally but a remote subnet cannot reach the VIP.
Likely causeUpstream routing, ARP/ND cache, anti-spoofing, ACL, asymmetric return path, or service bind policy differs from the local test.
ip route get {{externalProbeAddress}} from {{virtualIpAddress}}tcpdump -ni {{interfaceName}} host {{externalProbeAddress}}
ResolutionTrace request and return path hop by hop, restore the last working owner if needed, and fix network policy without changing election timers.
Security notes
- Expose health and state metrics only to authorized monitoring; infrastructure topology and transition controls are sensitive.
Alternatives
- Add synthetic probes from multiple routed zones when one external observer cannot cover all client paths.
Stop conditions
- Do not close the rollout until the system is observable during both normal and degraded operation.
Finish line
Verification checklist
ip -json address show dev {{interfaceName}} | jq '[.[].addr_info[] | select(.local == "{{virtualIpAddress}}") ] | length'The command returns 1 on the elected owner and 0 on the peer; never 1 on both.sudo timeout 5 tcpdump -ni {{interfaceName}} -c 3 -vv 'ip proto 112'VRRPv3 advertisements use VRID {{vrid}}, intended priority, one-second cadence, destination 224.0.0.18, and TTL 255 from only the expected owner.curl --fail --connect-timeout 2 http://{{virtualIpAddress}}:{{servicePort}}{{serviceHealthPath}}A client outside both hosts reaches the healthy service before, during, and after each approved transition within the availability objective.sudo keepalived --config-test --use-file=/etc/keepalived/keepalived.conf && systemctl is-active keepalived {{trackedService}}The installed configuration validates and both the election daemon and tracked service are active on both nodes.Recovery guidance
Common problems and safe checks
Both peers enter MASTER immediately after the second service starts.
Likely causeVRRP advertisements do not cross the host firewall, switch, VLAN, virtual network, or unicast route, so each peer believes no owner exists.
sudo tcpdump -ni {{interfaceName}} -vv 'ip proto 112'sudo nft list rulesetip route get {{secondaryAddress}}
ResolutionFence one peer at once, preserve captures from both sides, restore the exact control path, and repeat activation serially from a state with no ambiguous VIP owner.
Both peers receive packets but ignore each other's advertisements.
Likely causeVRRP version, VRID, address count, advertisement interval, interface, or authentication-related policy differs between rendered configurations.
sudo keepalived --config-test --use-file={{candidateConfig}}sudo tcpdump -ni {{interfaceName}} -c 5 -vv 'ip proto 112'sudo diff -u /etc/keepalived/keepalived.conf {{candidateConfig}}
ResolutionRender both peers from one reviewed source with only local router ID, address, and priority differences; validate and reactivate one peer at a time.
The expected MASTER advertises but never assigns the VIP.
Likely causeThe configured interface or prefix is wrong, address assignment is rejected, a track script forces fault state, or another local network manager owns the address.
journalctl -u keepalived -n 150 --no-pagerip -details address show dev {{interfaceName}}systemctl status systemd-networkd NetworkManager --no-pager
ResolutionCorrect interface ownership and the candidate while the peer remains authoritative; do not add the VIP manually as a permanent workaround.
The VIP moves but external clients continue sending traffic to the previous node.
Likely causeGateway or client neighbor caches reject or delay gratuitous ARP, switch security pins the previous MAC, or the VIP is proxied elsewhere.
ip neigh show {{virtualIpAddress}}sudo arping -I {{interfaceName}} -c 3 {{virtualIpAddress}}sudo tcpdump -ni {{interfaceName}} 'arp or ip host {{virtualIpAddress}}'
ResolutionRestore the approved neighbor-update and switch policy, measure convergence from representative clients, and use the platform-native address mechanism if movable ownership is unsupported.
The local readiness script succeeds interactively but fails under Keepalived.
Likely causeThe daemon uses a different user, PATH, working directory, permissions, timeout, environment, or mandatory-access-control context.
sudo -u {{scriptUser}} env -i PATH=/usr/bin:/bin {{healthScriptPath}}namei -l {{healthScriptPath}}journalctl -u keepalived | grep -E 'Script|SECURITY|track'
ResolutionUse absolute binaries and fixed inputs, grant only the minimum read/execute path, and prove the same unprivileged invocation before re-enabling tracking.
A single transient readiness failure causes frequent VIP movement.
Likely causeRise/fall thresholds, interval, timeout, and priority weight do not provide hysteresis for observed application behavior.
journalctl -u keepalived --since '-30 minutes' | grep -E 'Script|priority|STATE'time sudo -u {{scriptUser}} {{healthScriptPath}}
ResolutionMeasure normal latency and failure duration, then set bounded timeout plus consecutive fall/rise counts; do not mask a persistently unhealthy dependency with a very long delay.
Stopping the tracked service does not transfer ownership.
Likely causeThe health weight leaves the current node above the peer, tracking is attached to another instance, or the endpoint still returns success from a different process.
journalctl -u keepalived | grep -E 'effective priority|Script|STATE'curl --fail --max-time 2 http://127.0.0.1:{{servicePort}}{{serviceHealthPath}}sudo tcpdump -ni {{interfaceName}} -c 3 -vv 'ip proto 112'
ResolutionCorrect the readiness identity and reviewed weight so a confirmed failure places the healthy peer higher, then repeat the exercise in a controlled window.
The recovered preferred peer immediately takes the VIP back and causes a second interruption.
Likely causePreemption is enabled, `nopreempt` is absent or incompatible with the initial-state design, or an operator restored the peer before validating service readiness.
grep -nE 'state|priority|nopreempt|preempt_delay' /etc/keepalived/keepalived.confjournalctl -u keepalived --since '-15 minutes' | grep STATE
ResolutionKeep the peer fenced, restore a non-preemptive reviewed policy, synchronize the application first, and perform any handback as a separately observed maintenance action.
An unrelated host appears in packet captures for the same VRID.
Likely causeThe identifier was reused in the broadcast domain, a stale lab or retired node still runs Keepalived, or network segmentation differs from the inventory.
sudo tcpdump -eni {{interfaceName}} -c 20 -vv 'ip proto 112'ip neigh show dev {{interfaceName}}
ResolutionDo not outbid the unknown participant. Identify and retire or isolate it, reserve a collision-free VRID, and repeat duplicate-address checks before activation.
VRRP packets are visible but their TTL is not 255.
Likely causeTraffic is being routed, proxied, replayed, or generated by a non-compliant path rather than arriving from the intended on-link peer.
sudo tcpdump -ni {{interfaceName}} -c 5 -vv 'ip proto 112'ip route get {{primaryAddress}}tracepath {{primaryAddress}}
ResolutionReject the path, restore the documented same-link design or create a separately reviewed unicast architecture, and never relax the TTL expectation merely to make election proceed.
The VIP is unique but the application response differs after failover.
Likely causeCertificates, configuration, content, data access, session handling, firewall policy, or application revision drifted between peers.
curl -vk --resolve service.example.net:{{servicePort}}:{{virtualIpAddress}} https://service.example.net:{{servicePort}}{{serviceHealthPath}}systemctl status {{trackedService}} --no-pagersha256sum /etc/keepalived/keepalived.conf
ResolutionKeep clients on the known-good owner, reconcile the application release and trust material through its supported process, and repeat external identity and state tests before handback.
The pair flaps between MASTER and BACKUP while both applications are healthy.
Likely causeIntermittent packet loss, CPU scheduling delay, duplicated sources, mismatched intervals, overloaded health probes, or unstable links cross the master-down threshold.
sudo timeout 30 tcpdump -ni {{interfaceName}} -w /var/tmp/vrrp-flap.pcap 'ip proto 112'journalctl -u keepalived --since '-30 minutes' --no-pagerip -s link show dev {{interfaceName}}
ResolutionStabilize the control and probe paths, identify missing advertisement sequences from timestamps, and keep one peer fenced until the election remains stable for an agreed observation period.
Reference
Frequently asked questions
Does the higher priority node always own the VIP?
Only when peers exchange compatible advertisements and preemption policy allows it. A partition can produce independent masters, and non-preemptive recovery intentionally leaves a healthy lower-priority owner in place.
Why not use a very large health weight?
The weight should produce one deliberate ordering for the tested failure without making unrelated tracking combinations unpredictable or driving priority outside the reviewed range. Model every effective priority state explicitly.
Can VRRP protect a database primary?
It can move an address, but it cannot elect a data-consistent writer or prevent two database primaries. Use the database's supported consensus, replication, promotion, and fencing system first.
Is multicast VRRP preferable to unicast?
Multicast matches the standard on-link model and naturally reaches participants, but some environments do not carry it. Unicast can work when explicitly supported, yet every peer route and failure mode must be reviewed because omissions create invisible partitions.
Why keep an external fence if Keepalived already elects one MASTER?
Election establishes one owner only while participants can observe the same control plane. Fencing is the independent proof needed when reachability is ambiguous or the failed node cannot be trusted to obey the election.
Recovery
Rollback
Rollback returns the service address to one explicitly selected healthy node or to the prior platform owner. It does not merge application state, repair sessions, or make a network that forbids address movement support VRRP.
- Fence the non-authoritative peer first so it cannot advertise or answer through the VIP. Preserve both effective configurations, journals, firewall rules, address state, neighbor evidence, and packet captures.
- Stop and disable Keepalived on both nodes if ownership is ambiguous. Remove the VIP from any interface that still holds it, then verify silence for the target VRID and absence of duplicate ARP replies.
- Restore the last approved static address, platform floating-IP assignment, load-balancer target, or single Keepalived configuration on the selected healthy node. Validate service readiness before directing clients.
- Restore normal client routing only after an external probe and neighbor-table evidence identify one owner. Keep the second participant fenced until root cause and control-path tests are complete.
Evidence