OneLinersCommand workbench
Guides
Databases & Data / High Availability & Reliability / Security

Build Redis Sentinel high availability and test failover

Deploy three independent Redis Sentinels around a TLS/ACL-protected Redis Open Source 8.8 primary and two replicas, prove quorum versus majority, validate Sentinel-aware clients and test promotion, fencing and backup restore.

240 min12 stepsHigh-impact changeRevision 2
Save or explore
#redis#redis-sentinel#high-availability#replication#failover#quorum#tls#acl
Save to collectionCreate a collection in the sidebar first.
0 of 12 steps completed
Goal

Provide automatic non-clustered Redis failover with accurately bounded asynchronous data-loss semantics, independently verified majority behavior and a recovery path that does not confuse replication with backup.

Supported environments
  • Redis Open Source 8.8.x latest approved patch
  • Redis Sentinel Sentinel 2 bundled with Redis 8.8
  • Ubuntu Server 24.04 LTS
Prerequisites
  • Secure Redis nodes One primary and two replicas already pass TLS, ACL, persistence, capacity and off-host restore checks.
  • Independent failure domains Three Redis and three Sentinel placements are evaluated for host, rack, zone and network partitions.
  • Sentinel-aware client Validated client library supports multiple Sentinel seeds, logical master name, TLS and separate credentials.
  • Write-loss contract Owner accepts asynchronous replication exposure and approves min-replica/WAIT behavior.
  • Fencing A tested path prevents clients from writing to an isolated old primary.
  • Backup Recent independently stored artifact restores in quarantine inside approved RPO/RTO.
Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System
  • A non-clustered Redis Open Source 8.8 topology with one primary, at least two replicas and three independently placed Sentinel processes. The three Sentinels monitor one named primary set with quorum 2, while a reachable majority of 2 remains mandatory to authorize failover.
  • TLS and named ACL authentication for client, replication and Sentinel links; protected configuration rendering for replication and Sentinel credentials; private firewall paths; unique host identities; durable Sentinel configuration directories and supervised services.
  • A Sentinel-aware application connection design that seeds at least two or three Sentinel addresses, discovers the current primary by service name, validates TLS names, reconnects after `switch-master`, rejects writes to old/read-only nodes and never caches one primary address indefinitely.
  • A controlled failover and recovery program that measures replica lag, verifies `SENTINEL CKQUORUM`, triggers an administrative failover, validates epoch and role convergence, fences the old primary, checks application writes, reintroduces the recovered node as a replica, and separately restores a backup.
Observable outcome
  • Every Sentinel agrees on the primary, replica and Sentinel counts; `CKQUORUM` confirms both configured quorum and majority; loss of one Sentinel still permits failover, while a minority partition cannot authorize promotion.
  • A selected healthy replica is promoted during a planned test, clients discover it and resume bounded writes, remaining replicas follow it, the old primary cannot accept divergent writes, and the topology converges without manual primary-address edits.
  • Operators can state the exact acknowledged-write-loss exposure from asynchronous replication, demonstrate whether `min-replicas-to-write` and `WAIT` meet policy, and restore an independent backup. Sentinel is described as availability orchestration, never as a backup, consensus database or zero-data-loss system.

Architecture

How the parts fit together

A single writable Redis primary asynchronously replicates to two replicas in distinct failure domains. Three Sentinel processes run independently from the Redis data processes where possible and exchange observations. For a master set configured with quorum 2, two Sentinels must consider the primary unavailable for ODOWN; an elected Sentinel still needs authorization from a majority of all known Sentinels, also two in a three-Sentinel deployment, before failover. Quorum is failure-detection sensitivity, while majority is election authorization, and they must not be conflated. The promoted replica may lag because Redis replication is asynchronous, so Sentinel improves service availability but cannot guarantee preservation of the latest acknowledged writes. `min-replicas-to-write` and `min-replicas-max-lag` can stop some writes when replication health is poor, while client `WAIT` can request acknowledgements; neither creates strong consistency or replaces independent backup. Clients query multiple Sentinel endpoints for the named primary and react to topology change. TLS, ACLs and private networking protect every path. Sentinels persist discovered state by rewriting their writable configuration files, so filesystem durability and permissions matter.

Redis primaryAccepts writes and streams asynchronous replication to replicas. It is a role, not a permanently fixed host.
Two or more replicasMaintain candidate copies and normally reject writes. Replica priority, offset, health and placement influence promotion.
Three SentinelsMonitor instances, exchange subjective failure observations, reach ODOWN quorum, elect a leader by majority and reconfigure topology.
Sentinel service nameStable logical name such as `orders-primary` used by Sentinels and compatible clients to discover the current primary.
Sentinel-aware clientsSeed multiple Sentinels, authenticate/TLS to Sentinel and Redis, discover the primary and reconnect after failover.
Replication and Sentinel ACL identitiesSeparate machine users with only official required command/channel permissions, independently rotated from application identities.
Fencing and network controlPrevent an isolated old primary from serving divergent writes through routing, load balancer, firewall and client discovery behavior.
Independent backupProvides recovery from operator error, corruption, deletion and replication of bad data, which Sentinel cannot undo.
  1. Define availability SLO, RPO, client timeout/retry behavior, replica lag and data-loss acceptance. Determine which failure domains can communicate during realistic partitions.
  2. Deploy one primary and at least two replicas with identical Redis 8.8/TLS/ACL/persistence settings, unique identities, stable private addresses and healthy asynchronous replication.
  3. Deploy three Sentinels in independent failure domains with a writable protected config, TLS, auth-user/auth-pass, master set name, quorum 2 and reviewed timing.
  4. Wait for discovery, then query every Sentinel for master, replicas, sentinels and CKQUORUM. Confirm all know three Sentinels and two replicas before enabling automatic failover.
  5. Configure clients with multiple Sentinel seeds, logical primary name, TLS/ACL credentials, bounded retry, idempotency and stale-connection handling.
  6. Capture offsets, write a deterministic canary, run `WAIT` when part of the application contract, trigger `SENTINEL FAILOVER` and measure detection, promotion and client recovery.
  7. Verify new role, configuration epoch, replica convergence, old-primary fencing and no split writes. Rejoin the recovered old primary only as a replica after checking its dataset.
  8. Exercise loss of one Sentinel, minority isolation, replica unavailability, TLS/credential rotation, backup restore and complete site failure without claiming automatic success where majority cannot exist.

Assumptions

  • Redis Open Source 8.8 latest approved patch is installed consistently on every data and Sentinel node. Sentinel 2 ships with Redis and is the supported Sentinel generation.
  • The topology is non-clustered Redis. Redis Cluster has different sharding, discovery, bus and failover behavior and is outside this guide.
  • Three Sentinel processes are independently placed. Running all three on one host, VM, rack, Kubernetes node or failure domain does not create meaningful majority availability.
  • One primary and at least two replicas use distinct failure domains, stable private IP/DNS identities and reliable bidirectional networking. A two-data-node topology can fail over but has weaker maintenance and data-copy resilience.
  • The primary and replicas already satisfy the secure Redis guide: private TLS-only listeners, named ACLs, persistence, capacity, monitoring and tested off-host backup.
  • Redis replication is asynchronous. A promoted replica can be missing writes acknowledged by the old primary; partitions can produce divergent histories until the old primary is fenced and reconfigured.
  • `min-replicas-to-write 1` with an approved `min-replicas-max-lag` can reject writes when no sufficiently current replica exists, reducing but not eliminating loss. The business accepts this availability tradeoff.
  • Applications use an official or validated Sentinel-capable Redis client and provide at least two Sentinel endpoints, the exact service name, separate Sentinel and Redis credentials and TLS validation.
  • Sentinel configuration is writable by its unprivileged service because Sentinel rewrites state. A read-only container/config-map design needs an explicit persistent writable config copy.
  • Sentinel credentials are secrets and may be present in protected Sentinel configuration. A secrets agent renders files atomically with restrictive permissions; examples use placeholders only.
  • DNS, certificate SAN and hostname resolution are compatible with Sentinel announcements and client reachability. NAT/container address translation is explicitly tested.
  • Automatic failover is enabled only after a planned manual failover and one-Sentinel-loss test pass under production-equivalent load.

Key concepts

SDOWN
Subjectively down: one Sentinel has failed to receive an acceptable response from an instance for `down-after-milliseconds`.
ODOWN
Objectively down: enough Sentinels, at least the configured quorum, agree that the primary is down.
Quorum
The number of Sentinels required to agree on primary failure and reach ODOWN. It is not the leader-election majority.
Majority
More than half of all known Sentinels whose authorization elects a failover leader. A minority partition cannot fail over.
Configuration epoch
Monotonic version assigned to a failover configuration so Sentinels converge on the newer primary mapping.
Failover timeout
A multi-purpose timing bound affecting repeated failover attempts and replica reconfiguration, not a promise of exact outage duration.
Replica priority
A replica selection input. Priority zero prevents promotion; lower positive values are preferred after offset and connectivity considerations.
Replication offset
Position in the replication stream used to compare how current candidates are. A larger offset usually means fewer missing writes.
Fencing
Preventing an old or isolated primary from accepting production writes after another replica is promoted.
Client discovery
Querying Sentinels for the primary address by logical master name and refreshing connections when topology changes.

Before you copy

Values used in this guide

{{masterName}}

Stable Sentinel logical primary name.

Example: orders-primary
{{primaryHost}}

Initial primary private DNS name.

Example: redis01.internal.example
{{primaryIp}}

Initial primary private address used by monitor.

Example: 10.60.10.11
{{replicaOneHost}}

First replica private DNS name.

Example: redis02.internal.example
{{replicaTwoHost}}

Second replica private DNS name.

Example: redis03.internal.example
{{redisPort}}

TLS Redis data port.

Example: 6379
{{sentinelPort}}

TLS Sentinel port.

Example: 26379
{{sentinelOneHost}}

First Sentinel private DNS name.

Example: sentinel01.internal.example
{{sentinelTwoHost}}

Second Sentinel private DNS name.

Example: sentinel02.internal.example
{{sentinelThreeHost}}

Third Sentinel private DNS name.

Example: sentinel03.internal.example
{{sentinelConfig}}

Writable protected Sentinel configuration.

Example: /var/lib/redis-sentinel/sentinel.conf
{{redisTlsCert}}

Node certificate chain path.

Example: /etc/redis/tls/redis.crt
{{redisTlsKey}}secret

Node private key path.

Example: /etc/redis/tls/redis.key
{{redisTlsCa}}

Trusted CA bundle.

Example: /etc/redis/tls/ca.crt
{{sentinelUser}}

Redis ACL username used by Sentinels.

Example: sentinel-monitor
{{replicationUser}}

Redis ACL username used by replicas.

Example: replicator
{{downAfterMs}}

Tested subjective failure threshold.

Example: 10000
{{failoverTimeoutMs}}

Tested failover timeout.

Example: 60000
{{canaryKey}}

Synthetic failover test key.

Example: ha:test:epoch

Security and production boundaries

  • All Redis and Sentinel ports remain private and TLS-only. Sentinel changes primary topology and therefore is a privileged control plane.
  • Sentinel and replication credentials must be different from application and human administration. Render protected configs without printing secrets.
  • Sentinel's writable configuration must not be writable by unrelated users. Tampering can redirect clients or promote an attacker-controlled endpoint.
  • Do not use `resolve-hostnames` or announce hostnames until every Sentinel, Redis node and client validates DNS and certificate SAN consistently.
  • Firewall Sentinels so only data nodes, other Sentinels and approved clients can reach them. Do not expose port 26379 to the Internet.
  • Application clients must validate certificates for both Sentinel and Redis endpoints; do not disable hostname validation after failover.
  • Fence an isolated old primary. ACL and read-only replica state help, but routing/client policy must prevent stale direct connections.
  • Avoid permanent manual writes to replicas. `replica-read-only yes` is the safe default and does not alone prevent every administrative write path.
  • Failover testing changes the writable primary and can interrupt or duplicate application operations. Use idempotency, bounded retries and change approval.
  • Sentinel Pub/Sub events and logs expose hostnames, IPs and topology. Restrict retention and access.
  • Do not reduce quorum to make a degraded deployment fail over. Restore independent Sentinel capacity and verify majority.
  • Sentinel cannot recover deleted/corrupt keys or guarantee zero loss. Retain independently authorized, tested backups.

Stop before continuing if

  • Stop if fewer than three independently placed Sentinels are known.
  • Stop if CKQUORUM does not report OK for both quorum and majority.
  • Stop if fewer than two healthy replicas exist before the drill.
  • Stop if any node uses plaintext, public exposure, default user or shared credentials.
  • Stop if replica lag, offset, priority or persistence state is unknown.
  • Stop if clients have one Sentinel seed or a hard-coded primary address.
  • Stop if old-primary fencing cannot be demonstrated.
  • Stop if application operations are not retry-safe or observable.
  • Stop if write-loss tolerance is represented as zero without a separate consistency mechanism.
  • Stop if backup restore is stale, untested or stored in the same failure domain.
  • Stop if Sentinel configs cannot be durably rewritten.
  • Stop on any split writes, unexpected role, epoch disagreement or data mismatch.
01

instruction

Design failure domains, quorum, majority and data-loss bounds

read-only

Map three Sentinel placements and three Redis data-node placements against host, rack, zone, network and operator failures. Configure quorum 2 for three Sentinels, while documenting that two reachable Sentinels are also the majority required to authorize failover.

Why this step matters

A Sentinel deployment is safe only when the counted processes survive independent failures. Quorum controls ODOWN detection, but majority controls leader authorization. In a three-Sentinel deployment both commonly equal two, yet they remain separate mechanisms. Redis replication is asynchronous, so automatic promotion can select a replica missing acknowledged writes.

What to understand

Model partitions, not only process crashes. Three Sentinels in one zone can lose majority together; one Sentinel co-located with each Redis node may couple data and control failures.

Document replica priority and maximum acceptable lag. Decide whether `min-replicas-to-write` and application `WAIT` are worth rejecting writes during degraded replication.

Identify a fencing path that blocks old-primary writes without depending solely on the failed network segment.

System changes

  • No runtime changes; produces the HA, RPO and fencing design.

Syntax explained

quorum 2
Two Sentinels must agree on failure for ODOWN.
majority 2/3
At least two Sentinels must authorize a leader and failover.
Example output / evidence
Topology decision
Redis nodes: redis01 / redis02 / redis03 in three failure domains
Sentinels: sentinel01 / sentinel02 / sentinel03 in three failure domains
Known Sentinels: 3
Detection quorum: 2
Election majority: 2 of 3
Replica candidates: 2
Accepted acknowledged-write loss: <= 1 second target, not guaranteed

Checkpoint: Verify this layer before continuing

Continue whenThree independent Sentinels and three data nodes are mapped, quorum/majority are explicit, and write-loss/fencing/client assumptions are approved.

Stop whenStop if any two counted Sentinels share one failure domain, fencing is absent or stakeholders expect zero loss.

If this step fails

Design proposes two Sentinels with quorum 2.

Likely causeTwo processes cannot preserve a majority after one failure and create ambiguous partition behavior.

Safe checks
  • Review failure-domain and connectivity matrix.

ResolutionDeploy an odd number of at least three independent Sentinels and retest every partition case.

Security notes

  • Topology diagrams and addresses are sensitive; use sanitized examples outside the operational system.

Alternatives

  • Use manual failover or a managed service when independent majority cannot be provided.

Stop conditions

  • Stop if any two counted Sentinels share one failure domain, fencing is absent or stakeholders expect zero loss.
02

verification

Verify primary and two replicas before Sentinel

read-only

Confirm all nodes run the same approved Redis Open Source 8.8 patch, TLS and ACLs. On the primary, verify two online replicas, offsets, lag, priorities, persistence and read-only replica behavior.

Why this step matters

Sentinel cannot manufacture a safe replica. Promotion eligibility begins with current replication, matching software/security configuration, persistence health and correct candidate priority. A topology with one replica cannot tolerate maintenance and a simultaneous failure without losing redundancy.

What to understand

Query ROLE and INFO replication on every node. Verify replicas report the exact primary, link_up, low lag and `replica-read-only yes`.

Compare TLS certificates and ACL identity availability on every candidate. A promotion that clients cannot authenticate to is still an outage.

System changes

  • No changes; reads data-node roles and replication health.

Syntax explained

master_repl_offset
Primary replication stream position used with replica offsets to estimate lag.
Command
Fill variables0/2 ready

Values stay on this page and are never sent or saved.

redis-cli --tls --cacert {{redisTlsCa}} -h {{primaryHost}} --user <monitor-user> --askpass INFO replication
Example output / evidence
role:master
connected_slaves:2
slave0:ip=10.60.20.12,port=6379,state=online,offset=48299122,lag=0
slave1:ip=10.60.30.13,port=6379,state=online,offset=48299122,lag=0
master_repl_offset:48299122
repl_backlog_active:1

Checkpoint: Verify this layer before continuing

Continue whenOne primary and two online current replicas use identical Redis/TLS/ACL settings and healthy persistence.

Stop whenStop if any replica is stale, disconnected, priority zero unexpectedly, writable, mismatched or has persistence errors.

If this step fails

Replica link repeatedly enters down state.

Likely causeNetwork, TLS, replication ACL, disk, primary backlog or resource pressure breaks synchronization.

Safe checks
  • INFO replication on primary and replica
  • ACL LOG 20
  • journalctl -u redis-server -n 100

ResolutionRepair transport/auth/capacity and wait for offsets to converge before deploying or testing Sentinel.

Security notes

  • Replication credentials must not appear in INFO captures, config excerpts or command arguments.

Alternatives

  • Use a temporary replacement replica and complete full synchronization before maintenance on an unhealthy member.

Stop conditions

  • Stop if any replica is stale, disconnected, priority zero unexpectedly, writable, mismatched or has persistence errors.
03

config

Render a protected TLS Sentinel configuration on each node

caution

Create an identical bootstrap policy with unique local network identity and a writable runtime config. Secrets are rendered by a secrets agent; placeholders below are never committed with plaintext values.

Why this step matters

Sentinel rewrites its configuration with discovered replicas, peer Sentinels, epochs and current primary. The file therefore must be both protected and durably writable. TLS replication enables Sentinel's outgoing data-node links and TLS Sentinel listener. Quorum 2 is fixed in the reviewed bootstrap.

What to understand

Choose down-after from measured network and event-loop latency, not a desire for the shortest outage. Too low causes false SDOWN; too high increases detection time.

Failover timeout affects retries and reconfiguration. Parallel syncs limits how many replicas become temporarily unavailable while syncing to the new primary.

Sentinel auth-pass is sensitive. Render it atomically with restrictive permissions; config rewrite must preserve the secret and ownership.

System changes

  • Creates the Sentinel control-plane configuration, including a protected credential and writable state.

Syntax explained

sentinel monitor ... 2
Monitors the initial primary under the stable name with failure-detection quorum two.
tls-replication yes
Uses TLS for Sentinel-to-Redis and Sentinel-to-Sentinel communication.
File {{sentinelConfig}}
Configuration
Fill variables0/10 ready

Values stay on this page and are never sent or saved.

port 0
tls-port {{sentinelPort}}
tls-cert-file {{redisTlsCert}}
tls-key-file {{redisTlsKey}}
tls-ca-cert-file {{redisTlsCa}}
tls-auth-clients no
tls-replication yes
aclfile /etc/redis/sentinel-users.acl
bind <this-sentinel-private-ip> 127.0.0.1
protected-mode yes
daemonize no
supervised systemd
sentinel sentinel-user sentinel-peer
sentinel sentinel-pass <rendered-peer-secret>
sentinel monitor {{masterName}} {{primaryIp}} {{redisPort}} 2
sentinel auth-user {{masterName}} {{sentinelUser}}
sentinel auth-pass {{masterName}} <rendered-secret>
sentinel down-after-milliseconds {{masterName}} {{downAfterMs}}
sentinel failover-timeout {{masterName}} {{failoverTimeoutMs}}
sentinel parallel-syncs {{masterName}} 1
Example output / evidence
sentinel.conf
mode: TLS only
master set: orders-primary
monitor: 10.60.10.11:6379
quorum: 2
down-after: 10000 ms
failover-timeout: 60000 ms
parallel-syncs: 1
mode: 0640 redis-sentinel:redis-sentinel
writable directory: PASS

Checkpoint: Verify this layer before continuing

Continue whenEach node has correct unique bind identity, identical master policy, protected secret, TLS-only ports and a writable durable config.

Stop whenStop if config is shared read-only, secret is exposed, announce addresses are unreachable or timing is untested.

If this step fails

Sentinel logs Failed to rewrite config.

Likely causeConfig file or directory is read-only, ephemeral, incorrectly owned or replaced by configuration management.

Safe checks
  • namei -l {{sentinelConfig}}
  • journalctl -u redis-sentinel -n 100

ResolutionProvide a protected persistent writable copy and reconcile bootstrap versus runtime state; do not grant broad filesystem write.

Security notes

  • Never display auth-pass or private key. Restrict Sentinel configuration backups as credentials.

Alternatives

  • Use mTLS client certificates in addition to ACL authentication when certificate lifecycle is automated.

Stop conditions

  • Stop if config is shared read-only, secret is exposed, announce addresses are unreachable or timing is untested.
04

config

Protect Sentinel peer, operator and discovery identities with ACLs

caution

Render the same protected ACL identities on every Sentinel before startup. Disable the default user, reserve a superuser exclusively for peer-to-peer Sentinel authentication as required by Sentinel, give applications only the documented discovery API, and give the controlled operator identity only the inspection and planned-failover subcommands used by this runbook.

Why this step matters

Sentinel has two different authentication planes. `sentinel auth-user` and `sentinel auth-pass` authenticate Sentinel to the monitored Redis data nodes. `sentinel sentinel-user` and `sentinel sentinel-pass` authenticate one Sentinel to its peers. Incoming application and operator connections authenticate against the local Sentinel ACL. Conflating these identities either grants discovery clients control-plane authority or prevents the Sentinels from reaching majority.

What to understand

The peer identity is a Sentinel superuser because the official Sentinel protocol requires broad peer privileges. It is not distributed to applications, monitoring, or routine operators.

The discovery identity follows Redis' documented restricted Sentinel client profile and cannot invoke failover or configuration changes.

The operator identity adds only `CKQUORUM` and `FAILOVER` to the documented discovery/inspection surface. Store its original secret in a secrets manager; the ACL file contains only a SHA-256 hash.

All three Sentinels need equivalent ACLs and peer credentials. Rotate with an overlap identity or a controlled one-node-at-a-time procedure so peers never lose majority authentication.

System changes

  • Creates three named Sentinel ACL identities, disables anonymous/default access and places a peer credential in the protected writable Sentinel configuration.

Syntax explained

sentinel sentinel-user / sentinel-pass
Authenticates this Sentinel to the other Sentinel instances; it is separate from data-node auth-user/auth-pass.
+sentinel|failover
Allows only the controlled operator identity to request the planned failover used later in the guide.
default off
Requires every incoming Sentinel client to authenticate explicitly.
File /etc/redis/sentinel-users.acl
Configuration
user default off
user sentinel-peer on #<sha256-peer-password-hash> allchannels +@all
user sentinel-client on #<sha256-client-password-hash> resetchannels -@all +auth +client|getname +client|id +client|setname +command +hello +ping +role +sentinel|get-master-addr-by-name +sentinel|master +sentinel|myid +sentinel|replicas +sentinel|sentinels +sentinel|masters
user sentinel-operator on #<sha256-operator-password-hash> resetchannels -@all +auth +client|getname +client|id +client|setname +command +hello +ping +role +sentinel|get-master-addr-by-name +sentinel|master +sentinel|myid +sentinel|replicas +sentinel|sentinels +sentinel|masters +sentinel|ckquorum +sentinel|failover
Example output / evidence
Sentinel ACL validation
default: off
sentinel-peer: on, peer superuser only
sentinel-client: discovery-only API
sentinel-operator: inspection, CKQUORUM and planned FAILOVER
ACL file: 0640 redis-sentinel:redis-sentinel
Peer password in sentinel.conf: protected
Unauthenticated PING: NOAUTH Authentication required

Checkpoint: Verify this layer before continuing

Continue whenAll three ACL files match, the default user is off, discovery cannot invoke FAILOVER, the operator can run CKQUORUM, peers discover one another and no plaintext password appears in the ACL file.

Stop whenStop if an application identity can invoke FAILOVER/CONFIG/ACL, a peer secret is exposed, ACLs differ across Sentinels or disabling default breaks peer discovery.

If this step fails

Sentinels start but never discover one another after ACL enforcement.

Likely causeThe peer credentials are absent or inconsistent, `sentinel-user`/`sentinel-pass` are missing, or the ACL file cannot be read.

Safe checks
  • Inspect sanitized Sentinel authentication errors.
  • Verify ACL file ownership and digest on all three nodes.
  • Test AUTH and PING locally without printing the password.

ResolutionRestore the last protected ACL/config pair, confirm peer authentication on one node at a time, and wait for three-way discovery before continuing.

Security notes

  • The peer credential is deliberately powerful. Never reuse it for clients, expose it in command arguments, or store the plaintext in source control.

Alternatives

  • Use password-only `requirepass` only when every supported Sentinel client lacks ACL support; it provides coarser identity separation and requires the same secret on every Sentinel.

Stop conditions

  • Stop if an application identity can invoke FAILOVER/CONFIG/ACL, a peer secret is exposed, ACLs differ across Sentinels or disabling default breaks peer discovery.
05

command

Start Sentinels one at a time and wait for discovery

caution

Start the first Sentinel, verify it monitors the primary, then add the second and third with a convergence interval. Do not start automatic failover tests until every Sentinel sees all peers and replicas.

Why this step matters

Sequential introduction makes peer membership observable and avoids mistaking partial discovery for a stable majority. Sentinel learns peers through hello messages on monitored Redis instances. TLS, ACL and address errors can allow the process to run while preventing correct discovery.

What to understand

After each addition, wait for all existing Sentinels to report the new peer. Use unique run IDs and stable announced addresses.

Inspect firewall paths among all Sentinels and from each Sentinel to every Redis node. One-way reachability can produce asymmetric SDOWN views.

System changes

  • Starts Sentinel services and begins active monitoring/config rewrites; no failover should occur on a healthy primary.

Syntax explained

enable --now
Starts Sentinel and configures restart at boot under systemd supervision.
Command
sudo systemctl enable --now redis-sentinel && sudo systemctl --no-pager --full status redis-sentinel
Example output / evidence
Active: active (running)
+monitor master orders-primary 10.60.10.11 6379 quorum 2
+slave slave 10.60.20.12:6379 10.60.20.12 6379 @ orders-primary 10.60.10.11 6379
+slave slave 10.60.30.13:6379 10.60.30.13 6379 @ orders-primary 10.60.10.11 6379
+sentinel sentinel <runid> 10.60.20.21 26379 @ orders-primary 10.60.10.11 6379

Checkpoint: Verify this layer before continuing

Continue whenAll three services run unprivileged, monitor the same primary and progressively discover two replicas plus two peer Sentinels.

Stop whenStop on unexpected SDOWN/ODOWN, duplicate identity, config rewrite failure, plaintext listener or incomplete discovery.

If this step fails

Sentinel process is active but reports zero replicas.

Likely causeSentinel cannot authenticate/TLS to the primary, monitor address is wrong or replicas are not connected to the primary.

Safe checks
  • SENTINEL MASTER {{masterName}}
  • SENTINEL REPLICAS {{masterName}}
  • ACL LOG 20

ResolutionCorrect monitored address, TLS and ACL, then wait for discovery; never manually add fake replica entries.

Security notes

  • Sentinel logs expose topology. Restrict access and redact run IDs/IPs from public reports.

Alternatives

  • Run Sentinels on dedicated small hosts or independent application hosts rather than co-locating all with data nodes.

Stop conditions

  • Stop on unexpected SDOWN/ODOWN, duplicate identity, config rewrite failure, plaintext listener or incomplete discovery.
06

verification

Prove quorum, majority and consistent topology from every Sentinel

read-only

Query master, replicas, peers and CKQUORUM through TLS on each Sentinel. Compare primary address, flags, epoch, num-slaves and num-other-sentinels before client onboarding.

Why this step matters

CKQUORUM is the operational gate because it checks configured detection quorum and the majority required to authorize failover. Running it on all three Sentinels also catches asymmetric membership. Equal primary and epoch views establish the pre-failover baseline.

What to understand

A Sentinel counts other Sentinels, so `num-other-sentinels` should be two in a three-node deployment. Capture run IDs and addresses privately for duplicate detection.

Monitor CKQUORUM continuously. A healthy primary can hide that the control plane lost majority and would not fail over later.

System changes

  • No changes; reads Sentinel topology and authorization readiness.

Syntax explained

SENTINEL CKQUORUM
Checks reachable usable Sentinels against both ODOWN quorum and failover-majority requirements.
Command
Fill variables0/4 ready

Values stay on this page and are never sent or saved.

redis-cli --tls --cacert {{redisTlsCa}} -h {{sentinelOneHost}} -p {{sentinelPort}} --user sentinel-operator --askpass SENTINEL CKQUORUM {{masterName}}
Example output / evidence
Enter password: [hidden]
OK 3 usable Sentinels. Quorum and failover authorization can be reached

master: 10.60.10.11:6379
flags: master
num-slaves: 2
num-other-sentinels: 2
quorum: 2
config-epoch: 0

Checkpoint: Verify this layer before continuing

Continue whenEvery Sentinel reports OK, same primary/epoch, two replicas and two other Sentinels.

Stop whenStop if any view differs, flags contain s_down/o_down unexpectedly, or usable Sentinel count is below three.

If this step fails

One Sentinel reports a different config epoch.

Likely causeIt missed configuration propagation, is partitioned, has stale state or duplicate address/run ID.

Safe checks
  • SENTINEL MASTER {{masterName}} on all three
  • SENTINEL SENTINELS {{masterName}}

ResolutionDo not fail over; restore communication and converge on the highest valid epoch before client traffic depends on Sentinel.

Security notes

  • Restrict Sentinel command access to approved clients; topology disclosure assists lateral movement.

Alternatives

  • Expose CKQUORUM as a monitored health check rather than a one-time deployment test.

Stop conditions

  • Stop if any view differs, flags contain s_down/o_down unexpectedly, or usable Sentinel count is below three.
07

instruction

Configure Sentinel-aware clients with bounded retries

caution

Give clients all three Sentinel seeds, {{masterName}}, TLS CA/hostname validation, a narrow Sentinel credential and the separate application ACL credential. Configure finite connection, command and retry budgets plus idempotency.

Why this step matters

Sentinel can reconfigure servers, but availability reaches applications only when clients discover and reconnect to the new primary. Separate Sentinel and Redis authentication prevents topology-read permission from becoming data access. Bounded jittered retries avoid an outage becoming an unbounded request storm.

What to understand

Test client startup with each seed unavailable and with only a majority reachable. Validate that DNS/SAN names returned by Sentinel can be connected securely.

Redis retries can duplicate non-idempotent operations after timeout. Use request identifiers, transactions or application reconciliation rather than retrying blindly.

Close or invalidate pools connected to a node that becomes read-only or returns role errors. Never write directly to known replica addresses.

System changes

  • Changes application connection configuration and may recycle connection pools.

Syntax explained

{{masterName}}
Logical service name queried through Sentinel, not a hostname.
multiple seeds
Allows discovery when one Sentinel endpoint is unavailable.
Example output / evidence
Client HA configuration
Sentinel seeds: sentinel01:26379, sentinel02:26379, sentinel03:26379
Master name: orders-primary
Sentinel TLS: verify-full
Redis TLS: verify-full
Sentinel user: sentinel-client
Redis user: orders-api
Connect timeout: 1s
Command timeout: 2s
Retry budget: 10s with jitter
Hard-coded primary: none

Checkpoint: Verify this layer before continuing

Continue whenClient starts with any one seed down, discovers current primary, validates TLS and recovers within retry budget without duplicate canary writes.

Stop whenStop if client hard-codes a primary, disables TLS validation, has one seed or unbounded retries.

If this step fails

Client reports no master found.

Likely causeMaster name differs, Sentinel authentication fails, seed list is unreachable or client library lacks TLS/Sentinel support.

Safe checks
  • SENTINEL GET-MASTER-ADDR-BY-NAME {{masterName}} from each seed
  • Review client debug without secrets

ResolutionCorrect the logical name, TLS/auth and supported client options; do not bypass Sentinel with a static production address.

Security notes

  • Connection strings and debug output can contain both Sentinel and Redis secrets; redact and use secret injection.

Alternatives

  • Put a tested Sentinel-aware proxy in front only when the application client cannot support discovery and the proxy itself is highly available.

Stop conditions

  • Stop if client hard-codes a primary, disables TLS validation, has one seed or unbounded retries.
08

command

Capture offsets and write a failover canary

caution

Record primary and replica offsets, persistence status and wall-clock time. Write an incrementing synthetic canary through the application client and optionally WAIT for the approved replica acknowledgements.

Why this step matters

A sequence canary and captured offsets provide concrete evidence of what the promoted replica retained. WAIT asks for replica acknowledgements before returning, but it does not turn Redis into a strongly consistent database and does not guarantee persistence on replicas. The result must be interpreted, not advertised as zero loss.

What to understand

Use a dedicated synthetic key and record its value before/after failover. Do not use customer transactions as a test.

Capture INFO replication and persistence on all candidates. Acknowledgement count below policy is a stop signal, even if the write itself succeeded.

System changes

  • Writes one synthetic increment and creates replication traffic; WAIT may add bounded latency.

Syntax explained

WAIT 1 2000
Waits up to 2000 ms for at least one replica to acknowledge the preceding writes on this connection.
INCR
Produces a monotonic synthetic value that exposes missing or duplicate failover writes.
Command
Fill variables0/3 ready

Values stay on this page and are never sent or saved.

redis-cli --tls --cacert {{redisTlsCa}} -h {{primaryHost}} --user <test-user> --askpass MULTI INCR {{canaryKey}} EXEC WAIT 1 2000
Example output / evidence
1) OK
2) QUEUED
3) 1) (integer) 1042
4) (integer) 1

master_repl_offset: 48310221
replica1 offset: 48310221 lag: 0
replica2 offset: 48310221 lag: 0
UTC: 2026-07-28T10:15:00Z

Checkpoint: Verify this layer before continuing

Continue whenCanary sequence and UTC time are recorded, required WAIT acknowledgement returns, replicas are online and offsets converge.

Stop whenStop if WAIT returns below target, any replica lags/disconnects or persistence status is not ok.

If this step fails

WAIT returns zero despite connected replicas.

Likely causeReplication is delayed, command ran on a different connection, timeout is too short or replicas are not acknowledging.

Safe checks
  • INFO replication
  • ROLE on all nodes

ResolutionCancel failover, restore replication health and validate client connection semantics before retrying.

Security notes

  • Synthetic key must contain no customer or incident data. Do not print credentials in the capture.

Alternatives

  • Use application-owned idempotency and durable event logging when write-loss requirements exceed Redis asynchronous guarantees.

Stop conditions

  • Stop if WAIT returns below target, any replica lags/disconnects or persistence status is not ok.
09

command

Trigger a controlled Sentinel failover

danger

During an approved window, issue SENTINEL FAILOVER to one healthy Sentinel without stopping the primary. Observe events and time each phase; do not issue competing commands from multiple Sentinels.

Why this step matters

A planned administrative failover exercises election, candidate selection, promotion, epoch propagation, replica reconfiguration and client discovery without confusing it with host power loss. It still changes the writable node and can interrupt or duplicate application requests, so it is a production change.

What to understand

Watch every Sentinel event stream and application errors. Record time to elected leader, selected candidate, switch-master, client recovery and replica convergence.

Do not kill the primary first in the initial drill; controlled failover isolates topology/client behavior. A later fault-injection test can exercise down detection and fencing.

Ensure backup and rollback evidence exist. Sentinel may not choose the candidate operators expect if offset or priority differs.

System changes

  • Promotes a replica, demotes/reconfigures nodes, increments epoch and redirects Sentinel-aware clients.

Syntax explained

SENTINEL FAILOVER
Requests an immediate planned failover of the named master without requiring it to be ODOWN.
+switch-master
Event announcing the logical master address changed.
Command
Fill variables0/4 ready

Values stay on this page and are never sent or saved.

redis-cli --tls --cacert {{redisTlsCa}} -h {{sentinelOneHost}} -p {{sentinelPort}} --user sentinel-operator --askpass SENTINEL FAILOVER {{masterName}}
Example output / evidence
Enter password: [hidden]
OK
+new-epoch 1
+try-failover master orders-primary 10.60.10.11 6379
+elected-leader master orders-primary 10.60.10.11 6379
+selected-slave slave 10.60.20.12:6379
+promoted-slave slave 10.60.20.12:6379
+switch-master orders-primary 10.60.10.11 6379 10.60.20.12 6379

Checkpoint: Verify this layer before continuing

Continue whenOne replica is promoted, one epoch is issued, clients reconnect inside SLO and no competing writable path appears.

Stop whenStop/rollback traffic on split writes, wrong candidate, stalled election, severe application errors or data mismatch.

If this step fails

FAILOVER returns NOGOODSLAVE.

Likely causeNo known replica is healthy, sufficiently current, eligible or authenticated for promotion.

Safe checks
  • SENTINEL REPLICAS {{masterName}}
  • INFO replication on candidates

ResolutionCancel the test, repair and synchronize replicas, confirm priority/offset/TLS/ACL and retry only after CKQUORUM passes.

Security notes

  • Restrict FAILOVER permission to time-bound operators. It is a state-changing control-plane action.

Alternatives

  • Use a staging topology first and a traffic-drained production window when the application has never experienced failover.

Stop conditions

  • Stop/rollback traffic on split writes, wrong candidate, stalled election, severe application errors or data mismatch.
10

verification

Verify roles, epoch, client recovery and canary continuity

caution

Query all Sentinels and data nodes. Confirm new primary, replicas and epoch agree, then increment/read the canary through normal client discovery and inspect application error/latency metrics.

Why this step matters

Failover is complete only when the control plane, server roles and application behavior converge. A switch-master event alone may precede replica reconfiguration. Canary continuity exposes missing writes while application metrics expose retry storms or stale pools.

What to understand

Query ROLE directly on every node and Sentinel MASTER on every Sentinel. Confirm the old primary is a replica and cannot accept ordinary writes.

Compare canary and representative synthetic data, not only DBSIZE. Record any gap as data loss even if availability recovered.

Verify TLS certificate and ACL behavior on the promoted host exactly as on the original primary.

System changes

  • Writes one post-failover synthetic canary increment through discovered primary; all other checks are read-only.

Syntax explained

GET-MASTER-ADDR-BY-NAME
Returns the current logical primary endpoint from Sentinel's view.
ROLE
Returns direct Redis role and replication state for one node.
Command
Fill variables0/4 ready

Values stay on this page and are never sent or saved.

redis-cli --tls --cacert {{redisTlsCa}} -h {{sentinelTwoHost}} -p {{sentinelPort}} --user sentinel-operator --askpass SENTINEL GET-MASTER-ADDR-BY-NAME {{masterName}}
Example output / evidence
Enter password: [hidden]
1) "10.60.20.12"
2) "6379"

New primary ROLE: master
Old primary ROLE: slave of 10.60.20.12:6379
Sentinel config epoch: 1 on all three
Canary before: 1042
Canary after: 1043
Client recovery: 2.8 seconds
Duplicate application operations: 0

Checkpoint: Verify this layer before continuing

Continue whenAll Sentinels agree on address/epoch, one writable primary exists, replicas follow it, client recovery meets SLO and canary is continuous.

Stop whenStop traffic if any Sentinel differs, two nodes are writable, old node is reachable as primary, canary regresses or duplicates occur.

If this step fails

Old primary remains master after switch-master.

Likely causeSentinel cannot reach/reconfigure it, ACL lacks replication command, TLS fails or network partition persists.

Safe checks
  • ROLE on old primary
  • SENTINEL REPLICAS {{masterName}}
  • ACL LOG 20

ResolutionFence client access immediately, preserve divergent data, restore connectivity and rejoin only as a replica after analysis.

Security notes

  • Do not resolve divergence by discarding an old dataset before preserving forensic and possible recovery evidence.

Alternatives

  • Keep the old primary isolated and build a fresh replica from the new primary when its state cannot be trusted.

Stop conditions

  • Stop traffic if any Sentinel differs, two nodes are writable, old node is reachable as primary, canary regresses or duplicates occur.
11

command

Exercise one-Sentinel loss and minority behavior

danger

Stop one Sentinel and prove CKQUORUM remains OK on the other two. Restore it and converge. In a separate controlled network simulation, verify a single isolated Sentinel cannot authorize failover.

Why this step matters

The essential Sentinel safety property is that a minority partition cannot authorize failover. Losing one of three should retain majority, while isolating one must not create another primary. This validates process placement, peer accounting and network partition behavior rather than only planned promotion.

What to understand

Coordinate network simulation with infrastructure owners and preserve management access. Use a bounded rule with automatic rollback.

After restarting a Sentinel, wait for peer count and epoch convergence. Sentinel does not automatically forget removed peers; planned membership changes use the documented RESET sequence.

System changes

  • Stops one control-plane process and may apply temporary network isolation; data nodes remain running.

Syntax explained

2 usable Sentinels
The surviving two are both quorum and majority in a three-Sentinel deployment.
NOQUORUM in minority
A single Sentinel correctly refuses failover authorization.
Command
Fill variables0/4 ready

Values stay on this page and are never sent or saved.

sudo systemctl stop redis-sentinel && redis-cli --tls --cacert {{redisTlsCa}} -h {{sentinelTwoHost}} -p {{sentinelPort}} --user sentinel-operator --askpass SENTINEL CKQUORUM {{masterName}}
Example output / evidence
Enter password: [hidden]
OK 2 usable Sentinels. Quorum and failover authorization can be reached
Stopped Sentinel restored: active
All Sentinels known after convergence: 3

Minority isolation test:
single Sentinel: NOQUORUM
failover attempted: no
production writes during test: controlled

Checkpoint: Verify this layer before continuing

Continue whenTwo survivors retain CKQUORUM, isolated minority does not promote, restored Sentinel rejoins with correct epoch and all clients remain single-primary.

Stop whenStop if minority can promote, majority cannot be reached after one loss, or firewall rollback is uncertain.

If this step fails

CKQUORUM fails after one Sentinel stops.

Likely causeThe survivors cannot communicate, total known membership is wrong or configured quorum exceeds available majority.

Safe checks
  • SENTINEL SENTINELS {{masterName}} on survivors
  • SENTINEL MASTER {{masterName}}

ResolutionRestore control-plane connectivity and correct topology; do not lower quorum to force the drill.

Security notes

  • Network fault injection can lock out operators; require out-of-band recovery and an automatic expiry.

Alternatives

  • Use a lab that reproduces failure domains if production network isolation cannot be performed safely.

Stop conditions

  • Stop if minority can promote, majority cannot be reached after one loss, or firewall rollback is uncertain.
12

verification

Restore a backup and accept the HA service

read-only

Independently restore the latest verified backup into quarantine and validate application semantics. Record failover and recovery RPO/RTO, then establish monitoring for quorum, roles, lag, persistence and backup age.

Why this step matters

Sentinel availability and backup recovery are separate acceptance gates. Failover proves service continuity under a tested fault, while restore proves recovery from corruption, deletion or total topology loss. Reporting zero lost canary writes in one drill must not be generalized into a zero-loss guarantee.

What to understand

Alert on CKQUORUM, known peers/replicas, SDOWN/ODOWN, switch-master, repeated elections, replication lag/offset, min-replica write rejection, persistence errors and client reconnect latency.

Schedule quarterly failover and restore drills, annual failure-domain partition tests and review timing/placement after network or workload changes.

System changes

  • No state changes; captures acceptance and creates recurring operational controls.

Syntax explained

restore PASS
Proves an independently stored artifact can create a usable dataset, not that Sentinel protected it.
Command
Fill variables0/4 ready

Values stay on this page and are never sent or saved.

redis-cli --tls --cacert {{redisTlsCa}} -h {{sentinelThreeHost}} -p {{sentinelPort}} --user sentinel-operator --askpass SENTINEL CKQUORUM {{masterName}}
Example output / evidence
Enter password: [hidden]
HA acceptance
CKQUORUM: PASS on 3/3
Known replicas: 2
Planned failover RTO: 00:00:02.8
Canary loss: 0 in this test (not a zero-loss guarantee)
One-Sentinel loss: PASS
Minority no-failover: PASS
Independent backup restore: PASS
Backup RPO: 00:04:32
Open alerts: 0

Checkpoint: Verify this layer before continuing

Continue whenAll HA and backup gates pass, monitoring has owners and evidence explicitly states asynchronous write-loss limitations.

Stop whenStop release when failover passes but backup restore fails, or when data safety claims exceed measured semantics.

If this step fails

Failover passes repeatedly but backup age exceeds RPO.

Likely causeAvailability testing is operated while backup scheduling, transfer or monitoring has failed.

Safe checks
  • Review repository object time/digest and Redis persistence health.

ResolutionTreat the service as unrecoverable from logical loss, repair backup pipeline and repeat an isolated restore before acceptance.

Security notes

  • Sanitize event logs and backup reports; they contain topology and potentially sensitive key metadata.

Alternatives

  • Use managed multi-zone Redis with documented backup restore if operational staffing cannot sustain these drills.

Stop conditions

  • Stop release when failover passes but backup restore fails, or when data safety claims exceed measured semantics.

Finish line

Verification checklist

Quorum and majorityredis-cli --tls --cacert {{redisTlsCa}} -h {{sentinelOneHost}} -p {{sentinelPort}} --user sentinel-operator --askpass SENTINEL CKQUORUM {{masterName}}All three Sentinels report usable quorum and failover authorization.
Topology agreementredis-cli --tls --cacert {{redisTlsCa}} -h {{sentinelTwoHost}} -p {{sentinelPort}} --user sentinel-operator --askpass SENTINEL MASTER {{masterName}}All Sentinels agree on primary, epoch, two replicas and two other Sentinels.
Role convergenceredis-cli --tls --cacert {{redisTlsCa}} -h <current-primary> ROLEExactly one primary exists, all other data nodes follow it and the old primary is fenced from writes.
Client recoveryredis-cli --tls --cacert {{redisTlsCa}} -h {{sentinelThreeHost}} -p {{sentinelPort}} --user sentinel-operator --askpass SENTINEL GET-MASTER-ADDR-BY-NAME {{masterName}}Clients discover the promoted node, reconnect inside SLO and canary/application behavior has no unexplained loss or duplication.
Independent recoveryredis-check-rdb <restored-backup-copy>A separately stored backup validates and restores application semantics; evidence does not claim Sentinel is data protection.

Recovery guidance

Common problems and safe checks

CKQUORUM reports NOQUORUM.

Likely causeNot enough Sentinels are reachable for configured quorum or majority.

Safe checks
  • SENTINEL CKQUORUM {{masterName}}
  • SENTINEL SENTINELS {{masterName}}

ResolutionRestore independent Sentinel communication and membership; never lower quorum during an unexplained partition.

Only two Sentinels discover each other in a three-node design.

Likely causeFirewall, TLS, announce address, DNS or Sentinel authentication blocks hello exchange.

Safe checks
  • SENTINEL SENTINELS {{masterName}}
  • journalctl -u redis-sentinel -n 100

ResolutionCorrect reachability and identity, wait for convergence and require all three to report two peers.

Sentinel marks primary SDOWN but never ODOWN.

Likely causeLocal failure detection reached threshold but fewer than quorum Sentinels agree.

Safe checks
  • SENTINEL MASTER {{masterName}}
  • SENTINEL CKQUORUM {{masterName}}

ResolutionDiagnose asymmetric network and primary health; do not force failover until majority view is understood.

Primary is ODOWN but failover does not start.

Likely causeQuorum was reached but no majority can elect a leader, or no suitable replica exists.

Safe checks
  • SENTINEL CKQUORUM {{masterName}}
  • SENTINEL REPLICAS {{masterName}}

ResolutionRestore majority or a healthy promotable replica; do not create a competing primary manually in the minority.

Sentinel config changes disappear after restart.

Likely causeConfig is read-only, ephemeral, overwritten from a template or FLUSHCONFIG cannot persist.

Safe checks
  • SENTINEL FLUSHCONFIG
  • namei -l {{sentinelConfig}}

ResolutionProvide a durable writable protected runtime copy and keep desired bootstrap plus discovered state lifecycle explicit.

Sentinel cannot authenticate to primary.

Likely causeauth-user/auth-pass mismatch, ACL lacks monitoring commands or protected config rendered stale credentials.

Safe checks
  • SENTINEL MASTER {{masterName}}
  • ACL LOG 20

ResolutionRotate/render the exact Sentinel user on every node and grant only official required Sentinel ACL rules.

Replica link is down with NOAUTH.

Likely causemasteruser/masterauth does not match replication ACL user after rotation.

Safe checks
  • INFO replication
  • ACL LOG 20

ResolutionUse a dual-credential rotation window, update replicas, verify link_up, then revoke old credential.

TLS handshake fails only after promotion.

Likely causePromoted replica certificate SAN, CA chain or client hostname differs from primary expectations.

Safe checks
  • openssl s_client -connect {{replicaOneHost}}:{{redisPort}} -servername {{replicaOneHost}} -CAfile {{redisTlsCa}}

ResolutionIssue uniform valid node certificates and test every candidate before enabling promotion.

Client remains connected to old primary.

Likely causeClient hard-codes address, caches discovery forever or ignores switch/reconnect behavior.

Safe checks
  • SENTINEL GET-MASTER-ADDR-BY-NAME {{masterName}}
  • CLIENT LIST

ResolutionFix Sentinel-aware discovery, connection pool invalidation and bounded retry; fence direct old-primary routes.

Client cannot discover primary when one Sentinel is down.

Likely causeOnly one Sentinel endpoint is configured or client stops after first failure.

Safe checks
  • Review client Sentinel seed configuration.

ResolutionConfigure at least two, preferably all three, independent Sentinel seeds and test startup with each one unavailable.

Promoted replica is missing recent writes.

Likely causeAsynchronous replication lagged when the old primary failed.

Safe checks
  • Compare captured replication offsets and canary sequence.
  • INFO replication

ResolutionQuantify RPO breach, recover from application log/backup if possible and revise min-replica/WAIT or storage architecture.

Primary rejects writes before any failover.

Likely causemin-replicas-to-write is enabled and not enough replicas meet max lag.

Safe checks
  • INFO replication
  • CONFIG GET min-replicas-*

ResolutionRestore replica health; do not disable the data-safety gate unless availability owner explicitly accepts greater loss.

No replica is selected for promotion.

Likely causeAll replicas disconnected too long, have priority zero, stale offsets, errors or Sentinel cannot authenticate.

Safe checks
  • SENTINEL REPLICAS {{masterName}}
  • INFO replication

ResolutionRepair replica eligibility and synchronization before another failover; never promote an unknown stale copy.

Old primary returns as another writable primary.

Likely causeNetwork partition healed without effective fencing/client discovery or manual configuration conflicts.

Safe checks
  • ROLE on every node
  • SENTINEL MASTER {{masterName}}

ResolutionRemove client access to the old node, preserve divergent evidence, reconfigure it as replica only after data-loss analysis.

Sentinels disagree on primary after failover.

Likely causeConfiguration propagation, epochs, network or duplicate Sentinel identity/address is broken.

Safe checks
  • SENTINEL MASTER {{masterName}} on every Sentinel
  • SENTINEL SENTINELS {{masterName}}

ResolutionStop client writes if split routing exists, restore majority connectivity and converge on the highest valid epoch.

Failover takes longer than SLO.

Likely causedown-after, leader election, DNS, promotion, replica synchronization or client reconnect dominates.

Safe checks
  • Inspect Sentinel event timeline and application connection metrics.

ResolutionMeasure each phase under load and tune only with false-positive, replication and retry analysis.

Sentinel performs repeated failovers.

Likely causeFlapping network, aggressive down-after, overloaded primary or misannounced addresses causes instability.

Safe checks
  • SENTINEL MASTER {{masterName}}
  • LATENCY DOCTOR
  • Sentinel event logs

ResolutionStabilize network/host and increase thresholds based on evidence; do not mask an overloaded data node.

Replica serves stale reads after failover.

Likely causeApplications intentionally read replicas without bounded staleness or route caches did not update.

Safe checks
  • ROLE
  • INFO replication
  • Inspect client read preference.

ResolutionRoute consistency-sensitive reads to discovered primary or implement explicit staleness semantics and monitoring.

Planned FAILOVER returns NOGOODSLAVE.

Likely causeNo healthy sufficiently current promotable replica is known.

Safe checks
  • SENTINEL REPLICAS {{masterName}}

ResolutionCancel the drill, restore replication and persistence health, then retry after offsets converge.

Backup restore succeeds but Sentinel dataset contains prior bad writes.

Likely causeReplication and persistence copied logical corruption before backup selection.

Safe checks
  • Compare backup timestamp with incident timeline and application audit.

ResolutionSelect a known-good older backup, restore in isolation and replay only validated application changes.

After the procedure

Alternatives and next steps

Consider these alternatives

  • Use a managed Redis service with documented multi-zone failover and backup when the team cannot operate quorum, clients and fencing.
  • Use Redis Cluster for horizontal sharding and cluster-aware clients; do not layer Sentinel onto Redis Cluster.
  • Use an external orchestrator only when it provides equivalent majority, fencing, client discovery and failure-domain guarantees.
  • Choose manual failover when the environment cannot provide three independent Sentinels or safe automatic fencing.
  • Use stronger-consistency storage for data that cannot tolerate loss from asynchronous Redis replication.

Operate it safely

  • Automate topology configuration and verify every Sentinel independently through CI and monitoring.
  • Run quarterly planned failover plus annual failure-domain isolation while capturing offsets, epochs, application errors and RPO/RTO.
  • Test certificate and ACL rotation across primary, replicas, Sentinels and clients without disabling TLS or leaving shared credentials.
  • Add alerts for CKQUORUM, known Sentinel/replica counts, SDOWN/ODOWN, role change, replication offset/lag, rejected writes and backup age.
  • Document when Redis asynchronous semantics are unacceptable and migrate that data path to a stronger-consistency store.

Recovery

Rollback

There is no safe generic command that reverses a failover. Stabilize on the highest valid epoch, fence extra writers, preserve both datasets, restore majority, and rejoin nodes as replicas only after divergence analysis. Use backup restore when the promoted history is not acceptable.

  1. Stop or drain application writes when more than one writable node, epoch disagreement or canary regression is detected.
  2. Preserve Sentinel logs/configs, ROLE/INFO replication, offsets, epochs, client errors and persistence files from every node.
  3. Identify the majority-approved primary with the highest valid epoch; do not manually promote another node in a minority partition.
  4. Fence every other writable endpoint at client, network and process layers while preserving its data read-only.
  5. If the accepted primary dataset is valid, restore replication connectivity and rebuild old nodes as replicas one at a time.
  6. If required writes are missing or divergent, restore a known-good backup in isolation and reconcile only validated application events.
  7. Repeat CKQUORUM, role/epoch, client discovery, canary, one-Sentinel-loss and backup restore tests before resuming normal traffic.

Evidence

Sources and review

Verified 2026-07-24Review due 2026-10-22
Redis Sentinel high availabilityofficialRedis replicationofficialRedis ACLofficialRedis TLSofficialRedis persistenceofficialRedis securityofficialRedis administrationofficialRedis Open Source 8.8 release notesofficial