Build a MongoDB replica set and verify failover safely
Harden an existing three-member MongoDB 8.2 replica set, prove majority durability and client discovery, perform a planned election and bounded secondary outage, and reconcile real application-level evidence.
Demonstrate that MongoDB and its clients recover from one member failure inside an approved SLO without bypassing TLS, forcing unsafe quorum changes, losing acknowledged canaries, or confusing replication with backup.
- MongoDB Community Server 8.2.x
- Linux Supported MongoDB 8.2 production platforms
- Healthy three-member set Three independent voting data-bearing members with one PRIMARY, two SECONDARY members, stable private DNS, synchronized time, and no active maintenance.
mongosh '{{mongoUri}}' --eval 'rs.status()' - Authenticated TLS Member-to-member X.509 authentication and client TLS hostname validation are enforced; certificate rotation is monitored.
- Application readiness A supported driver, multi-host URI, retry/idempotency design, synthetic transaction, and correlated telemetry are available.
- Recovery and approval A current protected backup, tested restore runbook, out-of-band access, change window, incident owner, and abort thresholds are approved.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A production-ready failover test for a three-member, TLS/X.509-protected MongoDB 8.2 replica set.
- A measurable application recovery contract covering majority writes, topology discovery, retries, idempotency, and post-election reconciliation.
- A planned primary stepdown elects a different primary within the approved SLO without missing or duplicating canary operations.
- One secondary can leave and rejoin while the set stays writable, then returns to a fully converged three-member state.
- Operators can distinguish safe transient election errors from ambiguous writes, rollback, quorum loss, and client misconfiguration.
Architecture
How the parts fit together
Three independent voting data-bearing members communicate over mutually authenticated TLS. Drivers connect through a multi-host replica-set URI, validate the CA and hostnames, discover the current primary, and issue majority-aware idempotent writes. Monitoring correlates server terms, member states, lag, host saturation, and application outcomes.
- An application write reaches the current primary and is durably acknowledged by the calculated majority.
- The primary steps down; voting members exchange heartbeats and elect a caught-up eligible secondary in a higher term.
- The driver discovers the new primary and retries only operations whose semantics permit a safe retry.
- The former primary and any restarted secondary catch up to the winning oplog history before redundancy is declared restored.
Assumptions
- MongoDB Community Server 8.2 is already installed as a three-member self-managed replica set on independent supported hosts.
- TLS certificates, an internal X.509 member identity, SCRAM administrative credentials, private DNS, monitoring, and out-of-band host access already exist.
- The application uses a current supported MongoDB driver and exposes synthetic transaction, retry, and error-rate telemetry.
- A current protected backup and separately proven restore procedure exist; this guide tests availability and does not replace recovery testing.
Key concepts
- Election term
- A monotonically increasing epoch identifying the replica-set leadership generation.
- Voting majority
- More than half of configured voting members; a three-voter set needs two reachable voters.
- Write concern
- The requested durability acknowledgement; a timeout can leave the operation's final outcome ambiguous.
- Oplog window
- The span of operation history available for secondaries to catch up without initial sync.
- Retryable write
- A driver/server protocol for retrying supported writes; it is not a substitute for business idempotency.
- Rollback
- Removal of operations that are not in the winning history after a divergent former primary rejoins.
Before you copy
Values used in this guide
{{mongoUri}}secretSecret-free reference to the multi-host authenticated URI; load credentials from the approved secret mechanism.
Example: mongodb://ops@mongo-a.example.net,mongo-b.example.net,mongo-c.example.net/admin?replicaSet=rs0&tls=true&retryWrites=true{{mongoPrimaryUri}}secretAdministrative connection URI pinned to the confirmed current primary only for the planned stepdown.
Example: mongodb://ops@mongo-a.example.net/admin?directConnection=true&tls=true{{mongoA}}First advertised member DNS name.
Example: mongo-a.example.net{{mongoB}}Second advertised member DNS name.
Example: mongo-b.example.net{{mongoC}}Third advertised member DNS name.
Example: mongo-c.example.net{{mongoPort}}MongoDB TLS listener port.
Example: 27017{{caFile}}Local trusted CA bundle used to validate member certificates.
Example: /etc/ssl/certs/database-internal-ca.pem{{nodeCertificate}}Node public certificate inspected for SAN and lifetime.
Example: /etc/mongodb/tls/mongo-a.pem{{exerciseId}}Globally unique approved identifier used for canary writes and cleanup.
Example: chg-20260728-047Security and production boundaries
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Stop before continuing if
- Stop before a disruptive test when monitoring, a named incident commander, a current backup, or a tested recovery channel is missing.
- Stop when nodes disagree about identity, time, version, topology, or trust anchors; do not use force, trust bypasses, or quorum overrides to conceal the inconsistency.
- Stop when the application does not implement the retry, idempotency, timeout, and connection-discovery behavior required by the tested database.
- Stop after any unexplained data divergence, failed integrity check, authentication bypass, certificate validation failure, or loss of a majority.
verification
Record version, feature compatibility, and topology
Connect through the authenticated replica-set URI, record the exact server release and FCV, and enumerate every member before changing availability state.
Why this step matters
A failover exercise is meaningful only when the operator knows the actual binary, FCV, member identities, vote count, and priority policy. FCV can intentionally trail the binary during upgrade burn-in.
What to understand
Three voting, data-bearing members tolerate one member loss while preserving a majority of two; an arbiter does not store user data.
Use stable DNS names advertised by the replica-set configuration. Modern MongoDB rejects configurations that rely only on IP addresses in important deployment paths.
System changes
- This read-only baseline writes no database state; preserve the redacted output with the change record.
Syntax explained
buildInfo / featureCompatibilityVersion- Separates installed binary behavior from enabled on-disk feature semantics.
db.hello()- Reports the connection's current primary/secondary view and replica-set name.
Values stay on this page and are never sent or saved.
mongosh '{{mongoUri}}' --quiet --eval 'const b=db.getSiblingDB("admin").runCommand({buildInfo:1}); const f=db.getSiblingDB("admin").runCommand({getParameter:1,featureCompatibilityVersion:1}); printjson({version:b.version,fcv:f.featureCompatibilityVersion,hello:db.hello(),members:rs.conf().members.map(m=>({host:m.host,votes:m.votes,priority:m.priority}))})'{ version: "8.2.11", fcv: { version: "8.2" }, hello: { setName: "rs0", isWritablePrimary: true }, members: [ { host: "mongo-a.example.net:27017", votes: 1, priority: 2 }, { host: "mongo-b.example.net:27017", votes: 1, priority: 1 }, { host: "mongo-c.example.net:27017", votes: 1, priority: 1 } ] }Checkpoint: Checkpoint: Record version, feature compatibility, and topology
Continue whenAll three intended DNS members appear once, use the same supported release line, and exactly one reports writable primary.
Stop whenFCV, binary version, member count, DNS identity, votes, or priorities differ from the approved topology.
If this step fails
The set repeatedly elects a new primary during normal load.
Likely causeHeartbeat loss, time or DNS instability, host pressure, long storage stalls, process restarts, or poorly planned member priorities are destabilizing elections.
rs.status()db.adminCommand({replSetGetConfig: 1, commitmentStatus: true})db.adminCommand({serverStatus: 1}).connections
ResolutionStabilize the network, storage, time, and host resources before changing election settings. Do not mask recurring failures by lengthening timeouts without evidence.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
verification
Prove independent hosts, DNS, time, and capacity
From each database host, resolve every peer, check clock synchronization, free space, service state, and recent restart evidence. A three-process deployment on one failure domain is not high availability.
Why this step matters
Elections depend on timely heartbeats, stable names, responsive storage, and independent failure domains. A planned stepdown can expose a hidden shared dependency that normal steady-state traffic does not reveal.
What to understand
Measure disk latency and saturation with the normal observability stack as well as free capacity; sufficient free bytes do not prove healthy storage.
Confirm each member has independent compute and storage and that security policy permits member-to-member transport in both directions.
System changes
- No state changes; the commands read resolver, time, filesystem, and service status.
Syntax explained
NTPSynchronized- Confirms the host considers its clock synchronized; still review monitoring for offset and source quality.
df -h /var/lib/mongodb- Checks the filesystem that actually contains dbPath, not an unrelated root filesystem.
Values stay on this page and are never sent or saved.
getent hosts {{mongoA}} {{mongoB}} {{mongoC}}; timedatectl show -p NTPSynchronized --value; df -h /var/lib/mongodb; systemctl is-active mongod10.20.0.11 mongo-a.example.net 10.20.0.12 mongo-b.example.net 10.20.0.13 mongo-c.example.net yes /dev/nvme0n1p2 250G 83G 167G 34% /var/lib/mongodb active
Checkpoint: Checkpoint: Prove independent hosts, DNS, time, and capacity
Continue whenEvery hostname resolves consistently, NTP is synchronized, storage headroom meets policy, and mongod is active on all nodes.
Stop whenDNS differs by host, a clock is unsynchronized, a filesystem is pressured, or any member already has an unexplained restart or alert.
If this step fails
A member remains STARTUP2 or RECOVERING and never becomes SECONDARY.
Likely causeInitial sync is still copying data, the source is unavailable, the oplog window is too short, storage is too slow, or member authentication fails.
rs.status()db.getSiblingDB('admin').runCommand({replSetGetStatus: 1, initialSync: 1})journalctl -u mongod -n 200 --no-pager
ResolutionCorrect connectivity, capacity, TLS, or X.509 identity first. If initial sync cannot catch up, provision a clean data path and resync only after preserving diagnostic evidence.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
verification
Verify TLS identity and member authentication
Validate the CA chain and DNS SAN for every advertised member, then confirm the running process requires TLS and X.509 cluster authentication rather than trusting an open network.
Why this step matters
Replication availability is not a security boundary unless peers authenticate one another and clients validate the server identity. A successful encrypted connection with an invalid hostname is still vulnerable to impersonation.
What to understand
Certificates need SANs for the exact names in rs.conf; the certificate lifetime must exceed the exercise and a monitored rotation plan must exist.
X.509 member identity is preferable to a shared keyfile for production because it supports stronger identity and lifecycle management.
System changes
- The verification reads certificates and performs TLS handshakes; it does not modify certificate stores or MongoDB.
Syntax explained
-servername- Sends SNI and makes the observed certificate comparable to the intended DNS identity.
-verify_return_error- Terminates verification on chain errors instead of printing an error and continuing.
Values stay on this page and are never sent or saved.
for host in {{mongoA}} {{mongoB}} {{mongoC}}; do openssl s_client -connect "$host:{{mongoPort}}" -servername "$host" -CAfile {{caFile}} -verify_return_error </dev/null 2>/dev/null | grep -E 'subject=|issuer=|Verify return code'; donesubject=CN=mongo-a.example.net issuer=CN=Database Internal CA Verify return code: 0 (ok) subject=CN=mongo-b.example.net issuer=CN=Database Internal CA Verify return code: 0 (ok) subject=CN=mongo-c.example.net issuer=CN=Database Internal CA Verify return code: 0 (ok)
Checkpoint: Checkpoint: Verify TLS identity and member authentication
Continue whenEvery member returns code 0 with the intended DNS identity and approved CA.
Stop whenAny client requires invalid-certificate or invalid-hostname bypass, a certificate is near expiry, or the member certificate/private key permissions are unsafe.
If this step fails
A TLS connection works only with hostname validation disabled.
Likely causeThe certificate SAN does not contain the advertised replica-set hostname, the CA chain is wrong, or clients connect through an undocumented alias.
openssl s_client -connect {{mongoA}}:{{mongoPort}} -servername {{mongoA}} -CAfile {{caFile}} </dev/nullopenssl x509 -in {{nodeCertificate}} -noout -subject -issuer -ext subjectAltName
ResolutionIssue a certificate with the correct DNS SAN and distribute the trusted CA. Never keep allowInvalidHostnames or allowInvalidCertificates as a production workaround.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
verification
Establish replica health and replication lag
Capture state, optime, heartbeat messages, election term, and lag while the workload is healthy. The baseline becomes the comparison point after election.
Why this step matters
A deliberate election should start from an unambiguous healthy state. Testing while a secondary is lagged or recovering converts a controlled exercise into an uncontrolled availability event.
What to understand
Record the election term and primary so that the post-test evidence proves a real election rather than a client reconnect to the same member.
Replication lag must be evaluated against the workload's RPO and oplog window, not only against a generic zero-second sample.
System changes
- No database changes; diagnostic commands read replica-set state and optimes.
Syntax explained
stateStr / health- Human-readable member state and heartbeat health.
optimeDate- Last applied replicated operation time used to compare member progress.
Values stay on this page and are never sent or saved.
mongosh '{{mongoUri}}' --quiet --eval 'const s=rs.status(); printjson({set:s.set,term:s.term,members:s.members.map(m=>({name:m.name,state:m.stateStr,health:m.health,optimeDate:m.optimeDate,lastHeartbeatMessage:m.lastHeartbeatMessage||""}))}); rs.printSecondaryReplicationInfo()'{ set: "rs0", term: Long("42"), members: [ { name: "mongo-a.example.net:27017", state: "PRIMARY", health: 1 }, { name: "mongo-b.example.net:27017", state: "SECONDARY", health: 1 }, { name: "mongo-c.example.net:27017", state: "SECONDARY", health: 1 } ] }
source: mongo-b.example.net:27017
0 secs (0 hrs) behind the primaryCheckpoint: Checkpoint: Establish replica health and replication lag
Continue whenOne PRIMARY and two healthy SECONDARY members have negligible lag and no heartbeat error.
Stop whenAny member is STARTUP2, RECOVERING, ROLLBACK, UNKNOWN, unhealthy, or outside the approved lag threshold.
If this step fails
A member remains STARTUP2 or RECOVERING and never becomes SECONDARY.
Likely causeInitial sync is still copying data, the source is unavailable, the oplog window is too short, storage is too slow, or member authentication fails.
rs.status()db.getSiblingDB('admin').runCommand({replSetGetStatus: 1, initialSync: 1})journalctl -u mongod -n 200 --no-pager
ResolutionCorrect connectivity, capacity, TLS, or X.509 identity first. If initial sync cannot catch up, provision a clean data path and resync only after preserving diagnostic evidence.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
verification
Confirm oplog coverage and majority durability
Measure the oplog time window and inspect the default read/write concern. The oplog must cover the longest credible outage plus repair time.
Why this step matters
Failover can preserve availability while a returning member still needs sufficient history to catch up. Majority acknowledgement reduces rollback exposure but adds a latency and availability trade-off that must be explicit.
What to understand
A write-concern timeout is an ambiguous outcome: the write may exist even when the requested acknowledgement did not arrive.
Increasing oplog size changes storage allocation and should follow measured write volume, recovery duration, and disk headroom.
System changes
- This step reads oplog metadata and cluster-wide default concerns; it does not change either.
Syntax explained
w: majority- Requires durable acknowledgement by the calculated majority of voting data-bearing members.
wtimeout- Bounds acknowledgement wait but does not cancel or roll back an already applied write.
Values stay on this page and are never sent or saved.
mongosh '{{mongoUri}}' --quiet --eval 'printjson(db.getSiblingDB("local").oplog.rs.stats()); printjson(db.getSiblingDB("admin").runCommand({getDefaultRWConcern:1}))'{ ns: "local.oplog.rs", size: 21474836480, maxSize: 21474836480 }
{ defaultReadConcern: { level: "majority" }, defaultWriteConcern: { w: "majority", wtimeout: 5000 }, ok: 1 }Checkpoint: Checkpoint: Confirm oplog coverage and majority durability
Continue whenThe oplog window exceeds the approved recovery horizon and the application uses documented majority-aware durability.
Stop whenThe oplog is undersized, defaults are unknown, or the application assumes a timeout proves that no write occurred.
If this step fails
A write returns a write-concern timeout during or after failover.
Likely causeThe primary accepted the write but the requested majority did not acknowledge before the deadline; the outcome is ambiguous rather than safely failed.
db.getSiblingDB('admin').runCommand({getDefaultRWConcern: 1})rs.printSecondaryReplicationInfo()Query the operation's idempotency key after a primary is stable.
ResolutionReconcile the business operation by its idempotency key before retrying. Restore majority health and treat blind replay as a duplication risk.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
verification
Validate the application discovery and retry contract
Inspect the effective client configuration without exposing credentials. It must name multiple seed hosts, the exact replica set, TLS validation, bounded server selection, and supported retry behavior.
Why this step matters
Server redundancy does not produce application availability when a client pins one host or treats a transient election as a permanent error. Driver behavior belongs in the acceptance test.
What to understand
Use a current MongoDB-supported driver and a seed list; DNS or load-balancer behavior must not hide replica-set discovery.
Retryable writes cannot make arbitrary business workflows idempotent. Use unique request identifiers and reconcile unknown outcomes.
System changes
- The command opens an authenticated client connection and reads topology; no data is changed.
Syntax explained
replicaSet=rs0- Tells the driver to discover and enforce membership in the intended replica set.
retryWrites=true- Allows supported single-document writes to be retried under documented driver/server rules.
Values stay on this page and are never sent or saved.
mongosh '{{mongoUri}}' --quiet --eval 'const h=db.hello(); printjson({setName:h.setName,primary:h.primary,hosts:h.hosts,isWritablePrimary:h.isWritablePrimary})'{ setName: "rs0", primary: "mongo-a.example.net:27017", hosts: [ "mongo-a.example.net:27017", "mongo-b.example.net:27017", "mongo-c.example.net:27017" ], isWritablePrimary: true }Checkpoint: Checkpoint: Validate the application discovery and retry contract
Continue whenThe client sees all three members, validates TLS, discovers the primary, and has bounded retry/timeout behavior.
Stop whenOnly one seed is configured, the set name is missing, certificate checks are bypassed, or retries are unbounded/non-idempotent.
If this step fails
Clients continue sending writes to the old primary.
Likely causeThe connection string lists one host, omits replicaSet, uses stale DNS, or the driver lacks retryable-write and server-selection handling.
Inspect the effective connection string without printing its password.Check driver topology events and serverSelectionTimeoutMS failures.mongosh '{{mongoUri}}' --eval 'db.hello()'
ResolutionUse a seed list containing multiple members and the exact replica-set name, then verify the supported driver rediscovers the new primary.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
command
Create a majority-acknowledged canary
Write a uniquely identified synthetic record before the election and read it with majority concern. Use a dedicated operational database with retention, not an application collection.
Why this step matters
A canary gives the exercise a concrete durability assertion. It proves more than process health while avoiding customer data and ambiguous ad-hoc test writes.
What to understand
The unique _id makes a retry observable and prevents silent duplicate creation.
Grant the exercise identity only insert/find/remove access to the dedicated validation collection; do not use a cluster administrator from application hosts.
System changes
- Creates one synthetic document in ops_validation.failover_canary, replicated with majority write concern.
Syntax explained
w: majority- Waits for durable acknowledgement by a calculated majority.
wtimeout: 5000- Bounds the wait while preserving the possibility of an ambiguous completed write.
Values stay on this page and are never sent or saved.
mongosh '{{mongoUri}}' --quiet --eval 'const id="{{exerciseId}}-before"; const c=db.getSiblingDB("ops_validation").failover_canary; printjson(c.insertOne({_id:id,phase:"before",createdAt:new Date()},{writeConcern:{w:"majority",wtimeout:5000}})); printjson(c.findOne({_id:id})._id)'{ acknowledged: true, insertedId: "chg-20260728-047-before" }
chg-20260728-047-beforeCheckpoint: Checkpoint: Create a majority-acknowledged canary
Continue whenThe insert is acknowledged and the exact exercise ID is readable before stepdown.
Stop whenThe insert times out, duplicates an existing exercise ID, or cannot be read back with the approved concern.
If this step fails
A write returns a write-concern timeout during or after failover.
Likely causeThe primary accepted the write but the requested majority did not acknowledge before the deadline; the outcome is ambiguous rather than safely failed.
db.getSiblingDB('admin').runCommand({getDefaultRWConcern: 1})rs.printSecondaryReplicationInfo()Query the operation's idempotency key after a primary is stable.
ResolutionReconcile the business operation by its idempotency key before retrying. Restore majority health and treat blind replay as a duplication risk.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Use an existing application synthetic transaction only when it has an explicit owner, idempotency key, and cleanup policy.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
decision
Approve the fault, success criteria, and abort owner
Name the current primary, intended candidate set, test duration, application SLO, maximum error budget, monitoring panels, and the person authorized to abort. Do not improvise a failure injection.
Why this step matters
A failover test changes production availability and must have measurable acceptance criteria. An explicit abort owner prevents competing operators from issuing contradictory recovery actions.
What to understand
Prefer a planned primary stepdown before testing process or host loss; it exercises election without corrupting files or power state.
Observe application error rate, driver topology events, connection pool recovery, write latency, replication lag, and database logs on a synchronized timeline.
System changes
- Creates an approved operational decision and observation plan; it does not yet change the replica set.
Syntax explained
rs.stepDown(60)- Makes the current primary ineligible to become primary again for the requested period when a safe stepdown is possible.
Exercise chg-20260728-047 approved: primary=mongo-a, action=rs.stepDown(60), max write interruption=15s, abort owner=DBA on-call, customer traffic=synthetic canary only.
Checkpoint: Checkpoint: Approve the fault, success criteria, and abort owner
Continue whenThe window, success/abort thresholds, observers, recovery access, and synthetic workload are explicitly approved.
Stop whenNo single abort owner exists, production health is degraded, or the team cannot observe application and database behavior together.
If this step fails
rs.stepDown reports that no electable secondaries are caught up.
Likely causeReplication lag, maintenance mode, priority zero, hidden members, or unhealthy secondaries leave no safe election candidate.
rs.status()rs.printSecondaryReplicationInfo()rs.conf().members.map(m => ({host:m.host, priority:m.priority, votes:m.votes, hidden:m.hidden}))
ResolutionCancel the exercise, restore an electable synchronized secondary, and repeat the preflight. Do not force stepdown or reconfigure votes to manufacture availability.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
command
Step down the primary and measure election time
From the confirmed current primary, request a bounded stepdown. The shell connection may close; that is expected. Do not use force when no caught-up electable secondary exists.
Why this step matters
A controlled stepdown exercises the election protocol and client topology recovery while leaving the former primary process running and available as a secondary.
What to understand
A disconnected shell or not-primary response is normal because the command changes the role serving that connection.
The requested 60 seconds is a re-election exclusion period for the former primary, not an expected outage duration.
System changes
- The current primary relinquishes PRIMARY and triggers a replica-set election, temporarily interrupting writes.
Syntax explained
60- Seconds the stepped-down member should remain ineligible for primary election.
no force argument- Requires a safe, sufficiently caught-up election path instead of accepting avoidable rollback risk.
Values stay on this page and are never sent or saved.
date -u +%FT%TZ; mongosh '{{mongoPrimaryUri}}' --quiet --eval 'printjson(rs.stepDown(60))'; date -u +%FT%TZ2026-07-28T10:15:00Z MongoServerError: not primary and secondaryOk=false 2026-07-28T10:15:01Z
Checkpoint: Checkpoint: Step down the primary and measure election time
mongosh '{{mongoUri}}' --quiet --eval 'printjson(db.hello())'Continue whenA different member becomes PRIMARY within the approved SLO and the former primary becomes SECONDARY.
Stop whenNo new primary appears by the abort threshold, more than one member becomes unreachable, or application errors exceed the budget.
If this step fails
rs.stepDown reports that no electable secondaries are caught up.
Likely causeReplication lag, maintenance mode, priority zero, hidden members, or unhealthy secondaries leave no safe election candidate.
rs.status()rs.printSecondaryReplicationInfo()rs.conf().members.map(m => ({host:m.host, priority:m.priority, votes:m.votes, hidden:m.hidden}))
ResolutionCancel the exercise, restore an electable synchronized secondary, and repeat the preflight. Do not force stepdown or reconfigure votes to manufacture availability.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Run the same stepdown against a staging topology with production-like latency before the production exercise.
Stop conditions
- Stop before a disruptive test when monitoring, a named incident commander, a current backup, or a tested recovery channel is missing.
- Stop when nodes disagree about identity, time, version, topology, or trust anchors; do not use force, trust bypasses, or quorum overrides to conceal the inconsistency.
- Stop when the application does not implement the retry, idempotency, timeout, and connection-discovery behavior required by the tested database.
- Stop after any unexplained data divergence, failed integrity check, authentication bypass, certificate validation failure, or loss of a majority.
verification
Prove the new primary and stable membership
Reconnect through the multi-host URI, capture the new election term and member states, and compare them with the baseline instead of trusting a single green service indicator.
Why this step matters
The election is complete only when the cluster converges on one writable primary and all members agree on the new term. Process liveness alone cannot establish that.
What to understand
Capture timestamps from the application and server to calculate observed unavailability rather than relying on human perception.
Review logs for unexpected elections, rollback, slow storage, authentication errors, and repeated client discovery.
System changes
- No state change; this captures authoritative post-election evidence.
Syntax explained
term- Monotonically identifies election epochs and should advance after a real election.
stateStr- Shows the role each member currently reports.
Values stay on this page and are never sent or saved.
mongosh '{{mongoUri}}' --quiet --eval 'const s=rs.status(); printjson({term:s.term,electionCandidateMetrics:s.electionCandidateMetrics,members:s.members.map(m=>({name:m.name,state:m.stateStr,health:m.health}))})'{ term: Long("43"), electionCandidateMetrics: { lastElectionReason: "stepUpRequestSkipDryRun" }, members: [ { name: "mongo-a.example.net:27017", state: "SECONDARY", health: 1 }, { name: "mongo-b.example.net:27017", state: "PRIMARY", health: 1 }, { name: "mongo-c.example.net:27017", state: "SECONDARY", health: 1 } ] }Checkpoint: Checkpoint: Prove the new primary and stable membership
Continue whenThe term advanced, exactly one different member is PRIMARY, and all three members are healthy.
Stop whenThe topology oscillates, a member reports ROLLBACK/UNKNOWN, or heartbeats remain unhealthy.
If this step fails
The set repeatedly elects a new primary during normal load.
Likely causeHeartbeat loss, time or DNS instability, host pressure, long storage stalls, process restarts, or poorly planned member priorities are destabilizing elections.
rs.status()db.adminCommand({replSetGetConfig: 1, commitmentStatus: true})db.adminCommand({serverStatus: 1}).connections
ResolutionStabilize the network, storage, time, and host resources before changing election settings. Do not mask recurring failures by lengthening timeouts without evidence.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
command
Verify writes, reads, and idempotency after election
Write a second majority canary through the normal multi-host URI, then read both before and after records and verify the application synthetic transaction exactly once.
Why this step matters
A new primary that answers health checks may still reject writes, use stale client routing, or expose application retry defects. An end-to-end synthetic operation verifies the user-visible path.
What to understand
Search for both deterministic IDs to detect missing or duplicated outcomes around the election boundary.
Verify application-level invariants as well as database acknowledgement; a successful insert does not prove downstream processing.
System changes
- Creates one additional synthetic canary document in the dedicated validation collection.
Syntax explained
sort({_id:1})- Produces deterministic evidence containing both expected phase identifiers.
Values stay on this page and are never sent or saved.
mongosh '{{mongoUri}}' --quiet --eval 'const c=db.getSiblingDB("ops_validation").failover_canary; printjson(c.insertOne({_id:"{{exerciseId}}-after",phase:"after",createdAt:new Date()},{writeConcern:{w:"majority",wtimeout:5000}})); printjson(c.find({_id:/^{{exerciseId}}-/}).sort({_id:1}).toArray())'{ acknowledged: true, insertedId: "chg-20260728-047-after" }
[ { _id: "chg-20260728-047-after", phase: "after" }, { _id: "chg-20260728-047-before", phase: "before" } ]Checkpoint: Checkpoint: Verify writes, reads, and idempotency after election
Continue whenExactly one before and one after record exist, the new write has majority acknowledgement, and the application SLO has recovered.
Stop whenA record is missing/duplicated, writes still fail, or retry activity violates business idempotency.
If this step fails
The application reconnects but reports duplicate operations.
Likely causeIt retried an operation whose original outcome was ambiguous and the write was not naturally idempotent.
Search by the request id or unique business key.Inspect duplicate-key and retry logs around the election.
ResolutionUse unique idempotency keys or transactions designed for retry, reconcile duplicates, and add a test that covers lost acknowledgements.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
decision
Test one-member loss only after stepdown passes
Optionally stop one SECONDARY—not the new primary—to prove the set preserves a majority. Keep the fault bounded, observe degraded redundancy, then restart the same member.
Why this step matters
A controlled secondary outage proves the topology tolerates one member loss and that the returning member catches up, without combining primary election and host recovery into one opaque event.
What to understand
Execute locally on the explicitly named secondary through out-of-band access, and confirm the other two members are healthy immediately before stop.
The cluster is temporarily one failure away from losing its majority. Freeze unrelated maintenance until redundancy is restored.
System changes
- Stops and restarts mongod on exactly one secondary, temporarily reducing replica redundancy.
Syntax explained
systemctl stop/start mongod- Exercises clean process unavailability; it does not simulate disk corruption or abrupt power failure.
sudo systemctl stop mongod; sleep 10; sudo systemctl start mongod; systemctl is-active mongodactive
Checkpoint: Checkpoint: Test one-member loss only after stepdown passes
Continue whenThe primary stays writable, the remaining secondary stays healthy, and the restarted member returns to SECONDARY without rollback.
Stop whenEither surviving member degrades, application error rate rises, or the returning node cannot catch up within the approved time.
If this step fails
The set has no primary after one member is stopped.
Likely causeMore than one voting member is unreachable, votes were misconfigured, or the stopped node exposed a hidden dependency such as DNS or shared storage.
Reach each member's management endpoint independently.rs.status() from every reachable memberdb.adminCommand({replSetGetConfig: 1})
ResolutionRestore a majority using the documented member configuration. Use forced reconfiguration only for genuine disaster recovery with MongoDB's official procedure and accepted data-loss risk.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Skip this optional fault when the environment cannot safely operate with only two reachable voters; the planned stepdown remains valid evidence.
Stop conditions
- Stop before a disruptive test when monitoring, a named incident commander, a current backup, or a tested recovery channel is missing.
- Stop when nodes disagree about identity, time, version, topology, or trust anchors; do not use force, trust bypasses, or quorum overrides to conceal the inconsistency.
- Stop when the application does not implement the retry, idempotency, timeout, and connection-discovery behavior required by the tested database.
- Stop after any unexplained data divergence, failed integrity check, authentication bypass, certificate validation failure, or loss of a majority.
verification
Confirm catch-up, absence of rollback, and balanced capacity
Wait for the returning member to reach SECONDARY, compare optimes, review logs for rollback, and confirm the new primary has adequate storage and connection headroom.
Why this step matters
Availability recovery is incomplete until redundancy and data convergence are restored. A green primary while one member is stale leaves the next failure unsafe.
What to understand
Inspect host and database metrics through a full workload cycle; a recently promoted primary can have cold filesystem cache and transient latency.
Any rollback evidence requires business-level reconciliation even if the member eventually reports SECONDARY.
System changes
- No state changes; the step observes convergence, capacity, and logs.
Syntax explained
printSecondaryReplicationInfo- Displays human-readable lag estimates for secondaries relative to their sync source.
Values stay on this page and are never sent or saved.
mongosh '{{mongoUri}}' --quiet --eval 'rs.printSecondaryReplicationInfo(); const s=rs.status(); printjson(s.members.map(m=>({name:m.name,state:m.stateStr,optimeDate:m.optimeDate,health:m.health})))'source: mongo-a.example.net:27017
0 secs (0 hrs) behind the primary
source: mongo-c.example.net:27017
0 secs (0 hrs) behind the primary
[ { name: 'mongo-a.example.net:27017', state: 'SECONDARY', health: 1 }, { name: 'mongo-b.example.net:27017', state: 'PRIMARY', health: 1 }, { name: 'mongo-c.example.net:27017', state: 'SECONDARY', health: 1 } ]Checkpoint: Checkpoint: Confirm catch-up, absence of rollback, and balanced capacity
Continue whenBoth secondaries are healthy and within lag policy, logs contain no rollback, and capacity remains inside SLO thresholds.
Stop whenA node stays RECOVERING, lag grows, rollback appears, or the new primary lacks capacity.
If this step fails
The old primary enters ROLLBACK after reconnecting.
Likely causeWrites were acknowledged without a durable majority, or a partition allowed the former primary to accept operations that are absent from the winning history.
rs.status()Inspect mongod logs for rollback files and common point messages.Compare canary and application idempotency records on the current primary.
ResolutionFreeze application writes, preserve rollback artifacts, reconcile business operations, and restore from the authoritative history under an incident procedure.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
verification
Close the exercise with measurable evidence
Calculate database and application recovery times, preserve redacted evidence, clean synthetic records according to retention policy, and convert every surprise into an owned corrective action.
Why this step matters
A failover test creates value only when it produces auditable SLO evidence and improves the next response. Cleanup must be narrow and independently reviewable.
What to understand
Record time to new primary, time to successful client write, peak error rate, retries, duplicate count, lag, and time to full redundancy.
Do not declare success when the database elected correctly but the application missed its SLO or required manual endpoint changes.
System changes
- Deletes only the two validation documents whose IDs begin with the approved exercise identifier.
Syntax explained
/^{{exerciseId}}-/- Scopes cleanup to the unique exercise prefix; inspect matches before deletion in production.
Values stay on this page and are never sent or saved.
mongosh '{{mongoUri}}' --quiet --eval 'printjson(db.getSiblingDB("ops_validation").failover_canary.deleteMany({_id:/^{{exerciseId}}-/},{writeConcern:{w:"majority",wtimeout:5000}}))'{ acknowledged: true, deletedCount: 2 }Checkpoint: Checkpoint: Close the exercise with measurable evidence
Continue whenTwo canaries are removed, the report includes objective timings, and follow-up actions have owners and due dates.
Stop whenThe deletion filter matches unexpected records or evidence has not been preserved.
If this step fails
Failover succeeds but latency and queue depth remain elevated.
Likely causeThe new primary has colder caches, weaker storage, different placement, or clients are reconnecting too aggressively.
db.adminCommand({serverStatus: 1})Compare host CPU, disk latency, page faults, connections, and replication lag.
ResolutionThrottle reconnect storms, confirm symmetric member capacity, and wait for measured stabilization. Fail back only through another planned election, never by killing the new primary.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Retain canaries under an explicit TTL index when audit policy requires historical synthetic evidence.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
Finish line
Verification checklist
mongosh '{{mongoUri}}' --quiet --eval 'const s=rs.status(); print(s.members.filter(m=>m.stateStr==="PRIMARY"&&m.health===1).length)'Exactly 1.mongosh '{{mongoUri}}' --quiet --eval 'const s=rs.status(); print(s.members.filter(m=>m.health===1).length)'Exactly 3.mongosh '{{mongoUri}}' --quiet --eval 'print(db.getSiblingDB("ops_validation").failover_canary.countDocuments({_id:/^{{exerciseId}}-/}))'0 after approved cleanup.Run the owned application synthetic transaction through its normal endpoint.A single successful operation inside the SLO, with no duplicate request ID and no TLS bypass.Recovery guidance
Common problems and safe checks
A member remains STARTUP2 or RECOVERING and never becomes SECONDARY.
Likely causeInitial sync is still copying data, the source is unavailable, the oplog window is too short, storage is too slow, or member authentication fails.
rs.status()db.getSiblingDB('admin').runCommand({replSetGetStatus: 1, initialSync: 1})journalctl -u mongod -n 200 --no-pager
ResolutionCorrect connectivity, capacity, TLS, or X.509 identity first. If initial sync cannot catch up, provision a clean data path and resync only after preserving diagnostic evidence.
The set repeatedly elects a new primary during normal load.
Likely causeHeartbeat loss, time or DNS instability, host pressure, long storage stalls, process restarts, or poorly planned member priorities are destabilizing elections.
rs.status()db.adminCommand({replSetGetConfig: 1, commitmentStatus: true})db.adminCommand({serverStatus: 1}).connections
ResolutionStabilize the network, storage, time, and host resources before changing election settings. Do not mask recurring failures by lengthening timeouts without evidence.
A write returns a write-concern timeout during or after failover.
Likely causeThe primary accepted the write but the requested majority did not acknowledge before the deadline; the outcome is ambiguous rather than safely failed.
db.getSiblingDB('admin').runCommand({getDefaultRWConcern: 1})rs.printSecondaryReplicationInfo()Query the operation's idempotency key after a primary is stable.
ResolutionReconcile the business operation by its idempotency key before retrying. Restore majority health and treat blind replay as a duplication risk.
Clients continue sending writes to the old primary.
Likely causeThe connection string lists one host, omits replicaSet, uses stale DNS, or the driver lacks retryable-write and server-selection handling.
Inspect the effective connection string without printing its password.Check driver topology events and serverSelectionTimeoutMS failures.mongosh '{{mongoUri}}' --eval 'db.hello()'
ResolutionUse a seed list containing multiple members and the exact replica-set name, then verify the supported driver rediscovers the new primary.
A TLS connection works only with hostname validation disabled.
Likely causeThe certificate SAN does not contain the advertised replica-set hostname, the CA chain is wrong, or clients connect through an undocumented alias.
openssl s_client -connect {{mongoA}}:{{mongoPort}} -servername {{mongoA}} -CAfile {{caFile}} </dev/nullopenssl x509 -in {{nodeCertificate}} -noout -subject -issuer -ext subjectAltName
ResolutionIssue a certificate with the correct DNS SAN and distribute the trusted CA. Never keep allowInvalidHostnames or allowInvalidCertificates as a production workaround.
rs.stepDown reports that no electable secondaries are caught up.
Likely causeReplication lag, maintenance mode, priority zero, hidden members, or unhealthy secondaries leave no safe election candidate.
rs.status()rs.printSecondaryReplicationInfo()rs.conf().members.map(m => ({host:m.host, priority:m.priority, votes:m.votes, hidden:m.hidden}))
ResolutionCancel the exercise, restore an electable synchronized secondary, and repeat the preflight. Do not force stepdown or reconfigure votes to manufacture availability.
The old primary enters ROLLBACK after reconnecting.
Likely causeWrites were acknowledged without a durable majority, or a partition allowed the former primary to accept operations that are absent from the winning history.
rs.status()Inspect mongod logs for rollback files and common point messages.Compare canary and application idempotency records on the current primary.
ResolutionFreeze application writes, preserve rollback artifacts, reconcile business operations, and restore from the authoritative history under an incident procedure.
The set has no primary after one member is stopped.
Likely causeMore than one voting member is unreachable, votes were misconfigured, or the stopped node exposed a hidden dependency such as DNS or shared storage.
Reach each member's management endpoint independently.rs.status() from every reachable memberdb.adminCommand({replSetGetConfig: 1})
ResolutionRestore a majority using the documented member configuration. Use forced reconfiguration only for genuine disaster recovery with MongoDB's official procedure and accepted data-loss risk.
The application reconnects but reports duplicate operations.
Likely causeIt retried an operation whose original outcome was ambiguous and the write was not naturally idempotent.
Search by the request id or unique business key.Inspect duplicate-key and retry logs around the election.
ResolutionUse unique idempotency keys or transactions designed for retry, reconcile duplicates, and add a test that covers lost acknowledgements.
Failover succeeds but latency and queue depth remain elevated.
Likely causeThe new primary has colder caches, weaker storage, different placement, or clients are reconnecting too aggressively.
db.adminCommand({serverStatus: 1})Compare host CPU, disk latency, page faults, connections, and replication lag.
ResolutionThrottle reconnect storms, confirm symmetric member capacity, and wait for measured stabilization. Fail back only through another planned election, never by killing the new primary.
Reference
Frequently asked questions
Does a three-member replica set guarantee zero data loss?
No. Durability depends on write concern and failure timing. Majority-acknowledged writes have stronger guarantees, while weaker or ambiguous writes can be rolled back or require reconciliation.
Should we force rs.stepDown when it refuses?
No for a routine exercise. Refusal usually means no safe caught-up electable secondary exists. Correct that condition instead of converting a test into an avoidable data-risk event.
Can a load balancer replace replica-set discovery?
Usually no. MongoDB drivers need the replica-set topology and primary identity. Use the official multi-host connection model unless the platform's supported architecture explicitly says otherwise.
Recovery
Rollback
Leadership cannot be rolled back like a file edit. Stabilize the elected primary, restore failed members, and reconcile any ambiguous writes; never kill a healthy new primary merely to return to the original placement.
- Abort further fault injection and keep the currently stable majority serving.
- Restart only the deliberately stopped member and wait for healthy SECONDARY with acceptable lag.
- If configuration was changed outside this guide, restore the reviewed rs.conf only through the normal majority-safe reconfiguration procedure.
- If rollback or ambiguous writes appear, freeze affected business operations, preserve evidence, and reconcile by idempotency key.
- Clean only synthetic records after evidence is captured; leave customer data untouched.
Evidence