Create, monitor, degrade, and recover a Linux RAID 1 array with mdadm
Build a two-member data RAID 1 array, persist assembly and mounting, monitor synchronization and consistency, rehearse a controlled member failure, and rebuild onto a positively identified replacement.
Operate the complete lifecycle of a non-boot Linux software mirror without treating RAID as backup, formatting an uncertain disk, or declaring recovery complete before both members are active and consistent.
- Ubuntu Server 24.04 LTS
- mdadm 4.3+
- Linux md 6.8+
- Disposable lab or approved data-array change Rehearse create, failure, removal, and rebuild with disposable devices first. This guide does not convert an existing root or boot disk and does not preserve data already on member devices.
- Two matched source members and one replacement Provision equal-size partitions or devices, and identify each by serial or WWN. The replacement must be at least as large as the active member at the md data-offset level.
lsblk --paths --output NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS,SERIAL,WWN - Independent verified backup RAID 1 mirrors deletion, corruption, ransomware, and operator mistakes. Complete a restorable backup before creating or deliberately degrading the array.
- Failure-domain and alert plan Place members on independent devices or controllers where possible and define who receives degraded-array alerts. A silent one-member mirror is an unprotected single disk.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A named Linux md RAID 1 data array with metadata 1.2 on two positively identified empty members, one ext4 filesystem on the md device, stable UUID mounting, and a saved assembly definition.
- An operational monitoring path using /proc/mdstat, mdadm detail/test semantics, sysfs degraded and sync state, mdmonitor integration, kernel logs, and periodic non-repairing consistency checks.
- A rehearsed failure lifecycle that deliberately fails and removes one member only in an approved test, proves continued reads, adds a known replacement, observes rebuild, and accepts recovery only at [UU] with both members active sync.
- {{arrayDevice}} assembles after boot with the expected array UUID and name, the filesystem mounts at {{mountPoint}}, and the independent test checksum remains valid.
- Normal monitoring reports degraded=0, sync_action=idle, [UU], two active devices, and zero failed devices; a degraded transition is routed to an accountable operator.
- The team can replace a member by durable identity without confusing a stale md superblock, wrong slot, or smaller replacement for the failed device.
Architecture
How the parts fit together
Linux md combines two member block devices into one RAID 1 md device. Each normal write is mirrored to both active members, while reads may be served from either. An md superblock stores array UUID, role, events, and geometry; metadata 1.2 lives near the start of each member. A single ext4 filesystem owns the md device, never the individual members. mdadm manages metadata and membership, the kernel performs mirroring and recovery, sysfs and /proc expose state, and mdmonitor converts state changes into alerts. Backup remains independent because logical corruption and deletion are mirrored immediately.
- Identify three empty device partitions by serial/WWN and prove no signature, holder, mount, or md superblock is present.
- Create {{arrayDevice}} from {{memberA}} and {{memberB}}, then wait for initial resync to complete at [UU].
- Create ext4 on the md device, mount by filesystem UUID, persist the ARRAY identity, and verify monitoring.
- Run a non-repairing consistency check and record mismatch_cnt without assuming which mirror copy is authoritative.
- In an approved rehearsal, fail/remove {{memberB}}, verify degraded reads, add {{replacementMember}}, and wait for complete recovery before closing the alert.
Assumptions
- This is a data RAID 1 array, not a boot, root, EFI, swap, database quorum, LVM RAID, firmware RAID, or clustered md design. Bootability and root conversion require duplicated bootloader and initramfs planning.
- {{memberA}}, {{memberB}}, and {{replacementMember}} are equal-size prepared partitions or whole devices with durable identities. The guide does not partition them or erase discovered signatures.
- The controlled degrade step is run only in a disposable lab or an approved resilience test with a verified backup and no unrelated device faults.
- The ext4 filesystem and its application are consistent before the checksum fixture is created. RAID does not supply application transaction consistency.
- An accountable alert destination exists. Enabling mdmonitor without delivering events to a person or monitoring platform is not operational monitoring.
Key concepts
- RAID 1 mirror
- An md layout that stores equivalent data on each active member and can normally continue after one of two members fails.
- Array superblock
- Per-member metadata describing array UUID, geometry, role, and event counter. Stale superblocks can cause unsafe assembly assumptions.
- Resync versus recovery
- Resync establishes or rechecks member equivalence; recovery copies data into a newly added replacement for a missing role.
- Degraded
- The array remains active with fewer working members than its configured RAID devices. For a two-member mirror it has no remaining disk redundancy.
- mismatch_cnt
- The kernel's count of sectors that differ or would be rewritten during check/repair. On RAID 1 it does not identify which copy contains correct application data.
- RAID is not backup
- Mirroring improves availability for a member failure but immediately reproduces deletions, corruption, malware encryption, and many operator errors.
Before you copy
Values used in this guide
{{memberA}}First approved empty RAID member identified by serial/WWN.
Example: /dev/vdb1{{memberB}}Second approved empty RAID member used for the controlled failure rehearsal.
Example: /dev/vdc1{{replacementMember}}Approved empty replacement at least as large as an active member.
Example: /dev/vdd1{{arrayDevice}}md block device created for the mirror.
Example: /dev/md0{{arrayBlockName}}Basename of arrayDevice used under /sys/block.
Example: md0{{arrayName}}Short host-local md array name and evidence filename component.
Example: data{{arrayLabel}}ext4 filesystem label written to the md device.
Example: RAID_DATA{{mountPoint}}Restricted directory where the array filesystem is mounted.
Example: /srv/raid{{testFile}}Existing non-secret fixture created before the degrade test and known to be safely readable.
Example: recovery-proof.txt{{checksumFile}}Checksum manifest for the approved fixture, stored on the array and copied to backup evidence.
Example: recovery-proof.sha256Security and production boundaries
- mdadm --create and mkfs are destructive. A valid-looking device path is not authorization; require serial/WWN, signature, mount, holder, and asset-inventory agreement immediately before execution.
- A degraded RAID 1 array has no remaining member redundancy. Minimize writes and rebuild promptly; do not schedule unrelated upgrades or consistency repair during recovery.
- Metadata 1.2 at the start of a member changes how the raw device appears. Never mount, fsck, or format an individual member as though it were the array filesystem.
- mismatch repair chooses a copy according to kernel RAID behavior, not application truth. Investigate hardware, memory, cabling, and backup evidence before requesting repair.
Stop before continuing if
- Stop before create or add if any candidate has a signature, mount, holder, stale superblock, unexpected size, or identity mismatch.
- Stop if the array name, UUID, level, member count, or metadata shown by mdadm differs from the approved design.
- Stop before the degrade rehearsal unless initial resync is complete, backup is verified, monitoring is active, and the test is explicitly approved.
- Stop all discretionary writes if the degraded survivor reports I/O errors. Preserve it and restore or seek specialist recovery rather than repeatedly assembling.
- Stop before declaring success until [UU], degraded=0, sync_action=idle, two active sync members, data verification, and clean kernel evidence all agree.
verification
Prove all candidate members are unmounted and unclaimed
Record size, signatures, mounts, holders, serials, and WWNs for both initial members and the replacement. Any existing metadata or uncertain identity stops creation.
Why this step matters
The most dangerous RAID mistake is selecting the wrong block device. Identity, signatures, holders, mount state, size, and stale md metadata are independent signals that must all support the same ownership decision.
What to understand
Run discovery immediately before create and again before replacement add because device enumeration and attachments can change during a long maintenance window.
wipefs --no-act is read-only. Any listed GPT, filesystem, LVM, encryption, or RAID signature means the device needs a separate ownership and sanitization decision.
mdadm --examine can return nonzero for an empty device; guarding that result preserves evidence without treating a stale superblock as harmless.
System changes
- No persistent change; reads block inventory and on-device signatures.
Syntax explained
wipefs --no-act- Lists recognized signatures without erasing them.
mdadm --examine- Reads md member metadata without assembling an array.
SERIAL,WWN- Links transient Linux paths to durable device identities.
Values stay on this page and are never sent or saved.
lsblk --paths --output NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS,SERIAL,WWN {{memberA}} {{memberB}} {{replacementMember}}; sudo wipefs --no-act {{memberA}} {{memberB}} {{replacementMember}}; sudo mdadm --examine {{memberA}} {{memberB}} {{replacementMember}} || trueNAME SIZE TYPE FSTYPE MOUNTPOINTS SERIAL WWN /dev/vdb1 100G part disk-a 0x600a /dev/vdc1 100G part disk-b 0x600b /dev/vdd1 100G part disk-c 0x600c mdadm: No md superblock detected on /dev/vdb1. mdadm: No md superblock detected on /dev/vdc1.
Checkpoint: Approve three device identities
Continue whenAll candidates have expected size and serial/WWN, and no mount, holder, signature, or md superblock.
Stop whenAny signal is missing, inconsistent, or shows prior ownership.
If this step fails
A candidate shows an old md UUID.
Likely causeThe device was reused from another array or the wrong attachment was selected.
mdadm --examine {{memberA}}lsblk -f {{memberA}}udevadm info --query=property --name={{memberA}}
ResolutionStop and identify the former array and data owner. Sanitization is a separate destructive approval.
Security notes
- Treat cross-tenant or recycled storage as potentially sensitive even when no filesystem mounts.
Alternatives
- Provision fresh provider volumes with recorded attachment IDs and still repeat local identity checks.
Stop conditions
- No --force, wipe, zero-superblock, create, or format action follows uncertain discovery.
command
Install mdadm and confirm the kernel RAID 1 personality
Install the distribution package, print its version, and verify that RAID 1 support can be loaded before any member metadata is written.
Why this step matters
Installing the supported package and confirming the kernel personality establishes the managed toolchain before metadata exists, avoiding an array created by an untracked binary or unsupported environment.
What to understand
Ubuntu's mdadm package integrates udev, initramfs, monitoring, and configuration paths. Retain its package version with the change record.
modprobe raid1 loads the kernel personality if modular. /proc/mdstat lists available personalities even before an array exists.
Package installation can start services and update initramfs; review apt output and change controls.
System changes
- Installs mdadm and dependencies, may enable monitoring integration, and loads the RAID 1 kernel module.
Syntax explained
apt install --yes mdadm- Installs the distribution-managed md administration package.
modprobe raid1- Loads Linux md RAID 1 support when built as a module.
grep --word-regexp raid1- Requires the exact personality token in /proc/mdstat.
sudo apt update && sudo apt install --yes mdadm && sudo modprobe raid1 && mdadm --version && grep --word-regexp raid1 /proc/mdstatmdadm - v4.3 - 2024-02-15 Personalities : [raid1]
Checkpoint: Confirm supported md tooling
Continue whenmdadm reports its version and /proc/mdstat lists raid1.
Stop whenPackage configuration fails or the kernel lacks RAID 1 support.
Security notes
- Use signed distribution packages rather than an unverified recovery binary.
Alternatives
- Bake and validate mdadm in the server image before attaching production data devices.
Stop conditions
- Do not create an array while package or initramfs configuration is incomplete.
warning
Create the named two-member RAID 1 array
Create metadata version 1.2 on exactly the two approved empty members. The command writes md superblocks and begins copying one complete mirror; never run it against devices containing recoverable data.
Why this step matters
Creation writes superblocks to both members and starts a full mirror synchronization. Explicit level, member count, metadata, name, device, and ordered member list make the destructive intent reviewable.
What to understand
Metadata 1.2 is the modern default and resides near the beginning of each member; members are not directly mountable filesystems.
RAID 1 usable size is approximately one member, not the sum. Geometry and metadata overhead explain the small difference.
The create output and mdadm detail must immediately agree on level, two RAID devices, two active devices, and intended name.
System changes
- Writes md 1.2 superblocks to {{memberA}} and {{memberB}}, creates {{arrayDevice}}, and begins resync.
Syntax explained
--create {{arrayDevice}}- Creates the named md block device and new array metadata.
--metadata=1.2- Selects modern md superblock layout.
--level=1- Selects mirroring.
--raid-devices=2- Declares exactly two required member roles.
--name={{arrayName}}- Stores a human-readable host-local array name.
Values stay on this page and are never sent or saved.
sudo mdadm --create {{arrayDevice}} --metadata=1.2 --level=1 --raid-devices=2 --name={{arrayName}} {{memberA}} {{memberB}} && sudo mdadm --detail {{arrayDevice}}mdadm: array /dev/md0 started. /dev/md0: Raid Level : raid1 Array Size : 104726528 (99.87 GiB) Raid Devices : 2 Active Devices : 2 State : clean, resyncing
Checkpoint: Validate the new array identity
mdadm --detail {{arrayDevice}}Continue whenCorrect level, UUID/name, size, two devices, and resyncing/clean state.
Stop whenAny member, level, size, UUID context, or state differs from approval.
If this step fails
mdadm warns the devices have different sizes.
Likely causeProvisioning or partition boundaries do not match.
blockdev --getsize64 {{memberA}} {{memberB}}lsblk -b -o NAME,SIZE,START {{memberA}} {{memberB}}
ResolutionStop and reprovision matching members; do not accept unexplained lost capacity.
Security notes
- Use a second-person confirmation of both member identities immediately before this step.
Alternatives
- Create a degraded one-member array only in a separately designed migration; it is not a safe shortcut for new storage.
Stop conditions
- Any signature warning, identity mismatch, or unexpected geometry blocks creation.
verification
Monitor initial synchronization to a clean mirror
Track /proc/mdstat and detailed member state until resync finishes. Do not format or run a failure rehearsal while member data is still being copied.
Why this step matters
An array can be active while its second copy is incomplete. Waiting for [UU], degraded=0, and sync_action=idle prevents formatting load or deliberate failure from competing with the only initial synchronization.
What to understand
/proc/mdstat gives concise progress and [UU] member-slot state. mdadm detail gives member roles and event state.
sync_action reports resync, recovery, check, repair, or idle; degraded reports missing required members numerically.
Resync speed depends on device and workload. Do not change kernel speed limits reactively without an I/O-impact plan.
System changes
- No administrative metadata change; the kernel continues copying mirror contents until synchronized.
Syntax explained
[UU]- Both configured RAID 1 member slots are up.
sync_action=idle- No resync, recovery, check, or repair is active.
degraded=0- No required member is missing.
Values stay on this page and are never sent or saved.
cat /proc/mdstat; sudo mdadm --detail {{arrayDevice}}; cat /sys/block/{{arrayBlockName}}/md/sync_action; cat /sys/block/{{arrayBlockName}}/md/degradedmd0 : active raid1 vdc1[1] vdb1[0]
104726528 blocks super 1.2 [2/2] [UU]
unused devices: <none>
State : clean
idle
0Checkpoint: Require a complete initial mirror
Continue when[UU], two active sync members, degraded 0, and idle.
Stop whenA member logs I/O errors, drops, or resync cannot progress.
Security notes
- Do not mistake active for redundant; alerting should treat resync and degradation as operational states.
Alternatives
- Throttle or schedule creation in a maintenance window rather than using an incomplete array under full load.
Stop conditions
- No format, consistency check, or failure test before initial resync completes.
warning
Create one filesystem on the array and mount by UUID
Format the md device—not either member—with ext4, create the restricted mount point, save fstab, append the filesystem UUID entry, validate it, and mount the array.
Why this step matters
The filesystem belongs on the md device so every write follows mirror semantics. Stable filesystem UUID mounting and a preserved fstab prevent transient md numbering from becoming the only boot contract.
What to understand
mkfs.ext4 erases the selected block namespace; the command names arrayDevice explicitly and must never name memberA or memberB.
The fstab backup is required because an invalid entry can send the next boot to emergency mode. findmnt validates before mount.
nofail is appropriate only when this data mount is explicitly allowed to be absent at boot; required application storage should instead fail visibly with proper ordering.
System changes
- Creates ext4 on {{arrayDevice}}, creates {{mountPoint}}, saves and appends fstab, and mounts the filesystem.
Syntax explained
mkfs.ext4 -L- Creates ext4 and assigns the reviewed label on the md device.
blkid -s UUID -o value- Extracts the filesystem UUID for stable fstab identity.
defaults,nofail 0 2- Uses ordinary mount defaults, permits optional boot absence, disables dump, and schedules non-root fsck order.
Values stay on this page and are never sent or saved.
sudo mkfs.ext4 -L {{arrayLabel}} {{arrayDevice}} && sudo install --directory --mode=0750 {{mountPoint}} && sudo cp --archive /etc/fstab /etc/fstab.before-{{arrayName}} && ARRAY_UUID=$(sudo blkid -s UUID -o value {{arrayDevice}}) && printf 'UUID=%s %s ext4 defaults,nofail 0 2\n' "$ARRAY_UUID" '{{mountPoint}}' | sudo tee -a /etc/fstab && findmnt --verify --verbose && sudo mount {{mountPoint}} && findmnt {{mountPoint}}Creating filesystem with 26181632 4k blocks and 6545408 inodes Success, no errors or warnings detected TARGET SOURCE FSTYPE OPTIONS /srv/raid /dev/md0 ext4 rw,relatime
Checkpoint: Prove filesystem placement and mount
findmnt {{mountPoint}} && lsblk -f {{arrayDevice}}Continue whenExactly one ext4 filesystem on the md device is mounted at the approved path.
Stop whenA member device is formatted/mounted or fstab verification reports any error.
Security notes
- Select mount options such as nodev, nosuid, or noexec according to application requirements rather than copying defaults blindly.
Alternatives
- Use XFS through a separately sourced and tested filesystem procedure.
Stop conditions
- Never format a member or append duplicate fstab entries.
command
Persist array assembly and refresh the initramfs
Capture the ARRAY definition generated from the live device, append it only if its UUID is absent, update initramfs, and verify that the saved identity matches mdadm detail.
Why this step matters
Array UUID and metadata are stable; /dev/md numbering can vary. Saving the live ARRAY definition and refreshing initramfs lets early boot assemble the same identity without broad scans or naming assumptions.
What to understand
The generated line is saved as evidence before a guarded append. The UUID check avoids duplicating an existing record.
update-initramfs includes current md configuration in the boot image used to discover storage before the root system is fully available.
Review mdadm.conf for stale ARRAY entries that reference reused members or duplicate names.
System changes
- May append one ARRAY line to /etc/mdadm/mdadm.conf, creates evidence, and regenerates the current initramfs.
Syntax explained
mdadm --detail --scan- Prints an ARRAY definition from active md metadata.
grep -q UUID- Prevents appending a duplicate live array identity.
update-initramfs -u- Updates the existing initramfs with current assembly configuration.
Values stay on this page and are never sent or saved.
ARRAY_LINE=$(sudo mdadm --detail --scan {{arrayDevice}}) && printf '%s\n' "$ARRAY_LINE" | sudo tee /root/{{arrayName}}-array-line.txt && grep -q "$(sudo mdadm --detail {{arrayDevice}} | awk '/UUID/ {print $3}')" /etc/mdadm/mdadm.conf || printf '%s\n' "$ARRAY_LINE" | sudo tee -a /etc/mdadm/mdadm.conf && sudo update-initramfs -u && sudo mdadm --detail --scanupdate-initramfs: Generating /boot/initrd.img-6.8.0-71-generic ARRAY /dev/md0 metadata=1.2 name=host:data uuid=4cb9c8ef:7a2e1f05:8900d173:b15e3a91
Checkpoint: Match saved and live UUID
mdadm --detail --scan; grep '^ARRAY' /etc/mdadm/mdadm.confContinue whenOne matching UUID/name definition exists and initramfs generation succeeds.
Stop whenMultiple conflicting entries or a failed initramfs update appears.
Security notes
- Array names and UUIDs are infrastructure identifiers; protect evidence while making it accessible to responders.
Alternatives
- Use distribution-supported incremental assembly when policy intentionally avoids explicit ARRAY lines, then test reboot discovery.
Stop conditions
- Do not reboot into an unverified or failed initramfs update.
verification
Establish monitoring and run a read-only status probe
Verify mdmonitor service state, collect machine-readable degraded and sync indicators, and send a mdadm test event only after a real MAILADDR or monitoring integration has been configured.
Why this step matters
RAID 1 protects availability only when a degraded state is detected before the surviving member fails. A machine-readable baseline plus delivered test alerts establishes operational ownership.
What to understand
mdmonitor package integration varies; enabled/active must be interpreted with the installed unit and config rather than assumed.
mdadm --detail --test exits nonzero when the array is degraded and is suitable for supervised probes.
MAILADDR or monitoring integration must reach a real accountable destination; a local mailbox nobody reads is not sufficient.
System changes
- No change from status probes; separate monitoring configuration may enable alert delivery.
Syntax explained
mdadm --detail --test- Returns status suitable for monitoring without changing the array.
degraded- Kernel sysfs count of missing required members.
sync_action- Current synchronization or check activity.
Values stay on this page and are never sent or saved.
systemctl is-enabled mdmonitor.service; systemctl is-active mdmonitor.service; cat /sys/block/{{arrayBlockName}}/md/degraded; cat /sys/block/{{arrayBlockName}}/md/sync_action; sudo mdadm --detail --test {{arrayDevice}}; echo exit=$?enabled active 0 idle exit=0
Checkpoint: Deliver one normal baseline and test alert
Continue whenMonitoring records healthy state and the responsible operator can acknowledge a test event.
Stop whenThe service is inactive, destination is absent, or a real degraded state already exists.
Security notes
- Alert messages may expose device names and host identity; send through authorized monitoring channels.
Alternatives
- Export sysfs and mdadm state through the organization's node-monitoring agent with equivalent alert tests.
Stop conditions
- Do not begin a deliberate failure exercise without proven alert delivery.
command
Run and observe a scheduled consistency check
Request a non-repairing md check during a low-I/O window, monitor completion, then record mismatch_cnt. On RAID 1, a nonzero count does not by itself identify which copy is correct.
Why this step matters
A check reads mirror copies and counts mismatches without requesting repair, giving early evidence of divergence while preserving the decision about which data is trustworthy.
What to understand
Schedule checks away from rebuild, backup, snapshot, and peak I/O because they consume device bandwidth and can expose marginal hardware.
The loop waits on sysfs action rather than assuming a fixed duration. Operational automation should use bounded monitoring rather than an endless shell loop.
mismatch_cnt is sector-oriented and can be affected by page granularity. Nonzero requires investigation, not automatic repair.
System changes
- Requests a kernel md check that reads both members and updates mismatch_cnt; it does not request repair.
Syntax explained
echo check > sync_action- Starts a non-repairing md consistency pass.
mismatch_cnt- Reports sectors that differ or would be rewritten, subject to kernel accounting details.
Values stay on this page and are never sent or saved.
echo check | sudo tee /sys/block/{{arrayBlockName}}/md/sync_action && cat /proc/mdstat && while [ "$(cat /sys/block/{{arrayBlockName}}/md/sync_action)" != idle ]; do sleep 5; done; cat /sys/block/{{arrayBlockName}}/md/mismatch_cntcheck
md0 : active raid1 vdc1[1] vdb1[0]
[>....................] check = 3.1% (3276800/104726528) finish=8.4min speed=199980K/sec
0Checkpoint: Record completed consistency evidence
Continue whensync_action returns idle, array remains [UU], and mismatch_cnt is reviewed.
Stop whenA device error, degradation, or unexpected mismatch appears.
If this step fails
sync_action remains check far longer than expected.
Likely causeHeavy I/O, throttling, a slow/failing device, or suspended recovery.
cat /proc/mdstatjournalctl -k --since '-30 minutes'cat /sys/block/{{arrayBlockName}}/md/sync_completed
ResolutionReduce workload safely and diagnose devices; do not start another check or repair concurrently.
Security notes
- Do not run repair merely to make a monitoring number green; verify data authority first.
Alternatives
- Use scheduled distribution md check units when their cadence, load, and alert integration are documented.
Stop conditions
- Any degradation or hardware error cancels the elective consistency exercise.
warning
Fail and remove one known-good member in the approved rehearsal
Only in a disposable lab or explicit resilience test, mark {{memberB}} failed and remove it. Confirm the array remains active with [U_] and the application test file remains readable before touching the replacement.
Why this step matters
A controlled rehearsal proves the documented commands, monitoring, continued read path, and operator response before a real failure. It also intentionally removes all disk redundancy, so production use needs explicit risk approval.
What to understand
--fail changes the member state while it is still addressable; --remove detaches only the already failed member.
The expected [U_] assumes memberB occupied the second slot. Trust mdadm detail rather than pattern position if roles differ.
The fixture hash verifies a known read through the remaining member, not all application data or write consistency.
System changes
- Marks {{memberB}} faulty, removes it from {{arrayDevice}}, and leaves the array active but degraded on one member.
Syntax explained
--manage- Selects live member-management mode.
--fail- Marks the named member faulty in md metadata.
--remove- Detaches the already failed member from the array.
Values stay on this page and are never sent or saved.
sudo mdadm --manage {{arrayDevice}} --fail {{memberB}} --remove {{memberB}} && cat /proc/mdstat && sudo mdadm --detail {{arrayDevice}} && sudo sha256sum {{mountPoint}}/{{testFile}}mdadm: set /dev/vdc1 faulty in /dev/md0
mdadm: hot removed /dev/vdc1 from /dev/md0
md0 : active raid1 vdb1[0]
104726528 blocks super 1.2 [2/1] [U_]
State : clean, degraded
9f86d081884c7d65... /srv/raid/recovery-proof.txtCheckpoint: Confirm bounded degraded service
Continue whenOnly memberB is removed, array remains active degraded, alert fires, and fixture hash matches.
Stop whenThe survivor logs any error or application integrity/reachability fails.
If this step fails
mdadm refuses to remove the member.
Likely causeIt was not marked faulty, is the only active member, or device identity does not match a member.
mdadm --detail {{arrayDevice}}cat /proc/mdstat
ResolutionDo not force removal. Reconcile role/state and preserve the active survivor.
Security notes
- A degraded array is an availability incident; limit access and changes until rebuild completes.
Alternatives
- Rehearse on loop devices or disposable VMs using the same monitoring and runbook when production risk is unacceptable.
Stop conditions
- Never fail the only active member or continue when the survivor is unhealthy.
command
Add the positively identified replacement and monitor rebuild
Repeat signature, size, identity, mount, and holder checks for {{replacementMember}}, add it to the degraded array, and monitor until both slots are active sync.
Why this step matters
Replacement is another destructive identity boundary. Repeating discovery immediately before add prevents a new attachment or stale superblock from joining the wrong array, while progress observation protects the survivor.
What to understand
The replacement must be large enough for the array's component size, not merely marketed as the same nominal capacity.
mdadm may list it as spare rebuilding until recovery completes, then active sync in the missing role.
Recovery reads the entire survivor and writes the replacement; monitor kernel, SMART/platform, latency, and application health.
System changes
- Adds {{replacementMember}} to the degraded array and starts kernel recovery into the missing RAID 1 role.
Syntax explained
--manage --add- Adds the named device as a spare and triggers recovery when a required role is missing.
spare rebuilding- Intermediate state while data is copied to the replacement.
Values stay on this page and are never sent or saved.
lsblk --paths --output NAME,SIZE,FSTYPE,MOUNTPOINTS,SERIAL,WWN {{replacementMember}} && sudo wipefs --no-act {{replacementMember}} && sudo mdadm --examine {{replacementMember}} || true; sudo mdadm --manage {{arrayDevice}} --add {{replacementMember}} && cat /proc/mdstat && sudo mdadm --detail {{arrayDevice}}mdadm: added /dev/vdd1
md0 : active raid1 vdd1[2] vdb1[0]
[>....................] recovery = 4.8% (5033164/104726528) finish=8.0min speed=207800K/sec
Number Major Minor RaidDevice State
0 252 17 0 active sync /dev/vdb1
2 252 49 1 spare rebuilding /dev/vdd1Checkpoint: Observe one correct recovery
Continue whenReplacement identity is correct, recovery percentage advances, and survivor/application remain healthy.
Stop whenRecovery stalls, either device errors, or application integrity changes.
If this step fails
mdadm says the replacement is not large enough.
Likely causePartition geometry or actual sector capacity is smaller than the array component size.
blockdev --getsize64 {{replacementMember}} {{memberA}}mdadm --detail {{arrayDevice}}
ResolutionProvision a larger replacement; never shrink or recreate the active array during degradation.
Security notes
- Retain failed media under data-handling and disposal policy because it contains a recoverable copy of mirrored data.
Alternatives
- Add a prequalified hot spare before failure when capacity and failure-domain policy justify faster recovery start.
Stop conditions
- Do not clear signatures or force-add a replacement whose ownership or size is uncertain.
verification
Prove full redundancy, consistency, mount health, and alert closure
Wait for recovery to reach idle, require degraded=0 and [UU], verify both members are active sync, recheck the known data hash, and confirm no md or device errors appeared in the kernel journal.
Why this step matters
Recovery is complete only when independent kernel, mdadm, filesystem, data, and journal signals agree. A progress bar reaching 100 percent without active-sync state or clean data evidence is insufficient.
What to understand
[UU], degraded=0, and idle prove configured slot presence and no active copy operation; mdadm detail ties those slots to actual devices.
Checksum verification proves the known fixture survived the exercise. Application-native consistency and backup verification remain necessary for real datasets.
Clear the degraded alert only after this evidence and keep failed-device diagnostics with the incident record.
System changes
- No persistent change; reads final md, filesystem, fixture, and kernel evidence.
Syntax explained
sha256sum --check- Validates the approved fixture against its recorded digest.
journalctl --dmesg --priority=warning- Checks the recovery window for kernel device and md warnings.
Values stay on this page and are never sent or saved.
cat /proc/mdstat; cat /sys/block/{{arrayBlockName}}/md/degraded; cat /sys/block/{{arrayBlockName}}/md/sync_action; sudo mdadm --detail {{arrayDevice}}; sudo sha256sum --check {{mountPoint}}/{{checksumFile}}; sudo journalctl --dmesg --since '-30 minutes' --priority=warning --no-pagermd0 : active raid1 vdd1[2] vdb1[0]
104726528 blocks super 1.2 [2/2] [UU]
0
idle
Active Devices : 2
Failed Devices : 0
/srv/raid/recovery-proof.txt: OK
-- No entries --Checkpoint: Accept restored redundancy
Continue when[UU], degraded 0, idle, two active sync members, valid data evidence, and clean kernel journal.
Stop whenAny signal disagrees or the replacement is not active sync.
Security notes
- Do not expose fixture contents or device serials in public incident reports.
Alternatives
- Keep the service drained through a longer scrub and application consistency window for critical data.
Stop conditions
- No alert closure or failed-disk disposal before full evidence and backup owner acceptance.
Finish line
Verification checklist
test "$(cat /sys/block/{{arrayBlockName}}/md/degraded)" = 0 && test "$(cat /sys/block/{{arrayBlockName}}/md/sync_action)" = idle && grep --fixed-strings '[UU]' /proc/mdstatThe array is idle, not degraded, and both RAID 1 slots are up.sudo mdadm --detail {{arrayDevice}}State is clean, Active Devices is 2, Failed Devices is 0, and both approved members are active sync.findmnt {{mountPoint}} && sudo sha256sum --check {{mountPoint}}/{{checksumFile}}The ext4 filesystem is mounted from the md device by its UUID and the known test data hash verifies.sudo mdadm --detail --scan && grep '^ARRAY' /etc/mdadm/mdadm.confOne saved ARRAY record has the same UUID and name as the live RAID 1 device.Recovery guidance
Common problems and safe checks
mdadm --create warns that a device contains a filesystem or md superblock.
Likely causeA candidate is not empty, belongs to another array, or was selected by an unstable path.
wipefs --no-act {{memberA}} {{memberB}}mdadm --examine {{memberA}} {{memberB}}lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINTS,SERIAL,WWN
ResolutionStop and reconcile ownership and backups. Do not add --force or zero metadata as part of the same change.
/proc/mdstat remains [U_] after adding the replacement.
Likely causeRecovery is still active, the replacement is too small, has I/O errors, was added as a spare but cannot take the missing role, or recovery was paused.
cat /proc/mdstatmdadm --detail {{arrayDevice}}cat /sys/block/{{arrayBlockName}}/md/sync_actionjournalctl -k --since '-30 minutes'
ResolutionPreserve the survivor, resolve the specific replacement or device error, and allow recovery to complete. Never remove the active survivor to force slot ordering.
The array does not assemble after reboot, but both members are visible.
Likely causeThe initramfs or mdadm.conf lacks the correct ARRAY identity, members expose stale/conflicting event counters, or boot ordering differs.
mdadm --examine {{memberA}} {{replacementMember}}mdadm --detail --scangrep '^ARRAY' /etc/mdadm/mdadm.confjournalctl -b -u mdmonitor --no-pager
ResolutionAssemble by verified array UUID only after comparing member event counters and roles, then correct the saved ARRAY definition and regenerate initramfs.
A consistency check finishes with a nonzero mismatch_cnt.
Likely causeCopies differ because of interrupted writes, prior device errors, memory/controller problems, or a false assumption about write barriers; the count alone does not identify correct data.
cat /sys/block/{{arrayBlockName}}/md/mismatch_cntmdadm --detail {{arrayDevice}}journalctl -k --since '-24 hours'smartctl -x on each underlying whole disk
ResolutionFreeze discretionary changes, verify application data against backup or checksums, investigate hardware, and choose a repair/restore plan with explicit authority. Do not immediately echo repair.
Reference
Frequently asked questions
Can I use two partitions on the same physical disk?
mdadm can, but it does not protect against that disk's failure and can make performance worse. Members should occupy independent failure domains where practical.
Should I use /dev/md0 in fstab?
Mount the filesystem by its filesystem UUID and save the md ARRAY UUID/name for assembly. Transient md numbering should not be the only identity contract.
Is the array safe as soon as the replacement is added?
No. It remains degraded or recovering until both slots are active sync and [UU], degraded=0, and sync_action=idle all agree.
Recovery
Rollback
Array creation and filesystem formatting destroy prior member content. Roll back service by restoring data to replacement storage from backup; do not dismantle the only surviving mirror member during an incident.
- Before application data is written, unmount {{mountPoint}}, restore /etc/fstab.before-{{arrayName}}, remove the matching ARRAY line from mdadm.conf, update initramfs, and stop the array with mdadm --stop.
- Zero md superblocks only on positively identified disposable members after the array is stopped and backups are confirmed; this is destructive and outside ordinary incident recovery.
- After production use, provision replacement storage, restore or replicate data, verify application consistency, switch the mount during maintenance, and retain the md array until acceptance passes.
- During a degraded incident, preserve the surviving active member, add a known-good replacement, and complete rebuild. Do not attempt rollback by removing the survivor.
Evidence