Diagnose failing disks with SMART and replace a drive safely
Correlate kernel I/O symptoms, SMART or NVMe health data, self-test history, stable device identity, storage-stack membership, and application evidence before replacing a Linux data drive through the layer that actually owns redundancy.
Reach an evidence-based keep, monitor, or replace decision; preserve recoverable data before stress testing; remove only the positively identified physical device; and prove that the replacement is healthy and the owning storage stack has restored redundancy.
- Ubuntu Server 24.04 LTS
- smartmontools 7.4, 7.5
- A current recoverable copy of critical data SMART diagnostics can increase load on a marginal disk, and replacement can expose the last good replica. Restore representative data or validate the storage stack's independent backup before running extended tests.
- Stable device and physical-slot mapping Record /dev/disk/by-id, serial, WWN/EUI, model, controller path, enclosure bay, and the current kernel name. Never select a disk for removal from /dev/sdX alone.
udevadm info --query=property --name={{device}} | grep -E '^(ID_MODEL|ID_SERIAL|ID_WWN|ID_PATH)=' - Storage ownership and redundancy documentation Know whether the device belongs to mdraid, ZFS, LVM, device mapper, Ceph, a hardware RAID virtual disk, a plain filesystem, swap, or an application. The owning layer determines quiesce and replacement commands.
- Console access and approved downtime A replacement may require shutdown, enclosure work, controller rescan, or workload stop. Keep console access and an agreed failure escalation path.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A repeatable evidence packet that maps one physical drive through stable Linux identities, controller topology, mounts, RAID or volume membership, SMART/NVMe telemetry, kernel errors, self-test history, and application symptoms.
- A conservative decision gate that distinguishes interface or environmental faults from media failure, treats SMART PASSED as one signal rather than a warranty, and avoids running a long test when it could destroy the last readable copy.
- An owner-aware replacement runbook that quiesces the workload, uses the mdraid, ZFS, LVM, Ceph, hardware RAID, or plain-filesystem procedure actually responsible for data, verifies the new serial before rebuild, and closes with workload and kernel evidence.
- The operator can explain exactly why the device is being kept, monitored, or replaced, with captured SMART exit bits, raw logs, trends, kernel correlation, redundancy state, and a backup restore identifier.
- Remote hands remove only the disk whose serial, WWN, path, and bay all match the failed logical member; the new disk is equal-or-larger, empty, healthy, and positively identified before any rebuild begins.
- The owning storage layer reports restored redundancy, the replacement passes a post-install self-test, the surviving member did not accumulate new errors, and a representative workload read succeeds.
Architecture
How the parts fit together
SMART is telemetry supplied by a drive and interpreted through smartmontools; it is not the storage architecture. A Linux path may represent a physical disk, a controller-exposed virtual disk, a USB bridge, an NVMe namespace, or a member hidden below mdraid, ZFS, LVM, device mapper, Ceph, or a hardware RAID controller. Diagnosis therefore begins by mapping identity and ownership. Device telemetry is then correlated with kernel transport errors and the owning layer's data-integrity counters. Replacement reverses that map: the workload is quiesced, the owner safely releases the old member, hardware identity is confirmed, the new device is admitted, and reconstruction is observed before service verification.
- Stable aliases, serial, WWN, path, mount, holder, and storage membership identify what the kernel device actually represents.
- smartctl reads the supported device interface and preserves complete baseline and exit bits before tests change logs or load.
- Kernel messages, storage-owner counters, SMART trends, error LBAs, and self-tests are correlated into keep, monitor, or replace.
- The application and storage owner reach a safe quiesced state, and the old member is released only through the owning layer's documented process.
- The replacement identity and capacity are verified, the owner reconstructs redundancy, and post-change self-test, kernel, owner, and application checks close the incident.
Assumptions
- The operator can reach the raw physical device or the controller-specific interface that exposes meaningful health data. Some hardware RAID controllers hide drives behind virtual disks and require vendor or smartctl controller options.
- A current independent backup or verified redundant copy exists before self-tests and replacement. The guide prioritizes data survival over completing a diagnostic checklist.
- Stable identifiers and a bay map exist or can be created without unplugging devices. Hot swap is used only when the chassis, controller, drive carrier, operating system, and documented process all support it.
- The device hosts data rather than the running root or boot chain. Root-disk, encrypted-boot, multipath, Ceph OSD, SAN, and clustered replacements need stack-specific procedures beyond the general gates here.
- Attribute names and raw encodings differ by vendor and transport. Trend, error context, official device guidance, self-tests, kernel evidence, and workload behavior are evaluated together.
Key concepts
- SMART overall health
- A drive-provided threshold assessment. PASSED does not mean every sector is readable, and FAILED is a strong replacement signal rather than a repair action.
- Normalized and raw attributes
- ATA vendors expose normalized threshold values and vendor-specific raw fields. Raw counts such as reallocated, pending, and uncorrectable sectors need model-aware interpretation and trend comparison.
- Self-test
- A test executed by the device firmware. Short tests sample critical functions; extended tests read much more media. Command acceptance starts a test but the self-test log records its eventual result.
- Transport error
- An error caused outside the recording media, such as a cable, connector, HBA, backplane, power, thermal, firmware, or protocol problem. Interface CRC or reset patterns may affect a healthy disk.
- Stable device identity
- A serial, WWN, EUI, or controller-qualified identifier represented through /dev/disk/by-id and physical inventory. sdX and nvme index names are discovery-order labels.
- Storage owner
- The highest layer that defines redundancy and safe membership changes. Its status, not raw-disk presence alone, determines whether detaching and reconstruction are safe.
Before you copy
Values used in this guide
{{device}}Current raw block-device path being diagnosed. Prefer the stable whole-device by-id path when smartctl supports it.
Example: /dev/disk/by-id/wwn-0x5000c500bb01{{deviceKernelName}}Kernel basename used to correlate journal messages. Derive it at the time of diagnosis because it can change after reboot.
Example: sdc{{replacementDevice}}Stable whole-device path for the new disk after insertion and udev discovery.
Example: /dev/disk/by-id/wwn-0x5000c500bb99{{workloadService}}Exact systemd unit whose supported stop/start produces an application-consistent storage state.
Example: archive.service{{shortTestWaitSeconds}}Wait derived from smartctl's reported short-test completion estimate, with a small margin.
Example: 180{{longTestWaitSeconds}}Wait derived from `smartctl -c`; use only when the risk gate permits an extended test.
Example: 45000Security and production boundaries
- SMART and udev outputs include serials, WWNs, firmware revisions, topology, power-on age, and error history. Keep raw evidence restricted while preserving it for warranty, incident, and root-cause analysis.
- Raw-device, controller, enclosure, and rebuild authority can destroy or expose all data. Limit it to trained operators and require positive identity confirmation before physical work.
- A removed unencrypted disk retains data even when SMART fails. Use chain of custody, controlled storage, approved sanitization, and vendor-return terms that preserve confidentiality and possible forensic needs.
- Do not upload full SMART JSON to public analysis tools; it may identify assets and operational weaknesses. Share a redacted extract only after retaining the authoritative original.
Stop before continuing if
- Stop tests and minimize reads if critical data lacks a verified backup, the device is the last readable copy, errors are increasing, or the drive repeatedly disappears.
- Stop replacement if the serial, WWN, path, and physical bay do not all identify one target, or if more than one member of the redundancy group is unhealthy.
- Do not use a generic mdadm, zpool, pvremove, controller, or partition command until the owning storage stack and its safe replacement procedure are documented.
- Stop rebuild if the surviving member reports new uncorrectable errors, the replacement is smaller, or workload behavior suggests corruption. Preserve evidence and move to backup recovery.
- Do not hot-swap unless every hardware and software layer explicitly supports it and the selected member has been safely released.
verification
Map the device through every storage layer
Capture the block tree, mounts, holders, stable aliases, controller path, serial, WWN, and storage memberships before asking SMART for a verdict. This map identifies what must be quiesced and rebuilt.
Why this step matters
A SMART report is meaningful only when it belongs to the intended physical device and the operator understands which higher layer depends on it. Mapping first prevents diagnosis of a virtual disk or removal of the wrong member.
What to understand
lsblk shows parents and holders, but combine it with findmnt, pvs, mdadm, and zpool because no single command covers every possible owner.
A by-id link should resolve to the same kernel device whose serial and WWN appear in the inventory. Partition links and whole-disk links are different replacement targets.
Save the output before reboot because kernel names and missing-device links can change after the hardware is disturbed.
System changes
- No persistent changes; writes the device identity and ownership baseline into the maintenance evidence.
Syntax explained
find -L ... -samefile {{device}}- Finds stable aliases that resolve to the same current device without guessing from link text.
lsblk -e7- Excludes loop devices while preserving the real block hierarchy and selected identity columns.
mdadm --examine- Reads member metadata without assembling or changing an array.
Values stay on this page and are never sent or saved.
sudo lsblk -e7 -o NAME,PATH,SIZE,TYPE,FSTYPE,MOUNTPOINTS,PKNAME,MODEL,SERIAL,WWN && printf '\nStable aliases\n' && for link in /dev/disk/by-id/*; do [ "$(readlink -f "$link")" = "$(readlink -f {{device}})" ] && printf '%s -> %s\n' "$link" "$(readlink "$link")"; done && printf '\nOwnership\n' && sudo findmnt --source {{device}}; sudo pvs --noheadings -o pv_name,vg_name 2>/dev/null; sudo mdadm --examine {{device}} 2>&1 | sed -n '1,20p'; sudo zpool status -P 2>/dev/nullNAME PATH SIZE TYPE FSTYPE MOUNTPOINTS PKNAME MODEL SERIAL WWN sdc /dev/sdc 7.3T disk linux_raid_member ST8000NM ZA1A0001 0x5000c500bb01 └─md0 /dev/md0 7.3T raid1 xfs /srv/archive sdc Stable aliases /dev/disk/by-id/wwn-0x5000c500bb01 -> ../../sdc Ownership ARRAY /dev/md0 metadata=1.2 UUID=9e5b... name=archive:0
Checkpoint: Logical member and physical identity are unambiguous
udevadm info --query=property --name={{device}} | grep -E '^(ID_SERIAL|ID_WWN|ID_PATH)='Continue whenOne intended serial, WWN, controller path, bay, storage owner, and workload are recorded.
Stop whenThe path is a partition or virtual disk when a physical member is required, or any identity mapping is missing.
If this step fails
The device appears under a multipath or hardware RAID virtual disk.
Likely causeThe operating system is not directly addressing one physical drive.
multipath -ll 2>/dev/nulllspci -nn | grep -iE 'raid|sas|scsi'
ResolutionUse the owning multipath or controller management procedure and map the physical member through its official interface.
Security notes
- Restrict raw topology and serial inventory to operators who need physical storage access.
Alternatives
- Export the same map from configuration management or an enclosure inventory if it is current and verified against live udev data.
Stop conditions
- Do not proceed with tests or replacement while the device owner is unknown.
command
Install smartmontools and discover the correct device interface
Install the distribution package, record its version and drive database date, then scan supported devices. USB bridges, RAID controllers, SAS, ATA, and NVMe may require different smartctl device types.
Why this step matters
smartmontools translates several storage protocols and vendor conventions. Recording its release and scan result makes later interpretation reproducible and reveals when a controller-specific pass-through type is required.
What to understand
`--scan-open` attempts discovery and prints the suggested `-d` device type. That type may be essential behind RAID controllers or bridges.
The packaged drive database contains model-specific attribute interpretation. Keep it within the supported update path rather than downloading an unreviewed parser.
Installation does not automatically configure useful smartd schedules or alert routing; monitoring is a later owned change.
System changes
- Installs smartctl, smartd, the drive database, documentation, and packaged systemd integration.
Syntax explained
smartctl --version- Reports application release, build environment, and database information used to reproduce output.
smartctl --scan-open- Discovers devices it can open and suggests protocol/controller types without running self-tests.
sudo apt-get update && sudo apt-get install --yes smartmontools && smartctl --version | sed -n '1,8p' && sudo smartctl --scan-opensmartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.8.0-65-generic] Copyright (C) 2002-23, Bruce Allen, Christian Franke smartmontools release 7.4 dated 2023-08-01 smartmontools SVN rev 5530 dated 2023-08-01 smartmontools build host: x86_64-linux-gnu /dev/sdc -d scsi # /dev/sdc, SCSI device /dev/nvme0 -d nvme # /dev/nvme0, NVMe device
Checkpoint: The target appears through a supported device interface
sudo smartctl --scan-openContinue whenThe physical target or controller mapping appears with a usable type and no permission or pass-through error.
Stop whenThe tool sees only a virtual disk and the physical member cannot be mapped safely.
If this step fails
The scan omits a disk visible in lsblk.
Likely causeA bridge or controller blocks SMART pass-through, or the path requires an explicit device type.
smartctl -d test {{device}}lspci -nn | grep -iE 'raid|sas|sata|nvme'
ResolutionConsult the official controller or smartmontools support matrix and use its read-only physical-drive interface.
Security notes
- smartd can invoke notification scripts with privileges; do not enable unreviewed handlers as part of package installation.
Alternatives
- Use nvme-cli or the official RAID-controller tool when it is the authoritative physical-device interface, while retaining the same evidence gates.
Stop conditions
- Do not interpret a virtual-disk SMART summary as the health of each hidden physical member.
verification
Capture the complete health baseline and exit bits
Collect identification, capacity, overall health, attributes, error logs, self-test logs, temperature, power-on hours, and the smartctl exit code before starting another test. Store raw output in the restricted incident record.
Why this step matters
Complete pre-test output preserves the device's own evidence before a test, reset, reboot, or replacement changes logs. The smartctl exit code is a bitmask and can signal error logs even when human-readable health says PASSED.
What to understand
`-x` requests the extended information available for the detected transport: identity, attributes, logs, capabilities, temperature, and tests.
PIPESTATUS captures smartctl rather than tee's exit status. A value such as 192 combines set bits for error and self-test logs and must not be treated as a generic command failure.
JSON output supports controlled extraction and trend storage, while the raw text remains useful for vendor documentation and incident exchange.
System changes
- Creates root-owned mode-0600 evidence files under /var/tmp; it does not start a device test.
Syntax explained
-x- Prints all extended SMART information available for the device type.
-j- Produces structured JSON suitable for validated parsing and trend comparison.
PIPESTATUS[0]- Captures smartctl's status before the shell reports the downstream tee result.
Values stay on this page and are never sent or saved.
sudo smartctl -x {{device}} | sudo tee /var/tmp/smart-baseline.txt; status=${PIPESTATUS[0]}; printf 'smartctl_exit=%d\n' "$status"; sudo smartctl -j -x {{device}} > /var/tmp/smart-baseline.json; sudo chmod 0600 /var/tmp/smart-baseline.*SMART overall-health self-assessment test result: PASSED Power_On_Hours 0x0032 089 089 000 Old_age Always - 10422 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 8 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 2 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 2 smartctl_exit=192
Checkpoint: Raw and JSON baselines are complete and protected
sudo test -s /var/tmp/smart-baseline.txt -a -s /var/tmp/smart-baseline.json && sudo stat -c '%a %U %n' /var/tmp/smart-baseline.*Continue whenBoth files are non-empty, owned by root, mode 600, and the smartctl bitmask is recorded.
Stop whenThe device cannot be opened, identity differs, output is truncated, or critical data is not backed up.
If this step fails
The pipeline prints health output but reports a nonzero exit.
Likely causesmartctl encodes health, threshold, error-log, and self-test conditions as bit flags rather than one simple success code.
smartctl -x {{device}} >/dev/null; echo $?smartctl --help | tail -40
ResolutionDecode and retain each set bit using the current official manual; do not discard the report as a mere shell error.
Security notes
- Mode 0600 protects serial, firmware, topology, and failure evidence from ordinary local users.
Alternatives
- Stream the files into an approved restricted incident system after redaction while keeping a hash of the authoritative originals.
Stop conditions
- Do not start a self-test until baseline identity and backup readiness are confirmed.
verification
Correlate SMART evidence with kernel and storage errors
Compare attribute changes and self-test failures with kernel resets, link errors, filesystem reports, and redundancy-layer counters. A single normalized SMART value is not a universal cross-vendor diagnosis.
Why this step matters
Media, interface, controller, power, and filesystem failures can look alike. Time and LBA correlation across the kernel, SMART logs, and storage owner reduces both premature replacement and dangerous dismissal.
What to understand
Repeated read failures at the same LBA plus a failed extended self-test strongly implicate unreadable media. Interface CRC growth and bus resets may instead implicate cabling or controller paths.
Inspect a bounded time window so old unrelated boot messages do not overwhelm current evidence. Preserve the complete logs separately before filtering.
Redundancy layers may report corrected errors that applications never saw. Those corrections still show a component or path that needs attention.
System changes
- No persistent changes; correlates existing logs and self-test history.
Syntax explained
journalctl -k --since- Reads kernel messages from a defined incident window with timestamps.
smartctl -l error -l selftest- Reads device-maintained error and self-test histories without launching a new test.
Values stay on this page and are never sent or saved.
sudo journalctl -k --since '-24 hours' --no-pager | grep -iE '({{deviceKernelName}}|ata[0-9]|nvme|I/O error|medium error|reset|timeout|uncorrect|failed command)' | tail -80; sudo dmesg --level=err,warn | tail -80; sudo smartctl -l error -l selftest {{device}}Jul 28 03:17:06 node1 kernel: sd 3:0:0:0: [sdc] tag#17 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK Jul 28 03:17:06 node1 kernel: blk_update_request: I/O error, dev sdc, sector 782314048 op 0x0:(READ) SMART Self-test log # 1 Extended offline Completed: read failure 90% 10418 782314048
Checkpoint: Every replacement signal has device, time, and owner correlation
sudo smartctl -l error -l selftest {{device}} && sudo journalctl -k --since '-24 hours' --no-pager | tail -100Continue whenThe operator can distinguish media, transport, owner-layer, and application evidence and document confidence.
Stop whenMultiple devices share the symptom, power or controller errors are active, or evidence points to the wrong serial.
If this step fails
Several disks reset at the same timestamp.
Likely causeA shared HBA, expander, backplane, cable, power rail, firmware, or host event is more likely than simultaneous independent media failures.
journalctl -k --since '-30 minutes'lspci -vv -s $(lspci | grep -i 'SAS\|SATA\|RAID' | head -1 | cut -d' ' -f1)
ResolutionStabilize and diagnose the shared path before replacing multiple drives or starting concurrent stress tests.
Security notes
- Kernel logs can contain filesystem paths and tenant identifiers; protect the unredacted bundle.
Alternatives
- Add controller event logs, enclosure telemetry, and power monitoring when they are official sources of the physical path.
Stop conditions
- Do not call a disk healthy solely because overall SMART health is PASSED while correlated reads fail.
command
Run and inspect a short self-test
Start a short device self-test only after the backup and workload risk are understood. Wait for the device-reported completion estimate, then read the self-test log rather than treating command acceptance as success.
Why this step matters
A short self-test offers a bounded firmware-level check with less load than an extended surface scan. Reading its eventual log closes the asynchronous operation.
What to understand
smartctl reports an estimated completion time; the shell command returning successfully only means the device accepted the request.
Workload I/O can continue on many devices, but latency and error behavior should be monitored during the test.
A successful short test does not clear a previous extended-test failure or prove every LBA is readable.
System changes
- Starts a device-resident short self-test and adds its result to the self-test log.
Syntax explained
-t short- Requests the device's supported short offline self-test.
-l selftest- Reads the historical result after the estimated completion window.
-H- Also reads the overall health assessment without replacing detailed logs.
Values stay on this page and are never sent or saved.
sudo smartctl -t short {{device}} && sleep {{shortTestWaitSeconds}} && sudo smartctl -l selftest -H {{device}}Please wait 2 minutes for test to complete. Test will complete after Tue Jul 28 03:28:14 2026 UTC SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 10423 -
Checkpoint: The newest short test has a completed result
sudo smartctl -l selftest {{device}}Continue whenEntry number one refers to the current test and is completed without error, or its exact failure is captured.
Stop whenThe device disappears, kernel errors accelerate, temperature exceeds policy, or the test reports an unreadable region.
If this step fails
The latest log entry says interrupted by host reset.
Likely causeThe drive or bus reset, the host rebooted, or power management interrupted the operation.
journalctl -k --since '-15 minutes'smartctl -c {{device}}
ResolutionInvestigate transport and power stability before deciding whether a retry is safe.
Security notes
- Self-tests can affect latency; obtain workload-owner approval even though they do not intentionally overwrite user data.
Alternatives
- Skip directly to protected replacement when existing uncorrectable reads already justify action and another test would add risk.
Stop conditions
- Abort diagnostic load if the device becomes the last unstable copy.
decision
Run an extended self-test only when the failure risk is acceptable
An extended test reads the device surface and can take hours. Run it only when redundancy or backup can tolerate a device that worsens, and never delay replacement of a device already producing uncorrectable application reads merely to obtain another score.
Why this step matters
An extended test can localize surface read failures that a short test misses, but its sustained access may be the final workload a marginal disk tolerates. The data-protection gate therefore matters more than completing the test.
What to understand
`smartctl -c` provides the device's estimate; large nearline disks may require many hours and power-management policies must not interrupt them.
A repeated failure at the same LBA is actionable even if overall health remains PASSED. An interrupted test is diagnostic evidence rather than success.
Do not overlap an extended self-test with a ZFS scrub, RAID check, controller patrol read, backup scan, or rebuild unless the storage design explicitly permits the combined load.
System changes
- Starts a device-resident extended media self-test and records completion or failure in device logs.
Syntax explained
-c- Prints capabilities and recommended polling time without starting a test.
-t long- Requests the device's extended self-test, usually including a broad media read.
-l error- Reads recorded device errors to correlate the final self-test result.
Values stay on this page and are never sent or saved.
sudo smartctl -c {{device}} | grep -i 'Extended self-test routine'; sudo smartctl -t long {{device}} && sleep {{longTestWaitSeconds}} && sudo smartctl -l selftest -l error -H {{device}}Extended self-test routine recommended polling time: ( 735) minutes. SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 10424 782314048 # 2 Short offline Completed without error 00% 10423 -
Checkpoint: The extended result is recorded or deliberately skipped
sudo smartctl -l selftest -l error {{device}}Continue whenA completed result with no errors, a reproducible failure with location, or an approved skip because data risk outweighed diagnostic value.
Stop whenNew I/O errors, thermal alarms, disappearance, or failing redundancy occur during the test.
If this step fails
The test reaches the same remaining percentage and fails at the same LBA.
Likely causeA persistent unreadable media region exists.
smartctl -l selftest {{device}}smartctl -l error {{device}}
ResolutionProtect data and replace the device; repeated full tests are unlikely to restore the sector and add load.
Security notes
- Extended test timing can reveal maintenance windows; keep detailed fleet schedules restricted.
Alternatives
- Use a controller patrol read when it is the supported owner, but do not run two competing surface scans.
Stop conditions
- Never require an extended test before replacing a disk that is already losing required data.
decision
Classify the evidence as keep, monitor, or replace
Make a documented decision from trends, failed tests, uncorrectable reads, link symptoms, redundancy state, workload impact, age, warranty, and vendor guidance. SMART PASSED is not a guarantee that all reads succeed.
Why this step matters
A written evidence-based classification keeps SMART from becoming either an automatic panic button or an excuse to ignore real I/O failure. It also makes warranty, maintenance, and future trend review defensible.
What to understand
Replace when overall health fails, uncorrectable reads affect data, self-tests repeatedly fail, the owner ejects the member, or risk policy says the trend is unacceptable.
Monitor when evidence is noncritical and stable, backup and redundancy are sound, and a specific alert threshold and review time are assigned.
Investigate the path when interface errors, multiple-device resets, power, thermal, or controller symptoms dominate. Replacing media alone may reproduce the failure.
System changes
- No device changes; creates the formal keep, monitor, replace, or path-investigation decision and supporting evidence.
Syntax explained
smartctl -j -x- Provides structured comprehensive data; field availability depends on the transport and device.
jq '{...}'- Selects a review summary without replacing the protected raw JSON and text evidence.
Values stay on this page and are never sent or saved.
sudo smartctl -j -x {{device}} | jq '{model:.model_name,serial:.serial_number,passed:.smart_status.passed,power_on_hours:.power_on_time.hours,temperature:.temperature.current,ata_errors:.ata_smart_error_log.summary.count,self_tests:[.ata_smart_self_test_log.standard.table[]? | {type:.type.string,status:.status.string,lifetime_hours:.lifetime_hours,lba:.lba}]}'{
"model": "ST8000NM000A",
"serial": "ZA1A0001",
"passed": true,
"power_on_hours": 10424,
"temperature": 38,
"ata_errors": 3,
"self_tests": [
{"type":"Extended offline","status":"Completed: read failure","lifetime_hours":10424,"lba":782314048}
]
}
Decision: REPLACE — repeated uncorrectable read at the same LBA and kernel I/O error.Checkpoint: The decision states evidence, confidence, owner, and deadline
Continue whenA named operator approves keep/monitor/replace, references raw evidence and backup, and assigns the next action.
Stop whenThe decision relies on one attribute, an undocumented threshold, or the wrong physical identity.
If this step fails
Teams disagree because vendor raw attributes have unfamiliar values.
Likely causeRaw fields are model-specific and internet folklore is being treated as a universal specification.
smartctl -x {{device}}smartctl --version
ResolutionUse the current smartmontools interpretation, official vendor documentation, trends from the same model, self-tests, and observed I/O behavior.
Security notes
- Do not expose full asset serials in a public warranty or community post.
Alternatives
- Use a fleet risk score only when its inputs and escalation rules are explainable and raw evidence remains available.
Stop conditions
- Do not postpone replacement of a failed active member to improve diagnostic certainty while redundancy is degraded.
warning
Protect data and quiesce the owning workload
Before hardware work, stop new writes through the application's supported process, complete the most recoverable copy available, flush buffers, and record the final redundancy state. A failing disk may not survive a full copy, so prioritize irreplaceable data.
Why this step matters
Quiescing the workload and preserving a recoverable copy reduce the chance that replacement turns a component fault into inconsistent data or an unrecoverable outage.
What to understand
Use the application's supported stop or backup hook; `sync` flushes kernel buffers but does not create database or distributed-application consistency.
Confirm the redundancy state immediately before release. A second member may have degraded since the initial diagnosis.
Prioritize irreplaceable or business-critical data when a failing disk cannot sustain a complete backup read.
System changes
- Stops `{{workloadService}}`, flushes buffered writes, and leaves the dataset quiesced for owner-specific detach.
Syntax explained
systemctl stop {{workloadService}}- Requests the service's configured orderly stop rather than killing arbitrary processes.
sync- Requests writeback of dirty filesystem buffers; it is not application-level transaction consistency.
Values stay on this page and are never sent or saved.
sudo findmnt --source {{device}} -o TARGET,SOURCE,FSTYPE,OPTIONS; sudo systemctl stop {{workloadService}} && sudo sync && sudo systemctl is-active {{workloadService}}; sudo mdadm --detail /dev/md0 2>/dev/null || true; sudo zpool status -P 2>/dev/null || trueTARGET SOURCE FSTYPE OPTIONS
/srv/archive /dev/md0 xfs rw,relatime
inactive
/dev/md0:
Raid Level : raid1
Array Size : 7813895512
Active Devices : 2
Working Devices : 2
Failed Devices : 0Checkpoint: Workload is quiesced and the remaining copy is viable
systemctl is-active {{workloadService}}; cat /proc/mdstat; zpool status -x 2>/dev/null || trueContinue whenService is inactive, backup evidence is current, and the owner reports enough healthy data to perform the planned change.
Stop whenThe workload cannot stop cleanly, backup failed, or another member is unhealthy.
If this step fails
The service remains active or restarts automatically.
Likely causeA supervisor, dependency, container orchestrator, or restart policy owns it.
systemctl status {{workloadService}} --no-pagersystemctl list-dependencies --reverse {{workloadService}}
ResolutionCoordinate with the actual supervisor and verify application consistency before proceeding.
Security notes
- Backup copies created during failure response require the same confidentiality and access controls as production.
Alternatives
- Use an application-consistent snapshot or replica failover when it is already tested and safer than a full stop.
Stop conditions
- Do not detach a member while uncontrolled writes continue.
decision
Detach or offline the device through the owning storage stack
Follow the exact owner-specific procedure: fail and remove an md member, replace a ZFS leaf, migrate an LVM physical volume, remove swap, or use the hardware RAID controller. A plain single-disk filesystem has no redundant detach and normally requires restore to another device.
Why this step matters
Only the owning layer knows how to preserve or intentionally reduce redundancy. A generic raw-device removal can leave stale metadata, trigger reconstruction incorrectly, or destroy the wrong abstraction.
What to understand
For mdraid, the approved flow generally marks a member failed and removes it; for ZFS, offline or replace semantics differ; LVM may require pvmove and vgreduce; hardware RAID uses controller state.
A plain filesystem on one disk has no healthy mirror to rebuild. Its safe path is normally restore or image to new media, not pretending to detach redundantly.
Capture the owner's state before and after its exact documented action, and keep the workload stopped until the device is logically released.
System changes
- This decision authorizes an owner-specific membership change outside the generic guide; it may reduce redundancy or remove the old member.
Owner decision recorded: stack=mdraid container=/dev/md0 member=/dev/disk/by-id/wwn-0x5000c500bb01 pre_removal_state=clean approved_action="fail then remove member using mdadm" backup_restore_test=BR-2026-07-28-014 console_available=yes
Checkpoint: The old member is logically released and the remaining state is understood
Continue whenThe owning stack identifies the exact old serial/path as removable and reports the expected degraded or migration-complete state.
Stop whenNo stack-specific command has been reviewed, the selected member differs, or redundancy is already below the safe threshold.
If this step fails
The owner refuses to remove the member.
Likely causeThe path is wrong, the member is still active, a migration or rebuild runs, or removal would violate minimum redundancy.
cat /proc/mdstatzpool status -LPv 2>/dev/nullpvs -o pv_name,vg_name,pv_used
ResolutionDo not force removal. Resolve the owner state or move to backup-based recovery.
Security notes
- Membership commands are destructive privileged operations and require change approval plus peer verification.
Alternatives
- Power down and use the platform's cold-replacement procedure when hot removal is not explicitly supported.
Stop conditions
- Never substitute `wipefs`, `pvremove`, `mdadm --zero-superblock`, or `zpool detach` for an unverified release procedure.
instruction
Replace the positively identified physical device
Use the recorded serial, WWN, controller path, enclosure bay, and approved locator procedure. Power down when the controller, enclosure, disk, or platform does not explicitly support hot swap.
Why this step matters
Logical release does not identify the chassis slot by itself. Physical removal requires an independent serial and bay confirmation because pulling the surviving healthy member can immediately lose the dataset.
What to understand
Provide remote hands with model, serial, WWN, enclosure, slot, host, and whether power must be off. Require them to read back the serial before removal.
A locator LED is helpful but can be mis-addressed. Use it only after software identity and enclosure mapping agree.
Bag and label the removed drive immediately. Warranty return, forensics, data sanitization, and evidence retention may conflict and need an approved disposition.
System changes
- Removes the failed physical disk and inserts the approved replacement into the documented bay; may involve a host shutdown.
Remote-hands confirmation removed_serial=ZA1A0001 removed_bay=ENC1-SLOT07 inserted_serial=ZA1A0099 inserted_bay=ENC1-SLOT07 host_power_state=off chain_of_custody=HW-2026-1842
Checkpoint: Removed and inserted serials match the approved work order
Continue whenHost, enclosure, bay, old serial, new serial, power state, timestamp, and handler are recorded.
Stop whenThe visible serial differs, the wrong locator activates, or hot-swap support is uncertain.
If this step fails
The wrong disk disappears from the operating system.
Likely causeThe slot map or remote-hands identification was wrong.
lsblk -o NAME,SIZE,MODEL,SERIAL,WWNcat /proc/mdstatzpool status -LPv 2>/dev/null
ResolutionStop immediately, preserve power state and logs, reinsert only under the owner's recovery procedure, and escalate before touching another bay.
Security notes
- Treat the removed device as sensitive data regardless of whether the OS can still read it.
Alternatives
- Use vendor-qualified onsite service when enclosure design, warranty, or cluster policy forbids operator replacement.
Stop conditions
- Do not proceed if remote hands cannot read and confirm the target serial.
verification
Verify the replacement identity before rebuilding
Let udev settle, identify the new disk from serial and WWN, confirm its usable size, ensure no unexpected signatures exist, and capture a fresh SMART baseline. Do not add it to the storage stack until every identity check agrees.
Why this step matters
New hardware can be wrong, smaller, previously used, damaged in transit, or mapped through a different controller path. Verification before rebuild protects the surviving data copy from a bad candidate.
What to understand
Compare byte capacity, not marketing labels. Some equal-nominal disks differ enough in usable sectors to be rejected by the owner.
`wipefs -n` reports signatures without erasing them. Any signature requires provenance review even on a replacement from inventory.
Power-on hours, temperature, model, firmware, SMART support, and self-test history should match the expected new or approved refurbished status.
System changes
- No disk writes; waits for udev and reads identity, signatures, capacity, and initial health.
Syntax explained
udevadm settle- Waits for queued device events so identity links and properties are stable before selection.
lsblk -b- Shows exact byte capacity for comparison with the old member.
wipefs -n- Performs a no-act signature scan and preserves unexpected metadata for investigation.
Values stay on this page and are never sent or saved.
sudo udevadm settle && sudo lsblk -b -o NAME,PATH,SIZE,TYPE,FSTYPE,MODEL,SERIAL,WWN && sudo udevadm info --query=property --name={{replacementDevice}} | grep -E '^(ID_MODEL|ID_SERIAL|ID_WWN|ID_PATH)=' && sudo wipefs -n {{replacementDevice}} && sudo smartctl -x {{replacementDevice}} | sed -n '1,80p'sdd /dev/sdd 8001563222016 disk ST8000NM000A ZA1A0099 0x5000c500bb99 ID_MODEL=ST8000NM000A ID_SERIAL=ST8000NM000A_ZA1A0099 ID_WWN=0x5000c500bb99 ID_PATH=pci-0000:03:00.0-sas-phy7-lun-0 SMART overall-health self-assessment test result: PASSED Power_On_Hours: 0
Checkpoint: The replacement is the approved empty equal-or-larger device
blockdev --getsize64 {{replacementDevice}} && smartctl -i -H {{replacementDevice}}Continue whenSerial and WWN match inventory, capacity meets the owner requirement, no unexplained signatures exist, and baseline health is acceptable.
Stop whenIdentity, capacity, firmware policy, health, or signatures differ from the approved replacement.
If this step fails
The new by-id link does not appear.
Likely causeudev has not settled, the device failed enumeration, the controller needs rescan, or the hardware is absent.
udevadm settlejournalctl -k --since '-10 minutes'ls -l /dev/disk/by-id
ResolutionUse the platform's supported rescan or cold boot and diagnose enumeration; never reuse a neighboring existing path.
Security notes
- Do not erase unexpected replacement signatures until prior ownership and retention obligations are resolved.
Alternatives
- Prequalify spares with destructive burn-in on an isolated host before they enter the approved spare pool.
Stop conditions
- Do not admit a disk whose exact identity and usable capacity are not proven.
decision
Restore redundancy through the owning stack and monitor it
Partition or label the replacement exactly as required by the documented owner, add it through that owner's supported command, and watch reconstruction plus the surviving member. This guide does not substitute one universal rebuild command for mdraid, ZFS, LVM, Ceph, and hardware RAID.
Why this step matters
Reconstruction is the period when the surviving data and replacement are under maximum combined pressure. The owning stack must perform it, expose progress, and define success.
What to understand
Replicate partition tables only when the owner requires them and compare destination identity before writing. Whole-disk ZFS and partitioned mdraid use different layouts.
Monitor the surviving disk as closely as the replacement. A read error on the source may turn degraded availability into data loss.
Do not call the array healthy while resync, resilver, migration, or controller rebuild is still active, even if the new disk is listed.
System changes
- The owner-specific procedure writes labels or partitions to the new disk and reconstructs data or parity onto it.
Rebuild evidence stack=mdraid container=/dev/md0 new_member=/dev/disk/by-id/wwn-0x5000c500bb99 state=clean rebuild_status=completed finish=2026-07-28T18:42:11Z mismatch_count=0 active_devices=2 working_devices=2
Checkpoint: The owning layer reports completed reconstruction and full membership
Continue whenNo active rebuild remains, intended member count is restored, error counters are understood, and the new stable identity is active.
Stop whenThe source develops errors, progress stalls, capacity is rejected, or the owner reports mismatch or corruption.
If this step fails
Rebuild speed falls to zero while a member reports errors.
Likely causeThe surviving source is retrying unreadable data, the replacement is unstable, or workload/controller contention blocks progress.
cat /proc/mdstatzpool status -LPv 2>/dev/nulljournalctl -k --since '-15 minutes'
ResolutionReduce writes, preserve evidence, and move to backup or specialist recovery if the only source cannot complete reconstruction.
Security notes
- Do not copy raw array metadata or partition tables through an unaudited one-liner; a reversed source and destination can destroy the healthy member.
Alternatives
- Restore onto a newly built storage set when reconstruction from the marginal survivor has lower expected recoverability.
Stop conditions
- Never remove the last known-good member to retry a stalled rebuild.
verification
Validate the new disk, restored redundancy, and workload
After rebuild, run a short self-test on the replacement, confirm a clean redundancy state, inspect new kernel messages, restart the workload, and perform a representative read plus application check.
Why this step matters
A clean rebuild headline does not prove the replacement can self-test, the kernel path is stable, or the application can read real data. Closing all layers catches latent installation and consistency failures.
What to understand
Run a short test after reconstruction so it does not compete with the rebuild and its result refers to the installed path.
Review only the post-change kernel window and compare with the captured baseline. New resets, timeouts, or errors indicate the incident remains open.
Restart the workload through its normal supervisor and execute a representative health or checksum test, not only systemctl is-active.
System changes
- Starts a short self-test on the new disk and restarts `{{workloadService}}` after redundancy is restored.
Syntax explained
smartctl -t short- Launches a bounded firmware self-test on the installed replacement.
journalctl -k --since- Constrains review to the replacement and rebuild period.
systemctl start- Returns the quiesced workload to its managed service lifecycle.
Values stay on this page and are never sent or saved.
sudo smartctl -t short {{replacementDevice}} && sleep {{shortTestWaitSeconds}} && sudo smartctl -l selftest -H {{replacementDevice}}; sudo mdadm --detail /dev/md0 2>/dev/null || true; sudo zpool status -v 2>/dev/null || true; sudo systemctl start {{workloadService}} && sudo systemctl is-active {{workloadService}} && sudo journalctl -k --since '-30 minutes' --no-pager | grep -iE 'I/O error|reset|timeout|uncorrect' || true# 1 Short offline Completed without error 00% 0 - State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 active No new kernel storage errors found since replacement.
Checkpoint: Hardware, owner, kernel, and application all pass
smartctl -l selftest {{replacementDevice}} && systemctl is-active {{workloadService}}Continue whenNew self-test completed without error, full redundancy remains, no new kernel storage errors exist, and application checks pass.
Stop whenAny new error, degraded state, failed self-test, or application inconsistency appears.
If this step fails
The new disk passes SMART but the application fails reads.
Likely causeReconstruction preserved corrupted logical data, filesystem or application consistency is broken, or the issue was not the disk.
journalctl -u {{workloadService}} -n 100 --no-pagerdmesg --level=err,warn | tail -100
ResolutionKeep the workload controlled, verify the storage and filesystem layer, and restore affected application data from known-good backup.
Security notes
- Dispose of the old disk only after incident and recovery owners release it.
Alternatives
- Run a full owner-layer scrub or consistency check in a later controlled window after immediate service restoration.
Stop conditions
- Do not close the incident on device health alone without an application-level read test.
Finish line
Verification checklist
sudo smartctl -x {{replacementDevice}}The identity matches the installed serial and bay, overall health is supported and passing, no self-test or error-log failure exists, and temperature plus power-on hours are plausible for the new device.sudo mdadm --detail /dev/md0 2>/dev/null || sudo zpool status -P 2>/dev/null || sudo pvs -o pv_name,vg_name,pv_size,pv_freeThe actual owning stack reports its intended healthy and redundant state, the replacement is a member, and the removed identity is no longer active.systemctl is-active {{workloadService}} && journalctl -k --since '-30 minutes' --no-pager | grep -iE 'I/O error|medium error|reset|timeout|uncorrect' || trueThe workload is healthy and representative reads succeed, with no new device, link, timeout, or uncorrectable I/O errors after reconstruction.Recovery guidance
Common problems and safe checks
smartctl reports that the device is unsupported or returns little information through a USB or RAID bridge.
Likely causeThe path exposes a bridge or virtual disk and smartctl cannot infer the correct pass-through device type.
smartctl --scan-opensmartctl -d test {{device}}lsusb -t; lspci -nn | grep -iE 'raid|sas|sata|nvme'
ResolutionUse the device type identified by the official smartctl/controller documentation or the controller's official physical-drive tool. Do not assume an empty report means a healthy disk.
SMART says PASSED while the kernel and application report I/O errors.
Likely causeOverall health thresholds have not tripped, the fault is transport-related, or the failing region is recorded only in the error/self-test logs.
smartctl -x {{device}}journalctl -k --since '-24 hours' | grep -iE 'I/O error|reset|timeout'mdadm --detail /dev/md0 2>/dev/null || zpool status -v 2>/dev/null
ResolutionTreat reproducible I/O failures and owner-layer errors as real. Preserve LBA and path correlation, check cabling/controller/power, and replace or isolate according to data risk rather than the PASSED headline.
A self-test remains interrupted or never appears in the log.
Likely causeThe device reset, lost power, entered standby, another test superseded it, or the bridge does not pass self-test commands reliably.
smartctl -c -l selftest {{device}}journalctl -k --since '-2 hours' | tail -100smartctl --scan-open
ResolutionCorrect the transport or power issue and reconsider the risk before retrying. Repeated interruptions plus kernel resets may be stronger replacement evidence than another forced test.
The storage array remains degraded after the new disk appears healthy.
Likely causeThe replacement was not admitted by the owning layer, partition geometry differs, metadata is stale, or reconstruction stopped on an error from the surviving copy.
lsblk -o NAME,SIZE,TYPE,FSTYPE,MODEL,SERIAL,WWNcat /proc/mdstatzpool status -LPv 2>/dev/null
ResolutionFollow the owner-specific recovery path, compare size and layout with a healthy member, and preserve reconstruction errors. Never remove the remaining good member to make status look symmetric.
Reference
Frequently asked questions
Does SMART PASSED mean the drive is safe?
No. It means the device's overall threshold test has not reported failure. Kernel I/O errors, self-test failures, pending or uncorrectable sectors, controller resets, and application corruption still require investigation.
Should I run a long test before every replacement?
No. A long test adds sustained reads and can worsen a marginal last copy. Run it only when backup and redundancy tolerate failure and the result will change the decision.
Can I replace /dev/sdc with another /dev/sdc?
The kernel may reuse that name, but it is not identity. Use serial, WWN, stable by-id path, controller path, and bay mapping for both old and new devices.
Why does the guide not give one rebuild command?
Because the correct owner may be mdraid, ZFS, LVM, Ceph, hardware RAID, or no redundant layer. A universal command would be actively dangerous; the shared procedure is identity, risk, evidence, and verification.
Recovery
Rollback
Disk replacement is not generally reversible after a failed member has been removed and the new device has joined the storage stack. Preserve the old disk under chain of custody, keep the last good replica online, and use the independent backup when reconstruction cannot complete.
- Before physical removal, cancel the change by restarting the quiesced workload only if the storage owner still reports the original safe state and the device has not been failed or detached.
- If the new disk is rejected before rebuild, do not modify the surviving member. Remove the unaccepted replacement through the owning layer, retain the original evidence, and obtain a correct equal-or-larger device.
- If reconstruction fails because the surviving copy develops errors, stop nonessential writes, preserve logs and device images where policy permits, and restore prioritized data from the validated independent backup to a new storage set.
- Do not reinsert the old disk as an automatic rollback. Its metadata may be stale and reassembly can overwrite newer good state; any reuse requires an owner-specific recovery plan.
Evidence