Build and operate a ZFS storage pool with snapshots, scrubs, replacement, and recovery
Create a mirrored OpenZFS data pool from stable device identities, separate data into policy-aware datasets, recover individual files from snapshots, verify every block with scheduled scrubs, replace a failed member without confusing device names, and rehearse a read-only import path before an incident.
Produce a documented, observable ZFS pool whose redundancy, snapshot recovery, scrub history, device replacement, and import behavior have all been exercised without treating snapshots as backups or destructive force flags as routine recovery tools.
- Ubuntu Server 24.04 LTS
- OpenZFS 2.2, 2.3
- Two empty, same-size data devices Use dedicated disks or virtual block devices that contain no required partition table, filesystem, LVM signature, RAID metadata, encryption header, or previous ZFS label. The mirror is limited by its smaller member.
lsblk -e7 -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS,MODEL,SERIAL,WWN - A separate, tested backup A mirror protects availability after one member fails; it does not protect against deletion, ransomware, application corruption, pool destruction, fire, or operator error. Restore one representative file from backup before creating the pool.
- Console access and a maintenance window Pool creation and physical replacement can remove the wrong device permanently. Keep out-of-band console access, record bay-to-serial mapping, and pause workloads before any destructive storage action.
- Stable device paths Resolve both devices under /dev/disk/by-id and record serial numbers. Never build the procedure around /dev/sdX names because enumeration can change after reboot or replacement.
readlink -f {{diskA}} && readlink -f {{diskB}}
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A two-device mirrored pool whose members are named by persistent WWN paths, not boot-order-dependent sdX aliases. The pool uses an explicit sector alignment, automatic trim policy, and a root dataset that is never used as an undifferentiated dumping ground.
- A child dataset with explicit mounting, compression, access-time, quota, and reservation policy. A named snapshot protects a known maintenance point, while a file-level restoration exercise proves recovery without destroying newer data.
- An operations loop that records scrubs, maps a fault to physical identity, replaces one mirror member, observes the resilver to completion, and rehearses a read-only import on an isolated recovery path.
- Operators can explain the difference between a pool, top-level vdev, mirror leaf, dataset, snapshot, scrub, and resilver, and can identify which loss conditions redundancy can and cannot survive.
- The final evidence shows an ONLINE mirror, zero unrepairable errors, a completed scrub, an enabled periodic timer, reviewed dataset properties, a visible snapshot, and a canary recovered from that snapshot.
- A failed disk can be mapped from ZFS path to serial and enclosure path before removal, while recovery escalation begins from a no-mount read-only import rather than force-importing or rewinding a pool blindly.
Architecture
How the parts fit together
ZFS combines a volume manager with checksummed filesystems. The pool allocates data across top-level vdevs; this guide uses exactly one mirrored top-level vdev, so each allocated block has a copy on both leaf devices. Datasets sit above the pool and provide independent namespace and property boundaries. Snapshots preserve references to existing blocks through copy-on-write, while scrubs read all allocated blocks and compare checksums. If one copy is bad, the mirror supplies a verified copy for repair. A replacement disk receives current allocated data through resilvering. None of these mechanisms creates an off-host backup, so backup and restore evidence remains a separate prerequisite.
- Applications write to the mounted child dataset. ZFS compresses and checksums blocks, allocates them through the pool, and stores a copy on each mirror leaf.
- A snapshot preserves block references at a named transaction point. Later writes allocate new blocks, leaving snapshot-referenced blocks available for file-level recovery.
- A scrub walks allocated data, validates checksums, and repairs a bad copy from the surviving mirror copy when possible. Status and events become monitoring evidence.
- When a leaf fails, its stable identity is mapped to hardware, the replacement is introduced with zpool replace, and a resilver reconstructs current allocated data before redundancy is considered restored.
- In a recovery rehearsal, pool discovery happens through stable paths, import begins read-only with datasets unmounted, and normal read-write import occurs only after status and ownership are understood.
Assumptions
- The target is an Ubuntu 24.04 LTS data host using distribution-supported OpenZFS. Root-on-ZFS, clustered storage, dRAID, RAIDZ expansion, native encryption, special vdevs, deduplication, and boot-environment recovery require separate design decisions.
- Both selected devices are empty, dedicated, at least as large as the data set plus growth and snapshot headroom, and visible through persistent by-id identities that map to documented physical slots.
- A current independent backup has been restored outside this pool. Snapshots remain in the same failure domain as the live dataset and therefore do not satisfy backup or disaster-recovery requirements.
- The host has sufficient memory, cooling, power protection, and I/O capacity for the intended workload, and the scrub window has been measured rather than scheduled across latency-sensitive peaks.
- A human owns pool alerts, device replacement, scrub review, snapshot retention, capacity forecasting, package support, and recovery rehearsals. Enabling a timer without an alert owner does not create an operational control.
Key concepts
- Pool and top-level vdev
- A pool is the storage and dataset namespace. It stripes allocations across top-level vdevs, so every top-level vdev must have intentional redundancy. Adding a single unmirrored data vdev to a mirrored pool can make the entire pool depend on that one disk.
- Mirror leaf
- A physical or virtual block device inside a mirror. Either one of two healthy leaves can supply a block, but the mirror is not a second independent copy: controller faults, accidental commands, theft, fire, and simultaneous failures can still destroy both.
- Copy-on-write snapshot
- A read-only dataset state that preserves references to blocks as they existed at one transaction point. Space consumption grows as the live dataset diverges. Deleting a live file may not free its blocks while snapshots still reference them.
- Scrub
- A deliberate read of allocated data that verifies block checksums. On redundant vdevs, a verified alternate copy can repair corruption. The operation is I/O-intensive and must be monitored to completion.
- Resilver
- A reconstruction limited to data ZFS knows a new, replaced, or returned device is missing. Completion restores redundancy only if the remaining members stayed readable and status reports no unrepairable errors.
- Read-only no-mount import
- A recovery posture using readonly=on and -N to inspect pool metadata and datasets without mounting filesystems or accepting ordinary writes. It is evidence gathering, not permission to ignore an active owner or force import.
Before you copy
Values used in this guide
{{poolName}}Unique lowercase pool name used by zpool commands and the packaged scrub timer.
Example: atlas{{diskA}}Persistent /dev/disk/by-id path for the first mirror member; verify the resolved path, serial, WWN, and bay before use.
Example: /dev/disk/by-id/wwn-0x5000c500aa01{{diskB}}Persistent /dev/disk/by-id path for the second mirror member, on a separate physical disk of equal or greater usable size.
Example: /dev/disk/by-id/wwn-0x5000c500aa02{{replacementDisk}}Persistent path for the empty replacement device, verified to be the intended new serial and at least as large as the failed member.
Example: /dev/disk/by-id/wwn-0x5000c500aa03{{dataset}}Full child dataset name. Keep workloads in separate children so snapshot, quota, replication, and recovery policy can differ.
Example: atlas/services{{mountpoint}}Absolute directory where the child dataset is mounted. It must not conceal required files already present in that directory.
Example: /srv/atlas{{snapshotName}}Short, immutable operational label without @, normally naming the change boundary or timestamp.
Example: before-maintenanceSecurity and production boundaries
- Pool administration requires root-equivalent storage authority. Restrict zpool, zfs, raw-device, enclosure-management, and recovery-host access; a user who can replace or import devices can expose or destroy every dataset.
- Do not place secrets in unencrypted datasets or snapshots merely because the pool has checksums. Native encryption has key-loading and recovery implications not covered here, while disk-level encryption changes replacement and boot procedures.
- Snapshots can retain deleted confidential files beyond the application's retention expectation. Align snapshot expiry, legal hold, replication, and backup deletion with the data owner's policy.
- Treat device serials, pool history, event logs, and recovery output as infrastructure data. Redact tenant names and sensitive mount paths before sharing, but preserve the identifiers required to prove which disk was handled.
Stop before continuing if
- Stop before zpool create if either selected device has a signature, mount, swap entry, mdraid/LVM ownership, existing ZFS label, unexplained data, or an identity that cannot be mapped to a physical slot.
- Stop if the backup cannot restore a representative file independently of the new pool, or if no console and change window exist for destructive storage work.
- Stop replacement if the pool has no known-good surviving copy, multiple faulted members, active unrepairable errors, a replacement smaller than the old leaf, or uncertainty about the failed serial.
- Stop recovery import if another host may own the pool. Never use -f, rewind flags, or write-mode import merely to suppress an ownership or transaction warning.
- Stop a pool-layout change if the operator cannot explain every top-level vdev. `zpool add` is not the same as `zpool attach`, and an accidental nonredundant top-level vdev can create permanent pool-wide risk.
verification
Inventory disks and prove stable identities
Capture the complete block topology, mounted filesystems, existing signatures, serial numbers, and by-id symlinks before selecting either mirror member. The same evidence becomes the physical replacement map.
Why this step matters
Storage names are labels applied by several layers, not proof of physical identity. Recording topology and stable identifiers before a destructive action prevents the most common catastrophic error: operating on a healthy disk that happened to receive the expected sdX name.
What to understand
lsblk shows holders, filesystems, mountpoints, model, serial, and WWN in one tree. `wipefs -n` is intentionally non-destructive: it reports recognizable signatures without erasing them.
Resolve every by-id link and compare the reported serial with the chassis, enclosure, hypervisor attachment, or cloud-volume identifier. Photograph or export the bay map when remote hands will perform replacement.
An empty-looking mountpoint does not prove an unused block device. LVM, mdraid, swap, encryption, and an exported ZFS pool can own a device without a conventional mounted filesystem.
System changes
- No persistent state changes; the output becomes the approved before-state and device-to-slot evidence for this pool.
Syntax explained
lsblk -e7 -o ...- Excludes loop devices and selects identity, topology, filesystem, and mount columns instead of relying on the default view.
wipefs -n- Performs a no-act signature scan. Omitting -n in a later wipefs command could erase filesystem or RAID signatures.
readlink -f- Shows the current kernel device behind a stable symlink for correlation, not for use as the long-term pool label.
Values stay on this page and are never sent or saved.
sudo lsblk -e7 -o NAME,PATH,SIZE,TYPE,FSTYPE,FSVER,MOUNTPOINTS,MODEL,SERIAL,WWN && printf '\nSelected devices\n' && readlink -f {{diskA}} && readlink -f {{diskB}} && sudo wipefs -n {{diskA}} && sudo wipefs -n {{diskB}}NAME PATH SIZE TYPE FSTYPE MOUNTPOINTS MODEL SERIAL WWN sda /dev/sda 1.8T disk ST2000NM Z4ZAA001 0x5000c500aa01 sdb /dev/sdb 1.8T disk ST2000NM Z4ZAA002 0x5000c500aa02 Selected devices /dev/sda /dev/sdb offset type ---------------------------------------------------------------- No signatures found on /dev/disk/by-id/wwn-0x5000c500aa01. No signatures found on /dev/disk/by-id/wwn-0x5000c500aa02.
Checkpoint: Both selected paths resolve to the intended empty disks
readlink -f {{diskA}} {{diskB}} && sudo wipefs -n {{diskA}} {{diskB}}Continue whenTwo different resolved devices, matching approved serials and sizes, with no signatures or consumers.
Stop whenA path resolves unexpectedly, a serial or size differs, a signature exists, or either device is mounted or held by another storage layer.
If this step fails
The by-id directory contains several links for one disk.
Likely causeLinux exposes model/serial, WWN, and sometimes partition aliases for the same device.
ls -l /dev/disk/by-id | grep Z4ZAA001udevadm info --query=property --name=/dev/sda
ResolutionPrefer a unique whole-disk WWN identity when available, exclude `-partN` links, and record all aliases that resolve to the same device.
Security notes
- Inventory output may reveal storage serials and infrastructure paths; store it with the change record rather than a public ticket.
Alternatives
- On systems with enclosure services, add `lsscsi -g`, `sg_ses`, or the vendor's read-only slot locator to the evidence, but keep the stable by-id path as the command input.
Stop conditions
- Do not continue until a second person or an equivalent automated policy verifies the two destructive targets against the approved inventory.
command
Install OpenZFS and capture its supported feature set
Install the distribution package rather than an untracked third-party build, then record the userspace and kernel-module versions. A version mismatch is a stop condition because pool features enabled by newer software may prevent import on the intended recovery host.
Why this step matters
The pool's on-disk feature compatibility and the host's recovery behavior depend on the exact OpenZFS userspace and kernel module. Distribution packages provide a support and update boundary that must be recorded before enabling features.
What to understand
The `zfs version` output includes both command-line tools and module versions; they should agree. A DKMS build failure can leave old userspace talking to a missing or older module.
Do not run `zpool upgrade` as an installation habit. Enabling new pool features can make the pool impossible to import on older rescue media or a planned failover host.
Keep package updates inside normal kernel and storage maintenance, including a reboot or module-loading test when required by the distribution.
System changes
- Installs `zfsutils-linux`, its dependencies, command-line tools, systemd units, udev rules, and the ZFS kernel module for the running kernel.
Syntax explained
apt-get install --yes zfsutils-linux- Installs the Ubuntu-supported ZFS userland and module integration without accepting unrelated third-party repositories.
modinfo zfs- Reads metadata for the kernel module available to the running system; it does not load or change pool state.
sudo apt-get update && sudo apt-get install --yes zfsutils-linux && zfs version && modinfo zfs | sed -n '1,8p'zfs-2.2.2-0ubuntu9.5 zfs-kmod-2.2.2-0ubuntu9.5 filename: /lib/modules/6.8.0-65-generic/updates/dkms/zfs.ko version: 2.2.2-0ubuntu9.5 license: CDDL description: ZFS
Checkpoint: Userspace and module belong to the intended supported release
zfs version && modinfo zfs | grep -E '^(version|filename):'Continue whenBoth components report the reviewed OpenZFS release and the module path belongs to the running kernel.
Stop whenThe module is unavailable, DKMS failed, userspace and kernel versions materially differ, or the recovery host cannot import the resulting feature set.
If this step fails
modprobe or zfs commands report that the module is unavailable.
Likely causeThe running kernel lacks built modules, Secure Boot rejected the module, or a DKMS build failed.
dkms statusjournalctl -k -b | grep -iE 'zfs|module|secure boot'apt-cache policy zfsutils-linux zfs-dkms
ResolutionRepair the supported package and module-signing path, reboot into a supported kernel if required, and rerun the version checkpoint before touching disks.
Security notes
- Adding a third-party repository changes the kernel supply chain and is outside this guide. Verify repository provenance before installing storage code with kernel privileges.
Alternatives
- Use the vendor-supported package flow for Debian, RHEL-derived, or appliance platforms and preserve the same version and recovery-compatibility evidence.
Stop conditions
- Do not create or upgrade a pool when the installed feature set is newer than every available recovery environment.
decision
Run the destructive preflight and confirm the backup boundary
Re-check that neither selected disk is mounted, part of swap, claimed by device mapper, mdraid, LVM, or an imported pool. Confirm that the off-host backup has a recent restore test and that the pool name is unused.
Why this step matters
A final multi-layer ownership check catches devices that look empty in one tool but are active in another. It also separates the recoverability of existing data from the availability supplied by the future mirror.
What to understand
findmnt, LVM, mdraid, and ZFS each describe a different ownership layer. An absent result is expected for selected empty disks; a positive result requires investigation, not a force flag.
`zpool import` can reveal an exported pool whose labels are not visible as a mounted filesystem. Destroying those labels may remove the only map to existing data.
Record the timestamp, restore-test identifier, selected pool name, and approval in the change. The zpool name must not collide with an imported or importable pool.
System changes
- No storage state changes; this creates the go/no-go evidence immediately preceding pool creation.
Syntax explained
mdadm --examine- Reads possible md superblock metadata from each selected device without assembling or modifying an array.
zpool import- Lists importable pools without importing them; an existing label is an ownership signal.
Values stay on this page and are never sent or saved.
sudo findmnt --source {{diskA}} --source {{diskB}}; sudo pvs --noheadings -o pv_name 2>/dev/null; sudo mdadm --examine {{diskA}} {{diskB}} 2>&1 | sed -n '1,12p'; sudo zpool import; sudo zpool list {{poolName}}mdadm: No md superblock detected on /dev/sda. mdadm: No md superblock detected on /dev/sdb. no pools available to import cannot open 'atlas': no such pool
Checkpoint: No storage owner or pool name collision remains
sudo findmnt --source {{diskA}} --source {{diskB}}; sudo zpool list {{poolName}}Continue whenNo mounts reference either disk and the selected pool name is absent.
Stop whenAny layer reports ownership, the backup evidence is stale, or the named pool already exists or is importable.
If this step fails
mdadm reports a valid superblock on an otherwise unmounted disk.
Likely causeThe disk is or was a member of a software RAID array and may contain recoverable data.
mdadm --examine {{diskA}}cat /proc/mdstatlsblk -o NAME,TYPE,FSTYPE,MOUNTPOINTS
ResolutionIdentify the array and data owner. Assemble only through an approved recovery procedure; do not erase metadata until the old system is formally retired and backed up.
Security notes
- A backup statement without an actual restore record is not authorization for destructive work.
Alternatives
- For cloud volumes, add provider attachment ID, snapshot ID, encryption-key ownership, and availability-zone evidence to this checkpoint.
Stop conditions
- Do not allow schedule pressure to turn an unexplained signature into permission to use `-f`.
command
Create the mirrored pool from by-id paths
Create one two-way mirror with an explicit 4 KiB sector alignment, disable automatic root mounting, and preserve the stable device names in pool metadata. This is the irreversible boundary: zpool create overwrites labels on the selected devices.
Why this step matters
Pool topology is the durable failure-domain decision. Creating one explicit mirror with stable paths avoids accidental striping, while sector alignment and root-dataset policy prevent defaults from becoming hidden production assumptions.
What to understand
`mirror {{diskA}} {{diskB}}` creates one top-level mirrored vdev. Omitting `mirror` would create two top-level single-disk vdevs striped by the pool, where loss of either disk loses the pool.
ashift=12 selects 4096-byte sectors and is effectively permanent for the vdev. It is a conservative baseline for modern disks but should follow real device and workload requirements.
The pool root uses mountpoint=none. Workloads receive child datasets so quotas, snapshots, record sizes, and replication scope can be controlled independently.
The `-f` option is present only after the preceding no-signature gate because some platforms retain innocuous labels. It never authorizes overwriting unexplained data.
System changes
- Writes ZFS labels to both selected disks, creates pool metadata and one mirrored top-level vdev, and makes the pool importable under `{{poolName}}`.
Syntax explained
-f- Forces use of devices after explicit ownership checks; it can overwrite recognized labels and is therefore dangerous.
-o ashift=12- Sets the vdev allocation sector shift to 2^12 bytes; changing it later requires replacing or recreating the vdev.
-O mountpoint=none- Sets a filesystem property on the pool root so application data is placed in child datasets.
mirror- Groups both following leaves into one redundant top-level vdev rather than two striped single-disk vdevs.
Values stay on this page and are never sent or saved.
sudo zpool create -f -o ashift=12 -o autotrim=on -O mountpoint=none -O compression=zstd -O atime=off {{poolName}} mirror {{diskA}} {{diskB}} && sudo zpool status -P {{poolName}} pool: atlas
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
atlas ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
/dev/disk/by-id/wwn-0x5000c500aa01 ONLINE 0 0 0
/dev/disk/by-id/wwn-0x5000c500aa02 ONLINE 0 0 0
errors: No known data errorsCheckpoint: The created topology is exactly one healthy two-way mirror
sudo zpool status -P {{poolName}}Continue whenONLINE pool, one mirror-0, both intended by-id leaves ONLINE, zero errors.
Stop whenThe topology shows two top-level disks, a wrong path, DEGRADED state, or any read, write, or checksum error.
If this step fails
Pool status lists each disk directly under the pool instead of under mirror-0.
Likely causeThe `mirror` topology keyword was omitted or misplaced, creating a nonredundant stripe.
zpool status -P {{poolName}}zpool history {{poolName}}
ResolutionDo not put data on the pool. If it is newly created and empty, export, reconfirm identities, destroy it under the approved rollback, and recreate the intended mirror.
Security notes
- zpool history records administrative commands. Do not pass encryption keys or secrets directly on storage command lines.
Alternatives
- Run `zpool create -n ...` first on OpenZFS versions that support the requested dry-run details, then compare the proposed topology with the approved diagram.
Stop conditions
- Stop immediately if the post-create topology differs from the reviewed mirror; do not attempt to repair it by adding more top-level vdevs.
config
Create a dataset with explicit mount and space policy
Place application data in a child dataset instead of the pool root. A child dataset gives snapshots, quotas, reservations, permissions, compression, and replication an independent policy boundary.
Why this step matters
Datasets turn a pool into manageable policy boundaries. Explicit properties make capacity, mounting, performance trade-offs, and future recovery scope reviewable instead of inheriting silent defaults.
What to understand
quota limits the dataset and descendants, including snapshot-referenced space; reservation guarantees pool space to this dataset. Their combination must leave emergency free space for other datasets and metadata.
Compression often reduces physical I/O and capacity consumption, but the chosen algorithm must be supported by every recovery host. Measure it with representative data.
Disabling atime avoids write amplification for access-time updates. Applications that depend on access times need a different policy.
Before selecting a mountpoint, verify that the existing directory is empty or disposable; mounting a dataset can hide pre-existing files without deleting them.
System changes
- Creates `{{dataset}}`, mounts it at `{{mountpoint}}`, and assigns local quota, reservation, compression, and access-time properties.
Syntax explained
-o mountpoint={{mountpoint}}- Creates and mounts the filesystem at the explicit absolute path managed by ZFS.
-o quota=1.5T- Caps total space consumed by this dataset and its descendants according to ZFS accounting.
-o reservation=20G- Reserves pool capacity for this dataset even before it is logically used.
Values stay on this page and are never sent or saved.
sudo zfs create -o mountpoint={{mountpoint}} -o compression=zstd -o atime=off -o quota=1.5T -o reservation=20G {{dataset}} && sudo zfs get -H -o name,property,value,source mountpoint,compression,atime,quota,reservation {{dataset}}atlas/services mountpoint /srv/atlas local atlas/services compression zstd local atlas/services atime off local atlas/services quota 1.50T local atlas/services reservation 20G local
Checkpoint: The child dataset is mounted with local reviewed properties
sudo zfs get mountpoint,compression,atime,quota,reservation {{dataset}}Continue whenEvery property has the intended value and source `local`, and the mountpoint contains only the planned workload.
Stop whenThe mountpoint hides existing files, the quota is below current data size, or reservations consume unsafe pool headroom.
If this step fails
zfs create reports the mountpoint is busy or the application sees old data disappear.
Likely causeAnother filesystem is mounted there, a process uses the path, or existing directory contents became hidden under the new mount.
findmnt {{mountpoint}}lsof +D {{mountpoint}} 2>/dev/null | headmount | grep '{{mountpoint}}'
ResolutionStop the workload, choose a clean mountpoint or migrate data with a reviewed cutover, and verify the underlying directory after a safe unmount.
Security notes
- Dataset creation does not set application ownership or ACLs. Apply least-privilege permissions before copying sensitive data.
Alternatives
- Use `refquota` when only live referenced data, not descendant and snapshot usage, should be limited; document the accounting distinction.
Stop conditions
- Do not copy production data until mount ownership, permissions, capacity, and backup scope are verified.
command
Create a snapshot and restore one file without rolling back
Write a harmless canary, snapshot the dataset, alter the live copy, and restore only that file through the snapshot namespace. This demonstrates the normal recovery pattern without discarding unrelated changes made after the snapshot.
Why this step matters
A snapshot is useful only when operators can locate and recover the intended data. File-level restoration is safer than dataset rollback because it preserves unrelated valid writes made after the snapshot.
What to understand
Snapshot creation is atomic for the dataset but not automatically application-consistent. Databases and multi-file services may require a quiesce, transaction checkpoint, or native backup hook.
The `.zfs/snapshot` namespace exposes a read-only historical tree even when the `.zfs` directory is hidden from ordinary listings. Copying one file creates a new live version.
Snapshot `USED` represents unique blocks that would be freed by destroying that snapshot, not the original logical dataset size. Retention can silently consume pool headroom.
The canary deliberately demonstrates a change and recovery with non-sensitive data; do not learn the process first on the only copy of a production file.
System changes
- Creates `{{dataset}}@{{snapshotName}}`, changes the live canary, then writes the snapshot version back into the live dataset.
Syntax explained
zfs snapshot dataset@snapshot- Creates a read-only copy-on-write reference point; it does not copy all blocks or produce an off-host backup.
cp --preserve=mode,ownership,timestamps- Restores selected metadata with the file content; ACLs and extended attributes may require additional application-aware handling.
zfs list -t snapshot- Includes snapshots, which normal `zfs list` output omits by default.
Values stay on this page and are never sent or saved.
printf 'release=2026.07\n' | sudo tee {{mountpoint}}/oneliners-canary.txt >/dev/null && sudo zfs snapshot {{dataset}}@{{snapshotName}} && printf 'release=broken\n' | sudo tee {{mountpoint}}/oneliners-canary.txt >/dev/null && sudo cp --preserve=mode,ownership,timestamps {{mountpoint}}/.zfs/snapshot/{{snapshotName}}/oneliners-canary.txt {{mountpoint}}/oneliners-canary.txt && sudo cat {{mountpoint}}/oneliners-canary.txt && sudo zfs list -t snapshot -o name,used,refer {{dataset}}@{{snapshotName}}release=2026.07 NAME USED REFER atlas/services@before-maintenance 64K 96K
Checkpoint: The snapshot is visible and the live canary is restored
zfs list -t snapshot {{dataset}}@{{snapshotName}} && cat {{mountpoint}}/oneliners-canary.txtContinue whenThe named snapshot exists and the live file contains `release=2026.07`.
Stop whenThe snapshot is absent, the application could not reach a consistent point, or restoration would overwrite newer required data.
If this step fails
The snapshot directory appears missing in a normal `ls -a`.
Likely causeThe `snapdir` property defaults to hidden; the virtual `.zfs` path may still be directly accessible.
zfs get snapdir {{dataset}}ls {{mountpoint}}/.zfs/snapshot/{{snapshotName}}
ResolutionUse the direct path or temporarily set snapdir=visible if the exposure is acceptable. Do not assume the snapshot itself is missing.
Security notes
- A deleted secret remains readable in snapshots until all referencing snapshots expire. Restrict traversal and align retention with incident-response and deletion policy.
Alternatives
- Clone the snapshot into a separate writable dataset when an application needs an isolated multi-file recovery or validation environment.
Stop conditions
- Do not use zfs rollback for a single-file restore; stop if file metadata or application consistency requires a native restore process.
command
Run a scrub and retain completion evidence
Start a full checksum scrub, monitor progress, and wait for completion during a measured I/O window. Redundancy lets ZFS repair a damaged copy; a single-device pool can only report corruption.
Why this step matters
Checksums protect data only when blocks are read and a verified alternate copy exists. A scheduled, observed scrub discovers latent corruption while the mirror may still repair it.
What to understand
Scrub reads compete with workload I/O. Establish a baseline duration and latency impact, schedule outside critical peaks, and alert if the operation stalls or exceeds its window.
`zpool scrub -w` waits for completion, allowing an operator or automation job to capture final evidence. A status seen immediately after starting proves only that a scan began.
READ and WRITE counters reflect device I/O failures, while CKSUM means returned data did not match ZFS's checksum. CKSUM errors can also implicate cables, controllers, memory, or power.
Zero repaired bytes is normal for a healthy scrub. Repaired bytes are evidence of a fault that requires investigation even if the final pool state is ONLINE.
System changes
- Starts a pool-wide scrub that reads allocated blocks and may rewrite a damaged replica from a verified redundant copy.
Syntax explained
zpool scrub {{poolName}}- Starts or resumes a full checksum verification for the named pool.
zpool scrub -w- Waits until the scrub finishes so subsequent status is completion evidence.
zpool status -v- Shows scan result, device counters, and paths for known permanent data errors.
Values stay on this page and are never sent or saved.
sudo zpool scrub {{poolName}} && sudo zpool status {{poolName}} && sudo zpool scrub -w {{poolName}} && sudo zpool status -v {{poolName}} pool: atlas
state: ONLINE
scan: scrub repaired 0B in 00:08:42 with 0 errors on Tue Jul 28 02:18:14 2026
config:
NAME STATE READ WRITE CKSUM
atlas ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
errors: No known data errorsCheckpoint: The scrub completed and every error is accounted for
sudo zpool status -v {{poolName}}Continue whenA completed timestamp, zero unrepairable errors, expected repaired-byte count, and no unexplained device counters.
Stop whenA device faults, permanent errors appear, workload latency exceeds the agreed threshold, or the scrub cannot complete.
If this step fails
Scrub progress remains slow and production latency rises.
Likely causeThe pool lacks I/O headroom, a disk is retrying reads, or the scrub overlaps another intensive workload.
zpool iostat -v {{poolName}} 5 6zpool status {{poolName}}journalctl -k --since '-15 min' | tail -100
ResolutionPause the scrub when supported or move it outside the workload peak, then investigate slow or erroring leaves. Do not repeatedly cancel scans so no full scrub ever completes.
Security notes
- Permanent-error paths may reveal tenant or user names. Protect the full report while preserving enough detail for restoration.
Alternatives
- Use the packaged monthly timer as a baseline and adjust cadence from media class, capacity, replacement time, workload, and recovery objectives rather than folklore.
Stop conditions
- Treat any unrepairable error as a data incident; do not clear it and declare success.
config
Enable one periodic scrub schedule and verify the timer
Use the packaged systemd timer rather than creating a competing cron job. Enable exactly one cadence per pool, review its next activation, and ensure monitoring notices both a failed unit and a pool that has gone too long without a completed scrub.
Why this step matters
A manual scrub demonstrated once does not create durable protection. One owned systemd timer, paired with failure and scrub-age monitoring, makes the verification recurring and auditable.
What to understand
OpenZFS packages commonly provide weekly and monthly per-pool template timers. Enabling both doubles work without improving ownership; select one measured cadence.
The timer's existence does not prove completion. Monitor the instantiated service result and parse `zpool status` for the timestamp and error count.
Use randomized or fleet-staggered timing when many hosts share controllers, storage shelves, power, or monitoring capacity.
System changes
- Disables a possible weekly instance and enables the monthly per-pool systemd timer immediately and across reboots.
Syntax explained
disable --now- Stops and disables the competing cadence so only one schedule owns the pool.
enable --now- Enables the timer persistently and starts timer scheduling in the current boot.
list-timers --all- Shows future and previous activations, including inactive timers, for evidence.
Values stay on this page and are never sent or saved.
sudo systemctl disable --now zfs-scrub-weekly@{{poolName}}.timer 2>/dev/null || true; sudo systemctl enable --now zfs-scrub-monthly@{{poolName}}.timer && systemctl list-timers zfs-scrub-monthly@{{poolName}}.timer --allNEXT LEFT LAST PASSED UNIT Sat 2026-08-08 00:24:00 UTC 10 days - - zfs-scrub-monthly@atlas.timer 1 timers listed.
Checkpoint: Exactly one reviewed timer has a future activation
systemctl is-enabled zfs-scrub-monthly@{{poolName}}.timer && systemctl list-timers 'zfs-scrub-*@{{poolName}}.timer' --allContinue whenThe monthly instance is enabled and scheduled; no unintended duplicate cadence exists.
Stop whenAnother scheduler already owns scrubs or the packaged unit names differ on this platform.
If this step fails
The timer runs but the service fails or no new scrub timestamp appears.
Likely causeThe pool was unavailable, an old scrub/resilver blocked a new one, or the instantiated unit name does not match the actual pool.
systemctl status zfs-scrub-monthly@{{poolName}}.timer --no-pagerjournalctl -u zfs-scrub-monthly@{{poolName}}.service -n 80 --no-pagerzpool status {{poolName}}
ResolutionCorrect the pool or unit ownership, finish or diagnose the competing scan, and trigger one observed scrub before returning the timer to service.
Security notes
- Timer and service modification is privileged configuration; restrict who may change or mask the unit.
Alternatives
- Use configuration management or a fleet scheduler if it also verifies completion and prevents overlapping scans across shared infrastructure.
Stop conditions
- Do not enable a second schedule until the existing cron, appliance, or orchestration ownership has been identified.
verification
Map a faulted vdev to the physical disk before touching hardware
When a member reports errors, freeze the evidence and correlate the pool's stable path, serial, WWN, enclosure slot, and operating-system device. Do not clear counters or pull a disk merely because its current sdX name resembles an old diagram.
Why this step matters
The safest replacement begins with immutable evidence, not a blinking assumption. Pool status, event history, udev properties, and the physical map must all identify the same failed leaf.
What to understand
Use `-P` for full device paths and `-L` for resolved paths while retaining the WWN recorded at creation. Compare serial, WWN, controller path, and bay.
READ, WRITE, and CKSUM counters describe different failure symptoms. Preserve them and kernel events before clearing or rebooting because intermittent faults may disappear temporarily.
Do not infer disk failure solely from CKSUM. Cabling, backplane, HBA, memory, and power can corrupt or interrupt I/O and may affect multiple drives.
System changes
- No pool changes; captures diagnostic evidence and the exact physical replacement target.
Syntax explained
zpool status -LPv- Shows resolved full paths plus verbose permanent-error information for mapping and recovery.
zpool events -v- Displays detailed ZFS event history without resetting counters.
udevadm info --query=property- Reads device identity and connection-path properties used to correlate software and hardware.
Values stay on this page and are never sent or saved.
sudo zpool status -LPv {{poolName}} && sudo zpool events -v {{poolName}} | tail -40 && sudo udevadm info --query=property --name=$(readlink -f {{diskA}}) | grep -E '^(ID_SERIAL|ID_WWN|ID_PATH)=' pool: atlas
state: DEGRADED
status: One or more devices has experienced an unrecoverable error.
config:
NAME STATE READ WRITE CKSUM
atlas DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
wwn-0x5000c500aa01 DEGRADED 12 0 0
wwn-0x5000c500aa02 ONLINE 0 0 0
ID_SERIAL=ST2000NM_Z4ZAA001
ID_WWN=0x5000c500aa01
ID_PATH=pci-0000:03:00.0-sas-phy2-lun-0Checkpoint: One approved physical device matches the failed ZFS leaf
sudo zpool status -LPv {{poolName}} && sudo udevadm info --query=property --name=$(readlink -f {{diskA}})Continue whenPool path, WWN, serial, controller path, and physical slot all agree.
Stop whenMore than one member is unhealthy, the identity differs anywhere, or remote hands cannot positively identify the bay.
If this step fails
The old by-id link is gone after a complete device failure.
Likely causeudev cannot create a live symlink for hardware that no longer enumerates, while ZFS retains its recorded GUID or former path.
zpool status -gLP {{poolName}}zdb -C {{poolName}} | sed -n '1,160p'journalctl -k -b | grep -iE 'ata|scsi|nvme|reset|offline'
ResolutionUse the recorded creation inventory, ZFS GUID, serial-to-slot map, enclosure data, and kernel history. Do not choose a neighboring live disk by elimination without independent confirmation.
Security notes
- Only authorized storage operators or supervised remote hands should activate locator LEDs or handle bays.
Alternatives
- Where supported, use enclosure management to illuminate the slot after serial correlation, but never treat the LED alone as identity proof.
Stop conditions
- Do not remove hardware when there is any disagreement between the ZFS leaf, device properties, and physical map.
command
Replace the failed member and monitor resilvering
After confirming the exact old identity and the new disk has no required data, ask ZFS to replace the failed leaf with the new stable path. Keep the host powered and the workload controlled until the resilver completes.
Why this step matters
Replacement reconstructs the mirror from the surviving copy under elevated I/O risk. Explicit old and new stable identities, adequate new capacity, and continuous status monitoring protect the remaining data copy.
What to understand
`wipefs -n` on the replacement is a final no-act ownership check. A new-looking drive may contain another system's metadata or be the wrong bay.
`zpool replace pool old new` associates the intended failed leaf with the new device and begins resilvering. Do not use attach, detach, or add as approximate substitutes.
The loop waits while status reports active resilvering, but production monitoring should also track new errors and workload latency. A stalled resilver is not success.
After completion, the old leaf should disappear from the active mirror and the new by-id path should be ONLINE. A follow-up scrub and backup verification close the change.
System changes
- Writes ZFS labels to `{{replacementDisk}}`, associates it with the old leaf, reconstructs allocated data, and replaces the old member in the mirror.
Syntax explained
zpool replace pool old new- Performs an in-place logical replacement and starts resilvering onto the named new leaf.
while ... grep 'resilver in progress'- Polls status at a bounded interval; it does not suppress errors or prove a clean final state.
zpool status -v- Provides the required final scan result and permanent-error evidence after reconstruction.
Values stay on this page and are never sent or saved.
sudo wipefs -n {{replacementDisk}} && sudo zpool replace {{poolName}} {{diskA}} {{replacementDisk}} && sudo zpool status -P {{poolName}} && while sudo zpool status {{poolName}} | grep -q 'resilver in progress'; do sleep 30; done; sudo zpool status -v {{poolName}} pool: atlas
state: ONLINE
scan: resilvered 612G in 02:41:09 with 0 errors on Tue Jul 28 07:31:44 2026
config:
NAME STATE READ WRITE CKSUM
atlas ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
/dev/disk/by-id/wwn-0x5000c500aa03 ONLINE 0 0 0
/dev/disk/by-id/wwn-0x5000c500aa02 ONLINE 0 0 0
errors: No known data errorsCheckpoint: Resilver finished with the intended new member ONLINE
sudo zpool status -P {{poolName}}Continue whenCompleted resilver, both mirror leaves ONLINE, replacement path present, old path absent, and no known data errors.
Stop whenThe surviving member accumulates errors, the replacement is smaller or unstable, or status identifies a different old leaf.
If this step fails
zpool replace says the new device is too small.
Likely causeNominally similar disks have different sector counts, or the selected replacement path refers to a partition rather than the whole device.
blockdev --getsize64 {{diskA}} {{replacementDisk}}lsblk -b -o NAME,SIZE,TYPE,MODEL,SERIAL {{replacementDisk}}
ResolutionUse a verified whole disk with equal or greater usable capacity. Do not force or repartition the surviving production member to make a smaller replacement fit.
Security notes
- A removed disk may retain readable blocks. Apply the organization's chain-of-custody, encryption, secure erase, warranty-return, and disposal policy.
Alternatives
- When a hot spare is active, replace the failed physical member promptly so the spare can return to standby; a spare is not permission to leave degraded hardware indefinitely.
Stop conditions
- Escalate to backup recovery if the only surviving copy produces uncorrectable errors during resilvering.
decision
Rehearse discovery and a read-only, no-mount recovery import
On an isolated recovery host or scheduled rehearsal, export a quiesced test pool and discover it through /dev/disk/by-id. Import it read-only without mounting datasets, inspect health and datasets, then export cleanly. Never rehearse this against an active production pool from a second host.
Why this step matters
Import is an ownership and consistency decision, not merely a command that makes files appear. Read-only no-mount inspection reduces writes while operators establish pool health and whether another host is still active.
What to understand
A clean export flushes pending work and marks the pool as intentionally detached. Rehearse only with a quiesced test pool or approved outage; never export a live root or application pool casually.
`-d /dev/disk/by-id` constrains discovery to stable identities. `readonly=on` prevents ordinary pool writes and `-N` avoids mounting datasets before inspection.
A warning that the pool is active elsewhere requires fencing and ownership proof. `-f` can create simultaneous writers or hide a real recovery condition.
Rewind and checkpoint recovery can discard transactions. They belong to an incident plan with backup evidence and are not included in the routine rehearsal.
System changes
- Exports the rehearsed pool, imports it temporarily read-only without mounting datasets, exports it again, then returns it to normal import on the original authorized host.
Syntax explained
zpool import -d /dev/disk/by-id- Searches a controlled stable-device directory for importable pool labels.
-o readonly=on- Imports the pool property read-only for evidence gathering and restricted access.
-N- Imports the pool without mounting datasets, preventing path collisions and premature application access.
Values stay on this page and are never sent or saved.
sudo zpool export {{poolName}} && sudo zpool import -d /dev/disk/by-id && sudo zpool import -d /dev/disk/by-id -o readonly=on -N {{poolName}} && sudo zpool status -v {{poolName}} && sudo zfs list -r {{dataset}} && sudo zpool export {{poolName}} && sudo zpool import -d /dev/disk/by-id {{poolName}} pool: atlas
id: 14691277833012899140
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
atlas ONLINE
mirror-0 ONLINE
sda ONLINE
sdb ONLINE
NAME USED AVAIL REFER MOUNTPOINT
atlas/services 612G 840G 612G /srv/atlasCheckpoint: Recovery discovery and read-only inspection succeed without force
sudo zpool import -d /dev/disk/by-id -o readonly=on -N {{poolName}} && sudo zpool status -v {{poolName}}Continue whenThe correct pool ID and mirror are ONLINE in read-only mode, with no ownership warning or unrepairable errors.
Stop whenThe import requires force, offers rewind, reports missing top-level vdevs, or the original host may still own the pool.
If this step fails
The recovery host cannot discover the pool although both disks appear in lsblk.
Likely causeThe host lacks compatible OpenZFS support, discovery scans the wrong path, device permissions block labels, or the devices belong behind an unsupported controller layer.
zfs versionls -l /dev/disk/by-idzpool import -d /dev/disk/by-id
ResolutionMatch the supported OpenZFS feature set and expose the original whole devices through stable paths. Do not initialize, partition, or format disks on the recovery host.
Security notes
- Read-only import still exposes all unencrypted data to the recovery host's administrators. Use an authorized isolated environment.
Alternatives
- Restore to a newly created pool from independent backup when pool metadata or hardware is uncertain; this preserves the original media for forensic recovery.
Stop conditions
- Never connect the same writable pool to two non-clustered hosts or use force import before fencing the previous owner.
verification
Verify health, policy, snapshot visibility, and recovery evidence
Finish with one evidence bundle that identifies the pool topology, error counters, last scrub, dataset properties, snapshots, periodic timer, and the restored canary. A green service status alone is not proof that stored data is recoverable.
Why this step matters
A storage change is complete only when topology, integrity, policy, scheduling, and a real recovery result agree. Consolidated evidence prevents an ONLINE headline from hiding a disabled scrub, wrong mountpoint, or untested snapshot.
What to understand
Review both inherited and local property sources; a correct value inherited from an unexpected parent may change later when the parent policy changes.
Pool status is point-in-time evidence. Store the completion record with backup restore evidence, the serial map, replacement history, and next review date.
The canary proves file-level snapshot access, not application-consistent database recovery. Maintain native recovery tests for every workload using the dataset.
System changes
- No new changes; reads final pool, dataset, snapshot, timer, and recovered-file state into the change evidence.
Syntax explained
zpool get ashift,autotrim- Displays durable pool or vdev policy needed to compare the implementation with the design.
zfs get ...- Shows effective dataset values and whether each came from local, inherited, or default policy.
systemctl is-enabled- Verifies persistent scheduling state, not whether the last scrub succeeded.
Values stay on this page and are never sent or saved.
sudo zpool status -P {{poolName}} && sudo zpool get ashift,autotrim {{poolName}} && sudo zfs get mountpoint,compression,atime,quota,reservation {{dataset}} && sudo zfs list -t snapshot -r {{dataset}} && systemctl is-enabled zfs-scrub-monthly@{{poolName}}.timer && sudo cat {{mountpoint}}/oneliners-canary.txtpool: atlas state: ONLINE scan: scrub repaired 0B in 00:08:42 with 0 errors on Tue Jul 28 02:18:14 2026 errors: No known data errors atlas autotrim on local atlas/services compression zstd local atlas/services@before-maintenance 64K - 96K - enabled release=2026.07
Checkpoint: The complete operating contract is evidenced
sudo zpool status -P {{poolName}} && sudo zfs list -t snapshot -r {{dataset}}Continue whenHealthy intended mirror, completed clean scan, expected dataset and snapshot, and no unexplained error counters.
Stop whenAny required property, snapshot, timer, backup test, or device identity lacks evidence.
If this step fails
The pool is ONLINE but a required snapshot or reservation is absent.
Likely causeDevice health succeeded while the dataset policy step failed, was inherited unexpectedly, or was later changed.
zfs get all {{dataset}}zfs list -t snapshot -r {{dataset}}zpool history {{poolName}}
ResolutionCorrect only the missing dataset policy after capacity and retention review, then rerun the file recovery and final evidence checks.
Security notes
- Redact sensitive dataset names in broad operational dashboards while keeping full evidence in restricted storage records.
Alternatives
- Automate this evidence in monitoring and configuration management, but keep the human-reviewed recovery test and physical identity map.
Stop conditions
- Do not close the change when health is green but recovery, monitoring, or backup ownership remains unproven.
Finish line
Verification checklist
sudo zpool status -P {{poolName}}The pool and every mirror member are ONLINE, the most recent scrub or resilver completed with zero unrepairable errors, and READ, WRITE, and CKSUM counters are understood rather than blindly cleared.sudo zfs get mountpoint,compression,quota,reservation {{dataset}} && sudo zfs list -t snapshot -r {{dataset}} && sudo cat {{mountpoint}}/oneliners-canary.txtThe dataset mounts at the intended path with the reviewed policies, the named snapshot is listed, and the canary contains the restored release=2026.07 value.systemctl list-timers zfs-scrub-monthly@{{poolName}}.timer --allExactly one intended timer cadence is enabled for this pool and it has a future activation time; monitoring separately alerts on missed or failed scrubs.Recovery guidance
Common problems and safe checks
zpool create reports that a device contains a filesystem, partition table, RAID signature, or data from an exported pool.
Likely causeThe selected path is not empty, the wrong serial was chosen, or stale metadata exists from a previous use. Force creation would destroy the remaining ownership evidence.
lsblk -f {{diskA}} {{diskB}}wipefs -n {{diskA}} {{diskB}}zpool import -d /dev/disk/by-id
ResolutionStop and identify the prior owner. If the data is intentionally retired, obtain explicit approval and a separately verified backup before using the owner's documented decommissioning process; do not add -f as the first response.
A scrub completes with repaired bytes, persistent CKSUM errors, or files listed under permanent errors.
Likely causeZFS detected corruption. A mirror may repair a bad copy, but continuing checksum errors can originate from media, cables, controller, memory, power, or a damaged file with no valid replica.
zpool status -v {{poolName}}zpool events -v {{poolName}} | tail -80smartctl -x $(readlink -f {{diskA}})
ResolutionPreserve status and events, validate hardware and memory, and restore any named unrepairable file from a known-good backup. Run another scrub only after correcting the suspected cause; use zpool clear only after evidence is captured and the fault is resolved.
The replacement is visible but resilvering does not start, remains slow, or the pool becomes more degraded.
Likely causeThe old vdev identifier was wrong, the new disk is too small, another scrub/resilver owns the scan, the new path is unstable, or the surviving member is producing read errors under reconstruction load.
zpool status -LPv {{poolName}}zpool iostat -v {{poolName}} 5 3lsblk -o NAME,SIZE,MODEL,SERIAL,WWN
ResolutionDo not detach the surviving good member. Correct the path or capacity issue, reduce workload pressure, preserve new errors, and escalate to backup-based recovery if the remaining copy cannot complete the resilver.
The pool is importable but ZFS warns it is active on another system or offers a transaction rewind.
Likely causeThe pool was not exported cleanly, the original host is still active, cache and host identity disagree, or recent writes were interrupted.
zpool import -d /dev/disk/by-idps -ef | grep '[z]pool'systemctl status zfs-import-cache zfs-import-scan --no-pager
ResolutionFence the original owner and preserve the import report. Start with a read-only no-mount import on the authorized recovery host. A rewind can discard transactions and requires a backup-aware recovery decision, not an automatic force flag.
Reference
Frequently asked questions
Is a ZFS mirror a backup?
No. It improves availability when one member fails and allows checksum repair from another copy. Both members share the host, administrative commands, application corruption, and many physical hazards. Keep tested, versioned copies in another failure domain.
Why use /dev/disk/by-id instead of /dev/sda?
sdX names reflect discovery order and may change after reboot, controller changes, or replacement. A WWN or serial-backed path makes the pool status, change record, and physical bay map refer to the intended device identity.
Should I use zfs rollback to restore one deleted file?
Usually not. Copy that file from the snapshot namespace. Rollback discards live changes after the snapshot and may require destroying newer snapshots, bookmarks, or clones, so it is a dataset-wide recovery decision.
Can I clear error counters after a scrub?
Only after capturing evidence, identifying and correcting the underlying fault, restoring any unrepairable data, and confirming a clean scrub. Clearing counters hides history; it does not repair media, cabling, memory, or files.
Recovery
Rollback
A newly created empty pool can be exported and destroyed only after re-confirming its exact name and proving it contains no required data. Dataset property changes can be restored individually. Snapshot rollback is deliberately not the default because it discards every live change after the selected snapshot.
- For a property mistake, inspect the source with `zfs get <property> {{dataset}}`, then restore the recorded former value with `zfs set <property>=<value> {{dataset}}` or inherit the parent policy with `zfs inherit <property> {{dataset}}`.
- For a bad file change, copy only the affected files from `{{mountpoint}}/.zfs/snapshot/{{snapshotName}}/`; do not use `zfs rollback -r` unless the loss of newer data, snapshots, bookmarks, and dependent clones has been reviewed.
- To abandon a brand-new empty test pool, stop users, `zpool export {{poolName}}`, confirm the devices and backup again, re-import, and run `zpool destroy {{poolName}}`. Pool destruction is not recoverable and is excluded from the normal path.
- If replacement has started, keep the surviving member online and let the resilver complete when safe. Do not detach the last good copy; escalate with the captured status, events, serial map, and backup evidence.
Evidence