Create remote, versioned backups with rsync and SSH
Use a dedicated SSH identity, dry-run transfers, build hard-linked snapshots, enforce retention, and test restore.
Maintain space-efficient remote snapshots that can be restored without trusting the source host.
- rsync 3.2+
- OpenSSH 9.x
- Independent destination Use separate failure and access boundaries.
- Dedicated account Use a restricted account and SSH key.
- Capacity plan Measure source data and define retention.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A destination-owned snapshot tree in which each successful UTC timestamp is a complete restore view while unchanged files share disk blocks with the previous snapshot.
- Separate forced-command SSH identities for source-host writes, narrow promotion, and recovery reads so compromise of the protected source cannot silently erase every retained generation.
- A dry-run, transfer, atomic latest-pointer promotion, checksum verification, retention preview, and isolated restore drill with retained evidence.
- The first preview explains every create, update, metadata change, and exclusion before bytes are written.
- A successful transfer creates a new timestamped directory and only then advances latest; a failed transfer leaves the previous latest untouched.
- Critical files compare without unexplained differences, hard-link reuse is visible, and a historical snapshot restores into a clean path with expected metadata.
- The operator can prove that recovery access, destination capacity, retention, immutable/offline copies, and application-consistent capture are covered beyond the rsync file copy itself.
Architecture
How the parts fit together
The protected source can write only through rrsync into one destination root. Each run targets a new UTC directory and uses the previous latest snapshot as --link-dest, so identical files become hard links while changed files consume new blocks. A separate forced promotion command validates the completed timestamp before atomically changing latest, and a recovery key stored away from the source is read-only.
- The application first creates a consistent source state; volatile caches, sockets, mounts, and secrets outside scope are documented.
- The source runs a fully itemized dry-run through the write-only forced command.
- rsync writes a new timestamped directory and hard-links unchanged files against ../latest.
- Verification checks the completed snapshot; only a zero-exit run may invoke the narrow promotion command.
- The destination script validates the timestamp and snapshot marker, creates latest.new, and atomically renames it to latest.
- A recovery host reads a historical generation into an empty path and verifies content plus metadata before retention removes anything.
Assumptions
- The destination uses a filesystem that supports hard links and has enough inodes as well as free bytes.
- Source and destination rsync versions support every selected option, and both sides agree on xattr, ACL, ownership, sparse-file, and special-file requirements.
- The source path's trailing slash is intentional: its contents, not the enclosing directory, become the snapshot root.
- The application is quiesced or exports a consistent database/state snapshot before rsync reads mutable files. Rsync alone does not make a transactional backup.
- The destination account cannot modify rrsync, promotion scripts, authorized_keys, snapshot parent permissions, or retention policy.
- The recovery private key is not present on the protected source; source compromise must not grant read/delete access to every historical snapshot.
Key concepts
- Hard-linked snapshot
- A full directory view whose unchanged regular files reference the same inode and blocks as an earlier snapshot.
- Trailing slash
- In rsync source syntax, source/ transfers directory contents while source transfers the directory itself.
- Itemized changes
- A compact per-path code showing transfer type and metadata differences, used to review a dry-run.
- Forced command
- An SSH authorized_keys restriction that ignores the client command and runs only a server-controlled program.
- rrsync
- Rsync's restricted wrapper that validates server-side options and confines transfers to one directory.
- Application consistency
- A state that application software can recover, often requiring a database dump, snapshot hook, lock, or quiesce operation before file capture.
Before you copy
Values used in this guide
{{sourceHost}}Stable source identity embedded in key comments and evidence.
Example: source01{{excludeFile}}Reviewed rsync filter file containing paths outside backup scope.
Example: /etc/backup/source01.exclude{{source}}Exact local source root whose contents are captured.
Example: /srv/app{{user}}Restricted destination SSH account.
Example: backup_source01{{host}}Verified destination hostname.
Example: backup01.internal{{root}}Destination-owned absolute snapshot root used by server controls.
Example: /backup/source01{{timestamp}}UTC snapshot identifier created once per run.
Example: 2026-07-25T020000Z{{restorePath}}Relative path selected for a representative recovery drill.
Example: etc/app{{sourceAddress}}Exact source address or CIDR accepted for write and promotion keys.
Example: 192.0.2.10{{recoveryAddress}}Exact administrative recovery address or CIDR accepted for read access.
Example: 198.51.100.20{{writePublicKey}}Complete public key generated for write-only transfers.
Example: ssh-ed25519 AAAAC3... backup-write@source01{{promotePublicKey}}Complete public key generated for the narrow promotion helper.
Example: ssh-ed25519 AAAAC3... backup-promote@source01{{restorePublicKey}}Complete public key generated and retained on the separate recovery host.
Example: ssh-ed25519 AAAAC3... backup-restore@admin01Security and production boundaries
- Do not give the source a general shell on the backup host. A compromised source should be unable to inspect other clients, edit policy, or delete prior generations.
- Use rrsync supplied by the installed rsync release and protect its path root-owned. Its manual documents restrictions and a shell-startup risk when the destination login shell is writable bash.
- A write-only key can still fill storage with new files. Enforce quotas, capacity alerts, rate controls, and a staging/promotion policy.
- Hard links are not immutability. A destination administrator or unsafe in-place write can alter shared content; add storage snapshots or offline/immutable copies.
- Backup content may include credentials and personal data. Encrypt transport, protect destination disks, and keep recovery keys in a separate administrative boundary.
Stop before continuing if
- Stop if the source is not application-consistent or crosses an unreviewed mount.
- Stop if dry-run output contains unexpected deletions, ownership changes, path nesting, or excluded critical data.
- Stop if the destination identity offers an interactive shell or can overwrite prior timestamped snapshots.
- Stop if any rsync I/O, vanished-file, permission, xattr, ACL, or partial-transfer error is unresolved.
- Do not advance latest unless the exact new snapshot passed verification and completion-marker checks.
- Do not delete retention candidates until a historical restore and independent retained generation are proven.
command
Create dedicated backup identities
Generate separate Ed25519 identities for write and promotion; create recovery read access on a separate administrative host.
Why this step matters
Separate identities make write, promotion, and recovery privileges independently revocable and prevent a compromised source from reading or deleting every retained generation.
What to understand
Create the write and promotion keys on the protected source. Generate the recovery key on a separate administration host and never copy its private half back to the source.
Ed25519 provides a compact modern key. The KDF round count protects an encrypted private key at rest; unattended source keys need compensating host and forced-command controls.
Record every public-key fingerprint, owner, location, purpose, source-address restriction, and rotation date before installation.
System changes
- Creates three dedicated SSH key pairs in their respective operator boundaries; no destination authorization occurs yet.
Syntax explained
-t ed25519- Creates a modern Ed25519 public/private key pair.
-a 64- Uses 64 KDF rounds when a private-key passphrase is present.
-f- Writes each identity to an explicit purpose-specific path.
-C- Adds a non-authoritative purpose and host comment to the public key.
Values stay on this page and are never sent or saved.
ssh-keygen -t ed25519 -a 64 -f ~/.ssh/id_backup_write -C "backup-write@{{sourceHost}}" && ssh-keygen -t ed25519 -a 64 -f ~/.ssh/id_backup_promote -C "backup-promote@{{sourceHost}}"SHA256:7cQ... backup-write@source01 SHA256:Q8m... backup-promote@source01
Checkpoint: Inventory purpose-bound identities
ssh-keygen -lf ~/.ssh/id_backup_write.pub && ssh-keygen -lf ~/.ssh/id_backup_promote.pubContinue whenDistinct fingerprints are recorded for write and promotion; the recovery fingerprint is recorded from the separate recovery host.
Stop whenA private key is reused, recovery material exists on the source, or ownership and purpose are unclear.
If this step fails
Automation cannot unlock a passphrase-protected source key.
Likely causeThe unattended workflow has no approved agent or secret-unlock mechanism.
ssh-keygen -y -f ~/.ssh/id_backup_write >/dev/nullstat -c '%a %U:%G %n' ~/.ssh/id_backup_write
ResolutionUse an approved host-bound unlock mechanism or a no-passphrase key whose destination forced command and source address are tightly restricted.
Security notes
- Private keys use mode 600 and parent .ssh mode 700; public comments are not authorization controls.
Alternatives
- Use short-lived SSH certificates from a trusted CA when the environment supports automated issuance and revocation.
Stop conditions
- Do not use a personal administration key for unattended backup.
config
Restrict the destination account
Install separate write-only, promotion-only, and recovery-read keys with source restrictions and root-owned forced commands.
Why this step matters
The destination must enforce every privilege even when the source is hostile, so authorized_keys forces root-owned programs and disables generic SSH features.
What to understand
The write key runs rrsync -wo -no-overwrite -no-del under one absolute root. The options prevent reads, overwriting existing files, and receiver-side deletion; quotas still limit storage exhaustion.
The promotion key runs a reviewed helper that parses only `promote TIMESTAMP`, resolves paths beneath root, requires a completion marker, and atomically renames latest.new.
The recovery key uses rrsync -ro and an administrator source address. OpenSSH restrict disables forwarding, PTY, user rc, and agent/X11 forwarding unless explicitly re-enabled.
Keep the destination login shell and startup files root-controlled according to the current rrsync security guidance.
System changes
- Adds three restricted public keys and server-side forced-command policy to the destination backup account.
Syntax explained
restrict- Applies OpenSSH's broad authorized-key restrictions including no forwarding and no PTY.
from="address"- Limits key acceptance to the intended source or recovery network.
command="rrsync -wo -no-overwrite -no-del DIR"- Forces write-only access that cannot overwrite or delete retained receiver paths.
command="rrsync -ro DIR"- Forces read-only recovery access under the same root.
~{{user}}/.ssh/authorized_keysValues stay on this page and are never sent or saved.
from="{{sourceAddress}}",restrict,command="/usr/local/bin/rrsync -wo -no-overwrite -no-del {{root}}" {{writePublicKey}}
from="{{sourceAddress}}",restrict,command="/usr/local/sbin/backup-promote {{root}}" {{promotePublicKey}}
from="{{recoveryAddress}}",restrict,command="/usr/local/bin/rrsync -ro {{root}}" {{restorePublicKey}}Three restricted key lines installed; shell, PTY, forwarding, sibling paths, overwrite, and unapproved commands are rejected
Checkpoint: Prove restrictions before data transfer
ssh -i ~/.ssh/id_backup_write backup_source01@backup01.internal 'id'Continue whenThe general command is rejected by the forced rrsync policy; no interactive shell, PTY, forwarding, or arbitrary command is available.
Stop whenAny key opens a shell, accesses a sibling path, overwrites a completed snapshot, or gains unplanned forwarding.
If this step fails
The key still opens a shell.
Likely causeThe wrong authorized_keys file is active, the key line lacks the prefix, or sshd reads another key source.
sshd -T | grep -E 'authorizedkeys(file|command)'ssh-keygen -lf ~backup_source01/.ssh/authorized_keys
ResolutionRemove the unrestricted authorization, validate sshd configuration, and repeat negative tests before transferring data.
Security notes
- Forced commands must be root-owned and not writable through the restricted snapshot path.
Alternatives
- Use an rsync daemon-over-SSH module with a reviewed rsyncd.conf for multiple path policies.
Stop conditions
- Do not transfer production data until all negative authorization tests pass.
command
Preview the transfer
Review itemized changes, exclusions, trailing slash, and the new timestamp path while the server rejects receiver deletion.
Why this step matters
A fully itemized dry-run exposes path nesting, filter mistakes, and metadata churn before the destination changes, while the server rejects receiver-side deletion.
What to understand
-a preserves common metadata but does not imply ACLs, xattrs, or hard-link topology, so HAX are explicit and must be supported on both filesystems.
--numeric-ids preserves numeric identity rather than mapping names; this is useful for system restores but can create unsafe ownership if target identity plans differ.
Every run writes a previously unused timestamp, so paths absent from the sender are simply absent from that new view. The forced server policy rejects delete options and protects completed generations.
The trailing slash after source is intentional. Removing it adds another directory level and invalidates link-dest and restore paths.
System changes
- No destination data change because -n is active; reads source metadata and destination file lists.
Syntax explained
-aHAXn- Preserves archive metadata, hard links, ACLs, xattrs, and performs a dry-run.
--numeric-ids- Preserves numeric UID/GID values without name mapping.
--itemize-changes- Prints a compact reason code for every path difference.
--exclude-from- Loads reviewed filter rules from the named file.
-e "ssh -i ..."- Selects the dedicated write identity for the remote shell.
Values stay on this page and are never sent or saved.
rsync -aHAXn --numeric-ids --itemize-changes --exclude-from={{excludeFile}} -e "ssh -i ~/.ssh/id_backup_write" {{source}}/ {{user}}@{{host}}:{{timestamp}}/>f+++++++++ etc/app/config.yml .d..t...... var/lib/app/
Checkpoint: Approve the exact change set
rsync -aHAXn --numeric-ids --itemize-changes --exclude-from=/etc/backup/source01.exclude -e 'ssh -i ~/.ssh/id_backup_write' /srv/app/ backup_source01@backup01.internal:2026-07-25T020000Z/Continue whenEvery listed creation, update, metadata change, and exclusion matches the reviewed scope.
Stop whenThe destination is nested incorrectly, a critical path is absent, or any ownership change is unexplained.
If this step fails
The preview wants to copy the source directory under itself or an unexpected extra level.
Likely causeThe source trailing slash or destination relative path differs from the snapshot design.
rsync -n --list-only /srv/app/ backup_source01@backup01.internal:2026-07-25T020000Z/ | head
ResolutionCorrect source/destination semantics and rerun the full dry-run; never compensate during restore.
Security notes
- Filter files can accidentally exclude audit or recovery data; protect them in version control with review.
Alternatives
- Add --max-delete=0 during initial deployment to make any deletion fail closed.
Stop conditions
- Never remove -n until the itemized plan is accepted.
command
Create a dated snapshot
Link unchanged files to latest and write into a new timestamped directory.
Why this step matters
A new timestamped hierarchy isolates the run from previous generations, while link-dest reuses unchanged content without mutating the known-good snapshot.
What to understand
Use one UTC timestamp generated once and validated by both client and server. Never retry into a completed timestamp when -no-overwrite is enforced.
--link-dest=../latest is relative to the new destination directory, so it selects the sibling latest pointer. Files must match in content and preserved attributes to hard-link.
The first snapshot has no basis and should omit --link-dest or point at a reviewed empty bootstrap. A missing basis warning must not be ignored.
Capture rsync exit code and diagnostics. Codes 23 and 24 require explanation and must not trigger promotion automatically.
System changes
- Creates one new timestamped snapshot without permission to overwrite or delete any existing receiver path.
Syntax explained
-aHAX- Copies the reviewed content and preserves archive metadata, hard links, ACLs, and xattrs.
--link-dest=../latest- Hard-links unchanged files against the previous successful sibling snapshot.
--exclude-from- Applies the versioned scope filter while the server refuses delete operations.
{{timestamp}}/- Writes to a unique, not-yet-promoted destination directory.
Values stay on this page and are never sent or saved.
rsync -aHAX --numeric-ids --link-dest=../latest --exclude-from={{excludeFile}} -e "ssh -i ~/.ssh/id_backup_write" {{source}}/ {{user}}@{{host}}:{{timestamp}}/sent 14.2M bytes received 8.1K bytes total size is 8.4G
Checkpoint: Complete without touching latest
ssh -i ~/.ssh/id_backup_promote backup_source01@backup01.internal 'status 2026-07-25T020000Z'Continue whenThe server-side helper reports a complete new snapshot with plausible bytes/inodes while latest still points to the previous generation.
Stop whenrsync is nonzero, files vanished unexpectedly, capacity is low, the timestamp existed, or latest changed early.
If this step fails
rsync cannot create hard links or reports operation not permitted for ACL/xattr.
Likely causeDestination filesystem, mount options, account privileges, or metadata support do not match the selected preservation flags.
mount | grep '/backup'getfacl /backup/source01getfattr -d -m- /backup/source01
ResolutionDefine the required metadata semantics, provision compatible storage, and repeat into a new timestamp rather than dropping flags silently.
Security notes
- Write-only does not prevent storage exhaustion; monitor quota, bytes, inodes, and incomplete generations.
Alternatives
- Use destination-side filesystem snapshots instead of hard links when available.
Stop conditions
- A partial transfer never becomes latest.
command
Update latest only after success
Move the symlink atomically after rsync exits zero.
Why this step matters
Atomic promotion separates data transfer from the user-visible recovery pointer and preserves the previous known-good snapshot whenever transfer or validation fails.
What to understand
The server helper must read SSH_ORIGINAL_COMMAND, accept only a strict UTC timestamp, resolve the candidate under root, reject symlinks or missing markers, and log the old/new target.
ln creates latest.new and mv atomically replaces latest on the same filesystem. The helper, not client-supplied shell text, owns those exact operations.
Promotion does not make the snapshot immutable or prove application correctness; it marks the newest generation that passed the defined checks.
System changes
- Atomically changes the latest symlink from the previous verified snapshot to the exact completed timestamp.
Syntax explained
-i ~/.ssh/id_backup_promote- Selects the narrow forced-command identity rather than write or recovery access.
promote {{timestamp}}- Supplies one validated UTC identifier to the server-owned helper.
ln -sfn + mv -Tf- Server implementation stages a new symlink and atomically renames it over latest.
Values stay on this page and are never sent or saved.
ssh -i ~/.ssh/id_backup_promote {{user}}@{{host}} "promote {{timestamp}}"latest -> /backup/source01/2026-07-24T020000Z
Checkpoint: Confirm atomic pointer change
ssh -i ~/.ssh/id_backup_promote backup_source01@backup01.internal 'status latest'Continue whenlatest resolves to the exact new timestamp and the prior snapshot remains intact.
Stop whenThe helper accepts extra arguments, candidate is incomplete, latest resolves outside root, or prior generation changed.
If this step fails
Promotion rejects a snapshot after rsync exited zero.
Likely causeThe completion marker, timestamp policy, containment check, or post-transfer verification is missing.
ssh -i ~/.ssh/id_backup_promote backup_source01@backup01.internal 'status 2026-07-25T020000Z'
ResolutionKeep previous latest, inspect the server-side reason, and correct or repeat the snapshot without weakening validation.
Security notes
- The promotion helper must never eval SSH_ORIGINAL_COMMAND or interpolate it into a shell.
Alternatives
- Have a destination-side scheduled job detect and promote verified completion manifests.
Stop conditions
- Do not manually repoint latest around a failed helper.
verification
Verify source against snapshot
Use checksum dry-run for critical paths and inspect any differences.
Why this step matters
A checksum comparison from the recovery boundary proves critical content is readable through the recovery identity and detects silent differences hidden by size and timestamp shortcuts.
What to understand
-c makes rsync compare file checksums instead of relying only on size and modification time. It can be expensive, so prioritize critical paths and schedule broader scrubs.
The command uses the read-only recovery key from a separate host. An empty itemized result is expected; capture the exit status as evidence.
Also compare hard-link inode reuse, ACLs, xattrs, sparse allocation, and application-specific manifests where those properties matter.
System changes
- No intended file change because -n is active; reads source and destination content to compute checksums.
Syntax explained
-aHAXnc- Preserves comparison semantics, uses dry-run, and checks file contents.
--numeric-ids- Compares numeric ownership semantics.
id_backup_restore- Uses separately held read-only recovery access.
latest/- Reads the currently promoted complete snapshot.
Values stay on this page and are never sent or saved.
rsync -aHAXnc --numeric-ids -e "ssh -i ~/.ssh/id_backup_restore" {{source}}/ {{user}}@{{host}}:latest/No itemized differences; exit code 0
Checkpoint: Accept critical content verification
rsync -aHAXnc --numeric-ids -e 'ssh -i ~/.ssh/id_backup_restore' /srv/app/ backup_source01@backup01.internal:latest/Continue whenNo unexplained itemized differences and exit code zero.
Stop whenContent differs, metadata is unsupported, read access fails, or the source changed during verification without recorded application coordination.
If this step fails
Checksum comparison reports many files changed immediately after backup.
Likely causeThe source remained active, timestamps are not the issue because -c compares content, or the wrong snapshot was promoted.
readlink -f /backup/source01/latestdate -ulsof +D /srv/app | head
ResolutionReview application-consistency hooks and selected timestamp; create a new coherent generation rather than editing the snapshot.
Security notes
- Recovery access should be audited and unavailable to ordinary source automation.
Alternatives
- Verify a signed application manifest or repository checksum catalog at the destination.
Stop conditions
- An unexplained checksum difference blocks backup promotion and retention.
warning
Preview retention candidates
Print exact timestamped directories before any deletion.
Why this step matters
Retention is the only intentionally destructive storage step, so candidate selection must be printed, bounded, protected from latest, and separated from deletion.
What to understand
The find expression selects only direct child directories matching a strict timestamp shape and older than the defined age. It prints; it does not delete.
A robust destination job excludes latest's resolved target, preserves minimum daily/weekly/monthly generations, checks legal holds, and caps deletion count.
Hard-link accounting means apparent directory size is not reclaimable bytes. Measure filesystem allocation and inode pressure before and after a reviewed deletion.
System changes
- No change in preview form; a later separately approved destination-side action may permanently delete exact snapshot directories.
Syntax explained
-mindepth 1 -maxdepth 1- Restricts selection to direct children of the snapshot root.
-type d- Selects directories, not the latest symlink or ordinary files.
-name '20??-??-??T??????Z'- Limits candidates to the strict snapshot naming family.
-mtime +30- Selects entries older than thirty 24-hour periods by directory modification time.
-print- Previews exact candidates without deleting anything.
Values stay on this page and are never sent or saved.
find {{root}} -mindepth 1 -maxdepth 1 -type d -name '20??-??-??T??????Z' -mtime +30 -print/backup/source01/2026-06-10T020000Z
Checkpoint: Approve bounded retention candidates
find /backup/source01 -mindepth 1 -maxdepth 1 -type d -name '20??-??-??T??????Z' -mtime +30 -printContinue whenEvery candidate is an expired complete snapshot, none is latest, and required daily/weekly/monthly plus immutable copies remain.
Stop whenSelection is empty unexpectedly, includes latest, exceeds a deletion cap, or a restore drill has not passed.
If this step fails
Deleting an old directory recovers far less space than expected.
Likely causeMost files are hard-linked to retained snapshots or filesystem snapshots still reference the blocks.
du -sh /backup/source01/*du -sh --count-links /backup/source01/*df -h /backup/source01df -i /backup/source01
ResolutionUse filesystem-aware allocation reports and adjust retention based on generations, bytes, and inodes without deleting the last verified restore point.
Security notes
- Run actual deletion only on the destination under separate authorization unavailable to the source.
Alternatives
- Use storage-native retention and immutable snapshot policies.
Stop conditions
- Never append -delete or pipe preview output into rm as an unreviewed one-liner.
verification
Restore into a clean location
Copy representative data from a historical snapshot and compare metadata.
Why this step matters
A clean-path restore is the only practical proof that historical content is readable, recovery credentials work, and metadata can support the application.
What to understand
Keep -n for the first restore preview, then remove it only after target path, free space, ownership semantics, and selected historical timestamp are approved.
Restore into an empty non-production directory or disposable host. Never overlay a running application during a drill.
Compare content, ACLs, xattrs, ownership, hard links, sparse allocation, configuration parsing, and application read-only behavior. Record duration against recovery objectives.
System changes
- Dry-run changes nothing; the approved follow-up creates a representative restore tree in a clean test location.
Syntax explained
-aHAXn- Previews restoration while preserving the declared metadata model.
id_backup_restore- Uses read-only recovery access stored outside the protected source.
{{timestamp}}/{{restorePath}}/- Selects one exact historical generation and relative recovery scope.
./restore-test/- Targets an explicitly empty local rehearsal directory.
Values stay on this page and are never sent or saved.
rsync -aHAXn -e "ssh -i ~/.ssh/id_backup_restore" {{user}}@{{host}}:{{timestamp}}/{{restorePath}}/ ./restore-test/>f+++++++++ config.yml .d..t...... data/
Checkpoint: Prove historical recovery
diff -r expected/ restore-test/ && getfacl -R restore-test/ >/tmp/restore-test.aclContinue whenContent and required metadata match, the application parses restored configuration, and measured duration meets the objective.
Stop whenThe source timestamp is wrong, target is non-empty, metadata cannot be restored, or application verification fails.
If this step fails
Recovery cannot read a snapshot even though backups continue writing.
Likely causeThe separate recovery key expired, was never tested, or its source-address restriction no longer matches.
ssh-keygen -lf ~/.ssh/id_backup_restore.pubssh -vv -i ~/.ssh/id_backup_restore backup_source01@backup01.internal true
ResolutionRestore read-only access through the documented break-glass process and schedule recurring key-path tests independent of backup writes.
Security notes
- Restored data inherits production sensitivity; isolate and erase the drill target according to policy.
Alternatives
- Restore an entire application stack into a disposable recovery environment for stronger acceptance testing.
Stop conditions
- Do not overwrite live data during a routine drill.
Finish line
Verification checklist
rsync -aHAXnc source/ host:/backup/latest/No unexplained differences.ssh host readlink -f /backup/source/latestPoints to the newest successful snapshot.diff -r expected/ restore-test/No differences.Recovery guidance
Common problems and safe checks
rsync reports protocol version mismatch or unexplained text before the protocol starts.
Likely causeA destination shell startup file, banner, or forced-command wrapper writes output on the binary rsync channel.
ssh -i ~/.ssh/id_backup_write backup_source01@backup01.internal /bin/true > /tmp/rsync-ssh-outputwc -c /tmp/rsync-ssh-output
ResolutionRemove unauthorized startup output and ensure the forced command emits only the rsync protocol; do not bypass the restriction.
Unchanged files consume full additional space instead of sharing inodes.
Likely cause--link-dest points to the wrong relative directory, preserved attributes differ, or source/destination filesystems do not support the expected hard links.
readlink -f /backup/source01/lateststat -c '%i %h %n' /backup/source01/latest/etc/app/config.yml /backup/source01/2026-07-25T020000Z/etc/app/config.ymldf -i /backup/source01
ResolutionCorrect the basis path and metadata policy on a new snapshot; never rewrite completed generations in place.
rsync exits 23 or 24 with partial transfer or vanished files.
Likely causePermissions, I/O errors, source churn, disappearing temporary files, unsupported metadata, or an unreviewed mount interrupted completeness.
df -h /backup/source01df -i /backup/source01journalctl --since '1 hour ago' --no-pager | tail -n 100
ResolutionKeep latest unchanged, preserve the incomplete timestamp for diagnosis, correct the source consistency or destination error, and repeat into a new timestamp.
A restored file has content but the application still fails.
Likely causeOwnership, ACLs, xattrs, sparse layout, external secrets, database state, or dependent services were not captured or restored correctly.
getfacl -p ./restore-test/config.ymlgetfattr -d -m- ./restore-test/config.ymlstat ./restore-test/config.yml
ResolutionExtend backup scope and the restore acceptance test; do not weaken permissions simply to make the test pass.
Reference
Frequently asked questions
Does --link-dest make incremental backups?
Transfer and storage are incremental, but each timestamp is presented as a complete directory tree. Unchanged regular files share blocks through hard links.
Why not let the source update latest directly?
A narrow promotion command can validate completion and prevents a failed transfer from replacing the last known-good pointer. It also keeps arbitrary shell access out of the backup account.
Can rsync back up a running database directory?
A byte copy may be unusable or inconsistent. Use the database's supported backup or snapshot mechanism, then transfer the resulting consistent artifact.
Recovery
Rollback
A failed run must not move latest; remove only its incomplete snapshot.
- Confirm latest points elsewhere.
- Remove the exact incomplete timestamped directory.
- Keep the previous latest snapshot unchanged.
- Correct configuration and repeat the dry run.
Evidence