Recover an Ubuntu server with systemd rescue and emergency modes
Recover a non-booting Ubuntu Server by choosing the least invasive systemd target, preserving console evidence, identifying the failed boot dependency, applying a reversible correction, and proving a clean normal boot.
Restore an Ubuntu 24.04 LTS host to its normal target without guessing at the cause, repairing a mounted filesystem unsafely, or hiding a persistent failure behind an unconditional service disable.
- Ubuntu Server 24.04 LTS
- systemd 255.x
- GRUB 2.x
- Out-of-band console Keep provider, hypervisor, serial, or physical console access for the entire recovery. SSH and ordinary networking may never start, and a reboot can terminate the only remote session.
Confirm that the console displays the firmware or GRUB screen and accepts keyboard input. - Known-good backup or snapshot Record the most recent restorable backup and its recovery procedure before writing to the root filesystem. A hypervisor snapshot is useful only when its storage is healthy and the snapshot can actually be reverted.
- Change history Collect the last package, kernel, storage, mount, and service changes from the operator log or automation system. Recovery is faster and safer when the investigation begins with a bounded change set.
- Disk identity map Know whether root uses a plain partition, LVM, encryption, software RAID, or remote storage. Do not translate a device name from another host into this server.
lsblk --fs --output NAME,TYPE,FSTYPE,FSVER,LABEL,UUID,MOUNTPOINTS
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A repeatable decision path from a failed Ubuntu boot to rescue.target, emergency.target, or offline recovery media. The path deliberately starts with the least invasive environment that can expose the real failure.
- An evidence bundle containing the kernel command line, failed unit graph, current-boot journal, storage identity, mount graph, network state, and pre/post-recovery comparison.
- A reversible correction workflow that distinguishes a bad fstab entry or service dependency from filesystem corruption and hardware I/O failure, then proves an ordinary boot without permanent recovery flags.
- The host reaches its configured normal target after a fresh boot, systemctl reports running, and the failed-unit list is empty or contains only a separately approved non-critical exception.
- Required local filesystems validate against stable identifiers, application dependencies start in the intended order, and the current boot journal contains no repeated storage or dependency failure.
- The incident record names the first causal error, the bounded correction, the evidence used to verify it, and the recovery path if the problem returns.
Architecture
How the parts fit together
Firmware and GRUB load a selected kernel and initramfs. The initramfs discovers the root stack—possibly encryption, md RAID, and LVM—then hands control to systemd as PID 1. systemd constructs a transaction from the requested target and its dependencies. rescue.target starts a mostly local base system plus a rescue shell; emergency.target starts an even smaller shell and commonly keeps root read-only. The operator inspects the dependency graph and journal, corrects one proven cause, tests multi-user.target in the same boot, and finally performs a normal reboot with no one-time target override.
- Capture the on-screen failure and boot one kernel entry with systemd.unit=rescue.target; step down to emergency.target only if rescue cannot be reached.
- Authenticate locally, preserve the read-only state where possible, and inventory the active kernel command line, root stack, failed units, dependency graph, and current-boot journal.
- Classify the failure as configuration/dependency, mount identity, filesystem corruption, or lower-level device I/O. A corruption or I/O case leaves this workflow for offline repair or restore.
- Apply one reversible correction, reload systemd, and isolate multi-user.target while the recovery console is still available.
- Reboot without the temporary kernel argument, verify the normal target and service health, and retain the evidence with the incident record.
Assumptions
- The guide targets Ubuntu Server 24.04 LTS using systemd as PID 1 and GRUB as the ordinary bootloader. Cloud images may hide GRUB unless the provider console sends the correct key at the correct time.
- The operator possesses authorized root recovery credentials and can identify the host, tenant, and storage before entering commands. Recovery access is not a reason to bypass ownership or change-control requirements.
- The boot problem occurs after firmware can select a boot device. Missing disks, broken firmware, invalid Secure Boot signatures, and an unreadable GRUB installation require a lower-level platform procedure.
- A current restorable backup exists for important data. Rescue and emergency modes improve access; they do not make filesystem repair, package replacement, or storage writes reversible.
- The example failing unit is example-agent.service and the example filesystem device is /dev/mapper/ubuntu--vg-root. Both must be replaced only after this host's evidence identifies the real objects.
Key concepts
- rescue.target
- A special systemd target that brings up the base system and local mounts needed for a single-user rescue shell. It is preferable when those dependencies work because normal systemctl and journal facilities are more complete.
- emergency.target
- A minimal systemd target intended for failures so early that rescue cannot be reached. Root is normally mounted read-only, few units are active, and the operator must explicitly opt into writes.
- systemd transaction
- The set of unit jobs systemd computes to reach a target. A mount or device unit can fail and cause several dependent services to stop, so the last visible failure is often not the root cause.
- current-boot journal
- Records associated with boot ID zero. Restricting investigation to -b 0 prevents an old resolved error from being confused with the present boot failure.
- offline filesystem repair
- A check or repair performed while the target filesystem is unmounted. Metadata repair against a mounted read-write filesystem can race the kernel and cause additional damage.
- runtime mask
- A temporary /run/systemd/system link to /dev/null that makes one unit impossible to start until reboot. It is a diagnostic control, not a permanent unexplained fix.
Before you copy
Values used in this guide
{{failingUnit}}One non-essential unit proven by the current boot's journal and dependency graph to block progress. Never substitute local-fs.target, cryptsetup, storage, authentication, firewall, or another foundational unit.
Example: example-agent.service{{filesystemDevice}}Canonical block device containing the filesystem implicated by kernel or systemd-fsck evidence. Resolve it through lsblk, blkid, LVM, encryption, and md layers before use.
Example: /dev/mapper/ubuntu--vg-rootSecurity and production boundaries
- Anyone with an unrestricted local console and permission to edit the kernel command line may obtain powerful recovery access. Protect provider consoles, firmware settings, boot media, GRUB policy, and physical access according to the host threat model.
- Do not paste complete journals into a public ticket before redaction. Boot logs may contain hostnames, network addresses, device UUIDs, mount paths, user names, and arguments supplied to services.
- A service mask can restore availability while silently removing security, logging, backup, or integrity controls. Use a runtime mask only for a named non-essential unit and require a root-cause correction before incident closure.
- Filesystem repair is a data mutation. Image the failing media or prove a restore path first when the data is valuable or the device reports I/O errors.
Stop before continuing if
- Stop when the console identifies a different host, disk map, tenant, or boot entry than the approved target. Device names are not proof of identity.
- Stop in read-only mode if the kernel reports medium errors, repeated I/O resets, md failures, or an unstable storage path. Capture evidence and replace or image the device before writing.
- Stop before running fsck repair when the filesystem is mounted, when the filesystem type is unknown, or when no restorable backup exists for important data.
- Stop before masking a unit whose operational or security role is not understood. The correct action may be to restore a dependency rather than remove the consumer.
- Stop after a failed same-boot isolation test. Preserve the new journal instead of stacking unrelated edits until the system happens to start.
instruction
Establish a recovery boundary before changing boot state
Confirm console control, photograph or capture the exact failure, record the last successful boot, and attempt the previous installed kernel once when the failure began immediately after a kernel update.
Why this step matters
A boot failure removes ordinary observability and remote access at the same time. Establishing a trusted console, host identity, and evidence boundary prevents an urgent recovery from becoming an unaudited sequence of destructive guesses.
What to understand
Capture the exact first error, timestamps, unit names, device identifiers, and any preceding kernel messages. A photograph is often more useful than a paraphrased ticket when the journal is not yet writable.
A previous kernel is a bounded diagnostic when the failure immediately follows an update. It does not justify deleting the new kernel, rebuilding every initramfs, or changing Secure Boot until the comparison is recorded.
Confirm backup recency and the business owner before writes. A bootable server with silently damaged data is not a successful recovery.
System changes
- No persistent change; this step records access, identity, failure evidence, and the chosen recovery branch.
Syntax explained
Previous kernel- A one-boot comparison that can separate kernel/initramfs regression from persistent userspace configuration.
Console capture- Preserves messages emitted before persistent journaling or networking is available.
Console connected: tty1 Observed stop: Dependency failed for /srv/data Last successful boot: 2026-07-27 03:14 UTC Selected recovery path: systemd rescue target
Checkpoint: Confirm recovery prerequisites
Continue whenConsole input works, the target host is identified, evidence is captured, and a restore path is named.
Stop whenThe console or storage belongs to a different host, or no owner accepts the risk of a write.
Security notes
- Treat the provider or hypervisor console as privileged root access and close shared screen recordings after the incident.
Alternatives
- Boot a replacement instance from the last verified image when the console cannot be trusted or the recovery-time objective is shorter than diagnosis.
Stop conditions
- Do not edit GRUB, storage, or configuration before confirming host identity and recovery authority.
instruction
Choose rescue mode first and emergency mode only when needed
At GRUB, edit one boot entry and append systemd.unit=rescue.target to the linux line. Use systemd.unit=emergency.target only when rescue itself cannot complete because emergency starts a more minimal environment and normally leaves root read-only.
Why this step matters
Choosing rescue before emergency preserves more diagnostic facilities while still avoiding the normal service transaction. A one-time kernel argument also leaves the saved bootloader configuration unchanged.
What to understand
Append the argument to the existing linux line; do not replace root=, ro, encryption, console, or crash-kernel arguments. The selected kernel and initramfs still need the original storage description.
rescue.target normally mounts local filesystems and starts essential services before presenting a root shell. emergency.target is intentionally smaller and may provide fewer tools or no writable root.
The edit applies to one boot. If a target argument appears after the next ordinary boot, investigate persistent GRUB defaults or an automation change.
System changes
- Changes only the selected in-memory GRUB entry for one boot; no bootloader file is written.
Syntax explained
systemd.unit=rescue.target- Requests the rescue target as systemd's initial goal for this boot.
systemd.unit=emergency.target- Requests the minimal emergency shell when rescue dependencies cannot start.
Ctrl-x / F10- Boots the edited entry without saving the edit to GRUB configuration.
linux /boot/vmlinuz-6.8.0-71-generic root=/dev/mapper/ubuntu--vg-root ro systemd.unit=rescue.target Press Ctrl-x or F10 to boot this one-time entry.
Checkpoint: Prove the selected one-time target
cat /proc/cmdlineContinue whenThe command line contains exactly one intended systemd.unit recovery argument and the original root identity.
Stop whenThe root argument or other required kernel parameters were accidentally removed.
If this step fails
The edited entry immediately returns to GRUB or cannot find root.
Likely causeThe linux line was damaged while adding the argument, or the selected kernel/initramfs cannot assemble the storage stack.
Reopen the unmodified saved GRUB entry.Compare the root= and initrd lines character for character.Try the previously installed kernel once.
ResolutionDiscard the unsaved edit and retry with only the target suffix; use recovery media when neither installed initramfs can find root.
Security notes
- Do not add init=/bin/sh as a routine shortcut; it bypasses more of the managed boot and produces a different diagnostic environment.
Alternatives
- Use Ubuntu installation media when neither systemd target can be reached or when an unmounted root filesystem is required.
Stop conditions
- Do not continue if the edited line no longer names the known root stack.
verification
Identify the target, root mount, and boot transaction
After authenticating on the local console, record the actual kernel command line, system state, root source, filesystem type, and mount flags before attempting a repair.
Why this step matters
The prompt alone does not reveal whether the host is in rescue, emergency, initramfs, or a chroot. Recording PID 1 context, command line, root source, and storage topology prevents commands intended for one environment from damaging another.
What to understand
systemctl is-system-running commonly reports maintenance in a recovery target. A nonzero exit is expected and therefore guarded with || true rather than hidden from the evidence.
findmnt reads the kernel's effective mount table. Its source and options are more authoritative than assumptions based on /etc/fstab or legacy device names.
lsblk links partitions, LVM devices, filesystem UUIDs, and mount points. If encryption or md RAID exists, include those layers before selecting a repair device.
System changes
- No persistent change; reads kernel command line, manager state, mount table, and block-device metadata.
Syntax explained
is-system-running- Summarizes systemd manager health; maintenance is expected in rescue or emergency mode.
findmnt --noheadings --output- Returns only the requested effective root-mount fields.
lsblk --fs- Shows filesystem signatures and UUIDs across the device stack without mounting them.
cat /proc/cmdline && systemctl is-system-running || true; findmnt --noheadings --output TARGET,SOURCE,FSTYPE,OPTIONS /; lsblk --fs --output NAME,TYPE,FSTYPE,UUID,MOUNTPOINTSBOOT_IMAGE=/boot/vmlinuz-6.8.0-71-generic root=/dev/mapper/ubuntu--vg-root ro systemd.unit=rescue.target maintenance / /dev/mapper/ubuntu--vg-root ext4 ro,relatime ubuntu--vg-root lvm ext4 2c0e... /
Checkpoint: Map the installed root
findmnt / && lsblk --fsContinue whenOne understood root source, filesystem type, UUID, and current read/write state are visible.
Stop whenRoot comes from an unknown snapshot, clone, degraded array, or unexpected device.
Security notes
- Device UUIDs and host topology can be sensitive infrastructure data; redact them before external sharing.
Alternatives
- Use blkid and LVM/mdadm inventory tools when lsblk cannot expose an assembled layer clearly.
Stop conditions
- No write occurs until the root stack and current mount state are understood.
decision
Remount root read-write only for an identified file correction
Remain read-only while gathering evidence. If a specific configuration file must be corrected and the filesystem shows no I/O or corruption errors, remount only root read-write and immediately verify the effective flags.
Why this step matters
Emergency mode's read-only root is a safety boundary. Remounting is justified only after a specific editable file is implicated and kernel evidence does not suggest media or filesystem damage.
What to understand
The first findmnt output records the pre-change state. The second proves that the remount affected the intended root rather than a recovery-media filesystem.
A remount changes kernel mount flags but does not itself repair corruption. Repeated I/O errors, ext4 aborts, or XFS shutdown messages require offline handling.
Keep the write window short: back up the exact file, make one reviewed correction, sync if appropriate, and return to verification.
System changes
- Changes the installed root mount from read-only to read-write for the current boot.
Syntax explained
mount --options remount,rw /- Changes flags on the already mounted root without selecting a new source.
findmnt --output OPTIONS /- Shows the effective flags before and after the remount.
findmnt --noheadings --output OPTIONS / && sudo mount --options remount,rw / && findmnt --noheadings --output TARGET,SOURCE,FSTYPE,OPTIONS /ro,relatime / /dev/mapper/ubuntu--vg-root ext4 rw,relatime
Checkpoint: Confirm a controlled write window
findmnt --noheadings --output TARGET,SOURCE,FSTYPE,OPTIONS /Continue whenThe intended installed root is rw and no storage error appeared during remount.
Stop whenRemount fails, the source is unexpected, or the kernel logs a filesystem or device error.
If this step fails
mount reports that the filesystem cannot be remounted read-write.
Likely causeThe filesystem was forced read-only after an error, the device is unavailable, or this environment does not own the installed root.
journalctl -k -b --no-pager | tail -n 100findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS /lsblk --fs
ResolutionPreserve the read-only state and move to offline filesystem or hardware recovery; do not force a write mount.
Security notes
- A writable root lets recovery credentials alter authentication and boot policy; maintain console custody and an audit trail.
Alternatives
- Mount the installed filesystem under /mnt from trusted recovery media and edit only the proven file after preserving its original.
Stop conditions
- Never remount read-write merely to explore or because a later command expects writes.
verification
Trace failed units and the first causal journal error
List failed units, inspect the current boot at warning priority, and view the dependency chain for the first failed unit. Work from the earliest causal error rather than the final emergency-target message.
Why this step matters
systemd reports propagated dependency failures alongside their cause. Aligning failed units, precise current-boot timestamps, and the failed dependency graph identifies the first broken contract rather than the loudest downstream symptom.
What to understand
Limit journal review to boot zero and warning-or-higher priority for the initial pass, then widen around the first causal timestamp if informational context is needed.
A mount failure often prevents local-fs.target, which in turn prevents many applications. Repairing each application would not restore the missing mount.
Kernel I/O messages may precede the systemd unit failure. Review journalctl -k when the first userspace error names a block device or filesystem.
System changes
- No persistent change; reads failed-unit state, dependency relationships, and journal records.
Syntax explained
systemctl --failed- Lists units currently in failed state.
journalctl --boot=0- Restricts evidence to the current boot ID.
--priority=warning- Starts with warning, error, critical, alert, and emergency records.
list-dependencies --failed --all- Shows failure propagation across the target graph.
systemctl --failed --no-pager --plain; journalctl --boot=0 --priority=warning --no-pager --output=short-precise | tail -n 120; systemctl list-dependencies --failed --allUNIT LOAD ACTIVE SUB DESCRIPTION srv-data.mount loaded failed failed /srv/data Jul 28 08:41:13.420001 host systemd[1]: srv-data.mount: Mount process exited, code=exited, status=32 Jul 28 08:41:13.420991 host systemd[1]: Dependency failed for local-fs.target.
Checkpoint: Name the first causal failure
Continue whenThe incident notes one earliest causal unit or kernel error and the downstream units it blocked.
Stop whenEvidence points to hardware I/O or corruption that must be handled offline.
Security notes
- Service arguments and journal fields may expose tokens or private paths; redact before copying outside the authorized incident system.
Alternatives
- Use systemd-analyze critical-chain and journalctl fields such as _SYSTEMD_UNIT for a large or ambiguous transaction.
Stop conditions
- Do not change multiple downstream services when a shared failed dependency explains them.
verification
Validate fstab without mounting unknown filesystems
When the failure involves a mount, compare fstab identifiers with lsblk and use findmnt verification. Correct only an entry whose intended device, mount point, filesystem, and boot requirement are known.
Why this step matters
fstab errors are common boot blockers, but blindly commenting entries can remove durability or security guarantees. A preserved file plus stable-identifier comparison and parser verification makes the correction reviewable.
What to understand
cp --archive preserves timestamps, mode, ownership, ACLs, and extended attributes of the pre-recovery file at a clear adjacent path.
Compare UUID= or LABEL= with lsblk output. A kernel device name such as /dev/sdb1 can change across boots and should not be introduced as a shortcut.
findmnt --verify checks parsing, sources, target paths, and options without intentionally mounting every untrusted source. Review every warning, not only the exit status.
System changes
- Creates /etc/fstab.before-boot-recovery; any subsequent fstab correction is a separate reviewed write.
Syntax explained
cp --archive- Preserves the original file's metadata for rollback and comparison.
lsblk --fs- Displays the stable filesystem identifiers present on this host.
findmnt --verify --verbose- Parses fstab and reports source, target, and option problems with detail.
sudo cp --archive /etc/fstab /etc/fstab.before-boot-recovery && lsblk --fs --output NAME,FSTYPE,LABEL,UUID,MOUNTPOINTS && findmnt --verify --verboseSuccess, no errors or warnings detected 0 parse errors, 0 errors, 0 warnings
Checkpoint: Require a valid mount contract
findmnt --verify --verboseContinue whenZero parse errors and no unresolved required source or invalid option.
Stop whenThe intended device or business requirement for a mount cannot be established.
If this step fails
findmnt says a UUID is unreachable, while a similar disk appears in lsblk.
Likely causeThe filesystem was recreated, a clone duplicated identifiers, the wrong disk is attached, or the expected LVM/encryption layer is not assembled.
blkidlsblk -o NAME,SERIAL,WWN,FSTYPE,UUID,MOUNTPOINTSpvs; vgs; lvs
ResolutionIdentify the intended storage by asset identity and recovery documentation. Update fstab only when the replacement relationship is proven.
Security notes
- Do not use nofail to hide a missing mount that carries logs, authentication data, databases, or another required control.
Alternatives
- Create a properly ordered systemd mount unit when the dependency cannot be represented safely by a simple fstab entry.
Stop conditions
- Do not replace a stable identifier with a convenient device name from the current boot.
command
Temporarily mask one confirmed non-essential failing unit
If a non-essential service—not a mount required for data integrity, authentication, storage, or networking—blocks the boot, apply a runtime-only mask so the same boot can be tested without persisting an unexplained disable.
Why this step matters
A runtime mask is a reversible experiment that answers whether one confirmed non-essential unit blocks the transition to normal service. It disappears at reboot, preventing the experiment from silently becoming permanent policy.
What to understand
The placeholder must be a complete unit name taken from systemctl status and the current journal. Shell fragments or globs are not accepted input.
mask is stronger than disable: it prevents manual and dependency starts for this boot. That makes scope review essential before using it.
Record why the unit is non-essential, who owns it, and how its absence will be detected during the test.
System changes
- Creates a temporary /run/systemd/system link to /dev/null for the selected unit; it is removed by reboot.
Syntax explained
mask --runtime- Makes the unit unstartable only in volatile /run state.
is-enabled- Reports masked-runtime as evidence that the temporary control is active.
Values stay on this page and are never sent or saved.
sudo systemctl mask --runtime {{failingUnit}} && systemctl is-enabled {{failingUnit}}Created symlink /run/systemd/system/example-agent.service → /dev/null. masked-runtime
Checkpoint: Confirm narrow runtime isolation
systemctl is-enabled {{failingUnit}} && systemctl status {{failingUnit}} --no-pagerContinue whenOnly the reviewed unit is masked-runtime and no foundational dependency is affected.
Stop whenThe unit provides storage, networking, authentication, firewalling, logging, monitoring, or backups needed for safe operation.
Security notes
- Masking a security or audit agent can create an unobserved recovery session; require explicit owner approval for any such component.
Alternatives
- Correct and validate the unit configuration immediately when the root cause is already known and the correction is lower risk than isolation.
Stop conditions
- Never mask a target, mount, socket, path, or service whose downstream role has not been mapped.
decision
Stop and move filesystem repair to an offline environment
If the journal reports filesystem corruption or repeated block I/O errors, identify the device and prove whether it is mounted. Use fsck only from recovery media or an initramfs context where that filesystem is unmounted; never force repair against a mounted root filesystem.
Why this step matters
Filesystem repair changes metadata and can discard damaged objects. The safe decision is based on the exact device, filesystem type, mount state, kernel evidence, and backup—not on the desire to make the boot continue.
What to understand
fsck --no-act requests a non-repairing check where supported, but even read-only checks against a mounted changing filesystem may be inconsistent. Here it primarily demonstrates that the mounted root must leave this workflow.
For XFS, use XFS-specific diagnostic and repair guidance; generic fsck normally delegates differently. For encrypted, LVM, or md storage, select the filesystem-bearing upper device, not a lower member.
Repeated transport resets, SMART errors, or md degradation require hardware or array stabilization before filesystem repair.
System changes
- No intended persistent change; inventories the named filesystem device and refuses mounted repair.
Syntax explained
findmnt --source- Proves whether the exact device is currently mounted and where.
fsck --no-act- Requests a non-writing check when the filesystem checker supports it; it is not permission to inspect a busy mounted root carelessly.
Values stay on this page and are never sent or saved.
lsblk --fs {{filesystemDevice}}; findmnt --source {{filesystemDevice}} || true; sudo fsck --no-act {{filesystemDevice}}NAME FSTYPE FSVER LABEL UUID MOUNTPOINTS ubuntu--vg-root ext4 1.0 2c0e... / fsck from util-linux 2.39.3 /dev/mapper/ubuntu--vg-root is mounted.
Checkpoint: Choose offline repair or restore
Continue whenThe filesystem type, canonical device, mount state, device health, and backup are known before a repair command is planned.
Stop whenThe device is mounted, storage is unstable, type is unknown, or the backup cannot be restored.
If this step fails
fsck reports that the device is mounted and aborts.
Likely causeThe check targets the active root or another filesystem assembled in rescue mode.
findmnt --source {{filesystemDevice}}lsblk --fs {{filesystemDevice}}journalctl -k -b --no-pager | tail -n 100
ResolutionDo not override the protection. Boot trusted recovery media, leave the target unmounted, then follow the filesystem-specific repair plan.
Security notes
- Recovery media must be trusted and controlled; an unknown image can read every unencrypted secret on the attached storage.
Alternatives
- Restore to replacement storage and retain the original media for forensic or specialist recovery.
Stop conditions
- Never run repair mode against a mounted filesystem or an unstable block device.
verification
Reload configuration and test the normal target in the same boot
After a bounded correction, reload systemd, clear historical failed state, and isolate the server's normal multi-user target. This exercises mount ordering and service activation while console access is still present.
Why this step matters
Testing the normal target before reboot exercises the corrected dependency graph while the working recovery console remains attached. It reduces the chance that an unverified edit converts a recoverable host into another blind reboot.
What to understand
daemon-reload makes PID 1 reparse changed units and generated mount definitions. reset-failed clears historical failure state so new failures are distinguishable.
isolate stops units not required by multi-user.target and starts its dependencies. This can interrupt services, so the host must already be in an approved recovery window.
The wait form of is-system-running returns only after the manager reaches a stable state, but application-specific health still requires later checks.
System changes
- Reloads systemd definitions, clears failed markers, and changes the active unit set to multi-user.target.
Syntax explained
daemon-reload- Re-runs unit generators and reloads definitions after configuration changes.
reset-failed- Clears prior failure state and rate-limit counters before the controlled retest.
isolate multi-user.target- Starts the server target and stops units not part of its dependency closure.
is-system-running --wait- Waits for systemd to settle before reporting overall state.
sudo systemctl daemon-reload && sudo systemctl reset-failed && sudo systemctl isolate multi-user.target && systemctl is-system-running --waitrunning
Checkpoint: Reach a stable multi-user state
systemctl is-system-running --wait && systemctl --failed --no-pagerContinue whenrunning and zero newly failed required units.
Stop whenThe console becomes unstable, required services fail, or storage errors recur.
Security notes
- Do not start public-facing services until firewall and authentication dependencies are confirmed active.
Alternatives
- Start the corrected unit and its immediate target manually before full isolation when service impact must be narrower.
Stop conditions
- Do not reboot after a failed same-boot target test.
verification
Prove services, mounts, and networking before reboot
Check the default target, failed units, required mounts, addresses, routes, and listeners. Compare the state with the captured baseline and application runbook rather than assuming that a login prompt equals recovery.
Why this step matters
Systemd manager state proves only one layer. Mount validation, addresses, routes, and listeners establish whether the recovered host has the dependencies and exposure expected by its application runbook.
What to understand
Compare listeners with the pre-incident inventory and firewall policy. A recovered daemon bound to every address rather than loopback is a security regression.
findmnt verification must remain clean after generated units are reloaded. A parse-successful but missing optional mount should still be recorded.
Application health checks, queue checks, replication state, or cluster membership belong beside these host checks before traffic returns.
System changes
- No persistent change; reads target, failed-unit, mount, network, route, and socket state.
Syntax explained
get-default- Shows the target selected for ordinary boots.
ss --listening --numeric --tcp --udp- Lists listening TCP and UDP sockets without slow name resolution.
ip --brief address- Provides compact interface and address state.
systemctl get-default; systemctl --failed --no-pager --plain; findmnt --verify; ip --brief address; ip route show; ss --listening --numeric --tcp --udpmulti-user.target 0 loaded units listed. Success, no errors or warnings detected ens3 UP 192.0.2.20/24 default via 192.0.2.1 dev ens3 Netid State Local Address:Port
Checkpoint: Match the approved host baseline
Continue whenTarget, mounts, network, routes, firewall, listeners, and application checks match the documented role.
Stop whenA required mount or control is absent, or an unexpected listener is exposed.
Security notes
- Do not interpret network reachability as authorization; validate the firewall and service authentication independently.
Alternatives
- Keep the host isolated from load balancers and orchestration while completing application-specific checks.
Stop conditions
- No production traffic returns while a required dependency or security control is degraded.
warning
Reboot once and verify an unmodified normal boot
Remove only the one-time GRUB argument by rebooting normally. Keep the console attached, time the boot, and after login prove that the default target was reached without a runtime mask or new failed unit.
Why this step matters
A successful target isolation in a recovery boot is not final proof because runtime masks and one-time state disappear. One observed normal reboot verifies the persistent configuration and the full initramfs-to-systemd path.
What to understand
Use the attached console rather than relying on SSH reconnect. Record the time from reboot request to login and the first visible warning.
Do not save systemd.unit in GRUB defaults. An ordinary boot must use the existing default target and normal kernel command line.
The broadcast message is the last expected local evidence before processes stop; lack of immediate console output does not justify a power cycle.
System changes
- Stops services, syncs filesystems through the normal systemd reboot transaction, and starts a fresh boot.
Syntax explained
systemctl reboot- Requests an orderly reboot through PID 1 instead of forcing hardware reset.
sudo systemctl rebootBroadcast message from root@host on pts/0: The system will reboot now!
Checkpoint: Observe one complete normal boot
Continue whenThe console reaches the expected login without a recovery target argument or emergency prompt.
Stop whenStorage errors recur, boot stalls beyond the approved window, or the host selects an unexpected kernel/root.
Security notes
- Reboot is an availability change; notify service owners and keep the host out of traffic until post-boot verification passes.
Alternatives
- For a clustered service, drain or fail over workloads before the proof reboot.
Stop conditions
- Do not hard-reset while an orderly reboot is still syncing storage unless the platform incident procedure authorizes it.
verification
Capture post-recovery evidence and close the incident
After the normal boot, record boot health, failed units, the critical chain, journal errors, mount validation, and whether the temporary runtime mask disappeared. Document the exact root cause and permanent correction.
Why this step matters
Closing on a clean fresh-boot evidence set prevents a temporary recovery state from being mistaken for a durable fix and leaves future responders a precise cause-and-correction record.
What to understand
critical-chain explains boot ordering but not full duration of parallel jobs; retain it with failed units and the current-boot error journal.
The runtime mask should be absent after reboot. is-enabled should show the unit's persistent policy, which must be reviewed separately if the service remains problematic.
Document configuration hashes or diffs, operator, time, incident ID, backup reference, and application verification.
System changes
- No persistent change; captures final boot evidence for the incident and change records.
Syntax explained
systemd-analyze critical-chain- Shows timing and ordering for the target's time-critical dependency chain.
journalctl --priority=err- Lists current-boot errors that still require disposition.
is-enabled- Confirms the selected unit's persistent enablement after the runtime mask disappeared.
Values stay on this page and are never sent or saved.
systemctl is-system-running; systemctl --failed --no-pager --plain; systemd-analyze critical-chain --no-pager; journalctl --boot=0 --priority=err --no-pager; findmnt --verify; systemctl is-enabled {{failingUnit}} || truerunning 0 loaded units listed. graphical.target @8.221s └─multi-user.target @8.218s Success, no errors or warnings detected enabled
Checkpoint: Accept or reopen the recovery
Continue whenFresh boot, required services, application health, and evidence meet the owner-approved acceptance criteria.
Stop whenAny causal error, unexpected mask, failed unit, or application degradation remains unexplained.
Security notes
- Store evidence in the authorized incident system and remove unredacted console captures from personal devices.
Alternatives
- Keep the host quarantined and restore a replacement when evidence cannot demonstrate trustworthy state.
Stop conditions
- Do not close the incident based only on SSH availability or a green systemctl summary.
Finish line
Verification checklist
systemctl is-system-running && systemctl get-defaultThe manager reports running and the expected normal default target, not rescue.target or emergency.target.systemctl --failed --no-pager --plainThe table contains zero loaded failed units; any intentionally tolerated exception is separately documented and monitored.findmnt --verify --verboseThe current fstab and mount graph report no parse errors, unreachable required sources, duplicate mount points, or invalid options.journalctl --boot=0 --priority=err --no-pagerNo recurring storage, filesystem, dependency, authentication, or required-service error remains after the recovered boot.Recovery guidance
Common problems and safe checks
GRUB never appears, or the console skips directly from firmware to the failed Ubuntu boot.
Likely causeThe boot menu is hidden, the console did not deliver Shift or Escape early enough, a provider uses a serial menu, or GRUB itself is not the active boot path.
Confirm the provider console type and send its documented boot-menu key before the kernel splash.Check firmware boot order and the selected virtual disk without changing it.Attempt the provider's temporary boot or rescue-media function rather than rewriting GRUB blindly.
ResolutionUse the platform console's documented method to expose GRUB or boot Ubuntu installation media. Repair the bootloader only after mounting and identifying the intended root and EFI system partition.
rescue.target also fails and the console reports a local filesystem dependency failure.
Likely causeA required fstab mount is part of rescue dependencies, its UUID is stale, its backing storage is missing, or systemd-fsck left errors uncorrected.
Boot once with systemd.unit=emergency.target.Run lsblk --fs and blkid without changing signatures.Run findmnt --verify --verbose against the installed fstab after root is readable.
ResolutionCorrect a proven stale optional mount entry or restore its device. If filesystem errors remain, leave the filesystem unmounted and use an offline repair or restore procedure.
systemctl commands report that the system was not booted with systemd or cannot connect to the bus.
Likely causeThe operator booted init=/bin/sh, entered an initramfs shell, or is chrooted without the running host's systemd bus.
Read cat /proc/1/comm and cat /proc/cmdline.Inspect findmnt / and mount flags.Determine whether the prompt belongs to initramfs, installation media, or the installed system.
ResolutionUse direct file and mount tools appropriate to that environment, or reboot into systemd.unit=emergency.target when possible. Do not interpret unavailable systemctl output as a failed unit.
multi-user.target is reached in the recovery boot, but the next normal reboot returns to emergency mode.
Likely causeThe correction was only a runtime mask, the GRUB test still carried a recovery argument, a required device appears nondeterministically, or the persistent file was not saved.
Compare /proc/cmdline between the test and failed boot.Inspect systemctl is-enabled and systemctl cat for the implicated unit.Compare lsblk UUIDs, findmnt --verify, and both boot journals by boot ID.
ResolutionMake the smallest evidence-backed persistent correction, remove recovery-only arguments, and repeat the clean boot. Escalate intermittent storage discovery as a device-path incident rather than adding sleep commands.
Reference
Frequently asked questions
Should I start with emergency.target because it is more powerful?
No. Start with rescue.target because it provides more of the ordinary local system while still avoiding the normal service target. Emergency mode is the fallback when rescue dependencies fail.
Can I run fsck -y on root from the emergency shell?
Not while root is mounted. Identify the filesystem and use installation media or an initramfs recovery context where it is unmounted. Automatic answers also discard the chance to review unexpected destructive repairs.
Why not permanently disable the unit that blocks boot?
Because the unit may provide backups, security, storage, or another required control. A runtime mask is a diagnostic experiment; permanent policy follows root-cause analysis and an explicit service-owner decision.
Recovery
Rollback
Recovery changes are rolled back from the console by restoring the saved configuration and returning to the known boot target; filesystem writes cannot be undone without a backup or snapshot.
- If an fstab edit worsens the boot, enter emergency mode again, remount root read-write only after confirming storage health, restore /etc/fstab.before-boot-recovery, and run systemctl daemon-reload.
- Remove a persistent mask with systemctl unmask UNIT. A --runtime mask lives under /run and disappears on reboot, but verify that no persistent mask was created accidentally.
- If a service configuration was changed, restore its exact pre-change file from the configuration backup, validate it with the service's native checker, then start only that service before attempting another full boot.
- If storage corruption or hardware failure was discovered, stop normal boot attempts and restore to replacement storage from a tested backup; do not treat fsck output as a substitute for a recoverable copy.
Evidence