Services & Applications / Security

Configure journald retention, forwarding, and centralized remote logs

Make local journals persistent and bounded, receive per-host journal files on a dedicated collector, and upload structured events over mutually authenticated TLS while preserving a local buffer and auditable failure behavior.

105 min10 stepsChanges system stateRevision 2

Save or explore

#journald #systemd #logging #tls #centralized-logs #ubuntu

0 of 10 steps completed

Goal

Give Ubuntu 24.04 operators useful local retention plus an independently stored central copy of structured systemd journal events, with explicit disk budgets, certificate trust, service health checks, and a tested outage-and-recovery path.

Supported environments

Ubuntu Server 24.04 LTS
systemd-journal-remote 255

Prerequisites

Dedicated collector and capacity plan Use a separately administered collector with enough protected storage for measured event volume, retention, growth, and incident bursts. The collector should not share the failure domain of every source host.df -h /var/log/journal/remote; df -i /var/log/journal/remote
Mutual-TLS PKI Provision a collector certificate whose SAN contains {{collectorHost}}, one client certificate per source identity, private keys with mode 0600, and a trusted CA chain. Certificate issuance is outside this guide.openssl verify -CAfile /etc/systemd/journal-upload/pki/ca.crt /etc/systemd/journal-upload/pki/client.crt
Restricted network path Allow TCP/{{collectorPort}} from declared source networks to the collector only. Confirm DNS and time synchronization before TLS testing; this guide does not open a firewall.getent ahosts {{collectorHost}}; timedatectl show -p NTPSynchronized --value
Data governance Define who may read centralized logs, which fields may contain identifiers or secrets, the retention obligation, and the incident/legal hold process before collection begins.

Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System

A persistent local journal with explicit retention, free-space reserve, compression, sealing, and rate policy.
A dedicated `systemd-journal-remote` collector that stores structured native journals split by source host under a separate capacity budget.
A mutually authenticated `systemd-journal-upload` path with end-to-end canary evidence and a rehearsed central-outage recovery path.

Observable outcome

A source retains searchable local boot history within its declared disk envelope and passes `journalctl --verify` after journald restart.
Only a client certificate issued by the logging CA can reach the collector, whose server identity matches {{collectorHost}}.
A unique canary written on {{sourceHost}} appears centrally with structured hostname and identifier fields, and missed delivery catches up after a bounded collector outage.

Architecture

How the parts fit together

Each source writes structured events into its local system journal, whose persistent files provide the first buffer and the fastest host-level troubleshooting view. The uploader reads those events with a saved cursor and sends journal export data over HTTPS using a per-source client certificate. The collector's socket activates `systemd-journal-remote`, validates the client chain, and writes native journal files split by host. Separate storage budgets and independent access controls prevent either tier from silently consuming all disk or becoming the only copy.

systemd-journaldCollects kernel, service, stdout/stderr, audit-adjacent, and application events with structured metadata on each source.

Local persistent journalBuffers events across network outages and keeps immediate evidence across reboot within a bounded disk budget.

systemd-journal-uploadReads the source journal from a saved cursor and uploads export-formatted entries over authenticated HTTPS.

Mutual-TLS PKIAuthenticates the collector name to clients and each approved source identity to the collector.

systemd-journal-remoteAccepts uploaded event streams and writes centrally searchable native journal files.

Remote journal storageSeparates host files, enforces collector capacity, and provides an independently administered evidence copy.

A process writes an event to journald; metadata such as unit, PID, priority, boot ID, and hostname is stored locally.
Persistent local files are compressed, sealed when keys exist, rotated, and vacuumed according to age, maximum use, and free-space constraints.
The uploader resumes at its saved cursor, establishes a TLS connection to the certificate hostname, presents its client identity, and streams entries.
The collector socket accepts only the restricted network path and activates the receiver, which validates the client certificate and separates records by host.
Operators query local files for immediate response and central files for cross-host investigation, source-host loss, or independently retained evidence.
During collector outage, local retention grows within budget; after recovery, the uploader reconnects and sends entries after the saved cursor.

Assumptions

The collector is a dedicated Ubuntu 24.04 host or equivalent reviewed systemd environment, not a source host casually designated as its own independent evidence copy.
DNS for {{collectorHost}} and system time are trustworthy enough for TLS; the certificate SAN contains that exact name.
The PKI issues separate client certificates, protects private keys, supports revocation, and does not reuse the collector private key on clients.
TCP/{{collectorPort}} is allowed only from approved source networks by upstream and host firewalls; the service is not exposed as an unauthenticated public ingestion endpoint.
Log volume was measured under normal and incident load, and local/central disk budgets include other filesystem consumers and operational reserve.
Applications may still log sensitive values; centralization does not automatically redact credentials, tokens, personal data, or customer content.

Key concepts

Persistent journal: Journal files stored under `/var/log/journal`, retained across boot instead of only under volatile `/run/log/journal`.
Structured field: Metadata such as `_SYSTEMD_UNIT`, `_PID`, `_BOOT_ID`, priority, and hostname stored alongside message text and available for exact filtering.
Journal cursor: A stable position identifier that lets the uploader continue from a known event after restart or connection loss.
Forward Secure Sealing: Cryptographic sealing that can reveal retrospective modification of finalized journal data when verification keys and operational procedures are maintained.
Mutual TLS: TLS in which the source validates the collector certificate and the collector also validates a per-source client certificate.
SplitMode=host: Collector policy that stores received journals in files separated by source host, improving attribution and limiting mixed-file failure scope.
Backpressure boundary: The local journal and its disk budget are the buffer when the remote path is unavailable; exceeding the budget can rotate old evidence.

Before you copy

Values used in this guide

{{collectorHost}}

DNS name present in the collector certificate SAN and resolved by every source.

Example: logs01.example.net

{{collectorPort}}

Restricted TCP listener used by systemd-journal-remote.

Example: 19532

{{sourceHost}}

Stable hostname field expected for the source in central queries.

Example: app01.example.net

{{localJournalMaxUse}}

Maximum local persistent-journal disk use selected from measured rate and outage objective.

Example: 2G

{{localJournalKeepFree}}

Minimum filesystem space journald should preserve for other host functions.

Example: 1G

{{localRetention}}

Maximum age for local events when size constraints have not already rotated them.

Example: 30day

{{collectorMaxUse}}

Maximum disk use allocated to all received remote journals on this collector.

Example: 20G

{{collectorKeepFree}}

Minimum collector filesystem space reserved outside remote journal storage.

Example: 10G

Security and production boundaries

Logs are high-value security data and can also contain secrets. Apply least-privilege read access, encrypted backups, retention policy, query auditing, and redaction at the application source wherever possible.
Do not infer integrity from transport encryption alone. Protect the collector administrator plane, time, package supply chain, journal files, sealing material, and backups.
A shared client certificate destroys source attribution and makes revocation disruptive. Issue one identity per host or tightly governed workload.
Rate limiting protects the host but can suppress attack evidence during a flood. Monitor suppression messages and tune from measured legitimate peaks rather than disabling limits.
Remote collection increases data jurisdiction and privacy scope. Keep only fields and duration justified by operational and legal requirements.

Stop before continuing if

Stop if the source or collector clock is unsynchronized, DNS is ambiguous, the server SAN does not match, or any certificate chain validation fails.
Do not expose the remote socket until source network restrictions and mutual-TLS client validation are confirmed.
Stop when local or collector capacity, inode reserve, event rate, or outage buffer has not been measured.
Do not proceed if private keys are group/world-readable, shared across hosts, present in the repository, or copied through an unapproved channel.
Abort rollout when canary fields or hostname attribution are wrong, delivery gaps cannot be explained, or failure rehearsal rotates required local evidence.

verification

Measure current journal volume, durability, and rate

read-only

Record whether the source currently uses volatile or persistent storage, how much disk the journal consumes, the oldest and newest available entries, and an event-rate sample. Size limits from evidence rather than copying arbitrary numbers.

Why this step matters

Retention and outage buffering are capacity decisions. Measuring real boots, bytes, event rate, and effective defaults prevents a copied limit from either wasting storage or silently rotating incident evidence.

What to understand

Capture normal and known peak windows. Fifteen minutes is an initial sample, not a final capacity model; multiply measured rate by the required local outage duration and include growth.

`cat-config` merges vendor files and drop-ins in precedence order, exposing the policy journald will actually read rather than one file in isolation.

Oldest boot history reveals whether current storage is persistent; disk usage alone cannot prove that events survive reboot.

System changes

No persistent change; queries current journals and merged configuration.

Syntax explained

journalctl --disk-usage: Reports active and archived journal allocation.
--list-boots: Lists boot IDs and time ranges available in the current journal store.
systemd-analyze cat-config: Displays the merged vendor and administrator configuration in precedence order.

Command

journalctl --disk-usage && journalctl --list-boots --no-pager | tail -n 5 && journalctl --since '-15 minutes' --no-pager | wc -l && systemd-analyze cat-config systemd/journald.conf

Example output / evidence

Archived and active journals take up 612.0M in the file system.
 -2 2e4b0d7d2f7b4b1592a80df0c030d895 Mon 2026-07-27 08:11:03 UTC—Mon 2026-07-27 16:02:42 UTC
 -1 0a10ce9d97ab4387b4c2c8b36257bbfd Mon 2026-07-27 16:03:11 UTC—Tue 2026-07-28 06:13:22 UTC
  0 6d9bb2da8c04494db19dfb9c5e2aa70d Tue 2026-07-28 06:14:01 UTC—Tue 2026-07-28 08:10:17 UTC
1842
# /etc/systemd/journald.conf
[Journal]

Checkpoint: Approve retention inputs

Continue whenNormal and peak rate, current bytes, oldest evidence, filesystem free space, and outage objective are recorded.

Stop whenCapacity or retention ownership is unknown, or current logs already threaten the filesystem.

If this step fails

Only the current boot is listed.

Likely causeStorage is volatile, `/var/log/journal` is absent, or previous files were vacuumed/corrupt.

Safe checks

test -d /var/log/journal && sudo find /var/log/journal -maxdepth 2 -type f -ls
systemd-analyze cat-config systemd/journald.conf

ResolutionTreat historical evidence as unavailable, establish persistent storage, and verify it across a controlled reboot.

Security notes

Do not paste raw logs into an external sizing ticket; record aggregate counts and redact fields.

Alternatives

Use protected telemetry to estimate bytes and events over a longer representative period.

Stop conditions

No limits are selected without a measured rate and filesystem reserve.

config

Write a bounded persistent local journal policy

caution

Use a local drop-in rather than modifying vendor configuration. Persistent storage retains evidence across reboot, while maximum use, reserved free space, retention age, compression, and rate limits make the disk impact explicit.

Why this step matters

A local drop-in creates a reviewable durability and capacity boundary while leaving distribution defaults maintainable and preserving enough local evidence when the collector is unavailable.

What to understand

`Storage=persistent` writes under `/var/log/journal`. `SystemMaxUse` and `SystemKeepFree` work together, so the smaller effective constraint wins.

Age retention is a maximum, not a guarantee; size pressure may rotate older files sooner. Rate limits protect resources but can suppress noisy events.

Forward Secure Sealing needs keys and verification operations to add value; enabling the setting alone is not an evidence program.

System changes

Creates a journald drop-in that changes storage, compression, sealing, rotation, free-space, age, and rate policy after restart.

Syntax explained

Storage=persistent: Uses `/var/log/journal` so events survive reboot.
SystemMaxUse: Caps journal use on the persistent filesystem.
SystemKeepFree: Preserves a minimum amount of filesystem free space.
MaxRetentionSec: Removes archived journal files older than the declared maximum when rotation runs.
RateLimitIntervalSec/RateLimitBurst: Limits accepted messages per service inside a time window and reports suppression.

File /etc/systemd/journald.conf.d/60-oneliners-retention.conf

Configuration

Fill variables0/3 ready

Values stay on this page and are never sent or saved.

Local Journal Max Use{{localJournalMaxUse}}Maximum local persistent-journal disk use selected from measured rate and outage objective.Local Journal Keep Free{{localJournalKeepFree}}Minimum filesystem space journald should preserve for other host functions.Local Retention{{localRetention}}Maximum age for local events when size constraints have not already rotated them.

[Journal]
Storage=persistent
Compress=yes
Seal=yes
SystemMaxUse={{localJournalMaxUse}}
SystemKeepFree={{localJournalKeepFree}}
MaxRetentionSec={{localRetention}}
RateLimitIntervalSec=30s
RateLimitBurst=10000

Example output / evidence

[Journal]
Storage=persistent
Compress=yes
Seal=yes
SystemMaxUse=2G
SystemKeepFree=1G
MaxRetentionSec=30day
RateLimitIntervalSec=30s
RateLimitBurst=10000

Checkpoint: Review the effective drop-in

systemd-analyze cat-config systemd/journald.conf

Continue whenThe OneLiners file appears last with the measured limits and no conflicting later override.

Stop whenA later file overrides the policy or the configured budget exceeds the filesystem plan.

If this step fails

The effective output shows a different maximum.

Likely causeAnother higher-precedence drop-in or kernel option overrides it.

Safe checks

grep -R --line-number 'SystemMaxUse\|Storage=' /etc/systemd/journald.conf /etc/systemd/journald.conf.d /run/systemd/journald.conf.d 2>/dev/null

ResolutionChoose one authoritative administrator drop-in, remove conflicting unmanaged policy through change control, and recheck the merge.

Security notes

Mode the drop-in root-owned and do not use logs as a secret store.

Alternatives

Place `/var/log/journal` on a dedicated filesystem when host workload and evidence need stronger storage isolation.

Stop conditions

Do not restart journald with contradictory or unbounded capacity policy.

command

Validate and apply the local journal policy

caution

Inspect the merged configuration, create the persistent journal directory with systemd's supported mechanism, restart journald during a maintenance window, flush early-boot events to disk, and verify journal files before continuing.

Why this step matters

Applying retention changes deliberately and verifying journal files catches permission, storage, sealing, and corruption errors before remote transport adds another failure layer.

What to understand

`journalctl --setup-keys` creates sealing material where supported; protect verification material separately according to the evidence policy.

Restart may briefly affect event ingestion, so use a maintenance window and check service status immediately.

`--flush` moves volatile early-boot entries to persistent storage after it becomes available; `--verify` checks file consistency.

System changes

Restarts journald, creates persistent journal storage/sealing state if absent, and flushes volatile entries to disk.

Syntax explained

--setup-keys: Initializes forward-secure sealing keys for the local journal.
systemctl restart systemd-journald: Applies the merged configuration to the logging daemon.
journalctl --flush: Requests movement of runtime journal data into persistent storage.
journalctl --verify: Checks internal consistency and sealing where verification information is available.

Command

systemd-analyze cat-config systemd/journald.conf | sed -n '/60-oneliners-retention.conf/,$p'; sudo journalctl --setup-keys; sudo systemctl restart systemd-journald; sudo journalctl --flush; sudo journalctl --verify; journalctl --disk-usage

Example output / evidence

# /etc/systemd/journald.conf.d/60-oneliners-retention.conf
[Journal]
Storage=persistent
Compress=yes
Seal=yes
SystemMaxUse=2G
SystemKeepFree=1G
MaxRetentionSec=30day
PASS: /var/log/journal/4a65b1c7d1dc4d39b25bc4ab697a76a0/system.journal
Archived and active journals take up 640.0M in the file system.

Checkpoint: Accept local durability

Continue whenjournald is active, verification passes, a previous boot remains queryable after a controlled reboot, and usage stays within budget.

Stop whenThe service fails, journal verification fails, or filesystem reserve falls below policy.

If this step fails

journald restarts but `/var/log/journal` remains empty.

Likely causeStorage policy is not effective, directory permissions are wrong, filesystem is read-only, or flush failed.

Safe checks

systemctl status systemd-journald --no-pager
sudo journalctl -u systemd-journald -b --no-pager
sudo namei -l /var/log/journal

ResolutionCorrect the effective policy and systemd-managed permissions, then flush and verify again; do not create arbitrary modes manually.

Security notes

Sealing verification keys and journals must not share every failure domain if tamper evidence matters.

Alternatives

Use distribution-managed tmpfiles to establish journal directories instead of manual chmod.

Stop conditions

Do not configure upload while local ingestion or integrity is unhealthy.

command

Install the upstream journal remote components

caution

Install the Ubuntu package on the collector and each source, then inspect the packaged service, socket, and upload units before enabling them. Distribution unit files define the listening and output behavior that the local drop-ins refine.

Why this step matters

Using the signed Ubuntu package provides matched receiver, uploader, units, users, and dependencies, while inspecting packaged units prevents assumptions about ports and activation behavior.

What to understand

Install on collector and sources from approved Ubuntu repositories; record package origin and version for compatibility review.

The socket, service, and uploader have distinct roles. Do not enable every unit blindly: collector uses receiver socket, source uses uploader service.

System changes

Installs systemd journal remote binaries, units, service identities, and configuration defaults.

Syntax explained

apt-get update: Refreshes signed repository indexes for scripted package installation.
apt-get install --yes: Installs the reviewed package noninteractively.
systemctl cat: Shows complete packaged unit definitions and drop-ins.

Command

sudo apt-get update && sudo apt-get install --yes systemd-journal-remote && dpkg-query -W -f='${Package} ${Version}\n' systemd-journal-remote && systemctl cat systemd-journal-remote.socket systemd-journal-remote.service systemd-journal-upload.service

Example output / evidence

systemd-journal-remote 255.4-1ubuntu8.11
# /usr/lib/systemd/system/systemd-journal-remote.socket
[Socket]
ListenStream=19532
# /usr/lib/systemd/system/systemd-journal-upload.service
[Service]
ExecStart=/usr/lib/systemd/systemd-journal-upload --save-state

Checkpoint: Record package and roles

Continue whenThe expected Ubuntu package is installed; collector and source unit roles, executable paths, port, users, and output directory are understood.

Stop whenPackage origin is unexpected or existing remote logging units are active and unowned.

If this step fails

APT cannot locate systemd-journal-remote.

Likely causeRepository indexes are stale, the enabled Ubuntu components differ, or release support is incorrect.

Safe checks

apt-cache policy systemd-journal-remote
grep -R '^Components:' /etc/apt/sources.list.d/*.sources

ResolutionRestore supported signed Ubuntu sources and package indexes; do not download an unreviewed binary.

Security notes

Do not start network listeners during package installation without inspecting service state and firewall exposure.

Alternatives

Use the supported package/version for the target distribution and adapt unit paths with its official documentation.

Stop conditions

No rollout with mixed, unsupported systemd journal protocol implementations.

config

Configure per-host remote journal storage on the collector

caution

Keep received journals split by source host, set a collector-specific disk ceiling and reserve, and point the listener at the reviewed TLS identity. The collector private key must never be copied to source hosts.

Why this step matters

Collector policy separates host evidence, enforces an independent disk envelope, and binds TLS to protected files before a network socket accepts uploaded data.

What to understand

`SplitMode=host` preserves host attribution in file layout. It does not authorize hosts; client certificate validation and network policy do that.

`MaxUse` and `KeepFree` protect the collector filesystem; monitor both because retention can shorten during an incident burst.

Only the collector service identity needs key access, and the private key must remain 0600 or an equivalently restrictive service credential.

System changes

Creates a journal-remote drop-in defining TLS identity, trust, split mode, and collector retention limits.

Syntax explained

SplitMode=host: Writes received entries to files separated by originating host.
ServerKeyFile/ServerCertificateFile: Provides the collector TLS private key and certificate chain.
TrustedCertificateFile: Provides the CA used to validate uploading client certificates.
MaxUse/KeepFree: Bounds total remote journal storage and preserves filesystem reserve.
MaxFileSize/MaxFiles: Controls rotation granularity and archived file count.

File /etc/systemd/journal-remote.conf.d/60-oneliners-collector.conf

Configuration

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

Collector Max Use{{collectorMaxUse}}Maximum disk use allocated to all received remote journals on this collector.Collector Keep Free{{collectorKeepFree}}Minimum collector filesystem space reserved outside remote journal storage.

[Remote]
Seal=yes
SplitMode=host
ServerKeyFile=/etc/systemd/journal-remote/pki/collector.key
ServerCertificateFile=/etc/systemd/journal-remote/pki/collector.crt
TrustedCertificateFile=/etc/systemd/journal-remote/pki/ca.crt
MaxUse={{collectorMaxUse}}
KeepFree={{collectorKeepFree}}
MaxFileSize=256M
MaxFiles=200

Example output / evidence

[Remote]
Seal=yes
SplitMode=host
ServerKeyFile=/etc/systemd/journal-remote/pki/collector.key
ServerCertificateFile=/etc/systemd/journal-remote/pki/collector.crt
TrustedCertificateFile=/etc/systemd/journal-remote/pki/ca.crt
MaxUse=20G
KeepFree=10G
MaxFileSize=256M
MaxFiles=200

Checkpoint: Validate collector files and budget

sudo systemd-analyze cat-config systemd/journal-remote.conf && sudo namei -l /etc/systemd/journal-remote/pki/collector.key

Continue whenThe merged configuration references the intended PKI files and measured budgets; key access is restricted.

Stop whenKey permissions, CA trust, hostname certificate, or disk reserve is wrong.

If this step fails

The receiver reports permission denied reading its key.

Likely causeOwnership/mode does not match the packaged service identity or a sandbox path restriction applies.

Safe checks

systemctl show systemd-journal-remote.service -p User -p Group -p ReadOnlyPaths -p InaccessiblePaths
sudo namei -l /etc/systemd/journal-remote/pki/collector.key

ResolutionGrant the smallest read access to the packaged service identity through approved ownership or systemd credentials; never make the key world-readable.

Security notes

Keep collector administration and log-reading roles separate where possible.

Alternatives

Terminate TLS at a reviewed dedicated proxy only if end-to-end client identity and request-size controls remain explicit.

Stop conditions

Do not start a receiver with missing client-certificate validation.

command

Start the collector socket and prove the TLS listener

caution

Validate certificate ownership and dates, enable the socket-activated listener, and inspect both the bound port and service journal. Do not expose the socket until its network allowlist is already in force.

Why this step matters

Certificate inspection and socket verification prove what identity and port will be exposed before sources trust the endpoint.

What to understand

Check SAN, issuer, validity, and private-key path. The common name alone is not enough for hostname verification.

Socket activation may show systemd PID 1 as listener until a connection starts the service; that is expected and should be documented.

Confirm upstream and host policy restrict the port before enabling it.

System changes

Enables and starts the journal-remote socket, making TCP/{{collectorPort}} available to permitted networks.

Syntax explained

openssl x509 -noout: Prints certificate identity and validity without exposing private key material.
systemctl enable --now ...socket: Persists and immediately activates socket-based listening.
ss -lntp: Proves the numeric listening port and owning process.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Collector Port{{collectorPort}}Restricted TCP listener used by systemd-journal-remote.

sudo namei -l /etc/systemd/journal-remote/pki/collector.key; openssl x509 -in /etc/systemd/journal-remote/pki/collector.crt -noout -subject -issuer -dates -ext subjectAltName; sudo systemctl enable --now systemd-journal-remote.socket; systemctl is-active systemd-journal-remote.socket; sudo ss -lntp 'sport = :{{collectorPort}}'

Example output / evidence

f: /etc/systemd/journal-remote/pki/collector.key
-rw------- root systemd-journal-remote collector.key
subject=CN = logs01.example.net
issuer=CN = OneLiners Logging CA
notBefore=Jul 20 00:00:00 2026 GMT
notAfter=Jul 20 00:00:00 2027 GMT
X509v3 Subject Alternative Name:
    DNS:logs01.example.net
active
LISTEN 0 4096 0.0.0.0:19532 0.0.0.0:* users:(("systemd",pid=1,fd=57))

Checkpoint: Approve restricted TLS listener

Continue whenCertificate SAN matches {{collectorHost}}, validity is current, socket is active on {{collectorPort}}, and only approved sources can route to it.

Stop whenThe listener is public, certificate identity is wrong, or another process owns the port.

If this step fails

The socket fails with address already in use.

Likely causeAnother collector, proxy, or unrelated service binds the port.

Safe checks

sudo ss -lntp 'sport = :19532'
systemctl status systemd-journal-remote.socket --no-pager

ResolutionIdentify the existing owner and choose one documented architecture; do not kill an unknown production listener.

Security notes

A bound port is not proof that client authentication works; the next step must test mutual TLS.

Alternatives

Use a dedicated load balancer only with explicit client-certificate pass-through/validation and tested journal semantics.

Stop conditions

Stop if any unapproved network can reach the listener.

config

Configure a source host to upload over mutual TLS

caution

Point the source at the certificate name, install only that source's client identity, and trust the dedicated logging CA. The uploader tracks its cursor so reconnects continue from saved state rather than intentionally rereading the entire journal.

Why this step matters

A source-specific uploader configuration binds delivery to the certificate hostname and dedicated trust root while preserving separate private identities for attribution and revocation.

What to understand

Use the DNS name from the collector SAN, not an opportunistic IP address. Ensure each source receives only its own key and certificate.

The uploader saves cursor state through its packaged service behavior; deleting state can create gaps or duplicates and is not a routine fix.

Keep local retention even after upload works because network, collector, and certificate outages are expected failure modes.

System changes

Creates the upload endpoint and client TLS policy; no connection occurs until the service starts.

Syntax explained

URL=https://host:port: Selects encrypted journal upload to the certificate hostname.
ServerKeyFile: Provides this source's private client key.
ServerCertificateFile: Provides this source's client certificate.
TrustedCertificateFile: Provides the CA used to verify the collector.

File /etc/systemd/journal-upload.conf.d/60-oneliners-upload.conf

Configuration

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

Collector Host{{collectorHost}}DNS name present in the collector certificate SAN and resolved by every source.Collector Port{{collectorPort}}Restricted TCP listener used by systemd-journal-remote.

[Upload]
URL=https://{{collectorHost}}:{{collectorPort}}
ServerKeyFile=/etc/systemd/journal-upload/pki/client.key
ServerCertificateFile=/etc/systemd/journal-upload/pki/client.crt
TrustedCertificateFile=/etc/systemd/journal-upload/pki/ca.crt

Example output / evidence

[Upload]
URL=https://logs01.example.net:19532
ServerKeyFile=/etc/systemd/journal-upload/pki/client.key
ServerCertificateFile=/etc/systemd/journal-upload/pki/client.crt
TrustedCertificateFile=/etc/systemd/journal-upload/pki/ca.crt

Checkpoint: Review source identity

openssl x509 -in /etc/systemd/journal-upload/pki/client.crt -noout -subject -issuer -dates; sudo namei -l /etc/systemd/journal-upload/pki/client.key

Continue whenThe certificate identifies only this source, chains to the logging CA, is current, and its key is restricted.

Stop whenIdentity is shared, expired, unreadable by the service, or exposed to unrelated users.

If this step fails

The client key and certificate do not match.

Likely causeProvisioning copied files from different issuance requests.

Safe checks

openssl x509 -noout -modulus -in /etc/systemd/journal-upload/pki/client.crt | openssl sha256
sudo openssl rsa -noout -modulus -in /etc/systemd/journal-upload/pki/client.key | openssl sha256

ResolutionReprovision the correct pair through PKI inventory and revoke any uncertain identity.

Security notes

Never log, print, or copy private key contents during verification.

Alternatives

Use systemd credential delivery when the platform supports centrally protected service credentials.

Stop conditions

No service start with a mismatched or shared client identity.

verification

Verify name, chain, client identity, and time before upload

read-only

Perform an explicit mutual-TLS handshake from the source using the same key, certificate, CA, DNS name, and port that the uploader will use. A TCP connection without certificate verification is not sufficient evidence.

Why this step matters

An explicit handshake isolates DNS, routing, time, server name, CA chain, client identity, and key access before uploader state and journal protocol complicate diagnosis.

What to understand

`-servername` sends SNI for the same hostname that should appear in SAN; `-verify_return_error` makes trust failure fatal.

The client certificate and key exercise mutual authentication. A plain curl or TCP probe can succeed even when the collector will reject the source.

Capture only subject, issuer, and verification status, not session secrets or certificate bodies.

System changes

No persistent change; establishes and closes a test TLS session.

Syntax explained

-connect host:port: Targets the exact collector socket.
-servername: Sends the DNS service name for SNI and name-based selection.
-CAfile: Restricts server trust to the reviewed logging CA.
-cert/-key: Presents the source's mutual-TLS identity.
-verify_return_error: Stops on certificate verification errors instead of continuing diagnostically.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

openssl s_client -connect {{collectorHost}}:{{collectorPort}} -servername {{collectorHost}} -CAfile /etc/systemd/journal-upload/pki/ca.crt -cert /etc/systemd/journal-upload/pki/client.crt -key /etc/systemd/journal-upload/pki/client.key -verify_return_error </dev/null 2>/dev/null | grep -E 'subject=|issuer=|Verification'

Example output / evidence

subject=CN = logs01.example.net
issuer=CN = OneLiners Logging CA
Verification: OK

Checkpoint: Require mutual-TLS verification

Continue whenThe collector subject is {{collectorHost}}, issuer is expected, and Verification is OK using the source client identity.

Stop whenAny hostname, chain, validity, key, protocol, or time error appears.

If this step fails

Connection is refused before TLS output.

Likely causeSocket is inactive, firewall blocks/rejects, DNS resolves elsewhere, or route/port is wrong.

Safe checks

getent ahosts logs01.example.net
nc -vz logs01.example.net 19532
systemctl status systemd-journal-remote.socket --no-pager

ResolutionCorrect listener or the smallest network rule; keep certificate verification unchanged.

Security notes

Do not use `-verify 0`, `-k`, or an empty CA to bypass trust.

Alternatives

Use a PKI-approved TLS diagnostic tool that verifies the same SNI, CA, and client identity.

Stop conditions

A partial handshake is a failed checkpoint.

verification

Start upload and prove an end-to-end canary

read-only

Enable the uploader, emit a unique non-secret canary from the source, and query the collector's remote directory for the exact host and message. Retain local and remote timestamps to measure delivery delay.

Why this step matters

A unique canary proves service enablement, cursor reading, TLS transport, collector acceptance, structured field preservation, and central query—not merely port reachability.

What to understand

The canary contains no secret and includes a unique timestamp token. Emit it only after uploader service is active.

Run the central query on the collector using `--directory`, `_HOSTNAME`, and `SYSLOG_IDENTIFIER`; message grep alone can match unrelated data.

Record delivery latency and both unit journals. Repeated delay is an operational symptom even when eventual delivery succeeds.

System changes

Enables continuous upload and writes one harmless canary to local and central journals.

Syntax explained

enable --now: Starts upload now and on future boots.
logger --tag: Writes a uniquely identifiable journal message through syslog compatibility.
journalctl --directory: Queries journal files stored outside the collector's local machine journal.

Command

sudo systemctl enable --now systemd-journal-upload.service && logger --tag oneliners-log-canary 'central-journal-canary-20260728T083000Z' && sleep 3 && systemctl is-active systemd-journal-upload.service

Example output / evidence

active
Jul 28 08:30:00 app01 oneliners-log-canary[2841]: central-journal-canary-20260728T083000Z

Checkpoint: Prove source-to-collector delivery

sudo journalctl --directory=/var/log/journal/remote _HOSTNAME={{sourceHost}} SYSLOG_IDENTIFIER=oneliners-log-canary --since '-10 minutes' --no-pager

Continue whenExactly the unique canary appears with correct source identity within the accepted delivery interval.

Stop whenMessage, hostname, identifier, timestamp, or host separation is wrong.

If this step fails

Uploader is active but the canary never arrives.

Likely causeThe service is retrying, cursor state points after the canary, collector rejected the stream, or central query targets the wrong directory/host field.

Safe checks

journalctl -u systemd-journal-upload.service --since '-10 minutes' --no-pager
journalctl SYSLOG_IDENTIFIER=oneliners-log-canary --since '-10 minutes' --no-pager
journalctl -u systemd-journal-remote.service --since '-10 minutes' --no-pager

ResolutionResolve the exact transport, cursor, receiver, or query error and emit a new canary; do not erase state casually.

Security notes

Use synthetic canaries; never include credentials or customer data.

Alternatives

Use a structured application health event with a stable identifier if logger compatibility is prohibited.

Stop conditions

No additional source onboarding until a central canary is attributable and queryable.

decision

Rehearse collector outage and source recovery

caution

In a maintenance test, block only the source-to-collector path or stop the collector socket, generate numbered canaries, restore the path, and prove the uploader reconnects and the collector receives the backlog while local retention remains healthy.

Why this step matters

Central logging is reliable only if operators know how local buffering, cursor recovery, capacity, and alerting behave when the collector is unavailable.

What to understand

Stop the collector or block only the tested path, never the entire source network. Number canaries before, during, and after the outage.

Watch local disk growth and uploader retry frequency; confirm ordinary host services remain healthy.

After restoration, query all numbered canaries centrally in order and compare timestamps. Gaps require explanation before acceptance.

System changes

Temporarily interrupts the test delivery path and creates a small known backlog of synthetic events.

Syntax explained

journalctl -u systemd-journal-upload: Shows connection failure, retry, and reconnection evidence.
journalctl --disk-usage: Measures local buffer growth during outage.

Command

journalctl -u systemd-journal-upload.service --since '-15 minutes' --no-pager; journalctl --disk-usage; systemctl is-active systemd-journal-upload.service

Example output / evidence

Jul 28 08:42:11 app01 systemd-journal-upload[3011]: Upload to https://logs01.example.net:19532 failed: Connection refused
Jul 28 08:44:06 app01 systemd-journal-upload[3011]: Connected to https://logs01.example.net:19532
Archived and active journals take up 648.0M in the file system.
active

Checkpoint: Accept outage recovery

Continue whenLocal usage stays within budget, uploader reconnects automatically, and every numbered canary appears centrally without misattribution.

Stop whenRequired events rotate, source disk reserve is threatened, reconnect loops, or central gaps remain.

If this step fails

Only post-recovery canaries appear centrally.

Likely causeUploader cursor/state was reset, local retention rotated backlog, or service started with a current-only position.

Safe checks

journalctl --list-boots --no-pager
journalctl -u systemd-journal-upload.service --since '-1 hour' --no-pager
sudo ls -la /var/lib/systemd/journal-upload

ResolutionPreserve state and journals, determine cursor semantics, revise local buffer sizing, and repeat in a clean canary environment before production reliance.

Security notes

Do not create a real incident flood merely to test buffering.

Alternatives

Use a network namespace or staging collector to simulate refusal and timeout safely.

Stop conditions

Abort if the drill jeopardizes host disk or production alerting.

Finish line

Verification checklist

Verify local journal integrity and budgetsudo journalctl --verify && journalctl --disk-usageEvery local journal file passes verification and usage remains below {{localJournalMaxUse}} while at least {{localJournalKeepFree}} remains reserved.

Verify source uploader health

systemctl is-active systemd-journal-upload.service && journalctl -u systemd-journal-upload.service --since '-30 minutes' --no-pager

The unit is active with a successful recent connection and no repeated TLS, cursor, or HTTP errors.

Verify central evidence

sudo journalctl --directory=/var/log/journal/remote --since '-10 minutes' _HOSTNAME={{sourceHost}} SYSLOG_IDENTIFIER=oneliners-log-canary --no-pager

The collector returns the exact canary with hostname {{sourceHost}}, expected timestamp, identifier, and message.

Recovery guidance

Common problems and safe checks

The uploader repeatedly reports certificate verification failure.

Likely causeThe source connects by an address absent from SAN, trusts the wrong CA, has an incomplete chain, uses an expired certificate, or has incorrect time.

Safe checks

timedatectl status
getent ahosts logs01.example.net
openssl x509 -in /etc/systemd/journal-upload/pki/ca.crt -noout -subject -fingerprint -sha256
openssl s_client -connect logs01.example.net:19532 -servername logs01.example.net -verify_return_error </dev/null

ResolutionCorrect DNS/time or reissue the certificate with the exact service name and chain. Never set an insecure verification mode to make upload connect.

The TLS handshake works, but no remote journal file appears.

Likely causeThe socket did not activate the receiver, collector paths or permissions are wrong, client authorization fails, or the uploader has no readable cursor range.

Safe checks

systemctl status systemd-journal-remote.socket systemd-journal-remote.service --no-pager
journalctl -u systemd-journal-remote.service --since '-15 minutes' --no-pager
sudo find /var/log/journal/remote -maxdepth 2 -type f -ls

ResolutionFix the exact service or path error, emit a fresh unique canary, and query by its fields. Do not delete uploader state until its delivery semantics are understood.

The source filesystem fills during a collector outage.

Likely causeEvent volume exceeds the measured buffer, another consumer uses the same filesystem, or retention/free-space settings are not in the effective configuration.

Safe checks

journalctl --disk-usage
systemd-analyze cat-config systemd/journald.conf
df -h /var/log/journal
journalctl -u systemd-journald --since '-1 hour' --no-pager

ResolutionPreserve critical evidence, restore the remote path, reduce noisy logging at its source, and resize the explicit budget or isolate `/var/log` based on a revised outage objective.

Central queries show the collector hostname instead of the source identity.

Likely causeEvents were transformed through an incorrect path, the source hostname is unstable, or the query selects transport metadata rather than `_HOSTNAME`.

Safe checks

hostnamectl
journalctl -o verbose -n 1
sudo journalctl --directory=/var/log/journal/remote -o verbose -n 5

ResolutionStabilize source identity, confirm native journal upload, and document the authoritative field before onboarding more hosts.

After the procedure

Alternatives and next steps

Consider these alternatives

Use a maintained syslog daemon with RELP/TLS when downstream tooling, queues, or established policy requires RFC-style messages rather than native journal export.
Use a vendor-supported log agent when records need parsing, redaction, enrichment, durable disk queues, or delivery to object storage/SIEM; retain local journal health independently.
Use pull-based collection over a protected management network for constrained estates, understanding that source loss before pull can remove evidence.
Use immutable object storage or a security analytics platform as an additional retention tier; the journal collector is not by itself a compliance archive.

Operate it safely

Create alerts for uploader disconnect duration, collector service failure, certificate expiry, local/remote disk and inode pressure, journal corruption, and rate-limit suppression.
Back up and restore a representative remote journal, then prove `journalctl --directory` can query the restored files without changing evidence.
Automate certificate issuance, rotation, and revocation with per-source inventory and expiry lead time.
Define tiered retention, legal hold, redaction, and access review, then periodically sample logs for accidental secrets.
Load-test a nonproduction collector with realistic peak events and network interruption before expanding the fleet.

Reference

Frequently asked questions

Is `ForwardToSyslog=yes` required?

No. `systemd-journal-upload` reads the journal as a client and sends native export data. ForwardToSyslog is for a local syslog socket consumer and does not by itself provide central delivery.

Does `SystemMaxUse` guarantee that exact usage?

It is a ceiling considered with free-space constraints and rotation behavior, not a reservation. Active files and filesystem allocation can make observed usage differ slightly.

Can central logs replace local retention?

Not safely. Network and collector outages occur, and immediate host diagnosis may need local history. Keep a measured local buffer and monitor both tiers.

Why use mutual TLS on an internal network?

Network location is not reliable host identity. Mutual TLS authenticates both endpoints and supports per-source revocation, while encryption protects sensitive events in transit.

Recovery

Rollback

Rollback disables network upload and the collector listener first, preserves already received evidence, and then removes only the OneLiners drop-ins. It does not delete remote journals, revoke certificates automatically, or erase events already copied to backups.

On each source, stop and disable `systemd-journal-upload.service`; preserve its journal, cursor state, and the last successful central canary for incident review.
On the collector, stop and disable `systemd-journal-remote.socket` and service after confirming no other approved sources depend on them; retain `/var/log/journal/remote` read-only under the data-retention policy.
Remove or restore the three `60-oneliners-*.conf` drop-ins, restart journald only in a maintenance window, and verify local journal integrity plus capacity.
Revoke exposed client or collector certificates at the issuing PKI, remove obsolete private keys securely, and close the source-to-collector firewall rule if central logging is retired.

Evidence

Sources and review

Verified 2026-07-24Review due 2027-01-20

systemd journald configurationofficial systemd journal remote serviceofficial systemd journal upload serviceofficial systemd journal remote configurationofficial