Build centralized logging with Grafana Loki and Grafana Alloy
Deploy Loki 3.7 with Alloy 1.18, authenticated tenant isolation, bounded labels, pre-export redaction, object-store durability, compactor retention and an isolated recovery drill.
Operate a centralized log platform that remains queryable under load, minimizes sensitive data, prevents cross-tenant access and can prove retention and recovery rather than merely claiming them.
- Grafana Loki 3.7.3 or newer reviewed 3.7 patch
- Grafana Alloy 1.18
- Linux systemd-based supported distribution
- Object storage S3-compatible API with versioning and TLS
- Governance Approved log inventory, tenant ownership, sensitivity, retention, deletion/legal-hold and incident procedures.
- Capacity Peak ingestion, active streams, label values, retention, query concurrency, object requests and recovery bandwidth measured from representative load.
- Identity and TLS Private DNS, trusted certificates, workload identity and a gateway that derives tenant authorization server-side.
- Storage Dedicated encrypted versioned object storage with audit logs, separate component identities and a separate recovery bucket.
- Host access Dedicated unprivileged Alloy service account with only approved file/journal read access and persistent protected state.
- Recovery Documented RPO/RTO, independent configuration repository, versioned object inventory and a network-isolated restore environment.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A production-oriented Grafana Loki 3.7 deployment using TSDB schema v13 and dedicated S3-compatible object storage, with authenticated tenant isolation, compactor-managed retention, bounded queries and explicit disaster-recovery evidence.
- Grafana Alloy 1.18 collectors that read approved files and systemd journals as an unprivileged service, preserve positions, redact sensitive fields before export, use stable low-cardinality labels and send through an authenticated gateway.
- Operational controls for cardinality, queue pressure, rejected entries, query fairness, storage lifecycle, deletion, privacy, backup and an isolated restore drill with measured RPO and RTO.
- A synthetic canary line travels from host to Alloy, authenticated gateway, Loki object storage and an authorized LogQL query, while a second tenant cannot read it.
- Labels remain bounded and meaningful, volatile values stay in structured metadata, sensitive fixtures are redacted, and ingestion/query/object-store health is observable.
- Retention removes eligible data only through the compactor, object-store lifecycle cannot delete data early, and a separate recovery environment can query restored canary windows.
Architecture
How the parts fit together
Each host runs Grafana Alloy 1.18 with persistent positions and the narrow permissions needed for selected files or journals. Processing adds stable resource labels, removes or hashes approved sensitive fields, retains useful high-cardinality attributes as structured metadata, and sends batches through a TLS-authenticated gateway. The gateway derives Loki tenant identity from the authenticated workload instead of trusting a client-provided tenant header. Loki 3.7 writes TSDB index and chunks to a dedicated, encrypted, versioned S3-compatible bucket. The compactor owns retention and deletion; bucket lifecycle is deliberately longer than Loki retention. Query frontend, scheduler, queriers and caches are sized from measured workload and protected by per-tenant limits. Configuration, rule files, tenant policy and object manifests are backed up separately, while recovery is rehearsed in a separate bucket and network boundary before any production cutover.
- Classify log sources, owners, sensitivity, retention and tenant boundary; define a bounded label contract before installing collectors.
- Alloy reads only approved files/journals, preserves positions, redacts sensitive fields, adds stable labels and forwards over authenticated TLS.
- The gateway maps workload identity to tenant and sends the request to Loki; callers cannot choose another tenant by editing a header.
- Loki validates limits, stores TSDB index and chunks in object storage, and exposes metrics for accepted, rejected and delayed data.
- Authorized users query selective streams through Grafana or logcli; query fairness, range and concurrency controls protect other tenants.
- Compactor retention and audited deletion requests remove data according to policy; object lifecycle remains a safety backstop only after the application window.
- Versioned objects, configurations and manifests are restored into an isolated environment, canary ranges are queried, and RPO/RTO evidence is recorded.
Assumptions
- Grafana Loki 3.7.3 or a later reviewed 3.7 patch is the current supported line for this guide. Promtail is not deployed: it was removed in Loki 3.7.3 after deprecation, and Grafana Alloy is the supported collector path.
- Grafana Alloy 1.18 is installed from an official signed or digest-verified artifact. The exact patch is pinned, tested and upgraded only after reading release and migration notes.
- The reference environment uses Linux hosts, systemd journals and regular files. Kubernetes discovery can use the same principles but requires a separately reviewed DaemonSet, RBAC and pod-security design.
- An S3-compatible object store supports TLS, encryption, versioning, audit logs and distinct read/write/delete permissions. Filesystem storage is not treated as a production high-availability backend.
- A reverse proxy or gateway supplies authentication because open-source Loki does not provide a complete authentication layer. Tenant IDs come from server-side identity mapping.
- The organization has a data inventory, retention schedule, deletion process, legal-hold path and incident process for credentials or personal data accidentally logged.
- Stable service, environment, region, cluster and host dimensions are known. Request IDs, trace IDs, user IDs, client IPs, full paths and ephemeral container identifiers are not indexed labels.
- Object storage, gateway and Loki networks are private or tightly restricted, time is synchronized, and every service uses a dedicated least-privilege identity.
- Capacity estimates include peak bytes per second, burst size, active streams, label values, retention, replication, query concurrency, object request rate and recovery throughput.
- Backups preserve configuration, rules, tenant policy, object inventory and versioned objects; restore testing uses a separate bucket, tenant and network so compactor deletion cannot affect production.
Key concepts
- Stream label
- An indexed key/value pair that defines a Loki stream. Labels should be stable and low-cardinality because every distinct set creates another active stream and index burden.
- Structured metadata
- Non-indexed attributes stored with log entries. It is the appropriate home for useful high-cardinality context such as trace or request identifiers when supported by the query workflow.
- TSDB schema v13
- The current Loki index schema used with object storage. Schema start dates are immutable boundaries and must match recovered data.
- Compactor retention
- Loki's supported process for compacting index data and deleting chunks after the configured retention period, including tracked delete requests.
- Tenant isolation
- Separation of data and limits by tenant. It is only trustworthy when an authenticated gateway derives the tenant rather than accepting an arbitrary client header.
- Positions state
- Durable collector state recording how far a file or journal has been read. Loss can replay old lines or create gaps depending on source behavior.
- At-least-once delivery
- A delivery model in which retry can create duplicates. Operators design queries and alerts to tolerate bounded duplicates instead of inventing a high-cardinality deduplication label.
- Query fairness
- Limits and scheduling that prevent one large query or tenant from consuming disproportionate querier, cache and object-store capacity.
Before you copy
Values used in this guide
{{lokiVersion}}Approved Loki 3.7 patch.
Example: 3.7.3{{alloyVersion}}Approved Grafana Alloy 1.18 patch.
Example: 1.18.0{{lokiGateway}}Private TLS gateway URL used by agents and queries.
Example: https://loki.internal.example{{lokiCa}}Approved CA bundle for the Loki gateway.
Example: /etc/alloy/tls/loki-ca.pem{{alloyClientCert}}Alloy workload client certificate.
Example: /etc/alloy/tls/client.pem{{alloyClientKey}}secretProtected Alloy workload private key.
Example: /etc/alloy/tls/client-key.pem{{lokiTenant}}Synthetic validation tenant derived by the gateway.
Example: observability-test{{lokiBucket}}Dedicated object-store bucket or prefix.
Example: prod-observability-loki-eu1{{lokiRetention}}Approved application retention period.
Example: 744h{{alloyStorage}}Persistent local Alloy state directory.
Example: /var/lib/alloy{{logPath}}Approved application log glob.
Example: /var/log/example/*.log{{canaryLogFile}}Dedicated synthetic log file covered by the approved application glob.
Example: /var/log/example/canary.log{{canaryId}}Nonsecret unique identifier for a synthetic canary.
Example: ol-loki-20260728-001{{restoreBucket}}Separate bucket used only for recovery rehearsal.
Example: drill-observability-loki-eu1Security and production boundaries
- Logs commonly contain personal data, session identifiers, authorization headers and infrastructure secrets. Prevent collection at the source, redact before network export, restrict queries by role and audit access; retention is not a substitute for data minimization.
- Loki's multi-tenancy header is not authentication. A trusted gateway must authenticate the caller and overwrite or derive tenant identity. Negative cross-tenant tests are mandatory.
- Alloy receives only the operating-system groups and paths required for selected sources. Root, Docker socket and broad filesystem access materially expand the blast radius and require separate review.
- Object-store credentials are delivered through a secrets mechanism, never embedded in Alloy/Loki configuration, shell history, dashboards or backup manifests. Separate read, write and delete capabilities where the provider permits.
- Debug logging and sample exporters can reveal raw events. They are limited to synthetic fixtures in an isolated environment and disabled before production ingestion.
Stop before continuing if
- Stop before ingestion if tenant identity can be selected by an untrusted client, TLS verification is disabled, or a negative cross-tenant query succeeds.
- Stop if volatile or sensitive values are configured as labels, if cardinality is unknown, or if the redaction fixture reaches Loki unchanged.
- Stop if the object-store lifecycle is shorter than Loki retention, the compactor can access the production bucket during a restore drill, or versioning/audit evidence is unavailable.
- Stop if a collector needs root or broad host access without documented necessity, if persistent positions cannot be protected, or if queue growth can exhaust the host.
- Stop a rollout on rejected-entry growth, canary gaps, query SLO regression, unexpected cost, early deletion, or any sign of cross-tenant disclosure.
decision
Define tenants, data classes, retention and label budgets
Inventory every source before collection. Assign owner, tenant, purpose, sensitivity, retention, expected bytes per second, burst rate and query use. Approve a small label contract such as service, environment, region and host. Explicitly forbid secrets, user identifiers, request identifiers, client IPs, full paths and ephemeral IDs as indexed labels.
Why this step matters
A log platform becomes unsafe and expensive when collection precedes ownership, privacy and cardinality decisions. The contract makes tenancy, retention and bounded labels reviewable before any bytes leave a host.
What to understand
Estimate active streams from the Cartesian product of label values and include deployment churn. Validate with representative samples rather than production secrets.
Document whether high-cardinality context is dropped, redacted, retained as structured metadata or parsed only at query time.
Set separate retention where policy requires it, but do not create tenant fragmentation without an access or lifecycle reason.
System changes
- Creates a reviewed data-classification, tenant, label and capacity plan; it does not change running services.
Syntax explained
sha256sum- Produces a stable review fingerprint for the exact approved label list; it is not a security signature.
printf '%s\n' 'service,environment,region,host' | sha256sumb2a0e6f82ecb9fe92c84bc9f842c29a646b1823dc054fcf6e4d4ea8dcf4a6064 -
Checkpoint: Verify this layer before continuing
Continue whenEvery source has an owner and approved tenant, sensitivity, retention, volume and label set; the stream estimate fits capacity.
Stop whenStop if ownership, privacy basis, deletion policy, peak volume or a bounded label contract is missing.
If this step fails
The projected stream count is orders of magnitude above the host count.
Likely causeA volatile dimension was included in labels or one label has unbounded values.
List distinct values from synthetic or redacted samples.Calculate the upper bound for each label combination.
ResolutionRemove volatile labels, use structured metadata and repeat sizing before installation.
Security notes
- Never paste real tokens, emails, IP addresses or message bodies into planning artifacts.
Alternatives
- Retain only local journal logs until governance and capacity are approved.
Stop conditions
- Stop if ownership, privacy basis, deletion policy, peak volume or a bounded label contract is missing.
command
Verify pinned Loki and Alloy artifacts
Select Loki 3.7.3 or a newer reviewed 3.7 patch and Grafana Alloy 1.18 from official releases. Verify signatures or published digests in the organization's artifact pipeline, scan the immutable artifact, record the result and reject floating tags.
Why this step matters
Version evidence prevents an operator from following 3.7/1.18 behavior while deploying a different binary. Loki 3.7.3 removes Promtail, so a hidden legacy dependency must be found before rollout.
What to understand
Pin by digest or checksum and retain release-note review, vulnerability disposition and provenance in change control.
Run the exact binary used by the service; a package-manager candidate or workstation tool does not prove the host version.
Check configuration deprecations and storage-schema compatibility before moving between minor releases.
System changes
- Records approved immutable artifacts; no service is changed.
Syntax explained
-version / --version- Prints runtime build identity used to compare the deployed artifact with the reviewed release.
loki -version && alloy --versionloki, version 3.7.3 (branch: release-3.7.x, revision: 4fb960c) alloy, version v1.18.0
Checkpoint: Verify this layer before continuing
Continue whenExact reviewed versions and artifact digests match the deployment record; no Promtail dependency remains.
Stop whenStop on an unsigned/unverifiable artifact, floating tag, unexpected version or unresolved security advisory.
If this step fails
The binary reports an older release than the package inventory.
Likely causePATH selects a stale manual installation while the service unit uses another executable.
Inspect the service ExecStart path and resolved executable.Compare package files and immutable digest.
ResolutionRemove the ambiguous installation through change control and bind the service to the verified absolute path.
Security notes
- Do not download and execute unverified release artifacts directly on production hosts.
Alternatives
- Use the organization's mirrored, signed artifact repository.
Stop conditions
- Stop on an unsigned/unverifiable artifact, floating tag, unexpected version or unresolved security advisory.
config
Provision dedicated object storage and service identities
Create a Loki-only encrypted bucket or prefix with versioning and audit logs. Give ingest/storage components only required object operations and give compactor deletion through a separately reviewed policy. Keep bucket lifecycle longer than Loki retention and prevent the recovery environment from addressing the production bucket.
Why this step matters
Object storage is the durable data plane for Loki indexes and chunks. Dedicated scope, versioning and audit evidence reduce accidental deletion and make recovery possible, while separate delete permission limits compactor blast radius.
What to understand
Validate permissions with synthetic objects in the exact prefix: write/read for storage components and delete only for the compactor path.
Coordinate provider lifecycle with compactor retention. Lifecycle must never remove live chunks or indexes before Loki does.
Estimate request rate and recovery bandwidth as well as stored bytes. Region mismatch can dominate latency and cost.
System changes
- Creates bucket policy, encryption, lifecycle, versioning and service identities; these policies can delete or expose data if wrong.
Syntax explained
lifecycle_delete_after- Provider-side safety cleanup that must be longer than application retention and legal-hold requirements.
versioning- Retains prior object versions according to policy, enabling recovery from accidental overwrite or deletion.
object-storage-policyValues stay on this page and are never sent or saved.
bucket = "{{lokiBucket}}"
versioning = true
encryption = "provider-managed-or-approved-kms"
public_access = false
lifecycle_delete_after = "900d"
application_retention = "{{lokiRetention}}"Bucket: prod-observability-loki-eu1 Encryption: enabled Versioning: enabled Public access: blocked Audit logging: enabled Lifecycle minimum age: 900 days Loki retention: 744h Recovery principal production access: denied
Checkpoint: Verify this layer before continuing
Continue whenSynthetic object operations match least privilege, public access is blocked, lifecycle exceeds retention and recovery identity cannot reach production.
Stop whenStop if the bucket is shared without isolation, public, unencrypted, unversioned, unaudited or lifecycle can delete earlier than Loki.
If this step fails
Loki writes succeed but compactor deletion returns AccessDenied.
Likely causeThe compactor uses the ingest identity or the delete permission omits the exact prefix.
Inspect provider audit logs for the denied action and principal.Compare policy scope with the configured bucket/prefix.
ResolutionGrant only the missing delete action to the dedicated compactor identity, test on a disposable prefix and monitor the first retention cycle.
Security notes
- Credentials never appear in configuration, command lines or outputs; use workload identity or protected secret delivery.
Alternatives
- Use a managed Loki service when durable object-storage operations cannot be staffed.
Stop conditions
- Stop if the bucket is shared without isolation, public, unencrypted, unversioned, unaudited or lifecycle can delete earlier than Loki.
config
Configure Loki TSDB, retention and bounded limits
Render a Loki 3.7 configuration with TSDB schema v13, immutable schema start date, object-store references, compactor retention and explicit ingestion/query limits. Authentication is enforced at the gateway; Loki remains on a private network. Validate the complete rendered configuration before starting any component.
Why this step matters
The schema date, storage backend and retention configuration define how data can be read and deleted. A configuration can parse yet still be unsafe, so semantic review must accompany binary validation.
What to understand
Choose a future schema boundary only for new deployments or a staged migration. Never rewrite a past boundary after data exists.
Set per-tenant overrides where workloads differ, with documented owners and ceilings. Global unlimited ingestion or query range is not a capacity plan.
Keep Loki internal; `auth_enabled` enables tenant behavior but does not replace authenticating the caller at a gateway.
System changes
- Writes Loki storage schema, compactor and limit behavior. Incorrect values can make data unreadable, rejected or prematurely deleted.
Syntax explained
schema: v13- Selects the supported TSDB schema for index and chunk metadata.
retention_enabled- Allows compactor to apply application retention and tracked deletion.
reject_old_samples_max_age- Bounds historical ingestion and protects capacity; historical import needs a separate reviewed path.
/etc/loki/loki.yamlValues stay on this page and are never sent or saved.
auth_enabled: true
schema_config:
configs:
- from: 2026-08-01
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
compactor:
retention_enabled: true
delete_request_store: s3
limits_config:
retention_period: {{lokiRetention}}
reject_old_samples: true
reject_old_samples_max_age: 168h
ingestion_rate_strategy: global
max_query_length: 744hloki -verify-config -config.file=/etc/loki/loki.yamlconfig is valid
Checkpoint: Verify this layer before continuing
Continue whenThe exact Loki 3.7 binary validates the rendered config; schema, object store, retention and limits match review.
Stop whenStop on a past schema rewrite, missing delete store, unlimited tenant, unreviewed deprecated option or validation warning.
If this step fails
Loki fails with an unrecognized configuration field.
Likely causeThe candidate copied an option from an older release or the wrong component mode.
Run validation with the deployed binary.Compare the field against 3.7 configuration reference and upgrade notes.
ResolutionRestore the prior validated config, replace the removed option in staging and migrate only with supported procedure.
Security notes
- Rendered configuration must reference secrets without printing them and be readable only by the Loki service.
Alternatives
- Start with a smaller supported single-binary deployment for bounded noncritical load, retaining object storage and the same controls.
Stop conditions
- Stop on a past schema rewrite, missing delete store, unlimited tenant, unreviewed deprecated option or validation warning.
config
Enforce gateway authentication and tenant isolation
Place Loki behind a TLS gateway that authenticates workload and human identities, removes any inbound tenant header and sets the tenant from an allowlisted server-side mapping. Restrict Loki listeners to the gateway and internal components. Test positive access and explicit cross-tenant denial with synthetic tenants.
Why this step matters
Tenant boundaries are authorization boundaries. If callers can control X-Scope-OrgID or bypass the gateway, every other privacy and retention claim becomes unreliable.
What to understand
Use workload identity, mTLS or short-lived tokens. Map identity to exactly the permitted tenant and overwrite any inbound tenant header.
Apply the same gateway to Grafana, logcli, Alloy and automation; bypass listeners are blocked by network policy.
Create two synthetic tenants with unique canaries and prove each positive query plus both negative cross-tenant queries.
System changes
- Changes ingress authentication, authorization, headers and network paths. A mistake can expose all logs or stop ingestion.
Syntax explained
--org-id- Selects the test tenant for logcli, but the gateway must still authorize the authenticated caller for that tenant.
--addr- Uses the production-equivalent authenticated gateway instead of a direct Loki listener.
Values stay on this page and are never sent or saved.
logcli --addr={{lokiGateway}} --org-id={{lokiTenant}} labels2026/07/28 10:14:03 https://loki.internal.example/loki/api/v1/labels service environment host
Checkpoint: Verify this layer before continuing
Continue whenAuthorized synthetic tenant queries succeed; cross-tenant and direct-listener requests fail and are audited.
Stop whenStop immediately if a caller-selected tenant is accepted, a direct listener is reachable, TLS verification is disabled or negative tests expose data.
If this step fails
A valid token can query two unrelated tenants.
Likely causeThe gateway passes X-Scope-OrgID through or the identity mapping uses a wildcard.
Preserve gateway audit evidence and disable the credential.Inspect sanitized routing configuration and synthetic tenant tests.
ResolutionTreat as an incident, enforce header overwrite and one-to-one authorization, rotate affected credentials and repeat isolation tests.
Security notes
- Do not use real tenant data in boundary testing. Never place bearer tokens in shell history or screenshots.
Alternatives
- Use separate Loki deployments for regulatory boundaries that cannot rely on logical multi-tenancy.
Stop conditions
- Stop immediately if a caller-selected tenant is accepted, a direct listener is reachable, TLS verification is disabled or negative tests expose data.
config
Configure Alloy discovery, positions, labels and redaction
Run Alloy as a dedicated unprivileged account with persistent storage. Discover only approved files or journal units, normalize stable resource labels, remove forbidden labels and redact synthetic secret patterns before `loki.write`. Keep trace/request identifiers as structured metadata when needed. Validate and format the candidate configuration before reload.
Why this step matters
Collector policy is the last reliable point to minimize and redact data before it crosses a host boundary. Persistent positions limit replay after restart; narrow file permissions avoid turning a logging agent into a privileged surveillance process.
What to understand
Test redaction with synthetic plain, encoded and multiline fixtures. Regex replacement is not a complete data-loss-prevention system, so source applications must avoid secrets first.
Review file globs and journal matches for rotation behavior, symlinks and duplicate discovery. Persist storage on a protected local path.
Monitor component health, bytes read, entries forwarded/dropped, retry and process memory. Keep the local HTTP interface on loopback or authenticated private access.
System changes
- Installs a collector configuration, reads selected logs, stores positions locally and sends redacted events to Loki.
Syntax explained
local.file_match- Discovers only the approved file glob and attaches initial bounded labels.
stage.replace- Redacts a reviewed synthetic secret pattern before forwarding; it must be tested for bypasses.
--storage.path- Persists component state such as read positions under a protected service-owned directory.
/etc/alloy/config.alloyValues stay on this page and are never sent or saved.
local.file_match "application" {
path_targets = [{ "__path__" = "{{logPath}}", "service" = "example-api", "environment" = "production" }]
}
loki.source.file "application" {
targets = local.file_match.application.targets
forward_to = [loki.process.application.receiver]
}
loki.process "application" {
stage.replace {
expression = "(?i)(authorization|token|password)=([^ ]+)"
replace = "$1=[REDACTED]"
}
stage.static_labels { values = { service = "example-api", environment = "production" } }
forward_to = [loki.write.primary.receiver]
}
loki.write "primary" {
endpoint {
url = "{{lokiGateway}}/loki/api/v1/push"
tls_config {
ca_file = "{{lokiCa}}"
cert_file = "{{alloyClientCert}}"
key_file = "{{alloyClientKey}}"
}
}
}Values stay on this page and are never sent or saved.
alloy fmt /etc/alloy/config.alloy && alloy run --storage.path={{alloyStorage}} --server.http.listen-addr=127.0.0.1:12345 /etc/alloy/config.alloyts=2026-07-28T10:20:11Z level=info msg="starting Alloy" version=v1.18.0 ts=2026-07-28T10:20:11Z level=info msg="component evaluation complete" controller_path=/ ts=2026-07-28T10:20:11Z level=info msg="http server started" addr=127.0.0.1:12345
Checkpoint: Verify this layer before continuing
Continue whenAll components are healthy, only approved sources are discovered, positions persist and every synthetic secret fixture is redacted before Loki.
Stop whenStop if Alloy needs root, sees unrelated files, loses positions, emits a secret fixture, creates volatile labels or exposes its admin endpoint.
If this step fails
A canary line appears twice after every restart.
Likely causeThe storage path is ephemeral or two file targets overlap.
Inspect service arguments and storage ownership.Compare discovered target paths and labels.
ResolutionPersist the storage directory, remove overlapping discovery and document bounded at-least-once replay.
Security notes
- The Alloy account receives only required journal group or file ACL access; never grant Docker socket access as a shortcut.
Alternatives
- Use an OpenTelemetry Collector log pipeline if its selected components and persistence semantics are fully reviewed.
Stop conditions
- Stop if Alloy needs root, sees unrelated files, loses positions, emits a secret fixture, creates volatile labels or exposes its admin endpoint.
verification
Prove end-to-end ingestion, redaction and isolation
Append a synthetic line containing a unique canary and fake token to the dedicated canary file covered by the approved Alloy target. Query by bounded labels and canary through the authenticated gateway. Confirm the fake token is absent, the line appears once within the SLO and another tenant cannot retrieve it.
Why this step matters
An end-to-end canary simultaneously verifies discovery, positions, processing, authentication, tenancy, ingestion, storage and query. Synthetic secrets make redaction testable without risking a credential.
What to understand
Use a unique nonsecret identifier and an unmistakably fake token. Query a narrow time range and stable label selector.
Verify both positive and negative behavior: expected tenant sees exactly one line, alternate tenant sees none, and raw fake token is absent.
Record read-after-write latency and configuration revision so later regressions have a baseline.
System changes
- Appends one synthetic line to the dedicated canary file and queries it; no application log or production secret is modified.
Syntax explained
tee -a- Appends instead of replacing the dedicated canary file; the test writer needs access only to this file.
|=- Applies an exact line filter after selecting a bounded stream, limiting scanned data.
--since=10m- Bounds the query interval for deterministic verification and cost.
Values stay on this page and are never sent or saved.
printf '%s\n' 'canary={{canaryId}} token=synthetic-not-a-secret status=ok' | tee -a {{canaryLogFile}} >/dev/null && logcli --addr={{lokiGateway}} --org-id={{lokiTenant}} query '{service="example-api",environment="production"} |= "{{canaryId}}"' --since=10m2026-07-28T10:27:18Z {environment="production",host="app01",service="example-api"} canary=ol-loki-20260728-001 token=[REDACTED] status=ok
Cross-tenant query: 0 entries
Read-after-write latency: 2.4sCheckpoint: Verify this layer before continuing
Continue whenOne redacted canary is visible to the authorized tenant within SLO and invisible to the alternate tenant.
Stop whenStop on missing/duplicate canary, raw fake token, cross-tenant visibility, unbounded query or excessive latency.
If this step fails
The authorized query returns no canary.
Likely causeDiscovery, timestamp parsing, gateway routing, Loki ingestion or query range is wrong.
Check Alloy component health and write status.Inspect Loki rejected-entry and gateway metrics for the canary timestamp.
ResolutionRepair the first failing layer and repeat with a new canary; do not widen tenant access or disable validation.
Security notes
- Use synthetic values only; never test redaction by deliberately logging a real credential.
Alternatives
- Use Loki canary for continuous evidence after the manual acceptance test.
Stop conditions
- Stop on missing/duplicate canary, raw fake token, cross-tenant visibility, unbounded query or excessive latency.
verification
Measure labels, ingestion pressure and query cost
Analyze stream labels and value counts, then inspect Alloy queues, Loki rejection metrics, active streams, object-store requests, query bytes and latency under representative load. Set alerts and per-tenant limits from measured headroom rather than unlimited defaults.
Why this step matters
Cardinality and query cost are emergent production properties. Measuring them before broad onboarding detects unsafe label changes and provides evidence for limits, cache and capacity decisions.
What to understand
Compare active streams and values with the approved upper bound. A label can look harmless while a deployment changes it from stable service name to pod hash.
Track accepted/rejected bytes, queue age, retries, object requests and query scan volume during normal and burst conditions.
Set warnings before hard limits, then test the user-visible failure mode so teams understand how to reduce load safely.
System changes
- Reads metadata and metrics; subsequent alert/limit changes require separate reviewed configuration.
Syntax explained
series '{}'- Requests stream metadata for analysis; the one-hour window limits impact, but an empty selector can still be expensive at scale.
--analyze-labels- Summarizes label uniqueness and stream distribution to identify cardinality risk.
Values stay on this page and are never sent or saved.
logcli --addr={{lokiGateway}} --org-id={{lokiTenant}} series '{}' --since=1h --analyze-labelsTotal Streams: 84 Unique Labels: 4 Label Name Unique Values Found In Streams service 7 84 environment 3 84 region 2 84 host 12 84 Unbounded label candidates: 0
Checkpoint: Verify this layer before continuing
Continue whenObserved streams and label values fit the budget; queues are bounded, rejection is zero in baseline and query SLO has headroom.
Stop whenStop onboarding if values grow without bound, queues never drain, rejection rises, object throttling appears or representative queries exceed SLO.
If this step fails
A new `filename` label has tens of thousands of values.
Likely causeThe value includes rotated paths or container IDs rather than a stable source category.
Sample sanitized label values.Compare the Alloy config revision and deployment timestamp.
ResolutionRemove or normalize the label, keep the path as metadata, and add a cardinality regression alert.
Security notes
- Metadata can itself expose topology; restrict series analysis to operators and the authorized tenant.
Alternatives
- Use sampled, tenant-specific analysis if a global empty selector is too expensive.
Stop conditions
- Stop onboarding if values grow without bound, queues never drain, rejection rises, object throttling appears or representative queries exceed SLO.
warning
Validate retention and deletion on a disposable tenant
Create a disposable tenant with short test retention, ingest timestamped canaries, wait through the compactor interval and prove only eligible data disappears. Track a test deletion request to completion. Confirm object versions and audit events, then remove the test override without touching production prefixes.
Why this step matters
Retention and deletion are destructive and cannot be accepted from configuration review alone. A separate tenant and prefix prove eligibility, compactor operation and auditability without gambling production history.
What to understand
Use synthetic data and a tenant whose objects can be independently inventoried. Keep the production tenant and restore bucket outside the test identity.
Account for compactor delay: policy expiry is not necessarily immediate physical deletion. Record request and completion timestamps.
Verify the object-store lifecycle does not perform the deletion first, otherwise application retention evidence is invalid.
System changes
- Deletes synthetic test data through the supported compactor path; a wrong tenant or bucket can irreversibly delete production logs.
Syntax explained
--org-id=retention-test- Targets the disposable tenant, but gateway authorization and object-prefix isolation must also prove the boundary.
--from / --to- Queries exact UTC bounds around retained and expired canaries.
Values stay on this page and are never sent or saved.
logcli --addr={{lokiGateway}} --org-id=retention-test query '{service="retention-canary"}' --from='2026-07-27T00:00:00Z' --to='2026-07-29T00:00:00Z'2026-07-28T11:00:00Z retention-canary id=keep-001 Queryable entries older than test retention: 0 Compactor delete request: processed Production tenant object deletions: 0
Checkpoint: Verify this layer before continuing
Continue whenExpired synthetic entries and only those entries disappear through compactor; kept canary remains and production deletion count is zero.
Stop whenStop if target identity, tenant, bucket/prefix, time synchronization or object audit evidence is ambiguous.
If this step fails
Both expired and retained canaries disappear.
Likely causeRetention override, timestamps or object lifecycle is broader/shorter than intended.
Disable the test compactor path.Inspect versioned objects, effective limits and lifecycle audit events.
ResolutionRestore test object versions in isolation, correct the boundary and repeat; treat any production impact as an incident.
Security notes
- Require peer confirmation of tenant and bucket before deletion. Never use personal or production log content in the test.
Alternatives
- Validate retention first in a completely separate Loki deployment and bucket.
Stop conditions
- Stop if target identity, tenant, bucket/prefix, time synchronization or object audit evidence is ambiguous.
command
Back up configuration, rules and object manifests
Snapshot reviewed Loki/Alloy/gateway configuration, tenant overrides, rules and version metadata without secret values. Export a timestamped object inventory and provider versioning evidence. Store encrypted manifests and configuration in an independent repository with retention and restore ownership.
Why this step matters
Object data without matching schema, tenant policy and rules is difficult to interpret; configuration without object inventory is not a backup. A manifest joins both sides while excluding live secrets.
What to understand
Export normalized configuration references, schema dates, component versions, tenant overrides, gateway mappings and rule files.
Record object count, bytes, prefixes, versions and audit checkpoint. Do not copy a live bucket ad hoc while deletion is uncoordinated.
Encrypt the backup under an independent recovery identity and scan for embedded credentials before retention.
System changes
- Writes checksums and backup artifacts to the approved repository; careless copying can expose log data or secrets.
Syntax explained
sha256sum- Detects accidental configuration change or corruption; authenticity also needs a protected signed manifest or repository control.
sha256sum /etc/loki/loki.yaml /etc/alloy/config.alloy /etc/loki/rules/* > loki-config-manifest.sha2567f590bce30b0f49a4c0c17f2f10437d485c9649bfbe28152b5120a09439d12bb /etc/loki/loki.yaml 9bbda13e0cf9fb77bd5454cae16a84c18f49580114bb6086873bd32e76e3a351 /etc/alloy/config.alloy Object inventory: 184291 objects, 2.84 TiB, versioning enabled Secrets in archive scan: 0
Checkpoint: Verify this layer before continuing
Continue whenIndependent repository contains secret-free versioned configuration, object inventory and restore instructions with verified checksums.
Stop whenStop if secrets are embedded, object inventory is incomplete, deletion races are unaccounted for, or recovery identity shares the production failure domain.
If this step fails
The secret scan finds an object-store credential in rendered config.
Likely causeA deployment template inlined a secret rather than referencing protected delivery.
Quarantine and access-restrict the archive.Identify every copy and audit access without printing the value.
ResolutionRotate the credential, replace inline material with a reference, regenerate the archive and document exposure response.
Security notes
- Backups contain log-access policy and topology and must be protected at least as strongly as production.
Alternatives
- Use provider-native cross-account replication plus separately backed-up configuration and manifests.
Stop conditions
- Stop if secrets are embedded, object inventory is incomplete, deletion races are unaccounted for, or recovery identity shares the production failure domain.
verification
Restore into an isolated bucket and prove RPO/RTO
Create a network-isolated recovery Loki using a separate bucket and tenant identity. Restore versioned objects plus the matching schema/configuration, keep compactor deletion disabled during inspection, query known canary windows and compare stream counts. Record last recoverable timestamp, elapsed recovery time and all gaps before cleanup.
Why this step matters
Only an isolated restore demonstrates that storage, indexes, schema and authorization can produce useful queries after loss. A separate bucket prevents the recovery compactor or lifecycle from modifying production.
What to understand
Provision a distinct service identity, bucket, DNS name and tenant gateway. Deny every recovery component access to production storage.
Restore the exact schema periods and version-compatible configuration, initially disabling deletion and external notifications.
Query canaries across oldest, newest and boundary periods, compare manifest/counts and measure RPO/RTO. Clean up only after preserving evidence.
System changes
- Copies sensitive logs into an isolated recovery environment and may delete that disposable environment after evidence capture.
Syntax explained
--since=168h- Bounds the recovery query to the known evidence window instead of scanning the whole restored history.
Values stay on this page and are never sent or saved.
logcli --addr=https://loki-drill.internal.example --org-id={{lokiTenant}} query '{service="loki-canary"} |= "{{canaryId}}"' --since=168h2026-07-27T22:04:11Z {service="loki-canary",environment="production"} id=ol-loki-20260728-001
Object manifest: MATCH
Known canary windows: 12/12 PASS
Cross-tenant query: 0 entries
Measured RPO: 00:03:14
Measured RTO: 01:18:42Checkpoint: Verify this layer before continuing
Continue whenManifest and canary windows match, tenant isolation holds, no production objects change and measured RPO/RTO meet objectives.
Stop whenStop if any recovery principal can reach production, compactor deletion is enabled, schema/config is uncertain, manifest differs or sensitive access is uncontrolled.
If this step fails
Restored Loki reports missing chunks for an index.
Likely causeThe object restore is incomplete, versions mismatch or lifecycle deleted required chunks.
Compare object/version inventory with the signed manifest.Query a known narrow canary window with caches bypassed.
ResolutionRestore the missing versions into the recovery bucket, rebuild no indexes by guesswork, and repeat evidence checks.
Security notes
- Recovery contains production data. Apply production-equivalent encryption, access logging, retention and secure cleanup.
Alternatives
- Use provider-supported cross-region recovery, still rehearsed against a nonproduction destination.
Stop conditions
- Stop if any recovery principal can reach production, compactor deletion is enabled, schema/config is uncertain, manifest differs or sensitive access is uncontrolled.
Finish line
Verification checklist
logcli --addr={{lokiGateway}} --org-id={{lokiTenant}} query '{service="example-api"} |= "{{canaryId}}"' --since=10mExactly one redacted canary appears within the read-after-write SLO.logcli --addr={{lokiGateway}} --org-id=unauthorized-test query '{service="example-api"} |= "{{canaryId}}"' --since=10mThe unauthorized identity is denied or returns no data and the event is audited.logcli --addr={{lokiGateway}} --org-id={{lokiTenant}} series '{}' --since=1h --analyze-labelsOnly approved bounded labels exist and stream/value counts fit the documented budget.logcli --addr={{lokiGateway}} --org-id=retention-test query '{service="retention-canary"}' --since=168hOnly eligible synthetic data is deleted by compactor; kept canary remains and no production object is deleted.logcli --addr=https://loki-drill.internal.example --org-id={{lokiTenant}} query '{service="loki-canary"}' --since=168hObject manifest and known canary windows match, isolation holds and measured RPO/RTO meet objectives.Recovery guidance
Common problems and safe checks
Alloy reports HTTP 401 or 403 while writing to Loki.
Likely causeThe gateway rejected the workload identity, the tenant header is missing, or an agent can choose a tenant it is not authorized to use.
Inspect the gateway authentication decision and sanitized request headers without logging bearer tokens.Query Alloy write metrics for status code and tenant, then compare the workload identity mapping with the approved tenant registry.
ResolutionRepair the server-side identity-to-tenant mapping and rotate any exposed token. Never work around the failure by making Loki unauthenticated or allowing clients to select arbitrary X-Scope-OrgID values.
Loki rejects streams with too many labels or reports cardinality limits.
Likely causeA volatile field such as request ID, client IP, pathname, user, container hash or full filename was promoted to an indexed stream label.
Use logcli series --analyze-labels for the affected tenant and time window.Inspect the Alloy processing stage and sample structured metadata without printing sensitive values.
ResolutionKeep only bounded operational dimensions as labels, move high-cardinality fields to structured metadata or parsed query-time fields, and restart ingestion with a reviewed label budget.
Queries are slow although ingestion succeeds.
Likely causeThe query scans a broad time range, uses an unselective stream selector, performs expensive parsing before filtering, or the query frontend/cache is undersized.
Compare bytes scanned, query duration and cache hit metrics for a narrow and broad interval.Run the same query with the most selective stable labels and an early line filter.
ResolutionRewrite the query to select bounded labels and filter early, split long investigations, and scale the supported query path only after measuring storage and cache behavior.
Recent log lines are duplicated after an Alloy restart.
Likely causeThe positions state is not persistent, the same file is matched by two components, or a replacement agent replayed an earlier offset.
Check the configured Alloy storage path and its ownership.Compare discovery targets and stream labels for duplicate file matches.
ResolutionPersist Alloy state on durable local storage, remove overlapping targets, and accept bounded at-least-once replay rather than deduplicating by a new high-cardinality label.
Expected journal or file logs are missing.
Likely causeThe Alloy service lacks read permission, the journal cursor/path is wrong, discovery relabeling dropped the target, or the application writes somewhere else.
Run Alloy configuration validation and inspect component health.As the Alloy service account, list only the intended journal or log path and compare a known canary line.
ResolutionGrant the narrow group/path permission, correct discovery, and prove one canary line end to end. Do not run Alloy as root solely to bypass file permissions.
Loki rejects entries as too old or too far ahead.
Likely causeHost clocks differ, buffered entries exceeded the accepted age, a parser changed timestamps, or replayed files contain historical data.
Compare NTP status and UTC time on agents and Loki nodes.Inspect a sanitized rejected timestamp beside the source file timestamp and Loki limits.
ResolutionCorrect time synchronization and parser behavior, then use a controlled historical import path if required. Do not weaken global age limits without capacity and abuse review.
Alloy memory or queue usage grows continuously during a Loki outage.
Likely causeThe destination is unavailable longer than the bounded retry window, the producer rate exceeds recovery throughput, or local buffering limits are not explicit.
Inspect Alloy retry, dropped-entry, queue and process memory metrics.Measure object-store, gateway and distributor health before changing buffer limits.
ResolutionRestore the destination, apply intentional admission or drop policy for noncritical logs, and size bounded buffers from outage/RPO objectives. Never allow an unbounded queue to exhaust the host.
The compactor does not delete data after the retention period.
Likely causeRetention is disabled, delete-request storage is missing, the compactor ring is unhealthy, or object-store permissions prevent deletion.
Inspect compactor metrics, ring state and sanitized object-store authorization errors.Compare schema period, retention settings and object lifecycle policy.
ResolutionRepair the compactor and least-privilege delete permissions, validate on a test tenant, and preserve audit evidence. Do not manually delete index or chunk prefixes.
Data disappears earlier than Loki retention specifies.
Likely causeAn object-store lifecycle rule deletes objects before the compactor retention horizon or a shared bucket policy affects Loki prefixes.
Review bucket lifecycle rules and object versions for the Loki prefix.Compare earliest queryable data with compactor deletion metrics and audit logs.
ResolutionChange lifecycle policy so it is longer than application retention, isolate the bucket/prefix, and restore versioned objects into a quarantine environment before production recovery.
Queries return partial results or gaps after a storage incident.
Likely causeObjects, indexes or tenant prefixes are incomplete, a cache hides recovery progress, or the restored configuration uses a mismatched schema period.
Compare object inventory and checksums against the recovery manifest.Query known canary windows with caches bypassed where supported.
ResolutionRestore the complete versioned object set and matching schema/configuration in isolation, verify canaries, then perform a controlled cutover. Never reconstruct a production index by guessing.
A query exposes another tenant's logs.
Likely causeThe gateway trusts a caller-supplied tenant header, a service account is mapped to several tenants, or a cache key omits tenant identity.
Immediately disable affected access and preserve gateway and audit logs.Run negative tests using two synthetic tenants and distinct canary values.
ResolutionTreat this as a security incident, fix identity-derived tenancy and tenant-aware caching, rotate credentials, assess disclosure, and restore service only after negative isolation tests pass.
Sensitive tokens or personal data appear in Loki.
Likely causeThe source emitted secrets, the Alloy redaction expression missed an encoding or multiline form, or a debug path bypassed processing.
Restrict access, preserve minimal incident evidence, and query only a bounded affected window.Review source logging and processing rules with synthetic secret fixtures.
ResolutionStop the source leak, update redaction before export, revoke credentials, follow deletion and breach policy, and validate encoded/multiline fixtures without copying real sensitive data.
Loki startup fails after a configuration update.
Likely causeThe configuration contains a removed option, incompatible schema date, invalid storage credentials reference or unsupported deployment mode.
Run Loki configuration verification with the exact target binary and redacted environment.Compare the change with the official upgrade notes and previous rendered configuration.
ResolutionRestore the last validated configuration, keep data untouched, and stage the migration in an isolated environment before retrying.
Ingestion works but Grafana shows no logs.
Likely causeThe data source uses the wrong gateway, tenant mapping, time zone, stream selector or query range.
Use logcli through the same authenticated gateway and tenant.Query the exact canary label and UTC timestamp independently of dashboards.
ResolutionCorrect the Grafana data source and query, then compare the same canary through logcli and Grafana. Avoid widening tenant permissions.
Object-store requests are throttled or expensive.
Likely causeTenant/query concurrency, index layout, retention scans or cache misses generate excessive requests, or the bucket is in the wrong region.
Inspect Loki request metrics by operation and object-store provider metrics.Compare region, endpoint, cache hit rate and query concurrency with the approved design.
ResolutionCorrect region and supported cache/query settings, add capacity intentionally, and enforce query fairness. Do not hide cost by disabling audit or retention.
The restored environment starts deleting recovered objects.
Likely causeCompactor retention remained enabled against a copied production bucket or restore-time clock/schema settings make recovered data immediately eligible.
Stop the compactor and inspect its target bucket, tenant and retention configuration.Review object-store audit events and version history.
ResolutionUse a separate recovery bucket, disable deletion until validation completes, restore object versions, and require an explicit promotion checkpoint.
Alloy cannot reload a changed configuration.
Likely causeRiver syntax, component references or secret files are invalid, or the service lacks permission to read a referenced file.
Run `alloy fmt` and the supported validation/check command against the candidate file.Inspect the component graph and file ownership without exposing secrets.
ResolutionKeep the running configuration, repair the candidate offline, and reload only after all components report healthy.
A low-cardinality label suddenly has thousands of values.
Likely causeAn application changed label semantics, a relabel rule copied an unbounded field, or ephemeral resource identifiers replaced stable service identity.
Compare label value counts before and after the deployment.Inspect a sanitized sample and the configuration revision that introduced the change.
ResolutionRoll back the label mapping, preserve stable dimensions, move volatile attributes to structured metadata, and add a cardinality alert.
Log deletion requests remain pending.
Likely causeDeletion workers are unavailable, the request store is inaccessible, or the tenant/range does not match stored data.
Inspect deletion request status and compactor worker metrics.Verify request-store access and tenant identity without resubmitting duplicate requests.
ResolutionRepair the supported deletion path and track the original request to completion; do not bypass it with ad hoc bucket deletion.
The Loki canary reports intermittent missing entries.
Likely causeIngestion, query, storage or time synchronization is intermittently failing, or the canary's tenant route differs from production.
Correlate canary write and read latency with distributor, ingester, querier and object-store metrics.Check tenant routing and clock synchronization across the exact interval.
ResolutionRepair the failing layer and keep the canary representative of production authentication, tenancy and retention.
Reference
Frequently asked questions
Why not use Promtail?
Promtail was deprecated and removed as of Loki 3.7.3. Its collection capabilities moved to Grafana Alloy, which is the primary supported collector in this guide.
Should request_id be a Loki label?
No. It is normally unbounded and creates a stream per value. Keep it as structured metadata or parse it at query time, then select a bounded stream first.
Can object-store lifecycle replace Loki retention?
No. Loki's compactor understands indexes and chunks. An earlier bucket lifecycle can orphan or remove data unexpectedly; use lifecycle only beyond the application retention horizon.
Recovery
Rollback
Do not roll back by disabling TLS, authentication, tenant enforcement or retention safeguards. Stop new onboarding, preserve object versions and telemetry, restore the last validated component configuration one layer at a time, or redirect collectors to an approved bounded buffer while repairing the platform.
- Freeze schema, retention and deletion changes; disable only the affected rollout or compactor test while preserving object versions and audit evidence.
- Restore the previous signed Loki, gateway or Alloy configuration to a canary instance and validate it with the exact binary before staged rollout.
- If labels or redaction regressed, stop affected collection, rotate exposed credentials, correct source logging and processing, then resume with a new synthetic canary.
- If object data changed, use provider version history to restore into a separate recovery bucket and verify manifest/canaries before any production action.
- If tenant isolation failed, revoke access, preserve evidence, assess disclosure, fix server-side mapping and repeat two-tenant negative tests.
- Return normal ingestion only after queues drain within limits, rejection is zero, canaries pass, query SLO is restored and backup/retention state is known.
Evidence