OneLinersCommand workbench
Guides
Observability & Monitoring / Security / Services & Applications

Collect host and application telemetry with OpenTelemetry Collector

Deploy OpenTelemetry Collector Contrib 0.157.0 agents and gateways with least-privilege receivers, memory protection, privacy transforms, bounded persistent queues, TLS identity and failure drills.

270 min11 stepsHigh-impact changeRevision 2
Save or explore
Save to collectionCreate a collection in the sidebar first.
0 of 11 steps completed
Goal

Move traces, metrics and logs through a measurable, privacy-aware and backpressure-safe Collector path without confusing local queueing with durable telemetry storage.

Supported environments
  • OpenTelemetry Collector Contrib 0.157.0
  • Linux supported systemd-based distribution
  • OTLP gRPC and HTTP with TLS
Prerequisites
  • Telemetry contract Signal owners, stable resource identity, prohibited fields, volume, latency and accepted loss/duplicate behavior.
  • Capacity Known cgroup/service memory, dedicated disk quota, sustained/burst intake and accepted backend outage window.
  • Identity and PKI Private DNS, trusted CA, dedicated workload identities and tested certificate rotation/revocation.
  • Backends Authorized OTLP backends with durable storage, tenant controls, retention and query access.
  • Host permissions Dedicated non-root agent account and explicit read access only to approved telemetry sources.
  • Test environment Representative isolated agents, gateways and backend route for burst, outage, restart and redaction fixtures.
Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System
  • A pinned OpenTelemetry Collector Contrib 0.157.0 agent-and-gateway topology that receives OTLP traces, metrics and logs plus approved host/file telemetry without exposing debug or administration endpoints publicly.
  • Pipelines ordered for protection and correctness: memory limiter first, privacy transformations and filtering before export, resource normalization, bounded batching, TLS-authenticated exporters, finite retry and persistent queues.
  • Synthetic validation for all three signals, internal telemetry and backpressure alerts, a credential-rotation procedure, least-privilege host access and a recovery model that distinguishes local queue replay from backend durability.
Observable outcome
  • Each signal arrives at the authorized backend with stable service/environment identity, expected transformations and no synthetic secret, while an unauthorized exporter identity is denied.
  • Representative bursts trigger bounded memory/backpressure behavior rather than OOM, queues survive a controlled restart and drain within the accepted RPO without unbounded disk growth.
  • Operators can identify receive, process, queue and export failure from Collector self-telemetry and can restore the last validated configuration without enabling payload debug logging.

Architecture

How the parts fit together

OpenTelemetry Collector Contrib 0.157.0 agents run close to workloads for host/file access and local OTLP, while redundant gateways provide a stable authenticated egress point. Every pipeline begins with memory_limiter so load is refused before expensive batching, then applies privacy and resource policy, followed by batch. Exporters use TLS with a trusted CA and dedicated workload identity. Sending queues are finite and use file_storage when bounded restart/outage survival is required; retry has an explicit elapsed-time policy. The queue is at-least-once local buffering, not a telemetry backup. Administrative endpoints bind to loopback or a protected management network. Internal metrics and synthetic signals reveal receive, refusal, drop, queue, retry and export state without emitting production payloads to debug logs.

Workload instrumentationProduces OTLP signals with stable service identity and avoids collecting secrets at source.
Host agentsReceive local OTLP, collect approved host/file data with narrow access, redact and forward across a protected boundary.
Gateway poolCentralizes policy, batching, authentication and backend egress while exposing bounded health and self-telemetry.
Memory limiterApplies soft/hard memory protection before batching and queues can exhaust the runtime.
Persistent sending queueBuffers a finite amount of export work across retry/restart using protected local storage and at-least-once semantics.
Telemetry backendsProvide durable storage, tenant authorization, query and retention; their resilience is separate from Collector buffering.
Collector self-telemetryReports accepted/refused/dropped/exported items, queue capacity, retries, memory, CPU and configuration health.
  1. Inventory signals, fields, owners, sensitivity, volume and backend contract; approve stable resource identity and data minimization.
  2. Applications send OTLP to a local authenticated or loopback agent, and agents read only approved host/file sources.
  3. Memory limiter protects each pipeline before transform/filter/resource processors normalize and redact records.
  4. Batch groups bounded requests; exporter queue and retry absorb a finite downstream outage using protected persistent state where required.
  5. Gateways authenticate to the backend over verified TLS; no debug exporter or payload logging exists on the production path.
  6. Self-telemetry and synthetic events verify all three signals, backpressure and authorization, while recovery restores config and replays only bounded queues.

Assumptions

  • OpenTelemetry Collector Contrib 0.157.0 is the current official release documented in July 2026. The exact artifact is pinned and component stability is reviewed because the Collector has mixed stability across components.
  • Linux is the reference environment. Agents run as a dedicated non-root account with only required journal/file/host filesystem access; gateways require no host filesystem access.
  • OTLP/gRPC and OTLP/HTTP endpoints use TLS and workload authentication when crossing a trust boundary. Local loopback or protected Unix-domain paths are preferred inside one host.
  • Telemetry backends enforce tenant identity, durable retention and query authorization. The Collector never substitutes for backend replication or backup.
  • Container or service memory/disk limits are known. Memory limiter thresholds, queue capacity, batch size and retry horizon are derived from representative rate and outage objectives.
  • The organization defines prohibited fields, redaction behavior, retention and lawful use for traces, metrics and logs. A synthetic fixture suite tests transformations.
  • Secrets are supplied through protected environment/file/workload identity mechanisms and never embedded in YAML, endpoints, command lines or debug output.
  • Stateful processors such as tail sampling or cumulative-to-delta conversion require consistent routing and separate correctness review; the base design avoids claiming correctness without it.
  • Collector internal telemetry is scraped from a protected endpoint and alerting can distinguish process liveness from export success.

Key concepts

Distribution
A Collector binary containing a selected set of components. Core and Contrib differ; configuration only works when the exact distribution includes each receiver, processor, exporter and extension.
Pipeline
An ordered path for one signal from receivers through processors to exporters. A component configured but omitted from service.pipelines does nothing.
Memory limiter
A protective processor that refuses data when memory crosses configured thresholds. It should appear first so later buffers cannot exhaust the Collector.
Sending queue
An exporter-side finite buffer. With file_storage it can survive restart, but replay is at-least-once and host loss can still destroy it.
Backpressure
A condition where downstream capacity is below input. It appears as queue growth, retry, refusal or drops and must be bounded rather than hidden.
OTTL
OpenTelemetry Transformation Language used by transform/filter processors. Context and field semantics are versioned and must be validated with fixtures.
Agent and gateway
Agents collect near workloads; gateways centralize egress and policy. Stateful processing and load balancing determine whether related telemetry must reach the same gateway.
Self-telemetry
Collector's own metrics, logs and optional traces. It should reveal pipeline health without logging full production payloads.

Before you copy

Values used in this guide

{{otelVersion}}

Pinned Collector Contrib version.

Example: 0.157.0
{{otelGateway}}

Private TLS gateway OTLP endpoint.

Example: otel-gateway.internal.example:4317
{{otelBackend}}

Authorized backend OTLP endpoint.

Example: telemetry-backend.internal.example:4317
{{otelCa}}

Protected CA bundle path.

Example: /etc/otelcol/tls/ca.pem
{{otelCert}}

Collector client certificate path.

Example: /etc/otelcol/tls/client.pem
{{otelKey}}secret

Collector private key path.

Example: /etc/otelcol/tls/client-key.pem
{{otelStorage}}

Quota-controlled persistent queue directory.

Example: /var/lib/otelcol
{{otelMemoryLimit}}

Hard limiter below cgroup/service limit.

Example: 768
{{otelMemorySpike}}

Allowed short spike deducted from hard limit.

Example: 192
{{otelQueueSize}}

Finite exporter queue units sized from RPO.

Example: 10000
{{otelLogPath}}

Approved filelog include glob.

Example: /var/log/example/*.json

Security and production boundaries

  • Receivers are network services. Bind locally where possible, require TLS/authentication across trust boundaries, restrict source networks and cap request size/concurrency.
  • Hostmetrics and filelog access can reveal user, process, mount and application data. Agents get only required namespaces and paths; gateways get none.
  • Debug exporter, zPages, pprof and detailed logs can disclose payloads or topology. Keep them disabled in production or protected on loopback with synthetic data only.
  • Redaction occurs before every production exporter and is backed by source-side minimization. A failed transform is not allowed to silently forward unredacted data.
  • Private keys and tokens are not in YAML or environment dumps. Certificate rotation validates SAN, trust chain, file ownership and reload semantics.

Stop before continuing if

  • Stop if any receiver crosses a trust boundary without verified TLS and workload authentication or any admin/debug endpoint is publicly reachable.
  • Stop if memory limiter is absent/not first, queues or retry are unbounded, persistent storage shares an unmonitored root filesystem, or representative bursts OOM the process.
  • Stop if a synthetic secret reaches an exporter, transform errors are ignored without policy, or payload debug logging is enabled on production data.
  • Stop if an unauthorized backend identity succeeds, stable resource identity is not defined, or stateful processing lacks consistent routing evidence.
  • Stop rollout on sustained queue growth, export failure, refused telemetry above budget, disk pressure or missing end-to-end canaries.
01

decision

Inventory signals, fields, trust boundaries and SLOs

read-only

Map every producer, receiver protocol, signal, volume, sensitivity, retention, backend tenant and owner. Define stable service/environment/resource attributes and prohibited fields. Set accepted loss, duplicate, latency, outage buffer, memory and disk budgets before choosing topology.

Why this step matters

A Collector cannot invent a privacy, identity or durability contract. Explicit signal and failure budgets make queue, memory, retry and topology decisions measurable.

What to understand

Measure normal and burst items/bytes for each signal and include deployment/retry storms.

Classify every network hop as same-process, same-host, private network or external trust boundary.

Write prohibited attribute/body patterns and synthetic fixtures before building transforms.

System changes

  • Creates an approved telemetry contract; running systems are unchanged.

Syntax explained

sha256sum
Fingerprints the reviewed contract header so change records can reference an exact revision.
Command
printf '%s\n' 'traces metrics logs | owner | sensitivity | rate | RPO | latency' | sha256sum
Example output / evidence
880069c8ee2cf815956d9f2555b338fe4bdbdc12d3002a7efc03850f553f3961  -

Checkpoint: Verify this layer before continuing

Continue whenOwners, fields, rates, trust boundaries, loss/duplicate behavior and SLOs are explicit for all signals.

Stop whenStop if any signal has unknown owner, sensitivity, backend, volume or accepted loss.

If this step fails

Queue capacity cannot be calculated.

Likely causeNo sustained/burst rate or maximum outage buffer was defined.

Safe checks
  • Measure representative intake rate.
  • Set accepted outage and disk budget.

ResolutionComplete capacity measurements and decide intentional shedding before deployment.

Security notes

  • Never use raw production payloads in planning documents.

Alternatives

  • Keep direct SDK-to-backend export until an owned Collector contract exists.

Stop conditions

  • Stop if any signal has unknown owner, sensitivity, backend, volume or accepted loss.
02

command

Verify the exact Collector distribution and components

read-only

Pin official Collector Contrib 0.157.0 by checksum or digest, record provenance and list compiled components. Review each component's stability for the selected signal and reject floating tags or a configuration copied from another distribution.

Why this step matters

Collector names are not a guarantee that a binary contains a component. Distribution and stability evidence prevent startup surprises and unsupported production dependencies.

What to understand

Retain release notes, checksum/digest, vulnerability disposition and component list with the change.

Minimize components where a custom distribution is owned; every parser and receiver expands input attack surface.

Validate on the same CPU/OS and service arguments used in production.

System changes

  • Records artifact identity; no service changes.

Syntax explained

components
Lists compiled factories, proving that configuration component types exist in this binary.
Command
otelcol-contrib --version && otelcol-contrib components
Example output / evidence
otelcol-contrib version 0.157.0
receivers: filelog hostmetrics otlp
processors: batch filter memory_limiter resource transform
exporters: otlp debug
extensions: file_storage health_check

Checkpoint: Verify this layer before continuing

Continue whenPinned artifact and every required component/stability level match review.

Stop whenStop on unverifiable artifact, missing/experimental unaccepted component or version mismatch.

If this step fails

Validation says `unknown type: filelog`.

Likely causeThe core distribution was installed instead of Contrib.

Safe checks
  • Run components on the service binary.
  • Inspect ExecStart absolute path.

ResolutionInstall the approved Contrib or minimized official build, then repeat validation.

Security notes

  • Do not fetch unofficial prebuilt Collectors or plugins on production hosts.

Alternatives

  • Use an owned OpenTelemetry Collector Builder distribution with a signed supply chain.

Stop conditions

  • Stop on unverifiable artifact, missing/experimental unaccepted component or version mismatch.
03

decision

Choose agent and gateway placement

read-only

Place host/file receivers in per-host agents and central policy/backend egress in at least two gateways. Keep host permissions out of gateways. Document gRPC connection behavior, load distribution and whether any stateful processor needs consistent routing.

Why this step matters

Placement determines trust, blast radius and correctness. Agents need local visibility; gateways need no host access and can centralize authentication and egress.

What to understand

Long-lived gRPC connections can distribute unevenly; measure per-replica intake rather than assuming round-robin fairness.

Tail sampling and some metric transforms are stateful and require routing related telemetry together.

Separate production and nonproduction identities, queues and backends even when binaries are shared.

System changes

  • Defines deployment boundaries; no runtime change.

Syntax explained

agent
Collector close to workload with narrow host-local access.
gateway
Redundant central Collector receiving from agents and authenticating to backends.
Example output / evidence
Agents: 48 (host-local OTLP, hostmetrics, approved filelog)
Gateways: 3 across failure domains
Stateful processors: none
Receiver exposure: loopback on agents, private TLS on gateways
Backend egress identities: dedicated per environment

Checkpoint: Verify this layer before continuing

Continue whenEvery receiver/processor has justified placement, private exposure and failure behavior.

Stop whenStop if gateways require broad host access, one gateway is a single point of failure, or stateful routing is undefined.

If this step fails

One gateway handles most traffic.

Likely causeLong-lived connections or load-balancer policy prevent even distribution.

Safe checks
  • Compare intake per replica.
  • Inspect client connection count and LB policy.

ResolutionIncrease connections or use supported load-balancing/sharding and prove distribution under load.

Security notes

  • Do not expose gateway OTLP publicly without authentication, quotas and abuse controls.

Alternatives

  • Use agent-only deployment for a small bounded environment without central stateful processing.

Stop conditions

  • Stop if gateways require broad host access, one gateway is a single point of failure, or stateful routing is undefined.
04

config

Configure least-privilege receivers and protected endpoints

caution

Bind agent OTLP to loopback where possible, gateway OTLP to private TLS, and health/internal metrics to protected management addresses. Enable hostmetrics and filelog only on agents with explicit paths and service-account permissions. Keep pprof, zPages and detailed payload debug disabled.

Why this step matters

Receivers parse untrusted network/file input and host telemetry reveals sensitive topology. Narrow bindings, paths and privileges reduce attack surface before processors run.

What to understand

Use a dedicated service account and explicit file ACL/group. Never grant root or Docker socket access just to discover containers.

Protect health and metrics because they reveal topology and operational state, even when payloads are absent.

Test log rotation and receiver state with synthetic files; exclude archives and unrelated directories.

System changes

  • Opens local/private listeners and reads approved host/file sources; incorrect scope can expose or collect sensitive data.

Syntax explained

endpoint: 127.0.0.1
Prevents host-local agent receivers and admin endpoints from accepting remote traffic.
storage: file_storage
Persists filelog read state across restart under the protected storage directory.
File /etc/otelcol-contrib/config.yaml
Configuration
Fill variables0/2 ready

Values stay on this page and are never sent or saved.

extensions:
  health_check:
    endpoint: 127.0.0.1:13133
  file_storage:
    directory: {{otelStorage}}
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317
      http:
        endpoint: 127.0.0.1:4318
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu: {}
      memory: {}
      filesystem: {}
  filelog:
    include: ["{{otelLogPath}}"]
    storage: file_storage
Example output / evidence
OTLP gRPC: 127.0.0.1:4317 (agent)
OTLP HTTP: 127.0.0.1:4318 (agent)
Health: 127.0.0.1:13133
Metrics: 127.0.0.1:8888
filelog include targets: 3
Denied unrelated file read: PASS

Checkpoint: Verify this layer before continuing

Continue whenOnly intended listeners and paths are accessible; service runs non-root and survives a rotation without gaps/duplicates beyond policy.

Stop whenStop if a debug/admin endpoint is public, unrelated file is readable, receiver is unauthenticated across a network or state is ephemeral.

If this step fails

Collector returns permission denied for approved logs.

Likely causeService account lacks the narrow group/ACL or path traversal permission.

Safe checks
  • Test read access as the service account.
  • Inspect directory/file ACLs.

ResolutionGrant only the specific read/search access and repeat a synthetic rotation test.

Security notes

  • Host telemetry and filenames can contain sensitive identifiers; minimize attributes before export.

Alternatives

  • Instrument applications directly and omit filelog where structured OTLP logs are available.

Stop conditions

  • Stop if a debug/admin endpoint is public, unrelated file is readable, receiver is unauthenticated across a network or state is ephemeral.
05

config

Put memory protection, redaction and resource policy in order

caution

Configure memory_limiter as the first processor in every pipeline. Then drop prohibited telemetry, redact or hash approved attributes with tested OTTL, normalize resource identity without overwriting trusted SDK values, and place batch last. Set an explicit transform error policy and alert on errors.

Why this step matters

Processor order affects safety. Memory protection must run before buffers, and privacy policy must run before every exporter. `insert` preserves a trusted existing identity while adding a fallback.

What to understand

Size hard limit below cgroup/service memory and reserve spike headroom; test refusal rather than relying on defaults.

Use synthetic fixtures for each signal, encoding and missing/type variant. Propagating a redaction error is safer than silently exporting raw data.

Batch timeout trades latency for efficiency and must be tested with low-volume tenants as well as bursts.

System changes

  • Changes drop/redaction, resource attributes, memory refusal and batching for every pipeline.

Syntax explained

error_mode: propagate
Fails the affected transform path instead of silently ignoring a privacy-policy error.
action: insert
Adds fallback environment only when absent, avoiding accidental overwrite.
memory_limiter first
Refuses load before transform/batch allocations can exceed the process budget.
File /etc/otelcol-contrib/config.yaml
Configuration
Fill variables0/2 ready

Values stay on this page and are never sent or saved.

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: {{otelMemoryLimit}}
    spike_limit_mib: {{otelMemorySpike}}
  transform/privacy:
    error_mode: propagate
    trace_statements:
      - context: span
        statements:
          - delete_key(attributes, "enduser.email")
          - delete_key(attributes, "http.request.header.authorization")
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "enduser.email")
          - delete_key(attributes, "http.request.header.authorization")
    log_statements:
      - context: log
        statements:
          - replace_pattern(body, "(?i)(authorization|token|password)=([^ ]+)", "$1=[REDACTED]")
          - delete_key(attributes, "enduser.email")
          - delete_key(attributes, "http.request.header.authorization")
  resource/identity:
    attributes:
      - key: deployment.environment.name
        value: production
        action: insert
  batch:
    timeout: 5s
    send_batch_size: 1024
Command
otelcol-contrib validate --config=/etc/otelcol-contrib/config.yaml
Example output / evidence
Config is valid

Checkpoint: Verify this layer before continuing

Continue whenExact binary validates; synthetic secrets disappear, stable identity remains and burst testing stays below memory limits.

Stop whenStop if any pipeline bypasses privacy, ignores transform errors, puts limiter after batch or OOMs under accepted burst.

If this step fails

A synthetic email attribute remains in exported traces.

Likely causeThe transform statement only targets logs or the trace pipeline omits the processor.

Safe checks
  • Inspect every service pipeline.
  • Send one synthetic fixture per signal.

ResolutionAdd signal-appropriate transform/filter policy before its exporter and repeat negative tests.

Security notes

  • Never validate redaction by sending real credentials or personal data.

Alternatives

  • Drop entire high-risk bodies rather than regex-redacting when reliable minimization is impossible.

Stop conditions

  • Stop if any pipeline bypasses privacy, ignores transform errors, puts limiter after batch or OOMs under accepted burst.
06

config

Configure TLS, workload authentication and secret delivery

caution

Use trusted CA validation and a dedicated client certificate or approved authenticator for each environment. Reference protected key files, require server SAN validation and deny unauthorized identities. Keep secrets out of YAML, URLs, argv, diagnostics and environment dumps.

Why this step matters

Encryption without endpoint identity permits interception, and a shared token prevents tenant attribution. A dedicated workload identity provides revocation and least privilege.

What to understand

Validate chain, SAN, expiry and key match. Keep CA and client key readable only by the Collector account.

Stage rotation with an overlap window and confirm whether the exporter reloads files or requires restart.

Send a synthetic request with an unauthorized identity and prove backend denial/audit.

System changes

  • Installs trust and workload identity; mistakes can interrupt all export or disclose telemetry.

Syntax explained

-servername
Sends the expected TLS SNI so the server selects the intended certificate.
-verify_hostname
Verifies that the certificate identity matches the expected backend DNS name instead of checking only the CA chain.
-verify_return_error
Stops the handshake test on a certificate verification error instead of continuing to display diagnostic output.
-CAfile
Uses the approved trust anchor instead of disabling verification.
Configuration
Fill variables0/2 ready

Values stay on this page and are never sent or saved.

openssl s_client -connect {{otelBackend}} -servername telemetry-backend.internal.example -verify_hostname telemetry-backend.internal.example -verify_return_error -CAfile {{otelCa}} </dev/null
Example output / evidence
subject=CN=telemetry-backend.internal.example
issuer=CN=Example Internal Telemetry CA
Verification: OK
Verify return code: 0 (ok)

Checkpoint: Verify this layer before continuing

Continue whenChain and SAN validate, Collector identity has only intended tenant/write permission and unauthorized identity is denied.

Stop whenStop on expired/mismatched certificate, world-readable key, shared admin token, plaintext endpoint or disabled verification.

If this step fails

Handshake succeeds with `-k` but normal export fails.

Likely causeCertificate chain or SAN is wrong; insecure mode masks the defect.

Safe checks
  • Validate full chain and SAN.
  • Check system time and endpoint DNS.

ResolutionIssue/install a correct certificate and trust bundle; never retain insecure verification.

Security notes

  • Do not paste private keys or bearer values into test commands or support tickets.

Alternatives

  • Use provider workload identity with short-lived credentials instead of static keys.

Stop conditions

  • Stop on expired/mismatched certificate, world-readable key, shared admin token, plaintext endpoint or disabled verification.
07

config

Bound persistent queues, retry and batch behavior

caution

Enable exporter sending_queue with finite capacity and file_storage, and configure retry with a maximum elapsed time that matches accepted outage buffering. Put storage on a quota-controlled service-owned filesystem. Calculate capacity from measured item size/rate and prove disk/memory behavior under a backend outage.

Why this step matters

Queues and retry transform a downstream incident into local resource pressure. Finite capacity and an intentional horizon make loss, disk use and recovery predictable.

What to understand

Measure serialized batch size; queue_size units are not necessarily individual spans or bytes.

Use a dedicated quota and alerts before the host filesystem fills. Protect persistent state because it contains telemetry.

At-least-once replay can duplicate acknowledged-looking data around failure; verify downstream tolerance.

System changes

  • Writes telemetry to protected local queue files and retries backend export for a bounded period.

Syntax explained

storage: file_storage
Persists queued export requests across Collector restart.
queue_size
Bounds queued work units; size from measured request bytes rather than assuming it is bytes.
max_elapsed_time
Caps how long failed data is retried before explicit loss policy applies.
File /etc/otelcol-contrib/config.yaml
Configuration
Fill variables0/5 ready

Values stay on this page and are never sent or saved.

exporters:
  otlp/backend:
    endpoint: {{otelBackend}}
    tls:
      ca_file: {{otelCa}}
      cert_file: {{otelCert}}
      key_file: {{otelKey}}
    sending_queue:
      enabled: true
      storage: file_storage
      queue_size: {{otelQueueSize}}
    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 30s
      max_elapsed_time: 30m
Example output / evidence
Queue capacity: 10000 batches
Persistent storage quota: 20 GiB
Maximum retry elapsed time: 30m
Representative outage test: 15m
Peak queue utilization: 48%
Peak filesystem utilization: 36%
Drain time after recovery: 06m12s
Dropped telemetry: 0

Checkpoint: Verify this layer before continuing

Continue whenOutage test stays within memory/disk, queue persists restart, drains in SLO and duplicate/loss behavior is recorded.

Stop whenStop if queue/retry is unbounded, shares root without quota, contains unprotected data, or cannot drain faster than sustained intake.

If this step fails

Queue reaches 100% and export failures continue.

Likely causeBackend outage exceeded design or recovery throughput is below input.

Safe checks
  • Measure backend acceptance and queue growth.
  • Check disk quota and retry age.

ResolutionRestore/scale backend, shed documented noncritical telemetry and revise finite capacity; never remove disk safeguards.

Security notes

  • Queue files contain telemetry and need service-only permissions, encryption-at-rest where required and secure cleanup.

Alternatives

  • Use memory-only queue when restart survival is not required and disk privacy risk outweighs the bounded benefit.

Stop conditions

  • Stop if queue/retry is unbounded, shares root without quota, contains unprotected data, or cannot drain faster than sustained intake.
08

config

Assemble signal pipelines without bypasses

caution

Add receivers, processors and exporter to traces, metrics and logs explicitly. Keep memory_limiter first, privacy/resource processors next and batch last. Ensure no production signal reaches debug or an unauthenticated alternate exporter. Validate the rendered configuration and component graph.

Why this step matters

Collector component definitions are inert until referenced by a pipeline. Explicit graph review catches missing signals, bypass exporters and unsafe processor order.

What to understand

Render includes/templating before validation; validate what the service consumes, not a source fragment.

Metrics may not need body redaction but still require attribute policy. Maintain signal-specific fixtures.

Keep the production exporter set small and named for its authorization boundary.

System changes

  • Activates all three signal pipelines and their export behavior.

Syntax explained

service.pipelines
Connects configured components in actual execution order for each signal.
exporters: [otlp/backend]
Ensures production data has only the reviewed authenticated destination.
File /etc/otelcol-contrib/config.yaml
Configuration
service:
  extensions: [health_check, file_storage]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, transform/privacy, resource/identity, batch]
      exporters: [otlp/backend]
    metrics:
      receivers: [otlp, hostmetrics]
      processors: [memory_limiter, transform/privacy, resource/identity, batch]
      exporters: [otlp/backend]
    logs:
      receivers: [otlp, filelog]
      processors: [memory_limiter, transform/privacy, resource/identity, batch]
      exporters: [otlp/backend]
Command
otelcol-contrib validate --config=/etc/otelcol-contrib/config.yaml
Example output / evidence
Config is valid
pipelines:
  traces: otlp -> memory_limiter -> transform/privacy -> resource/identity -> batch -> otlp/backend
  metrics: otlp,hostmetrics -> memory_limiter -> resource/identity -> batch -> otlp/backend
  logs: otlp,filelog -> memory_limiter -> transform/privacy -> resource/identity -> batch -> otlp/backend

Checkpoint: Verify this layer before continuing

Continue whenRendered graph contains every intended signal, correct processor order and no bypass/debug exporter.

Stop whenStop if a signal omits protection, a debug exporter exists, rendered secrets appear or validation differs from service config.

If this step fails

Logs arrive unredacted while traces are correct.

Likely causeLogs pipeline omitted transform/privacy or points to another exporter.

Safe checks
  • Inspect rendered pipeline graph.
  • Compare per-signal synthetic fixtures.

ResolutionCorrect the logs path and repeat validation before reload.

Security notes

  • Treat rendered configuration as sensitive even when secret values are referenced externally.

Alternatives

  • Separate configurations per role if one combined file makes review ambiguous.

Stop conditions

  • Stop if a signal omits protection, a debug exporter exists, rendered secrets appear or validation differs from service config.
09

verification

Canary traces, metrics and logs end to end

read-only

Use telemetrygen or an equivalent pinned official generator to send one synthetic trace, metric and log containing stable resource identity and fake sensitive attributes. Query the backend for each signal, confirm redaction and latency, and verify an unauthorized client certificate is rejected.

Why this step matters

A config-valid Collector can still have broken trust, routing, processing or backend mapping. Signal canaries prove the complete contract and negative authorization.

What to understand

Use synthetic identifiers and a dedicated test tenant/namespace so results can be removed by policy.

Assert field values and absence, not just counts. Verify resource identity and timestamps.

Record intake-to-query latency, Collector revision and backend tenant for regression comparison.

System changes

  • Sends a small synthetic telemetry fixture and queries the backend.

Syntax explained

--otlp-insecure=false
Requires TLS; trust material is supplied through the approved generator mechanism.
--traces/--logs
Bounds synthetic event count for deterministic verification.
Command
Fill variables0/1 ready

Values stay on this page and are never sent or saved.

telemetrygen traces --otlp-endpoint {{otelGateway}} --otlp-insecure=false --traces 1 && telemetrygen metrics --otlp-endpoint {{otelGateway}} --otlp-insecure=false --duration 5s && telemetrygen logs --otlp-endpoint {{otelGateway}} --otlp-insecure=false --logs 1
Example output / evidence
traces generated: 1; backend received: 1; latency: 0.84s
metric points generated: 5; backend received: 5; latency: 1.12s
logs generated: 1; backend received: 1; latency: 0.91s
synthetic sensitive attributes exported: 0
unauthorized certificate: denied

Checkpoint: Verify this layer before continuing

Continue whenAll signals arrive once within SLO, sensitive fixtures are absent, identity is correct and unauthorized export fails.

Stop whenStop on missing/duplicate beyond policy, raw sensitive fixture, wrong tenant/resource or insecure connection.

If this step fails

Trace arrives but its service.name is `unknown_service`.

Likely causeInstrumentation omitted identity and fallback processor did not run or used the wrong action.

Safe checks
  • Inspect synthetic source resource.
  • Trace the resource processor output in isolation.

ResolutionSet source identity, add a bounded insert fallback and repeat all-signal canary.

Security notes

  • Never use real user data or credentials in telemetrygen fixtures.

Alternatives

  • Use backend-native synthetic checks when they exercise the same authenticated pipeline.

Stop conditions

  • Stop on missing/duplicate beyond policy, raw sensitive fixture, wrong tenant/resource or insecure connection.
10

verification

Exercise backpressure, restart and credential rotation

danger

In an isolated representative environment, make the backend unavailable, generate the accepted burst, observe limiter and queue behavior, restart the Collector, restore the backend and measure drain. Rotate a test certificate atomically and verify old identity revocation without losing the only recovery path.

Why this step matters

Failure testing reveals whether protective settings work before a real outage. Restart/replay and rotation are precisely where duplicate, loss and trust assumptions fail.

What to understand

Use synthetic telemetry and an isolated backend route. Never intentionally block a production backend or fill a production queue.

Observe RSS, refusal, queue/disk, retries and drain rate. Confirm alert thresholds fire before exhaustion.

Rotate with overlap and a rollback certificate, then revoke the old test identity after the new path is proven.

System changes

  • Temporarily interrupts test export, fills/replays a persistent queue, restarts a test Collector and changes a test identity.

Syntax explained

controlled backend outage
Tests bounded resilience without modifying production routing.
atomic certificate rotation
Prevents partial key/certificate reads and preserves a known-good rollback.
Example output / evidence
Backend outage: 15m00s
Peak RSS: 612 MiB / 768 MiB limiter
Peak queue: 4820 / 10000
Queue persisted restart: PASS
Drain after backend recovery: 06m12s
Expected bounded duplicates: 3
Unexpected drops: 0
New certificate export: PASS
Revoked certificate: DENIED

Checkpoint: Verify this layer before continuing

Continue whenMemory/disk remain bounded, queue persists and drains, loss/duplicates fit policy, alerts fire and rotated identity works.

Stop whenStop if target is production, disk/RSS approaches hard safety limit, queue cannot drain, secrets appear or rollback identity is unavailable.

If this step fails

Collector restarts but queue directory is empty.

Likely causefile_storage path is ephemeral, wrong, unmounted or unreadable by service account.

Safe checks
  • Inspect mount/path and ownership.
  • Compare pre/post queue metrics and files.

ResolutionFix protected persistent storage and repeat the isolated outage; update RPO claim if persistence is intentionally omitted.

Security notes

  • Persistent test data still follows telemetry retention and secure cleanup policy.

Alternatives

  • Use a disposable namespace/host and a mock OTLP backend to test failure behavior.

Stop conditions

  • Stop if target is production, disk/RSS approaches hard safety limit, queue cannot drain, secrets appear or rollback identity is unavailable.
11

warning

Operate self-telemetry, rollback and recovery evidence

caution

Scrape protected Collector internal metrics and alert on receive refusal, processing errors, queue utilization/age, export failures, retry, RSS, CPU and disk. Back up secret-free rendered configuration and manifests. Rehearse restoring the previous revision and bounded queue in isolation, documenting duplicates and the fact that backend recovery is separate.

Why this step matters

Process liveness is insufficient: a healthy Collector can accept and indefinitely queue or drop telemetry. Self-telemetry plus canaries establishes actionable health, while recovery evidence prevents local buffering from being mislabeled a backup.

What to understand

Protect the metrics endpoint and avoid labels containing credentials or uncontrolled resource values.

Alert on sustained rates and capacity percentages with runbooks that distinguish source, processing, queue and backend failure.

Store signed secret-free configuration and artifact identity; restore against synthetic telemetry and a disposable backend.

System changes

  • Adds internal metric scraping/alerts and backup artifacts; it does not back up downstream telemetry.

Syntax explained

curl --fail --silent
Fails on HTTP errors and avoids progress noise while reading a loopback endpoint.
grep -E
Selects key Collector metric families for a concise smoke check; monitoring should scrape the full approved endpoint.
Command
curl --fail --silent http://127.0.0.1:8888/metrics | grep -E 'otelcol_(receiver|processor|exporter|process)' | head
Example output / evidence
otelcol_receiver_accepted_spans_total{receiver="otlp"} 42018
otelcol_receiver_refused_spans_total{receiver="otlp"} 0
otelcol_exporter_sent_spans_total{exporter="otlp/backend"} 42018
otelcol_exporter_send_failed_spans_total{exporter="otlp/backend"} 0
otelcol_process_memory_rss_bytes 641728512

Checkpoint: Verify this layer before continuing

Continue whenAlerts distinguish receive/process/export pressure, config rollback passes, and recovery documentation states exact queue/backend boundaries.

Stop whenStop if internal telemetry is public, alerts include secrets, dashboards hide drops, or local queue is represented as durable backup.

If this step fails

Health endpoint is green while exporter failures grow.

Likely causeLiveness does not evaluate pipeline delivery.

Safe checks
  • Inspect exporter failure and queue metrics.
  • Run end-to-end canary.

ResolutionKeep liveness simple but page on sustained delivery/queue failure and canary breach.

Security notes

  • Restrict self-telemetry to operators; it reveals system topology and failure state.

Alternatives

  • Export Collector self-telemetry through a separate minimal path so the primary outage remains visible.

Stop conditions

  • Stop if internal telemetry is public, alerts include secrets, dashboards hide drops, or local queue is represented as durable backup.

Finish line

Verification checklist

Configuration graphotelcol-contrib validate --config=/etc/otelcol-contrib/config.yamlExact 0.157.0 distribution validates every component and pipeline with memory limiter first and no debug bypass.
All-signal canarytelemetrygen traces --otlp-endpoint {{otelGateway}} --traces 1Trace, metric and log fixtures reach the correct backend tenant within SLO with stable identity and redaction.
TLS authorizationopenssl s_client -connect {{otelBackend}} -servername telemetry-backend.internal.example -CAfile {{otelCa}} </dev/nullChain and SAN validate, approved Collector identity succeeds and unauthorized identity is denied.
Backpressurecurl --fail --silent http://127.0.0.1:8888/metricsAccepted burst and backend outage stay within memory/disk; queue persists, drains and reports bounded loss/duplicates.
Operational healthcurl --fail --silent http://127.0.0.1:13133/Liveness is healthy and separate delivery metrics/canary prove exporters are succeeding with queue headroom.

Recovery guidance

Common problems and safe checks

The Collector exits with `unknown type` for a configured component.

Likely causeThe configuration was written for a component that is not included in the selected core or contrib distribution, or the component name changed between releases.

Safe checks
  • Run the exact distribution's `components` command and configuration validation.
  • Compare component stability and configuration against the official registry for the pinned version.

ResolutionUse a distribution containing only reviewed components or build a pinned custom distribution. Do not download arbitrary third-party binaries to satisfy the configuration.

Applications receive `UNAVAILABLE`, HTTP 503 or connection refused on OTLP.

Likely causeThe agent is down, the receiver binds another interface, TLS trust fails, a firewall blocks the endpoint or memory protection is refusing load.

Safe checks
  • Check service health and the bound 4317/4318 sockets locally.
  • Inspect receiver refusal and memory-limiter metrics without enabling payload logging.

ResolutionRepair health, bind and trust configuration; scale or shed noncritical telemetry when memory pressure is intentional. Never disable TLS verification on a network receiver.

The Collector is killed by the OOM killer.

Likely causeMemory limiter is absent, placed after buffering processors, sized above the container limit, or persistent/export queues ingest faster than the backend can drain.

Safe checks
  • Inspect cgroup limits, process RSS, refused telemetry, queue size and backend latency before restart.
  • Confirm memory_limiter is the first processor in every pipeline.

ResolutionLower limiter thresholds below the runtime limit, bound queues and batches, restore export throughput and load-test a representative burst.

Persistent queue files grow until disk is full.

Likely causeThe backend outage exceeds queue capacity, storage shares the root filesystem, retry never gives up or downstream capacity is lower than sustained input.

Safe checks
  • Inspect exporter queue length/capacity, retry age and file_storage directory usage.
  • Measure backend acceptance rate and host free space/inodes.

ResolutionRestore the backend, enforce a finite queue and retry horizon based on RPO, isolate storage with a quota, and document which telemetry is dropped first.

Telemetry is duplicated after restart or failover.

Likely causePersistent queues provide at-least-once delivery and replay items whose acknowledgement was not durably observed.

Safe checks
  • Compare event identifiers/timestamps around the restart without adding identifiers as high-cardinality metric labels.
  • Inspect exporter retry and queue replay metrics.

ResolutionMake the backend and queries tolerant of bounded duplicates, preserve stable trace/span IDs, and do not promise exactly-once delivery.

Traces arrive but metrics or logs do not.

Likely causeThe signal lacks a service pipeline, receiver/exporter is omitted from that pipeline, or a filter/transform expression drops it.

Safe checks
  • Inspect the rendered service.pipelines section and per-signal accepted/refused/exported metrics.
  • Send one synthetic event of each signal to an isolated validation receiver.

ResolutionAdd the missing pipeline or correct processor logic, validate, then canary all three signals before production reload.

Backend rejects telemetry with authentication or authorization errors.

Likely causeClient certificate, CA, token or workload identity is expired, mapped to the wrong tenant, or absent from the exporter.

Safe checks
  • Validate certificate chain, SAN and expiry without printing private keys.
  • Inspect sanitized backend audit decisions and exporter response metrics.

ResolutionRestore the approved short-lived identity, rotate compromised material and test an unauthorized identity is denied. Do not use a shared administrator token.

Sensitive attributes remain after redaction.

Likely causeThe transform targets the wrong signal path, a new attribute key or encoding bypasses it, or a debug exporter emits before the processor.

Safe checks
  • Use synthetic plain, URL-encoded and multiline fixtures through every pipeline.
  • Inspect processor ordering and ensure no exporter bypasses redaction.

ResolutionStop the affected pipeline, revoke leaked credentials if applicable, correct source instrumentation and pre-export transform/filter logic, then retest synthetic fixtures.

The transform processor reports statement errors.

Likely causeOTTL syntax, context or field type does not match the pinned component version, or error_mode is propagating unexpected failures.

Safe checks
  • Validate the config with the exact binary and run a synthetic fixture.
  • Inspect processor error metrics and a sanitized error message.

ResolutionCorrect the OTTL statement and choose an explicit reviewed error_mode; never silently ignore redaction failures in production.

Host metrics have duplicate or unstable resource identities.

Likely causeAgents and gateways both overwrite resource attributes, hostname discovery varies, or ephemeral IDs are used as canonical host identity.

Safe checks
  • Compare resource attributes for one host across signals and restarts.
  • Inspect resource processor precedence in agent and gateway.

ResolutionAssign stable host/service/environment identity once, preserve SDK attributes intentionally, and delete only explicitly forbidden keys.

Batch latency exceeds the telemetry SLO.

Likely causeTimeout or batch size is too high for a low-volume tenant, queue congestion adds delay, or exporter connections are saturated.

Safe checks
  • Compare receive-to-export timestamps, batch send reason and queue latency.
  • Test representative low and high volume separately.

ResolutionTune batch timeout/size from measured volume, shard gateways by stable tenant key where required, and add backend/export capacity.

CPU rises sharply after enabling detailed logs or transforms.

Likely causeComplex regex/OTTL runs for every record, debug verbosity serializes full payloads, or filelog parses large multiline records.

Safe checks
  • Profile only in an isolated environment and compare processor CPU metrics by configuration revision.
  • Measure input size, record count and multiline limits.

ResolutionMove filtering earlier, simplify bounded expressions, cap record size and disable detailed debug export outside synthetic tests.

Filelog re-ingests old files or misses rotations.

Likely causeFile storage is not persistent, file fingerprints change, include/exclude patterns overlap or the rotation strategy is incompatible.

Safe checks
  • Inspect filelog storage extension state, file ownership and rotated filenames.
  • Run a controlled rotation with synthetic lines and count received records.

ResolutionPersist receiver state, correct include/exclude patterns and validate the application's actual rotation semantics before rollout.

Health endpoint reports healthy while telemetry is not reaching the backend.

Likely causeProcess liveness only proves the Collector is running, not that queues drain or exporters succeed.

Safe checks
  • Inspect exporter sent/failed and queue utilization metrics.
  • Use an end-to-end synthetic signal and backend query.

ResolutionKeep liveness simple but add readiness/operational alerts based on exporter and queue health plus signal canaries.

A gateway replica receives an uneven share and queues while others are idle.

Likely causeLoad balancing, sticky connections, gRPC connection longevity or tenant distribution concentrates traffic.

Safe checks
  • Compare accepted spans and queue utilization by replica.
  • Inspect client connection distribution and load-balancer algorithm.

ResolutionUse supported client/load-balancing behavior, additional connections or stable sharding while preserving trace/metric processing requirements.

Tail sampling or cumulative metric processing produces incorrect results.

Likely causeRelated telemetry is split across gateways without consistent routing, or stateful processors restart without an accepted loss model.

Safe checks
  • Inspect processor documentation/stability and routing key distribution.
  • Compare synthetic complete traces or cumulative series across replicas.

ResolutionRoute related data consistently, use supported load-balancing connectors, or remove the stateful processor until correctness is proved.

Collector internal metrics expose unexpected labels or secrets.

Likely causeResource attributes were promoted into self-telemetry, endpoint URLs contain credentials, or debug logs include headers.

Safe checks
  • Inspect internal metrics label names and sanitized logs.
  • Review exporter endpoint and authentication configuration references.

ResolutionRemove credentials from URLs, restrict internal telemetry endpoints and sanitize labels/logs before returning service.

TLS handshake fails only after certificate rotation.

Likely causeCollector did not reload files, chain order/SAN is wrong, CA bundle was replaced non-atomically or clock skew invalidates the new certificate.

Safe checks
  • Validate chain, SAN, key match, file permissions and system time.
  • Check whether the selected component supports dynamic certificate reload.

ResolutionRestore the prior valid certificate atomically, validate the new bundle offline and use a staged restart/reload procedure.

The backend shows unknown or conflicting service names.

Likely causeSDKs omit `service.name`, the resource processor inserts a generic value, or gateway normalization overwrites a correct value.

Safe checks
  • Trace one synthetic resource from SDK through agent and gateway.
  • Compare insert/upsert/delete actions in resource processors.

ResolutionRequire explicit service identity at instrumentation, add a bounded fallback only where absent, and never overwrite a trusted value accidentally.

No telemetry survives a full host loss despite persistent queueing.

Likely causeThe queue is local durability, not a backup; the host and queue filesystem share the same failure domain.

Safe checks
  • Review queue storage placement and RPO claims.
  • Confirm the downstream telemetry backend's own replication and retention.

ResolutionCorrect the architecture statement: persistent queues bridge bounded outages but do not replace backend durability. Use a durable backend and accept/document local loss on host failure.

After the procedure

Alternatives and next steps

Consider these alternatives

  • Run a gateway-only topology for applications that can securely export directly and do not need host-local collection, accepting the larger failure and network boundary.
  • Run agents only for a small environment with one backend, while retaining identity, bounded queues and per-agent operations.
  • Use a vendor-supported Collector distribution when component compatibility and support ownership are more important than a custom minimized binary.
  • Use Grafana Alloy when one collector must combine supported Prometheus/Loki and OpenTelemetry pipelines under its component model.

Operate it safely

  • Build alerts for refusal, dropped/export-failed items, queue utilization/age, retry duration, process memory/CPU, disk quota and synthetic signal latency.
  • Load-test normal, burst and backend-outage scenarios for each signal, then document the exact loss/duplicate behavior and tune finite limits.
  • Automate signed configuration validation and canary rollout across agents and gateways with a quick rollback to the previous rendered revision.
  • Review component stability and release notes every 90 days, especially before adopting stateful processors, new receivers or experimental authentication extensions.

Reference

Frequently asked questions

Does a persistent queue guarantee no telemetry loss?

No. It can survive a Collector restart and bounded backend outage, but host/disk loss destroys local state and replay can duplicate data. Durable backends and explicit RPO remain necessary.

Why must memory_limiter be first?

It must refuse load before batching, transformation or other buffering consumes the memory it is intended to protect.

Can the debug exporter stay enabled at basic verbosity?

Production policy should avoid payload exporters unless a bounded synthetic-only path is proven. Debug configuration can still expose topology or record content and creates operational temptation.

Recovery

Rollback

Do not roll back by disabling TLS, authentication, redaction, memory limits or finite queues. Preserve queue files and internal telemetry, stop only the failing rollout, restore the previous signed configuration/artifact, and accept that replay may duplicate data.

  1. Stop new configuration rollout while keeping unaffected Collectors running; capture component, queue, retry, RSS and disk metrics.
  2. Validate the previous rendered configuration with the exact pinned binary and deploy it to one isolated/canary Collector.
  3. Preserve persistent queue ownership and path; never delete queued telemetry merely to make the service start without a documented loss decision.
  4. If privacy failed, stop affected export, rotate exposed credentials, correct source instrumentation and transforms, then rerun synthetic negative tests.
  5. If identity rotation failed, restore the prior still-valid protected certificate, verify SAN/chain and stage a corrected rotation.
  6. Resume rollout only after all-signal canaries, unauthorized denial, bounded memory/queue behavior and backend delivery alerts pass.

Evidence

Sources and review

Verified 2026-07-24Review due 2026-10-22
OpenTelemetry Collector overviewofficialOpenTelemetry Collector deployment patternsofficialOpenTelemetry Collector configurationofficialOpenTelemetry Collector scalingofficialOpenTelemetry Collector resiliencyofficialOpenTelemetry Collector securityofficialOpenTelemetry Collector transform telemetryofficialOpenTelemetry Collector troubleshootingofficial