OneLinersCommand workbench
AI
Back to skills
SKILL.md

Distributed log correlation investigator

An evidence-first workflow to reconstruct one request or failure across services without retaining sensitive payloads.

Revision
2
Verified
2026-10-20
Save or explore
Save to collectionCreate a collection in the sidebar first.

Compatibility and paths

Codexskills/distributed-log-timeline/SKILL.md
Claude Code.claude/skills/distributed-log-timeline/SKILL.md
VS Code.github/skills/distributed-log-timeline/SKILL.md

Trust and provenance

Curated record reviewed 2026-10-20. Results still depend on the supplied context and target environment.

Generated assetReady to copy or download
---
name: distributed-log-timeline
description: Correlates one bounded request using exact trace/span context while preserving clock uncertainty, missing spans and privacy. Use when cross-service logs appear out of order; return a partial-order timeline and one discriminating query, not invented causality.
license: CC-BY-4.0
compatibility: Reviews bounded redacted evidence offline. Production queries, paging, configuration changes and mitigation require separately authorized targets and operators.
metadata:
  author: oneliners
  version: "2.0.0"
---

# Distributed log correlation investigator

## Select one request, not a whole customer history
Ask for exact environment, trace/request identifier, collection window, services and authorized metadata fields. Establish identifier namespace and propagation rules; two systems may reuse short request IDs. Keep raw evidence in its authorized store and produce a redacted ledger with stable evidence IDs. Do not join tenants or unrelated traces because a message or business label looks similar. If there is no reliable join key, label the association hypothetical.

## Keep clocks and relationships separate
Preserve event/source timestamp, collector observed timestamp, time zone, duration provenance, local sequence and trace/span IDs separately. An event's source clock can be wrong; collection can also be delayed. Neither a global timestamp sort nor collector arrival alone establishes causality. Use explicit parent relationships and documented in-process sequence where supplied, retaining uncertainty between services. Local durations require an appropriate clock/source; do not infer them from unrelated wall clocks or add nested span durations as if they were disjoint work.

## Worked skewed trace
This is a readable synthetic JSON export. The ISO fields are not the numeric nanosecond OTLP encoding. parentSpanId/localSequence/durationMs are supplied enrichment for this case, not assertions that every log record has those fields.

```json
{
  "provenance": "synthetic JSON view, not native OTLP wire data; ISO timestamps for human review",
  "targetTraceId": "11111111111111111111111111111111",
  "clockEvidence": "payments source clock offset unknown; collector arrival delays also unknown",
  "events": [
    {
      "id": "C1",
      "traceId": "11111111111111111111111111111111",
      "spanId": "a1a1a1a1a1a1a1a1",
      "service": "checkout",
      "timestamp": "2026-10-20T10:00:00.000Z",
      "observedTimestamp": "2026-10-20T10:00:00.020Z",
      "event": "request.start",
      "localSequence": 1
    },
    {
      "id": "P1",
      "traceId": "11111111111111111111111111111111",
      "spanId": "b2b2b2b2b2b2b2b2",
      "parentSpanId": "a1a1a1a1a1a1a1a1",
      "service": "payments",
      "timestamp": "2026-10-20T10:00:08.050Z",
      "observedTimestamp": "2026-10-20T10:00:00.090Z",
      "event": "upstream.call.start",
      "localSequence": 1
    },
    {
      "id": "P2",
      "traceId": "11111111111111111111111111111111",
      "spanId": "b2b2b2b2b2b2b2b2",
      "parentSpanId": "a1a1a1a1a1a1a1a1",
      "service": "payments",
      "timestamp": "2026-10-20T10:00:08.170Z",
      "observedTimestamp": "2026-10-20T10:00:00.210Z",
      "event": "upstream.call.error",
      "localSequence": 2,
      "durationMs": 120,
      "reason": "deadline"
    },
    {
      "id": "C2",
      "traceId": "11111111111111111111111111111111",
      "spanId": "a1a1a1a1a1a1a1a1",
      "service": "checkout",
      "timestamp": "2026-10-20T10:00:00.400Z",
      "observedTimestamp": "2026-10-20T10:00:00.450Z",
      "event": "request.end",
      "localSequence": 2,
      "durationMs": 400,
      "status": 504
    },
    {
      "id": "X1",
      "traceId": "22222222222222222222222222222222",
      "spanId": "c3c3c3c3c3c3c3c3",
      "service": "payments",
      "timestamp": "2026-10-20T10:00:00.100Z",
      "observedTimestamp": "2026-10-20T10:00:00.120Z",
      "event": "request.end",
      "localSequence": 1,
      "status": 200,
      "correlationLabel": "same-checkout-label-does-not-prove-same-trace"
    }
  ]
}
```

## Correlated result and bounded next query
Selection: C1,P1,P2,C2 have exact target trace11111111111111111111111111111111. Exclude X1: a coincident time or business label does not override a different trace ID. Supplied parent context links checkout span a1a1a1a1a1a1a1a1 to payments span b2b2b2b2b2b2b2b2. parentSpanId is supplied span-context enrichment here, not a mandatory top-level OpenTelemetry log field.

| Evidence | What can be ordered | What must stay uncertain |
| --- | --- | --- |
| C1 -> C2 | Same checkout span and supplied localSequence1→2; supplied local duration400ms, end504 | Which portion was network, queue or application work is not given. |
| P1 -> P2 | Same payments span and localSequence1→2; supplied local duration120ms, deadline result | The destination's own server trace and exact timeout budget are missing. |
| checkout -> payments | Supplied parent-span relationship establishes the request relationship | Parent/child alone does not prove every cross-service log ordering or a complete critical path. |
| payments wall time later than collector observation | P1/P2 source timestamps are about8s ahead of collector values | Clock offset and collection delay are unmeasured; this is not an8-second network call or permission to rewrite original timestamps. |

Use a partial-order view plus a separately labeled collector-arrival listing. Do not globally sort source timestamps and conclude payments happened after checkout ended. ObservedTimestamp is collector observation, not a second authoritative event time. Do not subtract timestamps from different clocks or sum overlapping parent/child durations into520ms end-to-end latency.

Next bounded request: selected trace, checkout/payments services, minimal event/span metadata and surrounding collector window wide enough for the unmeasured skew. Ask for parent context propagation and clock-synchronization evidence, plus downstream span when available. No request/response bodies, account emails, authorization headers or unrestricted logs are needed. Missing spans may reflect sampling/retention/propagation; they do not prove the downstream was never called. No trace query or clock adjustment was executed.

## Falsification checks
1. Change X1's message or correlationLabel to match checkout; it must still be excluded while its trace ID differs.
2. Remove the parentSpanId enrichment: retain exact-trace membership but report the parent edge as missing. Do not invent a call graph solely from service names.
3. Move P1/P2 source timestamps to plausible local times: the clock anomaly disappears, but the explicit local sequence and parent relationship stay the same. A prettier timeline is not new causal evidence.
4. Omit P2: the downstream outcome is now missing, not a proven success. A root504 does not identify a missing child's error by itself.
5. Supply only observed timestamps: present an arrival view and state that event order/durations are unknown. Do not label ingestion time as when the application published or emitted an event.

## Output and operational boundary
Return selection predicate, included/excluded evidence IDs, per-source ordering, supplied parent edges, clock anomalies, independent duration evidence, gaps and one bounded read-only query specification. Separate facts from inferred edges and state exactly which evidence would disprove the proposed sequence. Do not publish sensitive payloads to a ticket or change collector sampling, logging levels, retention, clocks or production request behavior during correlation. Such changes require their own review. The [OpenTelemetry log data model](https://opentelemetry.io/docs/specs/otel/logs/data-model/) distinguishes event/observed times and optional trace context; it does not make a partial export a complete distributed trace.

Real example

Input

Correlate the exact trace using partial ordering, not a global timestamp sort.

{
  "provenance": "synthetic JSON view, not native OTLP wire data; ISO timestamps for human review",
  "targetTraceId": "11111111111111111111111111111111",
  "clockEvidence": "payments source clock offset unknown; collector arrival delays also unknown",
  "events": [
    {
      "id": "C1",
      "traceId": "11111111111111111111111111111111",
      "spanId": "a1a1a1a1a1a1a1a1",
      "service": "checkout",
      "timestamp": "2026-10-20T10:00:00.000Z",
      "observedTimestamp": "2026-10-20T10:00:00.020Z",
      "event": "request.start",
      "localSequence": 1
    },
    {
      "id": "P1",
      "traceId": "11111111111111111111111111111111",
      "spanId": "b2b2b2b2b2b2b2b2",
      "parentSpanId": "a1a1a1a1a1a1a1a1",
      "service": "payments",
      "timestamp": "2026-10-20T10:00:08.050Z",
      "observedTimestamp": "2026-10-20T10:00:00.090Z",
      "event": "upstream.call.start",
      "localSequence": 1
    },
    {
      "id": "P2",
      "traceId": "11111111111111111111111111111111",
      "spanId": "b2b2b2b2b2b2b2b2",
      "parentSpanId": "a1a1a1a1a1a1a1a1",
      "service": "payments",
      "timestamp": "2026-10-20T10:00:08.170Z",
      "observedTimestamp": "2026-10-20T10:00:00.210Z",
      "event": "upstream.call.error",
      "localSequence": 2,
      "durationMs": 120,
      "reason": "deadline"
    },
    {
      "id": "C2",
      "traceId": "11111111111111111111111111111111",
      "spanId": "a1a1a1a1a1a1a1a1",
      "service": "checkout",
      "timestamp": "2026-10-20T10:00:00.400Z",
      "observedTimestamp": "2026-10-20T10:00:00.450Z",
      "event": "request.end",
      "localSequence": 2,
      "durationMs": 400,
      "status": 504
    },
    {
      "id": "X1",
      "traceId": "22222222222222222222222222222222",
      "spanId": "c3c3c3c3c3c3c3c3",
      "service": "payments",
      "timestamp": "2026-10-20T10:00:00.100Z",
      "observedTimestamp": "2026-10-20T10:00:00.120Z",
      "event": "request.end",
      "localSequence": 1,
      "status": 200,
      "correlationLabel": "same-checkout-label-does-not-prove-same-trace"
    }
  ]
}

Expected result

Selection: C1,P1,P2,C2 have exact target trace11111111111111111111111111111111. Exclude X1: a coincident time or business label does not override a different trace ID. Supplied parent context links checkout span a1a1a1a1a1a1a1a1 to payments span b2b2b2b2b2b2b2b2. parentSpanId is supplied span-context enrichment here, not a mandatory top-level OpenTelemetry log field.

| Evidence | What can be ordered | What must stay uncertain |
| --- | --- | --- |
| C1 -> C2 | Same checkout span and supplied localSequence1→2; supplied local duration400ms, end504 | Which portion was network, queue or application work is not given. |
| P1 -> P2 | Same payments span and localSequence1→2; supplied local duration120ms, deadline result | The destination's own server trace and exact timeout budget are missing. |
| checkout -> payments | Supplied parent-span relationship establishes the request relationship | Parent/child alone does not prove every cross-service log ordering or a complete critical path. |
| payments wall time later than collector observation | P1/P2 source timestamps are about8s ahead of collector values | Clock offset and collection delay are unmeasured; this is not an8-second network call or permission to rewrite original timestamps. |

Use a partial-order view plus a separately labeled collector-arrival listing. Do not globally sort source timestamps and conclude payments happened after checkout ended. ObservedTimestamp is collector observation, not a second authoritative event time. Do not subtract timestamps from different clocks or sum overlapping parent/child durations into520ms end-to-end latency.

Next bounded request: selected trace, checkout/payments services, minimal event/span metadata and surrounding collector window wide enough for the unmeasured skew. Ask for parent context propagation and clock-synchronization evidence, plus downstream span when available. No request/response bodies, account emails, authorization headers or unrestricted logs are needed. Missing spans may reflect sampling/retention/propagation; they do not prove the downstream was never called. No trace query or clock adjustment was executed.

Source evidence

Agent Skills specificationofficialOpenTelemetry logsofficialOpenTelemetry log data modelofficial