SKILL.md
Distributed log correlation investigator
An evidence-first workflow to reconstruct one request or failure across services without retaining sensitive payloads.
- Revision
- 2
- Verified
- 2026-10-20
Compatibility and paths
Codex
skills/distributed-log-timeline/SKILL.mdClaude Code
.claude/skills/distributed-log-timeline/SKILL.mdVS Code
.github/skills/distributed-log-timeline/SKILL.mdTrust and provenance
Curated record reviewed 2026-10-20. Results still depend on the supplied context and target environment.
Generated assetReady to copy or download
---
name: distributed-log-timeline
description: Correlates one bounded request using exact trace/span context while preserving clock uncertainty, missing spans and privacy. Use when cross-service logs appear out of order; return a partial-order timeline and one discriminating query, not invented causality.
license: CC-BY-4.0
compatibility: Reviews bounded redacted evidence offline. Production queries, paging, configuration changes and mitigation require separately authorized targets and operators.
metadata:
author: oneliners
version: "2.0.0"
---
# Distributed log correlation investigator
## Select one request, not a whole customer history
Ask for exact environment, trace/request identifier, collection window, services and authorized metadata fields. Establish identifier namespace and propagation rules; two systems may reuse short request IDs. Keep raw evidence in its authorized store and produce a redacted ledger with stable evidence IDs. Do not join tenants or unrelated traces because a message or business label looks similar. If there is no reliable join key, label the association hypothetical.
## Keep clocks and relationships separate
Preserve event/source timestamp, collector observed timestamp, time zone, duration provenance, local sequence and trace/span IDs separately. An event's source clock can be wrong; collection can also be delayed. Neither a global timestamp sort nor collector arrival alone establishes causality. Use explicit parent relationships and documented in-process sequence where supplied, retaining uncertainty between services. Local durations require an appropriate clock/source; do not infer them from unrelated wall clocks or add nested span durations as if they were disjoint work.
## Worked skewed trace
This is a readable synthetic JSON export. The ISO fields are not the numeric nanosecond OTLP encoding. parentSpanId/localSequence/durationMs are supplied enrichment for this case, not assertions that every log record has those fields.
```json
{
"provenance": "synthetic JSON view, not native OTLP wire data; ISO timestamps for human review",
"targetTraceId": "11111111111111111111111111111111",
"clockEvidence": "payments source clock offset unknown; collector arrival delays also unknown",
"events": [
{
"id": "C1",
"traceId": "11111111111111111111111111111111",
"spanId": "a1a1a1a1a1a1a1a1",
"service": "checkout",
"timestamp": "2026-10-20T10:00:00.000Z",
"observedTimestamp": "2026-10-20T10:00:00.020Z",
"event": "request.start",
"localSequence": 1
},
{
"id": "P1",
"traceId": "11111111111111111111111111111111",
"spanId": "b2b2b2b2b2b2b2b2",
"parentSpanId": "a1a1a1a1a1a1a1a1",
"service": "payments",
"timestamp": "2026-10-20T10:00:08.050Z",
"observedTimestamp": "2026-10-20T10:00:00.090Z",
"event": "upstream.call.start",
"localSequence": 1
},
{
"id": "P2",
"traceId": "11111111111111111111111111111111",
"spanId": "b2b2b2b2b2b2b2b2",
"parentSpanId": "a1a1a1a1a1a1a1a1",
"service": "payments",
"timestamp": "2026-10-20T10:00:08.170Z",
"observedTimestamp": "2026-10-20T10:00:00.210Z",
"event": "upstream.call.error",
"localSequence": 2,
"durationMs": 120,
"reason": "deadline"
},
{
"id": "C2",
"traceId": "11111111111111111111111111111111",
"spanId": "a1a1a1a1a1a1a1a1",
"service": "checkout",
"timestamp": "2026-10-20T10:00:00.400Z",
"observedTimestamp": "2026-10-20T10:00:00.450Z",
"event": "request.end",
"localSequence": 2,
"durationMs": 400,
"status": 504
},
{
"id": "X1",
"traceId": "22222222222222222222222222222222",
"spanId": "c3c3c3c3c3c3c3c3",
"service": "payments",
"timestamp": "2026-10-20T10:00:00.100Z",
"observedTimestamp": "2026-10-20T10:00:00.120Z",
"event": "request.end",
"localSequence": 1,
"status": 200,
"correlationLabel": "same-checkout-label-does-not-prove-same-trace"
}
]
}
```
## Correlated result and bounded next query
Selection: C1,P1,P2,C2 have exact target trace11111111111111111111111111111111. Exclude X1: a coincident time or business label does not override a different trace ID. Supplied parent context links checkout span a1a1a1a1a1a1a1a1 to payments span b2b2b2b2b2b2b2b2. parentSpanId is supplied span-context enrichment here, not a mandatory top-level OpenTelemetry log field.
| Evidence | What can be ordered | What must stay uncertain |
| --- | --- | --- |
| C1 -> C2 | Same checkout span and supplied localSequence1→2; supplied local duration400ms, end504 | Which portion was network, queue or application work is not given. |
| P1 -> P2 | Same payments span and localSequence1→2; supplied local duration120ms, deadline result | The destination's own server trace and exact timeout budget are missing. |
| checkout -> payments | Supplied parent-span relationship establishes the request relationship | Parent/child alone does not prove every cross-service log ordering or a complete critical path. |
| payments wall time later than collector observation | P1/P2 source timestamps are about8s ahead of collector values | Clock offset and collection delay are unmeasured; this is not an8-second network call or permission to rewrite original timestamps. |
Use a partial-order view plus a separately labeled collector-arrival listing. Do not globally sort source timestamps and conclude payments happened after checkout ended. ObservedTimestamp is collector observation, not a second authoritative event time. Do not subtract timestamps from different clocks or sum overlapping parent/child durations into520ms end-to-end latency.
Next bounded request: selected trace, checkout/payments services, minimal event/span metadata and surrounding collector window wide enough for the unmeasured skew. Ask for parent context propagation and clock-synchronization evidence, plus downstream span when available. No request/response bodies, account emails, authorization headers or unrestricted logs are needed. Missing spans may reflect sampling/retention/propagation; they do not prove the downstream was never called. No trace query or clock adjustment was executed.
## Falsification checks
1. Change X1's message or correlationLabel to match checkout; it must still be excluded while its trace ID differs.
2. Remove the parentSpanId enrichment: retain exact-trace membership but report the parent edge as missing. Do not invent a call graph solely from service names.
3. Move P1/P2 source timestamps to plausible local times: the clock anomaly disappears, but the explicit local sequence and parent relationship stay the same. A prettier timeline is not new causal evidence.
4. Omit P2: the downstream outcome is now missing, not a proven success. A root504 does not identify a missing child's error by itself.
5. Supply only observed timestamps: present an arrival view and state that event order/durations are unknown. Do not label ingestion time as when the application published or emitted an event.
## Output and operational boundary
Return selection predicate, included/excluded evidence IDs, per-source ordering, supplied parent edges, clock anomalies, independent duration evidence, gaps and one bounded read-only query specification. Separate facts from inferred edges and state exactly which evidence would disprove the proposed sequence. Do not publish sensitive payloads to a ticket or change collector sampling, logging levels, retention, clocks or production request behavior during correlation. Such changes require their own review. The [OpenTelemetry log data model](https://opentelemetry.io/docs/specs/otel/logs/data-model/) distinguishes event/observed times and optional trace context; it does not make a partial export a complete distributed trace.
Real example
Input
Correlate the exact trace using partial ordering, not a global timestamp sort.
{
"provenance": "synthetic JSON view, not native OTLP wire data; ISO timestamps for human review",
"targetTraceId": "11111111111111111111111111111111",
"clockEvidence": "payments source clock offset unknown; collector arrival delays also unknown",
"events": [
{
"id": "C1",
"traceId": "11111111111111111111111111111111",
"spanId": "a1a1a1a1a1a1a1a1",
"service": "checkout",
"timestamp": "2026-10-20T10:00:00.000Z",
"observedTimestamp": "2026-10-20T10:00:00.020Z",
"event": "request.start",
"localSequence": 1
},
{
"id": "P1",
"traceId": "11111111111111111111111111111111",
"spanId": "b2b2b2b2b2b2b2b2",
"parentSpanId": "a1a1a1a1a1a1a1a1",
"service": "payments",
"timestamp": "2026-10-20T10:00:08.050Z",
"observedTimestamp": "2026-10-20T10:00:00.090Z",
"event": "upstream.call.start",
"localSequence": 1
},
{
"id": "P2",
"traceId": "11111111111111111111111111111111",
"spanId": "b2b2b2b2b2b2b2b2",
"parentSpanId": "a1a1a1a1a1a1a1a1",
"service": "payments",
"timestamp": "2026-10-20T10:00:08.170Z",
"observedTimestamp": "2026-10-20T10:00:00.210Z",
"event": "upstream.call.error",
"localSequence": 2,
"durationMs": 120,
"reason": "deadline"
},
{
"id": "C2",
"traceId": "11111111111111111111111111111111",
"spanId": "a1a1a1a1a1a1a1a1",
"service": "checkout",
"timestamp": "2026-10-20T10:00:00.400Z",
"observedTimestamp": "2026-10-20T10:00:00.450Z",
"event": "request.end",
"localSequence": 2,
"durationMs": 400,
"status": 504
},
{
"id": "X1",
"traceId": "22222222222222222222222222222222",
"spanId": "c3c3c3c3c3c3c3c3",
"service": "payments",
"timestamp": "2026-10-20T10:00:00.100Z",
"observedTimestamp": "2026-10-20T10:00:00.120Z",
"event": "request.end",
"localSequence": 1,
"status": 200,
"correlationLabel": "same-checkout-label-does-not-prove-same-trace"
}
]
}Expected result
Selection: C1,P1,P2,C2 have exact target trace11111111111111111111111111111111. Exclude X1: a coincident time or business label does not override a different trace ID. Supplied parent context links checkout span a1a1a1a1a1a1a1a1 to payments span b2b2b2b2b2b2b2b2. parentSpanId is supplied span-context enrichment here, not a mandatory top-level OpenTelemetry log field. | Evidence | What can be ordered | What must stay uncertain | | --- | --- | --- | | C1 -> C2 | Same checkout span and supplied localSequence1→2; supplied local duration400ms, end504 | Which portion was network, queue or application work is not given. | | P1 -> P2 | Same payments span and localSequence1→2; supplied local duration120ms, deadline result | The destination's own server trace and exact timeout budget are missing. | | checkout -> payments | Supplied parent-span relationship establishes the request relationship | Parent/child alone does not prove every cross-service log ordering or a complete critical path. | | payments wall time later than collector observation | P1/P2 source timestamps are about8s ahead of collector values | Clock offset and collection delay are unmeasured; this is not an8-second network call or permission to rewrite original timestamps. | Use a partial-order view plus a separately labeled collector-arrival listing. Do not globally sort source timestamps and conclude payments happened after checkout ended. ObservedTimestamp is collector observation, not a second authoritative event time. Do not subtract timestamps from different clocks or sum overlapping parent/child durations into520ms end-to-end latency. Next bounded request: selected trace, checkout/payments services, minimal event/span metadata and surrounding collector window wide enough for the unmeasured skew. Ask for parent context propagation and clock-synchronization evidence, plus downstream span when available. No request/response bodies, account emails, authorization headers or unrestricted logs are needed. Missing spans may reflect sampling/retention/propagation; they do not prove the downstream was never called. No trace query or clock adjustment was executed.