SKILL.md
Data pipeline quality reviewer
An evidence-first workflow to verify freshness, completeness, uniqueness, lineage, and recovery behavior.
- Revision
- 2
- Verified
- 2026-10-20
Compatibility and paths
Codex
skills/data-pipeline-quality/SKILL.mdClaude Code
.claude/skills/data-pipeline-quality/SKILL.mdVS Code
.github/skills/data-pipeline-quality/SKILL.mdTrust and provenance
Curated record reviewed 2026-10-20. Results still depend on the supplied context and target environment.
Generated assetReady to copy or download
---
name: data-pipeline-quality
description: Reviews a fixed partition's schema, units, identity, event time and source completeness independently. Use when a recent ingestion timestamp or successful retry is being mistaken for complete, correctly deduplicated data.
license: CC-BY-4.0
compatibility: Complete owned CSV exercise requires Python 3.11+ standard library only. It reads an included StringIO fixture, writes no files, opens no database and does not execute an Airflow job.
metadata:
author: oneliners
version: "2.0.0"
---
# Data pipeline quality reviewer
## State the data contract and partition first
Request lineage/source identity, schema/units, event-time interval/timezone, key/version semantics, source completeness evidence, watermark/lateness contract, quality thresholds and recovery/checkpoint behavior. Distinguish job success, fresh ingestion, fresh event time and a complete business partition. Source manifests and watermarks are claims tied to a producer/version, not a conclusion inferred from the most recent row.
## Complete owned CSV and replay exercise
Save this reviewed block as pipeline-fixture.py and run python pipeline-fixture.py with Python 3.11+. All CSV rows, manifest IDs and fixed timestamps are included. There is no missing input file, dependency installation, network request or warehouse connection. The explicit ms/two-decimal/nonnegative rules belong to this fictional contract, not all telemetry systems.
```python
import csv
import io
import json
from datetime import datetime, timezone
from decimal import Decimal, InvalidOperation
CSV = """job_id,event_time,received_at,duration,unit
J1,2026-10-20T10:00:00+00:00,2026-10-20T10:01:00+00:00,12.50,ms
J2,2026-10-20T10:02:00+00:00,2026-10-20T10:03:00+00:00,7.50,ms
J2,2026-10-20T10:02:00+00:00,2026-10-20T10:04:00+00:00,7.50,ms
J3,2026-10-20T10:04:00+00:00,2026-10-20T10:05:00+00:00,NaN,ms
J4,2026-10-20T09:58:00+00:00,2026-10-20T10:04:00+00:00,9.00,ms
J5,2026-10-20T10:05:00+00:00,2026-10-20T10:05:00+00:00,3.00,ms
J6,2026-10-20T10:03:00+00:00,2026-10-20T10:04:00+00:00,5.00,s
"""
def instant(value):
parsed = datetime.fromisoformat(value)
if parsed.tzinfo is None:
raise ValueError("timezone required")
return parsed.astimezone(timezone.utc)
def review_partition(csv_text, manifest_ids, start, end, watermark, run_time):
start, end, watermark, run_time = map(instant, (start, end, watermark, run_time))
if start >= end:
raise ValueError("nonempty half-open interval required")
if manifest_ids is not None and (not manifest_ids or len(set(manifest_ids)) != len(manifest_ids)):
raise ValueError("unique nonempty source manifest required")
reader = csv.DictReader(io.StringIO(csv_text))
if reader.fieldnames != ["job_id", "event_time", "received_at", "duration", "unit"]:
raise ValueError("fixture schema mismatch")
groups, invalid, outside, received = {}, [], [], []
rows = 0
for row in reader:
rows += 1
identity = row.get("job_id")
try:
event, arrival = instant(row["event_time"]), instant(row["received_at"])
received.append(arrival)
if not identity:
raise ValueError("missing job ID")
if not start <= event < end:
outside.append(identity)
continue
value = Decimal(row["duration"])
if not value.is_finite() or value < 0 or value.as_tuple().exponent < -2:
raise ValueError("finite nonnegative two-decimal duration required")
if row["unit"] != "ms":
raise ValueError("contract unit is ms; do not mix or silently convert")
groups.setdefault(identity, []).append((event, value, row["unit"]))
except (ValueError, KeyError, TypeError, InvalidOperation) as error:
invalid.append({"id": identity, "reason": str(error)})
accepted, conflicts, duplicate_rows = {}, [], 0
for identity, records in groups.items():
if len(set(records)) != 1:
conflicts.append(identity)
continue
duplicate_rows += len(records) - 1
accepted[identity] = records[0]
# Owned in-memory idempotency control, not durable exactly-once storage.
store = {}
for _ in range(2):
store.update(accepted)
expected = set(manifest_ids) if manifest_ids is not None else None
missing = sorted(expected - set(accepted)) if expected is not None else None
event_age = (run_time - max(item[0] for item in accepted.values())).total_seconds() if accepted else None
receipt_age = (run_time - max(received)).total_seconds() if received else None
reasons = []
if missing is None:
reasons.append("UNKNOWN_SOURCE_COMPLETENESS")
elif missing:
reasons.append("MISSING_EXPECTED_IDS")
if watermark < end:
reasons.append("WINDOW_NOT_FINAL_BY_SUPPLIED_WATERMARK")
if invalid:
reasons.append("INVALID_ROWS")
if conflicts:
reasons.append("CONFLICTING_DUPLICATE_IDS")
if event_age is not None and event_age < 0:
reasons.append("CLOCK_ALIGNMENT_UNKNOWN")
return {"rows": rows, "acceptedIds": sorted(accepted), "duplicateRows": duplicate_rows, "invalid": invalid, "outsideIds": outside, "conflictIds": conflicts, "missingExpectedIds": missing, "acceptedExpectedCount": len(expected & set(accepted)) if expected is not None else None, "expectedCount": len(expected) if expected is not None else None, "latestAcceptedEventAgeSeconds": event_age, "latestReceiptAgeSeconds": receipt_age, "storedIdsAfterTwoPasses": len(store), "storedDurationMs": format(sum((item[1] for item in store.values()), Decimal("0")), ".2f"), "gate": "HOLD" if reasons else "FIXTURE_CHECKS_CLEAR_NOT_PIPELINE_PROOF", "reasons": reasons}
result = review_partition(CSV, ["J1", "J2", "J3", "J7"], "2026-10-20T10:00:00+00:00", "2026-10-20T10:05:00+00:00", "2026-10-20T10:04:00+00:00", "2026-10-20T10:06:00+00:00")
assert result["acceptedIds"] == ["J1", "J2"]
assert result["duplicateRows"] == 1
assert result["storedIdsAfterTwoPasses"] == 2
assert result["storedDurationMs"] == "20.00"
assert result["missingExpectedIds"] == ["J3", "J7"]
assert result["gate"] == "HOLD"
print(json.dumps(result, indent=2))
```
## Quality ledger and next action
For the half-open event-time window [10:00, 10:05) UTC, J1/J2 are accepted. The repeated J2 has the same business payload despite a later receipt and contributes one duplicate row, not a second job. J3's NaN is rejected; J6 uses seconds where this fixture requires milliseconds and is quarantined rather than added as 5 ms. J4 belongs to the previous window; J5 is exactly at the exclusive end and belongs to the next one.
Accepted expected IDs: 2/4; missing J3 and J7. This is completeness against the supplied source manifest, not a claim that every real source event is known. Latest valid event age: 240 s, while the latest receipt age is only 60 s. A recent receipt, including an invalid/out-of-window record, cannot establish fresh or complete business data. The supplied producer watermark 10:04 is before the window end; the window is not final under that producer's stated contract.
Processing the accepted records twice leaves 2 IDs and total fixture job-duration 20.00 ms in the owned dictionary, not 4 IDs/40.00 ms. The sum is a reconciliation of these job-duration values, not wall-clock latency or p95. An identical-key/different-payload replay is a conflict: quarantine that ID instead of silently keeping whichever row arrived first/last. This small in-memory control is not durable UPSERT, concurrent transaction handling or an Airflow exactly-once guarantee.
HOLD for missing expected IDs, an unfinalized watermark and invalid rows. Owner next steps: investigate J3's source value/unit contract and J7's lineage; compare the fixed partition against a newer authoritative source manifest/watermark; replay only an approved bounded partition into a staging target with real key/transaction/checkpoint semantics. Do not change the window to latest data, silently coerce NaN/units or rerun a whole production pipeline as a diagnostic action. No scheduler, warehouse or customer data was accessed.
## Verify retries without inventing exactly-once behavior
Use the same immutable input partition, business keys and expected output contract for each comparison. Test identical replay and conflicting replay separately; include boundaries, missing IDs, invalid/nonfinite values, units and partial failure. A real materialization needs transactional uniqueness/merge and checkpoint semantics at its durable target, plus controlled publication of a complete result. [Airflow best practices](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html) discuss deterministic partitions and reruns; the dictionary here is not an Airflow/DAG execution or substitute for storage guarantees.
## Denominators and time are part of the evidence
Report accepted/rejected/duplicate/conflict/outside counts with their rules; do not drop failures from a denominator to improve a score. Missing source inventory is UNKNOWN, not 100% complete. A half-open boundary must remain the same during a replay. Check clock/timezone alignment before interpreting negative ages or late events. [Python Decimal](https://docs.python.org/3/library/decimal.html) can parse nonfinite values too, so the fixture explicitly rejects them; decimal arithmetic alone does not validate the contract.
## Handoff and stop conditions
Return partition/source/schema identities, quality failures with row/key references, completeness and freshness scopes, replay result, owner, bounded recovery plan and unresolved guarantees. Redact values at source and avoid transferring raw private records to new systems. Do not backfill or overwrite production partitions, change acceptance rules or expose credentials to produce a green report. Unknown lineage, conflicting IDs, incomplete source evidence or unsafe replay requires an explicit hold and missing-evidence request.
Real example
Input
Review the complete seven-row CSV, fixed event-time window, source IDs and watermark included in SKILL.md. Distinguish duplicate, invalid, wrong-unit, outside-window and missing evidence; execute only the included stdlib-only fixture, not a production pipeline.
Expected result
For the half-open event-time window [10:00, 10:05) UTC, J1/J2 are accepted. The repeated J2 has the same business payload despite a later receipt and contributes one duplicate row, not a second job. J3's NaN is rejected; J6 uses seconds where this fixture requires milliseconds and is quarantined rather than added as 5 ms. J4 belongs to the previous window; J5 is exactly at the exclusive end and belongs to the next one. Accepted expected IDs: 2/4; missing J3 and J7. This is completeness against the supplied source manifest, not a claim that every real source event is known. Latest valid event age: 240 s, while the latest receipt age is only 60 s. A recent receipt, including an invalid/out-of-window record, cannot establish fresh or complete business data. The supplied producer watermark 10:04 is before the window end; the window is not final under that producer's stated contract. Processing the accepted records twice leaves 2 IDs and total fixture job-duration 20.00 ms in the owned dictionary, not 4 IDs/40.00 ms. The sum is a reconciliation of these job-duration values, not wall-clock latency or p95. An identical-key/different-payload replay is a conflict: quarantine that ID instead of silently keeping whichever row arrived first/last. This small in-memory control is not durable UPSERT, concurrent transaction handling or an Airflow exactly-once guarantee. HOLD for missing expected IDs, an unfinalized watermark and invalid rows. Owner next steps: investigate J3's source value/unit contract and J7's lineage; compare the fixed partition against a newer authoritative source manifest/watermark; replay only an approved bounded partition into a staging target with real key/transaction/checkpoint semantics. Do not change the window to latest data, silently coerce NaN/units or rerun a whole production pipeline as a diagnostic action. No scheduler, warehouse or customer data was accessed.