SKILL.md
Product experiment reviewer
An evidence-first workflow to review an experiment for hypothesis quality, metrics, guardrails, exposure, and decision rules.
- Revision
- 2
- Verified
- 2026-10-20
Compatibility and paths
Codex
skills/experiment-review/SKILL.mdClaude Code
.claude/skills/experiment-review/SKILL.mdVS Code
.github/skills/experiment-review/SKILL.mdTrust and provenance
Curated record reviewed 2026-10-20. Results still depend on the supplied context and target environment.
Generated assetReady to copy or download
---
name: experiment-review
description: Reviews an experiment's unit/denominators, missing outcomes and precommitted guardrails before any winner claim. Use when aggregate uplift may hide telemetry loss or safety regressions; distinguish arithmetic, statistical uncertainty and causal evidence.
license: CC-BY-4.0
compatibility: Complete fictional aggregate-ledger exercise uses Node.js 22.13+ only. No private user data, telemetry, experiment service or network. It computes bounds and an example guardrail, not a significance or power test.
metadata:
author: oneliners
version: "2.0.0"
---
# Product experiment reviewer
## Establish the decision before inspecting uplift
Record the actual hypothesis, population/unit, variants, assignment/exposure logic, outcome window, primary/guardrail/data-quality metrics and precommitted analysis/stop criteria. Check duration/power and interference/stable identity assumptions with the responsible analyst. Do not silently exclude missing outcomes, switch denominator or invent a baseline to improve a chart.
## Complete original missing-outcome case
The full synthetic ledger below gives assigned users, recorded binary outcomes/successes and an independently complete users-with-error measure. It contains no user identifiers. The treatment's missing outcome records are intentionally unknown, not known failures. The 5% limit is an original exercise rule, not a universal experimentation threshold.
Save as experiment-fixture.mjs and run node experiment-fixture.mjs with Node.js 22.13+. It performs actual count arithmetic over included data only. No experiment is started/stopped and no online measurement is claimed.
```js
import assert from "node:assert/strict";
const experimentLedger = {
"id": "bookmark-layout-owned-example",
"unit": "eligible assigned user, one assignment, one 24h outcome window",
"window": "fixed fictional 14-day enrollment; complete follow-up intended",
"expectedAssignment": [
0.5,
0.5
],
"precommittedMaxErrorUserRate": 0.05,
"variants": [
{
"name": "control",
"assigned": 1000,
"observedOutcomes": 1000,
"successfulOutcomes": 300,
"errorObservedUsers": 1000,
"usersWithError": 20
},
{
"name": "treatment",
"assigned": 1000,
"observedOutcomes": 500,
"successfulOutcomes": 200,
"errorObservedUsers": 1000,
"usersWithError": 90
}
]
};
function reviewExperiment(ledger) {
assert.ok(typeof ledger.unit === "string" && ledger.unit.length > 0 && typeof ledger.window === "string" && ledger.window.length > 0, "explicit unit and observation window required");
assert.ok(Array.isArray(ledger.variants) && ledger.variants.length === 2 && ledger.variants[0].name === "control" && ledger.variants[1].name === "treatment", "named control/treatment required");
assert.ok(Number.isFinite(ledger.precommittedMaxErrorUserRate) && ledger.precommittedMaxErrorUserRate >= 0 && ledger.precommittedMaxErrorUserRate <= 1, "precommitted guardrail required");
const issues = [], variants = ledger.variants.map((variant) => {
for (const key of ["assigned", "observedOutcomes", "successfulOutcomes", "errorObservedUsers", "usersWithError"]) assert.ok(Number.isInteger(variant[key]) && variant[key] >= 0, "integer unit counts required");
assert.ok(variant.assigned > 0 && variant.observedOutcomes <= variant.assigned && variant.successfulOutcomes <= variant.observedOutcomes && variant.errorObservedUsers <= variant.assigned && variant.usersWithError <= variant.errorObservedUsers, "inconsistent count denominators");
const missing = variant.assigned - variant.observedOutcomes;
if (missing > 0) issues.push(variant.name + ": OUTCOME_COVERAGE_INCOMPLETE");
if (variant.errorObservedUsers !== variant.assigned) issues.push(variant.name + ": GUARDRAIL_COVERAGE_UNKNOWN");
const errorUserRate = variant.errorObservedUsers === variant.assigned ? variant.usersWithError / variant.assigned : null;
if (errorUserRate !== null && errorUserRate > ledger.precommittedMaxErrorUserRate) issues.push(variant.name + ": PRECOMMITTED_ERROR_GUARDRAIL_EXCEEDED");
return { name: variant.name, assigned: variant.assigned, observedOutcomes: variant.observedOutcomes, observedOnlyRate: variant.observedOutcomes ? variant.successfulOutcomes / variant.observedOutcomes : null, missingOutcomes: missing, completionBounds: [variant.successfulOutcomes / variant.assigned, (variant.successfulOutcomes + missing) / variant.assigned], errorUserRate };
});
return { variants, issues, decision: issues.length ? "HOLD_NO_WINNER_CLAIM" : "READY_FOR_PLANNED_INFERENCE_NOT_A_WINNER", assignmentCounts: ledger.variants.map((v) => v.assigned), limitation: "Coverage bounds and a stated operational guardrail only. No randomization audit, SRM significance test, power calculation, confidence interval or causal estimate." };
}
const result = reviewExperiment(experimentLedger);
assert.equal(result.variants[0].observedOnlyRate, 0.3);
assert.equal(result.variants[1].observedOnlyRate, 0.4);
assert.deepEqual(result.variants[1].completionBounds, [0.2, 0.7]);
assert.equal(result.variants[1].errorUserRate, 0.09);
assert.equal(result.decision, "HOLD_NO_WINNER_CLAIM");
console.log(JSON.stringify(result, null, 2));
```
## Worked denominator review and decision
A naive logged-outcome comparison says control 300/1000=30% and treatment 200/500=40%, a 10 percentage-point difference (not 10% relative lift). But assignment counts are 1000/1000: treatment has 500 unobserved outcomes. Its full-population completion rate could range from 200/1000=20% to 700/1000=70% under the stated binary-outcome/count assumptions. That is a missing-data bound, not a confidence interval. Treating every unobserved user as a failure and claiming 20% would be another unsupported point estimate.
Equal supplied assignment counts do not prove correct randomization, stable identities or absence of leakage. The differential outcome coverage requires investigating eligibility/exposure/logging/follow-up. It is not by itself a performed assignment SRM significance test. Keep assignment, outcome-observed and guardrail denominators separate rather than filtering control until it resembles treatment.
The independently complete guardrail ledger has users-with-error 20/1000=2% in control and 90/1000=9% in treatment. Treatment exceeds this fictional precommitted 5% operational stop threshold. The output is HOLD for both incomplete outcome coverage and that guardrail breach, even though the naive metric looks better. It does not prove statistical significance or a causal treatment harm; it enforces the supplied safety rule while the data/design are reviewed.
Before an experiment, specify hypothesis, eligible population/randomization unit, exposure and outcome windows, primary metric/denominator, power/duration and analysis method, guardrails, data-quality checks and stop/decision rules. Here those statistical design checks are not executed, so even complete counters and a passing guardrail would mean ready for the planned inference, not a winner. Do not keep peeking until a favorable result appears, move goalposts after observation or roll out from these synthetic aggregates. No live users, traffic, telemetry collection or feature flag were accessed.
## Evidence and follow-up, not automatic rollout
[Microsoft ExP pre-experiment research](https://www.microsoft.com/en-us/research/articles/patterns-of-trustworthy-experimentation-pre-experiment-stage/) and [data-quality research](https://www.microsoft.com/en-us/research/articles/data-quality-fundamental-building-blocks-for-trustworthy-a-b-testing-analysis/) inform the distinction between a metric movement and trustworthy analysis; the counts/bounds here are original. Return the versioned experiment contract, eligibility/assignment/exposure/outcome reconciliation, guardrail result, actual statistical method/results if performed, limitations and owner decision. Missing data, unsupported inference or breached stop rules remains HOLD. Do not change traffic, tracking consent, retention or a feature flag merely to complete the review.
Real example
Input
Review the complete synthetic two-variant ledger in SKILL.md: equal assigned populations, unequal observed outcomes, and complete error-user counts. Calculate observed-only rates, missing-outcome bounds and the stated guardrail; do not label those bounds a confidence interval or declare a winner.
Expected result
A naive logged-outcome comparison says control 300/1000=30% and treatment 200/500=40%, a 10 percentage-point difference (not 10% relative lift). But assignment counts are 1000/1000: treatment has 500 unobserved outcomes. Its full-population completion rate could range from 200/1000=20% to 700/1000=70% under the stated binary-outcome/count assumptions. That is a missing-data bound, not a confidence interval. Treating every unobserved user as a failure and claiming 20% would be another unsupported point estimate. Equal supplied assignment counts do not prove correct randomization, stable identities or absence of leakage. The differential outcome coverage requires investigating eligibility/exposure/logging/follow-up. It is not by itself a performed assignment SRM significance test. Keep assignment, outcome-observed and guardrail denominators separate rather than filtering control until it resembles treatment. The independently complete guardrail ledger has users-with-error 20/1000=2% in control and 90/1000=9% in treatment. Treatment exceeds this fictional precommitted 5% operational stop threshold. The output is HOLD for both incomplete outcome coverage and that guardrail breach, even though the naive metric looks better. It does not prove statistical significance or a causal treatment harm; it enforces the supplied safety rule while the data/design are reviewed. Before an experiment, specify hypothesis, eligible population/randomization unit, exposure and outcome windows, primary metric/denominator, power/duration and analysis method, guardrails, data-quality checks and stop/decision rules. Here those statistical design checks are not executed, so even complete counters and a passing guardrail would mean ready for the planned inference, not a winner. Do not keep peeking until a favorable result appears, move goalposts after observation or roll out from these synthetic aggregates. No live users, traffic, telemetry collection or feature flag were accessed.