OneLinersCommand workbench
AI
Back to skills
SKILL.md

Product experiment reviewer

An evidence-first workflow to review an experiment for hypothesis quality, metrics, guardrails, exposure, and decision rules.

Revision
2
Verified
2026-10-20
Save or explore
Save to collectionCreate a collection in the sidebar first.

Compatibility and paths

Codexskills/experiment-review/SKILL.md
Claude Code.claude/skills/experiment-review/SKILL.md
VS Code.github/skills/experiment-review/SKILL.md

Trust and provenance

Curated record reviewed 2026-10-20. Results still depend on the supplied context and target environment.

Generated assetReady to copy or download
---
name: experiment-review
description: Reviews an experiment's unit/denominators, missing outcomes and precommitted guardrails before any winner claim. Use when aggregate uplift may hide telemetry loss or safety regressions; distinguish arithmetic, statistical uncertainty and causal evidence.
license: CC-BY-4.0
compatibility: Complete fictional aggregate-ledger exercise uses Node.js 22.13+ only. No private user data, telemetry, experiment service or network. It computes bounds and an example guardrail, not a significance or power test.
metadata:
  author: oneliners
  version: "2.0.0"
---

# Product experiment reviewer

## Establish the decision before inspecting uplift
Record the actual hypothesis, population/unit, variants, assignment/exposure logic, outcome window, primary/guardrail/data-quality metrics and precommitted analysis/stop criteria. Check duration/power and interference/stable identity assumptions with the responsible analyst. Do not silently exclude missing outcomes, switch denominator or invent a baseline to improve a chart.

## Complete original missing-outcome case
The full synthetic ledger below gives assigned users, recorded binary outcomes/successes and an independently complete users-with-error measure. It contains no user identifiers. The treatment's missing outcome records are intentionally unknown, not known failures. The 5% limit is an original exercise rule, not a universal experimentation threshold.

Save as experiment-fixture.mjs and run node experiment-fixture.mjs with Node.js 22.13+. It performs actual count arithmetic over included data only. No experiment is started/stopped and no online measurement is claimed.

```js
import assert from "node:assert/strict";
const experimentLedger = {
  "id": "bookmark-layout-owned-example",
  "unit": "eligible assigned user, one assignment, one 24h outcome window",
  "window": "fixed fictional 14-day enrollment; complete follow-up intended",
  "expectedAssignment": [
    0.5,
    0.5
  ],
  "precommittedMaxErrorUserRate": 0.05,
  "variants": [
    {
      "name": "control",
      "assigned": 1000,
      "observedOutcomes": 1000,
      "successfulOutcomes": 300,
      "errorObservedUsers": 1000,
      "usersWithError": 20
    },
    {
      "name": "treatment",
      "assigned": 1000,
      "observedOutcomes": 500,
      "successfulOutcomes": 200,
      "errorObservedUsers": 1000,
      "usersWithError": 90
    }
  ]
};
function reviewExperiment(ledger) {
  assert.ok(typeof ledger.unit === "string" && ledger.unit.length > 0 && typeof ledger.window === "string" && ledger.window.length > 0, "explicit unit and observation window required");
  assert.ok(Array.isArray(ledger.variants) && ledger.variants.length === 2 && ledger.variants[0].name === "control" && ledger.variants[1].name === "treatment", "named control/treatment required");
  assert.ok(Number.isFinite(ledger.precommittedMaxErrorUserRate) && ledger.precommittedMaxErrorUserRate >= 0 && ledger.precommittedMaxErrorUserRate <= 1, "precommitted guardrail required");
  const issues = [], variants = ledger.variants.map((variant) => {
    for (const key of ["assigned", "observedOutcomes", "successfulOutcomes", "errorObservedUsers", "usersWithError"]) assert.ok(Number.isInteger(variant[key]) && variant[key] >= 0, "integer unit counts required");
    assert.ok(variant.assigned > 0 && variant.observedOutcomes <= variant.assigned && variant.successfulOutcomes <= variant.observedOutcomes && variant.errorObservedUsers <= variant.assigned && variant.usersWithError <= variant.errorObservedUsers, "inconsistent count denominators");
    const missing = variant.assigned - variant.observedOutcomes;
    if (missing > 0) issues.push(variant.name + ": OUTCOME_COVERAGE_INCOMPLETE");
    if (variant.errorObservedUsers !== variant.assigned) issues.push(variant.name + ": GUARDRAIL_COVERAGE_UNKNOWN");
    const errorUserRate = variant.errorObservedUsers === variant.assigned ? variant.usersWithError / variant.assigned : null;
    if (errorUserRate !== null && errorUserRate > ledger.precommittedMaxErrorUserRate) issues.push(variant.name + ": PRECOMMITTED_ERROR_GUARDRAIL_EXCEEDED");
    return { name: variant.name, assigned: variant.assigned, observedOutcomes: variant.observedOutcomes, observedOnlyRate: variant.observedOutcomes ? variant.successfulOutcomes / variant.observedOutcomes : null, missingOutcomes: missing, completionBounds: [variant.successfulOutcomes / variant.assigned, (variant.successfulOutcomes + missing) / variant.assigned], errorUserRate };
  });
  return { variants, issues, decision: issues.length ? "HOLD_NO_WINNER_CLAIM" : "READY_FOR_PLANNED_INFERENCE_NOT_A_WINNER", assignmentCounts: ledger.variants.map((v) => v.assigned), limitation: "Coverage bounds and a stated operational guardrail only. No randomization audit, SRM significance test, power calculation, confidence interval or causal estimate." };
}
const result = reviewExperiment(experimentLedger);
assert.equal(result.variants[0].observedOnlyRate, 0.3);
assert.equal(result.variants[1].observedOnlyRate, 0.4);
assert.deepEqual(result.variants[1].completionBounds, [0.2, 0.7]);
assert.equal(result.variants[1].errorUserRate, 0.09);
assert.equal(result.decision, "HOLD_NO_WINNER_CLAIM");
console.log(JSON.stringify(result, null, 2));
```

## Worked denominator review and decision
A naive logged-outcome comparison says control 300/1000=30% and treatment 200/500=40%, a 10 percentage-point difference (not 10% relative lift). But assignment counts are 1000/1000: treatment has 500 unobserved outcomes. Its full-population completion rate could range from 200/1000=20% to 700/1000=70% under the stated binary-outcome/count assumptions. That is a missing-data bound, not a confidence interval. Treating every unobserved user as a failure and claiming 20% would be another unsupported point estimate.

Equal supplied assignment counts do not prove correct randomization, stable identities or absence of leakage. The differential outcome coverage requires investigating eligibility/exposure/logging/follow-up. It is not by itself a performed assignment SRM significance test. Keep assignment, outcome-observed and guardrail denominators separate rather than filtering control until it resembles treatment.

The independently complete guardrail ledger has users-with-error 20/1000=2% in control and 90/1000=9% in treatment. Treatment exceeds this fictional precommitted 5% operational stop threshold. The output is HOLD for both incomplete outcome coverage and that guardrail breach, even though the naive metric looks better. It does not prove statistical significance or a causal treatment harm; it enforces the supplied safety rule while the data/design are reviewed.

Before an experiment, specify hypothesis, eligible population/randomization unit, exposure and outcome windows, primary metric/denominator, power/duration and analysis method, guardrails, data-quality checks and stop/decision rules. Here those statistical design checks are not executed, so even complete counters and a passing guardrail would mean ready for the planned inference, not a winner. Do not keep peeking until a favorable result appears, move goalposts after observation or roll out from these synthetic aggregates. No live users, traffic, telemetry collection or feature flag were accessed.

## Evidence and follow-up, not automatic rollout
[Microsoft ExP pre-experiment research](https://www.microsoft.com/en-us/research/articles/patterns-of-trustworthy-experimentation-pre-experiment-stage/) and [data-quality research](https://www.microsoft.com/en-us/research/articles/data-quality-fundamental-building-blocks-for-trustworthy-a-b-testing-analysis/) inform the distinction between a metric movement and trustworthy analysis; the counts/bounds here are original. Return the versioned experiment contract, eligibility/assignment/exposure/outcome reconciliation, guardrail result, actual statistical method/results if performed, limitations and owner decision. Missing data, unsupported inference or breached stop rules remains HOLD. Do not change traffic, tracking consent, retention or a feature flag merely to complete the review.

Real example

Input

Review the complete synthetic two-variant ledger in SKILL.md: equal assigned populations, unequal observed outcomes, and complete error-user counts. Calculate observed-only rates, missing-outcome bounds and the stated guardrail; do not label those bounds a confidence interval or declare a winner.

Expected result

A naive logged-outcome comparison says control 300/1000=30% and treatment 200/500=40%, a 10 percentage-point difference (not 10% relative lift). But assignment counts are 1000/1000: treatment has 500 unobserved outcomes. Its full-population completion rate could range from 200/1000=20% to 700/1000=70% under the stated binary-outcome/count assumptions. That is a missing-data bound, not a confidence interval. Treating every unobserved user as a failure and claiming 20% would be another unsupported point estimate.

Equal supplied assignment counts do not prove correct randomization, stable identities or absence of leakage. The differential outcome coverage requires investigating eligibility/exposure/logging/follow-up. It is not by itself a performed assignment SRM significance test. Keep assignment, outcome-observed and guardrail denominators separate rather than filtering control until it resembles treatment.

The independently complete guardrail ledger has users-with-error 20/1000=2% in control and 90/1000=9% in treatment. Treatment exceeds this fictional precommitted 5% operational stop threshold. The output is HOLD for both incomplete outcome coverage and that guardrail breach, even though the naive metric looks better. It does not prove statistical significance or a causal treatment harm; it enforces the supplied safety rule while the data/design are reviewed.

Before an experiment, specify hypothesis, eligible population/randomization unit, exposure and outcome windows, primary metric/denominator, power/duration and analysis method, guardrails, data-quality checks and stop/decision rules. Here those statistical design checks are not executed, so even complete counters and a passing guardrail would mean ready for the planned inference, not a winner. Do not keep peeking until a favorable result appears, move goalposts after observation or roll out from these synthetic aggregates. No live users, traffic, telemetry collection or feature flag were accessed.

Source evidence

Agent Skills specificationofficialMicrosoft experimentation platform researchofficialMicrosoft ExP — pre-experiment trustworthinessofficialMicrosoft ExP — data quality in A/B analysisofficial