SKILL.md
Flaky test investigator
An evidence-first workflow to separate nondeterminism, shared state, timing, and environment failures.
- Revision
- 2
- Verified
- 2026-10-20
Compatibility and paths
Codex
skills/flaky-test-investigation/SKILL.mdClaude Code
.claude/skills/flaky-test-investigation/SKILL.mdVS Code
.github/skills/flaky-test-investigation/SKILL.mdTrust and provenance
Curated record reviewed 2026-10-20. Results still depend on the supplied context and target environment.
Generated assetReady to copy or download
---
name: flaky-test-investigation
description: Investigates a nondeterministic test using first-failure evidence, fixture ownership and controlled interleavings. Use for intermittent CI failures; separate a reproduced shared-state cause from timing/environment hypotheses without retries-until-green.
license: CC-BY-4.0
compatibility: Offline review and the included owned Node22.13+ exercises need no credentials, network or package installation. Untrusted repository execution requires separate review.
metadata:
author: oneliners
version: "2.0.0"
---
# Flaky test investigator
## Preserve the first failure
Collect exact test/revision, assertion, trace/output, run/attempt, runtime and runner, order/seed, worker count, fixture lifetime and teardown behavior. Compare failed and successful executions of the same target. Do not combine failures from different revisions into a made-up failure rate. A retry success preserves evidence of a failure; it does not diagnose or repair it. Redact credentials/customer data before sharing traces.
## Design one discriminating experiment
Separate shared mutable state, uncontrolled asynchronous ordering, stale UI observations, external service variability and environment/version differences. Change one cause at a time while preserving the assertion. Prefer an explicit readiness event/barrier or an assertion that waits for the required state over a fixed sleep. But waiting cannot repair a violated invariant: an object owned by another test can remain wrong forever. Do not increase timeouts before knowing what state should become true and which component owns it.
## Worked controlled schedule
```json
{
"provenance": "owned deterministic interleaving, not a measured CI flake rate",
"symptom": "Case A expects admin; another case may overwrite shared role to viewer before A asserts.",
"controlledSchedule": [
"A writes admin",
"A yields at explicit barrier",
"B writes viewer",
"B releases A",
"A checks its role"
],
"changedVariable": "shared object vs one object owned by each case"
}
```
## Complete offline reproduction
Save the entire following code as owned-test-state.mjs in a NEW practice directory. Node22.13+ is sufficient; no npm install, app checkout, credentials or network is needed. Review the code first. Run the two commands separately so the deliberately failing first mode does not stop the positive control.
```js
// Synthetic, process-local teaching fixture. No files, network or test service.
import assert from "node:assert/strict";
const mode = process.argv[2];
assert.ok(["shared", "isolated"].includes(mode), "Usage: node owned-test-state.mjs shared|isolated");
const shared = { role: "viewer" };
const fixture = () => mode === "shared" ? shared : { role: "viewer" };
let markAReady, continueA;
const aReady = new Promise((resolve) => { markAReady = resolve; });
const bHasWritten = new Promise((resolve) => { continueA = resolve; });
let checks = 0;
async function caseA() {
const state = fixture();
state.role = "admin";
markAReady();
await bHasWritten;
assert.equal(state.role, "admin", "case A must retain its own role");
checks++;
}
async function caseB() {
await aReady;
const state = fixture();
state.role = "viewer";
continueA();
assert.equal(state.role, "viewer", "case B must retain its own role");
checks++;
}
await Promise.all([caseA(), caseB()]);
assert.equal(checks, 2);
console.log(`${checks} checks passed; mode=${mode}`);
```
Commands (run separately): node owned-test-state.mjs shared; then node owned-test-state.mjs isolated.
## Observed fixture behavior and interpretation
Hypothesis demonstrated in the owned fixture: two asynchronous cases share one mutable object. Under the explicit A-write -> B-write -> A-read interleaving, B changes the value that A later asserts. This is a shared-state race, not evidence that the service is slow or that a random timeout must be increased.
Run shared mode: exits nonzero with ERR_ASSERTION and case A must retain its own role (actual viewer, expected admin). The schedule is deliberately forced, so this is a deterministic failure exposing one possible bad interleaving, NOT an observed probability or a claim that every real run fails.
Run isolated mode: the same barriers/interleaving execute, but each case receives its own object; both assertions pass and the program prints2 checks passed; mode=isolated. The positive control changes ownership, not assertion meaning, retries or sleep duration. No browser, pytest, network, database or real CI worker was involved.
Proposed investigation for a real suite: preserve first failing trace/log, run/seed/order/worker count, fixture setup/teardown ownership and the exact assertion. Compare an isolated case with the affected bounded group, then deliberately control the suspected interleaving. If isolation fixes this reproduction, inspect lifecycle/global caches/shared users/files before changing timeouts. Do not conclude every failure with the same assertion has this cause; absent selectors, service errors and differing environments need their own evidence.
Stability gate: keep the forced negative control against the shared variant and the positive control against isolated state, then run the real affected group under its supported parallel/order matrix and a predeclared repetition budget. Report all results; zero failures in a finite sample is not proof of zero flakiness. Do not retry until green, skip the failure, suppress the assertion or globally serialize the entire product as an unexplained fix.
## Counterexamples and real-suite follow-up
If a failure persists with truly per-case state and the same controlled schedule, this particular sharing hypothesis is not enough. Examine assertion semantics and other globals. If a UI element is simply not yet ready, use the framework's observable readiness assertion and preserve the trace; do not transfer this object-ownership fix without evidence. In pytest, inspect fixture scopes, mutable globals and order dependencies; this Node reproduction is not a native pytest experiment. In a database-backed suite, distinct process objects are insufficient when tests still share the same database rows or external account.
## Output and authority
Return observed failure, competing hypotheses, controlled variable/schedule, negative/positive result, fixture ownership correction, real-suite stability gate, untested environments and owner. Explain any temporary quarantine with expiry and tracked risk rather than silently calling it repaired. Do not disable checks or execute an unfamiliar test repository's hooks during review. [pytest's flakiness discussion](https://docs.pytest.org/en/stable/explanation/flaky.html) provides investigation context; [Node assertions](https://nodejs.org/api/assert.html) are the actual primitive used here, tested on Node22.22.2 rather than an assumed current runtime.
Real example
Input
Investigate this controlled shared-state schedule, without inventing a CI failure rate.
{
"provenance": "owned deterministic interleaving, not a measured CI flake rate",
"symptom": "Case A expects admin; another case may overwrite shared role to viewer before A asserts.",
"controlledSchedule": [
"A writes admin",
"A yields at explicit barrier",
"B writes viewer",
"B releases A",
"A checks its role"
],
"changedVariable": "shared object vs one object owned by each case"
}Expected result
Hypothesis demonstrated in the owned fixture: two asynchronous cases share one mutable object. Under the explicit A-write -> B-write -> A-read interleaving, B changes the value that A later asserts. This is a shared-state race, not evidence that the service is slow or that a random timeout must be increased. Run shared mode: exits nonzero with ERR_ASSERTION and case A must retain its own role (actual viewer, expected admin). The schedule is deliberately forced, so this is a deterministic failure exposing one possible bad interleaving, NOT an observed probability or a claim that every real run fails. Run isolated mode: the same barriers/interleaving execute, but each case receives its own object; both assertions pass and the program prints2 checks passed; mode=isolated. The positive control changes ownership, not assertion meaning, retries or sleep duration. No browser, pytest, network, database or real CI worker was involved. Proposed investigation for a real suite: preserve first failing trace/log, run/seed/order/worker count, fixture setup/teardown ownership and the exact assertion. Compare an isolated case with the affected bounded group, then deliberately control the suspected interleaving. If isolation fixes this reproduction, inspect lifecycle/global caches/shared users/files before changing timeouts. Do not conclude every failure with the same assertion has this cause; absent selectors, service errors and differing environments need their own evidence. Stability gate: keep the forced negative control against the shared variant and the positive control against isolated state, then run the real affected group under its supported parallel/order matrix and a predeclared repetition budget. Report all results; zero failures in a finite sample is not proof of zero flakiness. Do not retry until green, skip the failure, suppress the assertion or globally serialize the entire product as an unexplained fix.