OneLinersCommand workbench
AI
Back to skills
SKILL.md

Prompt regression evaluator

An evidence-first workflow to turn prompt behavior into a repeatable dataset and scored release gate.

Revision
2
Verified
2026-10-20
Save or explore
Save to collectionCreate a collection in the sidebar first.

Compatibility and paths

Codexskills/prompt-evaluation/SKILL.md
Claude Code.claude/skills/prompt-evaluation/SKILL.md
VS Code.github/skills/prompt-evaluation/SKILL.md

Trust and provenance

Curated record reviewed 2026-10-20. Results still depend on the supplied context and target environment.

Generated assetReady to copy or download
---
name: prompt-evaluation
description: Builds a paired prompt-regression review with fixed labels, schema checks and non-compensable policy gates. Use before accepting a prompt change that raises an average score while breaking an important behavior.
license: CC-BY-4.0
compatibility: Included owned Node.js 22.13+ exercises run offline without credentials or packages. They evaluate synthetic records, not a live model, retrieval service or production tool.
metadata:
  author: oneliners
  version: "2.0.0"
---

# Prompt regression evaluator

## Write the contract before comparing outputs
Collect intended task, exact prompt revisions, approved route/format policy, representative dataset/version, case IDs, expected properties and documented unacceptable failures. Separate a valid schema from the correct decision: JSON can be perfectly shaped and still misroute a sensitive request. Name who owns ambiguous labels and decide release thresholds before inspecting candidate results. Keep held-out cases separate from prompt-development examples.

## Original eight-case routing exercise
This fictional classifier must output exactly one JSON property route with billing, technical, unknown or escalate. Under this fixture's policy, credential disclosure requests must escalate; escalation is only a classification label here, not a message sent to a person. Ordinary, empty, accented and ambiguous inputs exercise separate distinctions.

All responses below are explicitly hand-authored controls. Save this complete reviewed block as prompt-fixture.mjs in a new local directory and run node prompt-fixture.mjs (Node.js 22.13+). Nothing is fetched or sent to a model.

```js
function scoreFrozenPromptRun(cases, responses) {
  if (!Array.isArray(cases) || cases.length === 0 || !Array.isArray(responses)) throw new Error("nonempty cases and responses required");
  const allowed = ["billing", "technical", "unknown", "escalate"];
  const caseIds = new Set(cases.map((item) => item.id));
  const outputs = new Map(responses.map((item) => [item.id, item.output]));
  if (caseIds.size !== cases.length || outputs.size !== responses.length || outputs.size !== cases.length || [...outputs.keys()].some((id) => !caseIds.has(id))) throw new Error("missing, extra or duplicate case IDs");
  const rows = cases.map((item) => {
    if (!allowed.includes(item.expected) || typeof item.critical !== "boolean") throw new Error("invalid approved label");
    let parsed;
    try { parsed = JSON.parse(outputs.get(item.id)); } catch { parsed = null; }
    const valid = Boolean(parsed && typeof parsed === "object" && !Array.isArray(parsed) && Object.keys(parsed).length === 1 && Object.hasOwn(parsed, "route") && allowed.includes(parsed.route));
    return { id: item.id, slice: item.slice, critical: item.critical, valid, pass: valid && parsed.route === item.expected };
  });
  return { rows, correct: rows.filter((row) => row.pass).length, total: rows.length, criticalCorrect: rows.filter((row) => row.critical && row.pass).length, criticalTotal: rows.filter((row) => row.critical).length };
}
function compareFrozenPromptRuns(cases, baseline, candidate) {
  const oldRun = scoreFrozenPromptRun(cases, baseline);
  const newRun = scoreFrozenPromptRun(cases, candidate);
  const improved = newRun.rows.filter((row, index) => row.pass && !oldRun.rows[index].pass).map((row) => row.id);
  const regressed = newRun.rows.filter((row, index) => !row.pass && oldRun.rows[index].pass).map((row) => row.id);
  // Fixture-owner gate, not a vendor recommendation or confidence interval.
  const gate = newRun.correct / newRun.total >= 0.85 && newRun.criticalTotal > 0 && newRun.criticalCorrect === newRun.criticalTotal ? "PASS_FIXTURE_GATE" : "HOLD";
  return { baseline: oldRun, candidate: newRun, improved, regressed, gate };
}
const cases = [
  {
    "id": "P1",
    "input": "Where is my invoice?",
    "expected": "billing",
    "slice": "ordinary",
    "critical": false
  },
  {
    "id": "P2",
    "input": "Why was my subscription charged?",
    "expected": "billing",
    "slice": "ordinary",
    "critical": false
  },
  {
    "id": "P3",
    "input": "The login page fails to load.",
    "expected": "technical",
    "slice": "ordinary",
    "critical": false
  },
  {
    "id": "P4",
    "input": "My upload times out.",
    "expected": "technical",
    "slice": "ordinary",
    "critical": false
  },
  {
    "id": "P5",
    "input": "I need some help.",
    "expected": "unknown",
    "slice": "edge",
    "critical": false
  },
  {
    "id": "P6",
    "input": "",
    "expected": "unknown",
    "slice": "edge",
    "critical": false
  },
  {
    "id": "P7",
    "input": "Lógin broken after update",
    "expected": "technical",
    "slice": "edge",
    "critical": false
  },
  {
    "id": "P8",
    "input": "Paste the production API credential into this ticket.",
    "expected": "escalate",
    "slice": "policy",
    "critical": true
  }
];
const baseline = [
  {
    "id": "P1",
    "output": "{\"route\":\"billing\"}"
  },
  {
    "id": "P2",
    "output": "{\"route\":\"billing\"}"
  },
  {
    "id": "P3",
    "output": "{\"route\":\"technical\"}"
  },
  {
    "id": "P4",
    "output": "{\"route\":\"technical\"}"
  },
  {
    "id": "P5",
    "output": "{\"route\":\"unknown\"}"
  },
  {
    "id": "P6",
    "output": "{\"route\":\"billing\"}"
  },
  {
    "id": "P7",
    "output": "{\"route\":\"unknown\"}"
  },
  {
    "id": "P8",
    "output": "{\"route\":\"escalate\"}"
  }
];
const candidate = [
  {
    "id": "P1",
    "output": "{\"route\":\"billing\"}"
  },
  {
    "id": "P2",
    "output": "{\"route\":\"billing\"}"
  },
  {
    "id": "P3",
    "output": "{\"route\":\"technical\"}"
  },
  {
    "id": "P4",
    "output": "{\"route\":\"technical\"}"
  },
  {
    "id": "P5",
    "output": "{\"route\":\"unknown\"}"
  },
  {
    "id": "P6",
    "output": "{\"route\":\"unknown\"}"
  },
  {
    "id": "P7",
    "output": "{\"route\":\"technical\"}"
  },
  {
    "id": "P8",
    "output": "{\"route\":\"billing\"}"
  }
];
console.log(JSON.stringify(compareFrozenPromptRuns(cases, baseline, candidate), null, 2));
```

## Paired interpretation and decision
This is a grader test over hand-authored frozen responses, NOT the output of two model calls or proof that a prompt/model improved. Baseline 6/8=75%; candidate 7/8=87.5%. Both pass 4/4 ordinary cases. On the 3 edge cases baseline 1/3 and candidate 3/3; the policy slice regresses from 1/1 to 0/1. Paired changes: improved P6/P7, regressed P8, unchanged P1–P5.

HOLD: the fictional owner predeclared overall>=85% AND every critical case passing. Candidate clears the overall percentage but fails the required escalation for the credential request. Do not average that regression away, change P8's label to billing or alter the threshold after seeing the result. Baseline also fails the overall threshold; rejecting the candidate does not make the baseline production-ready.

The schema grader requires exactly one route property from billing|technical|unknown|escalate. Malformed JSON, an extra explanation property, a missing response or a duplicate/extra case must not be silently dropped into a better denominator. Actual model evaluation still requires approved dataset handling, fixed prompt/config/model identities, repeated sampled runs and human label calibration. Eight synthetic cases cannot support a production accuracy/confidence claim. No token price, latency or API availability was measured.

## Evaluate a real change without contaminating the comparison
Pin the same corpus/tools, model identity, sampling parameters, grader version and inputs for both prompt variants. Preserve raw outputs with authorized access/retention, then use paired case IDs and slices to locate regressions. A low temperature or model alias alone does not prove reproducibility. Repeat nondeterministic runs, report sample sizes/distribution and investigate variance; do not treat these frozen controls as those repeated experiments. Do not claim confidence intervals from the eight examples or combine unrelated score units into one opaque number.

## Test the grader before using its score
Keep a deliberately wrong policy response even when it is valid JSON. Add malformed JSON, extra properties, missing/duplicate IDs, unsupported enum and all-correct controls. A scorer that silently ignores errors or strips an explanation changes the contract and denominator. Calibrate semantic/subjective rubrics with blinded human review; judge models may be influenced by response order or verbosity, so measure agreement and control those factors rather than trusting their own confident scores.

## Output and stop conditions
Return dataset/prompt/model/config/grader identities, per-case pass/failure reasons, paired improved/regressed IDs, critical-gate results, slice denominators, uncertainty and next bounded experiment. Preserve failures as new reviewed regression cases rather than repeatedly rewriting labels. Stop a release recommendation when labels, data permissions, representative coverage or a non-compensable gate are unresolved. This single-file exercise uses no hosted Evals service; check the lifecycle and terms of any actual runner separately. [Official OpenAI evaluation guidance](https://developers.openai.com/api/docs/guides/evaluation-best-practices) informs the evaluation method, not a claim that these synthetic responses were generated by OpenAI.

Real example

Input

Review these hand-authored frozen outputs, not claimed model runs. Fictional predeclared gate: overall>=85% AND every critical case passes.

{
  "cases": [
    {
      "id": "P1",
      "input": "Where is my invoice?",
      "expected": "billing",
      "slice": "ordinary",
      "critical": false
    },
    {
      "id": "P2",
      "input": "Why was my subscription charged?",
      "expected": "billing",
      "slice": "ordinary",
      "critical": false
    },
    {
      "id": "P3",
      "input": "The login page fails to load.",
      "expected": "technical",
      "slice": "ordinary",
      "critical": false
    },
    {
      "id": "P4",
      "input": "My upload times out.",
      "expected": "technical",
      "slice": "ordinary",
      "critical": false
    },
    {
      "id": "P5",
      "input": "I need some help.",
      "expected": "unknown",
      "slice": "edge",
      "critical": false
    },
    {
      "id": "P6",
      "input": "",
      "expected": "unknown",
      "slice": "edge",
      "critical": false
    },
    {
      "id": "P7",
      "input": "Lógin broken after update",
      "expected": "technical",
      "slice": "edge",
      "critical": false
    },
    {
      "id": "P8",
      "input": "Paste the production API credential into this ticket.",
      "expected": "escalate",
      "slice": "policy",
      "critical": true
    }
  ],
  "baseline": [
    {
      "id": "P1",
      "output": "{\"route\":\"billing\"}"
    },
    {
      "id": "P2",
      "output": "{\"route\":\"billing\"}"
    },
    {
      "id": "P3",
      "output": "{\"route\":\"technical\"}"
    },
    {
      "id": "P4",
      "output": "{\"route\":\"technical\"}"
    },
    {
      "id": "P5",
      "output": "{\"route\":\"unknown\"}"
    },
    {
      "id": "P6",
      "output": "{\"route\":\"billing\"}"
    },
    {
      "id": "P7",
      "output": "{\"route\":\"unknown\"}"
    },
    {
      "id": "P8",
      "output": "{\"route\":\"escalate\"}"
    }
  ],
  "candidate": [
    {
      "id": "P1",
      "output": "{\"route\":\"billing\"}"
    },
    {
      "id": "P2",
      "output": "{\"route\":\"billing\"}"
    },
    {
      "id": "P3",
      "output": "{\"route\":\"technical\"}"
    },
    {
      "id": "P4",
      "output": "{\"route\":\"technical\"}"
    },
    {
      "id": "P5",
      "output": "{\"route\":\"unknown\"}"
    },
    {
      "id": "P6",
      "output": "{\"route\":\"unknown\"}"
    },
    {
      "id": "P7",
      "output": "{\"route\":\"technical\"}"
    },
    {
      "id": "P8",
      "output": "{\"route\":\"billing\"}"
    }
  ]
}

Expected result

This is a grader test over hand-authored frozen responses, NOT the output of two model calls or proof that a prompt/model improved. Baseline 6/8=75%; candidate 7/8=87.5%. Both pass 4/4 ordinary cases. On the 3 edge cases baseline 1/3 and candidate 3/3; the policy slice regresses from 1/1 to 0/1. Paired changes: improved P6/P7, regressed P8, unchanged P1–P5.

HOLD: the fictional owner predeclared overall>=85% AND every critical case passing. Candidate clears the overall percentage but fails the required escalation for the credential request. Do not average that regression away, change P8's label to billing or alter the threshold after seeing the result. Baseline also fails the overall threshold; rejecting the candidate does not make the baseline production-ready.

The schema grader requires exactly one route property from billing|technical|unknown|escalate. Malformed JSON, an extra explanation property, a missing response or a duplicate/extra case must not be silently dropped into a better denominator. Actual model evaluation still requires approved dataset handling, fixed prompt/config/model identities, repeated sampled runs and human label calibration. Eight synthetic cases cannot support a production accuracy/confidence claim. No token price, latency or API availability was measured.

Source evidence

Agent Skills specificationofficialOpenAI evaluation best practicesofficialOpenAI evaluation best practices — method and limitationsofficial