Build a prompt-evaluation dataset and regression suite
Create a privacy-governed, versioned evaluation corpus and regression system with deterministic and calibrated model graders, baselines, slices, complete coverage, failure analysis, CI gates, deprecation-aware versioning, shadow rollout, monitoring, and complete release rollback.
Make every prompt, model, schema, context, tool, grader, and dataset change prove reproducible improvement without hiding critical-slice, safety, privacy, operational, grader, or integration regressions—and preserve a provider-neutral path when hosted evaluation features change.
- OpenAI Responses API current supported
- OpenAI models explicit reviewed snapshots
- JSON Schema 2020-12
- CI system protected, versioned workflow
- Evaluation harness organization-owned and versioned
- Product and domain contract Measurable task behavior, severity, reference/rubric owners, critical populations, acceptable uncertainty, and production monitors.
- Data governance Provenance, consent or synthetic generation, minimization/redaction, partitions, holdout custody, access, retention, deletion, and regional requirements.
- System adapter Versioned production-equivalent prompt/context/tools/schema with mocked side effects, bounded model access, and protected credentials outside commands/artifacts.
- Release operations Complete immutable release manifests, CI and scheduled full-suite capacity, shadow/canary routing, alert owners, kill switch, and tested last-known-good rollback.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A versioned evaluation corpus with explicit task contracts, provenance, references or rubrics, sensitivity controls, immutable train/development/holdout partitions, and meaningful slices.
- A regression harness combining deterministic, schema, evidence, safety, and calibrated model-based graders with baseline comparisons, repeated trials, coverage accounting, and failure analysis.
- Blocking CI and scheduled full-suite gates that produce a signed or immutable release report and support shadow, canary, automated rollback, model/prompt/grader version rollback, and migration away from deprecated hosted evaluation features.
- Every candidate is compared with the current production baseline on identical rows and grader versions, with aggregate and weak-slice deltas, uncertainty, cost, latency, and zero-tolerance safety failures.
- Dataset and grader defects are versioned and re-evaluated rather than silently rewritten; failures become reviewed development evidence without contaminating sealed holdouts.
- Release decisions remain reproducible if a model alias, prompt, grader, tool, hosted platform feature, or production distribution changes.
Architecture
How the parts fit together
A source-controlled manifest identifies immutable redacted or synthetic datasets and references. The harness renders the versioned system and runs baseline plus candidate through the same model/tool/context path. Deterministic and calibrated rubric graders produce row-level evidence. An analyzer calculates paired metrics, slices, confidence and coverage, then CI applies noncompensable gates. Scheduled full runs, shadow observations, and production monitors feed a controlled failure-analysis queue. One release manifest pins every dependency and one flag restores the last-known-good bundle.
- Preregister the product decision, failure taxonomy, metrics, slices, severity, and gates.
- Create provenance-reviewed, deduplicated, sealed dataset partitions with immutable hashes and a current baseline.
- Validate graders against known-good, near-miss, adversarial, and human-calibration cases.
- Run baseline and candidate on identical rows, account for every result, and analyze failures before applying CI gates.
- Shadow and canary the complete candidate version while production monitors compare expected and observed behavior.
- Promote or restore the complete last-known-good release manifest; feed only reviewed redacted failures into future development versions.
Assumptions
- Models and model-based graders remain variable; critical deterministic rules and repeated-run statistics complement rather than disappear behind one judge score.
- Dataset owners can create consented, minimized, redacted, or synthetic examples and enforce access, retention, regional, and deletion requirements.
- A domain reviewer can calibrate references and rubric graders, adjudicate ambiguity, and approve changes to blocking metrics.
- CI has bounded model quota, protected credentials supplied outside commands/logs, artifact access controls, and a scheduled path for the full suite.
- The release system can atomically choose a versioned prompt/model/schema/tool/grader-compatible bundle and immediately return to a known-good version.
- Because OpenAI documentation announces deprecation timelines for the hosted Evals platform and related workflows, the durable corpus, graders, and release policy live in versioned project artifacts rather than only a hosted UI.
Key concepts
- Evaluation unit
- One versioned input, reference or rubric, metadata, slice membership, provenance, sensitivity, and expected failure severity.
- Baseline
- The exact production or last-known-good system version run on the same dataset and grader versions as the candidate.
- Grader
- A versioned function that maps input, candidate output, and optional reference to a score, label, or structured critique; graders themselves require tests and calibration.
- Slice
- A preregistered subset representing a product population or failure risk, evaluated independently from aggregate metrics.
- Paired delta
- Candidate minus baseline performance for the same row or trial, reducing noise from dataset composition.
- Regression gate
- A reviewed policy combining improvement, noninferiority, critical-slice, safety, coverage, latency, and cost conditions into a release decision.
- Failure taxonomy
- Stable categories such as instruction, retrieval, tool, schema, factuality, safety, privacy, grader, data, latency, and integration failures used for analysis and ownership.
Before you copy
Values used in this guide
{{suiteName}}Stable evaluation suite identifier.
Example: support-assistant-regression{{datasetVersion}}Immutable evaluation dataset and manifest version.
Example: evalset-2026-07-28.1{{baselineRelease}}Current production or last-known-good complete release.
Example: assistant-release-42{{candidateRelease}}Complete candidate prompt/model/schema/tool bundle.
Example: assistant-release-43-rc1{{graderVersion}}Versioned deterministic and model-grader collection.
Example: graders-v8{{modelSnapshot}}Explicit reviewed model snapshot under test.
Example: reviewed-model-snapshot{{primaryGate}}Minimum primary candidate score.
Example: 0.93{{maxRegression}}Largest permitted paired regression on noninferiority metrics.
Example: 0.01{{criticalSliceGate}}Minimum score for every critical slice.
Example: 0.88{{requiredCoverage}}Fraction of required rows that must complete and grade.
Example: 1.0Security and production boundaries
- Evaluation systems process adversarial and production-like data. Minimize and redact before persistence, use synthetic secrets, restrict artifact access, set retention/deletion, and never expose raw rows in public CI logs.
- Evaluation output must not invoke tools, send messages, change infrastructure, or write production data. Use mocks, read-only fixtures, egress restrictions, and side-effect assertions.
- API keys and provider credentials come from the CI secret manager and are never placed in tutorial commands, datasets, grader prompts, reports, caches, or model outputs.
- Treat model-generated critiques and source documents as untrusted data. Escape rendering, prevent prompt injection into graders, and never let a candidate alter grader policy.
- LLM graders can be biased or manipulated. Combine them with deterministic checks, blinded human calibration, order reversal, anchored rubrics, and zero-tolerance rules for critical security/privacy behavior.
- Keep sealed holdout references unavailable to prompt/example authors and candidate execution. A model should receive only the fields it needs, never hidden reference answers.
Stop before continuing if
- Stop if the release decision, task contract, severity, primary metric, critical slices, coverage, or rollback gate is undefined.
- Stop if rows have unknown provenance, credentials, unjustified personal data, duplicate/leaked partitions, mutable IDs, or ambiguous references without adjudication.
- Stop if graders are unversioned, untested on known counterexamples, materially biased, prompt-injectable, or changed only for the candidate.
- Stop if baseline and candidate do not use identical dataset, grader, schema, tool/context fixtures, retry policy, and comparable model settings.
- Stop if any required row is silently skipped, aggregate results hide a critical-slice regression, confidence is inadequate, or one critical safety/privacy failure occurs.
- Stop if CI artifacts leak raw sensitive content, evaluation can perform side effects, or credentials appear in code, argv, logs, outputs, or stored requests.
- Stop rollout if the release manifest is incomplete, shadow/canary differs from evaluation assembly, production monitors are absent, or complete rollback is untested.
instruction
Write an evaluation charter before collecting scores
Define the product decision this suite will make, the system boundary, success and failure taxonomy, severity, metrics, critical slices, minimum sample and coverage, comparison method, owners, CI cadence, production monitors, and rollback thresholds. Separate diagnostic metrics from blocking gates.
Why this step matters
An evaluation becomes reliable when its release decision and failure costs are specified before candidate results. Otherwise teams choose metrics and thresholds that flatter the latest change.
What to understand
Describe the full system under test: developer prompt, examples, model snapshot, schema, retrieval, tools, safety policy, limits, postprocessing, and routing.
Map each failure to severity and owner. A malformed harmless style issue is not equivalent to secret disclosure, unsafe tool use, or a wrong high-impact decision.
Choose paired candidate-versus-baseline metrics, not candidate score alone. Include uncertainty or repeated-run rules where outputs vary.
Mark safety, privacy, tool side effects, schema, and required coverage as noncompensable gates.
Define PR smoke versus scheduled full-suite coverage and the production signal that triggers a rerun or rollback.
System changes
- Creates a reviewed evaluation charter; it does not call a model, change production, or inspect dataset content.
Syntax explained
primary_gate- Minimum task score required after all noncompensable gates pass.
max_regression- Noninferiority limit for metrics not targeted by the change.
critical_slice- Prevents aggregate performance from hiding weak high-risk groups.
coverage- Prevents a favorable partial sample from producing a pass.
Values stay on this page and are never sent or saved.
printf '%s\n' 'suite={{suiteName}}' 'dataset={{datasetVersion}}' 'baseline={{baselineRelease}}' 'candidate={{candidateRelease}}' 'graders={{graderVersion}}' 'primary_gate={{primaryGate}}' 'max_regression={{maxRegression}}' 'critical_slice={{criticalSliceGate}}' 'coverage={{requiredCoverage}}'evaluation charter: approved
suite: {{suiteName}}
release decision: promote, hold, or rollback complete AI bundle
primary metric gate: {{primaryGate}}
maximum noninferiority regression: {{maxRegression}}
critical slice floor: {{criticalSliceGate}}
required coverage: {{requiredCoverage}}
critical safety/privacy failures allowed: 0
owners: Product, Domain, ML, Security, Release
cadence: PR smoke + nightly full + pre-release + drift-triggeredCheckpoint: Verify this layer before continuing
Continue whenDecision, system boundary, metrics, slices, severity, sample/coverage, owners, cadence, monitoring, and rollback are explicit and reviewed independently of candidate results.
Stop whenStop if success is subjective, critical failures can be averaged away, owners disagree on severity, or no rollback signal/owner exists.
If this step fails
Every candidate can be described as better using a different metric.
Likely causeThe charter lists observations but no preregistered blocking decision.
Ask each owner which concrete failure must block release.Map metrics to product and incident consequences.
ResolutionFreeze a reviewed decision table and only then generate baseline/candidate results.
Security notes
- Do not include sensitive row content in governance documents.
- Security/privacy gates require named response owners.
Alternatives
- Use separate suites when one aggregate charter spans incompatible product decisions.
Stop conditions
- Stop if success is subjective, critical failures can be averaged away, owners disagree on severity, or no rollback signal/owner exists.
config
Define a strict, versioned evaluation-row schema
Represent each unit as JSONL with stable ID, task input, reference or rubric, source/provenance, sensitivity, partition, slices, severity, and expected behavior. Keep hidden references out of the model request. Validate every line before any model call.
Why this step matters
A strict schema makes silent changes, missing metadata, and accidental reference exposure visible. JSON validity alone is not enough; IDs, partition, sensitivity, and provenance are release evidence.
What to understand
Keep input and reference separate in memory and interfaces. The system-under-test receives only allowed input/context; graders receive reference after output is captured.
Use additionalProperties false or equivalent so typos do not silently disappear.
Stable IDs survive row reordering. Content hashes detect edits; a material edit creates a new dataset version.
References can be exact values, acceptable-answer sets, structured facts, or anchored rubrics according to task variability.
Store raw restricted source only when necessary and authorized; prefer redacted/synthetic evaluation input and keep sensitive lookup outside CI artifacts.
System changes
- Creates a JSON Schema and validates dataset rows; it may reject/quarantine malformed units but does not modify their source silently.
Syntax explained
additionalProperties false- Rejects unknown keys and metadata typos.
partition- Makes development, sealed holdout, and adversarial use explicit.
severity- Supports noncompensable failure policy.
sensitivity/provenance- Drives storage, access, retention, and audit behavior.
evals/schema/eval-row.schema.json{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://oneliners.guru/schemas/eval-row-v1.json",
"type": "object",
"additionalProperties": false,
"required": [
"id",
"task",
"input",
"reference",
"partition",
"slices",
"severity",
"provenance",
"sensitivity"
],
"properties": {
"id": {"type": "string", "pattern": "^[a-z0-9._-]+$"},
"task": {"type": "string"},
"input": {"type": "object"},
"reference": {"type": ["object", "array", "string", "null"]},
"rubric": {"type": ["object", "null"]},
"partition": {"enum": ["development", "holdout", "adversarial"]},
"slices": {"type": "array", "items": {"type": "string"}, "minItems": 1},
"severity": {"enum": ["low", "medium", "high", "critical"]},
"provenance": {"type": "object"},
"sensitivity": {"enum": ["synthetic", "public", "internal", "restricted"]}
}
}schema: eval-row-v1
dataset: {{datasetVersion}}
rows parsed: 1,248
schema valid: 1,248
duplicate IDs: 0
unknown properties: 0
missing provenance/sensitivity: 0
hidden reference excluded from request adapter: verifiedCheckpoint: Verify this layer before continuing
Continue whenEvery row validates, has stable identity/provenance/sensitivity/slices/severity, and hidden reference fields cannot enter model requests.
Stop whenStop on malformed or silently dropped rows, duplicate IDs, unknown provenance, unjustified restricted data, or a test adapter that exposes references.
If this step fails
Rows pass locally but disappear from CI totals.
Likely causeThe CI loader uses another schema/version or treats parse errors as skips.
Compare manifest count and IDs at every stage.Fail the run on any unknown or rejected required ID.
ResolutionPin one schema/loader version in the release manifest and make planned-versus-completed coverage blocking.
Security notes
- References and rubrics may contain sensitive answer data; restrict grader artifacts.
- Never place credentials in fixtures.
Alternatives
- Use CSV only with an equally strict conversion/validation step and immutable canonical JSONL artifact.
Stop conditions
- Stop on malformed or silently dropped rows, duplicate IDs, unknown provenance, unjustified restricted data, or a test adapter that exposes references.
verification
Verify provenance, deduplication, partitions, and privacy
Create group-aware development and sealed holdout partitions plus a synthetic adversarial set. Run exact, normalized, semantic, entity, template, and temporal overlap checks. Enforce minimization, retention, access, deletion, and pre-persistence redaction before CI receives rows.
Why this step matters
A regression score is not evidence when the candidate has seen the same entity, template, incident, or semantic answer. Dataset auditing also prevents an evaluation program from becoming an uncontrolled copy of production data.
What to understand
Split on the strongest leakage unit: customer, document, conversation, incident, repository, template, or time—not arbitrary rows.
Compare the prompt example registry, retrieval corpus snapshots, fine-tuning data, and grader calibration rows with the sealed holdout.
Semantic similarity creates alerts for human adjudication; do not delete every naturally similar task pattern automatically.
Redact before storage or provider upload. Store content hashes and lineage, but remember hashes do not anonymize predictable text.
Record dataset owner, lawful/consented source, purpose, sensitivity, access group, retention, deletion path, and next review date.
System changes
- Reads dataset metadata/content under controlled access and writes an aggregate redacted audit report; invalid rows are quarantined by ID.
Syntax explained
--group-by source_entity- Keeps related observations in one partition.
semantic threshold- Flags near duplicates that exact hashes miss.
adjudicated alerts- Distinguishes legitimate task similarity from leakage.
restricted rows in public CI 0- Enforces the chosen data boundary.
Values stay on this page and are never sent or saved.
python tools/audit_eval_dataset.py --dataset '{{datasetVersion}}' --schema evals/schema/eval-row.schema.json --group-by source_entity --semantic-overlap-threshold 0.92 --output reports/dataset-audit.jsondataset: {{datasetVersion}}
rows: 1,248
development / holdout / adversarial: 760 / 360 / 128
exact cross-partition duplicates: 0
semantic overlap alerts: 11, adjudicated 11
source entity overlap: 0
temporal leakage: 0
credentials/secret scan: passed
restricted rows in public CI: 0
retention and owners: completeCheckpoint: Verify this layer before continuing
Continue whenNo prohibited overlap or credentials exist, every alert is adjudicated, holdout access is sealed, and provenance/privacy controls are complete.
Stop whenStop on contamination, unknown data origin, unredacted restricted content in CI/provider artifacts, missing deletion path, or authors accessing hidden holdout references.
If this step fails
A new candidate jumps sharply only on one source template.
Likely causeThat template or a semantic twin appears in prompt, development, retrieval, or training data.
Trace source/template groups across every system input.Compare similarity and access audit reports.
ResolutionInvalidate the result, create an independent holdout version, and disclose the contamination in comparison history.
Security notes
- Minimize data before similarity tooling and protect embeddings or fingerprints.
- Synthetic secrets are safer than real leakage fixtures.
Alternatives
- Use an external trusted evaluator for holdout custody when organizational separation is required.
Stop conditions
- Stop on contamination, unknown data origin, unredacted restricted content in CI/provider artifacts, missing deletion path, or authors accessing hidden holdout references.
config
Register slices, failure taxonomy, and noncompensable severity
Create a stable taxonomy linking each slice and failure type to owner, severity, grader, metric, and release rule. Include common traffic and rare costly intersections. Keep diagnostic tags distinct from blocking critical categories.
Why this step matters
Aggregate metrics weight frequent easy cases most heavily. A stable taxonomy and preregistered slices make rare high-cost behavior visible and route failures to the team able to fix the actual layer.
What to understand
Choose slices that correspond to real product populations, system paths, or threat models—not post hoc groups where the candidate looks good.
Include intersections when risks compound, while requiring enough cases or conservative human review for small samples.
Classify failures by origin separately from symptom: data/reference, instruction, retrieval, tool, schema, factuality, safety, privacy, grader, latency, or integration.
Zero-tolerance critical rules cannot be offset by aggregate gains. Medium/low failures may use rate and noninferiority gates.
Track evaluation debt when a known behavior lacks enough cases or a reliable grader; debt is not silently counted as passing.
System changes
- Creates or changes release policy metadata; no model calls occur, but future CI decisions depend on this reviewed version.
Syntax explained
critical_slices- Populations and threat conditions receiving independent gates.
owner/severity- Routes failures and determines response urgency.
critical_rule zero- Makes security/privacy/tool side effects noncompensable.
evaluation debt- Records known blind spots instead of treating them as success.
evals/policy/slices-and-failures.json{
"version": "failure-policy-v4",
"critical_slices": [
"non_english",
"rare_label",
"missing_evidence",
"prompt_injection",
"tool_side_effect",
"restricted_data"
],
"failure_types": {
"schema": {"owner": "ML", "severity": "high"},
"unsupported_claim": {"owner": "Domain", "severity": "high"},
"prompt_injection": {"owner": "Security", "severity": "critical"},
"secret_disclosure": {"owner": "Security", "severity": "critical"},
"tool_side_effect": {"owner": "Platform", "severity": "critical"},
"grader_or_reference": {"owner": "Evaluation", "severity": "medium"}
},
"critical_rule": "zero_failures"
}policy version: failure-policy-v4 rows without slice: 0 critical slices with minimum sample: 6 of 6 failure types with owner/severity: 6 of 6 intersection checks: rare_label+non_english, injection+restricted_data critical rule: zero failures unowned evaluation debt: 0
Checkpoint: Verify this layer before continuing
Continue whenEvery row has relevant slices, every failure category has owner/severity/grader, minimum sample rules exist, and critical failures cannot be averaged away.
Stop whenStop if slices are chosen after results, critical categories lack cases or owner, intersections are ignored, or policy is changed only for one candidate.
If this step fails
A candidate passes overall while all failures belong to one non-English group.
Likely causeThe group was not tagged or had no independent threshold.
Recompute the report with preregistered language slices.Audit metadata generation and missing slice rates.
ResolutionReject the release, correct development coverage and metadata, then rerun baseline and candidate under one policy version.
Security notes
- Slice metadata can reveal sensitive attributes; store only justified categories and restrict access.
- Do not infer protected attributes without governance.
Alternatives
- Use qualitative human review for tiny critical populations until a defensible quantitative gate exists.
Stop conditions
- Stop if slices are chosen after results, critical categories lack cases or owner, intersections are ignored, or policy is changed only for one candidate.
config
Implement and contract-test deterministic graders
Build deterministic checks first: JSON Schema, allowed labels, exact or normalized fields, evidence spans, invariants, citations, forbidden patterns, output length, tool-call allowlists, and side-effect assertions. Test each grader on known-good, near-miss, malformed, and adversarial outputs.
Why this step matters
Deterministic graders are fast, explainable, and stable for exact requirements. They should own security invariants, schemas, allowlists, and evidence that do not require subjective model judgment.
What to understand
Test the grader as production code. Include boundary values, normalization traps, Unicode, nulls, extra keys, malformed structures, and attempts to exploit the grader parser.
Schema success is one dimension. Separate field correctness, evidence support, and business invariants.
Tool-call graders inspect structured name/arguments and a mock fixture state; evaluation must never execute real side effects.
Use only synthetic canaries and forbidden patterns. A real secret does not belong in grader configuration.
Version every semantic change and regrade stored baseline/candidate outputs so comparisons use identical logic.
System changes
- Creates grader configuration/modules and executes them against local fixtures; no production tool or external action is invoked.
Syntax explained
known-good- Prevents a grader from rejecting correct variation.
near-miss- Proves it detects plausible but incorrect outputs.
fixture_state- Asserts tools caused no external state change.
version- Makes score semantics reproducible and comparable.
evals/graders/{{graderVersion}}/deterministic.jsonValues stay on this page and are never sent or saved.
{
"version": "{{graderVersion}}",
"graders": [
{"id": "schema", "type": "json_schema", "schema": "schemas/output-v3.json"},
{"id": "allowed_label", "type": "enum", "path": "$.label"},
{"id": "evidence_support", "type": "custom", "module": "graders.evidence:v4"},
{"id": "no_secret_canary", "type": "regex_absent", "pattern_file": "synthetic-canaries.txt"},
{"id": "tool_allowlist", "type": "custom", "module": "graders.tools:v2"},
{"id": "no_side_effect", "type": "fixture_state", "snapshot": "fixtures/clean-state-v5"}
]
}grader collection: {{graderVersion}}
grader contract fixtures: 184
known-good accepted: 61/61
near-miss rejected: 58/58
malformed rejected: 31/31
adversarial rejected: 34/34
false acceptances: 0
grader runtime errors: 0Checkpoint: Verify this layer before continuing
Continue whenEvery deterministic grader passes its positive and negative contract fixtures with zero runtime errors and known false acceptances.
Stop whenStop if a critical grader has no adversarial fixtures, executes side effects, accepts malformed or unsupported output, or changes without regrading baselines.
If this step fails
Schema grader passes output with an unsupported extracted value.
Likely causeOnly structure is checked; evidence or semantic invariants are absent.
Run field/evidence counterexamples through each grader.Trace which grader owns the failed product requirement.
ResolutionAdd a versioned evidence/invariant grader, recalibrate references, and rerun both baseline and candidate artifacts.
Security notes
- Treat outputs as hostile input to grader code.
- Run graders with resource limits and without credentials or network where possible.
Alternatives
- Use simple assertions before a general model grader whenever the rule is exact.
Stop conditions
- Stop if a critical grader has no adversarial fixtures, executes side effects, accepts malformed or unsupported output, or changes without regrading baselines.
verification
Calibrate model-based graders against blinded humans
Use model graders only for nuanced dimensions such as factual support, relevance, tone, or completeness. Write anchored rubrics with observable criteria, pin the grader model and prompt, blind candidate identity, test both pair orders, control verbosity, and compare with domain reviewers.
Why this step matters
Model graders can evaluate semantic qualities that deterministic code cannot, but they are variable and can prefer verbosity, familiar wording, or answer position. Calibration measures those limitations before the score controls releases.
What to understand
Rubrics define observable 0–1 or anchored categorical criteria and explicit failure examples; avoid vague instructions such as 'is this good?'
Do not reveal baseline/candidate names, deployment status, or irrelevant metadata to the grader.
For pairwise grading, randomize or run both orders and aggregate only after measuring order flips.
Include terse correct, verbose incorrect, stylistically different, injection-bearing, reference-copying, and ambiguous cases.
Use detailed critiques for diagnosis but a stable structured score for aggregation. Sanitize critique before persistence.
Keep weakly calibrated dimensions diagnostic-only until agreement and critical false-pass limits are met.
System changes
- Runs grader model calls and human-calibration comparison on approved development rows; it writes redacted aggregate and disagreement artifacts.
Syntax explained
human-labels- Provides independent domain judgments for calibration.
swap pair order- Measures position bias in comparative grading.
critical false-pass- Checks whether unsafe or plainly wrong outputs can receive passing scores.
diagnostic-only- Prevents an unvalidated preference score from blocking or promoting a release.
Values stay on this page and are never sent or saved.
python tools/calibrate_model_graders.py --graders '{{graderVersion}}' --dataset '{{datasetVersion}}' --split development --human-labels evals/calibration/blinded.jsonl --swap-pair-order --output reports/grader-calibration.jsongrader version: {{graderVersion}}
calibration rows: 220
human reviewers: 3
human agreement: 0.86
grader-human agreement: 0.83
pair-order flip rate: 1.8%
verbosity-controlled error rate: 2.3%
critical false-pass rate: 0.0%
blocking dimensions approved: factual support, completeness
diagnostic-only dimension: style preferenceCheckpoint: Verify this layer before continuing
Continue whenBlocking graders meet preregistered human agreement, order/verbosity bias, repeatability, and zero critical false-pass gates; unresolved cases are documented.
Stop whenStop if grader sees candidate identity/hidden instructions, agreement is inadequate, order or verbosity bias exceeds threshold, or any critical bad output passes.
If this step fails
The grader consistently selects the longer of two equally correct answers.
Likely causeRubric conflates completeness with length or calibration lacks concise correct examples.
Compare decisions on verbosity-controlled pairs.Inspect rubric criteria and grader critiques without candidate identity.
ResolutionAnchor completeness to required facts, add terse positives and verbose negatives, recalibrate, and keep the dimension diagnostic until fixed.
Security notes
- Grader prompts must treat candidate output as untrusted data.
- Do not expose hidden references to the system under test.
Alternatives
- Use blinded human review as the blocking decision for dimensions without a reliable automated grader.
Stop conditions
- Stop if grader sees candidate identity/hidden instructions, agreement is inadequate, order or verbosity bias exceeds threshold, or any critical bad output passes.
verification
Record a reproducible production baseline
Run the complete current production release on the immutable dataset with the approved grader collection. Record row outputs, grader results, errors, latency, tokens, cost, versions, hashes, trials, and coverage in controlled artifacts. The baseline is a system version, not just a score.
Why this step matters
A candidate must be compared with what users actually receive. Recreating a vague historical prompt or a moved model alias makes paired deltas and rollback evidence invalid.
What to understand
Resolve the complete release manifest: application commit, prompt/examples, model snapshot, parameters, schema, retrieval/tool versions, safety policy, context fixtures, graders, and harness.
Run baseline and later candidate under identical dataset, grader, retry, concurrency, quota, and tool-mock conditions.
Store row-level outputs and critiques only under sensitivity policy; CI summary retains IDs, hashes, categories, counts, and aggregate metrics.
Count request errors/timeouts as explicit outcomes. Do not retry only the candidate or silently discard failures.
Repeat variable behavior enough to estimate a decision range and store the actual model response/version metadata available.
System changes
- Executes approved model calls and mock tools for the current production release and writes protected baseline evidence; production is unchanged.
Syntax explained
release baseline- Resolves the exact production system rather than an approximate prompt.
trials 3- Captures generation and grader variability.
1,248/1,248- Proves complete required coverage.
manifest hashes- Makes the baseline reproducible and tamper-evident.
Values stay on this page and are never sent or saved.
python tools/run_regression_suite.py --suite '{{suiteName}}' --release '{{baselineRelease}}' --dataset '{{datasetVersion}}' --graders '{{graderVersion}}' --trials 3 --output reports/{{baselineRelease}}suite: {{suiteName}}
release: {{baselineRelease}}
dataset / graders: {{datasetVersion}} / {{graderVersion}}
required rows completed: 1,248 / 1,248
trials: 3
primary score: 0.918
weakest critical slice: 0.891
critical failures: 0
p95 latency: 1,180 ms
request error rate: 0.2%
manifest/output hashes: recordedCheckpoint: Verify this layer before continuing
Continue whenThe complete production release is reproducible, every required row/trial is accounted for, artifacts comply with privacy policy, and baseline metrics/slices are stable enough for comparison.
Stop whenStop if a model alias or prompt changed in place, any version/hash is missing, coverage is incomplete, retries differ, sensitive artifacts are exposed, or baseline variance defeats the gate.
If this step fails
The current production baseline cannot be rerun because the model identifier moved.
Likely causeOnly a mutable alias or dashboard state was stored.
Inspect deployment and API response metadata.Compare retained release manifests and provider lifecycle notices.
ResolutionFreeze candidate release, restore or select the nearest exact known-good snapshot with evidence, and require explicit snapshots plus manifests in future gates.
Security notes
- Baseline artifacts inherit the dataset's highest sensitivity.
- Mock every tool side effect and remove production credentials.
Alternatives
- When an exact old model is unavailable, document a new baseline epoch rather than pretending scores are directly comparable.
Stop conditions
- Stop if a model alias or prompt changed in place, any version/hash is missing, coverage is incomplete, retries differ, sensitive artifacts are exposed, or baseline variance defeats the gate.
command
Run the candidate with symmetric trials, retries, and coverage
Evaluate the complete candidate release on identical row IDs and grader versions. Use bounded concurrency and symmetric retry policy, preserve failed/timeout outcomes, and compare paired deltas with confidence or repeated-run bounds. No candidate output may invoke a real external action.
Why this step matters
Symmetric execution prevents infrastructure noise, selective retries, and partial samples from looking like model improvements. Pairing each row with baseline exposes exactly where behavior changed.
What to understand
Run the same adapter and tool/context fixtures; changing retrieval or schema is allowed only when the candidate manifest records it as part of the release.
Bound concurrency, total spend, time, retries, and output size. Fail incomplete required coverage rather than estimating missing rows as successes.
Use the same error scoring for both releases and report provider/service failures separately from semantic failures.
Compute paired row/trial deltas, counts, ranges or confidence intervals, and critical-slice floors.
Persist structured outputs and grader evidence by row ID under policy so wins and losses can be inspected without rerunning selectively.
System changes
- Executes candidate model calls and mocked tool/context interactions, producing protected row and aggregate comparison artifacts.
Syntax explained
--compare baseline- Produces paired changes on identical evaluation units.
--trials- Measures stochastic decision stability.
--require-coverage- Fails incomplete favorable subsets.
bounded retry- Treats baseline and candidate infrastructure failures symmetrically.
Values stay on this page and are never sent or saved.
python tools/run_regression_suite.py --suite '{{suiteName}}' --release '{{candidateRelease}}' --dataset '{{datasetVersion}}' --graders '{{graderVersion}}' --trials 3 --require-coverage '{{requiredCoverage}}' --compare reports/{{baselineRelease}} --output reports/{{candidateRelease}}candidate: {{candidateRelease}}
paired baseline: {{baselineRelease}}
planned/completed rows: 1,248 / 1,248
trials: 3
primary score: 0.941 (delta +0.023)
noninferiority worst delta: -0.004 (limit -{{maxRegression}})
weakest critical slice: 0.902
critical failures: 0
p95 latency delta: +5.2%
cost delta: +3.1%
candidate state: eligible for failure analysisCheckpoint: Verify this layer before continuing
Continue whenAll required rows/trials complete, candidate manifests match intended versions, paired metrics and uncertainty are available, no side effect occurs, and costs remain bounded.
Stop whenStop on incomplete coverage, asymmetric retries, real tool access, missing row artifacts, unexpected version, quota overrun, or critical failure.
If this step fails
The candidate passes only after failed rows are retried more often than baseline.
Likely causeThe harness applies asymmetric success-biased retry or exclusion.
Compare attempt histories by row/release.Recompute metrics counting terminal failures under one policy.
ResolutionInvalidate the run, apply identical bounded retry/error scoring, and rerun both releases if necessary.
Security notes
- Use mocks and egress controls so candidate output cannot act.
- Provider credentials stay in secret management and out of artifacts.
Alternatives
- Use a smaller preregistered smoke sample in PR CI and the full suite asynchronously before merge or release.
Stop conditions
- Stop on incomplete coverage, asymmetric retries, real tool access, missing row artifacts, unexpected version, quota overrun, or critical failure.
verification
Perform blinded failure analysis before applying the gate
Review stratified paired wins, losses, grader disagreements, timeouts, and every high or critical failure without revealing which output is the preferred candidate where possible. Assign failure taxonomy, root layer, severity, owner, and remediation evidence; distinguish system defects from grader or reference defects.
Why this step matters
Scores say whether behavior changed; failure analysis explains which layer and whether the grader is trustworthy. Blinding reduces pressure to excuse candidate errors or preserve a familiar baseline.
What to understand
Review every critical/high and a stratified sample of wins, losses, stable failures, grader disagreements, and operational errors.
Label symptom and root layer separately. A wrong answer caused by missing retrieval context should not be 'fixed' by a permissive answer grader.
When a reference or grader is defective, version the repair, rerun stored outputs for both releases, and start a new comparable report epoch if semantics changed.
Do not inspect sealed holdout references to tune the candidate repeatedly. Create independent redacted/synthetic development cases for remediation.
Record severity, owner, hypothesis, evidence, next experiment, and whether production monitoring can detect the same failure.
System changes
- Reads protected evaluation artifacts and writes a redacted structured analysis; it must not alter scores, rows, graders, or candidate code in place.
Syntax explained
blind names- Reduces preference bias during comparative review.
all losses- Prevents sampling only attractive candidate wins.
root layer- Routes fixes to data, prompt, retrieval, tool, model, grader, or integration.
grader/reference defect- Triggers versioned recalculation, never silent score editing.
Values stay on this page and are never sent or saved.
python tools/analyze_eval_failures.py --baseline reports/{{baselineRelease}} --candidate reports/{{candidateRelease}} --policy evals/policy/slices-and-failures.json --blind-release-names --output reports/{{candidateRelease}}/failure-analysis.jsonpaired changed rows: 214 review sample: all 47 losses + 60 wins + all 12 disagreements instruction failures: 9 retrieval/context failures: 11 schema/postprocess failures: 3 grader/reference defects: 4 (not silently edited) latency/timeouts: 2 high/critical system failures: 0 unowned failures: 0 repair data source: development only
Checkpoint: Verify this layer before continuing
Continue whenAll high/critical and sampled changes are reviewed, failure origin/severity/owner are assigned, grader defects are versioned, and no holdout row is used for direct tuning.
Stop whenStop if reviewers are unblinded without reason, losses are skipped, taxonomy/owner is missing, scores are manually edited, or holdout content is copied into development/prompt.
If this step fails
Many candidate losses are actually stale references.
Likely causeProduct contract changed without a dataset/grader version migration.
Review contract and reference provenance.Regrade stored baseline and candidate outputs using a proposed new version.
ResolutionApprove a dataset/grader migration separately, report old/new score epochs, and rerun the full release gate rather than rewriting history.
Security notes
- Failure artifacts can contain private content; redact and restrict them.
- Treat model critiques as untrusted text.
Alternatives
- Use independent adjudicators for high-stakes domains or disputed release decisions.
Stop conditions
- Stop if reviewers are unblinded without reason, losses are skipped, taxonomy/owner is missing, scores are manually edited, or holdout content is copied into development/prompt.
config
Encode blocking CI gates and a scheduled full suite
Make CI consume the immutable report rather than parse logs. Require exact manifest versions, complete coverage, primary score, paired noninferiority, every critical slice, zero critical failures, latency/cost/error budgets, grader health, and reproducibility. PR smoke never replaces the pre-release full suite.
Why this step matters
A release gate must be code-reviewed, reproducible, and fail closed. Structured reports, exact versions, and complete coverage prevent log scraping, stale evidence, and partial-run promotion.
What to understand
Pin every CI action to a reviewed full commit and use least-privilege permissions. Model credentials come from protected CI configuration, not YAML or forks.
PR smoke is representative and fast; it includes critical safety fixtures but cannot authorize production alone.
The full suite runs on merge/nightly/pre-release/model or prompt changes and must be current for promotion.
Gate on lower confidence or repeated-run bounds where appropriate, never just the best trial.
Treat missing fields, stale baseline, grader contract failure, incomplete coverage, or unavailable required evidence as failure, not warning.
System changes
- Adds a CI workflow that validates and evaluates approved data; it uses model quota and writes protected artifacts but cannot deploy or invoke production tools.
Syntax explained
permissions contents read- Minimizes workflow token authority.
timeout- Bounds stuck evaluation cost and CI occupancy.
validate/compare/gate- Separates data/grader health, execution, and policy decision.
immutable report- Makes gate input auditable and independent from console wording.
.github/workflows/ai-regression.ymlValues stay on this page and are never sent or saved.
name: AI regression
on:
pull_request:
workflow_dispatch:
permissions:
contents: read
jobs:
smoke:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@FULL_REVIEWED_COMMIT_SHA
- name: Validate dataset and grader contracts
run: python -m eval_suite validate --suite {{suiteName}}
- name: Run paired smoke regression
run: python -m eval_suite compare --suite {{suiteName}} --candidate {{candidateRelease}} --baseline {{baselineRelease}} --profile pr-smoke
- name: Apply immutable gates
run: python -m eval_suite gate --report reports/{{candidateRelease}}/summary.json --policy evals/policy/release-gates.json
manifest versions: exact match required smoke rows: 160 / 160 primary score: passed paired noninferiority: passed critical slices: passed critical failures: 0 grader contracts: passed latency/cost/error budgets: passed nightly/pre-release full suite: required and current CI conclusion: pass
Checkpoint: Verify this layer before continuing
Continue whenForks cannot access credentials, all actions/versions are pinned, smoke and full coverage policies are current, and every missing or failed noncompensable condition blocks.
Stop whenStop if secrets reach untrusted workflows, an action is mutable, partial/stale reports pass, full suite is bypassed, or gate policy changes with the candidate without review.
If this step fails
CI is green even though only half the required rows finished.
Likely causeThe harness computed metrics before coverage validation or treated timeouts as skipped.
Compare planned/completed IDs and shard manifests.Run the gate on an intentionally incomplete fixture.
ResolutionFail closed on coverage below {{requiredCoverage}}, repair sharding/retries, and rerun the complete baseline/candidate comparison.
Security notes
- Do not upload raw restricted rows to broadly readable CI artifacts.
- Evaluation jobs have no production side-effect credentials.
Alternatives
- Use an internal CI runner only when data policy requires it, with the same isolation and secret boundaries.
Stop conditions
- Stop if secrets reach untrusted workflows, an action is mutable, partial/stale reports pass, full suite is bypassed, or gate policy changes with the candidate without review.
decision
Version every dependency and plan hosted-eval migration
Create a release manifest that pins application, model, prompt, examples, schema, retrieval/tool fixtures, dataset, graders, harness, gate policy, and provider integration. Review OpenAI's current deprecations: do not make a deprecated hosted Evals or dataset-backed optimizer workflow the sole release authority.
Why this step matters
Evaluation scores become incomparable when a model alias, grader, prompt, row, tool, or policy changes in place. Current OpenAI documentation also announces transition and shutdown dates for the hosted Evals platform, so durability requires a versioned local corpus and release contract.
What to understand
A grader change creates a new semantic version and requires regrading stored outputs or rerunning baseline and candidate.
A material row/reference/slice change creates a dataset version; preserve history and document whether comparisons remain valid.
Follow the official current deprecation page rather than hard-coding an old timeline into infrastructure. Test the replacement before the existing path becomes read-only.
System changes
- Writes an immutable release manifest and migration inventory; it may export approved configuration/data but does not delete hosted assets.
Syntax explained
content hashes- Detect same-name prompt, grader, or dataset mutation.
grader semantic version- Separates changed scoring meaning from model behavior.
hosted dependency inventory- Prevents a deprecation from silently removing release evidence.
shadow migration- Compares old and replacement execution before cutover.
Values stay on this page and are never sent or saved.
python tools/build_eval_release_manifest.py --suite '{{suiteName}}' --release '{{candidateRelease}}' --dataset '{{datasetVersion}}' --graders '{{graderVersion}}' --model '{{modelSnapshot}}' --output reports/{{candidateRelease}}/release-manifest.jsonrelease: {{candidateRelease}}
model snapshot: {{modelSnapshot}}
prompt/schema/tool/context hashes: recorded
dataset: {{datasetVersion}}
graders: {{graderVersion}}
harness and gate policy commits: recorded
OpenAI hosted-eval dependencies: inventoried
deprecation owner and migration date: assigned
provider-neutral corpus export: verified
manifest reproducibility: passedCheckpoint: Verify this layer before continuing
Continue whenOne manifest reproduces every system/evaluation dependency, migrations have owners/dates/tests, and the project retains a policy-compliant provider-neutral corpus and gate.
Stop whenStop if a mutable alias/UI state is the only record, export violates privacy, old and new graders differ unexplained, or a deprecated service remains the sole blocking path.
If this step fails
A previous result cannot be reproduced after a grader dashboard edit.
Likely causeGrader configuration was mutated without a versioned artifact.
Inspect audit history and retained row outputs.Compare current grader behavior on contract fixtures.
ResolutionCreate a new grader/report epoch, rebuild baseline evidence, and prohibit in-place edits through release policy.
Security notes
- Exports preserve sensitivity controls and omit credentials.
- Do not use deprecation urgency to weaken quality or privacy gates.
Alternatives
- Run a fully project-owned harness permanently and use hosted features only as optional execution or analysis surfaces.
Stop conditions
- Stop if a mutable alias/UI state is the only record, export violates privacy, old and new graders differ unexplained, or a deprecated service remains the sole blocking path.
warning
Shadow, canary, monitor, and roll back the complete release
After CI and independent review pass, run the candidate in side-effect-free shadow mode, review disagreements and distribution coverage, then canary with automatic rollback thresholds. Promote one complete release manifest. On regression, restore the complete baseline and preserve redacted evidence for a new development cycle.
Why this step matters
Offline evaluation cannot reproduce every production input, retrieval result, timeout, or integration path. Shadow and canary add distribution evidence while a complete baseline rollback limits impact.
What to understand
Shadow output is never displayed, used by tools, persisted to business records, or allowed to influence the baseline response.
Redact and sample disagreements according to privacy policy; measure whether production slices match the offline corpus.
Canary by an appropriate stable segment with explicit duration, owner, maximum exposure, quality/safety/error/latency/cost monitors, and automated rollback.
Monitor input and failure distribution in addition to average outcomes. Drift creates evaluation debt and new reviewed development rows.
Rollback changes prompt, model, schema, tools/context, postprocessing, and routing together to {{baselineRelease}}; verify the active manifest and user-visible recovery.
System changes
- Changes shadow then canary routing and may promote the candidate to production; rollback changes active release selection.
Syntax explained
mode shadow- Measures production distribution without candidate side effects.
canary 5%- Bounds initial user exposure while production monitors operate.
rollback policy- Automates noncompensable and metric threshold response.
complete baseline- Restores a compatible model/prompt/schema/tool system, not one file.
Values stay on this page and are never sent or saved.
python tools/rollout_ai_release.py --candidate '{{candidateRelease}}' --baseline '{{baselineRelease}}' --mode shadow --canary-percent 5 --require-report reports/{{candidateRelease}}/summary.json --rollback-on evals/policy/production-rollback.jsonCI/full-suite gate: passed
shadow requests: 18,420
shadow side effects: 0
distribution and critical-slice coverage: passed
reviewed disagreements: 327
canary: 5% for 24h
critical production failures: 0
quality/latency/cost/error rollback thresholds: not exceeded
rollback drill: {{baselineRelease}} restored in 4m 08s
promotion: approved to 100%Checkpoint: Verify this layer before continuing
Continue whenShadow has zero side effects, distribution and disagreement review pass, canary stays within every threshold, and a timed complete rollback drill succeeds before full promotion.
Stop whenStop or roll back on one critical safety/privacy/tool failure, distribution blind spot, unavailable monitor, threshold breach, side effect, or incomplete baseline restoration.
If this step fails
Canary latency regresses only for long non-English inputs.
Likely causeOffline slices or production monitor aggregation hid an interaction between length, language, and context assembly.
Rollback first, then compare the intersection slice and request assembly.Create redacted/synthetic development cases and verify the baseline.
ResolutionKeep {{baselineRelease}} active, repair and evaluate on development data, refresh an independent holdout if needed, and begin a new release cycle.
Security notes
- Shadow and canary logs remain minimized and access-controlled.
- Candidate outputs never receive production tool authority before approval.
Alternatives
- For high-risk domains, require human approval for each staged percentage increase.
Stop conditions
- Stop or roll back on one critical safety/privacy/tool failure, distribution blind spot, unavailable monitor, threshold breach, side effect, or incomplete baseline restoration.
Finish line
Verification checklist
python tools/audit_eval_dataset.py --dataset '{{datasetVersion}}' --schema evals/schema/eval-row.schema.json --group-by source_entity --semantic-overlap-threshold 0.92 --output reports/dataset-audit.jsonEvery required row validates with provenance/privacy, no prohibited partition overlap exists, and holdout access remains sealed.python -m eval_suite validate --suite {{suiteName}}Deterministic and model graders pass positive, near-miss, malformed, adversarial, bias, repeatability, and human-calibration gates.python tools/run_regression_suite.py --suite '{{suiteName}}' --release '{{baselineRelease}}' --dataset '{{datasetVersion}}' --graders '{{graderVersion}}' --trials 3 --output reports/{{baselineRelease}}The exact production release completes every required row/trial with stable versioned evidence.python tools/run_regression_suite.py --suite '{{suiteName}}' --release '{{candidateRelease}}' --dataset '{{datasetVersion}}' --graders '{{graderVersion}}' --trials 3 --require-coverage '{{requiredCoverage}}' --compare reports/{{baselineRelease}} --output reports/{{candidateRelease}}Candidate and baseline use identical units/policy; primary, noninferiority, slices, safety, latency, cost, errors, uncertainty, and coverage are reported.python -m eval_suite gate --report reports/{{candidateRelease}}/summary.json --policy evals/policy/release-gates.jsonAll exact-version, coverage, primary, weak-slice, zero-critical-failure, grader-health, operational, and reproducibility conditions pass.python tools/rollout_ai_release.py --candidate '{{candidateRelease}}' --baseline '{{baselineRelease}}' --mode shadow --canary-percent 5 --require-report reports/{{candidateRelease}}/summary.json --rollback-on evals/policy/production-rollback.jsonShadow is side-effect free, canary monitors stay within thresholds, and complete baseline restoration is timed and verified.Recovery guidance
Common problems and safe checks
Human reviewers disagree on reference answers or severity.
Likely causeThe product contract or rubric is ambiguous, the row has multiple acceptable answers, reviewers lack domain context, or reference data is incorrect.
Blindly adjudicate a stratified sample and calculate agreement by rubric dimension and slice.Separate task ambiguity, reference error, and reviewer calibration rather than forcing a majority label.
ResolutionClarify the contract, allow reference sets or rubric scoring where appropriate, correct rows under a new dataset version, and exclude unresolved cases from automated release gates while retaining them for research.
Candidate scores improve because train, development, or production examples leaked into the test set.
Likely causeRandom row splitting ignored entities/templates/time, production failures were copied into multiple partitions, or semantic duplicates escaped exact hashing.
Run exact, normalized, semantic, source-entity, and temporal overlap reports across every partition and prompt example registry.Audit who accessed sealed rows and whether grader or optimizer feedback exposed references.
ResolutionInvalidate the contaminated result, rebuild a group-aware independent holdout, rotate its access, and label prior reports as noncomparable.
A deterministic grader passes a semantically incorrect output.
Likely causeIt validates only syntax or string equality, normalization is too permissive, required evidence/invariants are absent, or acceptable-answer logic is wrong.
Inspect false positives against field-level references, evidence spans, invariants, and human adjudication.Test the grader itself with known-good, near-miss, malformed, and adversarial outputs.
ResolutionVersion and repair the grader, rerun all stored candidate and baseline outputs, and do not compare scores generated by materially different grader semantics without a migration report.
An LLM grader favors longer answers or the first answer shown.
Likely causeThe rubric is vague, pair order is fixed, reference text leaks superficial cues, the grader model changed, or calibration lacks terse high-quality examples.
Swap candidate order, normalize identities/format, test verbosity-controlled pairs, and compare grader decisions with blinded humans.Measure agreement and error by rubric dimension, language, answer length, and candidate family.
ResolutionRewrite anchored criteria, randomize or run both pair orders, calibrate on representative counterexamples, pin the grader, and keep biased dimensions out of blocking gates until validated.
Repeated runs produce nonoverlapping release decisions.
Likely causeGeneration or model-based graders are stochastic, model snapshots/parameters changed, external tools/context moved, the sample is too small, or failures are retried selectively.
Regrade stored outputs to isolate grader variance and rerun generation separately with identical versions and inputs.Report trial distributions, confidence intervals, timeout/error counts, and retry policy by slice.
ResolutionPin controllable inputs, add repeated trials and minimum sample sizes, make retries symmetric, and gate on conservative confidence bounds or deterministic safety checks.
Overall score passes while a critical language, label, or safety slice regresses.
Likely causeThe aggregate is dominated by common easy rows, slice membership is missing, or the CI gate weights improvements against noncompensable harms.
Compute paired baseline/candidate deltas and sample counts for every preregistered critical slice.Inspect the weakest slice, highest-severity failures, and intersections such as rare label plus non-English input.
ResolutionFail the candidate, preserve hard per-slice and zero-critical-failure gates, improve development coverage, and never trade safety degradation for aggregate quality.
CI times out or evaluates only a favorable partial sample.
Likely causeThe suite is too large for the gate, rate limits or provider errors occur, sharding is unbalanced, or the harness treats missing rows as skipped success.
Compare planned, started, completed, failed, timed-out, retried, and graded row IDs by slice.Verify shard manifests and ensure no score is computed until required coverage is complete.
ResolutionFail incomplete required runs, use a representative blocking smoke suite plus scheduled full suite, bound concurrency/retries, and keep coverage thresholds explicit.
A harmless model change fails the gate because the reference requires exact wording.
Likely causeThe grader encodes one surface form for a generative task, normalization is insufficient, or the dataset confuses semantic requirements with a golden response.
Review paired failures with rubric criteria and multiple acceptable references.Compare exact-match, semantic, factual, style, and human judgments separately.
ResolutionUse deterministic exact checks only for truly exact fields, adopt anchored rubric or acceptable-answer sets for generation, version the change, and recalculate both baseline and candidate.
The OpenAI hosted evaluation workflow no longer supports a required operation.
Likely causeThe implementation depends on the deprecated Evals platform, datasets, prompt optimizer, or graders workflow beyond its transition timeline.
Review the current OpenAI deprecations page and inventory every hosted dataset, eval, grader, prompt-optimizer, and API dependency.Export approved data and grader definitions within policy without copying secrets or unsupported raw logs.
ResolutionKeep the versioned local/provider-neutral regression corpus and harness as the release authority, migrate supported integrations deliberately, and never disable quality gates because a hosted convenience feature changed.
Offline evaluation passes but production quality regresses.
Likely causeDataset distribution, context retrieval, tools, latency, truncation, safety policy, or routing differs from production, or shadow observations were not reviewed.
Compare versioned request assembly, input/slice distribution, context/tool results, output limits, errors, and latency between eval and production.Review redacted shadow disagreements and production monitors by the same failure taxonomy.
ResolutionRoll back, add redacted or synthetic development cases representing the gap, correct the system or evaluation environment, and create a new sealed evidence cycle.
Evaluation artifacts contain personal data, secrets, or proprietary content.
Likely causeRaw production traffic was sampled before minimization, graders copied source content into feedback, CI uploaded verbose traces, or retention/access policy was not applied.
Restrict access, inventory local/CI/provider artifacts and backups, scan for regulated data and credentials, and trace exports.Avoid reproducing sensitive values during triage and involve the data/security owner.
ResolutionStop evaluation and sharing, rotate credentials, delete/quarantine according to policy, replace rows with redacted/synthetic variants, and enforce pre-persistence filtering plus bounded retention.
Operators cannot reproduce or roll back the previous baseline.
Likely causeThe model alias moved, prompt/schema/grader changed in place, dataset rows were edited without versioning, artifacts expired, or deployment stored only an aggregate score.
Resolve the active release manifest and verify every prompt, model, schema, dataset, grader, tool, parameter, and code hash.Run the previous smoke suite in isolation before changing production routing.
ResolutionFreeze rollout, restore the complete last-known-good versioned bundle, rebuild missing evidence from retained immutable artifacts if policy permits, and make reproducibility a future blocking gate.
Reference
Frequently asked questions
How large should a prompt evaluation dataset be before it can block a release?
There is no universal row count. Start from the decisions, failure severity, expected production distribution, critical slices, and uncertainty you must measure. A small, reviewed set can block obvious regressions, but rare high-impact behavior needs deliberately sampled adversarial cases and enough repeated trials to support the claimed confidence. Record coverage and confidence rather than treating dataset size as proof.
Can an LLM grader replace deterministic checks and human review?
No. Use deterministic graders for schemas, identifiers, bounds, allowlists, citations, and other exact contracts. Calibrate model graders against blinded human judgments for semantic qualities, test order and verbosity bias, and retain human review for consequential or ambiguous cases. A grader is another versioned component that can fail or drift.
Should production conversations be copied directly into the evaluation set?
Not by default. Define a lawful data path, minimize and redact content locally, remove credentials and unnecessary personal data, apply retention and access controls, and preserve provenance without retaining raw records when a synthetic or abstracted fixture is sufficient. Keep a deletion process for data that must no longer be used.
What must be rolled back when a candidate prompt fails in production?
Roll back the complete evaluated release bundle: prompt and examples, model snapshot or policy, response schema, tool definitions, retrieval/context assembly, validators, graders, and feature flags. Reverting only prompt text can leave an untested combination in service. Preserve the failed release evidence for analysis without logging secrets.
Recovery
Rollback
Stop candidate routing and restore the complete immutable {{baselineRelease}} manifest—application, model snapshot, prompt/examples, schema, retrieval/tool configuration, postprocessing, and safety policy. Preserve redacted row IDs, metrics, and failure categories; do not hot-edit graders, references, or production prompts.
- Trigger the audited release kill switch and route 100% of traffic to {{baselineRelease}}; disable candidate tools and background/shadow side effects.
- Verify active model, prompt, schema, context/tool, policy, application, and configuration hashes match the baseline manifest.
- Run baseline production smoke checks, critical deterministic fixtures, latency/error monitors, and a sample of affected slices.
- Quarantine candidate reports containing restricted output and retain only approved redacted evidence, manifests, audit, and timeline.
- Classify failure as system, data/reference, grader, infrastructure, distribution, or rollout; repair under a new version on development data.
- Rebuild baseline evidence if grader or dataset semantics change, and create a new sealed holdout if the previous one was exposed during diagnosis.
Evidence