Optimize LLM cost, latency, caching and context safely
Measure production token economics and user latency, design cacheable prompts and context budgets, and ship only quality-gated optimizations.
Reduce cost per successful task and tail latency without sacrificing evidence, privacy, safety or task quality.
- OpenAI Responses API current
- Node.js 20 LTS, 22 LTS
- Quality baseline Representative task evals and critical safety gates exist.
Review {{evalDataset}}. - Usage telemetry Response usage, timing, retries and release metadata are captured without raw sensitive logging.
Audit {{controlRelease}} telemetry. - Versioned pricing Current official pricing is represented in operational config with effective dates.
Verify pricing configuration. - Controlled rollout Shadow/canary routing and complete release rollback are tested.
Verify {{controlRelease}} restore.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A production measurement model that joins user outcome, model and prompt revisions, request path, input/output/reasoning/cached token usage, time to first useful output, end-to-end latency, retries, cache writes and reads, and calculated cost. It reports distributions and cohorts rather than misleading averages.
- A context budget and prompt-caching design. Stable instructions, tools, schemas and examples form an exact reusable prefix; dynamic user and retrieved data come later. Retrieval selects only current, authorized and relevant evidence, while every cache key and retention choice receives privacy review.
- A quality-gated optimization ladder: remove unnecessary model calls, shorten generated output, select smaller models only after baseline quality, combine or parallelize independent work, stream useful progress, move asynchronous work to Batch or flex processing where appropriate, and roll back any change that breaches task quality or safety.
- The team knows cost per successful task, p50/p95/p99 latency, time to first useful output, token composition, cache read/write economics, retry amplification and quality by workflow, model and release.
- Requests stay within a reviewed context budget. Oversized, stale, duplicate or unauthorized context is rejected or pruned by deterministic policy, and context-window availability is never mistaken for a recommendation to fill it.
- Cache improvements are proven from reported cached_tokens and cache_write_tokens. Exact prefixes and stable keys are designed intentionally, high-volume keys are partitioned, and cache retention is assessed under the organization's current data policy.
- No cheaper or faster release ships without candidate-versus-control evals, critical safety review and canary observation. A smaller model, shorter output or pruned context is accepted only when task-specific quality remains inside approved gates.
Architecture
How the parts fit together
Every model request passes through a release-aware adapter and a context assembler. The assembler validates the task, selects authorized evidence, enforces per-section token budgets and renders a stable prefix followed by dynamic content. The adapter pins model and service settings, records wall-clock phases and usage fields, and streams when partial output is useful. A cost ledger applies versioned pricing configuration outside the prompt. An evaluation harness compares candidate and control on representative tasks and critical cases. The rollout system selects one optimization at a time, runs shadow and canary cohorts, observes quality, cost and latency jointly, and restores the previous complete release when a gate fails.
- Define user-visible SLOs, quality gates and cost per successful task before selecting optimizations.
- Instrument the existing control and collect a representative baseline with phase timing and complete usage metadata.
- Build a context ledger, set per-section budgets and remove duplicate, stale, low-value and unauthorized inputs.
- Render static instructions, examples, schemas and tool definitions first; append dynamic history, retrieval and user request afterward.
- Measure cache reads and writes, output length, request count, time to first useful output and full task latency.
- Evaluate candidates in a ladder: deterministic path, fewer calls, smaller model, shorter output, pruning, caching, parallelization and asynchronous processing.
- Run each candidate against frozen quality/safety evals, then shadow and canary it while comparing cost and latency distributions.
- Promote only improvements that preserve gates, continuously monitor drift and restore the previous bundle when quality or privacy regresses.
Assumptions
- The application can capture response usage and timestamps in a protected backend and can join them to an immutable model/prompt/context release.
- A task owner can define success, critical failures and acceptable quality changes; cost and latency are secondary when the candidate cannot meet accuracy or safety.
- Pricing is stored in a versioned operational configuration reviewed against official current pricing, not embedded permanently in tutorial text.
- Production-derived examples can be aggregated or redacted under approved retention and access rules. Raw prompts are not required for ordinary cost dashboards.
- The team can route candidates to shadow and canary cohorts, retain a known-good release and prevent an optimization experiment from changing authorization.
- Token counting is measured with appropriate tooling for planning but final billing and cache evidence come from API response usage.
Key concepts
- Cost per successful task
- Total model and retry cost divided by tasks that pass product quality and safety criteria, not merely requests that returned HTTP 200.
- Time to first useful output
- Elapsed time until the user receives content they can act on; it differs from connection time and full completion latency.
- Context budget
- A reviewed allocation of input tokens by instruction, history, retrieval, tools and reserve for output/reasoning rather than filling the maximum window.
- Exact prefix match
- Prompt caching condition where the reusable beginning of requests, including ordering of relevant content, remains identical.
- Cache write/read economics
- Comparison of cache_write_tokens and cached_tokens under the current model's pricing and expected reuse, rather than assuming every cache write saves money.
- Quality gate
- Task-specific threshold, critical-case rule and human-review requirement that an optimization must pass before rollout.
Before you copy
Values used in this guide
{{controlRelease}}Current known-good model/prompt/context release.
Example: support-ai/4.8.2{{candidateRelease}}One-change optimization candidate.
Example: support-ai/4.9.0-rc2{{evalDataset}}Reviewed representative and critical task corpus.
Example: evals/support-optimization-v12.jsonl{{contextBudget}}Maximum reviewed total input-token budget.
Example: 24000{{outputBudget}}Maximum generated-token budget for the workflow.
Example: 700{{canaryPercent}}Eligible traffic share for the candidate.
Example: 5Security and production boundaries
- Cost reduction never weakens authorization, content isolation, approval or critical safety checks. Security controls are measured as part of the quality gate.
- Do not log raw prompts, retrieved secrets, credentials or full customer documents merely to calculate tokens. Usage fields and approved aggregate metadata are sufficient for routine dashboards.
- Prompt caches are organization-scoped, but retention and eligibility depend on current model and data settings. Review the official data guide and organizational policy before sensitive workloads.
- Cache keys must not contain raw email, account numbers or secrets. Use stable privacy-safe bucketing and avoid cross-tenant shared prefixes when content includes tenant-confidential policy.
- Context pruning must preserve evidence provenance and mandatory policy. Never let an optimization model remove safety instructions or selectively hide conflicting evidence.
- A smaller model or lower reasoning configuration receives the same adversarial, privacy and critical-case evaluation as the control.
Stop before continuing if
- Stop if success, critical failures, latency boundaries or cost accounting are undefined; optimization without an objective invites misleading wins.
- Stop if telemetry records secrets or raw sensitive content beyond approved policy, or if cache retention conflicts with organizational requirements.
- Stop when context pruning removes required evidence, policy or citations, or when a smaller model breaches any critical quality gate.
- Stop if candidate and control differ in several untracked variables, use different datasets or cannot report effective release and usage metadata.
- Roll back on dangerous false positives, privacy leakage, sustained quality regression, retry amplification, cache-write losses or p95/p99 latency breach.
instruction
Define quality, latency and cost objectives together
Name the user task, success measure, critical failures, time to first useful output, p95 completion SLO, throughput and monthly budget. Optimize cost per successful task rather than raw request price.
Why this step matters
Optimization requires a multi-dimensional objective. Cost and latency wins are invalid when the workflow no longer solves the task safely.
What to understand
Define user-perceived timing, not only API duration.
Separate critical safety gates from average quality.
Include retries and human correction in task cost.
Assign an owner and review cadence.
System changes
- Creates a reviewed SLO and budget document.
- Changes no runtime.
Syntax explained
TTFU- Time to first useful output.
p95- Tail boundary experienced by five percent of requests.
cost_per_success- Total cost divided by quality-approved tasks.
workflow=grounded_support_answer quality_gate=expert_score>=0.92 dangerous_false_positives=0 ttfu_p95_ms<=900 completion_p95_ms<=3500 cost_per_success_usd<=0.018 availability>=99.9%
Checkpoint: Approve measurable objectives
Continue whenProduct, engineering and safety owners can explain every gate and tradeoff.
Stop whenStop if quality or success is undefined.
If this step fails
Teams optimize different numbers.
Likely causeNo shared objective.
Compare dashboards.Review release criteria.
ResolutionAdopt one versioned metric contract.
Security notes
- Critical safety is non-negotiable.
- Do not expose customer data in SLO labels.
Alternatives
- Keep the existing system if it already meets objectives.
- Use deterministic output for constrained tasks.
Stop conditions
- Stop if quality or success is undefined.
config
Instrument one complete task path
Capture phase timing, status, attempts, model, release and response usage fields. Store privacy-safe dimensions and calculate cost with versioned pricing outside request code.
Why this step matters
A baseline identifies the actual bottleneck and prevents changes based on intuition. Usage metadata is the authoritative token evidence.
What to understand
Measure end-to-end and phase timing.
Report distributions and per-workflow cohorts.
Join sampled quality without retaining all raw content.
Version pricing separately so historical reports remain reproducible.
System changes
- Adds telemetry and cost calculation.
- Does not change model behavior.
Syntax explained
cached_tokens- Input tokens read from prompt cache.
cache_write_tokens- Tokens written to cache where reported.
attempts- Retry amplification for one task.
telemetry/llm-task-event.jsonValues stay on this page and are never sent or saved.
{
"release": "{{controlRelease}}",
"workflow": "grounded_support_answer",
"status": "completed",
"latency_ms": 2280,
"time_to_first_useful_ms": 610,
"attempts": 1,
"usage": {
"input_tokens": 6120,
"output_tokens": 318,
"cached_tokens": 4096,
"cache_write_tokens": 0,
"reasoning_tokens": 94
},
"quality_sampled": true
}events=10000 missing_usage=0.1% ttfu_p50=420ms ttfu_p95=880ms completion_p50=1710ms completion_p95=3260ms cache_read_ratio=61.4% retry_amplification=1.04 raw_prompt_logging=disabled
Checkpoint: Validate baseline completeness
node telemetry/audit-baseline.mjs --release {{controlRelease}}Continue whenUsage, phase timing and release coverage meet the measurement threshold without sensitive raw logging.
Stop whenStop if usage or quality cannot be joined or telemetry violates privacy policy.
If this step fails
Cost differs from billing.
Likely causeStale pricing or wrong token class.
Inspect effective model.Compare usage fields.
ResolutionUpdate versioned pricing and recalculate.
Security notes
- Do not log API keys or raw prompts.
- Use privacy-safe cohort identifiers.
Alternatives
- Use aggregate OpenAI dashboards for a first baseline.
- Sample detailed traces under strict access.
Stop conditions
- Stop if usage or quality cannot be joined or telemetry violates privacy policy.
config
Create a context ledger and reserve
Allocate tokens to stable instructions, tools, history, retrieval and user input, plus output and reasoning reserve. Reject or deliberately reduce low-value sections before the model's maximum context is approached.
Why this step matters
Context capacity is finite and noisy context can reduce quality. A ledger makes pruning accountable and preserves output reserve.
What to understand
Measure tokens by section.
Filter authorization before relevance.
Deduplicate and prefer current primary evidence.
Fail or clarify when mandatory evidence does not fit.
System changes
- Adds deterministic context selection and budgets.
- May reject oversized requests rather than truncate silently.
Syntax explained
retrieval- Largest dynamic evidence allocation.
reserve- Safety margin for rendering variation.
fail_or_request_clarification- Explicit overflow behavior.
config/context-budget.jsonValues stay on this page and are never sent or saved.
{
"total_input_budget": {{contextBudget}},
"sections": {
"instructions": 2800,
"tools_and_schema": 4200,
"recent_history": 3500,
"retrieval": 11000,
"user_request": 1500,
"reserve": 1000
},
"max_output_tokens": {{outputBudget}},
"overflow": "fail_or_request_clarification"
}input_budget={{contextBudget}}
actual_input=18242
retrieval_candidates=18 selected=7
duplicates_removed=4
stale_removed=3
unauthorized_removed=4
output_reserve={{outputBudget}}
budget=PASSCheckpoint: Run overflow and evidence fixtures
node context/test-budget.mjs --budget {{contextBudget}}Continue whenMandatory evidence is preserved, unauthorized context removed and overflow is explicit.
Stop whenStop if pruning removes safety policy or conflicting evidence silently.
If this step fails
A required source disappears.
Likely causeRanking ignored mandatory provenance.
Inspect candidate ledger.Compare citations.
ResolutionPin required records and fail when they cannot fit.
Security notes
- Authorization precedes retrieval ranking.
- Never summarize secrets into supposedly safe context automatically.
Alternatives
- Retrieve older detail on demand.
- Split one task into explicit user-approved phases.
Stop conditions
- Stop if pruning removes safety policy or conflicting evidence silently.
config
Render static prefix before dynamic data
Place stable instructions, examples, tool definitions and schemas first in consistent order. Append tenant-authorized retrieval, history and the current request afterward. Use immutable artifact rendering to prevent insignificant changes from destroying exact-prefix reuse.
Why this step matters
Prompt caching depends on an exact reusable beginning. Stable rendering lowers avoidable misses while keeping dynamic sensitive data outside broad shared prefixes.
What to understand
Sort tools deterministically.
Version examples and schemas.
Do not put timestamps or random IDs in the prefix.
Separate tenant-confidential static policy when cache sharing would be inappropriate.
System changes
- Changes prompt ordering without changing reviewed meaning.
- Creates a stable prefix artifact.
Syntax explained
prefix_sha256- Detects accidental rendering drift.
static_tokens- Potentially reusable exact prefix.
dynamic_tokens- Per-request content after the prefix.
src/prompts/render.tsexport function renderPrompt(release, request) {
return [
release.instructions,
release.examples,
release.tools,
release.outputSchema,
{ role: "user", content: request.authorizedContext },
{ role: "user", content: request.currentQuestion },
];
}prefix_release=prompt-prefix/5.1.0 prefix_sha256=8f92... static_tokens=7210 dynamic_tokens=11032 ordering_drift=0 tenant_data_in_shared_prefix=false
Checkpoint: Compare rendered prefixes
node prompts/check-prefix.mjs --release {{candidateRelease}} --samples 100Continue whenEligible requests have identical approved prefixes and dynamic content begins afterward.
Stop whenStop if stabilization changes semantics or shares confidential tenant policy improperly.
If this step fails
Prefixes differ on every request.
Likely causeTimestamp or unordered tools appear early.
Diff serialized prompts.Inspect ordering.
ResolutionMove dynamic fields later and sort registry artifacts.
Security notes
- Do not maximize reuse across confidentiality boundaries.
- Hash comparisons must not expose content.
Alternatives
- Rely on automatic caching after stable ordering.
- Skip caching for low-reuse or sensitive paths.
Stop conditions
- Stop if stabilization changes semantics or shares confidential tenant policy improperly.
config
Configure and measure prompt caching deliberately
For eligible current models, apply a stable privacy-safe prompt_cache_key and optional supported explicit breakpoint after a reusable prefix. Measure reads and writes; do not assume a cache hit or savings.
Why this step matters
Current prompt caching exposes measurable read and write usage. Explicit breakpoints give control but can cost more when reuse is poor, so economics must be measured.
What to understand
Confirm model supports the exact options.
Keep cacheable prefix at least eligible length.
Use stable keys and partition high-volume traffic.
Review current retention and data policy before sensitive use.
System changes
- May write prompt cache state and changes request billing.
- Does not reuse final responses.
Syntax explained
prompt_cache_key- Stable routing bucket for common prefixes.
explicit- Uses only supplied cache breakpoints when configured.
cached_tokens- Measured tokens read from cache.
src/openai/cached-request.tsconst response = await client.responses.create({
model: release.model,
prompt_cache_key: "support:prefix-v5:shard-03",
prompt_cache_options: { mode: "explicit", ttl: "30m" },
input: [
{
role: "developer",
content: [{
type: "input_text",
text: release.stablePrefix,
prompt_cache_breakpoint: { mode: "explicit" },
}],
},
...dynamicInput,
],
});request=first input_tokens=8120 cache_write_tokens=7210 cached_tokens=0 request=second input_tokens=8044 cache_write_tokens=0 cached_tokens=7168 prompt_cache_key=support:prefix-v5:shard-03 prefix_match=exact quality_delta=0
Checkpoint: Calculate cache economics
node telemetry/cache-report.mjs --release {{candidateRelease}}Continue whenRead savings exceed write cost for approved workloads and quality is unchanged.
Stop whenStop if options are unsupported, cache writes dominate reads or retention conflicts with policy.
If this step fails
Cache writes recur without reads.
Likely causePrefix or key changes before reuse.
Compare hashes and keys.Inspect request rate.
ResolutionRemove breakpoint or stabilize and shard the prefix.
Security notes
- Keys contain no raw identity.
- Review organization cache and data settings.
Alternatives
- Use automatic prompt caching only.
- Skip caching and reduce request/context instead.
Stop conditions
- Stop if options are unsupported, cache writes dominate reads or retention conflicts with policy.
config
Reduce generated output without removing evidence
Define a concise response contract, bound arrays and output tokens, and render stable UI labels outside the model. Test that required conclusions, citations, uncertainty and escalation remain present.
Why this step matters
Generated tokens often dominate latency and cost. Concision is valuable only when the response still supports the user's decision and audit needs.
What to understand
Move labels and boilerplate to UI.
Use required structured fields.
Bound verbose arrays.
Evaluate evidence and escalation recall, not style alone.
System changes
- Changes response contract and output budget.
- Performs no external action.
Syntax explained
90 words- Observable summary limit.
evidence_ids- Preserves grounding.
max_output_tokens- Hard upper bound, not target quality.
prompts/concise-output.mdReturn:
- decision: one allowed enum
- summary: at most 90 words
- evidence_ids: at most 6 supplied IDs
- missing_evidence: at most 4 short items
- next_action_id: one allowlisted ID or null
Do not repeat the input or policy text.control_output_tokens_p50=438 candidate_output_tokens_p50=246 reduction=43.8% evidence_recall=100.0% critical_quality_delta=0 completion_p95_delta=-31.2%
Checkpoint: Evaluate concise output
node evals/run.mjs --dataset {{evalDataset}} --candidate concise-outputContinue whenOutput and latency fall while evidence, safety and usefulness gates pass.
Stop whenStop if truncation or concision removes required facts or produces incomplete responses.
If this step fails
Responses stop mid-object.
Likely causeHard budget is below valid output size.
Inspect incomplete reason.Measure valid lengths.
ResolutionRaise budget or simplify schema while retaining required fields.
Security notes
- Do not omit warnings to save tokens.
- Keep citations and uncertainty mandatory.
Alternatives
- Stream a longer answer.
- Generate summary first and optional detail on demand.
Stop conditions
- Stop if truncation or concision removes required facts or produces incomplete responses.
decision
Remove, combine or parallelize model calls
Map the request dependency graph. Replace formal steps with code, combine sequential model tasks that share context, and parallelize only independent branches. Measure full-task quality and latency after every topology change.
Why this step matters
Every request adds network and queue latency. Conventional code is faster for formal logic, while independent calls can overlap. Combining too much can hurt accuracy, so topology is an evaluated tradeoff.
What to understand
Draw dependencies from data, not implementation order.
Replace classification when deterministic rules suffice.
Combine tasks that use the same context and output contract.
Parallel branches must share an immutable snapshot and reconcile failures.
System changes
- Changes orchestration and request count.
- May alter concurrency and rate-limit pressure.
Syntax explained
deterministic_replacements- Steps no longer requiring a model.
combined_calls- Sequential round trips merged.
parallel_branches- Independent tasks executed concurrently.
node perf/analyze-dag.mjs workflows/support-answer.jsonmodel_calls_before=4 deterministic_replacements=1 combined_calls=1 parallel_branches=2 model_calls_after=2 critical_path_before=4210ms critical_path_after=2520ms quality_gate=PASS
Checkpoint: Load-test the new dependency graph
node perf/test-dag.mjs --release {{candidateRelease}}Continue whenCritical path improves, snapshots remain consistent and quality gates pass.
Stop whenStop if branches are dependent, concurrency breaches rate limits or quality falls.
If this step fails
Parallel answers disagree.
Likely causeBranches used different data snapshots.
Compare source versions.Inspect timestamps.
ResolutionUse one immutable context snapshot or serialize dependent work.
Security notes
- Parallelization does not duplicate writes.
- Carry tenant context explicitly to every branch.
Alternatives
- Keep sequential flow when dependencies are real.
- Precompute constrained results.
Stop conditions
- Stop if branches are dependent, concurrency breaches rate limits or quality falls.
decision
Compare smaller models and bounded reasoning
After the control meets quality, evaluate smaller or lower-effort configurations on the same corpus. Route only intents whose per-slice gates pass and keep uncertain or consequential cases on the proven control.
Why this step matters
Smaller models often improve speed and price, but quality determines whether the saving is real. Intent-specific routing captures proven gains without forcing one configuration across all risk levels.
What to understand
Use identical context and output contracts.
Measure per-slice quality and calibration.
Include routing classifier errors in evaluation.
Fallback must be bounded and visible.
System changes
- Changes model routing for approved low-risk intents.
- Keeps high-risk requests on control.
Syntax explained
low_risk_only- Candidate is restricted to proven cohort.
critical_false_positives- Blocking safety metric.
quality_delta- Candidate-control task score difference.
Values stay on this page and are never sent or saved.
node evals/compare-models.mjs --dataset {{evalDataset}} --control {{controlRelease}} --candidate {{candidateRelease}}cases=1200 candidate_cost_delta=-54.1% candidate_p95_delta=-38.7% overall_quality_delta=-0.4pp critical_false_positives=0 low_risk_gate=PASS high_risk_gate=FAIL routing=low_risk_only
Checkpoint: Approve routing matrix
node routing/verify.mjs --release {{candidateRelease}}Continue whenOnly evaluated slices use the candidate and uncertain cases return to control.
Stop whenStop on critical failure or unmeasured routing ambiguity.
If this step fails
Cheap route escalates more work.
Likely causeTask cost omitted correction labor.
Measure escalation.Calculate cost per success.
ResolutionNarrow routing or restore the control.
Security notes
- High-risk authority never moves based on cost alone.
- Route using privacy-safe features.
Alternatives
- Use one proven model.
- Use deterministic paths for low-risk formal tasks.
Stop conditions
- Stop on critical failure or unmeasured routing ambiguity.
config
Stream useful output and real progress
Stream when partial content can be safely shown, process bounded chunks and expose real tool or retrieval states. Measure time to first useful output and user completion, not merely first byte.
Why this step matters
Streaming reduces perceived waiting and can reduce user completion time. Raw token events are not automatically safe or useful, so the backend still applies output policy.
What to understand
Define which events are user-visible.
Batch rendering to avoid UI overhead.
Handle disconnect and cancellation.
Never show tool arguments as approved actions before validation.
System changes
- Enables streaming network responses.
- Does not change model authority.
Syntax explained
first_useful_text_ms- First safe content meaningful to user.
client_render_batches- Bounded UI updates.
processBoundedEvent- Server policy before display.
src/openai/stream.tsconst stream = await client.responses.create({
model: release.model,
input,
stream: true,
});
for await (const event of stream) {
const safeUpdate = processBoundedEvent(event);
if (safeUpdate) sendToClient(safeUpdate);
}connection_ms=84 first_event_ms=192 first_useful_text_ms=468 full_completion_ms=2410 chunks=19 client_render_batches=6 moderation_boundary=PASS
Checkpoint: Test stream lifecycle
node perf/test-stream.mjs --release {{candidateRelease}}Continue whenUseful output appears sooner, cancellation works and full quality is unchanged.
Stop whenStop if partial content bypasses safety or tool approval.
If this step fails
UI freezes during streaming.
Likely causeEvery token causes rendering.
Count events and renders.Profile client.
ResolutionBatch safe chunks and reduce update frequency.
Security notes
- Encode streamed text.
- Do not expose hidden reasoning or secrets.
Alternatives
- Show deterministic progress states.
- Return a complete response for short outputs.
Stop conditions
- Stop if partial content bypasses safety or tool approval.
decision
Move tolerant workloads to Batch or flex processing
Classify workloads by deadline. Keep interactive paths on suitable service behavior, use Batch API for asynchronous groups and flex processing for lower-priority tasks that tolerate slower responses and possible resource unavailability.
Why this step matters
Not every request needs interactive capacity. Official asynchronous and lower-priority processing options reduce cost when the product can tolerate their semantics.
What to understand
Document deadlines and availability expectations.
Preserve request IDs and per-item validation.
Never route emergency or user-blocking tasks to slower processing.
Reconcile partial batch results and expiry.
System changes
- Changes scheduling and possibly service tier.
- Introduces asynchronous result lifecycle.
Syntax explained
batch- Asynchronous grouped processing.
flex- Lower-cost, slower and occasionally unavailable processing.
standard- Interactive path under normal service behavior.
node routing/classify-workloads.mjs config/workloads.jsoninteractive_support=standard nightly_enrichment=batch offline_evals=flex deadline_violations=0 interactive_queue_impact=-28% quality_contract=unchanged
Checkpoint: Verify workload deadlines
node routing/test-workloads.mjsContinue whenEvery class meets product deadline and result validation requirements.
Stop whenStop if an interactive or safety-critical request can enter a tolerant queue.
If this step fails
A user request waits in Batch.
Likely causeRouting ignored interactive deadline.
Trace workload classification.Inspect queue metadata.
ResolutionRestore interactive routing and make deadline a mandatory input.
Security notes
- Apply the same data and authorization policy.
- Protect batch files and results.
Alternatives
- Use scheduled local deterministic processing.
- Keep all traffic interactive if volume is low.
Stop conditions
- Stop if an interactive or safety-critical request can enter a tolerant queue.
verification
Evaluate one optimization at a time
Compare one candidate variable with the control on the same dataset and runtime conditions. Report quality, critical failures, output completeness, latency distribution, usage, cache economics and cost per successful task.
Why this step matters
One-variable experiments make causality and rollback understandable. Joint quality and operational evaluation prevents cheap but unsuccessful requests from appearing optimal.
What to understand
Freeze dataset and release artifacts.
Repeat probabilistic cases as policy requires.
Review every critical difference.
Report uncertainty and per-slice results.
System changes
- Sends bounded evaluation requests.
- Writes reports but changes no production routing.
Syntax explained
--control- Current known-good release.
--candidate- One-change optimized release.
cost_per_success_delta- Economic gain after quality.
Values stay on this page and are never sent or saved.
node evals/run-optimization.mjs --dataset {{evalDataset}} --control {{controlRelease}} --candidate {{candidateRelease}}cases=1500 candidate_success=94.8% control_success=94.9% dangerous_false_positives=0 p95_latency_delta=-27.4% cost_per_success_delta=-31.8% cache_net_savings=positive human_reviewed_critical=100% gate=PASS
Checkpoint: Approve release evidence
node evals/review-optimization.mjs reports/{{candidateRelease}}.jsonContinue whenAll gates, critical review and operational budgets pass with one attributable change.
Stop whenStop if several variables changed, critical cases fail or grader agreement is weak.
If this step fails
Candidate result cannot be reproduced.
Likely causeArtifacts or pricing were mutable.
Print hashes.Inspect manifests.
ResolutionPin every component and rerun.
Security notes
- Use redacted fixtures.
- Human-calibrate semantic graders.
Alternatives
- Use deterministic benchmark for formal outputs.
- Keep control when gain is immaterial.
Stop conditions
- Stop if several variables changed, critical cases fail or grader agreement is weak.
decision
Canary, monitor drift and preserve rollback
Shadow the candidate, then route a bounded eligible cohort. Monitor quality samples, critical events, token composition, cache writes/reads, attempts, p95/p99 latency and cost per success. Restore the complete previous release when any blocking gate fires.
Why this step matters
Production traffic reveals distribution and caching behavior absent offline. A bounded canary protects users and tests real economics, while complete rollback restores known quality quickly.
What to understand
Use stable cohort assignment.
Sample quality by risk and language.
Alert on retry and cache-write amplification.
Keep the control and dashboards healthy through observation.
System changes
- Changes model/prompt/context release for a bounded cohort.
- Keeps prior release available.
Syntax explained
--percent- Maximum eligible candidate traffic.
cache_read_write_ratio- Observed reuse economics.
automatic_stop- Predefined rollback triggers.
Values stay on this page and are never sent or saved.
node deploy/optimize.mjs canary --release {{candidateRelease}} --percent {{canaryPercent}}release={{candidateRelease}}
phase=canary
percent={{canaryPercent}}
quality_delta=-0.1pp
dangerous_false_positives=0
p95_latency_delta=-24.9%
cost_per_success_delta=-29.7%
cache_read_write_ratio=5.4
rollback={{controlRelease}}
automatic_stop=armedCheckpoint: Review canary and rollback
node deploy/optimize.mjs status --release {{candidateRelease}} --verify-rollbackContinue whenQuality and privacy gates pass, gains persist and control restores successfully.
Stop whenStop on critical failure, quality drift, privacy issue, retry storm or negative cache economics.
If this step fails
Offline cache savings disappear.
Likely causeProduction prefixes or reuse differ.
Compare hashes.Slice by key.
ResolutionRemove costly breakpoints or redesign stable prefixes before promotion.
Security notes
- Canary never changes authorization.
- Restrict raw sample access.
Alternatives
- Reviewer-only pilot.
- Retain control when production gain is uncertain.
Stop conditions
- Stop on critical failure, quality drift, privacy issue, retry storm or negative cache economics.
Finish line
Verification checklist
node telemetry/audit-baseline.mjs --release {{controlRelease}}Usage, phase timing, retries and quality join successfully.node context/test-budget.mjs --budget {{contextBudget}} && node prompts/check-prefix.mjs --release {{candidateRelease}} --samples 100Mandatory authorized evidence fits and reusable prefixes are exact.node evals/run-optimization.mjs --dataset {{evalDataset}} --control {{controlRelease}} --candidate {{candidateRelease}}Quality gates and critical safety pass with improved cost/latency.node telemetry/cache-report.mjs --release {{candidateRelease}}Cache reads justify writes and cost per success improves.node deploy/optimize.mjs rollback --from {{candidateRelease}} --to {{controlRelease}} --dry-runPrevious complete release restores and passes quality smokes.Recovery guidance
Common problems and safe checks
Average latency improved but users still complain.
Likely causeTail latency or time to first useful output regressed.
Inspect p50/p95/p99.Measure first useful streamed event.
ResolutionOptimize the critical tail path and report distributions by workflow.
Token cost rose after enabling caching.
Likely causeCache writes exceed useful reads or prefixes rarely repeat.
Compare cache_write_tokens and cached_tokens.Group by cache key and prefix release.
ResolutionRemove low-reuse breakpoints or stabilize/partition prefixes based on measured reuse.
cached_tokens remains zero.
Likely causePrompt is ineligible or prefixes differ before the reusable boundary.
Compare serialized prefixes byte-for-byte.Inspect input-token count and key.
ResolutionMove static content first, stabilize ordering and use the documented key/breakpoint configuration.
Cache hit rate drops at high traffic.
Likely causeOne key exceeds useful routing volume or mixes many prefixes.
Measure requests per key.Count distinct prefixes.
ResolutionPartition traffic with a stable mapping while keeping identical prefixes together.
Shorter responses omit critical evidence.
Likely causeOutput optimization changed the product contract.
Run critical evidence fixtures.Compare reviewer scores.
ResolutionRestore the prior budget and redesign a concise structured contract with required evidence.
A smaller model is cheaper but escalates more tasks.
Likely causeAccuracy or calibration is insufficient, increasing downstream cost.
Calculate cost per successful task.Review escalation and correction labor.
ResolutionKeep the control or route only proven low-risk intents to the smaller model.
Pruned context produces unsupported claims.
Likely causeRelevant or conflicting evidence was removed.
Trace citations to retrieved candidates.Inspect pruning scores and mandatory records.
ResolutionFail closed when evidence does not fit, strengthen retrieval policy and include the failure in evals.
A parallelized workflow returns inconsistent state.
Likely causeSteps were not independent or used different snapshots.
Compare dependency graph.Inspect data versions.
ResolutionSerialize dependent steps or pass one immutable snapshot to each branch.
Streaming starts quickly but final latency worsens.
Likely causeChunk handling, moderation or client rendering adds overhead.
Measure each streaming phase.Compare full task completion.
ResolutionBatch client updates, process bounded chunks and retain streaming only when it improves user completion.
Retry cost spikes during outages.
Likely causeRetries lack deadlines, jitter or load shedding.
Inspect attempts per task.Graph queue depth.
ResolutionCap attempts and concurrency, add jitter/circuit breaking and expire low-priority work.
Usage cost calculation disagrees with invoices.
Likely causePricing config or token-class mapping is stale.
Compare effective model and usage fields.Review pricing configuration date.
ResolutionUpdate versioned pricing from official sources and preserve historical rates for prior usage.
One long conversation exceeds context limits.
Likely causeHistory grows without durable state or relevance policy.
Measure tokens by section.Identify repeated and stale turns.
ResolutionPreserve durable facts and open questions, retrieve older details on demand and ask for clarification rather than truncating blindly.
A candidate looks better in evals but worse in production.
Likely causeDataset distribution or quality grader is unrepresentative.
Compare privacy-safe production slices.Calibrate graders with humans.
ResolutionPause rollout, add adjudicated failures and rerun candidate/control.
Reference
Frequently asked questions
Should I always use the smallest model?
No. Establish the quality target first, then compare smaller models on representative evals. A cheaper request that fails or escalates more work can cost more per successful task.
Should I fill the entire context window?
No. The maximum is a capacity boundary, not a quality recommendation. Include only authorized, relevant and current context while reserving capacity for output and system behavior.
Does prompt caching return the same answer?
No. It reuses processed prompt prefixes; the model still generates a new response and nondeterministic outputs can differ.
Are fewer input tokens always the best latency optimization?
Not usually. Generated tokens, model choice and request topology often dominate. Measure phase timing before changing context that may support quality.
Recovery
Rollback
Atomically restore the prior model, prompt, context assembler, cache configuration, output contract and routing release. A cost optimization is not worth keeping while quality, privacy or availability is uncertain.
- Set candidate routing to zero and stop new asynchronous candidate jobs.
- Preserve privacy-safe metrics, release IDs and adjudicated failing samples.
- Restore the complete control release and verify effective hashes.
- Run critical quality, context, streaming, retry and cache configuration smokes.
- Confirm p95/p99 latency, success and retry metrics recover to the control baseline.
- Version the failure fixture and create a new candidate rather than mutating the failed release.
Evidence