Build a source-cited RAG pipeline with verified evidence
Govern source revisions, preserve ACLs through parsing and chunking, pin embeddings, combine lexical and semantic retrieval, validate claim-level citations, evaluate abstention, and release immutable indexes with safe rollback.
Answer from authorized, current, reproducible evidence while making unsupported claims, stale sources, cross-tenant retrieval, prompt injection, cost drift and deletion failures visible and blockable.
- OpenAI embeddings example text-embedding-3-small verified 2026-07-28
- Self-managed retrieval pipeline versioned source, chunk, index and prompt revisions
- Source and rights owners Authoritative documents, revision rules, processing/quotation rights, retention, deletion and review owners are identified.
- Trusted identity The application derives tenant, ACL and data class from authenticated server-side identity and can test two-tenant isolation.
- Evaluation ownership Subject-matter reviewers can label relevant evidence, contradictions, unsupported claims and high-severity outcomes.
- Versioned storage and recovery Immutable source/index artifacts, encrypted backups, atomic serving aliases and an isolated restore environment are available.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A governed source register with immutable revisions, content hashes, effective dates, tenant/ACL policy, supersession and deletion ownership.
- A sandboxed acquisition, deterministic parsing and stable chunking pipeline that preserves evidence coordinates and permissions.
- A pinned embedding revision with measurable privacy, vector dimension, token/cost and idempotent candidate-index behavior.
- Authorized hybrid retrieval, bounded reranking, grounded structured generation and claim-level citation resolution.
- Separate retrieval, citation, answer, abstention, privacy, latency and cost evaluations with zero cross-tenant leakage.
- An incremental dual-index release process with tombstones, monitoring, encrypted backups and revocation-aware rollback.
- Every material claim in a supported answer maps to an authorized immutable source span; missing or contradictory evidence produces a useful abstention.
- Retrieval never relies on post-filtering to hide another tenant's candidates, and caches include effective permission plus index revision.
- A model or chunking change creates a new evaluated vector space rather than mixing incompatible embeddings.
- Document updates and deletions propagate measurably without re-embedding unchanged content or losing historical evidence.
- Operators can diagnose freshness, recall, citation failure, latency and cost by revision without copying raw documents or questions into telemetry.
Architecture
How the parts fit together
A quarantined acquisition worker creates immutable sanitized source revisions. A deterministic parser produces permission-homogeneous chunks and stable evidence handles. An embedding worker writes a non-serving candidate index. At query time trusted identity constrains lexical and vector retrieval before fusion/reranking; a generator receives only bounded evidence and returns structured claims. A citation gate resolves and verifies every claim before rendering. Versioned manifests, evaluations and serving aliases support monitoring and rollback.
- An approved source is acquired into quarantine, scanned, sanitized, hashed and recorded with owner, rights, dates and access policy.
- The parser creates stable chunks whose evidence coordinates and ACL never cross source or permission boundaries.
- Approved chunks receive vectors from one pinned embedding revision and enter a complete non-serving candidate index.
- An authenticated query is normalized, constrained by trusted tenant/ACL and sent through bounded lexical plus semantic retrieval and reranking.
- The model receives untrusted numbered evidence, returns a structured disposition and cites handles; deterministic and evaluated gates validate support.
- Evaluation and canary evidence promote one serving alias; updates use new revisions, tombstones and a complete rollback bundle.
Assumptions
- The organization owns or is authorized to process, embed, retain and quote each source according to its data classification and provider terms.
- Tenant and ACL identity comes from trusted authentication middleware; users cannot assert broader permissions through a query.
- The example embedding request uses OpenAI `text-embedding-3-small`; production teams reverify current model, pricing and data controls before adoption.
- The retrieval and generation applications do not execute instructions found in documents and do not expose arbitrary tools.
- Human subject-matter owners can label high-risk retrieval evidence and adjudicate citation/answer evaluation.
- Source, chunk, index, retrieval, prompt, model and evaluation revisions can be stored and promoted as one immutable release.
Key concepts
- Retrieval-augmented generation
- Pipeline that retrieves authorized evidence and supplies it to a generator; retrieval improves access to sources but does not guarantee factual support.
- Chunk
- Stable permission-homogeneous text/structure unit with source revision, coordinates, hash and embedding.
- Hybrid retrieval
- Combination of lexical candidates for exact identifiers and semantic candidates for paraphrased intent under the same authorization filter.
- Claim-level citation
- Mapping from each material generated statement to exact immutable evidence spans rather than a general document list.
- Abstention
- Useful explicit result when evidence is missing, conflicting, unauthorized or outside scope instead of a generated guess.
- Index revision
- Immutable source/chunk/vector/retrieval configuration set evaluated and promoted together.
- Tombstone
- Explicit deletion, supersession or access-revocation record propagated through indexes, caches and serving policy.
Security and production boundaries
- Authorization occurs before candidate retrieval, not after generation. Apply tenant, ACL, status and jurisdiction filters to lexical, vector, rerank and cache paths consistently.
- Retrieved text never becomes system or developer instruction. It cannot authorize tools, secrets, writes, network destinations or changes to citation policy.
- Embeddings and excerpts are derived from source content and may remain sensitive. Encrypt them, scope access, apply retention/deletion and include them in incident analysis.
- Use allowlisted metadata for logs: revisions, opaque IDs, timings, counts, disposition and validation results. Raw documents, queries, evidence and answers require exceptional governed handling.
- Do not present similarity, reranker or model self-confidence as probability without empirical calibration. Show evidence, limitations and disposition.
- External model and embedding providers receive only data approved under their current data controls, region, account and retention policy.
- A citation proves neither source authority nor claim correctness by existence alone. Validate current-state applicability, exact span support and authorization.
- Downstream commands, access changes, financial actions and communications require deterministic application authorization and human approval appropriate to consequence.
Stop before continuing if
- Do not ingest a source whose owner, rights, current revision, tenant/ACL, retention or deletion procedure is unknown.
- Do not serve an index with mixed embedding dimensions, mixed permission chunks, unresolved tombstones or a failed cross-tenant evaluation.
- Do not release answers whose material claims can cite arbitrary IDs, unresolved spans, superseded sources or unauthorized metadata.
- Do not enable model tools merely because retrieved text asks for them; tool authorization belongs to a separate deterministic control plane.
- Do not promote when worst-slice retrieval/citation/answer thresholds fail or an unacceptable high-severity confident error remains.
- Do not roll back to an index that would reintroduce deleted or access-revoked content.
instruction
Define the supported-answer and citation contract
Before selecting an embedding model, define which questions the system may answer, which sources are authoritative, what tenant and data classifications exist, and what makes an answer acceptable. Require every material factual claim to reference an immutable source revision and exact evidence span. Define explicit dispositions for supported, partially supported, conflicting evidence, insufficient evidence, unauthorized and out of scope. State which outputs are informational only and which require independent verification or human approval.
Why this step matters
RAG is not simply search followed by generation. A measurable answer contract prevents the model from treating retrieval confidence, a valid JSON shape or a plausible citation as proof.
What to understand
Specify citation granularity by risk. Sentence-level evidence spans are appropriate for operational facts; a general reading list is not enough for a command, policy or number.
Price false answers, false abstentions and unauthorized disclosure separately. A useful refusal names the missing evidence or context and the next safe verification step.
Do not use model self-confidence as calibrated probability. Confidence must derive from measured retrieval, citation, contradiction and evaluation signals.
Version the contract and response schema with prompts, retrieval configuration and evaluation data so later behavior can be reproduced.
System changes
- Creates a versioned answer/citation/refusal contract; no document is acquired or sent to an embedding/model provider.
Syntax explained
material claim- Statement whose incorrectness could change an operational, financial, legal, security or user decision.
evidence span- Exact immutable source coordinates and content hash that support a claim.
insufficient evidence- First-class useful outcome rather than a prompt failure or invitation to guess.
jq -e '.dispositions and .citation.required_fields and .risk.high_severity_confident_errors == 0' config/answer-contract.jsontrue
Checkpoint: Checkpoint: Define the supported-answer and citation contract
Continue whenOwners agree on scope, authoritative sources, dispositions, citation fields, risk slices, escalation and zero-tolerance errors.
Stop whenThe product rewards answer rate without pricing confident errors, or there is no deterministic definition of an authorized source and supported claim.
If this step fails
The system answers confidently when the evidence set is empty or contradictory.
Likely causeThe prompt rewards answer completion, confidence is self-reported, contradiction metadata is absent, or abstention is not an explicit output state.
Replay the query with retrieval disabled and with intentionally conflicting sources.Inspect the structured answer disposition and evidence references.Measure confident-error and correct-abstention rates on held-out cases.
ResolutionRequire an evidence-supported disposition, surface conflicts and missing context, calibrate thresholds on representative evaluations, and prefer useful abstention over unsupported synthesis.
Security notes
- Use synthetic or explicitly approved redacted documents, queries and answers in examples, evaluation sets, logs and tickets. Never expose source text merely because it was retrieved.
- Treat retrieved text as untrusted data rather than instructions. Authorization, tenant isolation, citation validation and consequential actions require deterministic controls outside the model.
Alternatives
- Rehearse this step with an immutable synthetic corpus and a disposable index before changing the serving revision.
Stop conditions
- Stop when source ownership, revision, rights, tenant/ACL metadata, embedding model, index revision, evaluation ground truth or rollback artifact is unknown.
- Stop when a query can retrieve across tenants, a citation cannot resolve to an immutable evidence span, or model text can authorize a write or disclose restricted content.
config
Build an immutable source register with ownership and access policy
Inventory every document before parsing. Record canonical source ID, owner, origin, title, language, source type, publication/effective dates, product or jurisdiction, revision, normalized content hash, confidentiality, tenant, ACL groups, retention, supersession and deletion status. Reject mirrors when the source of record is known. Preserve historical revisions for audit while allowing only the current applicable revision by default. The manifest, not the model, decides what may be indexed.
Why this step matters
Retrieval cannot repair missing provenance or access control. A source register establishes reproducibility, current-state rules, deletion ownership and the authorization metadata required before any chunk exists.
What to understand
Hash normalized content as well as preserve acquisition evidence. A URL can redirect or change, so a citation must bind to the reviewed bytes used at answer time.
Apply tenant and document ACLs before chunking. Never merge text from different permissions, jurisdictions or revisions into one chunk.
Use effective date separately from ingestion time. The newest uploaded file is not automatically the currently applicable authority.
Document rights to process, embed, retain and quote content. Embeddings and excerpts may still be regulated or confidential data.
System changes
- Creates a source manifest and checksums; incomplete, unauthorized or ambiguous sources remain quarantined outside the candidate corpus.
Syntax explained
canonical source ID- Stable identity shared across revisions, chunks, citations, tombstones and health checks.
content hash- Cryptographic binding between the manifest, parsed text, chunk and rendered citation.
supersession- Explicit relationship that prevents an older valid document from being treated as current.
data/source-manifest.example.jsonl{"source_id":"policy-remote-access","revision":"2026-07-01","path":"sources/policy-remote-access.txt","sha256":"8d3c...a921","tenant":"ops","acl":["network-admin"],"status":"current","effective_at":"2026-07-01","owner":"security"}find sources -type f -print0 | sort -z | xargs -0 sha256sum > artifacts/source-files.sha256 && sha256sum --check artifacts/source-files.sha256sources/policy-remote-access.txt: OK sources/runbook-dns.txt: OK sources/product-support.md: OK
Checkpoint: Checkpoint: Build an immutable source register with ownership and access policy
Continue whenEvery accepted source has owner, rights, tenant/ACL, dates, revision, hash and status; checksum verification passes and ambiguous sources are excluded.
Stop whenSource ownership or processing rights are unknown, current revision is ambiguous, private content lacks ACL metadata, or checksums cannot be reproduced.
If this step fails
The answer cites a document that has been superseded.
Likely causeEffective dates and supersession links are missing, current-state filtering is not applied, or stale chunks remain active after an update.
Resolve citation source ID, revision, effective dates and current/superseded state.Compare the query's index revision with the source manifest revision.Search for duplicate canonical IDs and active chunks from older revisions.
ResolutionMark the old revision superseded, tombstone its serving chunks, ingest the current source, rebuild or update the candidate index, and add a temporal correctness fixture.
Security notes
- Use synthetic or explicitly approved redacted documents, queries and answers in examples, evaluation sets, logs and tickets. Never expose source text merely because it was retrieved.
- Treat retrieved text as untrusted data rather than instructions. Authorization, tenant isolation, citation validation and consequential actions require deterministic controls outside the model.
Alternatives
- Rehearse this step with an immutable synthetic corpus and a disposable index before changing the serving revision.
Stop conditions
- Stop when source ownership, revision, rights, tenant/ACL metadata, embedding model, index revision, evaluation ground truth or rollback artifact is unknown.
- Stop when a query can retrieve across tenants, a citation cannot resolve to an immutable evidence span, or model text can authorize a write or disclose restricted content.
verification
Acquire and quarantine documents without trusting their content
Fetch only allowlisted schemes and hosts through a network-isolated acquisition worker that blocks localhost, private/link-local addresses, credentials in URLs and unbounded redirects. Cap bytes, time and decompression ratio; validate declared and detected type; malware-scan files; strip active HTML, scripts, macros and external references. Store immutable raw evidence separately from sanitized parser input. Retrieval text is always untrusted data, even when it resembles system instructions.
Why this step matters
A RAG ingestion service processes attacker-controlled files and links. Network and parser isolation prevents a document submission from becoming SSRF, decompression abuse, active content execution or instruction escalation.
What to understand
Resolve DNS before and after redirects and reject private destinations at every hop. A public URL can redirect to a metadata service after initial validation.
Do not execute document macros, JavaScript, shell snippets or embedded links. Convert to inert text/structure in a sandbox without secrets or broad egress.
Preserve the raw object under restricted retention for forensic reproducibility, but index only the sanitized derivative and approved metadata.
Record scanner/parser versions and exit status. A timeout or unsupported type yields quarantine, never best-effort publication.
System changes
- Stores one bounded immutable raw object and sanitized candidate; it does not parse or index content that failed network, type or malware policy.
Syntax explained
--proto '=https'- Restricts curl to HTTPS instead of allowing a redirected request to switch protocols.
--max-filesize- Bounds downloaded bytes; the acquisition service must also enforce streamed and decompressed limits.
quarantine- Non-serving state used until network, type, malware, rights and metadata checks complete.
curl --proto '=https' --tlsv1.2 --max-time 20 --max-filesize 10485760 --location --max-redirs 3 --output quarantine/source.bin https://example.org/approved-document.txt && file --brief --mime-type quarantine/source.bin && sha256sum quarantine/source.bintext/plain 8d3c7c579e78c0f08a49b2250e4565a6f52f5bba5b6d3d883c19d6b371a4a921 quarantine/source.bin
Checkpoint: Checkpoint: Acquire and quarantine documents without trusting their content
Continue whenApproved input downloads within limits, detected type and checksum are recorded, active content cannot execute, and unsafe/unsupported inputs remain quarantined.
Stop whenThe worker has production secrets, follows private-network redirects, executes active content, cannot bound decompression, or publishes after scanner/parser failure.
If this step fails
A prompt-injection string in a retrieved document changes tool behavior or answer policy.
Likely causeThe system treats retrieved text as instructions, exposes privileged tools in the same context, or relies on the model to distinguish authority.
Replay the source with a synthetic injection canary and tools disabled.Inspect the final prompt structure and deterministic tool authorization logs.Test whether the content can alter allowed citations, tenant filters or output schema.
ResolutionQuarantine the source, separate data from instruction channels, remove unnecessary tools, enforce tool and data authorization deterministically, and add the injection to adversarial evaluation.
Security notes
- Use synthetic or explicitly approved redacted documents, queries and answers in examples, evaluation sets, logs and tickets. Never expose source text merely because it was retrieved.
- Treat retrieved text as untrusted data rather than instructions. Authorization, tenant isolation, citation validation and consequential actions require deterministic controls outside the model.
Alternatives
- Rehearse this step with an immutable synthetic corpus and a disposable index before changing the serving revision.
Stop conditions
- Stop when source ownership, revision, rights, tenant/ACL metadata, embedding model, index revision, evaluation ground truth or rollback artifact is unknown.
- Stop when a query can retrieve across tenants, a citation cannot resolve to an immutable evidence span, or model text can authorize a write or disclose restricted content.
config
Parse structure and create stable, permission-homogeneous chunks
Parse headings, paragraphs, tables, lists, code blocks, captions, page numbers and footnotes into a deterministic intermediate representation. Chunk on semantic boundaries with modest overlap, retaining parent source, revision, structure path, character/page coordinates, content hash, language, effective dates and ACL. Never combine different sources or access rules. Derive stable chunk IDs from canonical source ID, revision, structure path and normalized span hash—not mutable list position.
Why this step matters
A decisive exception separated from its heading or date can invert meaning. Stable structural chunks make retrieval, deletion, citation display and evaluation reproducible across repeated ingestion.
What to understand
Keep tables intact or serialize rows with column headings. A value without its header is not reliable evidence.
Store a citation-ready display span separately from embedding text if normalization is needed. The displayed evidence must still map exactly to the source hash.
Measure chunk-size distributions by document type and language. One global token window is rarely appropriate for policy prose, code and reference tables.
Re-ingest the same immutable fixture twice and assert identical chunk IDs, boundaries and hashes; any intended parser change creates a new index revision.
System changes
- Creates deterministic parsed records and chunk candidates with provenance and ACL metadata; no vector is generated yet.
Syntax explained
heading-aware- Uses document structure to keep a passage with the title and scope that explain it.
overlap_tokens- Repeats limited context across adjacent chunks while increasing storage and duplicate-result risk.
span_sha256- Stabilizes chunk identity and citation integrity around normalized evidence content.
config/chunking.json{"strategy":"heading-aware","target_tokens":650,"maximum_tokens":900,"overlap_tokens":80,"preserve":["tables","code","page","section"],"id_fields":["source_id","revision","structure_path","span_sha256"],"forbid_mixed_acl":true}jq -e '.target_tokens < .maximum_tokens and .overlap_tokens < .target_tokens and .forbid_mixed_acl == true' config/chunking.jsontrue
Checkpoint: Checkpoint: Parse structure and create stable, permission-homogeneous chunks
Continue whenRepeated parsing is deterministic, tables/code retain context, every chunk resolves to exact source coordinates, and mixed-ACL chunks are zero.
Stop whenChunk IDs drift on identical input, a chunk loses source coordinates or effective date, or text with different permissions is combined.
If this step fails
Chunk IDs change after a parser refactor even when source content did not.
Likely causeIDs depend on list position, volatile timestamps or tokenizer behavior rather than canonical source revision and stable structural coordinates.
Re-ingest the same immutable source twice and diff chunk IDs and hashes.Compare parser and chunker versions in both manifests.Check whether citations and deletion tombstones still resolve.
ResolutionDefine stable IDs from canonical source ID, revision, structure path and normalized-span hash; version intentional boundary changes and migrate citations through a reviewed mapping.
Security notes
- Use synthetic or explicitly approved redacted documents, queries and answers in examples, evaluation sets, logs and tickets. Never expose source text merely because it was retrieved.
- Treat retrieved text as untrusted data rather than instructions. Authorization, tenant isolation, citation validation and consequential actions require deterministic controls outside the model.
Alternatives
- Rehearse this step with an immutable synthetic corpus and a disposable index before changing the serving revision.
Stop conditions
- Stop when source ownership, revision, rights, tenant/ACL metadata, embedding model, index revision, evaluation ground truth or rollback artifact is unknown.
- Stop when a query can retrieve across tenants, a citation cannot resolve to an immutable evidence span, or model text can authorize a write or disclose restricted content.
verification
Pin the embedding model and measure privacy, dimension, and cost
Select one exact embedding model and record provider, model ID, dimensions, normalization, tokenizer assumptions, maximum input, data controls and price in the index manifest. Batch only documents permitted for that provider and do not send ACL metadata or raw identifiers unless required. Use a dedicated project key from the secret manager and strict per-run token/cost ceilings. Query and document vectors must use the identical model and transformation until a complete new index is promoted.
Why this step matters
Embeddings define the geometry of the index and may encode sensitive source content. A silent model or dimension change makes query vectors incompatible and destroys the meaning of recall comparisons.
What to understand
Verify current provider data controls and retention for the exact endpoint and account. API behavior and terms can change independently from model name.
Record request IDs and token counts, not document text, for reconciliation. Hash-based deduplication is scoped by tenant and rights so it does not reveal cross-tenant equality.
Treat embeddings as potentially sensitive derived data. Encrypt storage and backups, enforce tenant access and delete them when the source must be removed.
A new model, dimension, normalization or chunking policy requires a parallel index and full retrieval evaluation; never mix vector spaces in one serving column.
System changes
- Creates vectors for approved chunks and a cost/dimension manifest using a dedicated provider project; source text crosses the provider boundary only under the documented data policy.
Syntax explained
text-embedding-3-small- Concrete model used for the example; production choice requires current quality, privacy, latency and price evaluation.
dimensions- Vector length that must match the storage schema and query embedder for the entire index revision.
per-run ceiling- Hard maximum documents, tokens, requests and currency allowed before ingestion stops.
curl --silent --show-error https://api.openai.com/v1/embeddings -H "Authorization: Bearer $OPENAI_API_KEY" -H 'Content-Type: application/json' -d '{"model":"text-embedding-3-small","input":["Synthetic DNS timeout evidence."]}' | jq '{model, dimensions:(.data[0].embedding|length), prompt_tokens:.usage.prompt_tokens}'{
"model": "text-embedding-3-small",
"dimensions": 1536,
"prompt_tokens": 5
}Checkpoint: Checkpoint: Pin the embedding model and measure privacy, dimension, and cost
Continue whenThe manifest records exact model and dimension, synthetic probe returns the expected vector length, approved data controls are documented and batch cost stays within the hard ceiling.
Stop whenThe provider/data policy is not approved, vectors have mixed dimensions, model alias is moving, or request/token/cost accounting is incomplete.
If this step fails
Embedding requests fail with a dimension or model mismatch.
Likely causeThe embedding model changed without a new index, stored vectors have different dimensions, or query and document embeddings use different settings.
Read model, dimensions and normalization from the immutable index manifest.Inspect a candidate vector's dimension without logging document text.Compare the query embedder revision with the serving index revision.
ResolutionStop serving the mixed revision, restore the matching query embedder, or build a complete new index under the new model and promote it only after retrieval evaluation.
Security notes
- Use synthetic or explicitly approved redacted documents, queries and answers in examples, evaluation sets, logs and tickets. Never expose source text merely because it was retrieved.
- Treat retrieved text as untrusted data rather than instructions. Authorization, tenant isolation, citation validation and consequential actions require deterministic controls outside the model.
Alternatives
- Rehearse this step with an immutable synthetic corpus and a disposable index before changing the serving revision.
Stop conditions
- Stop when source ownership, revision, rights, tenant/ACL metadata, embedding model, index revision, evaluation ground truth or rollback artifact is unknown.
- Stop when a query can retrieve across tenants, a citation cannot resolve to an immutable evidence span, or model text can authorize a write or disclose restricted content.
config
Create an immutable candidate index with idempotent upserts and tombstones
Write chunks and embeddings under a new index revision rather than modifying the live alias in place. Use chunk ID as the idempotency key; persist source revision, content hash, embedding revision, tenant/ACL, dates and citation coordinates. Upsert only new or changed chunks, mark removed/superseded chunks with tombstones, and record counts plus costs in the build manifest. The serving alias changes only after all verification passes.
Why this step matters
A versioned candidate isolates partial builds and enables atomic promotion. Idempotent writes prevent retries from duplicating chunks or embedding cost, while tombstones make revocation measurable.
What to understand
Do not delete the previous immutable index during the build or initial serving window. It is the fastest reliable rollback artifact.
Distinguish unchanged source hash from unchanged chunk ID; a parser revision may intentionally change chunks even when raw bytes match.
Validate referential integrity from index record to chunk, source revision and citation coordinates before answer generation.
Keep build manifests free of source text and secrets. Counts, hashes, revisions, timings, token use and price are sufficient for audit and cost.
System changes
- Creates a non-serving candidate index and tombstone set; the live serving alias and previous index remain unchanged.
Syntax explained
candidate revision- Complete isolated corpus/index version evaluated before an atomic serving-alias change.
idempotent upsert- Repeating the same chunk write yields one record and does not create duplicate cost or citations.
tombstone- Explicit revocation/deletion marker propagated across serving indexes and caches.
artifacts/index-manifest.example.json{"index_revision":"rag-2026-07-28.1","source_manifest_sha256":"2cc4...5fe1","chunker":"heading-aware-v3","embedding_model":"text-embedding-3-small","dimensions":1536,"new":184,"changed":22,"unchanged":8410,"deleted":9,"duplicate_ids":0,"mixed_acl":0}jq -e '.duplicate_ids == 0 and .mixed_acl == 0 and (.new + .changed + .unchanged + .deleted) > 0' artifacts/index-manifest.json && sha256sum artifacts/index-manifest.jsontrue c49c0fa22971b8216e0f293faf9fbef0f1db6c370ba040214ab3c6df3e46938e artifacts/index-manifest.json
Checkpoint: Checkpoint: Create an immutable candidate index with idempotent upserts and tombstones
Continue whenCandidate manifest has zero duplicate IDs and mixed ACLs, changed/unchanged/deleted counts reconcile, every record resolves to provenance, and previous index remains intact.
Stop whenA partial build is serving, retries create duplicates, deletion counts cannot reconcile, or an index record lacks source/chunk/ACL/citation metadata.
If this step fails
Index cost rises sharply after an apparently small document update.
Likely causeThe pipeline re-embeds unchanged chunks, lacks content-hash deduplication, retries batches without idempotency, or indexes duplicated source mirrors.
Compare new, changed, unchanged, deleted and duplicate chunk counts.Reconcile embedding request IDs and tokens with the candidate manifest.Inspect retry and idempotency behavior for the failed batch.
ResolutionStop the update, deduplicate by normalized content hash within policy, embed only changed chunks, use idempotent upserts, and enforce per-run token/cost ceilings.
Security notes
- Use synthetic or explicitly approved redacted documents, queries and answers in examples, evaluation sets, logs and tickets. Never expose source text merely because it was retrieved.
- Treat retrieved text as untrusted data rather than instructions. Authorization, tenant isolation, citation validation and consequential actions require deterministic controls outside the model.
Alternatives
- Rehearse this step with an immutable synthetic corpus and a disposable index before changing the serving revision.
Stop conditions
- Stop when source ownership, revision, rights, tenant/ACL metadata, embedding model, index revision, evaluation ground truth or rollback artifact is unknown.
- Stop when a query can retrieve across tenants, a citation cannot resolve to an immutable evidence span, or model text can authorize a write or disclose restricted content.
config
Normalize queries and enforce authorization before retrieval
Authenticate the caller and derive tenant, ACL groups, environment and data class from trusted identity—not request text. Normalize Unicode and whitespace, preserve quoted phrases, error codes, version numbers, hostnames and product identifiers, and bound query length. Apply tenant/ACL/current-state filters in the retrieval engine before candidates are returned. Include all authorization and index revision fields in cache keys; never cache private results globally.
Why this step matters
Filtering after vector search can disclose metadata and contaminate ranking; a global cache can cross tenants even when the underlying query path is safe. Trusted identity must constrain retrieval before content leaves storage.
What to understand
Reject client-supplied tenant or ACL overrides. An administrator preview needs a separately authorized role and an audit record.
Preserve high-information literals through query rewriting. Semantic paraphrase is helpful for intent but harmful when it removes the exact error code or version that identifies evidence.
Include permission-set hash and serving index in cache identity; invalidate on access changes and tombstones.
Log normalized query category or hash only when policy permits. Raw user questions can contain secrets and personal data.
System changes
- Creates a bounded query and cache policy with pre-retrieval authorization; it may reduce results that were previously visible through permissive post-filtering.
Syntax explained
trusted identity- Authenticated server-side claims used to derive tenant and ACL rather than values typed by the user.
pre-retrieval filter- Constrains candidate generation so unauthorized chunks never reach ranking, generation or logs.
ACL hash in cache key- Separates cached results for different effective permissions without exposing group names.
config/query-policy.json{"maximum_characters":4096,"preserve":["quoted_phrases","error_codes","versions","hostnames"],"required_filters":["tenant","acl","status=current"],"cache_key":["tenant","acl_hash","index_revision","normalized_query","retrieval_revision"],"reject_client_acl":true}jq -e '.required_filters|index("tenant") and index("acl") and . as $f | $f' config/query-policy.json >/dev/null; printf '%s\n' 'tenant and ACL filters configured before retrieval'tenant and ACL filters configured before retrieval
Checkpoint: Checkpoint: Normalize queries and enforce authorization before retrieval
Continue whenSynthetic cross-tenant and revoked-access queries return zero unauthorized candidates, exact identifiers survive normalization, and cache entries are permission/index scoped.
Stop whenAuthorization happens after retrieval, ACL values are user-controlled, private cache keys omit permission scope, or raw queries are logged without an approved purpose.
If this step fails
A user retrieves a chunk belonging to another tenant or ACL group.
Likely causeAuthorization is applied after vector search, metadata is user-controlled, a cache key omits tenant/ACL scope, or an index contains mixed-permission chunks.
Disable answer generation and preserve request, identity and source IDs for incident response.Replay the query with two synthetic tenants and inspect pre-retrieval filters.Audit cache keys, chunk ACL metadata and denied-document fixtures.
ResolutionQuarantine the serving revision, invalidate affected caches, fix authorization before retrieval, rebuild permission-homogeneous chunks, notify security owners and require zero cross-tenant leaks before release.
Security notes
- Use synthetic or explicitly approved redacted documents, queries and answers in examples, evaluation sets, logs and tickets. Never expose source text merely because it was retrieved.
- Treat retrieved text as untrusted data rather than instructions. Authorization, tenant isolation, citation validation and consequential actions require deterministic controls outside the model.
Alternatives
- Rehearse this step with an immutable synthetic corpus and a disposable index before changing the serving revision.
Stop conditions
- Stop when source ownership, revision, rights, tenant/ACL metadata, embedding model, index revision, evaluation ground truth or rollback artifact is unknown.
- Stop when a query can retrieve across tenants, a citation cannot resolve to an immutable evidence span, or model text can authorize a write or disclose restricted content.
decision
Combine lexical and semantic candidates, then rerank
Retrieve a bounded lexical candidate set for exact identifiers and a semantic set for paraphrased intent under identical authorization filters. Fuse ranks with a deterministic method such as reciprocal-rank fusion, deduplicate by chunk/source, and rerank only the top bounded set. Preserve raw component ranks and scores for evaluation, not as claims of probability. Cap per-stage latency and cost; if reranking fails, use the tested deterministic fallback or abstain rather than silently changing behavior.
Why this step matters
Semantic retrieval captures paraphrase, while lexical retrieval preserves exact codes and names. A bounded fusion stage reduces dependence on either failure mode and provides evidence for diagnosis.
What to understand
Tune candidate counts, fusion constant and reranker depth on held-out data. More candidates increase latency and may introduce distractors or sensitive text.
Deduplicate overlapping chunks and diversify by source when repeated passages would crowd out independent evidence.
Do not expose raw similarity scores as confidence percentages. Their scale depends on model, index and query and must be calibrated empirically.
Version query rewriting, lexical analyzer, embedding model, fusion and reranker together as the retrieval revision.
System changes
- Adds bounded hybrid retrieval and reranking under the existing authorization filter; no answer is generated yet.
Syntax explained
lexical candidates- Match exact terms, codes and names using an inverted/text index.
semantic candidates- Retrieve nearby embeddings for paraphrased meaning within the authorized corpus.
reciprocal-rank fusion- Combines rank positions without pretending unlike lexical and vector scores share one calibrated scale.
printf '%s\n' 'lexical rank: [dns-12,dns-09,net-44]' 'semantic rank: [net-44,dns-12,vpn-03]' | awk 'NF'lexical rank: [dns-12,dns-09,net-44] semantic rank: [net-44,dns-12,vpn-03]
Checkpoint: Checkpoint: Combine lexical and semantic candidates, then rerank
Continue whenHeld-out identifier and paraphrase slices meet recall targets, unauthorized candidates remain zero, reranking stays within latency/cost budgets, and fallback behavior is explicit.
Stop whenCandidate paths use different ACL filters, fusion/reranking is unversioned, a failed reranker silently changes production behavior, or score is presented as calibrated confidence.
If this step fails
Retrieval returns fluent but irrelevant chunks for exact product names or error codes.
Likely causeSemantic search dominates lexical evidence, query rewriting removed identifiers, chunking lost headings, or reranking was not trained/evaluated for the domain.
Compare lexical, semantic and hybrid candidate lists for the frozen query.Inspect identifier preservation in query normalization and chunk metadata.Measure recall and ranking metrics on identifier-heavy evaluation slices.
ResolutionPreserve quoted identifiers, combine lexical and semantic candidates, tune fusion/reranking on held-out fixtures, and keep a safe exact-search fallback for codes and names.
Security notes
- Use synthetic or explicitly approved redacted documents, queries and answers in examples, evaluation sets, logs and tickets. Never expose source text merely because it was retrieved.
- Treat retrieved text as untrusted data rather than instructions. Authorization, tenant isolation, citation validation and consequential actions require deterministic controls outside the model.
Alternatives
- Rehearse this step with an immutable synthetic corpus and a disposable index before changing the serving revision.
Stop conditions
- Stop when source ownership, revision, rights, tenant/ACL metadata, embedding model, index revision, evaluation ground truth or rollback artifact is unknown.
- Stop when a query can retrieve across tenants, a citation cannot resolve to an immutable evidence span, or model text can authorize a write or disclose restricted content.
config
Generate only from evidence handles with explicit abstention
Provide the model with the answer contract, user question and a bounded set of numbered evidence records containing source ID, revision, title, effective scope and exact text span. Delimit evidence as untrusted data and prohibit it from changing instructions or tool permissions. Require structured output with disposition, answer, material claims and cited evidence IDs. When evidence is absent, contradictory, unauthorized or outside scope, return a useful abstention that names the missing information without inventing facts.
Why this step matters
Retrieved text can be wrong, stale or malicious. Evidence handles plus a structured disposition make unsupported claims and injection attempts detectable before an answer reaches the user.
What to understand
Do not ask the model to create URLs or source metadata from memory. The application resolves cited IDs through the immutable manifest.
Keep tools unavailable in the answer-generation stage unless separately justified and deterministically authorized. Retrieved instructions never grant tool permission.
Limit evidence to the smallest sufficient authorized spans and keep sensitive metadata out of prompts. Retrieval authorization does not imply unlimited disclosure.
Label generated synthesis separately from quoted evidence. The user should see which statement is interpretation and which span is the source.
System changes
- Adds a structured grounded-generation contract and abstention path; it does not yet certify that citations support claims.
Syntax explained
evidence handle- Opaque application-issued ID resolving to an immutable authorized source span.
disposition- Machine-checkable answer state that includes conflict, insufficiency and unauthorized outcomes.
untrusted delimiter- Makes document boundaries explicit while deterministic controls prevent the content from gaining authority.
config/grounded-response.schema.json{"type":"object","required":["disposition","answer","claims"],"properties":{"disposition":{"enum":["supported","partially_supported","conflicting","insufficient_evidence","unauthorized","out_of_scope"]},"answer":{"type":"string"},"claims":{"type":"array","items":{"type":"object","required":["text","evidence_ids"],"properties":{"text":{"type":"string"},"evidence_ids":{"type":"array","items":{"type":"string"},"minItems":1}}}}}}jq -e '.required == ["disposition","answer","claims"] and (.properties.disposition.enum|index("insufficient_evidence"))' config/grounded-response.schema.json"insufficient_evidence"
Checkpoint: Checkpoint: Generate only from evidence handles with explicit abstention
Continue whenResponses validate against schema, all material claims cite supplied handles, no URL/source metadata is invented, and missing/conflicting evidence produces useful abstention.
Stop whenThe model can cite arbitrary IDs, tools are authorized by retrieved text, claims lack evidence IDs, or the product suppresses abstention to maximize answer rate.
If this step fails
A prompt-injection string in a retrieved document changes tool behavior or answer policy.
Likely causeThe system treats retrieved text as instructions, exposes privileged tools in the same context, or relies on the model to distinguish authority.
Replay the source with a synthetic injection canary and tools disabled.Inspect the final prompt structure and deterministic tool authorization logs.Test whether the content can alter allowed citations, tenant filters or output schema.
ResolutionQuarantine the source, separate data from instruction channels, remove unnecessary tools, enforce tool and data authorization deterministically, and add the injection to adversarial evaluation.
Security notes
- Use synthetic or explicitly approved redacted documents, queries and answers in examples, evaluation sets, logs and tickets. Never expose source text merely because it was retrieved.
- Treat retrieved text as untrusted data rather than instructions. Authorization, tenant isolation, citation validation and consequential actions require deterministic controls outside the model.
Alternatives
- Rehearse this step with an immutable synthetic corpus and a disposable index before changing the serving revision.
Stop conditions
- Stop when source ownership, revision, rights, tenant/ACL metadata, embedding model, index revision, evaluation ground truth or rollback artifact is unknown.
- Stop when a query can retrieve across tenants, a citation cannot resolve to an immutable evidence span, or model text can authorize a write or disclose restricted content.
verification
Resolve citations and validate every material claim
After generation, resolve each evidence ID server-side under the same caller authorization and serving revision. Verify source/chunk existence, current or intentionally historical status, content hash, exact coordinates and permitted display. Check that every material claim has evidence and that each cited span actually supports its subject, predicate, quantity, date and qualifiers. Use deterministic checks for IDs, numbers, dates and quotations, plus an evaluated entailment grader and human review for high-risk answers.
Why this step matters
A syntactically present citation may point to an unrelated paragraph, stale revision or unauthorized source. Validation after generation prevents citation decoration from being mistaken for grounding.
What to understand
Separate citation validity, correctness, completeness and source quality. One aggregate 'citation score' hides different operational remedies.
Never reveal an unauthorized title or snippet in an error. Return an unauthorized disposition while preserving evidence internally for audit.
For quotations, compare normalized exact text and punctuation against the source span. For numbers and dates, deterministic extraction catches many high-impact errors cheaply.
Do not treat a model grader as independent truth. Calibrate it against blinded human labels and retain adjudication for high-severity claims.
System changes
- Adds a blocking post-generation citation gate and authorized renderer; unsupported or stale claims are rejected or returned for safe regeneration.
Syntax explained
citation validity- Evidence ID resolves to the expected immutable source revision and authorized span.
citation correctness- The cited span entails the attached claim rather than merely sharing a topic.
citation completeness- Every material claim, number, date and operational recommendation has sufficient evidence.
jq -e '[.claims[] | select((.evidence_ids|length)==0)] | length == 0' fixtures/grounded-answer.jsontrue
Checkpoint: Checkpoint: Resolve citations and validate every material claim
Continue whenAll citations resolve under authorization, hashes and coordinates match, material claims are completely supported, and high-risk disagreements reach human review.
Stop whenA citation resolves against a different revision, a material claim lacks support, an unauthorized source would be disclosed, or grader quality is unmeasured.
If this step fails
A citation opens but does not support the sentence attached to it.
Likely causeThe system cites a whole document instead of a span, citations were added after generation, or retrieval returned topically related but non-entailing text.
Resolve the stored chunk and exact start/end coordinates against the content hash.Compare the claim's subject, predicate, qualifiers, date and scope with the evidence span.Run citation correctness separately from citation presence.
ResolutionReject or relink the claim, generate from immutable evidence handles, require claim-level support, and block publication of answers whose material claims lack an entailing span.
Security notes
- Use synthetic or explicitly approved redacted documents, queries and answers in examples, evaluation sets, logs and tickets. Never expose source text merely because it was retrieved.
- Treat retrieved text as untrusted data rather than instructions. Authorization, tenant isolation, citation validation and consequential actions require deterministic controls outside the model.
Alternatives
- Rehearse this step with an immutable synthetic corpus and a disposable index before changing the serving revision.
Stop conditions
- Stop when source ownership, revision, rights, tenant/ACL metadata, embedding model, index revision, evaluation ground truth or rollback artifact is unknown.
- Stop when a query can retrieve across tenants, a citation cannot resolve to an immutable evidence span, or model text can authorize a write or disclose restricted content.
verification
Evaluate retrieval, citations, answers, abstention, and isolation separately
Create versioned train-free evaluation sets with known relevant source IDs and spans, hard lexical identifiers, paraphrases, missing evidence, contradictions, superseded material, multilingual cases, adversarial instructions, deleted sources and two-tenant ACL probes. Measure retrieval recall and ranking, citation validity/correctness/completeness, factual support, contradiction handling, correct abstention, confident error, latency and cost. Report worst slices and confidence intervals; manually adjudicate all high-severity errors.
Why this step matters
A high answer score cannot distinguish a retrieval miss from unsupported synthesis, broken citation or authorization leak. Separate metrics direct fixes and prevent aggregate gains from hiding catastrophic slices.
What to understand
Keep evaluation cases out of prompt examples and development feedback where possible. Track lineage to reduce leakage and overfitting.
Use exact retrieval ground truth for recall and independent human review for claim support. Model graders assist scale but do not replace risk ownership.
Set release thresholds before seeing candidate results. Require zero cross-tenant disclosure and zero unacceptable confident high-severity errors.
Record model snapshots, prompts, source/index revisions, code/config hashes, grader versions, latency environment and provider cost with every result.
System changes
- Creates a reproducible evaluation suite and release thresholds; failing candidates remain non-serving regardless of subjective demo quality.
Syntax explained
retrieval recall@k- Fraction of cases where at least one known relevant evidence span appears in the first k authorized candidates.
confident error- Unsupported or wrong answer presented without appropriate uncertainty or abstention.
worst slice- Lowest-performing risk-relevant subset used as a release gate rather than hidden by an average.
jq -r '.case_id + "\t" + .slice + "\t" + .expected_disposition' evals/rag-eval.jsonl | head -5dns-001 lexical-identifier supported policy-014 superseded supported missing-003 no-evidence insufficient_evidence acl-tenant-b cross-tenant unauthorized inject-009 prompt-injection supported
Checkpoint: Checkpoint: Evaluate retrieval, citations, answers, abstention, and isolation separately
Continue whenCandidate meets predefined retrieval, citation, answer, abstention, latency and cost thresholds; cross-tenant leaks and unacceptable high-severity confident errors are zero.
Stop whenGround truth is unavailable, evaluation data is leaked/unrepresentative, only aggregate metrics pass, or a high-severity error lacks adjudication.
If this step fails
Evaluation passes, but production questions still produce unsupported answers.
Likely causeThe dataset is synthetic, leaked, too easy, stale or scored only by the same model family; important tenants, languages and temporal cases are missing.
Compare privacy-safe production issue clusters with evaluation coverage.Review worst-slice metrics and grader disagreements against blinded human labels.Check dataset lineage and whether evaluation examples appeared in prompts or training artifacts.
ResolutionAdd representative hard negatives and production-derived redacted cases, separate retrieval/citation/answer metrics, use human adjudication for high-impact slices, and gate on worst slices.
Security notes
- Use synthetic or explicitly approved redacted documents, queries and answers in examples, evaluation sets, logs and tickets. Never expose source text merely because it was retrieved.
- Treat retrieved text as untrusted data rather than instructions. Authorization, tenant isolation, citation validation and consequential actions require deterministic controls outside the model.
Alternatives
- Rehearse this step with an immutable synthetic corpus and a disposable index before changing the serving revision.
Stop conditions
- Stop when source ownership, revision, rights, tenant/ACL metadata, embedding model, index revision, evaluation ground truth or rollback artifact is unknown.
- Stop when a query can retrieve across tenants, a citation cannot resolve to an immutable evidence span, or model text can authorize a write or disclose restricted content.
decision
Apply incremental updates and deletions through a dual-index release
Compare the new source manifest with the serving revision by canonical ID, revision and content hash. Re-parse and re-embed only new or changed chunks, create tombstones for removed, superseded or access-revoked content, and build a complete candidate manifest. Run source integrity, ACL, retrieval, citation and answer evaluations, then atomically move the serving alias. Keep the previous immutable index and source snapshot for rollback while enforcing deletion deny-lists immediately across both.
Why this step matters
Incremental updates control embedding cost and freshness, but partial in-place mutation makes rollback and citation resolution ambiguous. Dual revisions provide atomic promotion and reproducible evidence.
What to understand
Apply an immediate retrieval deny-list for legal deletion or revoked access before slower vector deletion finishes; verify caches and replicas observe it.
Do not reuse an index revision after a failed build. Preserve the manifest and failure evidence, then create a new revision.
Canary candidate retrieval by shadowing authorized queries with content suppressed from logs; compare source IDs, ranks, latency and cost rather than raw private text.
After promotion, monitor miss, abstention, citation and stale-source indicators through an observation window before retiring the previous revision.
System changes
- Builds and atomically promotes a new source/chunk/vector revision, propagates tombstones and retains a complete previous revision for rollback.
Syntax explained
manifest diff- Deterministically identifies new, changed, unchanged, superseded and deleted source revisions.
dual index- Keeps candidate and serving revisions separate until acceptance and supports fast alias rollback.
immediate deny-list- Blocks revoked content before asynchronous deletion from every physical index and cache completes.
comm -3 <(jq -r '.source_id+"@"+.revision+" "+.sha256' manifests/serving.jsonl | sort) <(jq -r '.source_id+"@"+.revision+" "+.sha256' manifests/candidate.jsonl | sort) | headpolicy-remote-access@2026-07-01 8d3c...a921 policy-remote-access@2026-01-15 611f...9e02
Checkpoint: Checkpoint: Apply incremental updates and deletions through a dual-index release
Continue whenChanged/deleted counts reconcile, deny-list probes return zero, candidate passes all gates, alias promotion is atomic and previous revision remains verifiably usable.
Stop whenDeletion cannot be proven, a candidate is partially serving, source/index revisions are mixed, or promotion lacks a one-operation rollback.
If this step fails
Deleted or access-revoked content remains retrievable.
Likely causeThe source tombstone did not propagate, caches retain old results, the serving alias still points to an old index, or deletion was treated as an ordinary re-ingest.
Query the canonical source ID across source, chunk, index and cache manifests.Run a denied-content probe under an authorized test identity.Check event offsets and serving alias revision.
ResolutionImmediately deny the source ID at retrieval, invalidate scoped caches, complete tombstone propagation, verify absence across all active revisions, and record deletion latency against policy.
Security notes
- Use synthetic or explicitly approved redacted documents, queries and answers in examples, evaluation sets, logs and tickets. Never expose source text merely because it was retrieved.
- Treat retrieved text as untrusted data rather than instructions. Authorization, tenant isolation, citation validation and consequential actions require deterministic controls outside the model.
Alternatives
- Rehearse this step with an immutable synthetic corpus and a disposable index before changing the serving revision.
Stop conditions
- Stop when source ownership, revision, rights, tenant/ACL metadata, embedding model, index revision, evaluation ground truth or rollback artifact is unknown.
- Stop when a query can retrieve across tenants, a citation cannot resolve to an immutable evidence span, or model text can authorize a write or disclose restricted content.
config
Monitor quality, freshness, privacy, latency, and cost without raw content
Record bounded metadata by tenant class and revision: ingestion status, source age, parser failures, chunk counts, embedding tokens/cost, candidate and serving revisions, retrieval latency, reranker latency, answer disposition, citation validation, abstention, cache result, deletion propagation and evaluation trend. Use opaque request/source IDs and sampled synthetic probes. Alert on stale authoritative sources, unauthorized candidate count above zero, deletion objective breach, cost variance and worst-slice regression.
Why this step matters
RAG quality changes when sources, parsing, embeddings, retrieval, prompts or models change. Metadata and synthetic probes reveal drift without creating a shadow database of private queries and documents.
What to understand
Track freshness by authoritative source owner and effective date, not merely last ingestion timestamp.
Measure cost per new/changed chunk and per answered query. Re-embedding unchanged data or excessive candidate depth should be visible.
Use privacy-preserving production issue categories to select new evaluation cases; raw content requires an exceptional governed workflow.
Tie every metric and alert to source, index, retrieval, prompt and model revisions so mixed fleets cannot hide behind averages.
System changes
- Adds metadata-only dashboards, synthetic probes and release-quality alerts; raw documents, queries and answers remain outside telemetry.
Syntax explained
freshness- Age and current-state validity of authoritative source revisions, not simply pipeline activity.
deletion objective- Maximum time from revocation decision to absence from retrieval, caches and serving revisions.
revision labels- Bounded identifiers for source/index/retrieval/model releases used to attribute a regression.
config/rag-alerts.yamlalerts:
unauthorized_candidates: 0
stale_authoritative_sources: 1
deletion_propagation_minutes: 15
embedding_cost_variance_percent: 5
citation_validation_failure_percent: 1
high_severity_confident_errors: 0sed -n '1,20p' config/rag-alerts.yamlalerts: unauthorized_candidates: 0 stale_authoritative_sources: 1 deletion_propagation_minutes: 15 embedding_cost_variance_percent: 5 citation_validation_failure_percent: 1 high_severity_confident_errors: 0
Checkpoint: Checkpoint: Monitor quality, freshness, privacy, latency, and cost without raw content
Continue whenOperators can detect stale, unauthorized, costly, slow or unsupported behavior by revision without reading customer content; synthetic alerts have been exercised.
Stop whenTelemetry stores raw content by default, user-controlled labels are unbounded, deletion cannot be timed, or quality metrics cannot identify the active revision.
If this step fails
Index cost rises sharply after an apparently small document update.
Likely causeThe pipeline re-embeds unchanged chunks, lacks content-hash deduplication, retries batches without idempotency, or indexes duplicated source mirrors.
Compare new, changed, unchanged, deleted and duplicate chunk counts.Reconcile embedding request IDs and tokens with the candidate manifest.Inspect retry and idempotency behavior for the failed batch.
ResolutionStop the update, deduplicate by normalized content hash within policy, embed only changed chunks, use idempotent upserts, and enforce per-run token/cost ceilings.
Security notes
- Use synthetic or explicitly approved redacted documents, queries and answers in examples, evaluation sets, logs and tickets. Never expose source text merely because it was retrieved.
- Treat retrieved text as untrusted data rather than instructions. Authorization, tenant isolation, citation validation and consequential actions require deterministic controls outside the model.
Alternatives
- Rehearse this step with an immutable synthetic corpus and a disposable index before changing the serving revision.
Stop conditions
- Stop when source ownership, revision, rights, tenant/ACL metadata, embedding model, index revision, evaluation ground truth or rollback artifact is unknown.
- Stop when a query can retrieve across tenants, a citation cannot resolve to an immutable evidence span, or model text can authorize a write or disclose restricted content.
verification
Back up, restore, and roll back the coupled RAG release
Back up the source manifest and allowed raw/sanitized artifacts according to retention, parsed chunks, vector/text indexes, tombstones, embedding and retrieval manifests, prompts/schemas, evaluation data/results, authorization policy and serving-alias history. Encrypt artifacts and keep restore keys independent. Restore into isolation, verify hashes and ACLs, run retrieval/citation/deletion fixtures and resolve historical citations. A rollback atomically restores the complete previous serving bundle, not only the application.
Why this step matters
An index without its source and model manifests cannot explain citations; an application rollback against a new index may mix incompatible IDs. Coupled recoverable artifacts preserve both availability and evidence integrity.
What to understand
Test restore with no cached production credentials. Use replacement model/provider credentials and synthetic queries to prove wiring without processing live content.
Preserve deletion and revoked-access state across restore. A backup that reintroduces prohibited content is not recoverable.
Record RPO/RTO, artifact sizes, checksum result, active revisions and unresolved exceptions. A successful archive command is not a restore test.
If rolling back after new source updates, reconcile legal deletions and ACL revocations into the previous index before making it serving.
System changes
- Creates encrypted coupled backup and an isolated restored release; rollback may switch serving alias but must preserve newer deletions and access revocations.
Syntax explained
coupled bundle- Source, chunks, indexes, policies, prompts, schemas and evaluation evidence required to reproduce one answer revision.
historical citation resolution- Ability to map a past evidence ID to the exact source span and hash used at answer time.
revocation-forward rollback- Restores older search behavior while carrying forward newer deletion and ACL restrictions.
sha256sum --check backups/rag-2026-07-28.1/SHA256SUMSsource-manifest.jsonl: OK chunks.jsonl.zst: OK index.snapshot: OK release-manifest.json: OK evals/results.json: OK
Checkpoint: Checkpoint: Back up, restore, and roll back the coupled RAG release
Continue whenChecksums and isolated restore pass, historical citations resolve, unauthorized/deleted content remains absent, evaluation meets baseline and alias rollback is atomic.
Stop whenRestore keys or manifests are missing, source rights forbid the backup, deleted content returns, citation hashes fail, or application/index/prompt revisions cannot be matched.
If this step fails
A rollback restores the old application but citations resolve against the new index.
Likely causeApplication, source manifest, chunker, embeddings, index, prompt and citation resolver were released independently without a coupled revision.
Read revision identifiers from the response, serving alias and release manifest.Resolve a baseline citation against both candidate and previous indexes.Verify that the previous immutable index and source snapshot remain available.
ResolutionRestore the complete coupled release, invalidate mixed caches, verify citation hashes and retrieval fixtures, and prevent independent promotion of application and index aliases.
Security notes
- Use synthetic or explicitly approved redacted documents, queries and answers in examples, evaluation sets, logs and tickets. Never expose source text merely because it was retrieved.
- Treat retrieved text as untrusted data rather than instructions. Authorization, tenant isolation, citation validation and consequential actions require deterministic controls outside the model.
Alternatives
- Rehearse this step with an immutable synthetic corpus and a disposable index before changing the serving revision.
Stop conditions
- Stop when source ownership, revision, rights, tenant/ACL metadata, embedding model, index revision, evaluation ground truth or rollback artifact is unknown.
- Stop when a query can retrieve across tenants, a citation cannot resolve to an immutable evidence span, or model text can authorize a write or disclose restricted content.
Finish line
Verification checklist
sha256sum --check backups/rag-candidate/SHA256SUMSSource manifest, sanitized artifacts, chunks, index and release manifest match immutable checksums and every chunk resolves to authorized coordinates.jq -e '.cross_tenant_leaks == 0 and .retrieval_recall_at_8 >= .thresholds.retrieval_recall_at_8' evals/results.jsonCross-tenant candidates are zero and lexical, semantic, temporal, identifier and deletion slices meet predefined retrieval thresholds.jq -e '.high_severity_confident_errors == 0 and .citation_validity == 1 and .citation_completeness >= .thresholds.citation_completeness' evals/results.jsonAll evidence IDs resolve, material claims are supported, missing/conflicting evidence abstains correctly and high-severity confident errors are zero.jq -e '.duplicate_chunk_ids == 0 and .deleted_sources_retrievable == 0 and .restore_passed == true' artifacts/release-evidence.jsonIncremental counts and embedding spend reconcile, deletions are absent from retrieval/caches, and isolated restore plus alias rollback pass.Recovery guidance
Common problems and safe checks
A current authoritative document is absent from search results.
Likely causeAcquisition failed, the manifest quarantined the file, parsing produced no chunks, ACL metadata is wrong, or the serving index points at an older revision.
Look up the canonical source ID and content hash in the manifest.Inspect parser status, chunk count and index revision without reading restricted text.Run exact lexical search for a known harmless phrase under an authorized tenant.
ResolutionRepair the acquisition, parser or metadata mapping, rebuild a new immutable index revision, add the miss to retrieval evaluation, and promote only after authorization plus recall tests pass.
The answer cites a document that has been superseded.
Likely causeEffective dates and supersession links are missing, current-state filtering is not applied, or stale chunks remain active after an update.
Resolve citation source ID, revision, effective dates and current/superseded state.Compare the query's index revision with the source manifest revision.Search for duplicate canonical IDs and active chunks from older revisions.
ResolutionMark the old revision superseded, tombstone its serving chunks, ingest the current source, rebuild or update the candidate index, and add a temporal correctness fixture.
A citation opens but does not support the sentence attached to it.
Likely causeThe system cites a whole document instead of a span, citations were added after generation, or retrieval returned topically related but non-entailing text.
Resolve the stored chunk and exact start/end coordinates against the content hash.Compare the claim's subject, predicate, qualifiers, date and scope with the evidence span.Run citation correctness separately from citation presence.
ResolutionReject or relink the claim, generate from immutable evidence handles, require claim-level support, and block publication of answers whose material claims lack an entailing span.
A user retrieves a chunk belonging to another tenant or ACL group.
Likely causeAuthorization is applied after vector search, metadata is user-controlled, a cache key omits tenant/ACL scope, or an index contains mixed-permission chunks.
Disable answer generation and preserve request, identity and source IDs for incident response.Replay the query with two synthetic tenants and inspect pre-retrieval filters.Audit cache keys, chunk ACL metadata and denied-document fixtures.
ResolutionQuarantine the serving revision, invalidate affected caches, fix authorization before retrieval, rebuild permission-homogeneous chunks, notify security owners and require zero cross-tenant leaks before release.
Embedding requests fail with a dimension or model mismatch.
Likely causeThe embedding model changed without a new index, stored vectors have different dimensions, or query and document embeddings use different settings.
Read model, dimensions and normalization from the immutable index manifest.Inspect a candidate vector's dimension without logging document text.Compare the query embedder revision with the serving index revision.
ResolutionStop serving the mixed revision, restore the matching query embedder, or build a complete new index under the new model and promote it only after retrieval evaluation.
Retrieval returns fluent but irrelevant chunks for exact product names or error codes.
Likely causeSemantic search dominates lexical evidence, query rewriting removed identifiers, chunking lost headings, or reranking was not trained/evaluated for the domain.
Compare lexical, semantic and hybrid candidate lists for the frozen query.Inspect identifier preservation in query normalization and chunk metadata.Measure recall and ranking metrics on identifier-heavy evaluation slices.
ResolutionPreserve quoted identifiers, combine lexical and semantic candidates, tune fusion/reranking on held-out fixtures, and keep a safe exact-search fallback for codes and names.
The system answers confidently when the evidence set is empty or contradictory.
Likely causeThe prompt rewards answer completion, confidence is self-reported, contradiction metadata is absent, or abstention is not an explicit output state.
Replay the query with retrieval disabled and with intentionally conflicting sources.Inspect the structured answer disposition and evidence references.Measure confident-error and correct-abstention rates on held-out cases.
ResolutionRequire an evidence-supported disposition, surface conflicts and missing context, calibrate thresholds on representative evaluations, and prefer useful abstention over unsupported synthesis.
Chunk IDs change after a parser refactor even when source content did not.
Likely causeIDs depend on list position, volatile timestamps or tokenizer behavior rather than canonical source revision and stable structural coordinates.
Re-ingest the same immutable source twice and diff chunk IDs and hashes.Compare parser and chunker versions in both manifests.Check whether citations and deletion tombstones still resolve.
ResolutionDefine stable IDs from canonical source ID, revision, structure path and normalized-span hash; version intentional boundary changes and migrate citations through a reviewed mapping.
Index cost rises sharply after an apparently small document update.
Likely causeThe pipeline re-embeds unchanged chunks, lacks content-hash deduplication, retries batches without idempotency, or indexes duplicated source mirrors.
Compare new, changed, unchanged, deleted and duplicate chunk counts.Reconcile embedding request IDs and tokens with the candidate manifest.Inspect retry and idempotency behavior for the failed batch.
ResolutionStop the update, deduplicate by normalized content hash within policy, embed only changed chunks, use idempotent upserts, and enforce per-run token/cost ceilings.
Evaluation passes, but production questions still produce unsupported answers.
Likely causeThe dataset is synthetic, leaked, too easy, stale or scored only by the same model family; important tenants, languages and temporal cases are missing.
Compare privacy-safe production issue clusters with evaluation coverage.Review worst-slice metrics and grader disagreements against blinded human labels.Check dataset lineage and whether evaluation examples appeared in prompts or training artifacts.
ResolutionAdd representative hard negatives and production-derived redacted cases, separate retrieval/citation/answer metrics, use human adjudication for high-impact slices, and gate on worst slices.
Deleted or access-revoked content remains retrievable.
Likely causeThe source tombstone did not propagate, caches retain old results, the serving alias still points to an old index, or deletion was treated as an ordinary re-ingest.
Query the canonical source ID across source, chunk, index and cache manifests.Run a denied-content probe under an authorized test identity.Check event offsets and serving alias revision.
ResolutionImmediately deny the source ID at retrieval, invalidate scoped caches, complete tombstone propagation, verify absence across all active revisions, and record deletion latency against policy.
A prompt-injection string in a retrieved document changes tool behavior or answer policy.
Likely causeThe system treats retrieved text as instructions, exposes privileged tools in the same context, or relies on the model to distinguish authority.
Replay the source with a synthetic injection canary and tools disabled.Inspect the final prompt structure and deterministic tool authorization logs.Test whether the content can alter allowed citations, tenant filters or output schema.
ResolutionQuarantine the source, separate data from instruction channels, remove unnecessary tools, enforce tool and data authorization deterministically, and add the injection to adversarial evaluation.
A rollback restores the old application but citations resolve against the new index.
Likely causeApplication, source manifest, chunker, embeddings, index, prompt and citation resolver were released independently without a coupled revision.
Read revision identifiers from the response, serving alias and release manifest.Resolve a baseline citation against both candidate and previous indexes.Verify that the previous immutable index and source snapshot remain available.
ResolutionRestore the complete coupled release, invalidate mixed caches, verify citation hashes and retrieval fixtures, and prevent independent promotion of application and index aliases.
Reference
Frequently asked questions
Does adding citations stop hallucinations?
No. A model can attach an irrelevant or stale citation. Validate that every material claim is completely supported by exact authorized evidence spans and allow abstention.
Why keep lexical search if embeddings understand meaning?
Error codes, versions, identifiers and names often require exact matching. Hybrid retrieval covers both literal and paraphrased questions and must be evaluated by slice.
Can authorization be applied after vector search?
No for a secure design. Unauthorized candidates can leak through ranking, logs, caches or timing. Apply trusted tenant/ACL filters before candidates leave storage.
Should every document update re-embed the entire corpus?
No. Compare canonical IDs and hashes, embed only changed chunks, build a complete candidate revision and use full re-embedding only when the model/chunking space changes.
Can retrieved documents contain operational instructions?
They can contain text that looks like instructions, but it remains untrusted evidence. Deterministic application policy controls tools, secrets and actions.
What is the safest rollback unit?
The coupled source, chunks, vectors, retrieval config, prompt/schema, model and citation resolver revision, with newer deletions and ACL revocations carried forward.
Recovery
Rollback
Atomically restore the complete previous source/chunk/vector/retrieval/prompt/model/citation bundle and invalidate caches, while carrying forward every newer deletion and ACL revocation. Verify authorization, retrieval, citations and evaluation before reopening; never roll back only application code against an incompatible index.
- Stop candidate promotion or return the serving alias to the last accepted immutable release; preserve failed manifests, redacted request IDs and evaluation evidence.
- Apply the current deletion and access-revocation deny-list to the previous revision before exposing it, then verify zero results for every affected canonical source ID.
- Restore matching parser/chunker, embedding query model, lexical/vector indexes, fusion/reranker, grounded prompt/schema and citation resolver revisions.
- Invalidate caches whose keys reference candidate source, ACL, retrieval or prompt revisions; confirm no mixed revision remains in responses.
- Run two-tenant isolation, identifier/paraphrase retrieval, superseded/deleted source, prompt-injection, citation correctness, abstention and cost fixtures.
- Resolve representative historical citations to exact source spans and hashes, measure rollback RTO, and record any availability or freshness gap.
- Fix the failure under a new immutable revision and repeat full evaluation; never mutate or reuse the rejected release manifest.
Evidence