AI/ML / Databases & Data / Security

Operate the RAG knowledge lifecycle: lineage, ACLs, PII, retention, deletion, and recovery

Build a governed RAG knowledge lifecycle for document updates and deletion, immutable lineage, tenant ACLs, PII minimization before embeddings, retention and legal hold, multi-index consistency, deletion verification, encrypted backup, isolated restore, derived-index rebuild, and rollback that never resurrects prohibited data.

420 min12 stepsHigh-impact changeRevision 2

Save or explore

#rag #governance #lineage #acl #pii #retention #deletion #backup #restore

0 of 12 steps completed

Goal

Make every canonical and derived knowledge representation attributable, authorized, minimal, updateable, deletable, recoverable, and independently verifiable while separating search indexes from policy authority and preserving current tombstones and ACL revocations through restore or rollback.

Supported environments

PostgreSQL current supported release
Elasticsearch current supported release
OpenAI vector stores current API where used

Prerequisites

Data and records ownership Privacy, security, records, legal, and service owners have approved classification, purpose, retention, deletion, hold, residency, and recovery policies.Review the signed governance manifest and escalation contacts.
Authoritative identity and ACL service Tenant, groups, denies, revocations, and policy revision come from authenticated server state.Run positive and cross-tenant negative authorization fixtures.
Versioned isolated generations Canonical events and derived stores support idempotency, stable identity, reconciliation, isolated rebuild, and alias-style promotion.Rebuild a synthetic generation and compare stable IDs.
Recovery environment Encrypted backups, managed keys, compatible software, an isolated restore target, current security ledger, and RPO/RTO are available.Complete one isolated restore without production routing.

Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System

A governance and lineage layer that identifies every canonical and derived representation, tenant, ACL revision, transformation, lifecycle state, policy owner, retention rule, and recovery objective.
An idempotent update and deletion pipeline with immediate tombstone denial, pre-embedding minimization, isolated generations, reconciliation, legal-hold deferral, and multi-method deletion verification.
An encrypted backup and isolated recovery process that rebuilds derived indexes, applies current security ledgers to historical data, proves RPO/RTO, and rolls back relevance without rolling back security.

Observable outcome

Operators can answer where a chunk came from, who may retrieve it, which transformations touched it, when it should expire, what deletion targets remain, and which release or backup can recover it.
Personal and secret-bearing data is reduced before duplication; cross-tenant defaults fail closed; deletion status reflects verified absence rather than an accepted request.
Updates, restores, and rollback cannot silently mix generations or revive deleted, expired, quarantined, or ACL-revoked content.

Architecture

How the parts fit together

Canonical governed content and an authorization service feed a transactional outbox. A lifecycle worker minimizes and redacts, generates stable chunks, and writes isolated lexical, vector, summary, and cache generations with immutable lineage. A separate current security ledger carries tombstones, holds, expiries, quarantines, and ACL revocations through all rebuilds and restores. Reconciliation, deletion verification, encrypted backups, isolated recovery, canary control, and audit make every transition observable.

Canonical store and row securityOwn authoritative content metadata, tenant boundary, current revision, and transactional change events.

Lineage and security ledgersTrace transformations and preserve current deletion, hold, expiry, quarantine, and authorization state independently of historical search snapshots.

Minimization and lifecycle workerRedact before derivation, process idempotent updates and deletions, and record per-target outcomes.

Derived lexical, vector, summary, and cache viewsServe retrieval as reconstructable authorized projections rather than policy authorities.

Reconciler and deletion verifierProve identity parity, ACL freshness, and absence across every online representation and replica.

Backup and recovery controllerEncrypt and expire recoverable state, restore in isolation, replay current security, rebuild indexes, and enforce canary and rollback gates.

Policy owners classify sources and representations; canonical writes and idempotent events bind stable tenant identity, revision, ACL, and retention.
The worker minimizes and redacts before chunking or embedding, then writes a gated isolated generation and promotes only after parity and security checks.
Deletion commits a durable tombstone and immediate gateway deny, propagates purge to every representation, and remains verifying until negative tests pass.
Reconciliation compares a stable high-water mark; ambiguous drift is quarantined while safe idempotent work is replayed.
Backups retain canonical, lineage, security, mappings, and configuration under policy; isolated restore replays current security and rebuilds derived views before canary exposure.

Assumptions

The organization supplies retention and legal-hold requirements; this guide is technical implementation guidance, not legal advice.
A stable tenant-scoped identity can follow a document through canonical, chunk, embedding, index, summary, cache, export, and backup representations.
Search indexes are replaceable derived views and can be isolated, rebuilt, reconciled, and routed through versioned generations.
The authorization service and current security ledger remain available during restore and rollback, or recovery stays isolated until they are recovered.
Backup media can be encrypted, access-controlled, expired, and tested in an isolated environment with compatible software and keys.

Key concepts

Lineage: The stable identity and transformation history connecting source revision to redacted text, chunks, embeddings, indexes, summaries, caches, and releases.
Security ledger: Durable current tombstones, expiries, quarantines, legal-hold transitions, and ACL revocations applied to all historical rebuilds and restores.
Data minimization: Excluding or replacing data not needed for the retrieval purpose before it is duplicated into derived representations.
Tombstone: A minimal durable denial record that immediately blocks retrieval and prevents later replay or restore from resurrecting deleted identity.
Legal hold: An authorized scoped preservation override that defers eligible purge but does not expand access or replace normal ACLs.
High-water mark: An ordered event boundary that lets reconciliation compare changing systems against one consistent canonical state.
RPO and RTO: Accepted recoverable data-loss interval and service-restoration time, verified through actual isolated restores.

Before you copy

Values used in this guide

{{governanceManifest}}

Dataset ownership, representation, retention, hold, and recovery policy.

Example: /srv/rag-governance/datasets.json

{{lineageSchema}}

Stable identity and transformation lineage schema.

Example: /srv/rag-governance/schema/lineage-v7.json

{{rlsMigration}}

Reviewed PostgreSQL tenant row-security migration.

Example: /srv/rag-governance/migrations/20260728_rls.sql

{{redactionPolicy}}

Versioned pre-derivation minimization and redaction policy.

Example: /srv/rag-governance/policies/pii-v6.json

{{updatePolicy}}

Outbox, idempotency, target, promotion, and previous-generation policy.

Example: /srv/rag-governance/policies/update-v4.json

{{governanceDatabaseUrl}}secret

Restricted PostgreSQL URI for canonical metadata, the security ledger, purge acknowledgements, and reconciliation state.

Example: postgresql://rag_lifecycle@db.internal/rag_governance

{{documentId}}

Stable tenant-scoped opaque document identity.

Example: tenant-a:policy-42

{{deletionEvent}}

Idempotent authorized deletion request ID.

Example: del-771

{{highWaterMark}}

Canonical event boundary used for reconciliation.

Example: 99200

{{reconcileReport}}

Access-controlled opaque discrepancy and remediation report.

Example: /srv/rag-governance/reports/reconcile-99200.json

{{backupManifest}}

Encrypted backup contents, high-water marks, compatibility, retention, and restore contract.

Example: /srv/rag-backup/2026-07-28/manifest.json

{{backupArchive}}

Encrypted-at-rest PostgreSQL custom-format archive selected by the reviewed backup manifest.

Example: /srv/rag-backup/2026-07-28/canonical.dump

{{backupChecksums}}

SHA-256 manifest covering the backup archive, schemas, configuration, and security-ledger export.

Example: /srv/rag-backup/2026-07-28/SHA256SUMS

{{currentSecurityLedgerDump}}

Current SQL export of tombstones, expiries, quarantines, holds, and ACL revocations applied after restoring the older archive.

Example: /srv/rag-backup/current-security-ledger.sql

{{restoreDatabaseUrl}}secret

PostgreSQL URI for the isolated no-traffic recovery database.

Example: postgresql://rag_restore@recovery-db.internal/rag_recovery_20260728

{{restoreEnvironment}}

Isolated no-traffic recovery account, cluster, or namespace.

Example: rag-recovery-20260728

{{candidateGeneration}}

Verified update or recovery generation for canary.

Example: kb-1843

{{lifecycleCandidateDir}}

Immutable lifecycle application release prepared for the explicit candidate systemd instance.

Example: /srv/rag-lifecycle/releases/kb-1843

{{lifecycleReleaseChecksums}}

SHA-256 manifest for the lifecycle candidate, policies, schemas, and service configuration.

Example: /srv/rag-lifecycle/releases/kb-1843/SHA256SUMS

{{lifecycleCanaryReport}}

JSON report from the explicit candidate service and governed lifecycle canary.

Example: /srv/rag-governance/reports/kb-1843-canary.json

{{policyFixtures}}

Positive, denied, cross-tenant, malformed, and revoked authorization fixtures.

Example: /srv/rag-governance/fixtures/policy-v7.jsonl

Security and production boundaries

Use a canonical authorization service and durable tenant-scoped lineage ledger. Search indices are derived views and must not become the authority for identity, permissions, retention, legal hold, or deletion state.
Minimize and redact personal or secret-bearing content before chunking and embedding. Embeddings and derived summaries can still reveal information, so they remain protected data with the same tenant and lifecycle controls as their source.
A deletion is complete only after every online representation is inaccessible and a verification job proves absence by stable IDs and content fingerprints. Backups follow a documented expiry and restore-time tombstone process rather than being silently treated as already purged.
Legal hold preserves specifically governed records but does not broaden who may search them. Hold creation, release, retention overrides, restores, and destructive lifecycle actions require auditable authorization and separation of duties.

Stop before continuing if

Stop if a source document lacks an authoritative tenant, stable identity, revision, ACL version, retention class, and accountable owner before ingestion.
Stop if an update, restore, or rollback can reactivate tombstoned, expired, quarantined, or ACL-revoked content.
Stop if deletion verification examines only the user interface or canonical database while vector, keyword, cache, generated-summary, export, and backup paths remain unaccounted for.
Stop if legal-hold scope or release authority is ambiguous; escalate to the organization's records and legal process rather than improvising policy from this technical guide.

decision

Define ownership, classification, retention, legal hold, and recovery objectives

read-only

Inventory every source and derived representation before ingestion: canonical file or row, normalized text, chunks, embeddings, lexical records, summaries, caches, exports, logs, evaluation fixtures, and backups. Assign tenant, data owner, system owner, classification, permitted purposes, residency, retention trigger and duration, deletion deadline, legal-hold authority, review interval, RPO, RTO, and recovery owner. Document which policy or contract supplies each requirement; this guide implements controls but does not invent legal obligations.

Why this step matters

Deletion, backup, cost, and restore decisions conflict unless ownership and policy are explicit. A representation inventory prevents derived vectors or caches from falling outside the lifecycle.

What to understand

Use policy-approved opaque tenant and subject identifiers rather than personal data in manifests.

Distinguish operational recovery retention from product search retention.

Have records, privacy, security, and service owners review exceptional holds and long retention.

System changes

Creates a governed lifecycle and recovery manifest; no content is ingested or deleted.

Syntax explained

purge_within_hours: Maximum verified online-deletion interval after an eligible request.
RPO/RTO: Accepted recoverable data loss and service restoration objectives.
representations: Complete derived-data inventory included in updates, deletion, and restore.

Command

Fill variables0/1 ready

Values stay on this page and are never sent or saved.

Governance Manifest{{governanceManifest}}Dataset ownership, representation, retention, hold, and recovery policy.

jq '.datasets[] | {id,owner,tenant,classification,retention,hold_policy,rpo,rto,representations}' {{governanceManifest}}

Example output / evidence

{"id":"support-kb","owner":"knowledge-ops","tenant":"per-record","classification":"internal","retention":{"trigger":"source-deletion","purge_within_hours":24},"hold_policy":"records-v3","rpo":"4h","rto":"2h","representations":9}

Checkpoint: Checkpoint: Define ownership, classification, retention, legal hold, and recovery objectives

jq '.datasets[] | {id,owner,tenant,classification,retention,hold_policy,rpo,rto,representations}' {{governanceManifest}}

Continue whenEvery dataset and representation has accountable policy, owner, lifecycle, hold, and recovery decisions.

Stop whenRetention or hold authority is ambiguous, a representation lacks ownership, or recovery objectives have never been accepted.

If this step fails

Storage and embedding cost grows although canonical document count is stable.

Likely causeSuperseded generations never expire, retries duplicate vectors, overlap increased, legal holds are misapplied, or temporary stores lack expiration.

Safe checks

Group bytes and vectors by tenant, generation, state, and lineage revision.
Reconcile stable IDs for duplicates.
Review expiration, hold, and snapshot retention decisions.

ResolutionCorrect idempotency and lifecycle policies, expire only data eligible under retention and hold rules, and rebuild a compact verified generation; never delete evidence solely to meet a cost target.

Security notes

Use a canonical authorization service and durable tenant-scoped lineage ledger. Search indices are derived views and must not become the authority for identity, permissions, retention, legal hold, or deletion state.
Minimize and redact personal or secret-bearing content before chunking and embedding. Embeddings and derived summaries can still reveal information, so they remain protected data with the same tenant and lifecycle controls as their source.
A deletion is complete only after every online representation is inaccessible and a verification job proves absence by stable IDs and content fingerprints. Backups follow a documented expiry and restore-time tombstone process rather than being silently treated as already purged.
Legal hold preserves specifically governed records but does not broaden who may search them. Hold creation, release, retention overrides, restores, and destructive lifecycle actions require auditable authorization and separation of duties.

Alternatives

Rehearse the decision against a frozen, non-production corpus and retain the candidate-versus-control evidence before changing live traffic.
Keep the last accepted retrieval or knowledge release available behind an immutable alias so rollback does not depend on rebuilding during an incident.

Stop conditions

Stop if a source document lacks an authoritative tenant, stable identity, revision, ACL version, retention class, and accountable owner before ingestion.
Stop if an update, restore, or rollback can reactivate tombstoned, expired, quarantined, or ACL-revoked content.
Stop if deletion verification examines only the user interface or canonical database while vector, keyword, cache, generated-summary, export, and backup paths remain unaccounted for.
Stop if legal-hold scope or release authority is ambiguous; escalate to the organization's records and legal process rather than improvising policy from this technical guide.

config

Create immutable lineage and a current security ledger

read-only

Give each tenant-scoped document and chunk a stable identity. Record source locator, source revision, normalized content hash, parent identity, parser, redactor, chunker, embedding, index generation, ACL version, lifecycle state, event ID, and timestamps. Store deletions, expiries, quarantines, hold state, and ACL revocations in a durable append-only or equivalently auditable security ledger independent of search snapshots. The ledger is the safety overlay applied to rebuilds, restores, and rollback.

Why this step matters

Immutable lineage explains which transformation created every representation, while a current independent security ledger ensures historical data cannot bypass later revocations or deletion.

What to understand

Tenant scope belongs in stable identity to prevent cross-tenant collisions.

Hashes prove normalized content identity but should not be exposed as public lookup or correlation keys.

Lineage and security events need idempotency and monotonic ordering appropriate to the source.

System changes

Creates lineage and security-ledger schemas and records used by all lifecycle workers.

Syntax explained

event_id: Idempotent ordered change identity used for replay and reconciliation.
state: Active, tombstoned, held, quarantined, expired, or superseded lifecycle decision.
redactor / embedding: Transformation versions required to identify affected derived generations.

File {{lineageSchema}}

Configuration

CREATE TABLE knowledge_security_ledger (
  sequence_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  event_id text NOT NULL UNIQUE,
  tenant_id text NOT NULL,
  document_id text NOT NULL,
  event_type text NOT NULL CHECK (event_type IN ('tombstone','expiry','quarantine','hold','hold_release','acl_revoke')),
  reason text NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now(),
  applied_at timestamptz
);
CREATE TABLE knowledge_purge_targets (
  event_id text NOT NULL REFERENCES knowledge_security_ledger(event_id),
  target text NOT NULL,
  acknowledged_at timestamptz,
  identity_hits integer NOT NULL DEFAULT 0,
  fingerprint_hits integer NOT NULL DEFAULT 0,
  semantic_hits integer NOT NULL DEFAULT 0,
  PRIMARY KEY (event_id, target)
);
CREATE TABLE knowledge_outbox (
  event_id text NOT NULL,
  target text NOT NULL,
  status text NOT NULL DEFAULT 'pending',
  attempts integer NOT NULL DEFAULT 0,
  last_error_code text,
  PRIMARY KEY (event_id, target)
);
CREATE TABLE knowledge_representation_state (
  target text NOT NULL,
  document_id text NOT NULL,
  source_revision text NOT NULL,
  content_sha256 text NOT NULL,
  acl_version integer NOT NULL,
  state text NOT NULL,
  observed_at timestamptz NOT NULL,
  PRIMARY KEY (target, document_id)
);

Example output / evidence

schema=PASS stable_documents=42180 stable_chunks=128430 duplicate_ids=0 orphan_chunks=0 ledger_high_water=99182

Checkpoint: Checkpoint: Create immutable lineage and a current security ledger

Continue whenEvery active chunk maps to one document and complete transformation lineage; security events have a durable high-water mark.

Stop whenStable tenant identity, transformation versions, or an independent current tombstone ledger cannot be guaranteed.

If this step fails

Index reconciliation repeatedly reports missing or extra chunks.

Likely causeCanonical snapshots are changing during comparison, chunker revisions differ, retries are not idempotent, or tombstones and active records use inconsistent high-water marks.

Safe checks

Freeze a canonical revision and compare stable IDs.
Inspect duplicate event idempotency keys and chunker manifest.
Separate expected tombstones from unexplained extras.

ResolutionRebuild an isolated generation from one canonical high-water mark, replay later events in order, fix idempotency, and promote only after parity reaches zero unexplained differences.

Security notes

Use a canonical authorization service and durable tenant-scoped lineage ledger. Search indices are derived views and must not become the authority for identity, permissions, retention, legal hold, or deletion state.
Minimize and redact personal or secret-bearing content before chunking and embedding. Embeddings and derived summaries can still reveal information, so they remain protected data with the same tenant and lifecycle controls as their source.
A deletion is complete only after every online representation is inaccessible and a verification job proves absence by stable IDs and content fingerprints. Backups follow a documented expiry and restore-time tombstone process rather than being silently treated as already purged.
Legal hold preserves specifically governed records but does not broaden who may search them. Hold creation, release, retention overrides, restores, and destructive lifecycle actions require auditable authorization and separation of duties.

Alternatives

Rehearse the decision against a frozen, non-production corpus and retain the candidate-versus-control evidence before changing live traffic.
Keep the last accepted retrieval or knowledge release available behind an immutable alias so rollback does not depend on rebuilding during an incident.

Stop conditions

Stop if a source document lacks an authoritative tenant, stable identity, revision, ACL version, retention class, and accountable owner before ingestion.
Stop if an update, restore, or rollback can reactivate tombstoned, expired, quarantined, or ACL-revoked content.
Stop if deletion verification examines only the user interface or canonical database while vector, keyword, cache, generated-summary, export, and backup paths remain unaccounted for.
Stop if legal-hold scope or release authority is ambiguous; escalate to the organization's records and legal process rather than improvising policy from this technical guide.

config

Enforce tenant authorization at canonical and derived stores

caution

Protect canonical metadata with database privileges and row-level policy, then project the same opaque tenant, allow, deny, and ACL revision into every searchable representation. Normal application roles must not bypass row-level security; privileged maintenance uses separate audited roles and explicit procedures. Missing tenant or ACL metadata denies ingestion and retrieval. Test positive access and wrong-tenant, denied-group, revoked-group, malformed, null, and owner-bypass cases before indexing real content.

Why this step matters

Authorization must exist at the canonical system of record and at retrieval. Defense in depth and explicit bypass testing reduce the chance that a privileged owner or incomplete derived record crosses tenant boundaries.

What to understand

Set tenant context from authenticated server state inside a transaction, never from user-provided SQL.

Audit roles with BYPASSRLS and table ownership because PostgreSQL documents bypass behavior.

Keep search filters pre-retrieval and reconcile their ACL version with canonical state.

System changes

Enables and forces row-level security and introduces a tenant policy for canonical knowledge metadata.

Syntax explained

ENABLE ROW LEVEL SECURITY: Activates per-row policy evaluation for normal access.
FORCE ROW LEVEL SECURITY: Subjects the table owner to row security in ordinary operation.
USING / WITH CHECK: Restricts both visible existing rows and rows accepted by writes.

File {{rlsMigration}}

Configuration

ALTER TABLE knowledge_documents ENABLE ROW LEVEL SECURITY;
ALTER TABLE knowledge_documents FORCE ROW LEVEL SECURITY;
CREATE POLICY tenant_docs ON knowledge_documents
  USING (tenant_id = current_setting('app.tenant_id', true))
  WITH CHECK (tenant_id = current_setting('app.tenant_id', true));

Example output / evidence

migration=applied rls=enabled forced=true app_role_bypass=false positive_cases=24 negative_cases=96 mismatches=0

Checkpoint: Checkpoint: Enforce tenant authorization at canonical and derived stores

Continue whenApplication roles access only the authenticated tenant and derived records carry matching fail-closed ACL metadata.

Stop whenThe application role bypasses RLS, tenant context is client-controlled, or missing ACL metadata is treated as public.

If this step fails

An unauthorized tenant sees another tenant's chunk after ingestion.

Likely causeThe source lacked tenant identity, an ingestion default classified missing tenant as public, IDs collided across tenants, or a privileged worker wrote the wrong ACL metadata.

Safe checks

Disable affected retrieval and preserve opaque audit IDs.
Trace canonical source, ingestion actor, lineage, and every derived record.
Test whether caches or exports expanded exposure.

ResolutionTreat it as a security incident, quarantine affected generations, correct fail-closed schema and tenant-scoped identity, notify accountable responders, and rebuild from reviewed canonical data.

Security notes

Use a canonical authorization service and durable tenant-scoped lineage ledger. Search indices are derived views and must not become the authority for identity, permissions, retention, legal hold, or deletion state.
Minimize and redact personal or secret-bearing content before chunking and embedding. Embeddings and derived summaries can still reveal information, so they remain protected data with the same tenant and lifecycle controls as their source.
A deletion is complete only after every online representation is inaccessible and a verification job proves absence by stable IDs and content fingerprints. Backups follow a documented expiry and restore-time tombstone process rather than being silently treated as already purged.
Legal hold preserves specifically governed records but does not broaden who may search them. Hold creation, release, retention overrides, restores, and destructive lifecycle actions require auditable authorization and separation of duties.

Alternatives

Rehearse the decision against a frozen, non-production corpus and retain the candidate-versus-control evidence before changing live traffic.
Keep the last accepted retrieval or knowledge release available behind an immutable alias so rollback does not depend on rebuilding during an incident.

Stop conditions

Stop if a source document lacks an authoritative tenant, stable identity, revision, ACL version, retention class, and accountable owner before ingestion.
Stop if an update, restore, or rollback can reactivate tombstoned, expired, quarantined, or ACL-revoked content.
Stop if deletion verification examines only the user interface or canonical database while vector, keyword, cache, generated-summary, export, and backup paths remain unaccounted for.
Stop if legal-hold scope or release authority is ambiguous; escalate to the organization's records and legal process rather than improvising policy from this technical guide.

config

Minimize and redact before chunking or embedding

read-only

Classify fields at source and exclude data unnecessary for the retrieval purpose. Replace approved identifiers with opaque references, redact secrets and personal data using deterministic rules plus reviewed detectors, and quarantine uncertain records for human review. Perform these controls before normalization output reaches chunking, embedding, search, model prompts, logs, or evaluation. Version redaction policy and store only minimal audit facts. Test names, emails, account numbers, keys, tokens, free text, OCR errors, Unicode variants, and false-positive business terms.

Why this step matters

Once sensitive text is embedded or copied into multiple indexes, deletion and access scope become harder to prove. Minimizing before derivation reduces breach impact, storage, model exposure, and lifecycle work.

What to understand

Embeddings remain protected derived data even when the original strings are not directly readable.

Do not store redacted values in ordinary audit logs; store detector category, opaque record ID, policy revision, and outcome.

Security credentials should be excluded, not merely masked and embedded.

System changes

Introduces a pre-derivation minimization, redaction, and quarantine gate.

Syntax explained

default: Fail-closed treatment when the redactor cannot make a confident safe decision.
exclude_fields: Data that never enters the knowledge pipeline.
audit_content: false: Prevents the lifecycle audit trail from becoming another sensitive copy.

File {{redactionPolicy}}

Configuration

{"revision":"pii-v6","default":"quarantine-on-uncertain","exclude_fields":["raw_email","access_token","private_key"],"replace":{"customer_id":"opaque_ref"},"detectors":["email","phone","credential","account"],"audit_content":false}

Example output / evidence

records=10000 excluded_fields=18422 redactions=2187 quarantined=43 false_negative_fixture=0 false_positive_rate=0.004 policy=pii-v6

Checkpoint: Checkpoint: Minimize and redact before chunking or embedding

Continue whenReviewed fixtures show zero sensitive false negatives and uncertain content remains outside derived stores.

Stop whenRedaction occurs after embedding, secrets are only masked cosmetically, or uncertain findings are automatically accepted.

If this step fails

PII remains visible in embeddings or chunks after redaction rules changed.

Likely causeRedaction occurred after embedding, only raw text was updated, detectors missed structured variants, or old generations and caches remain retained.

Safe checks

Trace the content hash through pre-redaction, redacted, chunk, vector, and cache lineage.
Run reviewed detector fixtures without exposing findings broadly.
List all generations created under the former rule.

ResolutionQuarantine affected representations, reprocess from minimized source before embedding, purge superseded derived data under policy, verify negative fingerprints, and document residual limitations.

Security notes

Use a canonical authorization service and durable tenant-scoped lineage ledger. Search indices are derived views and must not become the authority for identity, permissions, retention, legal hold, or deletion state.
Minimize and redact personal or secret-bearing content before chunking and embedding. Embeddings and derived summaries can still reveal information, so they remain protected data with the same tenant and lifecycle controls as their source.
A deletion is complete only after every online representation is inaccessible and a verification job proves absence by stable IDs and content fingerprints. Backups follow a documented expiry and restore-time tombstone process rather than being silently treated as already purged.
Legal hold preserves specifically governed records but does not broaden who may search them. Hold creation, release, retention overrides, restores, and destructive lifecycle actions require auditable authorization and separation of duties.

Alternatives

Rehearse the decision against a frozen, non-production corpus and retain the candidate-versus-control evidence before changing live traffic.
Keep the last accepted retrieval or knowledge release available behind an immutable alias so rollback does not depend on rebuilding during an incident.

Stop conditions

Stop if a source document lacks an authoritative tenant, stable identity, revision, ACL version, retention class, and accountable owner before ingestion.
Stop if an update, restore, or rollback can reactivate tombstoned, expired, quarantined, or ACL-revoked content.
Stop if deletion verification examines only the user interface or canonical database while vector, keyword, cache, generated-summary, export, and backup paths remain unaccounted for.
Stop if legal-hold scope or release authority is ambiguous; escalate to the organization's records and legal process rather than improvising policy from this technical guide.

config

Process updates through an idempotent outbox and isolated generation

caution

Commit canonical content, source revision, ACL revision, and a change event atomically. A worker consumes the event idempotently, minimizes and redacts, creates deterministic chunk IDs, and writes lexical, vector, summaries, and caches to a new generation. Reconcile counts and identities, run authorization and retrieval smoke tests, then switch aliases. Keep the former generation for bounded rollback. Never update one derived store synchronously and hope the others eventually catch up without observable state.

Why this step matters

A transactional event plus isolated generation turns multi-store dual writes into observable replayable work and prevents users from seeing a half-updated corpus.

What to understand

Use stable chunk identities where structure is unchanged and explicit supersession where it changes.

Acknowledge the outbox only after target state is durable or retryable.

Bind caches to corpus and ACL revisions so promotion cannot reuse stale answers.

System changes

Writes a new derived generation and atomically changes aliases only after gates pass.

Syntax explained

idempotency: Makes retries safe after worker or network failure.
promote_when: Minimum completeness, security, and usefulness gates before exposure.
retain_previous_hours: Bounded rollback window subject to deletion and retention overlays.

File {{updatePolicy}}

Configuration

{"event":"document.upserted","idempotency":"event_id+target_generation","targets":["lexical","vector","summary"],"promote_when":["id_parity","acl_pass","smoke_pass"],"retain_previous_hours":48}

Example output / evidence

event=evt-99183 source_revision=1843 lexical=complete vector=complete summary=complete parity=PASS acl=PASS alias_switch=kb-1843

Checkpoint: Checkpoint: Process updates through an idempotent outbox and isolated generation

Continue whenAll derived targets represent the same source and ACL revision before the alias switch.

Stop whenEvents can be lost, targets cannot be reconciled, or aliases would expose mixed revisions.

If this step fails

A source was updated but search still returns the old paragraph.

Likely causeThe change event was lost, one derived index failed, aliases point to mixed generations, or cache keys omit corpus and source revisions.

Safe checks

Compare canonical revision and hash with lineage, lexical, vector, summary, and cache manifests.
Inspect outbox attempts and dead letters by stable document ID.
Query candidate and active generations directly using opaque IDs.

ResolutionReplay the idempotent event into an isolated generation, reconcile every representation, invalidate revisioned caches, and switch only after parity and authorization tests pass.

Security notes

Use a canonical authorization service and durable tenant-scoped lineage ledger. Search indices are derived views and must not become the authority for identity, permissions, retention, legal hold, or deletion state.
Minimize and redact personal or secret-bearing content before chunking and embedding. Embeddings and derived summaries can still reveal information, so they remain protected data with the same tenant and lifecycle controls as their source.
A deletion is complete only after every online representation is inaccessible and a verification job proves absence by stable IDs and content fingerprints. Backups follow a documented expiry and restore-time tombstone process rather than being silently treated as already purged.
Legal hold preserves specifically governed records but does not broaden who may search them. Hold creation, release, retention overrides, restores, and destructive lifecycle actions require auditable authorization and separation of duties.

Alternatives

Rehearse the decision against a frozen, non-production corpus and retain the candidate-versus-control evidence before changing live traffic.
Keep the last accepted retrieval or knowledge release available behind an immutable alias so rollback does not depend on rebuilding during an incident.

Stop conditions

Stop if a source document lacks an authoritative tenant, stable identity, revision, ACL version, retention class, and accountable owner before ingestion.
Stop if an update, restore, or rollback can reactivate tombstoned, expired, quarantined, or ACL-revoked content.
Stop if deletion verification examines only the user interface or canonical database while vector, keyword, cache, generated-summary, export, and backup paths remain unaccounted for.
Stop if legal-hold scope or release authority is ambiguous; escalate to the organization's records and legal process rather than improvising policy from this technical guide.

warning

Tombstone deletion immediately and propagate purge everywhere

danger

When an eligible deletion arrives, authenticate and authorize it, resolve stable IDs, and append a tombstone before asynchronous purge. The retrieval gateway denies tombstoned IDs immediately. Propagate idempotent deletion to canonical content as policy permits, raw objects, parsed text, chunks, embeddings, lexical records, summaries, caches, exports, evaluation copies, and temporary vector stores. Record per-target acknowledgement and eventual-consistency deadline. A successful request or missing UI row is not completion.

Why this step matters

An immediate deny separates user-visible safety from backend convergence, while representation-by-representation acknowledgements make complete deletion measurable.

What to understand

Do not log deleted content or include it in support tickets.

Preserve minimal tombstone evidence long enough to prevent resurrection according to policy.

Escalate missed convergence deadlines and keep the gateway block active.

System changes

Commits a tombstone and starts irreversible purge of eligible online representations.

Syntax explained

--document: Stable tenant-scoped identity resolved by an authorized service.
--idempotency: Prevents duplicate requests from producing inconsistent lifecycle state.
--verify-all-targets: Requires observable absence across the representation inventory.

Command

Fill variables0/3 ready

Values stay on this page and are never sent or saved.

Governance Database Url{{governanceDatabaseUrl}}Restricted PostgreSQL URI for canonical metadata, the security ledger, purge acknowledgements, and reconciliation state.Deletion Event{{deletionEvent}}Idempotent authorized deletion request ID.Document Id{{documentId}}Stable tenant-scoped opaque document identity.

psql {{governanceDatabaseUrl}} -v ON_ERROR_STOP=1 -c "BEGIN; SET LOCAL app.tenant_id='tenant-a'; INSERT INTO knowledge_security_ledger(event_id,tenant_id,document_id,event_type,reason) VALUES ('{{deletionEvent}}','tenant-a','{{documentId}}','tombstone','source-deleted') ON CONFLICT (event_id) DO NOTHING; UPDATE knowledge_documents SET state='tombstoned' WHERE tenant_id='tenant-a' AND document_id='{{documentId}}'; INSERT INTO knowledge_purge_targets(event_id,target) SELECT '{{deletionEvent}}', target FROM unnest(ARRAY['raw','parsed','chunks','lexical','vector','summary','cache','export','evaluation']) AS target ON CONFLICT DO NOTHING; INSERT INTO knowledge_outbox(event_id,target) SELECT event_id,target FROM knowledge_purge_targets WHERE event_id='{{deletionEvent}}' ON CONFLICT DO NOTHING; COMMIT;" && psql {{governanceDatabaseUrl}} -Atc "SELECT json_build_object('event',event_id,'targets',count(*),'acknowledged',count(acknowledged_at),'pending',count(*)-count(acknowledged_at)) FROM knowledge_purge_targets WHERE event_id='{{deletionEvent}}' GROUP BY event_id" | jq -e '.event == "{{deletionEvent}}" and .targets == 9'

Example output / evidence

tombstone=committed gateway=blocked targets=9 acknowledged=8 pending=vector-store deadline=2026-07-29T12:00:00Z status=VERIFYING

Checkpoint: Checkpoint: Tombstone deletion immediately and propagate purge everywhere

psql {{governanceDatabaseUrl}} -v ON_ERROR_STOP=1 -c "BEGIN; SET LOCAL app.tenant_id='tenant-a'; INSERT INTO knowledge_security_ledger(event_id,tenant_id,document_id,event_type,reason) VALUES ('{{deletionEvent}}','tenant-a','{{documentId}}','tombstone','source-deleted') ON CONFLICT (event_id) DO NOTHING; UPDATE knowledge_documents SET state='tombstoned' WHERE tenant_id='tenant-a' AND document_id='{{documentId}}'; INSERT INTO knowledge_purge_targets(event_id,target) SELECT '{{deletionEvent}}', target FROM unnest(ARRAY['raw','parsed','chunks','lexical','vector','summary','cache','export','evaluation']) AS target ON CONFLICT DO NOTHING; INSERT INTO knowledge_outbox(event_id,target) SELECT event_id,target FROM knowledge_purge_targets WHERE event_id='{{deletionEvent}}' ON CONFLICT DO NOTHING; COMMIT;" && psql {{governanceDatabaseUrl}} -Atc "SELECT json_build_object('event',event_id,'targets',count(*),'acknowledged',count(acknowledged_at),'pending',count(*)-count(acknowledged_at)) FROM knowledge_purge_targets WHERE event_id='{{deletionEvent}}' GROUP BY event_id" | jq -e '.event == "{{deletionEvent}}" and .targets == 9'

Continue whenRetrieval is blocked immediately and every target reaches verified absence within policy deadline.

Stop whenIdentity or authority is uncertain, legal hold applies, target inventory is incomplete, or the gateway cannot deny during convergence.

If this step fails

A deleted document no longer appears in the UI but can be found by vector similarity.

Likely causeDeletion removed the catalog row but not embeddings, the vector backend is eventually consistent, a cache retained results, or verification tested only keyword search.

Safe checks

Search every store by stable document/chunk IDs and content fingerprints.
Inspect tombstone acknowledgements and backend consistency status.
Repeat negative searches after the declared convergence window.

ResolutionBlock the tombstoned ID at the retrieval gateway immediately, retry idempotent purge across all stores, invalidate caches and derived outputs, and retain the incident until negative verification passes.

Security notes

Use a canonical authorization service and durable tenant-scoped lineage ledger. Search indices are derived views and must not become the authority for identity, permissions, retention, legal hold, or deletion state.
Minimize and redact personal or secret-bearing content before chunking and embedding. Embeddings and derived summaries can still reveal information, so they remain protected data with the same tenant and lifecycle controls as their source.
A deletion is complete only after every online representation is inaccessible and a verification job proves absence by stable IDs and content fingerprints. Backups follow a documented expiry and restore-time tombstone process rather than being silently treated as already purged.
Legal hold preserves specifically governed records but does not broaden who may search them. Hold creation, release, retention overrides, restores, and destructive lifecycle actions require auditable authorization and separation of duties.

Alternatives

Quarantine and revoke retrieval while authority or hold state is resolved; do not destructively purge ambiguous records.
Expire a temporary vector store using its supported policy when the entire store is disposable and no held records share it.

Stop conditions

Stop if a source document lacks an authoritative tenant, stable identity, revision, ACL version, retention class, and accountable owner before ingestion.
Stop if an update, restore, or rollback can reactivate tombstoned, expired, quarantined, or ACL-revoked content.
Stop if deletion verification examines only the user interface or canonical database while vector, keyword, cache, generated-summary, export, and backup paths remain unaccounted for.
Stop if legal-hold scope or release authority is ambiguous; escalate to the organization's records and legal process rather than improvising policy from this technical guide.

decision

Apply legal hold without broadening access

caution

Evaluate hold scope through the organization's authorized records process, not application guesswork. A valid hold changes purge eligibility for specified identities and representations, records authority and review date, and converts deletion requests into deferred actions. It does not make content searchable by new users, cancel ACL revocations, or justify indefinite generic retention. Hold details and access are restricted. Release requires authorized dual control where policy requires it and automatically resumes pending lifecycle actions.

Why this step matters

Legal hold and authorization solve different problems. Separating preservation from access prevents a retention exception from becoming a disclosure exception.

What to understand

This technical workflow is not legal advice; hold creation and release follow approved organizational authority.

Record deferred deletions durably so they execute after release.

Minimize broadly visible status to avoid exposing case or subject details.

System changes

Reads or updates restricted hold and pending-action state; content access remains unchanged.

Syntax explained

--redacted: Shows operational state without exposing privileged case details.
pending_deletion: Ensures an eligible request is deferred rather than silently discarded.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

psql {{governanceDatabaseUrl}} -v ON_ERROR_STOP=1 -Atc "SELECT json_build_object('document',document_id,'hold_active',bool_or(event_type='hold' AND applied_at IS NOT NULL),'pending_deletion',EXISTS (SELECT 1 FROM knowledge_security_ledger d WHERE d.document_id=l.document_id AND d.event_type='tombstone' AND d.applied_at IS NULL),'access_policy','unchanged') FROM knowledge_security_ledger l WHERE document_id='{{documentId}}' GROUP BY document_id" | jq -e '.document == "{{documentId}}" and .access_policy == "unchanged"'

Example output / evidence

document=tenant-a:policy-42 hold=active scope=canonical+derived authority=records-case-opaque review_due=2026-10-01 pending_deletion=true access_policy=unchanged

Checkpoint: Checkpoint: Apply legal hold without broadening access

psql {{governanceDatabaseUrl}} -v ON_ERROR_STOP=1 -Atc "SELECT json_build_object('document',document_id,'hold_active',bool_or(event_type='hold' AND applied_at IS NOT NULL),'pending_deletion',EXISTS (SELECT 1 FROM knowledge_security_ledger d WHERE d.document_id=l.document_id AND d.event_type='tombstone' AND d.applied_at IS NULL),'access_policy','unchanged') FROM knowledge_security_ledger l WHERE document_id='{{documentId}}' GROUP BY document_id" | jq -e '.document == "{{documentId}}" and .access_policy == "unchanged"'

Continue whenHeld records remain preserved but no broader access is granted and pending deletion is scheduled for authorized release.

Stop whenHold authority or scope is uncertain, access would expand, or deferred deletion cannot be retained reliably.

If this step fails

A legal hold prevents an expected deletion with no understandable status.

Likely causeHold scope is broad, release authority is missing, lifecycle workers treat hold as a generic error, or the user-facing status exposes sensitive case details.

Safe checks

Inspect the authorized hold record and scope using a separation-of-duties role.
Confirm the deletion request is recorded as deferred rather than lost.
Check that access remains governed independently.

ResolutionPreserve the scoped record, record deferred deletion and reason in restricted audit data, execute purge automatically after authorized hold release, and avoid disclosing privileged legal details.

Security notes

Use a canonical authorization service and durable tenant-scoped lineage ledger. Search indices are derived views and must not become the authority for identity, permissions, retention, legal hold, or deletion state.
Minimize and redact personal or secret-bearing content before chunking and embedding. Embeddings and derived summaries can still reveal information, so they remain protected data with the same tenant and lifecycle controls as their source.
A deletion is complete only after every online representation is inaccessible and a verification job proves absence by stable IDs and content fingerprints. Backups follow a documented expiry and restore-time tombstone process rather than being silently treated as already purged.
Legal hold preserves specifically governed records but does not broaden who may search them. Hold creation, release, retention overrides, restores, and destructive lifecycle actions require auditable authorization and separation of duties.

Alternatives

Rehearse the decision against a frozen, non-production corpus and retain the candidate-versus-control evidence before changing live traffic.
Keep the last accepted retrieval or knowledge release available behind an immutable alias so rollback does not depend on rebuilding during an incident.

Stop conditions

Stop if a source document lacks an authoritative tenant, stable identity, revision, ACL version, retention class, and accountable owner before ingestion.
Stop if an update, restore, or rollback can reactivate tombstoned, expired, quarantined, or ACL-revoked content.
Stop if deletion verification examines only the user interface or canonical database while vector, keyword, cache, generated-summary, export, and backup paths remain unaccounted for.
Stop if legal-hold scope or release authority is ambiguous; escalate to the organization's records and legal process rather than improvising policy from this technical guide.

verification

Reconcile canonical state, outbox, indexes, caches, and ledger

read-only

Run periodic reconciliation at a declared canonical high-water mark. Compare active stable IDs, revisions, hashes, ACL versions, and lifecycle states against outbox completion, lexical and vector generations, summaries, cache namespaces, and security-ledger application. Classify missing, extra, duplicate, stale, tombstoned-present, and authorization-drift records. Automatically replay safe idempotent work; quarantine ambiguous differences. Reconciliation must not dump source text into logs.

Why this step matters

Asynchronous multi-store pipelines eventually drift through retries, partial failures, and operator changes. A stable-ID reconciliation loop finds silent inconsistency before users or deletion audits do.

What to understand

Use one high-water mark and account explicitly for later events.

Separate expected eventual work from overdue failures and page only on policy-relevant conditions.

Apply security-ledger state before declaring an extra record harmless.

System changes

Reads all manifests, replays safe idempotent jobs, and quarantines ambiguous inconsistencies.

Syntax explained

--as-of: Freezes the comparison boundary across changing systems.
--targets: Enumerates every canonical, derived, cache, and security representation.
--report: Persists opaque discrepancy evidence and remediation status.

Command

Fill variables0/3 ready

Values stay on this page and are never sent or saved.

Governance Database Url{{governanceDatabaseUrl}}Restricted PostgreSQL URI for canonical metadata, the security ledger, purge acknowledgements, and reconciliation state.High Water Mark{{highWaterMark}}Canonical event boundary used for reconciliation.Reconcile Report{{reconcileReport}}Access-controlled opaque discrepancy and remediation report.

psql {{governanceDatabaseUrl}} -v ON_ERROR_STOP=1 -Atc "WITH boundary AS (SELECT max(created_at) AS observed_before FROM knowledge_security_ledger WHERE sequence_id <= {{highWaterMark}}), active AS (SELECT document_id,source_revision,content_sha256,acl_version,state FROM knowledge_documents WHERE state='active'), compared AS (SELECT r.target,count(*) FILTER (WHERE a.document_id IS NULL) AS extra,count(*) FILTER (WHERE a.document_id IS NOT NULL AND (r.source_revision,r.content_sha256,r.acl_version,r.state) IS DISTINCT FROM (a.source_revision,a.content_sha256,a.acl_version,a.state)) AS stale FROM knowledge_representation_state r LEFT JOIN active a USING(document_id), boundary b WHERE r.observed_at <= b.observed_before GROUP BY r.target), missing AS (SELECT count(*) AS value FROM active a WHERE EXISTS (SELECT 1 FROM (VALUES ('lexical'),('vector'),('summary'),('cache')) required(target) WHERE NOT EXISTS (SELECT 1 FROM knowledge_representation_state r, boundary b WHERE r.document_id=a.document_id AND r.target=required.target AND r.observed_at <= b.observed_before))) SELECT json_build_object('as_of',{{highWaterMark}},'missing',(SELECT value FROM missing),'extra',coalesce(sum(extra),0),'stale',coalesce(sum(stale),0),'pending_retry',(SELECT count(*) FROM knowledge_outbox WHERE status='pending')) FROM compared" | tee {{reconcileReport}} | jq -e '.missing == 0 and .extra == 0 and .stale == 0'

Example output / evidence

as_of=99200 canonical_active=128410 missing=0 extra=0 duplicate=0 stale_revision=0 stale_acl=0 tombstoned_present=0 pending_retry=3 quarantined=0 PASS

Checkpoint: Checkpoint: Reconcile canonical state, outbox, indexes, caches, and ledger

psql {{governanceDatabaseUrl}} -v ON_ERROR_STOP=1 -Atc "WITH boundary AS (SELECT max(created_at) AS observed_before FROM knowledge_security_ledger WHERE sequence_id <= {{highWaterMark}}), active AS (SELECT document_id,source_revision,content_sha256,acl_version,state FROM knowledge_documents WHERE state='active'), compared AS (SELECT r.target,count(*) FILTER (WHERE a.document_id IS NULL) AS extra,count(*) FILTER (WHERE a.document_id IS NOT NULL AND (r.source_revision,r.content_sha256,r.acl_version,r.state) IS DISTINCT FROM (a.source_revision,a.content_sha256,a.acl_version,a.state)) AS stale FROM knowledge_representation_state r LEFT JOIN active a USING(document_id), boundary b WHERE r.observed_at <= b.observed_before GROUP BY r.target), missing AS (SELECT count(*) AS value FROM active a WHERE EXISTS (SELECT 1 FROM (VALUES ('lexical'),('vector'),('summary'),('cache')) required(target) WHERE NOT EXISTS (SELECT 1 FROM knowledge_representation_state r, boundary b WHERE r.document_id=a.document_id AND r.target=required.target AND r.observed_at <= b.observed_before))) SELECT json_build_object('as_of',{{highWaterMark}},'missing',(SELECT value FROM missing),'extra',coalesce(sum(extra),0),'stale',coalesce(sum(stale),0),'pending_retry',(SELECT count(*) FROM knowledge_outbox WHERE status='pending')) FROM compared" | tee {{reconcileReport}} | jq -e '.missing == 0 and .extra == 0 and .stale == 0'

Continue whenNo unexplained identity, revision, ACL, lifecycle, or tombstone differences remain at the high-water mark.

Stop whenComparison uses changing snapshots, text must be broadly logged, or ambiguous differences would be deleted automatically.

If this step fails

Index reconciliation repeatedly reports missing or extra chunks.

Likely causeCanonical snapshots are changing during comparison, chunker revisions differ, retries are not idempotent, or tombstones and active records use inconsistent high-water marks.

Safe checks

Freeze a canonical revision and compare stable IDs.
Inspect duplicate event idempotency keys and chunker manifest.
Separate expected tombstones from unexplained extras.

ResolutionRebuild an isolated generation from one canonical high-water mark, replay later events in order, fix idempotency, and promote only after parity reaches zero unexplained differences.

Security notes

Use a canonical authorization service and durable tenant-scoped lineage ledger. Search indices are derived views and must not become the authority for identity, permissions, retention, legal hold, or deletion state.
Minimize and redact personal or secret-bearing content before chunking and embedding. Embeddings and derived summaries can still reveal information, so they remain protected data with the same tenant and lifecycle controls as their source.
A deletion is complete only after every online representation is inaccessible and a verification job proves absence by stable IDs and content fingerprints. Backups follow a documented expiry and restore-time tombstone process rather than being silently treated as already purged.
Legal hold preserves specifically governed records but does not broaden who may search them. Hold creation, release, retention overrides, restores, and destructive lifecycle actions require auditable authorization and separation of duties.

Alternatives

Rehearse the decision against a frozen, non-production corpus and retain the candidate-versus-control evidence before changing live traffic.
Keep the last accepted retrieval or knowledge release available behind an immutable alias so rollback does not depend on rebuilding during an incident.

Stop conditions

Stop if a source document lacks an authoritative tenant, stable identity, revision, ACL version, retention class, and accountable owner before ingestion.
Stop if an update, restore, or rollback can reactivate tombstoned, expired, quarantined, or ACL-revoked content.
Stop if deletion verification examines only the user interface or canonical database while vector, keyword, cache, generated-summary, export, and backup paths remain unaccounted for.
Stop if legal-hold scope or release authority is ambiguous; escalate to the organization's records and legal process rather than improvising policy from this technical guide.

verification

Verify deletion by identity, fingerprint, and semantic probes

read-only

After target acknowledgements and documented consistency windows, query every online representation by stable document and chunk IDs, exact normalized fingerprints, cache keys, and safe synthetic semantic probes designed during ingestion. Confirm canonical eligibility, lexical hits, vector hits, summaries, exports, and application responses are absent. Verify from more than one replica or region where applicable. Record only opaque evidence and operation IDs. Keep tombstone denial active even after verification to protect future restore and replay.

Why this step matters

Backend acknowledgements and UI behavior do not prove absence. Multiple lookup methods and replicas reduce false completion, while the durable tombstone prevents later resurrection.

What to understand

Semantic probes must be synthetic and approved; do not reinsert deleted sensitive phrases into logs.

Define maximum verification time and escalation rather than polling forever.

Backup expiry is tracked separately because online verification does not rewrite historical snapshots.

System changes

Performs read-only negative verification and marks the deletion complete when every online target passes.

Syntax explained

--targets all-online: Uses the governed representation inventory rather than a hard-coded single database.
--replicas all: Detects lagging replicas or regions.
--repeat 3: Checks stable absence after eventual consistency convergence.

Command

Fill variables0/2 ready

Values stay on this page and are never sent or saved.

bash -eu -o pipefail -c 'for attempt in 1 2 3; do psql {{governanceDatabaseUrl}} -Atc "SELECT json_build_object('"'"'event'"'"',event_id,'"'"'targets'"'"',count(*),'"'"'unacknowledged'"'"',count(*) FILTER (WHERE acknowledged_at IS NULL),'"'"'identity_hits'"'"',sum(identity_hits),'"'"'fingerprint_hits'"'"',sum(fingerprint_hits),'"'"'semantic_hits'"'"',sum(semantic_hits)) FROM knowledge_purge_targets WHERE event_id='"'"'{{deletionEvent}}'"'"' GROUP BY event_id" | jq -e '"'"'.targets == 9 and .unacknowledged == 0 and .identity_hits == 0 and .fingerprint_hits == 0 and .semantic_hits == 0'"'"'; done'

Example output / evidence

event=del-771 targets=9 replicas=6 attempts=3 id_hits=0 fingerprint_hits=0 semantic_hits=0 cache_hits=0 application_disclosures=0 status=VERIFIED

Checkpoint: Checkpoint: Verify deletion by identity, fingerprint, and semantic probes

bash -eu -o pipefail -c 'for attempt in 1 2 3; do psql {{governanceDatabaseUrl}} -Atc "SELECT json_build_object('"'"'event'"'"',event_id,'"'"'targets'"'"',count(*),'"'"'unacknowledged'"'"',count(*) FILTER (WHERE acknowledged_at IS NULL),'"'"'identity_hits'"'"',sum(identity_hits),'"'"'fingerprint_hits'"'"',sum(fingerprint_hits),'"'"'semantic_hits'"'"',sum(semantic_hits)) FROM knowledge_purge_targets WHERE event_id='"'"'{{deletionEvent}}'"'"' GROUP BY event_id" | jq -e '"'"'.targets == 9 and .unacknowledged == 0 and .identity_hits == 0 and .fingerprint_hits == 0 and .semantic_hits == 0'"'"'; done'

Continue whenAll lookup modes across online targets and replicas return zero evidence, with tombstone protection retained.

Stop whenAny hit remains, a target is unobservable, or verification would disclose deleted content.

If this step fails

Deletion verification reports success immediately but the backend documents eventual consistency.

Likely causeThe verifier checked an acknowledgement rather than observable absence, used one query shape, or ignored caches and replicas.

Safe checks

Record backend operation IDs and consistency guarantees.
Search by IDs, exact fingerprints, and semantic probes across replicas.
Repeat until the policy deadline or escalation threshold.

ResolutionKeep the gateway tombstone deny active, poll bounded verification across all representations, escalate missed deadlines, and never translate an accepted request into completed deletion prematurely.

Security notes

Use a canonical authorization service and durable tenant-scoped lineage ledger. Search indices are derived views and must not become the authority for identity, permissions, retention, legal hold, or deletion state.
Minimize and redact personal or secret-bearing content before chunking and embedding. Embeddings and derived summaries can still reveal information, so they remain protected data with the same tenant and lifecycle controls as their source.
A deletion is complete only after every online representation is inaccessible and a verification job proves absence by stable IDs and content fingerprints. Backups follow a documented expiry and restore-time tombstone process rather than being silently treated as already purged.
Legal hold preserves specifically governed records but does not broaden who may search them. Hold creation, release, retention overrides, restores, and destructive lifecycle actions require auditable authorization and separation of duties.

Alternatives

Rehearse the decision against a frozen, non-production corpus and retain the candidate-versus-control evidence before changing live traffic.
Keep the last accepted retrieval or knowledge release available behind an immutable alias so rollback does not depend on rebuilding during an incident.

Stop conditions

Stop if a source document lacks an authoritative tenant, stable identity, revision, ACL version, retention class, and accountable owner before ingestion.
Stop if an update, restore, or rollback can reactivate tombstoned, expired, quarantined, or ACL-revoked content.
Stop if deletion verification examines only the user interface or canonical database while vector, keyword, cache, generated-summary, export, and backup paths remain unaccounted for.
Stop if legal-hold scope or release authority is ambiguous; escalate to the organization's records and legal process rather than improvising policy from this technical guide.

config

Back up recoverable state with retention and tombstone-aware manifests

caution

Back up canonical content eligible under policy, schemas, lineage, security ledger, ACL policy, index mappings, prompts, application configuration, and keys through the approved encrypted key-management path. Use supported snapshots for search indexes but treat indexes as rebuildable derived state. Record corpus and ledger high-water marks, software compatibility, encryption key reference, retention, expiry, and restore procedure. Historical backups may contain later-deleted data; restrict access, expire them according to policy, and require current tombstones during restore.

Why this step matters

A file copy is not a usable recovery point. Complete manifests and current security overlay requirements preserve both recoverability and deletion guarantees.

What to understand

Never log or embed encryption keys in the manifest; store a managed key reference.

Test snapshot compatibility with the target software version.

Deletion from online systems does not imply immediate physical removal from governed immutable backups; document and enforce expiry.

System changes

Creates encrypted governed backups and search snapshots subject to explicit retention.

Syntax explained

high_water: Orders restored canonical and security events deterministically.
retention_days: Maximum backup retention unless an authorized scoped hold applies.
restore_mode: Requires isolation and current tombstone replay before exposure.

File {{backupManifest}}

Configuration

{"backup":"rag-2026-07-28T00","canonical_high_water":99190,"security_ledger_high_water":99203,"includes":["canonical","lineage","security-ledger","acl-policy","index-mappings","app-config"],"encrypted":true,"retention_days":30,"restore_mode":"isolated+tombstone-replay"}

Example output / evidence

backup=rag-2026-07-28T00 objects=43810 bytes=918273645 encrypted=true integrity=PASS retention_expiry=2026-08-27 restore_test=scheduled

Checkpoint: Checkpoint: Back up recoverable state with retention and tombstone-aware manifests

Continue whenBackup integrity passes and the manifest contains everything required for isolated authorized recovery and policy-compliant expiry.

Stop whenKeys, security ledger, mappings, compatibility, retention, or restore ownership is missing.

If this step fails

A backup completes successfully but an isolated restore cannot rebuild usable search.

Likely causeThe snapshot omitted lineage, mappings, embedding model, ACL ledger, tombstones, keys, or application configuration, or compatibility was never tested.

Safe checks

Inventory backup contents against the recovery manifest.
Restore to an isolated account or cluster.
Run identity, ACL, deletion, corpus, and retrieval fixtures.

ResolutionExpand the backup and recovery manifest, retain compatible software and keys under policy, repeat isolated restore, and do not claim the recovery objective until end-to-end verification passes.

Security notes

Use a canonical authorization service and durable tenant-scoped lineage ledger. Search indices are derived views and must not become the authority for identity, permissions, retention, legal hold, or deletion state.
Minimize and redact personal or secret-bearing content before chunking and embedding. Embeddings and derived summaries can still reveal information, so they remain protected data with the same tenant and lifecycle controls as their source.
A deletion is complete only after every online representation is inaccessible and a verification job proves absence by stable IDs and content fingerprints. Backups follow a documented expiry and restore-time tombstone process rather than being silently treated as already purged.
Legal hold preserves specifically governed records but does not broaden who may search them. Hold creation, release, retention overrides, restores, and destructive lifecycle actions require auditable authorization and separation of duties.

Alternatives

Rehearse the decision against a frozen, non-production corpus and retain the candidate-versus-control evidence before changing live traffic.
Keep the last accepted retrieval or knowledge release available behind an immutable alias so rollback does not depend on rebuilding during an incident.

Stop conditions

Stop if a source document lacks an authoritative tenant, stable identity, revision, ACL version, retention class, and accountable owner before ingestion.
Stop if an update, restore, or rollback can reactivate tombstoned, expired, quarantined, or ACL-revoked content.
Stop if deletion verification examines only the user interface or canonical database while vector, keyword, cache, generated-summary, export, and backup paths remain unaccounted for.
Stop if legal-hold scope or release authority is ambiguous; escalate to the organization's records and legal process rather than improvising policy from this technical guide.

verification

Restore in isolation and rebuild derived indexes safely

read-only

Restore canonical, lineage, ACL, and security-ledger state to an isolated account, cluster, or namespace with no user traffic. Validate backup integrity and software compatibility, then apply all current tombstones, expiries, quarantines, hold transitions, and ACL revocations newer than the backup. Rebuild lexical and vector indexes from minimized canonical data using pinned transformations; do not assume an index snapshot is authoritative. Run parity, positive and negative ACL, deletion, retrieval, and cost/capacity tests before considering promotion.

Why this step matters

Isolation prevents historical content from becoming searchable before current policy is applied. Rebuilding derived indexes from canonical minimized data proves lineage and avoids restoring stale hidden state.

What to understand

Test that operators and application roles in the restore cannot bypass tenant controls.

Record restore duration and compare with RTO and capacity assumptions.

Keep restored snapshots isolated until every security and deletion fixture passes.

System changes

Creates an isolated recovery environment and rebuilt derived indexes; it does not route production traffic.

Syntax explained

--replay-security-through latest: Applies current deletion and authorization state after historical data recovery.
--rebuild-derived: Regenerates search views from canonical governed content.
--verify: Runs identity, access, deletion, retrieval, and integrity gates.

Command

Fill variables0/5 ready

Values stay on this page and are never sent or saved.

Backup Checksums{{backupChecksums}}SHA-256 manifest covering the backup archive, schemas, configuration, and security-ledger export.Restore Database Url{{restoreDatabaseUrl}}PostgreSQL URI for the isolated no-traffic recovery database.Backup Archive{{backupArchive}}Encrypted-at-rest PostgreSQL custom-format archive selected by the reviewed backup manifest.Current Security Ledger Dump{{currentSecurityLedgerDump}}Current SQL export of tombstones, expiries, quarantines, holds, and ACL revocations applied after restoring the older archive.Restore Environment{{restoreEnvironment}}Isolated no-traffic recovery account, cluster, or namespace.

sha256sum -c {{backupChecksums}} && pg_restore --exit-on-error --no-owner --dbname={{restoreDatabaseUrl}} {{backupArchive}} && psql {{restoreDatabaseUrl}} -v ON_ERROR_STOP=1 -f {{currentSecurityLedgerDump}} && sudo systemctl start rag-rebuild@{{restoreEnvironment}}.service && sudo systemctl is-active rag-rebuild@{{restoreEnvironment}}.service && psql {{restoreDatabaseUrl}} -Atc "SELECT count(*) FROM knowledge_documents d JOIN knowledge_security_ledger l USING(document_id) WHERE l.event_type IN ('tombstone','expiry','quarantine','acl_revoke') AND d.state='active'" | grep -x 0

Example output / evidence

restore=complete canonical=42170 security_events_replayed=137 tombstones_applied=19 acl_revocations=7 lexical_chunks=128201 vector_chunks=128201 parity=PASS deletion=PASS acl=PASS

Checkpoint: Checkpoint: Restore in isolation and rebuild derived indexes safely

sha256sum -c {{backupChecksums}} && pg_restore --exit-on-error --no-owner --dbname={{restoreDatabaseUrl}} {{backupArchive}} && psql {{restoreDatabaseUrl}} -v ON_ERROR_STOP=1 -f {{currentSecurityLedgerDump}} && sudo systemctl start rag-rebuild@{{restoreEnvironment}}.service && sudo systemctl is-active rag-rebuild@{{restoreEnvironment}}.service && psql {{restoreDatabaseUrl}} -Atc "SELECT count(*) FROM knowledge_documents d JOIN knowledge_security_ledger l USING(document_id) WHERE l.event_type IN ('tombstone','expiry','quarantine','acl_revoke') AND d.state='active'" | grep -x 0

Continue whenIsolated recovery meets RPO/RTO, current security state is applied, and rebuilt indexes pass all gates.

Stop whenThe restore cannot remain isolated, current ledgers are unavailable, or any deleted or unauthorized fixture is searchable.

If this step fails

Restoring a snapshot makes previously deleted knowledge searchable again.

Likely causeThe restore contains historical data and current tombstones, expiries, quarantines, or ACL revocations were not replayed before the restored index was exposed.

Safe checks

Keep the restored generation isolated.
Compare its high-water mark with the security ledger.
Run deletion and authorization fixtures before any alias switch.

ResolutionReplay the current immutable tombstone and authorization ledgers, rebuild derived views as needed, verify absence, and expose only after the restored generation passes security and lineage gates.

Security notes

Use a canonical authorization service and durable tenant-scoped lineage ledger. Search indices are derived views and must not become the authority for identity, permissions, retention, legal hold, or deletion state.
Minimize and redact personal or secret-bearing content before chunking and embedding. Embeddings and derived summaries can still reveal information, so they remain protected data with the same tenant and lifecycle controls as their source.
A deletion is complete only after every online representation is inaccessible and a verification job proves absence by stable IDs and content fingerprints. Backups follow a documented expiry and restore-time tombstone process rather than being silently treated as already purged.
Legal hold preserves specifically governed records but does not broaden who may search them. Hold creation, release, retention overrides, restores, and destructive lifecycle actions require auditable authorization and separation of duties.

Alternatives

Rehearse the decision against a frozen, non-production corpus and retain the candidate-versus-control evidence before changing live traffic.
Keep the last accepted retrieval or knowledge release available behind an immutable alias so rollback does not depend on rebuilding during an incident.

Stop conditions

Stop if a source document lacks an authoritative tenant, stable identity, revision, ACL version, retention class, and accountable owner before ingestion.
Stop if an update, restore, or rollback can reactivate tombstoned, expired, quarantined, or ACL-revoked content.
Stop if deletion verification examines only the user interface or canonical database while vector, keyword, cache, generated-summary, export, and backup paths remain unaccounted for.
Stop if legal-hold scope or release authority is ambiguous; escalate to the organization's records and legal process rather than improvising policy from this technical guide.

decision

Canary recovery or update and preserve security during rollback

caution

Expose a verified new or recovered generation through a small stable canary. Monitor corpus and security high-water marks, missing and extra IDs, ACL mismatches, tombstoned hits, update and deletion lag, cache version, retrieval quality, latency, and storage cost. If a content or ranking regression occurs, restore the previous generation only after applying the current security ledger. Never roll back a tombstone, expiry, quarantine, legal-hold decision, or ACL revocation merely to recover relevance or availability.

Why this step matters

Search content and security state have different rollback semantics. A canary and explicit current-ledger overlay allow relevance rollback without reversing deletion or authorization.

What to understand

Use stable traffic assignment and synthetic deletion/ACL probes.

Alert on policy deadlines, not only worker errors.

Retain failed generation evidence under policy and require a new immutable revision before retry.

System changes

Routes bounded traffic to the verified generation and rehearses a security-preserving rollback.

Syntax explained

--abort-on: Automatically rejects security, lineage, and operational boundary violations.
--preserve-security-ledger: Prevents historical generations from reviving prohibited content.

Command

Fill variables0/5 ready

Values stay on this page and are never sent or saved.

Lifecycle Release Checksums{{lifecycleReleaseChecksums}}SHA-256 manifest for the lifecycle candidate, policies, schemas, and service configuration.Lifecycle Candidate Dir{{lifecycleCandidateDir}}Immutable lifecycle application release prepared for the explicit candidate systemd instance.Candidate Generation{{candidateGeneration}}Verified update or recovery generation for canary.Lifecycle Canary Report{{lifecycleCanaryReport}}JSON report from the explicit candidate service and governed lifecycle canary.Governance Database Url{{governanceDatabaseUrl}}Restricted PostgreSQL URI for canonical metadata, the security ledger, purge acknowledgements, and reconciliation state.

sha256sum -c {{lifecycleReleaseChecksums}} && sudo ln -sfn {{lifecycleCandidateDir}} /srv/rag-lifecycle/candidate && sudo systemctl restart rag-lifecycle@candidate.service && curl -fsS http://127.0.0.1:8501/healthz | jq -e '.generation == "{{candidateGeneration}}" and .ready == true' && jq -e '.tombstone_hits == 0 and .acl_mismatches == 0 and .lineage_drift == 0 and .p95_ms <= 350' {{lifecycleCanaryReport}} && sudo ln -sfn "$(readlink -f /srv/rag-lifecycle/current)" /srv/rag-lifecycle/previous && sudo ln -sfn {{lifecycleCandidateDir}} /srv/rag-lifecycle/current && sudo systemctl restart rag-lifecycle@current.service && sudo ln -sfn "$(readlink -f /srv/rag-lifecycle/previous)" /srv/rag-lifecycle/current && sudo systemctl restart rag-lifecycle@current.service && psql {{governanceDatabaseUrl}} -Atc "SELECT count(*) FROM knowledge_security_ledger WHERE applied_at IS NULL" | grep -x 0

Example output / evidence

traffic=5% requests=18440 tombstone_hits=0 acl_mismatches=0 lineage_drift=0 p95_ms=227
rollback_drill=PASS restored_generation=kb-1842 security_high_water=99203 restore_seconds=43

Checkpoint: Checkpoint: Canary recovery or update and preserve security during rollback

sha256sum -c {{lifecycleReleaseChecksums}} && sudo ln -sfn {{lifecycleCandidateDir}} /srv/rag-lifecycle/candidate && sudo systemctl restart rag-lifecycle@candidate.service && curl -fsS http://127.0.0.1:8501/healthz | jq -e '.generation == "{{candidateGeneration}}" and .ready == true' && jq -e '.tombstone_hits == 0 and .acl_mismatches == 0 and .lineage_drift == 0 and .p95_ms <= 350' {{lifecycleCanaryReport}} && sudo ln -sfn "$(readlink -f /srv/rag-lifecycle/current)" /srv/rag-lifecycle/previous && sudo ln -sfn {{lifecycleCandidateDir}} /srv/rag-lifecycle/current && sudo systemctl restart rag-lifecycle@current.service && sudo ln -sfn "$(readlink -f /srv/rag-lifecycle/previous)" /srv/rag-lifecycle/current && sudo systemctl restart rag-lifecycle@current.service && psql {{governanceDatabaseUrl}} -Atc "SELECT count(*) FROM knowledge_security_ledger WHERE applied_at IS NULL" | grep -x 0

Continue whenCanary passes security and operations, and rollback restores content behavior while keeping current security state.

Stop whenRelease telemetry lacks stable revisions, automatic abort is unavailable, or rollback would resurrect prohibited data.

If this step fails

Rollback fixes bad relevance but revives stale ACLs or deleted documents.

Likely causeThe team restored a historical search snapshot as a complete truth rather than a derived view that must consume current security and tombstone ledgers.

Safe checks

Keep the rollback generation isolated.
Compare its corpus high-water mark to current authorization and deletion ledgers.
Run all security and deletion fixtures before routing.

ResolutionApply current revocations, holds, expiries, and tombstones to the former content generation, verify negative searches, and then restore traffic; choose availability loss over prohibited resurrection.

Security notes

Use a canonical authorization service and durable tenant-scoped lineage ledger. Search indices are derived views and must not become the authority for identity, permissions, retention, legal hold, or deletion state.
Minimize and redact personal or secret-bearing content before chunking and embedding. Embeddings and derived summaries can still reveal information, so they remain protected data with the same tenant and lifecycle controls as their source.
A deletion is complete only after every online representation is inaccessible and a verification job proves absence by stable IDs and content fingerprints. Backups follow a documented expiry and restore-time tombstone process rather than being silently treated as already purged.
Legal hold preserves specifically governed records but does not broaden who may search them. Hold creation, release, retention overrides, restores, and destructive lifecycle actions require auditable authorization and separation of duties.

Alternatives

Rehearse the decision against a frozen, non-production corpus and retain the candidate-versus-control evidence before changing live traffic.
Keep the last accepted retrieval or knowledge release available behind an immutable alias so rollback does not depend on rebuilding during an incident.

Stop conditions

Stop if a source document lacks an authoritative tenant, stable identity, revision, ACL version, retention class, and accountable owner before ingestion.
Stop if an update, restore, or rollback can reactivate tombstoned, expired, quarantined, or ACL-revoked content.
Stop if deletion verification examines only the user interface or canonical database while vector, keyword, cache, generated-summary, export, and backup paths remain unaccounted for.
Stop if legal-hold scope or release authority is ambiguous; escalate to the organization's records and legal process rather than improvising policy from this technical guide.

Finish line

Verification checklist

Lineage and consistencyjq -e '.missing == 0 and .extra == 0 and .stale == 0' {{reconcileReport}}Zero unexplained missing, extra, duplicate, stale-revision, stale-ACL, orphan, or tombstoned-present records.

Tenant isolation and minimization

psql {{governanceDatabaseUrl}} -v ON_ERROR_STOP=1 -f {{rlsMigration}} && jq -e '.default == "quarantine-on-uncertain" and .audit_content == false and (.exclude_fields | index("private_key")) != null' {{redactionPolicy}} && psql {{governanceDatabaseUrl}} -Atc "SELECT relrowsecurity::int || ':' || relforcerowsecurity::int FROM pg_class WHERE oid='knowledge_documents'::regclass" | grep -x '1:1'

All cross-tenant and denied cases fail closed, authorized cases pass, and reviewed sensitive fixtures are excluded or redacted before derivation.

Deletion evidence

psql {{governanceDatabaseUrl}} -Atc "SELECT count(*) FROM knowledge_purge_targets WHERE event_id='{{deletionEvent}}' AND (acknowledged_at IS NULL OR identity_hits<>0 OR fingerprint_hits<>0 OR semantic_hits<>0)" | grep -x 0

Stable IDs, fingerprints, semantic probes, caches, exports, and application responses show zero online evidence.

Isolated recovery

sha256sum -c {{backupChecksums}} && psql {{restoreDatabaseUrl}} -Atc "SELECT count(*) FROM knowledge_documents d JOIN knowledge_security_ledger l USING(document_id) WHERE l.event_type IN ('tombstone','expiry','quarantine','acl_revoke') AND d.state='active'" | grep -x 0 && systemctl is-active rag-rebuild@{{restoreEnvironment}}.service

Recovery meets RPO/RTO and passes current tombstone, ACL, lineage, corpus, and retrieval gates before exposure.

Recovery guidance

Common problems and safe checks

A source was updated but search still returns the old paragraph.

Likely causeThe change event was lost, one derived index failed, aliases point to mixed generations, or cache keys omit corpus and source revisions.

Safe checks

Compare canonical revision and hash with lineage, lexical, vector, summary, and cache manifests.
Inspect outbox attempts and dead letters by stable document ID.
Query candidate and active generations directly using opaque IDs.

ResolutionReplay the idempotent event into an isolated generation, reconcile every representation, invalidate revisioned caches, and switch only after parity and authorization tests pass.

A deleted document no longer appears in the UI but can be found by vector similarity.

Likely causeDeletion removed the catalog row but not embeddings, the vector backend is eventually consistent, a cache retained results, or verification tested only keyword search.

Safe checks

Search every store by stable document/chunk IDs and content fingerprints.
Inspect tombstone acknowledgements and backend consistency status.
Repeat negative searches after the declared convergence window.

Restoring a snapshot makes previously deleted knowledge searchable again.

Likely causeThe restore contains historical data and current tombstones, expiries, quarantines, or ACL revocations were not replayed before the restored index was exposed.

Safe checks

Keep the restored generation isolated.
Compare its high-water mark with the security ledger.
Run deletion and authorization fixtures before any alias switch.

A user loses access to valid documents after a group or policy change.

Likely causeACL projection lagged, allow/deny precedence changed, a cache retained an old group revision, or the search generation lacks current policy metadata.

Safe checks

Compare canonical authorization output with indexed ACL version.
Inspect policy and membership cache keys.
Run positive and negative fixtures for the principal archetype.

ResolutionRepair the projection, replay the change, invalidate revisioned caches, and prove positive access plus cross-tenant denial; never remove deny enforcement to recover recall.

An unauthorized tenant sees another tenant's chunk after ingestion.

Likely causeThe source lacked tenant identity, an ingestion default classified missing tenant as public, IDs collided across tenants, or a privileged worker wrote the wrong ACL metadata.

Safe checks

Disable affected retrieval and preserve opaque audit IDs.
Trace canonical source, ingestion actor, lineage, and every derived record.
Test whether caches or exports expanded exposure.

PII remains visible in embeddings or chunks after redaction rules changed.

Likely causeRedaction occurred after embedding, only raw text was updated, detectors missed structured variants, or old generations and caches remain retained.

Safe checks

Trace the content hash through pre-redaction, redacted, chunk, vector, and cache lineage.
Run reviewed detector fixtures without exposing findings broadly.
List all generations created under the former rule.

A legal hold prevents an expected deletion with no understandable status.

Likely causeHold scope is broad, release authority is missing, lifecycle workers treat hold as a generic error, or the user-facing status exposes sensitive case details.

Safe checks

Inspect the authorized hold record and scope using a separation-of-duties role.
Confirm the deletion request is recorded as deferred rather than lost.
Check that access remains governed independently.

Index reconciliation repeatedly reports missing or extra chunks.

Likely causeCanonical snapshots are changing during comparison, chunker revisions differ, retries are not idempotent, or tombstones and active records use inconsistent high-water marks.

Safe checks

Freeze a canonical revision and compare stable IDs.
Inspect duplicate event idempotency keys and chunker manifest.
Separate expected tombstones from unexplained extras.

ResolutionRebuild an isolated generation from one canonical high-water mark, replay later events in order, fix idempotency, and promote only after parity reaches zero unexplained differences.

Deletion verification reports success immediately but the backend documents eventual consistency.

Likely causeThe verifier checked an acknowledgement rather than observable absence, used one query shape, or ignored caches and replicas.

Safe checks

Record backend operation IDs and consistency guarantees.
Search by IDs, exact fingerprints, and semantic probes across replicas.
Repeat until the policy deadline or escalation threshold.

A backup completes successfully but an isolated restore cannot rebuild usable search.

Likely causeThe snapshot omitted lineage, mappings, embedding model, ACL ledger, tombstones, keys, or application configuration, or compatibility was never tested.

Safe checks

Inventory backup contents against the recovery manifest.
Restore to an isolated account or cluster.
Run identity, ACL, deletion, corpus, and retrieval fixtures.

Storage and embedding cost grows although canonical document count is stable.

Likely causeSuperseded generations never expire, retries duplicate vectors, overlap increased, legal holds are misapplied, or temporary stores lack expiration.

Safe checks

Group bytes and vectors by tenant, generation, state, and lineage revision.
Reconcile stable IDs for duplicates.
Review expiration, hold, and snapshot retention decisions.

Rollback fixes bad relevance but revives stale ACLs or deleted documents.

Likely causeThe team restored a historical search snapshot as a complete truth rather than a derived view that must consume current security and tombstone ledgers.

Safe checks

Keep the rollback generation isolated.
Compare its corpus high-water mark to current authorization and deletion ledgers.
Run all security and deletion fixtures before routing.

After the procedure

Alternatives and next steps

Consider these alternatives

Use short-lived disposable vector stores with explicit expiration for temporary low-risk corpora, while still tracking source authority, access, and verified expiry.
Keep highly sensitive data out of RAG entirely and retrieve an opaque reference to a separately authorized system when minimization cannot make derivation acceptable.
Rebuild search indexes from canonical data rather than backing them up when rebuild time meets RTO and reduces compatibility and deletion complexity.
Use physically or account-separated tenant stores for boundaries that cannot safely share metadata filtering, while preserving common lineage and lifecycle contracts.

Operate it safely

Automate weekly source-to-derived reconciliation, daily overdue deletion checks, backup expiry, hold review reminders, ACL lag, redaction drift, orphan detection, and storage cost by generation.
Run quarterly deletion, cross-tenant, lost-event, partial-index, legal-hold release, corrupt-backup, isolated restore, and security-preserving rollback exercises.
Review every new derived use—summaries, fine-tuning, evaluation, analytics, exports, caches—as a lifecycle representation before enabling it.
Measure actual deletion convergence and restore RPO/RTO by provider and corpus size, then update capacity and escalation thresholds from observed evidence.

Reference

Frequently asked questions

Are embeddings anonymous after text is converted to vectors?

No. They are protected derived data and may reveal or correlate source information. Apply the same tenant, access, retention, deletion, and backup controls as the source.

Is deletion complete when the source row disappears?

No. It requires immediate retrieval denial, purge across every derived and cache representation, and negative verification after consistency windows.

Should backups be rewritten for every online deletion?

That depends on approved policy and storage design. Commonly immutable backups remain restricted until scheduled expiry, and current tombstones are applied during restore before exposure.

Does legal hold make a record searchable by administrators?

No. Hold preserves scoped data; access remains governed independently. Privileged hold metadata and release actions require authorized audit and separation of duties.

Can we roll back to yesterday's search snapshot?

Only in isolation and only after applying all current tombstones, expiries, quarantines, and ACL revocations. Historical relevance state must not reverse current security.

Why keep a tombstone after verified purge?

A minimal policy-approved tombstone prevents event replay, restore, or old index generations from resurrecting a prohibited stable identity.

Should vector and lexical indexes be backed up?

They can be snapshotted, but treat them as derived. Preserve enough canonical data, lineage, transformations, mappings, keys, and capacity to rebuild and prove policy.

What if deletion verification finds one lagging replica?

Keep retrieval denial active, retry or quarantine the target, escalate the missed deadline, and do not report completion until observable absence passes.

Recovery

Rollback

Rollback content or ranking by restoring the previous verified generation in isolation, then apply the current security ledger before traffic. Deletion, expiry, quarantine, hold state, and ACL revocations are forward-only security decisions and must never be reverted by a historical snapshot.

Stop candidate routing and preserve opaque lineage, event, alert, and reconciliation evidence under incident-retention policy.
Restore the previous canonical or derived generation only in an isolated environment and compare its high-water marks with current state.
Replay every newer tombstone, expiry, quarantine, hold transition, and authorization revocation; invalidate candidate and historical caches.
Rebuild derived indexes where lineage or compatibility is uncertain, then run parity, minimization, positive and negative ACL, deletion, retrieval, and latency tests.
Route a small canary, verify security-ledger high-water mark in live telemetry, and expand only after the recovery owner accepts RPO/RTO evidence.

Evidence

Sources and review

Verified 2026-07-24Review due 2026-10-22

OpenAI Retrieval guide: vector store files, deletion consistency, attributes, and expirationofficial PostgreSQL row security policiesofficial Elastic create, monitor, and delete snapshotsofficial Elastic restore a snapshotofficial