Run production semantic search with PostgreSQL and pgvector
Pin PostgreSQL 18 and pgvector 0.8.5, enforce tenant RLS, store one validated vector dimension, ingest idempotently, measure HNSW and IVFFlat against exact recall, and prove maintenance, PITR, restore and rollback.
Provide fast semantic retrieval without sacrificing tenant isolation, evidence provenance, measurable recall, deletion guarantees or database recoverability.
- PostgreSQL 18.x
- pgvector 0.8.5 exact tag
- Embedding schema vector(1536), one pinned model revision
- Database release and recovery ownership PostgreSQL 18 primary/replicas, encrypted backups, WAL archive, isolated restore target, capacity budget and database owner are available.
- Embedding contract The exact embedding model, 1536 dimensions, normalization, data controls, source provenance, cost and replacement procedure are approved.
- Trusted tenant and ACL identity Serving middleware derives tenant/ACL from authentication and supports transaction-local parameterized database context.
- Representative evaluation A frozen multi-tenant corpus and exact top-k query set cover filter selectivity, identifiers, deletions and expected production scale.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A PostgreSQL 18 semantic-search database with pgvector 0.8.5 pinned and recorded as part of one release manifest.
- A provenance-rich `vector(1536)` chunk schema with stable idempotency keys, lifecycle state and citation coordinates.
- A forced row-level security boundary using transaction-local trusted tenant and ACL context for a non-owner runtime role.
- Measured exact, HNSW and optional IVFFlat search with filtered iterative scans, real plans and recall/latency gates.
- A monitored maintenance and upgrade process plus tested pg_dump, physical PITR, restore and revocation-aware rollback.
- Wrong-dimension vectors fail before serving, while an embedding change creates a distinct evaluated vector space.
- Every application query is tenant/ACL constrained inside PostgreSQL and pooled connections cannot retain a previous request's identity.
- ANN speed is accepted only against exact authorized top-k ground truth and worst-filter slices.
- Upsert retries remain idempotent and deletion or access revocation has a measurable primary, replica and cache verification path.
- Operators can reproduce versions, query plans, recall and restore state without logging raw vectors or private source excerpts.
Architecture
How the parts fit together
An authenticated application begins a database transaction, derives tenant and ACL groups from trusted identity and stores them with transaction-local `set_config`. Forced RLS constrains a non-owner runtime role before rows reach the application. Stable chunk records contain provenance, citation coordinates, a pinned embedding revision and `vector(1536)`. Exact search provides evaluation truth; HNSW or evaluated IVFFlat provides serving speed. PostgreSQL statistics, backups, WAL and external release manifests support maintenance and recovery.
- The migration identity installs pinned pgvector and creates owner/runtime separation, schema, forced RLS and reviewed indexes.
- Ingestion validates one exact embedding revision and writes stable chunks in a transaction; retries update one row and supersession/deletion remains explicit.
- The application begins a request transaction and sets tenant plus ACL from authenticated server-side identity using parameters.
- PostgreSQL applies RLS, then exact or ANN search orders authorized active rows with the operator class matching the chosen distance.
- Plans, exact recall, p95 latency, returned count and cross-tenant probes gate index/search settings and every material release.
- Monitoring, logical backup, physical PITR and coupled manifests keep the database recoverable without reintroducing revoked data.
Assumptions
- PostgreSQL 18.x packages, pgvector 0.8.5, a secret manager, encrypted storage, WAL archive and isolated restore environment are available.
- The example schema uses 1536-dimensional vectors from one pinned embedding model; operators choose and verify their exact model before creating the type.
- Tenant and ACL values are derived from authenticated server-side identity and cannot be overridden by request text or document metadata.
- The application driver supports parameterized SQL and transaction-local settings; examples using psql illustrate database behavior rather than secret delivery.
- An approved source/chunk manifest outside the database can reproduce provenance, citation hashes, embedding revision and mandatory deletions.
- A representative frozen evaluation set supplies exact relevant top-k and tenant/filter slices before approximate search is accepted.
Key concepts
- Exact nearest-neighbor search
- Orders all authorized candidate rows by a vector-distance operator and provides perfect recall for the stored vectors; used as evaluation ground truth.
- HNSW
- Graph-based approximate index with strong speed/recall behavior, slower builds and higher memory than IVFFlat.
- IVFFlat
- Centroid/list approximate index with faster build and lower memory, but requiring representative training data and measured lists/probes.
- Operator class
- Index definition connecting a distance operation such as cosine to the corresponding query operator; mismatches prevent correct index use.
- Iterative scan
- pgvector 0.8.x behavior that visits additional ANN candidates when filters remove early results, bounded by search and memory settings.
- Row-level security
- PostgreSQL policies that restrict rows visible or writable to a role; table ownership and `BYPASSRLS` must be controlled.
- Recall@k
- Overlap between approximate and exact authorized top-k results for the same query, evaluated by tenant and filter slice.
- PITR
- Physical base backup plus continuous WAL archive used to restore a PostgreSQL cluster to a chosen recovery point.
Security and production boundaries
- Use a non-owner runtime role with `NOSUPERUSER NOBYPASSRLS`; migration and owner privileges never belong to the serving application.
- Set tenant and ACL only after `BEGIN` with parameterized transaction-local `set_config(..., true)`. Commit or roll back every request and test alternating tenants on reused pool connections.
- RLS protects database rows, not application caches, logs, exported archives or privileged reporting identities. Carry tenant scope through each of those systems.
- Treat vectors as potentially sensitive derived data. Encrypt storage/backups, restrict exports, apply retention and prove deletion from primary, replicas, caches and restored revisions.
- Never log complete vectors, private excerpts, secrets or caller-controlled identifiers as metric labels. Use opaque IDs, revisions, plans, counts and timings.
- A valid HNSW/IVFFlat index does not guarantee recall. Gate approximate search against exact authorized results and zero cross-tenant leakage.
- Do not force the planner globally or raise memory settings without production-sized rehearsal. PostgreSQL may correctly prefer a sequential scan for small/selective data.
- Carry mandatory deletions and ACL revocations forward through PITR and rollback even when restoring an older database point.
Stop before continuing if
- Do not create the serving schema until PostgreSQL 18 and pgvector 0.8.5 provenance plus restore compatibility are verified.
- Do not ingest when embedding model/dimensions are moving or one table would mix vector spaces.
- Do not serve when runtime owns the table, has `BYPASSRLS`, or a missing/stale pooled tenant setting can expose rows.
- Do not accept ANN search without exact ground truth, predefined worst-slice recall/latency gates and zero unauthorized results.
- Do not run reindex or extension upgrade without disk/WAL/replica capacity, restored rehearsal and an accepted rollback path.
- Do not declare backup success until isolated logical and physical restore reproduces extension, roles, forced RLS, indexes, recall and mandatory deletions.
verification
Verify PostgreSQL 18 and pin pgvector 0.8.5
Start on PostgreSQL 18.x with the pgvector 0.8.5 extension package or source artifact built specifically for that PostgreSQL major. Record server version, extension default/installed version, package provenance and image/package digest in the release manifest. Do not use an unpinned container tag or install from the repository default branch. Rehearse extension installation and any update on a restored database before production.
Why this step matters
PostgreSQL major and extension version determine available types, index behavior, iterative scans, upgrade paths and backup compatibility. Capturing them before schema work makes the build reproducible.
What to understand
pgvector publishes tags rather than GitHub Releases; v0.8.5 is the latest upstream tag verified on 2026-07-28. Pin that tag/package and record its immutable commit or package digest.
Keep the extension installed on every primary, replica and restore target before restoring dependent objects.
A package signature verifies provenance, not suitability. Review packaging, vulnerability and operational policy separately.
System changes
- Reads server and extension metadata and creates a release record; no schema or data changes occur.
Syntax explained
PostgreSQL 18.x- Supported database major for this guide; apply current minor updates through the normal database patch process.
pgvector 0.8.5- Exact extension tag whose iterative-scan and index behavior is documented and tested here.
psql --set=ON_ERROR_STOP=1 --dbname=semantic -c "SELECT version();" -c "SELECT name, default_version, installed_version FROM pg_available_extensions WHERE name='vector';"PostgreSQL 18.0 on x86_64-pc-linux-gnu, compiled by gcc, 64-bit name | default_version | installed_version --------+-----------------+------------------- vector | 0.8.5 | 0.8.5 (1 row)
Checkpoint: Checkpoint: Verify PostgreSQL 18 and pin pgvector 0.8.5
Continue whenPostgreSQL reports major 18, vector reports installed/default 0.8.5, provenance is recorded and the same versions exist on the isolated restore target.
Stop whenThe extension is absent/mismatched, package provenance is unknown, production and restore targets differ, or an update has not been rehearsed.
If this step fails
The extension cannot be created or shows a version older than 0.8.5.
Likely causeSee the matching full troubleshooting entry.
Stop promotion and reproduce safely with synthetic tenant data.
ResolutionApply the documented resolution and repeat all release gates.
Security notes
- Use synthetic data, parameterized SQL, trusted tenant context, and source-equivalent embedding controls.
Alternatives
- Rehearse with an immutable synthetic corpus in a disposable restored database.
Stop conditions
- Stop on unknown versions/dimensions, RLS bypass, tenant leakage, or unverified recovery.
config
Create the extension and separate owner, migrator, and runtime roles
Create login roles outside the application schema according to your secret-management process. Use a no-login owner for tables, a controlled migration identity for DDL and a runtime identity with only required DML. The runtime must not own protected tables, be superuser or have `BYPASSRLS`. Revoke broad public schema privileges. Create the vector extension under controlled migration and verify role attributes before the application connects.
Why this step matters
PostgreSQL table owners and `BYPASSRLS` roles normally bypass row security. Separating ownership from runtime prevents an otherwise correct policy from becoming optional in the application path.
What to understand
Inject login passwords or certificates through the deployment secret channel, not SQL committed to source or command-line arguments.
Disable or tightly govern migration credentials outside a change window. The application should never inherit owner membership.
Review existing `PUBLIC` grants at database and schema level; defaults vary between upgraded and newly created databases.
System changes
- Creates three least-privilege roles, revokes public schema creation and installs vector 0.8.5; secrets are supplied separately.
Syntax explained
NOLOGIN owner- Owns schema objects without providing a credential usable by the application.
NOBYPASSRLS- Prevents the role from ignoring row-level security policies.
sql/001_roles_extension.sqlREVOKE CREATE ON SCHEMA public FROM PUBLIC;
CREATE ROLE semantic_owner NOLOGIN NOSUPERUSER NOBYPASSRLS;
CREATE ROLE semantic_migrator LOGIN NOSUPERUSER NOBYPASSRLS;
CREATE ROLE semantic_app LOGIN NOSUPERUSER NOBYPASSRLS;
GRANT semantic_owner TO semantic_migrator;
CREATE EXTENSION IF NOT EXISTS vector WITH VERSION '0.8.5';psql --set=ON_ERROR_STOP=1 --dbname=semantic --file=sql/001_roles_extension.sql && psql --dbname=semantic -c "SELECT rolname, rolsuper, rolbypassrls FROM pg_roles WHERE rolname IN ('semantic_owner','semantic_migrator','semantic_app') ORDER BY 1;"REVOKE
CREATE ROLE
CREATE ROLE
CREATE ROLE
GRANT ROLE
CREATE EXTENSION
rolname | rolsuper | rolbypassrls
--------------------+----------+--------------
semantic_app | f | f
semantic_migrator | f | f
semantic_owner | f | fCheckpoint: Checkpoint: Create the extension and separate owner, migrator, and runtime roles
Continue whenThe extension is 0.8.5; runtime and migrator are non-superuser, `NOBYPASSRLS`; runtime does not inherit owner; public cannot create schema objects.
Stop whenRuntime owns tables, has superuser/BYPASSRLS/owner membership, secrets appear in SQL, or extension installation differs from the reviewed artifact.
If this step fails
A runtime query returns chunks from another tenant.
Likely causeSee the matching full troubleshooting entry.
Stop promotion and reproduce safely with synthetic tenant data.
ResolutionApply the documented resolution and repeat all release gates.
Security notes
- Use synthetic data, parameterized SQL, trusted tenant context, and source-equivalent embedding controls.
Alternatives
- Rehearse with an immutable synthetic corpus in a disposable restored database.
Stop conditions
- Stop on unknown versions/dimensions, RLS bypass, tenant leakage, or unverified recovery.
config
Create a provenance-rich schema with an exact vector dimension
Choose vector dimension from the exact embedding model and freeze it for this table/release. The example uses 1536 dimensions. Store stable tenant, source, revision and chunk IDs; normalized content hash; source coordinates; ACL groups; active/superseded state; embedding revision and timestamps. Put a unique constraint on tenant/source/revision/chunk so retries are idempotent. Do not store raw source text unless the database is approved for its classification; a redacted excerpt is optional.
Why this step matters
Vector dimension is part of the physical type and must match both document and query embeddings. Provenance and stable keys make update, deletion, citations and restore auditable rather than treating vectors as anonymous rows.
What to understand
A new embedding model/dimension gets a parallel table or column and index. Do not cast or pad vectors to force compatibility.
Keep source text outside this table when classification or retention differs; store only coordinates and a content hash for citation resolution.
Use UUID tenant IDs and stable IDs from trusted ingestion, not values extracted from document instructions.
System changes
- Creates the semantic schema, 1536-dimensional chunk table, stable primary key and tenant/active B-tree index.
Syntax explained
vector(1536)- Enforces the exact dimension used by the pinned example embedding model.
citation jsonb- Stores validated source coordinates and display metadata; the application must validate its schema.
sql/002_schema.sqlCREATE SCHEMA semantic AUTHORIZATION semantic_owner;
CREATE TABLE semantic.chunks (
tenant_id uuid NOT NULL,
source_id text NOT NULL,
source_revision text NOT NULL,
chunk_id text NOT NULL,
acl_groups text[] NOT NULL DEFAULT '{}',
content_sha256 char(64) NOT NULL,
citation jsonb NOT NULL,
excerpt text,
embedding_model text NOT NULL,
embedding vector(1536) NOT NULL,
active boolean NOT NULL DEFAULT true,
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
updated_at timestamptz NOT NULL DEFAULT clock_timestamp(),
PRIMARY KEY (tenant_id, source_id, source_revision, chunk_id)
);
ALTER TABLE semantic.chunks OWNER TO semantic_owner;
CREATE INDEX chunks_tenant_active_idx ON semantic.chunks (tenant_id, active);psql --set=ON_ERROR_STOP=1 --dbname=semantic --file=sql/002_schema.sql && psql --dbname=semantic -c "\d+ semantic.chunks"CREATE SCHEMA
CREATE TABLE
ALTER TABLE
CREATE INDEX
Table "semantic.chunks"
Column | Type | Nullable | Default
-------------------+-----------------------+----------+---------------------
tenant_id | uuid | not null |
embedding | vector(1536) | not null |
active | boolean | not null | true
Indexes:
"chunks_pkey" PRIMARY KEY, btree (tenant_id, source_id, source_revision, chunk_id)
"chunks_tenant_active_idx" btree (tenant_id, active)Checkpoint: Checkpoint: Create a provenance-rich schema with an exact vector dimension
Continue whenSchema exists under the no-login owner, vector dimension is 1536, stable primary key prevents duplicates, and tenant/active index is present.
Stop whenEmbedding dimensions are not fixed, table owner is runtime, provenance or tenant key is missing, or raw text storage violates data policy.
If this step fails
PostgreSQL reports `expected 1536 dimensions, not 3072` during insert or query.
Likely causeSee the matching full troubleshooting entry.
Stop promotion and reproduce safely with synthetic tenant data.
ResolutionApply the documented resolution and repeat all release gates.
Security notes
- Use synthetic data, parameterized SQL, trusted tenant context, and source-equivalent embedding controls.
Alternatives
- Rehearse with an immutable synthetic corpus in a disposable restored database.
Stop conditions
- Stop on unknown versions/dimensions, RLS bypass, tenant leakage, or unverified recovery.
config
Enforce tenant and ACL policy with forced RLS
Enable and force row-level security. Create a restrictive policy for the runtime role that requires tenant equality from a transaction-local setting, active state and overlap with trusted ACL groups. Use `NULLIF(current_setting(..., true),'')` so missing context denies access rather than becoming a default tenant. Grant only SELECT/INSERT/UPDATE/DELETE on the table. Test missing, valid and wrong tenant contexts as the actual runtime role.
Why this step matters
RLS places the tenant boundary inside the data engine, but only if the runtime cannot bypass it and request context is scoped safely. Missing context must return zero rows, never all rows.
What to understand
Set tenant and ACL values from authenticated identity through parameters after `BEGIN`; never interpolate strings into `SET` statements.
Connection pools reuse sessions, so transaction-local settings and guaranteed rollback/commit are required for every request.
RLS does not protect application caches, logs or replicas accessed through privileged roles. Apply the same tenant scope throughout the system.
System changes
- Enables forced RLS, creates tenant/ACL policy and grants bounded DML to runtime; missing context now denies rows.
Syntax explained
FORCE ROW LEVEL SECURITY- Makes table owners subject to policy during ordinary access, adding defense in depth.
current_setting(..., true)- Returns NULL instead of error when the setting is missing so policy can deny safely.
sql/003_rls.sqlALTER TABLE semantic.chunks ENABLE ROW LEVEL SECURITY;
ALTER TABLE semantic.chunks FORCE ROW LEVEL SECURITY;
CREATE POLICY chunks_tenant_acl ON semantic.chunks
AS RESTRICTIVE FOR ALL TO semantic_app
USING (
tenant_id = NULLIF(current_setting('app.tenant_id', true), '')::uuid
AND active
AND acl_groups && string_to_array(NULLIF(current_setting('app.acl_groups', true), ''), ',')
)
WITH CHECK (
tenant_id = NULLIF(current_setting('app.tenant_id', true), '')::uuid
AND acl_groups && string_to_array(NULLIF(current_setting('app.acl_groups', true), ''), ',')
);
GRANT USAGE ON SCHEMA semantic TO semantic_app;
GRANT SELECT, INSERT, UPDATE, DELETE ON semantic.chunks TO semantic_app;psql --set=ON_ERROR_STOP=1 --dbname=semantic --file=sql/003_rls.sql && psql --dbname=semantic -c "SELECT c.relrowsecurity, c.relforcerowsecurity, p.polname, p.polroles::regrole[] FROM pg_class c JOIN pg_namespace n ON n.oid=c.relnamespace LEFT JOIN pg_policy p ON p.polrelid=c.oid WHERE n.nspname='semantic' AND c.relname='chunks';"ALTER TABLE
ALTER TABLE
CREATE POLICY
GRANT
GRANT
relrowsecurity | relforcerowsecurity | polname | polroles
----------------+---------------------+----------------------+---------------------
t | t | chunks_tenant_acl | {semantic_app}Checkpoint: Checkpoint: Enforce tenant and ACL policy with forced RLS
Continue whenRLS and FORCE are true, the runtime role is bound to the restrictive policy, missing/wrong tenant returns zero and valid tenant returns only authorized active rows.
Stop whenRuntime can bypass/own the table, context is session-global or client-controlled, ACL policy is permissive by accident, or caches are not tenant scoped.
If this step fails
A runtime query returns chunks from another tenant.
Likely causeSee the matching full troubleshooting entry.
Stop promotion and reproduce safely with synthetic tenant data.
ResolutionApply the documented resolution and repeat all release gates.
Security notes
- Use synthetic data, parameterized SQL, trusted tenant context, and source-equivalent embedding controls.
Alternatives
- Rehearse with an immutable synthetic corpus in a disposable restored database.
Stop conditions
- Stop on unknown versions/dimensions, RLS bypass, tenant leakage, or unverified recovery.
config
Use parameterized idempotent upserts and transactional deletion
Validate dimension, finite numeric values, model revision, tenant, ACL and citation schema before SQL. In one transaction set trusted tenant/ACL context, mark superseded chunks inactive, and upsert stable keys with the new hash, citation, excerpt and vector. Use application driver parameters; the psql example demonstrates variable binding for a synthetic row. For deletion, delete by tenant and stable source/chunk ID under RLS, invalidate tenant caches and verify zero rows on primary and replicas.
Why this step matters
Stable keys make retries safe, while the fixed vector type prevents mixed spaces. The full synthetic unit vector is deterministic, harmless and executable without an embedding-provider call.
What to understand
Production supplies all 1536 finite values through driver parameters; Python creates only this deterministic synthetic fixture.
Use one transaction for supersession and upsert so readers do not see both old and new active revisions.
Deletions require an immediate application deny-list when regulatory or access urgency is shorter than replica/cache propagation.
System changes
- Defines the idempotent upsert and writes one authorized synthetic row inside transaction-local RLS context.
Syntax explained
ON CONFLICT- Makes retry of the same stable chunk update one row rather than create duplicates.
driver parameters- Keep identifiers, metadata and vectors separate from SQL syntax and prevent injection.
sql/004_upsert.sqlBEGIN;
SELECT set_config('app.tenant_id', :'tenant_id', true);
SELECT set_config('app.acl_groups', :'acl_groups', true);
INSERT INTO semantic.chunks
(tenant_id, source_id, source_revision, chunk_id, acl_groups, content_sha256, citation, excerpt, embedding_model, embedding)
VALUES
(:'tenant_id'::uuid, :'source_id', :'source_revision', :'chunk_id', string_to_array(:'acl_groups', ','), :'content_sha256', :'citation'::jsonb, :'excerpt', :'embedding_model', :'embedding'::vector)
ON CONFLICT (tenant_id, source_id, source_revision, chunk_id) DO UPDATE SET
acl_groups=EXCLUDED.acl_groups, content_sha256=EXCLUDED.content_sha256,
citation=EXCLUDED.citation, excerpt=EXCLUDED.excerpt,
embedding_model=EXCLUDED.embedding_model, embedding=EXCLUDED.embedding,
active=true, updated_at=clock_timestamp();
COMMIT;QUERY_VECTOR=$(python3 -c "print('[' + ','.join(['1'] + ['0'] * 1535) + ']')") && psql --set=ON_ERROR_STOP=1 --dbname=semantic --username=semantic_app --variable=tenant_id=11111111-1111-1111-1111-111111111111 --variable=source_id=dns-runbook --variable=source_revision=2026-07-28 --variable=chunk_id=dns-001 --variable=acl_groups=network-admin --variable=content_sha256=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa --variable=citation='{"page":4,"start":120,"end":302}' --variable=excerpt='Synthetic DNS evidence' --variable=embedding_model=text-embedding-3-small --variable=embedding="$QUERY_VECTOR" --file=sql/004_upsert.sqlBEGIN
set_config
--------------------------------------
11111111-1111-1111-1111-111111111111
(1 row)
set_config
---------------
network-admin
(1 row)
INSERT 0 1
COMMITCheckpoint: Checkpoint: Use parameterized idempotent upserts and transactional deletion
Continue whenThe command upserts one authorized row; repeating it returns `INSERT 0 1` while the stable key keeps row count unchanged.
Stop whenThe application concatenates vector/tenant SQL, retries duplicate rows, mixed embedding models enter one column, or deletions cannot be verified through replicas and caches.
If this step fails
Ingestion creates duplicate chunks or an update leaves old embeddings active.
Likely causeSee the matching full troubleshooting entry.
Stop promotion and reproduce safely with synthetic tenant data.
ResolutionApply the documented resolution and repeat all release gates.
Security notes
- Use synthetic data, parameterized SQL, trusted tenant context, and source-equivalent embedding controls.
Alternatives
- Rehearse with an immutable synthetic corpus in a disposable restored database.
Stop conditions
- Stop on unknown versions/dimensions, RLS bypass, tenant leakage, or unverified recovery.
config
Establish exact-search ground truth, then create HNSW concurrently
Load a representative synthetic or approved corpus and preserve an exact nearest-neighbor query as evaluation ground truth before adding approximate indexes. For cosine distance, order directly by `<=>` and limit the result; do not wrap the indexed expression. Create HNSW concurrently with `vector_cosine_ops` so writes continue, beginning with upstream defaults `m=16` and `ef_construction=64`. Monitor progress and memory instead of raising `maintenance_work_mem` beyond safe host capacity.
Why this step matters
Exact search provides perfect-recall truth for a frozen corpus. HNSW trades some recall for speed and memory; starting from documented defaults and measuring against exact results avoids tuning by anecdote.
What to understand
HNSW can be created before or after data because it has no training phase, but bulk load followed by index creation is often faster.
The operator class must match the query distance: cosine uses `vector_cosine_ops` with `<=>`; L2, inner product and L1 need their own classes and indexes.
Partial `WHERE active` keeps inactive historical rows out of ANN search, while tenant/ACL filters remain query predicates under RLS.
System changes
- Creates a partial HNSW cosine index concurrently; exact queries and the tenant/active B-tree index remain available.
Syntax explained
m = 16- Default maximum HNSW graph connections per layer; higher values can improve recall at greater memory/build cost.
ef_construction = 64- Default build candidate-list size; higher values can improve index quality while slowing build and insert.
sql/005_hnsw.sqlCREATE INDEX CONCURRENTLY chunks_embedding_hnsw_cosine_idx
ON semantic.chunks USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64)
WHERE active;psql --set=ON_ERROR_STOP=1 --dbname=semantic --file=sql/005_hnsw.sql && psql --dbname=semantic -c "SELECT indexname, indexdef FROM pg_indexes WHERE schemaname='semantic' AND indexname='chunks_embedding_hnsw_cosine_idx';"CREATE INDEX
indexname | indexdef
-----------------------------------------+------------------------------------------------------------
chunks_embedding_hnsw_cosine_idx | CREATE INDEX chunks_embedding_hnsw_cosine_idx ON semantic.chunks USING hnsw (embedding vector_cosine_ops) WITH (m='16', ef_construction='64') WHERE activeCheckpoint: Checkpoint: Establish exact-search ground truth, then create HNSW concurrently
Continue whenThe HNSW index is valid, partial on active rows, uses cosine operator class and documented initial parameters, while exact ground-truth results remain stored for comparison.
Stop whenDistance/operator class does not match the embedding contract, build exceeds rehearsed memory/disk/WAL, index is invalid, or exact ground truth is unavailable.
If this step fails
Index creation consumes memory or takes much longer than planned.
Likely causeSee the matching full troubleshooting entry.
Stop promotion and reproduce safely with synthetic tenant data.
ResolutionApply the documented resolution and repeat all release gates.
Security notes
- Use synthetic data, parameterized SQL, trusted tenant context, and source-equivalent embedding controls.
Alternatives
- Rehearse with an immutable synthetic corpus in a disposable restored database.
Stop conditions
- Stop on unknown versions/dimensions, RLS bypass, tenant leakage, or unverified recovery.
decision
Evaluate IVFFlat and enable iterative scans for filtered ANN
Use HNSW by default when its speed/recall trade-off and memory fit. Evaluate IVFFlat when faster builds and lower memory matter, but create it only after representative data exists because it trains centroids. Start lists near `rows/1000` up to one million rows or `sqrt(rows)` above that, and probes near `sqrt(lists)`, then measure. Under tenant/ACL filters, approximate indexes apply filtering to visited candidates; pgvector 0.8.x iterative scans can visit more candidates until enough authorized rows are found.
Why this step matters
ANN filtering can return too few rows when only a fraction of visited candidates satisfy tenant/ACL predicates. Strict iterative scanning improves filtered recall while retaining ordered results, but must be bounded and evaluated.
What to understand
Use `SET LOCAL` inside the same transaction as tenant context and query; connection-pool sessions must not retain search or identity settings.
Increase `ef_search` only with recall and latency evidence. If increasing scan tuples no longer helps, inspect `hnsw.scan_mem_multiplier`, dead tuples and filter selectivity.
For a small number of large, stable tenants, partitions or partial indexes may improve filtered search; hundreds of per-tenant indexes create operational complexity.
System changes
- Defines transaction-local tenant, ACL and HNSW iterative-scan settings; it does not change global PostgreSQL defaults.
Syntax explained
hnsw.ef_search = 100- Visits a larger candidate set than default 40, generally improving recall at higher latency.
strict_order- Iteratively scans more candidates while preserving exact distance ordering.
sql/006_search_settings.sqlBEGIN;
SELECT set_config('app.tenant_id', '11111111-1111-1111-1111-111111111111', true);
SELECT set_config('app.acl_groups', 'network-admin', true);
SET LOCAL hnsw.ef_search = 100;
SET LOCAL hnsw.iterative_scan = strict_order;
SELECT source_id, chunk_id, 1 - (embedding <=> :'query_vector'::vector) AS cosine_similarity
FROM semantic.chunks
WHERE active
ORDER BY embedding <=> :'query_vector'::vector
LIMIT 10;
COMMIT;QUERY_VECTOR=$(python3 -c "print('[' + ','.join(['1'] + ['0'] * 1535) + ']')") && psql --set=ON_ERROR_STOP=1 --dbname=semantic --username=semantic_app --variable=query_vector="$QUERY_VECTOR" --file=sql/006_search_settings.sqlBEGIN
set_config
--------------------------------------
11111111-1111-1111-1111-111111111111
(1 row)
set_config
---------------
network-admin
(1 row)
SET
SET
source_id | chunk_id | cosine_similarity
------------+----------+-------------------
dns-runbook| dns-001 | 0.913842
(1 row)
COMMITCheckpoint: Checkpoint: Evaluate IVFFlat and enable iterative scans for filtered ANN
Continue whenAuthorized query returns up to ten ordered results, tenant/ACL settings are transaction-local, iterative scan improves filtered recall within the p95 latency budget and no unauthorized row appears.
Stop whenQuery vector dimension is wrong, settings leak across requests, result count/recall is below threshold, or IVFFlat is built before representative data.
If this step fails
HNSW returns fewer authorized rows than the requested LIMIT under tenant filtering.
Likely causeSee the matching full troubleshooting entry.
Stop promotion and reproduce safely with synthetic tenant data.
ResolutionApply the documented resolution and repeat all release gates.
Security notes
- Use synthetic data, parameterized SQL, trusted tenant context, and source-equivalent embedding controls.
Alternatives
- Rehearse with an immutable synthetic corpus in a disposable restored database.
Stop conditions
- Stop on unknown versions/dimensions, RLS bypass, tenant leakage, or unverified recovery.
verification
Measure EXPLAIN plans, exact recall, latency, and tenant slices
Use a frozen query set with known relevant chunks and realistic tenant/ACL selectivity. Capture `EXPLAIN (ANALYZE, BUFFERS, SETTINGS, WAL)` for canonical parameterized query shapes. For each query, run exact top-k inside a transaction with index scans disabled, then run HNSW/IVFFlat settings and calculate overlap recall@k. Measure warm and cold latency, rows removed by filter, returned count, buffer hits/reads and planning estimates. Gate on worst tenant and selective-filter slices, not only the average.
Why this step matters
An index being used does not prove that its answers are good, and perfect recall does not prove latency is acceptable. Exact ground truth plus real plans quantify the trade-off and reveal filter behavior.
What to understand
Do not force `enable_seqscan=off` in production. Use it only inside the exact-evaluation transaction to generate ground truth.
Parameterize the query vector in application code. The abbreviated EXPLAIN display is normal; never log the complete sensitive vector.
Track empty/underfilled result sets separately from overlap recall. Under restrictive filters, ANN may return fewer than k even when the top returned rows are relevant.
System changes
- Reads query plans and runs bounded evaluation transactions; no global planner or HNSW setting changes occur.
Syntax explained
ANALYZE- Executes the query to report actual timing and rows; use only safe read-only fixtures.
BUFFERS- Shows shared/local/temp buffer activity used to distinguish cache and I/O behavior.
QUERY_VECTOR=$(python3 -c "print('[' + ','.join(['1'] + ['0'] * 1535) + ']')") && psql --dbname=semantic --username=semantic_app --variable=query_vector="$QUERY_VECTOR" -c "BEGIN; SELECT set_config('app.tenant_id','11111111-1111-1111-1111-111111111111',true); SELECT set_config('app.acl_groups','network-admin',true); SET LOCAL hnsw.ef_search=100; SET LOCAL hnsw.iterative_scan=strict_order; EXPLAIN (ANALYZE, BUFFERS, SETTINGS) SELECT source_id,chunk_id FROM semantic.chunks WHERE active ORDER BY embedding <=> :'query_vector'::vector LIMIT 10; ROLLBACK;"Limit (cost=78.21..82.46 rows=10 width=72) (actual time=3.812..4.266 rows=10 loops=1)
-> Index Scan using chunks_embedding_hnsw_cosine_idx on chunks
Order By: (embedding <=> '[...]'::vector)
Filter: ((tenant_id = '11111111-1111-1111-1111-111111111111'::uuid) AND active)
Buffers: shared hit=124
Settings: hnsw.ef_search = '100', hnsw.iterative_scan = 'strict_order'
Execution Time: 4.341 msCheckpoint: Checkpoint: Measure EXPLAIN plans, exact recall, latency, and tenant slices
Continue whenCanonical query plan is understood, exact and approximate results are reproducible, worst-slice recall meets target, p95 latency stays within budget and unauthorized results equal zero.
Stop whenEvaluation lacks exact ground truth, plans are captured under privileged bypass, full private vectors enter logs, or aggregate results hide failing tenant/filter slices.
If this step fails
Recall falls after switching from exact search to HNSW or IVFFlat.
Likely causeSee the matching full troubleshooting entry.
Stop promotion and reproduce safely with synthetic tenant data.
ResolutionApply the documented resolution and repeat all release gates.
Security notes
- Use synthetic data, parameterized SQL, trusted tenant context, and source-equivalent embedding controls.
Alternatives
- Rehearse with an immutable synthetic corpus in a disposable restored database.
Stop conditions
- Stop on unknown versions/dimensions, RLS bypass, tenant leakage, or unverified recovery.
config
Monitor indexes and perform vacuum, reindex, and extension upgrades safely
Enable standard PostgreSQL statistics appropriate to policy and monitor query latency, calls, rows, buffer reads, table/index size, live/dead tuples, autovacuum progress, replication lag and index-build progress. Keep raw vectors and source excerpts out of query logs. Tune autovacuum from measured churn. pgvector notes that HNSW vacuum can be accelerated by reindexing first; use `REINDEX INDEX CONCURRENTLY`, then `VACUUM (ANALYZE)` under a rehearsed change window. Rehearse extension upgrades and recall tests on a restored copy.
Why this step matters
Vector indexes can be larger than their table, and update/delete churn affects both result count and performance. Observable maintenance prevents blind parameter changes or unnecessary reindexing.
What to understand
Use `pg_stat_progress_create_index` during builds and `pg_stat_activity`/locks during maintenance. Do not cancel an operation without understanding cleanup state.
Keep `log_statement` and parameter capture compatible with data policy; complete vectors and excerpts are sensitive and high volume.
Reindex only with disk, WAL, replica and time budgets. `CONCURRENTLY` reduces write blocking but uses extra resources and has restrictions.
System changes
- Adds a metadata-only maintenance query. Reindex, vacuum or extension update remain separate approved operations after rehearsal.
Syntax explained
n_dead_tup- Estimated dead rows indicating churn and possible vacuum pressure, interpreted with workload and autovacuum history.
REINDEX ... CONCURRENTLY- Builds a replacement index while reducing write blocking, at additional time, storage and WAL cost.
sql/007_maintenance.sqlSELECT pg_size_pretty(pg_relation_size('semantic.chunks')) AS table_size,
pg_size_pretty(pg_relation_size('semantic.chunks_embedding_hnsw_cosine_idx')) AS hnsw_size;
SELECT n_live_tup, n_dead_tup, last_autovacuum, last_autoanalyze
FROM pg_stat_user_tables WHERE schemaname='semantic' AND relname='chunks';
SELECT extversion FROM pg_extension WHERE extname='vector';psql --set=ON_ERROR_STOP=1 --dbname=semantic --file=sql/007_maintenance.sql table_size | hnsw_size
------------+-----------
812 MB | 1.6 GB
(1 row)
n_live_tup | n_dead_tup | last_autovacuum | last_autoanalyze
------------+------------+-------------------------------+-------------------------------
1000000 | 12450 | 2026-07-28 02:14:07+00 | 2026-07-28 02:17:41+00
extversion
------------
0.8.5Checkpoint: Checkpoint: Monitor indexes and perform vacuum, reindex, and extension upgrades safely
Continue whenOperators know table/index size, churn, autovacuum, extension and replica state; maintenance thresholds and recall regression gates are documented and no content enters telemetry.
Stop whenDisk/WAL/replica capacity is insufficient, a concurrent index is invalid, query logs expose vectors/content, or extension upgrade lacks restored rehearsal and rollback.
If this step fails
Embedding/index storage grows faster than source content.
Likely causeSee the matching full troubleshooting entry.
Stop promotion and reproduce safely with synthetic tenant data.
ResolutionApply the documented resolution and repeat all release gates.
Security notes
- Use synthetic data, parameterized SQL, trusted tenant context, and source-equivalent embedding controls.
Alternatives
- Rehearse with an immutable synthetic corpus in a disposable restored database.
Stop conditions
- Stop on unknown versions/dimensions, RLS bypass, tenant leakage, or unverified recovery.
verification
Prove pg_dump restore, PITR, index recovery, and release rollback
Use physical backups plus WAL archiving for point-in-time recovery according to RPO/RTO and a logical custom-format `pg_dump` for portable schema/data inspection. Back up role/grant definitions through the approved cluster process, source/embedding/release manifests outside the database, and encryption keys independently. Restore PostgreSQL 18 with pgvector 0.8.5 into isolation, run `pg_restore --list`, restore, verify extension/RLS/indexes, execute two-tenant and deletion probes, compare exact recall and plans, then test application release rollback against the matching database point.
Why this step matters
A backup command does not prove recovery. Vector extension availability, RLS policies, owners, ANN indexes and external embedding/source revisions must align before a restored database can serve safely.
What to understand
Restore custom archives with reviewed ownership and role mapping. PostgreSQL documentation notes special RLS behavior during dump/restore; use a controlled administrative identity and verify policies afterward.
Physical PITR recovers a consistent database point, but external source manifests, embedding model/configuration, caches and serving alias require coordinated restoration.
Indexes are restorable/rebuildable artifacts, not the only copy of source truth. Keep stable chunks, vectors and manifests sufficient to rebuild while meeting RTO.
System changes
- Creates a logical custom archive and recovery evidence. An actual PITR or production rollback changes database state and must follow the reviewed runbook.
Syntax explained
custom format- Compressed pg_dump archive that supports `pg_restore --list` and selective/parallel restore where appropriate.
PITR- Physical base backup plus WAL used to recover the complete PostgreSQL cluster to a target time.
pg_dump --format=custom --file=backups/semantic.dump --dbname=semantic && pg_restore --list backups/semantic.dump | sed -n '1,12p'; ; Archive created at 2026-07-28 12:00:00 UTC ; dbname: semantic ; TOC Entries: 42 ; Compression: gzip ; 2; 3079 0 EXTENSION - vector 220; 1259 16420 TABLE semantic chunks semantic_owner 221; 1259 16430 INDEX semantic chunks_embedding_hnsw_cosine_idx semantic_owner 222; 3256 16440 POLICY semantic chunks chunks_tenant_acl semantic_owner
Checkpoint: Checkpoint: Prove pg_dump restore, PITR, index recovery, and release rollback
Continue whenLogical and physical restore complete within objectives; vector 0.8.5, forced RLS, policies, owners, indexes and manifests match; tenant/deletion/recall/plan tests pass.
Stop whenArchive is the only backup, WAL/key recovery is untested, target lacks compatible vector extension, restored runtime bypasses RLS, deleted content returns, or application and embedding revisions do not match.
If this step fails
A pg_dump archive restores but vector queries or RLS policies are missing.
Likely causeSee the matching full troubleshooting entry.
Stop promotion and reproduce safely with synthetic tenant data.
ResolutionApply the documented resolution and repeat all release gates.
Security notes
- Use synthetic data, parameterized SQL, trusted tenant context, and source-equivalent embedding controls.
Alternatives
- For very large indexes, restore base data and rebuild ANN indexes from stable vectors when measured RTO and WAL/storage costs are better than restoring index pages.
Stop conditions
- Stop a production restore when the target PostgreSQL/pgvector versions, encryption keys, role mapping, source manifest or mandatory deletion list cannot be verified.
- Stop rollback when it would reintroduce cross-tenant access, deleted content, an incompatible embedding dimension or a query/index operator mismatch.
Finish line
Verification checklist
psql --dbname=semantic -c "SELECT current_setting('server_version'), extversion FROM pg_extension WHERE extname='vector';" -c "SELECT rolname, rolsuper, rolbypassrls FROM pg_roles WHERE rolname='semantic_app';"PostgreSQL reports 18.x, vector 0.8.5 and runtime remains non-superuser/NOBYPASSRLS with the expected 1536-dimensional schema.psql --dbname=semantic --username=semantic_app --file=sql/verify_two_tenants.sqlMissing/wrong tenant returns zero, valid tenant sees only authorized active chunks, alternating tenants on one connection never cross and writes outside policy fail.psql --dbname=semantic --username=semantic_app --file=sql/verify_exact_ann.sqlHNSW/IVFFlat canonical plans are understood, worst-slice recall@10 and p95 latency meet predefined thresholds, underfilled results are tracked and unauthorized results equal zero.pg_restore --list backups/semantic.dump | grep -E 'EXTENSION|TABLE|INDEX|POLICY'Archive contains vector extension dependency, chunks, indexes and RLS policy; isolated logical/PITR restore plus revocation-forward application rollback pass.Recovery guidance
Common problems and safe checks
PostgreSQL reports `expected 1536 dimensions, not 3072` during insert or query.
Likely causeThe embedding model or dimensions changed while the table and serving query still use `vector(1536)`, or vectors from two revisions were mixed.
Read `atttypmod` and the release manifest for the embedding column.
ResolutionStop the mixed ingest, restore the matching query embedder, or build a separate table/index for the new dimension and promote it only after a complete recall and isolation evaluation.
The extension cannot be created or shows a version older than 0.8.5.
Likely causeThe server package is missing, `shared_preload_libraries` assumptions are wrong, the extension control files are from another package, or the database was not upgraded.
Run `SELECT * FROM pg_available_extension_versions WHERE name='vector';`.
ResolutionInstall the official pgvector package/build for PostgreSQL 18, restart only if packaging requires it, rehearse `ALTER EXTENSION vector UPDATE` on a restored copy, and verify 0.8.5 before schema creation.
A runtime query returns chunks from another tenant.
Likely causeRLS is disabled, the runtime role owns the table or has `BYPASSRLS`, tenant context is missing/stale, a security-definer function bypasses policy, or a cache crosses tenants.
Remove the endpoint from service and preserve request and role evidence.
ResolutionEnable and force RLS, use a non-owner role without `BYPASSRLS`, set tenant context transaction-locally from trusted identity, fix cache scope, rotate exposed credentials if necessary, and require zero leaks before reopening.
A pooled connection returns no rows or retains the previous tenant.
Likely causeTenant context was set with session scope, a transaction was not started, `SET LOCAL` expired before the query, or connection reset behavior is inconsistent.
Inspect `current_setting('app.tenant_id', true)` inside the failing transaction.
ResolutionRequire one transaction per request, set tenant ID locally with a parameterized statement, clear/discard connections after exceptional exits, and add alternating-tenant regression tests.
The planner performs a sequential scan after the HNSW index was created.
Likely causeThe query orders by an expression incompatible with the indexed operator, casts the column, uses a different distance operator/operator class, the table is small, statistics are stale, or the LIMIT/shape is unsuitable.
Run `EXPLAIN (ANALYZE, BUFFERS, SETTINGS)` with a synthetic vector.
ResolutionWrite the canonical nearest-neighbor query with matching operator class, refresh statistics, evaluate realistic data volume, and accept a sequential scan when the cost model correctly finds it cheaper.
HNSW returns fewer authorized rows than the requested LIMIT under tenant filtering.
Likely causeApproximate candidates are visited before the filter, `hnsw.ef_search` is too low, dead tuples reduce candidates, or iterative scanning is disabled.
Compare the result with an exact search in a transaction using `SET LOCAL enable_indexscan=off`.
ResolutionEnable strict iterative scans, increase `ef_search` within latency/memory budgets, vacuum as appropriate, and consider tenant partitioning or exact search for highly selective filters.
Recall falls after switching from exact search to HNSW or IVFFlat.
Likely causeApproximate search is expected to trade recall for speed, parameters are too small, IVFFlat was trained with insufficient data, or the evaluation query distribution changed.
Run the same frozen query set with index scans disabled to obtain exact top-k.
ResolutionTune on held-out data, rebuild an under-trained IVFFlat index after representative loading, increase search effort within latency budget, or keep exact search where correctness matters more than speed.
Ingestion creates duplicate chunks or an update leaves old embeddings active.
Likely causeUpserts lack a stable unique key, source revision is not part of identity, retries are not idempotent, or deletion/supersession does not tombstone prior rows.
Group by tenant, source ID, revision and chunk ID to find duplicates.
ResolutionAdd a stable unique constraint, use parameterized `INSERT ... ON CONFLICT DO UPDATE`, mark superseded rows inactive in the same transaction, and reconcile affected sources before rebuilding the candidate index.
Embedding/index storage grows faster than source content.
Likely causeDeleted rows are not vacuumed, duplicate ingestion occurs, HNSW retains bloat, vectors use unnecessarily large dimensions, or old revisions remain indefinitely.
Compare live/dead tuple counts, relation/index sizes and source/chunk counts.
ResolutionFix duplicate and retention policy first, delete expired rows, vacuum, use `REINDEX INDEX CONCURRENTLY` when justified, and evaluate smaller vectors or partition lifecycle only through measured recall.
Index creation consumes memory or takes much longer than planned.
Likely causeHNSW graph exceeds `maintenance_work_mem`, too many parallel workers compete for RAM, writes continue heavily, or index parameters are oversized.
Read PostgreSQL logs for the HNSW memory notice.
ResolutionCancel only through the controlled change procedure, reduce concurrency/parameters or schedule more memory, load bulk data before index creation, and repeat on a production-sized restored copy.
A deleted or access-revoked chunk still appears in search.
Likely causeThe row delete did not commit, a replica lags, application cache is stale, a previous index/table remains serving, or soft-delete filtering is missing.
Query the stable chunk ID on primary and replicas under an administrative audit role.
ResolutionApply an immediate deny-list at the application boundary, complete transactional delete/tombstone propagation, invalidate tenant-scoped caches, verify all replicas and record deletion latency.
A pg_dump archive restores but vector queries or RLS policies are missing.
Likely causeThe extension is unavailable in the target, pre-data/schema/post-data sections were restored selectively, owners/roles differ, policies were omitted, or runtime grants were not recreated.
Inspect the archive with `pg_restore --list`.
ResolutionProvision compatible PostgreSQL 18 and pgvector 0.8.5 first, restore all required sections with reviewed ownership, recreate external roles safely, verify RLS/indexes and do not promote the restore until the full acceptance suite passes.
Reference
Frequently asked questions
Should HNSW or IVFFlat be the default?
Start with exact search as truth. HNSW often provides the better speed/recall trade-off but uses more memory and builds slower. IVFFlat can fit lower-memory/faster-build cases after representative data exists. Measure both on your filtered corpus.
Can the application set tenant ID once when opening a pooled connection?
No. Begin a transaction and set trusted tenant/ACL locally for every request. Session-global context can leak into the next borrower of the same connection.
Why can filtered HNSW return fewer than LIMIT?
ANN visits candidates before filters remove rows. Increase evaluated `ef_search`, enable iterative scans, maintain dead tuples and consider exact search or partitioning for highly selective filters.
Should we use `enable_seqscan=off` to force HNSW?
Not in production. Use it transaction-locally only to generate exact evaluation results. Fix query/operator shape and statistics, and allow PostgreSQL to choose a sequential scan when cheaper.
Does pg_dump replace physical backups and WAL?
No. Custom-format pg_dump is useful for portable logical recovery and inspection; physical backups plus WAL provide cluster PITR. Test both with compatible extension and external manifests.
What must a rollback preserve?
Restore matching PostgreSQL/extension, schema, embedding/query, source and application revisions while carrying forward every newer deletion and ACL revocation.
Recovery
Rollback
Restore the previous coupled PostgreSQL/schema/pgvector/embedding/query/application release and, when required, the verified pre-change database point. Carry every newer deletion and ACL revocation forward, verify forced RLS and role attributes, then compare exact and ANN results before reopening.
- Stop writes or candidate promotion, drain the changed application/database path and preserve plans, migration logs, WAL position, release manifest and privacy-safe request evidence.
- Determine whether the prior application is compatible with current schema and extension. If not, restore the verified physical/logical baseline in isolation before production cutover.
- Provision PostgreSQL 18 and pgvector 0.8.5, restore roles/grants through the reviewed process, database objects/data and matching external source/embedding/query manifests.
- Apply the current mandatory deletion and ACL revocation list to the restored point and invalidate tenant-scoped application caches.
- Verify runtime is non-owner/NOBYPASSRLS, RLS and FORCE are enabled, policies and operator-class indexes are valid, and embedding dimension/model match.
- Run alternating two-tenant probes, idempotent upsert/delete, exact top-k, ANN recall, EXPLAIN plans and citation/source-count reconciliation.
- Resume with a bounded canary, retain the failed release for analysis and fix under a new immutable revision rather than mutating the rejected manifest.
Evidence