Build a container-image pipeline with BuildKit, SBOMs, signing, and provenance
Release OCI images from protected source through a disposable digest-pinned BuildKit builder, native SPDX and SLSA provenance, optional CycloneDX inventory, Cosign keyless or KMS signing, exact identity verification, digest-only admission, and evidence-backed recovery.
Make every deployed container image traceable to reviewed source and immutable build inputs, inventoried in SPDX/CycloneDX, accompanied by accurate provenance, signed by an approved short-lived identity, independently verified, and deployed or rolled back only by digest.
- Docker Buildx current supported stable
- BuildKit current supported stable
- Cosign current supported stable
- OCI registry attestation/referrer compatible
- Linux CI runner isolated disposable worker
- Protected source Reviewed full commit, immutable CI dependencies, required approvals, lockfiles, clean context, and untrusted pull requests excluded from release authority.
- Isolated build platform Disposable dedicated Linux worker, reviewed Docker/Buildx/BuildKit versions, trust-scoped cache, narrow network, monitoring, and no production socket/credential access.
- OCI registry Staging repository supporting BuildKit attestations, immutable retention, referrer discovery, audit, short-lived scoped credentials, and tested promotion without digest change.
- Signing identity Protected keyless OIDC workflow with exact identity/issuer or KMS/HSM immutable key version with workload identity; no private key stored in CI.
- Policy and incident operations SPDX/CycloneDX/provenance validators, vulnerability/license/secret policy, admission enforcement, negative tests, release ledger, kill switch, quarantine, and verified rollback digest.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- An image-release control plane rather than a convenient docker build command: protected source enters a disposable, isolated BuildKit builder whose base, builder, scanner, and CI dependencies are immutable references.
- A dual-format inventory and provenance path: BuildKit attaches native SPDX SBOM and SLSA provenance, while a second pinned scanner can produce CycloneDX from the same pushed digest when consumers require it.
- A signing and deployment gate that captures the exporter digest, signs repository@digest with Cosign keyless OIDC or approved KMS workload identity, verifies exact identity/issuer and evidence, and deploys only the accepted digest.
- Platform, release, security, and DevOps operators can trace every deployed digest to reviewed source, locked inputs, builder identity, inventories, policy decisions, signer, and approval.
- Untrusted pull requests, mutable tags, leaked secrets, missing or ambiguous attestations, broad signer identities, and unsigned or wrong-digest deployments fail closed.
- Operators can freeze signing, quarantine builders/cache/artifacts, rotate authority, scope affected releases, and recover using an independently verified incident-safe rollback digest.
Architecture
How the parts fit together
Protected source and lockfiles enter an isolated disposable CI runner and digest-pinned BuildKit builder. The builder pushes to staging with OCI attestations, then a separate verifier extracts the immutable digest and checks native SPDX/provenance plus optional CycloneDX. A protected signing identity signs only accepted content. Independent verification and admission precede digest-only deployment. Logs, transparency evidence, release ledger, and policy results remain outside disposable build state.
- A protected workflow selects a reviewed full commit and immutable build inputs.
- A disposable pinned BuildKit builder pushes the multi-platform image with native SPDX and SLSA provenance to staging.
- The verifier extracts exporter digest, inspects subjects/referrers, optionally creates CycloneDX from the same digest, and evaluates policy plus negative fixtures.
- A separate protected job signs the accepted digest with keyless OIDC or KMS workload identity.
- An independent verifier constrains signer identity/issuer or key version and rechecks evidence.
- Registry promotion preserves the digest, admission accepts only that digest, runtime identity is verified, and evidence is retained outside the builder.
Assumptions
- The source repository uses protected branches/tags, required review, pinned CI dependencies, a controlled release environment, and a commit SHA selected before the build.
- The runner is single-purpose and disposable or rebuilt between trust domains; it does not share a Docker daemon, writable cache, credentials, or workspace with untrusted pull requests.
- The OCI registry supports attestations/referrers used by current BuildKit and retains manifests, signatures, SBOMs, provenance, and audit events according to release policy.
- Every base image and tool image is promoted through review and referenced by digest. Language dependencies use integrity-checked lockfiles and package-manager verification.
- The protected signing job receives short-lived OIDC or narrow KMS workload identity only after review gates. No private signing key is stored in the repository, CI variable, cache, artifact, or runner image.
- Rollback is an immutable, previously verified digest; tags are discovery labels only and never authorization or deployment identity.
Key concepts
- OCI image digest
- A content-addressed sha256 identity returned by the registry exporter. Sign, verify, promote, deploy, and roll back repository@digest—not a mutable tag.
- SBOM
- An inventory, not a vulnerability verdict. BuildKit natively emits SPDX; this tutorial optionally produces CycloneDX from the same pushed digest for downstream systems and compares both with the runtime filesystem.
- Build provenance
- Structured evidence describing a build subject, builder, inputs/materials, parameters, and source context. BuildKit can emit SLSA provenance, but a particular SLSA Build level also depends on platform guarantees and must be assessed separately.
- Attestation
- A statement attached to an OCI artifact, such as SPDX SBOM or SLSA provenance. Presence is insufficient: the subject and expected predicate fields must be validated and connected to a trusted signing/promotion decision.
- Cosign keyless signing
- Cosign uses a short-lived OIDC identity certificate, Fulcio, and Rekor instead of a long-lived private key in CI. Verification must constrain both expected certificate identity and OIDC issuer.
- SLSA level
- A statement about the build process and platform guarantees, not a command-line switch. Provenance existence supports SLSA Build L1 evidence; higher levels require hosted, signed, hardened platform properties defined by the SLSA specification.
Before you copy
Values used in this guide
{{registryHost}}Approved staging/production OCI registry hostname without scheme.
Example: registry.example.com{{imageRepository}}Approved repository path below the registry host.
Example: platform/payments-api{{releaseTag}}Non-authoritative release label; deployment still uses digest.
Example: 2026.07.28-1{{sourceCommit}}Full Git commit SHA selected by the protected release workflow.
Example: 6d3f7a138f0c9ae294f9616e14ac2ba09be48321{{baseImageRepository}}Fully qualified approved base-image repository without digest.
Example: docker.io/library/alpine{{baseImageDigest}}Reviewed sha256 digest, including sha256 prefix.
Example: sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa{{buildkitImage}}Approved BuildKit image repository.
Example: docker.io/moby/buildkit{{buildkitDigest}}Reviewed BuildKit image sha256 digest.
Example: sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb{{cycloneDxImage}}Approved CycloneDX-capable scanner image repository.
Example: ghcr.io/anchore/syft{{cycloneDxDigest}}Reviewed scanner image sha256 digest.
Example: sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc{{cosignIdentity}}Exact or narrowly reviewed workflow identity expected from keyless signing.
Example: https://github.com/example/payments/.github/workflows/release.yml@refs/heads/main{{oidcIssuer}}Exact identity-provider issuer trusted by verification.
Example: https://token.actions.githubusercontent.com{{metadataPath}}Workspace-relative JSON path used for exporter metadata.
Example: out/build-metadata.jsonSecurity and production boundaries
- Keep untrusted pull-request testing, image building, signing, verification, and production deployment as separate trust decisions. An untrusted job must never reach builder sockets, registry push credentials, OIDC signing permission, or production admission bypass.
- Treat Docker daemon access and most container-builder control sockets as root-equivalent for the runner host. Isolate the builder on a disposable dedicated worker and never point it at production Docker/Kubernetes.
- Pin base images, builder/scanner images, actions, and deployment artifacts by digest or full reviewed commit. Tags may move legitimately or maliciously.
- Never pass secrets using Dockerfile ARG/ENV. BuildKit mode=max provenance records build arguments, and image history/layers can expose them. Use secret/SSH mounts and ensure commands do not copy or echo secret material.
- Do not store a Cosign private key in CI. Prefer keyless OIDC with exact identity/issuer verification or a KMS/HSM key with short-lived workload identity, immutable key version, and audit logs.
- The build job should not automatically receive signing authority. Place signing behind protected environment/release approval after SBOM, provenance, vulnerability, policy, and negative tests pass.
Stop before continuing if
- Stop if any base, builder, scanner, action, or deployed image reference is not immutable and reviewed.
- Stop if a secret is present in build context, ARG/ENV, logs, metadata, provenance, image history, layers, cache, or exported artifact.
- Stop if the registry exporter does not return an exact digest or fails to retain the expected SBOM/provenance attestations.
- Stop if SPDX/CycloneDX inventories omit critical runtime components without an accepted documented exception.
- Stop if keyless/KMS identity is unavailable, unexpectedly broad, or verification cannot constrain the expected signer and issuer/key version.
- Stop if provenance subject, source commit, materials, builder, or build parameters do not match the release record.
- Stop if deployment accepts tags, an unsigned digest, a signature without required attestations/policy, or an emergency bypass without an explicit incident process.
instruction
Freeze the release intent and trust boundaries
Before building, select a reviewed full commit, document the target platforms and registry, assign separate build/sign/deploy roles, and prove that untrusted pull requests cannot reach this pipeline. Choose the SLSA evidence goal explicitly; do not label the pipeline L2/L3 merely because it emits a provenance predicate.
Why this step matters
A secure image pipeline begins with an immutable reviewed subject and explicit principals. If the commit, builder, signer, or deployment identity is decided after output exists, an attacker can substitute the release while every individual tool still reports success.
What to understand
Resolve branch/tag to a full commit in the protected workflow and record repository plus commit before checkout. Re-check HEAD after checkout and before build.
Map identities for source approval, builder execution, registry push, signing OIDC/KMS, policy verification, and deployment. Avoid one broad CI credential spanning every role.
Define target platforms, build context, Dockerfile path, base-image locks, dependency locks, output repository, retention, and rollback digest.
Assess SLSA Build requirements from the official specification. BuildKit provenance helps create evidence but does not by itself prove hosted signing, isolation, or hardened build-platform properties.
System changes
- Creates a reviewed release record and trust-boundary decision; it does not build, push, sign, or deploy.
Syntax explained
source- Full reviewed Git commit; branches and tags are mutable selectors.
registry/repository- Approved artifact namespace used for push, signing, verification, and admission.
build/sign- Separate trust decisions with different short-lived permissions.
digest-only- Promotion, deploy, and rollback use registry content identity.
Values stay on this page and are never sent or saved.
printf '%s\n' 'source={{sourceCommit}}' 'registry={{registryHost}}/{{imageRepository}}' 'tag={{releaseTag}}' 'build=isolated' 'sign=protected-oidc' 'deploy=digest-only'release intent: approved
source commit: {{sourceCommit}}
target repository: {{registryHost}}/{{imageRepository}}
human tag: {{releaseTag}} (non-authoritative)
untrusted PR access to builder/signing: denied
build and signing roles: separated
deployment identity: repository@sha256
SLSA claim: evidence assessed separately; no level inferredCheckpoint: Verify this layer before continuing
Continue whenThe record contains one full commit, one repository, exact platforms, reviewed inputs/tool digests, role boundaries, evidence requirements, stop conditions, and rollback owner.
Stop whenStop if source is unreviewed/mutable, a fork can trigger protected jobs, one broad credential spans build/sign/deploy, or a claimed SLSA level lacks a documented requirement assessment.
If this step fails
A release later cannot explain which commit, builder, signer, or policy result produced the deployed digest.
Likely causeThe pipeline started from a branch/tag and inferred evidence or roles only after the build.
Compare checkout SHA with {{sourceCommit}}.Trace every credential and trigger from untrusted contribution through deployment.
ResolutionCancel the release, revoke unnecessary permissions, approve an immutable record, and restart from a clean isolated workspace.
Security notes
- Never make signing available to pull_request/fork jobs.
- Treat workflow/config changes as production code with required review.
Alternatives
- For mature platforms, encode this record as a signed release manifest checked by policy.
Stop conditions
- Stop if source is unreviewed/mutable, a fork can trigger protected jobs, one broad credential spans build/sign/deploy, or a claimed SLSA level lacks a documented requirement assessment.
config
Pin the Dockerfile inputs and exclude unwanted context
Use an immutable base-image digest, a reviewed Dockerfile frontend, deterministic dependency locks, and a minimal .dockerignore. The example is intentionally generic: adapt package installation to the application, but keep base and downloaded dependencies verifiable and ensure the runtime runs as nonroot.
Why this step matters
The image digest is a function of every build input. Pinning the base and dependencies prevents silent upstream movement, while a small context reduces secret leakage and makes SBOM/provenance easier to interpret.
What to understand
Record the base repository and sha256 as one reviewed reference. Verify it supports every target platform before release.
Pin language dependencies with ecosystem lockfiles and integrity hashes where supported. Disable implicit lockfile rewrites during CI.
Use multi-stage builds to exclude compilers and credentials, then independently verify copied runtime files appear in the SBOM.
Create .dockerignore entries for .git, local environment files, credentials, build output not required by COPY, editor caches, test secrets, and incident bundles. Test the effective context rather than trusting the file name.
Run the final process as a fixed nonroot UID/GID, avoid setuid binaries/capabilities where possible, and make runtime files read-only in the deployment policy.
System changes
- Defines immutable base/runtime behavior and changes files entering the image build context.
Syntax explained
# syntax- Selects Dockerfile frontend behavior; review/pin according to builder policy.
ARG BASE_IMAGE- Allows policy-controlled substitution while the supplied value remains repository@digest.
FROM ${BASE_IMAGE}- Consumes the reviewed content-addressed base.
USER 65532:65532- Makes nonroot the default process identity.
ENTRYPOINT JSON- Avoids implicit shell parsing for the runtime process.
DockerfileValues stay on this page and are never sent or saved.
# syntax=docker/dockerfile:1
ARG BASE_IMAGE={{baseImageRepository}}@{{baseImageDigest}}
FROM ${BASE_IMAGE} AS runtime
WORKDIR /app
COPY --chown=65532:65532 ./dist/ /app/
USER 65532:65532
ENTRYPOINT ["/app/service"]Dockerfile policy:
frontend: reviewed syntax
base: {{baseImageRepository}}@{{baseImageDigest}}
runtime user: 65532:65532
remote downloads in Dockerfile: none
mutable package inputs: none in example
secret-bearing ARG or ENV: noneCheckpoint: Verify this layer before continuing
Continue whenPolicy finds no mutable base/tool/dependency, no secret path enters context, final process is nonroot, and every runtime file has an owner and provenance path.
Stop whenStop if base/platform digest is wrong, dependency locks change during build, a remote download lacks integrity, a secret enters context, or runtime requires root without an approved exception.
If this step fails
A rebuild changes unexpectedly or the image contains a local secret/file absent from source review.
Likely causeA mutable input, nondeterministic generation, broad build context, or hidden cache input changed the output.
Compare provenance materials/parameters and Docker context inventories.Rebuild once without cache using identical tool/platform digests.
ResolutionNarrow and lock the input, rotate exposed secrets, regenerate the reviewed context, and rebuild as a new release digest.
Security notes
- Do not put credentials in ARG/ENV even if a later layer deletes them.
- Review base updates as dependency changes rather than silently following tags.
Alternatives
- Use policy-as-code to reject mutable FROM, root USER, ADD URL, unverified curl pipes, and forbidden context paths.
Stop conditions
- Stop if base/platform digest is wrong, dependency locks change during build, a remote download lacks integrity, a secret enters context, or runtime requires root without an approved exception.
command
Create a disposable digest-pinned BuildKit builder
Create a dedicated docker-container Buildx builder from the approved BuildKit image digest. Run this only on an isolated builder host; Docker control remains host-root equivalent even when the build output itself is nonroot. Bootstrap and inspect the builder before accepting work.
Why this step matters
Pinning and isolating the builder prevents an unnoticed BuildKit update or lower-trust workload from changing release behavior. A disposable builder also makes cache/workspace teardown a normal control instead of an exceptional cleanup.
What to understand
The host must be dedicated to protected builds, patched, monitored, capacity-limited, and separated from production daemons/networks. Destroy it or the builder state at the end of the trust window.
Resolve the BuildKit image digest for the host architecture and retain its approval/scanner record. A tag beside the digest is documentation only.
Do not enable network.host, security.insecure, privileged entitlements, host mounts, SSH agent forwarding, or broad registry credentials unless a separately reviewed build requires and contains them.
Constrain egress to source/dependency/registry/Sigstore endpoints appropriate for the build. Reproducible/offline builds may use a promoted internal mirror.
Use a fresh builder or trust-scoped cache for releases. Never import writable cache produced by untrusted pull-request jobs.
System changes
- Creates a BuildKit container, local builder metadata, cache state, and network activity on the dedicated runner.
Syntax explained
--driver docker-container- Runs a managed BuildKit daemon in a container instead of the default daemon-integrated builder.
--driver-opt image=...@digest- Selects an immutable reviewed BuildKit implementation.
inspect --bootstrap- Starts the builder and reports effective version/platforms before release work.
Values stay on this page and are never sent or saved.
docker buildx create --name ol-release-builder --driver docker-container --driver-opt image={{buildkitImage}}@{{buildkitDigest}} --use && docker buildx inspect --bootstrap ol-release-builderName: ol-release-builder
Driver: docker-container
Status: running
Buildkit: v0.x.y
Platforms: linux/amd64, linux/arm64
Driver image: {{buildkitImage}}@{{buildkitDigest}}
Insecure entitlements: none approvedCheckpoint: Verify this layer before continuing
Continue whenBuilder reports the approved digest/version, only intended platforms and endpoints, no insecure entitlements, and no state shared with lower-trust jobs.
Stop whenStop if Docker context is unexpected, builder image/tag is mutable, host is shared with untrusted/production work, platform is missing, or an insecure entitlement appears.
If this step fails
The builder starts from an unapproved image or silently reaches a production/internal service.
Likely causeThe CI host reused a default/shared Docker context, driver option was ignored, or network policy is too broad.
Run docker context show and docker buildx inspect.Compare container image digest, mounts, capabilities, and network with baseline.
ResolutionDestroy the builder/host, correct context/digest/network, and create a fresh instance before processing source.
Security notes
- Docker socket access is root-equivalent; runner isolation is mandatory.
- Do not reuse the builder after suspected hostile source or cache.
Alternatives
- Rootless BuildKit can reduce privilege but still requires kernel/network/cache isolation and feature testing.
Stop conditions
- Stop if Docker context is unexpected, builder image/tag is mutable, host is shared with untrusted/production work, platform is missing, or an insecure entitlement appears.
verification
Prove secrets cannot enter the build record
Run a fail-closed preflight over tracked files, Dockerfile instructions, build arguments, context manifest, and planned credentials. Build-time secrets must use BuildKit secret or SSH mounts and the consuming command must not persist or print them. The example demonstrates only a nonsecret fixture name.
Why this step matters
BuildKit protects mounted secret contents from normal image layers, but only when the secret is outside context/ARG/ENV and the Dockerfile command handles it safely. Provenance mode=max includes build arguments, making ARG especially dangerous for credentials.
What to understand
The heuristic grep is only one negative test; use an approved secret scanner with history scanning and repository-specific patterns. Treat every hit as sensitive until triaged.
Inventory the effective build context, including generated files and symlink targets. Ensure CI does not create credential files inside the context before build.
Authenticate the registry through the CI credential helper or docker login using protected stdin/credential store; do not embed registry auth in Dockerfile or image config.
For package access, pass a file descriptor/BuildKit --secret or --ssh mount only to the exact RUN step. Delete package-manager credential caches in that same step and prevent shell tracing.
After a canary build, inspect history, layers, SBOM, provenance parameters, metadata, logs, cache export, and filesystem for synthetic canary strings.
System changes
- Reads repository files and may stop the pipeline; it must not emit matched secret content to shared logs.
Syntax explained
git diff --exit-code- Ensures generated/unreviewed workspace changes are not silently included.
grep -Il- Reports file names only, avoiding accidental secret value output.
ARG|ENV negative- Blocks credential-shaped Dockerfile variables before mode=max provenance records them.
git diff --exit-code && git ls-files -z | xargs -0 -r grep -IlE '(PRIVATE KEY|password[[:space:]]*=|token[[:space:]]*=)' || true; grep -nE '^(ARG|ENV)[[:space:]].*(TOKEN|PASSWORD|SECRET|KEY)' Dockerfile && exit 1 || trueworking tree: clean tracked-secret heuristic: no findings secret-bearing ARG/ENV: none context allowlist: passed BuildKit secret mount IDs: reviewed names only secret values in command line/logs: none release may continue: yes
Checkpoint: Verify this layer before continuing
Continue whenWorkspace is clean, context is enumerated, approved scanner passes, secret mounts are narrow, synthetic canaries are absent from all outputs/evidence, and no secret value is logged.
Stop whenStop on any real credential/private key, unknown generated context path, secret-bearing ARG/ENV, shell tracing, synthetic canary leak, or scanner failure.
If this step fails
A canary token appears in image history, provenance, SBOM, cache, or logs.
Likely causeThe value entered context/ARG/ENV or the consuming build step copied/echoed/cached it.
Quarantine every artifact/log/cache containing the canary.Trace Dockerfile step and metadata field without reproducing the secret.
ResolutionTreat real equivalents as compromised, rotate them, correct secret mounts/command behavior, clear suspect cache, and rebuild without cache from a clean commit.
Security notes
- Never print OIDC tokens or docker config during diagnosis.
- A clean secret scan does not authorize broad credentials.
Alternatives
- Prefer vendored/promoted dependencies that remove build-time secret/network access where practical.
Stop conditions
- Stop on any real credential/private key, unknown generated context path, secret-bearing ARG/ENV, shell tracing, synthetic canary leak, or scanner failure.
command
Build a local evidence canary before registry publication
Exercise Dockerfile policy and native SPDX SBOM generation without creating a deployable tag. Use a local exporter for evidence inspection, then validate the generated SPDX JSON and runtime filesystem. This canary is separate from the final pushed build and cannot be signed/promoted.
Why this step matters
A nondeployable canary catches Dockerfile, scanner, platform, secret, and inventory problems before registry/signing authority is used. It also establishes what BuildKit natively emits: SPDX, not an assumed CycloneDX document.
What to understand
Run on a representative platform; multi-platform releases require per-platform evidence checks because packages and base layers can differ.
Inspect the local exporter layout produced by the current BuildKit version rather than hard-coding silently; fail if the expected SPDX file is absent.
Validate SPDX schema/required identifiers using an approved validator and compare a sample of runtime binaries/packages with the package list.
Exercise negative fixtures: intentionally forbidden mutable FROM, synthetic secret in context, missing lock integrity, root runtime, and unsupported platform must fail policy.
Delete the canary output after retaining sanitized policy results. It is not the final OCI image and must never be promoted by renaming files.
System changes
- Builds source in the isolated builder, reads dependencies, writes a local root filesystem and SPDX evidence under out/canary, and populates trust-scoped cache.
Syntax explained
SOURCE_DATE_EPOCH- Nonsecret reproducibility hint derived from reviewed commit time.
--sbom=true- Requests BuildKit's native SPDX SBOM generation.
--provenance=mode=max- Requests detailed build provenance; build arguments become sensitive metadata.
--output type=local- Exports files for canary inspection and intentionally does not create the release digest.
jq -e- Fails when required SPDX fields/package inventory are absent.
Values stay on this page and are never sent or saved.
rm -rf out/canary && mkdir -p out/canary && SOURCE_DATE_EPOCH="$(git show -s --format=%ct {{sourceCommit}})" docker buildx build --builder ol-release-builder --platform linux/amd64 --sbom=true --provenance=mode=max --output type=local,dest=out/canary --build-arg SOURCE_DATE_EPOCH="$SOURCE_DATE_EPOCH" . && test -s out/canary/sbom.spdx.json && jq -e '.spdxVersion and (.packages | length > 0)' out/canary/sbom.spdx.json >/dev/nullcanary exporter: local
source commit: {{sourceCommit}}
platform: linux/amd64
SPDX document: out/canary/sbom.spdx.json
SPDX schema/package check: passed
synthetic secret canary: absent
runtime package spot-check: passed
deployable registry digest: intentionally not createdCheckpoint: Verify this layer before continuing
Continue whenCanary builds on each target platform, SPDX validates and covers expected runtime components, secret/policy negatives fail, and no deployable artifact exists.
Stop whenStop on missing/invalid SBOM, unexpected package/binary, secret-canary leak, policy negative success, network/input drift, or platform mismatch.
If this step fails
The SPDX file is absent even though the build succeeded.
Likely causeBuilder/exporter version/layout differs, scanner failed, or final stage has no recognizable package database.
Inspect BuildKit progress and exporter files.Confirm supported Buildx/BuildKit versions and --sbom flag.
ResolutionCorrect builder/scanner/exporter and Dockerfile metadata; rerun canary. Do not waive the final SBOM requirement.
Security notes
- Treat exported root filesystem as source-derived untrusted data; never execute its entrypoint on the CI host.
- Sanitize evidence before broad distribution.
Alternatives
- Use a registry staging namespace for canaries only if local exporter cannot exercise required registry features; enforce retention/no-deploy policy.
Stop conditions
- Stop on missing/invalid SBOM, unexpected package/binary, secret-canary leak, policy negative success, network/input drift, or platform mismatch.
command
Build and push the immutable image with native attestations
Build the reviewed commit for the approved platforms, push to the staging repository, attach BuildKit SPDX and mode=max SLSA provenance attestations, and write structured exporter metadata. Registry authentication is supplied by the protected job outside this command and never printed.
Why this step matters
The registry exporter creates the immutable multi-platform digest that all later evidence must reference. Asking for both native attestation types in the same build links them to the exporter subject, while structured metadata avoids scraping console text.
What to understand
Authenticate only to the narrow staging repository with short-lived push permission. The builder does not need production deployment credentials.
Use --push. Docker documents that attestations are not reliably preserved by the default local image store/--load path; absence must fail the release.
Build each required platform under equivalent policy and inspect per-platform manifests. The index digest is the release identity, while platform manifest digests remain important evidence.
Mode=max provides richer parameters/materials but can expose build-argument values. The secret preflight is mandatory.
Retain metadata as a controlled CI artifact after scanning it for secrets. Validate registry referrers before signing.
System changes
- Runs multi-platform builds, pulls dependencies, writes cache/metadata, pushes image manifests/layers and native attestation manifests to staging registry.
Syntax explained
--platform- Builds explicit platform manifests under one OCI index.
--sbom=true- Attaches BuildKit-generated SPDX SBOM attestations.
--provenance=mode=max- Attaches detailed SLSA provenance predicates; not itself a SLSA level claim.
--metadata-file- Writes structured exporter data including the resulting image digest.
--push- Publishes manifests and attestations to the OCI registry; required for this evidence path.
Values stay on this page and are never sent or saved.
mkdir -p "$(dirname '{{metadataPath}}')" && docker buildx build --builder ol-release-builder --platform linux/amd64,linux/arm64 --tag {{registryHost}}/{{imageRepository}}:{{releaseTag}} --sbom=true --provenance=mode=max --metadata-file {{metadataPath}} --push .multi-platform build: passed
platforms: linux/amd64, linux/arm64
push: {{registryHost}}/{{imageRepository}}:{{releaseTag}}
manifest digest: sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd
SPDX attestation: attached
SLSA provenance attestation: attached
metadata: {{metadataPath}}
promotion status: staged; unsigned and not deployableCheckpoint: Verify this layer before continuing
Continue whenExactly the approved platforms are present, metadata contains one valid index digest, registry retains SPDX/provenance for every subject, and the artifact remains staged/unavailable to deployers.
Stop whenStop on build/input drift, mutable pull, unexpected platform, missing digest/attestation, registry rewrite, secret in metadata/provenance, or any policy failure.
If this step fails
The image is visible by tag but the registry has no attestations.
Likely causeThe build used --load, registry lacks required OCI support, or exporter flags/version were wrong.
Inspect metadata JSON and registry subject/referrers.Repeat cosign tree and imagetools inspection without signing.
ResolutionQuarantine/delete the staged release under policy, fix BuildKit/registry/exporter, and rebuild from the same clean commit; never bolt unverifiable evidence onto an ambiguous tag.
Security notes
- Do not expose registry auth through command argv or logs.
- Signing is intentionally not performed in the builder step.
Alternatives
- Use registry-native immutable staging repositories and promotion APIs when available, always preserving digest/evidence linkage.
Stop conditions
- Stop on build/input drift, mutable pull, unexpected platform, missing digest/attestation, registry rewrite, secret in metadata/provenance, or any policy failure.
verification
Extract the digest and inspect SPDX, provenance, and referrers
Parse the exporter metadata instead of console text, require an exact sha256 digest, construct one immutable repository reference, and inspect its index plus attached artifacts before signing. The gate must compare attestation subjects, platforms, source commit, builder, parameters, and materials with the release record.
Why this step matters
A structured digest extraction closes the gap between the image that was built and the image that is signed. Attestation presence is useful only when each statement describes the correct subject and expected build.
What to understand
Treat a missing, null, non-sha256, or wrong-repository digest as a hard failure. Do not resolve the tag again in a later job.
Inspect the OCI index and every platform manifest. The index digest may have attestation manifests associated differently by registry/BuildKit version; base policy on documented effective structure.
Parse SPDX for document namespace, creation info, package/file inventory, relationships, licenses, checksums, and the correct subject. Reject placeholder/empty inventories.
Parse provenance for subject digest, source URI/commit, builder ID, invocation parameters, resolved dependencies/materials, and completeness metadata. Define allowed builders and parameters explicitly.
Do not claim Build L2/L3 from predicate shape. Maintain a separate SLSA assessment referencing platform controls, signer, isolation, and change evidence.
System changes
- Reads metadata and registry manifests/referrers. It creates only short-lived shell variables and verification logs.
Syntax explained
jq -er- Extracts one required structured metadata value and fails on missing/null data.
sha256 regex- Prevents tags, truncated digests, whitespace, and unexpected algorithms entering the signing reference.
repository@$IMAGE_DIGEST- Binds registry namespace to exact image content.
cosign tree- Enumerates related signature/attestation artifacts for review; it does not verify trust by itself.
Values stay on this page and are never sent or saved.
IMAGE_DIGEST="$(jq -er '.[\"containerimage.digest\"]' {{metadataPath}})" && printf '%s' "$IMAGE_DIGEST" | grep -Eq '^sha256:[0-9a-f]{64}$' && IMAGE_REF="{{registryHost}}/{{imageRepository}}@$IMAGE_DIGEST" && docker buildx imagetools inspect "$IMAGE_REF" && cosign tree "$IMAGE_REF"immutable reference: {{registryHost}}/{{imageRepository}}@sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd
index platforms: linux/amd64, linux/arm64
SPDX subjects: match platform manifests
provenance subjects: match platform manifests
source revision: {{sourceCommit}}
builder/material/parameter policy: passed
unexpected referrers: none
release status: evidence verified; not yet signedCheckpoint: Verify this layer before continuing
Continue whenDigest matches exporter and registry, every approved platform exists, SPDX/provenance subjects and required fields match release intent, no unexpected artifact appears, and policy output is retained.
Stop whenStop on tag resolution, malformed/mismatched digest, missing platform, missing/empty attestation, wrong source/builder/material/parameter, secret metadata, or unexplained referrer.
If this step fails
The attestation exists but references a platform digest different from the image being promoted.
Likely causeThe verifier compared only the index, registry rewrote artifacts, or evidence from another build/tag was selected.
Enumerate the index and per-platform digests.Compare each attestation subject with exporter metadata and registry manifest graph.
ResolutionReject the release and rebuild/reattach only through the reviewed build process. Never edit an attestation's subject after the fact.
Security notes
- Do not execute the image to inspect evidence.
- Attestation content is untrusted input until identity and policy are verified.
Alternatives
- Use an admission/policy engine that evaluates OCI subjects and predicates consistently in CI and cluster.
Stop conditions
- Stop on tag resolution, malformed/mismatched digest, missing platform, missing/empty attestation, wrong source/builder/material/parameter, secret metadata, or unexplained referrer.
command
Generate and validate CycloneDX from the pushed digest
When consumers require CycloneDX, scan the same immutable pushed digest with a separately promoted, digest-pinned generator. The example runs Syft as an unprivileged container without a host socket; it pulls the registry artifact through its own narrow read-only credentials. Compare the result with native SPDX rather than assuming the formats are identical.
Why this step matters
BuildKit's native SBOM is SPDX. Generating CycloneDX independently from the pushed digest serves consumers that require CycloneDX and catches blind spots, while an immutable subject prevents accidentally scanning a moved tag.
What to understand
Provide only repository pull permission through the runner's protected credential mechanism. Do not mount docker.sock, a home directory, cloud credentials, source workspace, or SSH agent.
Use the invoking unprivileged host UID/GID so the scanner can write only the pre-created bounded output directory; retain read-only root, dropped capabilities, no-new-privileges, bounded scratch, registry-only network, and disposable execution.
Normalize and compare high-value runtime inventory—names, versions, package URLs, hashes, operating-system packages, application dependencies—between SPDX and CycloneDX. Formatting differences are expected; missing critical components are not.
System changes
- Pulls/scans the staged image, writes CycloneDX JSON under out, and uses narrow registry/network access. It does not execute the image entrypoint.
Syntax explained
--user $HOST_UID:$HOST_GID- Runs as the invoking unprivileged host identity so output is writable without world-writable permissions.
--cap-drop ALL- Removes Linux capabilities not required for registry/file analysis.
--read-only/--tmpfs- Keeps scanner root immutable and bounds scratch state.
repository@digest- Scans exactly the image already inspected, never a moved tag.
cyclonedx-json- Emits the requested machine-readable CycloneDX BOM.
Values stay on this page and are never sent or saved.
install -d -m 0750 out && HOST_UID="$(id -u)" && HOST_GID="$(id -g)" && test -w out && docker run --rm --user "$HOST_UID:$HOST_GID" --env HOME=/tmp --cap-drop ALL --security-opt no-new-privileges --network bridge --read-only --tmpfs /tmp:rw,noexec,nosuid,size=64m -v "$PWD/out:/out" {{cycloneDxImage}}@{{cycloneDxDigest}} registry:{{registryHost}}/{{imageRepository}}@"$(jq -er '.[\"containerimage.digest\"]' {{metadataPath}})" -o cyclonedx-json=/out/cyclonedx.json && test -O out/cyclonedx.json && jq -e '.bomFormat == "CycloneDX" and (.components | length > 0)' out/cyclonedx.json >/dev/nullscanner: {{cycloneDxImage}}@{{cycloneDxDigest}}
subject: {{registryHost}}/{{imageRepository}}@sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd
CycloneDX document: out/cyclonedx.json
bomFormat: CycloneDX
components: 47
schema validation: passed
SPDX/CycloneDX critical inventory comparison: passed
unexplained difference: 0Checkpoint: Verify this layer before continuing
Continue whenPinned scanner reads only the immutable subject, CycloneDX validates with nonempty inventory, high-value components agree with SPDX/runtime, and no credential or host socket enters the sandbox.
Stop whenStop if scanner is mutable/privileged, needs host socket/broad credentials, scans a tag/wrong platform, schema fails, inventory is empty, or a critical SPDX/CycloneDX difference is unexplained.
If this step fails
CycloneDX contains fewer application components than SPDX or the runtime inventory.
Likely causeScanner cataloger differs, wrong architecture was scanned, package metadata is absent, or file permissions prevented inspection.
Compare scanner logs/catalogers and exact subject/platform.Spot-check known runtime packages and package databases without running the image.
ResolutionCorrect scanner configuration/image metadata, document justified format differences, and block release until critical coverage is restored.
Security notes
- SBOM generators process hostile artifact data; sandbox them.
- Never grant push/delete permission to an inventory job.
Alternatives
- If CycloneDX is not an organizational requirement, retain validated native SPDX and avoid an unnecessary second scanner.
Stop conditions
- Stop if scanner is mutable/privileged, needs host socket/broad credentials, scans a tag/wrong platform, schema fails, inventory is empty, or a critical SPDX/CycloneDX difference is unexplained.
command
Sign the digest with protected keyless identity
After all evidence and content policies pass, a protected signing job obtains a short-lived OIDC identity and signs the exact immutable reference. The job must be restricted to the approved repository, workflow/ref/environment, require minimal id-token permission, and have no untrusted checkout after approval.
Why this step matters
Signing only after independent evidence checks turns the signer into an explicit promotion authority. Keyless OIDC avoids distributing a long-lived private key, while certificate and transparency evidence lets verifiers constrain the workflow identity.
What to understand
Place this step in a protected deployment/release environment with required approval and exact branch/tag/repository conditions. Do not expose id-token permission to general build/test jobs.
Transfer only the immutable reference and verified release evidence into the signing job. Do not check out or run untrusted source after the approval gate.
Grant registry permission to attach the signature to only the target repository and no deletion/admin access.
Never print ambient OIDC token/environment, enable shell tracing, or persist credential helpers in artifacts/cache.
If using KMS instead, configure Cosign with an immutable KMS key URI/version and short-lived workload identity. Validate public key fingerprint/audit events and never export private material.
System changes
- Requests a short-lived identity certificate, writes a signature artifact to the OCI registry/transparency system, and records promotion evidence.
Syntax explained
repository@$IMAGE_DIGEST- Signs exact content rather than whichever digest a tag resolves to later.
cosign sign- Creates a signature using ambient keyless identity in the protected job.
--yes- Allows noninteractive transparency-log acknowledgement; policy/approval happened before the command.
Values stay on this page and are never sent or saved.
IMAGE_DIGEST="$(jq -er '.[\"containerimage.digest\"]' {{metadataPath}})" && IMAGE_REF="{{registryHost}}/{{imageRepository}}@$IMAGE_DIGEST" && cosign sign --yes "$IMAGE_REF"subject: {{registryHost}}/{{imageRepository}}@sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd
signing mode: keyless OIDC
certificate identity: {{cosignIdentity}}
OIDC issuer: {{oidcIssuer}}
transparency log inclusion: recorded
private key stored in CI: no
tag signed as authority: noCheckpoint: Verify this layer before continuing
Continue whenOnly the protected job can sign, exact digest and intended identity/issuer are recorded, transparency evidence exists, registry contains the signature, and no long-lived private key or OIDC token is retained.
Stop whenStop if subject is a tag/malformed, OIDC permission exists on untrusted jobs, identity/ref/environment is unexpected, evidence policy did not pass, registry scope is broad, or transparency/KMS audit is absent.
If this step fails
Signature was created by a different workflow/ref identity than expected.
Likely causeThe signing job trigger/environment is too broad, reusable workflow identity changed, or wrong OIDC provider/job executed.
Do not deploy; run exact cosign verification and inspect certificate/transparency metadata.Audit workflow/ref approvals and id-token grants.
ResolutionQuarantine the release, remove/mark the signature according to registry policy, revoke/bound permissions, and sign only through the corrected reviewed identity after revalidation.
Security notes
- Signing authority is production authority even when it cannot deploy directly.
- A signature does not waive vulnerability or provenance failures.
Alternatives
- Use KMS/HSM signing with workload identity when keyless public identity/transparency is incompatible with policy.
Stop conditions
- Stop if subject is a tag/malformed, OIDC permission exists on untrusted jobs, identity/ref/environment is unexpected, evidence policy did not pass, registry scope is broad, or transparency/KMS audit is absent.
verification
Verify exact signer, evidence, and negative cases before deploy
Use an independent verifier to check the exact digest, certificate identity, and OIDC issuer, then re-evaluate SBOM/provenance and release policy. Run negative tests with a moved tag, unsigned digest, wrong identity, missing attestation, unexpected material, and disallowed vulnerability; every one must block.
Why this step matters
Verification must express who is allowed to sign what, not merely that some cryptographic signature exists. Rechecking attestations and content policy after signing prevents a signature from laundering incomplete or unsafe evidence.
What to understand
Run verification in a separate protected component/image from the builder and signer when possible. Pin the verifier and policy bundle, and protect policy changes with review.
Specify exact certificate identity and issuer. If a regular expression is necessary for supported workflow variations, anchor it tightly and test unrelated repository/ref identities are rejected.
Evaluate native BuildKit provenance as data: subject, source commit, builder ID, parameters, materials, completeness. Do not confuse cosign image signature with an independent signature over every predicate field unless your attestation workflow explicitly provides that.
Evaluate SPDX/CycloneDX, vulnerability, malware, license, secret, base-policy, and exception expiry. SBOM absence/ambiguity blocks rather than being interpreted as zero vulnerabilities.
Run negative fixtures continuously: unsigned digest, other repository signer, other ref/workflow, tag drift, missing predicate, wrong source/material, critical policy failure, and admission bypass.
System changes
- Reads registry/Sigstore evidence and policy data and emits a signed/controlled verification decision; it does not modify image content.
Syntax explained
--certificate-identity- Constrains the accepted keyless workload to the reviewed workflow identity.
--certificate-oidc-issuer- Constrains the identity provider that authenticated that workload.
repository@digest- Checks the exact release subject and digest claim.
negative matrix- Proves the verifier fails closed, not merely that the happy path succeeds.
Values stay on this page and are never sent or saved.
IMAGE_DIGEST="$(jq -er '.[\"containerimage.digest\"]' {{metadataPath}})" && IMAGE_REF="{{registryHost}}/{{imageRepository}}@$IMAGE_DIGEST" && cosign verify --certificate-identity '{{cosignIdentity}}' --certificate-oidc-issuer '{{oidcIssuer}}' "$IMAGE_REF" >/dev/null && printf '%s\n' 'signature=passed' 'identity=passed' 'issuer=passed' 'run-attestation-and-policy-verifier=required'signature cryptography: passed
certificate identity: {{cosignIdentity}}
OIDC issuer: {{oidcIssuer}}
transparency verification: passed
subject digest: exact match
SPDX/CycloneDX policy: passed
provenance subject/source/builder/materials: passed
vulnerability/license/secret policy: passed
negative test matrix: 6 of 6 blocked
admission dry run: passedCheckpoint: Verify this layer before continuing
Continue whenExact signature identity/issuer/digest, transparency, attestation subject/content, vulnerability/license/secret rules, exception expiry, and every negative test pass; admission dry-run accepts only this digest.
Stop whenStop on any verification/service ambiguity, broad identity rule, missing evidence, wrong source/material/builder, unsigned/moved tag acceptance, critical content policy failure, or negative test success.
If this step fails
Cosign verify succeeds but admission rejects the image.
Likely causeAdmission uses different trust roots, identity syntax, policy bundle, registry mirror/digest, or cannot fetch required referrers.
Compare exact digest and policy revision in CI and admission.Run admission dry-run with sanitized decision logs.
ResolutionAlign pinned trust/policy bundles through review, restore referrer availability, and repeat all negative tests; never bypass production admission to match CI.
Security notes
- Verification output and policy exceptions are security evidence; restrict mutation.
- Do not accept identity wildcards spanning unrelated repositories.
Alternatives
- Use Sigstore policy-controller, Kyverno, Gatekeeper, or registry-native admission where appropriate, with equivalent exact policies.
Stop conditions
- Stop on any verification/service ambiguity, broad identity rule, missing evidence, wrong source/material/builder, unsigned/moved tag acceptance, critical content policy failure, or negative test success.
warning
Promote, deploy, retain evidence, and recover by digest
Promote only the verified digest without rebuilding, deploy repository@digest through enforced admission, verify runtime health and identity, and record a previously verified rollback digest. On compromise, stop signing/deployment, quarantine subjects/cache/builders, rotate identities, and rebuild rather than retagging or trusting the nearest signed image.
Why this step matters
The value of supply-chain evidence is realized only when deployment consumes the same digest under an enforcing policy. Recovery likewise needs an independently verified known-good digest and must replace compromised builder/signing state rather than applying a reassuring tag.
What to understand
Use registry-side copy/promotion that preserves the manifest digest and related signatures/attestations, or verify the destination digest/evidence again. Never rebuild for promotion.
Commit deployment manifests with repository@sha256 and optional tag/comment for humans. Admission independently verifies accepted identity, issuer/key, attestations, vulnerability policy, and exception expiry.
Run post-deploy verification: workload reports expected version/commit, orchestrator imageID matches release digest, health/SLO checks pass, and no mutable tag is resolved by the runtime.
Retain a release ledger: source commit, repository/digests per platform, base/tool/action digests, SBOM/provenance and their hashes, signature/certificate/transparency/KMS evidence, scans, policy revision/results, approvals, deployment, and rollback.
For compromise, freeze signing/promotion, isolate builder and affected artifacts, preserve external audit evidence, rotate runner/registry/OIDC/KMS permissions as applicable, quarantine cache, assess signed release scope, and rebuild from trusted inputs.
Before rollback, verify signature, SBOM/provenance, vulnerability/incident scope, configuration/data compatibility, and health in staging. A valid older signature alone is insufficient.
System changes
- Copies/promotes registry artifacts, changes deployment manifests/workloads, writes durable release evidence, and may delete/dispose builder/cache state. Rollback changes production.
Syntax explained
promote exact digest- Moves approved content without rebuilding or resolving a tag.
deploy exact digest- Ensures runtime consumes the verified subject.
admission enforced- Prevents deployment outside the signed evidence policy.
rollback verified digest- Returns only to evidence-backed content known outside current incident scope.
printf '%s\n' 'promote exact repository@digest' 'deploy exact repository@digest' 'verify runtime and admission decision' 'retain signature SBOM provenance policy approvals' 'rollback only to independently verified digest'promotion source digest: sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd production digest: identical deployment manifest uses tag: no admission signature/evidence policy: enforced runtime health and version endpoint: passed release evidence retention: confirmed rollback digest: previously verified and incident-safe builder/cache after release: destroyed or trust-scoped
Checkpoint: Verify this layer before continuing
Continue whenDestination digest/evidence match, deployment and runtime imageID equal the verified digest, admission cannot be bypassed, health passes, evidence is immutable/retained, and rollback/recovery drills succeed.
Stop whenStop/rollback if promotion changes digest or loses evidence, manifest/runtime uses a tag/different digest, admission is bypassed, health fails, evidence is mutable/missing, or rollback digest intersects incident scope.
If this step fails
Production runs a different digest from the release ledger after a successful pipeline.
Likely causePromotion rebuilt/rewrote the image, deployment used a tag, registry mirror changed identity, or admission was bypassed.
Freeze rollout and compare source/destination/deployment/runtime digests.Inspect registry and admission audit records without restarting compromised workloads.
ResolutionRestore the exact approved digest or a verified rollback, re-enable admission, investigate the divergence, rotate compromised authority, and run the full release process again.
Security notes
- Treat emergency admission bypass as a security incident with explicit owner/expiry/audit.
- Do not delete evidence needed to scope signed compromised releases.
Alternatives
- Blue/green or canary deployment reduces rollback pressure but does not relax digest/evidence policy.
Stop conditions
- Stop/rollback if promotion changes digest or loses evidence, manifest/runtime uses a tag/different digest, admission is bypassed, health fails, evidence is mutable/missing, or rollback digest intersects incident scope.
Finish line
Verification checklist
jq -er '.["containerimage.digest"]' {{metadataPath}}Returns one sha256 digest matching the registry index and every later sign/verify/deploy record.IMAGE_REF='{{registryHost}}/{{imageRepository}}@sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd'; docker buildx imagetools inspect "$IMAGE_REF"; cosign tree "$IMAGE_REF"Expected SPDX and SLSA provenance artifacts exist for the exact subject/platforms and no unexplained referrer is present.jq -e '.bomFormat == "CycloneDX" and (.components | length > 0)' out/cyclonedx.jsonCycloneDX validates, describes the exact pushed digest, and agrees with SPDX/runtime on critical components.cosign verify --certificate-identity '{{cosignIdentity}}' --certificate-oidc-issuer '{{oidcIssuer}}' '{{registryHost}}/{{imageRepository}}@sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd' >/dev/nullSignature, exact certificate identity, issuer, transparency evidence, and digest claim pass.printf '%s\n' 'unsigned denied' 'wrong identity denied' 'tag denied' 'missing provenance denied' 'critical policy failure denied' 'approved digest accepted'Admission accepts only the verified digest and every negative fixture fails closed.printf '%s\n' 'signing kill switch tested' 'builder/cache quarantine tested' 'credentials rotated' 'verified rollback digest staged'Operators can freeze authority, replace build/sign state, scope evidence, and recover with an independently verified unaffected digest.Recovery guidance
Common problems and safe checks
A secret value appears in provenance, image history, a layer, or logs.
Likely causeThe secret was passed as ARG/ENV, copied into the build context, echoed by a RUN command, stored by a package manager, or included in mode=max provenance build arguments.
Cancel publication, restrict logs/attestations, scan the Dockerfile, .dockerignore, build arguments, history, exported filesystem, and BuildKit secret mounts from a trusted workstation.Identify every credential and downstream system exposed; do not repeat the value in tickets or test commands.
ResolutionRevoke and rotate the credential, delete or quarantine affected artifacts according to registry policy, remove it from context/ARG/ENV, use a BuildKit secret or SSH mount, rebuild without cache from a clean commit, and re-run leak tests.
The published image has no SBOM or provenance attestation.
Likely causeThe build used --load or a local image store that does not preserve attestations, the flags were omitted, the registry rejected OCI referrers/attestation manifests, or the selected exporter cannot attach them.
Inspect build metadata and the pushed registry reference with docker buildx imagetools inspect --format and cosign tree.Confirm --push, --sbom=true, --provenance=mode=max, BuildKit version, exporter, and registry OCI artifact support.
ResolutionKeep the image in a nondeployable staging repository, fix the exporter/registry path, rebuild and push from the same reviewed commit, and require both attestation types before signing or promotion.
Cosign keyless signing cannot obtain an identity token.
Likely causeThe CI job lacks id-token permission, the provider audience/issuer is wrong, the job is not on a protected ref/environment, or outbound access to Fulcio/Rekor is unavailable.
Inspect only token-availability metadata, job permissions, protected-environment approval, OIDC issuer/audience configuration, DNS/TLS, and Sigstore service status.Never print the OIDC token, Authorization header, ambient cloud credentials, or signing challenge.
ResolutionRestore the least-privilege OIDC permission and approved network path on the protected signing job. Do not fall back to an unprotected long-lived private key in CI.
Cosign signs successfully but verification rejects certificate identity or issuer.
Likely causeThe signing workflow/ref/repository identity differs from policy, the wrong OIDC issuer was expected, a reusable workflow changes the certificate subject, or the verifier is checking a different digest.
Run cosign verify against the exact repository@digest with explicit certificate-identity and certificate-oidc-issuer values.Inspect the returned certificate and transparency-log metadata, then compare it with the approved workflow identity pattern.
ResolutionCorrect the signing workflow or exact verifier policy through review. Never broaden identity matching to a repository-wide wildcard simply to accept an unexplained certificate.
Provenance exists but policy cannot match its subject, materials, or builder.
Likely causeThe attestation describes a different image/platform, source metadata was unavailable, untrusted build arguments changed, the builder identity is not allowlisted, or the policy incorrectly assumes a SLSA level.
Extract the attestation predicate without executing the image and compare subject digest, source repository/commit, parameters, materials, and builder ID with the release record.Check the documented BuildKit provenance mode and limitations before assigning any SLSA claim.
ResolutionReject promotion, rebuild from the reviewed commit on the approved builder, and fix the predicate policy. Describe evidence precisely; emitting SLSA-shaped provenance alone does not establish SLSA Build L2 or L3.
A deployment resolves a tag to a digest that was never verified.
Likely causeThe deployment manifest uses a mutable tag, promotion retagged an image after verification, the registry was rewritten, or admission policy is absent/bypassed.
Resolve the production reference, compare its digest with the signed release record, and verify signature plus attestations before changing anything.Inspect deployment history and admission-policy decisions for tag mutation or emergency bypass.
ResolutionStop rollout, pin the approved repository@digest, restore admission enforcement, and investigate tag movement. A human-readable tag may annotate a release but must not be the deployment identity.
Reference
Frequently asked questions
Does an SBOM prove that a container image is secure?
No. An SBOM is an inventory that improves review, vulnerability correlation, licensing analysis, and incident response. It can be incomplete, stale, or ambiguous, and it does not prove that components are configured safely or free from malicious behavior. Validate coverage, retain provenance, scan with current intelligence, and keep runtime controls.
Why must deployment use an image digest when the release tag is already protected?
A tag is a mutable name, while a registry digest identifies the exact image manifest selected by the release gate. Sign, attest, verify, and deploy the digest so every stage refers to the same artifact. Keep the human-friendly tag as metadata, not as the production trust anchor.
Is a valid Cosign signature enough to authorize production deployment?
No. Verification must also constrain the expected signer identity and issuer or key version, match the exact image digest, validate required SBOM and provenance attestations, and apply vulnerability and policy gates. A cryptographically valid signature from an unexpected identity is not an approved release.
Can BuildKit secrets safely be passed through Dockerfile ARG or ENV?
No. Build arguments, environment values, history, layers, logs, cache metadata, and provenance can expose them. Use BuildKit secret or SSH mounts, keep secret files out of the build context, prevent commands from echoing or copying them, and include negative leakage tests before release.
Recovery
Rollback
Stop rollout and signing first. Preserve external registry, CI, OIDC/KMS, transparency, admission, and deployment evidence. Roll back only to a previously signed, attested, policy-compliant digest shown to be outside the incident and compatible with current data/configuration; never select the nearest tag.
- Disable protected signing/promotion and block new deployment admissions except the explicit incident path; isolate the builder and suspect cache/artifacts.
- Compare failed production imageID, release ledger, registry index, signatures, SPDX/CycloneDX, provenance, policy revision, source commit, and materials to scope affected releases.
- Choose a known-good repository@digest and independently re-run exact identity, transparency/KMS, attestation, vulnerability/license/secret, compatibility, and incident-indicator checks.
- Canary the rollback digest with production-like configuration/data compatibility, then deploy by digest under the normal admission policy and verify runtime imageID plus health.
- Rotate runner/registry/OIDC/KMS authority as indicated, rebuild disposable workers from verified images, quarantine cache, and invalidate any compromised signing path.
- Publish a corrected release through the complete build-SBOM-provenance-sign-verify pipeline; retain incident and superseded-release evidence according to policy.
Evidence